Even though the form of the past subjunctive is the same as the form of the simple past, you can distinguish the two by their function:
1. He went to London.
2. If he went to London, he would...
The "went" in #2 has nothing to do with the past in terms of time. (Though you might say that the "wents" in #1 and #2 have "distance" or "remoteness" in common.)
I would also say that tense is largely irrelevant, in the subjunctive mood. How can we talk about "pastness", in the context of the potential or the unreal? The subjunctive inhabits "no-time".
In German, for instance, one of the roles of the present subjunctive is indirect speech, while the past subjunctive is used for contrary-to-fact statements. They are the "present" and "past" subjunctives not by virtue of their relationship to each other, but by their respective morphological relationships with the present and past indicative.
MrP