I think one of the meanings of these phrases is "the first item of a multi-item list". The other meaning does not seem to be related to multi-item list. For example,
How did I get infected in the first place?
The meaning here is somewhat like 'initially' or 'in the beginning'. With this meaning, can I still interchange these phrases?
'How did I get infected to start with?' or 'To start with, how did I get infected?'