Given that the party is over and that you met the person a couple of days later, then we must use sentence 1, as taiwandave said, with or without inversion.
Had I known your telephone number, I would have invited you to the party.
OR
If I had known your telephone number, I would have invited you...