Hi, equivocal,
I think Mr. P. has a different reading of "I might have gone sailing" than you do.
You are thinking of "may", "might", and "could" as synonyms for "it is possible that". So your reading of it is something like this:
I to go sailing". In other words, he sees "I might have gone...