“Face to face” is one of those phrases that sometimes confuse people because of it’s “made up” nature, like “one on one”.
I may not be completely correct about the hyphens but this is what I understand.
When the phrase is used as adjective, hyphen is supposed to be used.
When it’s used as adverbial phrase, hyphen is not needed. Other experts may correct me if this understanding is wrong.
Paul and Mary had a face-to-face discussion. (Used as adjective)
Paul and Mary had long discussion face to face (used as adverb)