.
The indefinite article certainly helps in the determination, though I doubt that it is much of a deciding point. In any case, your clause is most likely restrictive (defining) because the gentleman in this isolated sentence has no definition otherwise-- it is as simple as that.
Even with 'the', it could just as well be a defining clause in this isolated situation, as no other information exists about the gentleman or the situation.
A gentleman whom I had never seen before smiled at me. (completely new information)
The gentleman whom I had never seen before smiled at me. (There were 2 gentlemen-- one I knew and one I didn't know.)
A gentleman, whom (by the way) I had never seen before, smiled at me.
The gentleman (that I mentioned before), whom I had never seen before, smiled at me.In this series, the first, restrictive (defining), case is the most likely for the indefinite article and the last, non-restrictive (non-defining), case is the most likely for the definite article.
All in all, not a good quiz question.
.