A: How do you like your coffee?
B: I'm not a coffee drinker.

Does it mean B doesn't drink coffee or rarely?
He doesn't drink coffee.

I'm not much of a coffee drinker - only rarely
I'm not a big coffee drinker. - I don't drink much coffee - so the implication is rarely.

I'm not a big cigar smoker. - I don't smoke many cigars - again, implication is rarely.
Teachers: We supply a list of EFL job vacancies
I believe the same logic applies to "I'm not a cigar smoker". But I remember a Hollywood star said that when he meant he rarely smoked cigars. I could have misunderstood him. Please confirm. Thank you.
New2grammarhe rarely smoked cigars
It means that he seldom smoked cigars.
Site Hint: Check out our list of pronunciation videos.
 Optilang's reply was promoted to an answer.