I agree with you completely, Clive.
Actually the analysis in terms of whether it's polite or impolite doesn't have much appeal to me. It can be either, depending on the context, IMO.
I think your rule of thumb, or subconscious knowledge, really works, and the politeness is indeed a secondary...