Hi Philip,
I love your explanation. Can it be also explained as "no offense"?
Your key point is "it isn't directed you." But in actuality, is it usually the direct opposite? That's, the speaker does indeed mean to direct to the intended listener. That's, the nothing personal is often meant to be personal. Please correct me if you think otherwise.