He could be angry with Tomeee, or he could have simply decided that Tomeee's opinion doesn't matter.
It's mildly rude, but few people I know are offended by the word "hell." What's more impolite about it is the way you discounting another person or his ideas in a very high-handed manner.
A: Are you ready to go to lunch?
B: Wait, Barbara wanted to come too.
A: Oh, to hell with Barbara. Let's go. I'm hungry now.
Yes, that would be considered quite rude!