Well yes, male refers to men; the male of any species. But also it can refer to the pokey bit of an electrical plug e.g. headphone plugs have a 'male' jack which plugs into a corresponding 'female' hole... lol ;)
Why can't you label a door meant for men as male? I guess the door isn't male ..is it? haha