To have the nerve to do something can either mean that you are particularly brave, or that you are partcularly rude.
In this context, Mr Dursley feels that the person wearing an emerald green cloak is showing a degree of rudeness AND courage in wearing an emerald green cloak unsuited to his age.