Wrong can be used in either context and your meaning will be clear. I wouldn't worry about the usage unless you're planning on writing a novel or technical manual. Wrong and incorrect are pretty much interchangeable, you will not sound rude.
I'll try to explain why a native English speaker would prefer one over the other. The explanation for this can get a bit complicated. It has to do with the word's traditional usage. Style and history. A native English speaker will have picked it up through hearing it spoken or written in this style, even if they haven't studied entymology (history of words in language).
Wrong is from Old English (about 600 years ago) it is used in many classical literatures with stories that contain morale lessons. During the evolution of modern English (200 years ago) it gained more general use.
Since "wrong" is less formal, you will encounter it in everyday speech more often. It is perfectly ok to say:
"My address is wrong."
"I called the wrong number."
"The answer is wrong."
"You are going the wrong way."
Incorrect is slightly more formal because of its Latin origin. Incorrectus - in (not), correctus (correct). You will see it used in legal, business, or science documents. It is generally used more by people who are well-educated or who have studied etymology.
Here are some examples of sentences where an English speaker would choose to use incorrect:
"This is the result of incorrect gamma interpretation." - Technical manual
"Man torches woman's clinic on the incorrect assumption that they perform abortions." - Newspaper headline
"It purports to debunk what it describes as the historically incorrect myths propagated by revisionist historians." - Book review
"Most of it has turned out to be incorrect," a diplomat at the IAEA with detailed knowledge of the agency's investigations said." -Newspaper article
If you are ever unsure about a word's usage you can get clues from an etymology dictionary (there are a few online). Although, really, this level of debate on a word usage is considered university level. Native English speakers will hear it and tell you they think it sounds better but unless they have studied etymology or have a degree in English they may not be able to tell you why.