Note the difference between asking a question and asking someone to do something.
Asking a question is asking for an answer.
Examples: Are you taller than six feet? What is your name? Is that a black eye?
Asking someone to do something is not just asking for an answer. It's expecting the other person to behave in a certain way, to accomplish some task.
Examples: Can you wash the dishes? Won't you please mow the lawn? I'd like you to prepare a report on the election results. Could you buy me a diamond ring?
"You're asking too much of me" is only possible as a response to the second kind of asking.
If I ask you, "What color is the sky?", for example, you would not answer, "You're asking too much of me" because I have not asked you to do anything (except answer the question).
CJ