About the expression "...what was the matter" or "what the matter was"

Please look at the following question:
My mother asked___ with me.
A. what was the wrong     B. what the matter was    C. what matter was           D. what was the matter

"D" is right because in spoken language poeple often say "what's the matter" and it is analysed that here "what" is a subject.

And in the sentence" He asked what the matter was" the "matter" means substance. Right?

But I noticed a sentence "He came in and asked what the matter was with us." Is it right?  Thanks.

The exercise sentence is in reported speech, also called 'indirect speech/question'.  Therefore, the correct answer is B: 'My mother asked what the matter was with me'.  The direct question could have been: 'What is the matter with you?'. 

Your last example is fine.