(1) "Did you try calling him? -- Yes, I did. But he wasn't answering."
(2) "Did you try to call him? -- Yes, I did. But he wasn't answering."
(2) should be right, in my opinion. However, what is right and why the wrong sentence is wrong?

Thanks in advance.
Yes, I did. But he didn't answer.
I think the acknowledgement of an American native speaker should suffice, shouldn't it?
Thanks.
The reason is that if you tried calling him, you did actually call him (to see if something could be solved by the call). You could say, "Did you try calling him? -- Yes, I did. But he wasn't very helpful."
(There is also the slight suggestion that several tries may be necessary.)
The "try to" form is used when the speaker is talking about an actual attempt, pure and simple.
(There is also the slight suggestion that only one try is meant.)
Did you try calling him? ~ I think that calling him would have been a good idea. Did you do that?
Did you try to call him? ~ Did you make an attempt to call him?
___________
He wasn't answering implies "over a period of time", therefore probably consisting of several tries.
He didn't answer typically implies one try.
The differences are subtle. Others may have different views on the subject.
CJ
Even supposing that's a typo, I'm lost.
-- Did you try calling him?
-- Yes, but he didn't answer.
Sounds normal to me.
CJ
In the sentence we are discussing, the difference is quite subtle, and the fact that the speaker is responding to a question does not help. The speaker may be responding to the question s/he thought she heard. Had there been no previous question, I (and I can speak only for myself) would have said, depending on what had happened:
I tried to call him, but he didn't answer.
or:
I tried calling him, but he wasn't very helpful.
(Sorry - I am repeating myself. However, we can compare this woth what follows.)
In the situation below, the difference is clearer:
The room was hot. I tried to open the winow,, but it was stuck.
The room was hot, I tried opening the window, but it made no difference.
I'm not sure I always use "try -ing" that way, but maybe I should!
CJ