1. I talked to Joe on the phone while playing tennis. -- Here,
"while" is a conjunction, I think, and what comes after should be a
clause, but why "playing tennis"? It doesn't look like a clause. -- 'Playing tennis' is a non-finite clause: I talked to Joe on the phone while [I was] playing tennis. 'I' because the unspoken subject of the nonfinite clause should be the same as the subject of the main clause.
2. I talked to Joe on the phone while I was playing tennis. -- Here,
an appropriate clause seems to follow the conjunction "while." Good
sentence -- Yes.
3. I talked to Joe on the phone while Jim (or I) playing tennis --
What is wrong is this? Is "playing tennis" a noun phrase? I think so. -- No. The sentence is incorrect. You have added the subject (Jim, I), which now demands a finite verb: while Jim was playing tennis.