You don't care for visiting me. And

You don't care to visiting me.

Are both useable?

Thank you!
You don't care for visiting me.= 'Generally you don't like doing it.'
You don't care to visit me = 'You don't want to.' Your original was not possible grammar.
Your first sentence is in the present progressive tense. You are talking about an event that has happened in the past and will probably happen in the future so the -ing suffix is correct. However, "You don't care to visiting me" is in the present (or possibly future, depending on the context of the sentence) tense and therefore needs to have a matching verb. The sentence should read: "You don't care to visit me."
 Mister Micawber's reply was promoted to an answer.