Go to the doctor. Or

Go to the doctor's.
The first is ok, the second needs 'office' at the end.
Yes, I'd say it can.

And the other odd thing about this is that we use "the" even though we don't even know which doctor we might see.
Teachers: We supply a list of EFL job vacancies

The construction you have in mind is usually used with shops or people's homes. I'm not sure about AmE, but it is used in BrE.

I bought it at the baker's.
I picked it up at the butcher's.

I saw her at Jack's.

I don't think it can be used with 'doctor', but it takes a native speaker to be sure. Emotion: big smile

 BarbaraPA's reply was promoted to an answer.
Students: We have free audio pronunciation exercises.
Thanks GG,

I didn't know it went also with 'doctor'. I wasn't even sure it was used in AmE. Emotion: smile