"John and me went to town" may be common, but I would consider it incorrect usage. Many native speakers use it, but I would definitely not advise an English learner to adopt it. It's not based on "shaky grounds" but on the difference between grammatical subjects and grammatical objects.
...