Just a few thoughts:
I don't think most native speakers would ever actually think "I can understand what he said, but I don't speak like that". However, if your English is quite broken and/or hard to understand, native speakers might think to themselves (or comment to others) "Man! He's...