What is the meaning of I don't hang around in the following text?

"Those folks any happier over in Bowmore?"

"Damned if I know . I don't hang around."



I can't be sure with no context, but it sounds like when he does go to Bowmore, he only stays long enough to do what he went there to do and then leaves.