I've lived and worked a lot of places in Thailand over 12 years. Phuket a few times. In Patong as a bit of a delinquent years ago and then in Phuket Town as more of a eat/sleep/work kind of life. In my experience yeh I find people in Patong less desirable and just out to get money or sometimes rude and abrupt. However, living at Phuket Town I rarely experienced it. I eat out every meal and always go out doing stuff. It must have to do with your habits and what you do where you go and who you interact with as I find people pretty normal.
Pattaya I felt was the rudest most unfriendly locals.