My wife lived in Phitsanulok for 25 years but we when we moved from Samui we came up here in the mountains. I have never lived in the city but I do hate driving there. The sois seem to be much smaller than, say Phetchbun, and the traffic more congested.
Apart from that we do not mind the place and the shopping facilities are there for most things. There is a choice of private hospitals and some are not expensive.
The traveling out of the country is really a hassle as you say, having to fly first to DMK and out with Air Asia or Nok Scoot, or bus to BKK for the other airlines. They are doing a lot of work at the airport terminal so maybe there could be some international flights in the future.
When we first came here we had to use the Phitsanulok immigration (now Phetchabun) and found them very helpful (compared to Samui).