Humidity. Oh, GOD, the humidity kills me! I have friends from San Diego County that now live in GA and they love it. But! They DO tell me the summers are difficult.
One of my sons is considering South Carolina for a permanent spot, but he's still in the USMC so will not settle for a long time, yet.
Our extended family is in CA, TX, and PA, for the most part. Our children will be spread out throughout CA. My wife, being from Philly originally, is partial to the northeast, but the appeal of the southeast is the history there - we are both into Civil War reenacting and have enjoyed our occasional sojourns there.
But, the southeast is not off the list ... it's just not yet on MY shortlist - solely because of that humidity.