Nope. I would say that your ex is going to be very unhappy at having his relationship with his child stopped dead and reduced to mere vacation periods. Since he has the child so much, he will be able to bring a very good case at why you should not be allowed to relocate with the child.. and could, because of the time he does spends with the child, file for a change of physical custody so as to keep the child in CA.
Depending on the judge, he may decide that it is your decision to move, so you should be the one to suffer, not your ex. At best, (if you ex does fight this with a decent attorney) expect to have to change the visitation so that your ex get the child for much longer periods and most of the summer AND for you to have to pay all travel costs assocaited with visitation.
Relocating is easy when the NCP has little contact with the child.. in your case, it is going to be difficult and costly.. and may result in you losing physical custody.
Before you panic, this is worse case so be prepared ! If your ex does not know the law or gets a bad attorney then you will be fine.. if he gets a good attorney then you may have to stay in CA if you want to keep physical custody.
Remember, no one is stopping YOU from relocating. You are free to locate anywhere you want.. they can stop you from taking the child though, that is the issue.