Please wait a moment. Most people here are very big on the idea that parents are allowed to "transfer" their parenting time to other people. That is absolutely not correct. Parents are allowed to share their parenting time with their extended families, but they can't go out of town for extended periods of time, and leave the kids with a babysitter rather than leaving the children with the other parent, whether ROFR is in effect or not. His new wife's parents would be babysitters. His parents, would not.
Believe me, if you kept the children with you rather than with someone like the new wife's parents, the judge would NOT be on dad's side if he tried to take it to court for contempt. In fact, the judge would wonder why dad didn't talk to you ahead of time and arrange a trade in the summer vacation schedule.
However, if he had planned for his parents to have that time with the kids, particularly if they didn't see them often, then you would be the bad guy.....even if you had an ROFR in place.