I agree with (most of) the others...go. I get technically Friday (last day of month) is the start of visitation. However, I agree it would help you IF dad took you to court for this, that IF you have documented other times when a similar weekend was interpreted differently...and the fact that the 1st and 2nd is sat and sun and thus in many peoples books, the first weekend of the month, AND its mothers day. And, the fact dad has known about this a long time and just now raising problems...its not the law but judges are human (to some degree) and dad will NOT look good trying to deprive kids of this. I hope you go, relax, have a good time.
Make sure beforehand you email him, let him know your plans (not enough details to stalk you in any way or prevent the kids), and proceed.
I was convinced when enough seniors spelled out what would likely happen in court IF dad takes you, and sounds like he won't.