Are you saying that you have physical custody in a legal sense, as in you have been awarded physical custody through the courts, or that you have physical custody because the kids live with you right now and there IS no court ordered custody? Big difference.
As far as the kids leaving the state, without the consent of the other parent, is he objecting to them going, or simply doesn't know?
If he is objecting, you may want to wait until you have court ordered custody of some sort, and determine if it is within the bounds of your custody, before you send them. If he simply doesn't know, and you are not attempting to interfere with his visitation or relationship with the children in any way, then you could probably send them so long as you have them back in a timely manner that will not interfere with him having visitation.
In the eyes of the court, it's all about the parents cooperating with one another to ensure that the children are allowed to develop a relationship with both parents.
Good luck!