I am in this gray area with a loose cannon husband, and I sympathize with how painful it can be.
We now have a visitation schedule that has been agreed upon by us via our lawyers. But prior to this, since the girls are very young, live with me, and I am and have always been the primary caretaker, it was my understanding that I would not be viewed very favorably by the court if I allowed visitation that was harmful for the children. Also that being their primary caretaker, I was in the best position to decide what was in their best interest. Now, if I had forced that at any point I could have been slated as denying their father access to the kids. Luckily I didn't have to do that so much (exception was the era of the TRO) but I was not going to let his apparent attempts to slate me as interfering with his access to them put them in a bad situation. In my case, the fear is of him falling asleep while they're in his care, leading me to steer the visits toward earlier in the day and not do overnights until he tries to get to the bottom of his sleep issues. So my actions have to match my concern here. I believe that is important both to establish status quo and so that the issue can be addressed.
Even with the temporary agreement involving lawyers, he made off with our daughter for a bit. Luckily by then his lawyer was well involved and talked him down from that ledge. Mind you, he sees them 6 days a week. He just can't deal with structure and is impulsive. He fails to realize that having this very liberal parenting schedule ensures him regular time with them. He just fixates on the part where he can't make plans with them during my time, and rebels against that like a child. Anyway...
If I had to do this over again I think I would have tried to get temporary orders a LOT sooner. Good luck to you.