You do not have to allow any visitation unless court ordered. He can not involve the police unless there is a court order in place and then most of the time they will not get involved anyway. However, if he turns up at the baby sitter and takes the child then you also will have problems getting the child via the police.
------------------
Psst... I do not work for Macdonald’s or Burger King, and even if I did, I would not tell you. For sloppy bread, tired tomatoes, frozen onions, watered down mayonnaise and imitation meat, please find a drive through window with a person who openly admits they make that stuff.
My advice above is equal to the advice they would give if you asked that person a legal question.