I presume he and you have not filed an affidavit of paternity where he and you agree he is the father, legally, either.
Personally, I would play a bit of hardball. Tell him that if he wants any visitation that he needs to complete the establishment of paternity through the courts. Without that he actually has absolutely no legal right to demand any visitation. He also is allowed to ignore his obligation to provide support for the child.
I was watching a people’s court show today. The judge was actually chastising a mother for not stepping up and seeking child support from the father of her child. Her statement made a very strong point
The mother is seeking what the child has a right to; that both parents provide support for the child and it is the mothers responsibility to represent the child’s best interest in seeking the father pay his share.
Stand up for your child’s best interest and get this taken care of through the courts.
I would tell him
No legal paternity, NO visitation.