You state the father is "around"; does that mean he signed the birth certificate or paternity papers at the time of birth so that paternity is established?
You should probably allow him limited and supervised visitation with the child at your home only for the time being. If he wants more, he'll have to petition the court for visitation rights. They'll decide the amount and conditions of future visitation. Until he does that, I wouldn't allow him to take the child from your home.