Talk with an attorney before making any big moves. For example, if you deny him access to the daughter, then it could be used against you in a custody determination.
The ideal situation is to file for divorce, and then request a temporary custody and visitation determination from the court.
Make sure you follow all the court rules wrt money, etc.