Single Mother with Custody Concern Question?

I am a single mother. My daughter is 7 years old and he father had just been issued to pay child support the past year. I made the mistake of the droping the case and had a notorized legal contract made up and now he is not abiding by the guidelines. My question is before I go any futher with legal actions, if he has never been a part of her life & had not paid child support prior & I have not stated as the father on the birth certificate if he attempted to get any custody would he succeed. Knows and admits to being her father but only paid Child Support because he was ordered by the DA's. He also has a Drug criminal record. Any advice is appreciated. He did threaten to get a lawyer and go for %50 custody if I did not drop the DA charges. I don't think he would be able to get any type of custody but would like to know what my chances are if I am to persew any further. I live in San Bernardino County CA.