Huntersmom,
I think the point was that too often people get caught up in egos instead of thinking about what's best for their child. You didn't mention any abuse from the child's father initially and unfortunately, some women don't seem to care that in most circumstances it's in the best interests of their child that the child have as much contact with the father as possible. The courts almost always favor the mother and if we're honest, we know this isn't always in the child's best interest. I believe this is was he was trying to point out to you. Sometimes the father is the best caretaker.
If he is abusive, you need to gather as much info regarding his behavior as you can. Write things down as soon as they happen noting times, dates etc and try to have people you trust with you as much as possible so they can witness the behaviors you're concerned about.
Finally, I don't think you have to worry about him getting full custody unless he can show somehow that you're unfit. I was threatened with this by my kids' father and even though I knew it could never happen, it still put fear in me just hearing him say it, so I know how you feel.
The best advice you can receive is to always have the best interest of your child in mind. Always. And if it means putting your ego aside sometimes, that's what you need to do.
Good luck to you. Hope this helped a little.