If he threatened to hurt you (physically) and you feel as if he is serious about that....how come you haven't filed a police report???
And yes, you REALLY need to settle down and breath. If/When he petitions the court for visitation or a parenting plan, you will have the ability (and hopefully a family law attorney) to provide your side of things.
It's obvious (to me) that you are a first time mom, and probably rather young (I am 40+, so anyone under 35 is rather young!
)
But really, people go through this sort of thing all the time--and by and large it is usually handled well by the courts.
Good Luck....you've got at least 18 years to co-parent with this man. How good those 18 years are, are up to you.