While you are not sure if the harrassment is illegal or not, and while you are waiting for the company to do something, what you can do is go back to work, resting in the assurance that if he fires you, you will be able to draw unemployment. He probably does not have the power to fire you. So he can't threaten that.
So the harrassment he is doing is probably verbal abuse. If he yells at you, screams at you, accuses you of being lazy, stealing, etc. don't pay any attention. it's not true, his saying it won't make it true, and you don't have to defend your honor or set him straight. Do keep HR informed of each occasion. It would be good to take notes of each time this happens.
If he curses you, tell him quietly that you don't find it acceptable for someone to talk to you that way. Then, immediately, report this incident to your HR department. Tell them again that you do not find this acceptable. Make it clear to them that you are going to press charges against him if he assaults you physically in any way. Tell them clearly but politely that it is their responsibility to make this behavior stop. WHATEVER you do, do not curse or yell back at him or argue with him even though what he is saying is not the truth (Like you deliberately screwed this up, you're no good, your mamma's a bad person).
" It was so bad he almost got physically hurt" In other words, he made an employee so mad they probably took a swing at him and he got to fire them. if you retaliate physically, you'll be fired for good cause. If you quit, that's fine with them, because it was your choice to quit. If you are going to quit, wait till you have another job lined up, because you will not be eligible for unemployment benefits, most likely.
The secret to dealing with verbal harrassment from a supervisor like him is to develop selective deafness. Be pleasant, cool, do your job to the best of your ability, and basically ignore his words. His bluster, his language, all this is his management technique. If it works, then he is a successful supervisor. The company has hired him, it is their responsibility to control his behavior.
If he is calling you a "lilly white old handicapped Catholic male" or something, or telling you you're too old to do this job (and you're over 40) this is maybe going to be EEOC covered harrassment. you can complain to the EEOC while still employed there, and you should tell HR that you plan to do so.
But in any case, it's just words. Don't let him control your behavior, make you angry, or cost you your job simply because he runs his mouth. If he strikes you or assaults you physically, call the police immediately and press charges againt him for assault. Do not pick up a wrench and defend yourself unless your life is actually threatened. you can improve this situation by your behavior. Do not let him be in control. You do have the personal right to be treated fairly and not harrassed. Hopefully, you and your company can work this out in some way. The more mature and dignified you act, the less you let him "occupy space in your mind," the better the outcome will be for you mentally, even if you have to leave and get another job.