It depends upon EXACTLY what he is saying.
Contrary to popular belief, there is nothing illegal about a negative reference. It is only illegal to give an UNTRUE reference. Note that "untrue" and "a reference I disagree with" are not necessarily the same thing.
The employer is legally entitled to say anything that is true or that represents his honest opinion. If what he is saying is true, then you have no case whatsoever, no matter how negative and no matter how it affects you. So if he says you were frequently late, and you WERE frequently late, you can't sue him. If he says you got negative performance reviews, and you DID get negative performance reviews, you can't sue him.
If what he is saying is something he honestly believes to be true, then again you can't sue him. So if he says that you had a bad management style, and he honestly believes you had a bad management style, you can't sue him.
But if he says something that is FALSE, such as that you were fired for stealing (when you were fired for a totally different reason, or you quit), then you MAY be able to take legal action.
The burden of proof is on you to show ALL of the following:
1.) What he is saying is FALSE. Not an opinion that you disagree with, not a misunderstanding, not a differing view of what transpired, but FALSE.
2.) It was broadcast to a third party.
3.) As a result of what he said, you suffered damages.
It's not enough for you to say, Well, I didn't get the job, so he must have said something bad about me. You need to show that he said something UNTRUE and that as a direct result, you didn't get the job. In other words, BUT FOR what he said, you would have been hired.
If you believe you can meet this criteria, see a local attorney.