Tumbleweed, while Jet is sweeping out the trash let me give you some advice. Or better still, answer the question you never asked.
Yes, your boss can sue you for defamation. However, as Jet also said, truth is the ultimate defense. The problem for you would be in proving that your statements were true.
So, in the worst case anyone can sue anyone over just about anything and if there are actually tapes of your statements the plaintiff has proof that the statements were made. You will then need proof that the statements were truthful to defend against a case of defamation.
Now, back to the facts of your case. What your boss has is heresay at best. Not only would he be need to have his wife testify that you made such statements, the OTHER element of defamation which Jet also gave you would require that he PROVE he has suffered damages.
On the surface it sounds exactly as you guess, a case of intimidation on the part of your boss. HOWEVER, I would advise in the future that you keep your mouth shut in a similar situation.
As you can see, it only opens you to legal recourse and while you may always have a defense, do you really want to spend the money, time and effort continually proving your defense in court?