There is an avenue for holding the company and/or the participants liable if you can identify what is being said, by whom and to whom.
If what is being said is false and defamatory, and if, as a result of the false and defamatory statements made about you, you suffer reputational injury, then those making the statements can potentially be held liable.
Reputational injury can be shown through an economic loss, as Andy implied with his question, or through a showing of a loss of esteem in your community, a directed hatred, ridicule or contempt toward you in your personal or professional life, humiliation suffered as a result of the defamatory comments made, or any significant damage to you in your profession or trade.
The fact that the gossip or rumors MAY affect you in the future cannot be used as the basis for any defamation action.
If your former employer passes on any false and defamatory information about you to a prospective new employer via a reference, THEN you should consult with an attorney, as there may be an action worth pursuing in court.