breakaway -
There are five necessary conditions for a slander or libel suit. The first is publication, which means that there has to be some sort of circulation of a defamatory statement. This publication can only occur if the person who is defamed knows about it and the person who made the defamatory statement communicates it to at least one other person.
You can slander someone just by saying something derogatory about him, but if he doesn't know it, and the person you say it to doesn't say anything, then, obviously, the person who has been defamed cannot sue you because he is unaware that anything bad has been said about him. If you say to him directly something derogatory, only you and he know it and no damage has been done to his reputation, so he cannot sue you (although he may punch you in the nose
)
The other conditions that are necessary for defamation that can lead to a lawsuit against you are identification (or making sure that people know who you are talking about), defamation (the derogatory words that are spoken or written), fault (which is complicated - but basically means that the person who communicated the defamatory statement knew it was false) and injury (what the person suffered as a result of the defamation).
Hope that helps.