April,
Some questions first:
You ask if you are 'doing this right?' That depends on what you are hoping to accomplish.
What is the current custody decree? Does he have any court ordered visitation? If he does, it doesn't matter whether or not he excersized that option or not, you have to comply. Otherwise, you will be the one in contempt. Courts dont necessarily penalize those who choose not to spend time with their child, but you can be sure you will be penalized for not allowing it.
If you answer these questions, I, and others, may be able to respond more fully.