Phone visitation

Background - My ex and I live in GA - divorced three years ago. I have physical custody of our son - Ex has filed Petition for Custody Modification - I need to know what I can and can't do - In original divorce/custody decree it says nothing about phone visitation - Ex is now saying (in Petition) that I am denying him phone privledges with our son. Our son is 11 and answering and calling his Dad is just not a priority, in the past I have actually picked the phone up, dialed and handed it to him. His dad calls sometimes calls five times daily starting at 8:00 in the a.m. (I'm at work, our son is at school/daycare) If I answer his call and tell his dad he's playing, he won't leave a message he says he'll call back (usually cell phone) he calls back in minutes, and when he finally reaches Cory he interrogates him as to what he was doing when he called. This phone situation is total out of hand, I feel like I can't even step outside of my home. My ex even went to our son's counselor at school and to his day care asking him to have Cory call him. I am not denying him phone rights in any way, but why do I have to answer calls all evening, it's summer, children are out playing and such and I feel like I'm a hostage to the phone. The sad thing is his dad doesn't get him for visitation as he should often turning down days (holidays) because it is not convenient with his plans.
What rights do I have at this point (I am documenting as my attorney as instructed me to) but this is getting to be too much.

Just document everything.. What is the modification for ?..ie, what is he after ? Is he filing for a change in physical custody ?

With all you have said I do not see what he can win... Document everything.. When you get in court specify the problems and ask that phone calls are at set times.. ie, 7pm Monday and Thursday etc..