Let me throw this scenario out there.....lets say that SHE wants to take him out of state and on vacation for a week, does she need to ask for my permission since it would interfer with his and my visitation time? And what can happen if I say no, and she still does so? I want to work with her on this and we are both going to have do things or approve requests we do not necessarily agree with, but do so knowing what is best for the child.
No, she probably wouldn't need your permission. She would need to demonstrate that she tried to work with you to make up the visitation (and since she is giving you extra time already, that's a cinch for her). She has primary custody of the child, its expected that she will be able to vacation with the child.
You have two hour visits. It is not expected, at this time, that you would be able to vacation with the child.
What you really need to understand here dad is your current legal position. Your rights are limited to what the court order gives you, or what mom agrees to give you over and above the court order. If the court order is not giving you what you feel is fair, then you need to take things back to court to try for a modification.
You cannot just unilaterally keep the child beyond what the court order says without mom's permission, and doing that, PLUS taking the child out of state would seriously get you into a world of hurt.