Well, I'll take the other side on this one.
First, I say check with the friend of the court or other parenting time and post-judgment dispute arm of your court. (Don't know what's its called in VA.) Discuss the rules and standards where you are.
In the courts I know, if a parent voluntarily misses parenting time/visitation three times in a row- skips or is late chronically- you can go to court to have the schedule changed. And that is deemed to be acting in the best interests of the child.
The reasoning is this. Basically it isn't about you or dad, it's about structure, predictability, respect and a child being able to schedule a life and not have to go through the repeated distress of having a parent bail on them over and over and over. That isn't ok. It sure wouldn't elicit the sympathy of any court personnel I know.
Other reasoning- no one is entitled to have anyone repeatedly sit around and wait for them. People who are considerate and respectful don't do it to others. In addition, this sows seeds of conflict and, just like you said, endorses violating rules. It is passive aggression. No parent is so incredibly important that they are given carte blanche to jerk around other parent and, especially, child!
With a cooperative parent, the need to adjust schedule is a piece of cake. Dad works late often or has a flex schedule, so a new schedule is put in place by consent to provide predictability for child, ability to comply and provide structure to child by Dad (or Mom, if that were the case), and out of respect for other parent.
I'd make an appointment with the referee (or equivalent) to see how to proceed to get his schedule changed to meet status quo so you and your child are not living on puppet strings.
No, this is not ok in my book.