Im also not attempting to take custody, unless of course CPS needs to remove him from the home. I'm just trying to establish some ground rules, and ensure my child is safe when he is at home. If i did nothing, and received a a phone call, saying something happened while he was unattended (ie. fell out the window), wouldn't i be every bit as responsible for not acting?
Stop. Just for a second, please stop.
Once more, with feeling: You cannot base anything on "what if". He's FIVE years old - most kids have absolutely no concept of the difference between 5 minutes (when they're doing something fun) and an hour (when they're bored).
The court trusts both parents to make sure as much as possible that the child won't come to harm.
Sometimes harm happens anyway. We cannot control every single aspect of our lives and, again, you're going to come out looking less-than-favorable if this went to court.
What I'm getting is that the child is being somewhat pressed to answer. A 5 year old generally isn't going to volunteer, "OMG Dad..I'm so sick of Mom she won't let me do anything and they leave me alone ALL DAY", unless he is being asked a somewhat leading question. Kids have this inherent need to tell a grown-up what she/he thinks the grown-up wants to hear.
You trusted Mom at one point. You need to trust her until and unless you get actual proof that she's knowingly endangering the child. Otherwise, she'll come out on top every. single. time.