My parents had a pool, and although they had a fence, they had like 5 doors that led out to the pool. They did not watch my twins when they were little. They were very mad, and thought it was because we didn't trust them...but we didn't trust the twin terrors that could divide and conquer, not them. It was a battle I fought for several years and stood my ground.
Then we bought a house with a pool. Fenced, with locks and alarms on both doors leading out to it, and the alarm was always activated. Further, we had a fence around the pool inside the fence.
I used to live in Vegas, and there was a drowning like every other day. It only takes a few seconds and you live with the consequences forever. I was totally on your side for more visitation until I got to the fence thing. If I were mom, I would freak the heck out over the lack of fence and fight you tooth and nail. Invest in a fence.