What caused the difficulties? If the children were going to the same dr they always have then they would already know the situation from speaking to mom and dad before hand. Same with the schools.
This past year, it was really difficult for me to get the children care even with a specific power of attorney and a general power of attorney.
The only scenario where they wouldn't , that I can think of off the top of my head would be the ER, and they could easily phone mom asking for permission (as has happened with me before where they phoned my husband. )
ETA: and Ditto what ECM said
Just because you had a child together it does not give you a lifetime pass to harass them
No you cannot terminate rights because you just realized you were stupid.