change in custody

My ex and I were divorced in 1993 and we have joint legal with me having physical custody of the children. He has reasonable visitation or if we don't agree we use the ammended parenting policy. He remarried in 1996 and I remarried in 1997. In 1997 he filed a claim with protective services that myself and my husband were abusing the children so the court gave him an ex-parte change of custody order pending the investigation by protective services. They did the investigation and did not substantiate it. I got my children back after 1 and a half months with an attorney of course. Now, I believe he is trying to build a case against me and try and get a change of custody. He has a Criminal Sexual Conduct 3rd degree on his record, he doesn't see the kids for his midweek visit he is entitled to, he doesn't go to baseball games or parent teacher conferences, he and his wife are not willing to cooperate with me regarding the children. When I have issues regarding the children and visits, he doesn't talk with me, his wife tries to run the show. I don't fee it is her place. He lives with his wifes mother, two sisters and his wifes mothers boyfriend. I personally don't know whether or not he stands a chance of gaining custody of the children or not, but I sure could use some advice. Please respond. Thank you!