My X is hurting the kids

I live in Calif. been divorced for 7 yrs. with two kids
ages 13 & 8. We have had 50-50 custody all this time.
My X has always had an unstable residence and has not
been able to keep a job since we divorced. Now I believe
he is on disability and maybe unemployment. His health is poor as well. Now he is residing with his Mother so she can help him raise and support the children on his visitation days. I have been remarried for 5 years and have the same residence since the divorce. My problem is, that I am trying to modify the original divorce decree since my children are growing older and their needs have changed...plus they have requested it. My X & I have had trouble communicating off and on through the 7 years and its been a constant battle...and the kids are in the middle of our disputes. now the kids have certain activities that he won't take them to. I put them in swim lessons and he doesn't take them on "his" days. He lives about 15 miles away from me and says the distance traveling is too much for him.Although his Mother is the one who transports the kids everywhere.We are in the process of going to mediation (court ordered) but nothing has gotten resolved...for we have about 2 more appts. to go to. In mediation I am asking to have the kids M-F during the school year giving up my weekends to him...and giving him the kids in the Summer. I am willing to waive the support under the condition he helps purchase things for the children.I don't know if that is such a smart decision. I thought maybe he could try to get Spousal support from me. With the 50-50 custody he hasn't bought clothes, haircuts, shoes etc...Although his Mother has helped purchase misc. things from Thrift stores. I don't know if he will agree to this offer. It hurts me that my X won't take the
kids to their activities on "his" days. What can I do? Is this fair? He has been impossible to talk to. He has so much anger and resentment for me and my husband. I am only concerned for the childrens well being. I don't want to end up in court and have to pay the high cost of a lawyer. I am looking for any suggestions of what I should do.