Public Drainage Easement Problem

What is the name of your state?California. My husband and I own a home with a public drainage easement located to the left of our house. There is a service road located on our property that ends several houses after our easement ends. At our request, the builder placed a gate at the beginning of our property so the public would not drive on our private property. The gate is ours to maintain. Our problem is our neighbor. The backyard of their home faces the side of our home where our easement ends and their easement begins. They claim that they have the right to travel on our property (via the public drainage easement) because some of their property is slightly up a hill past our property line. We say, they have no right to use our property as it is a public drainage easement to share with county for maintenance of the drain. We also say if they want to access their hillside, they should install a gate in their backyard fence. Currently, without our permission, they are building a pool and have given all of the contractors access via our gate and property. Are we right to insist they have no rights to access their property via our property by way of our public drainage easement?

Also, no ingress/egress easement

Also, there is no recorded ingress/egress easement for our neighbors to drive through our property.