Pay for a survey to determine exactly where the property line and fence were. If the fence was on your property, you can sue neighbor for trespass and damage to property. Your damages would be the cost of putting up a new fence. If the fence was on his land, you have no damages. If the fence was actually right on the property line (which I highly doubt), you get damages for any parts that were on your side of the line, but not for anything on his side of the line.
Depending on the cost of a new fence and the cost of a survey, it may be cheaper to just put up a new fence that's on your side of the line.
Check your plat maps and deed to see if there is any mention of the fence being a shared fence or if it designates the property line. He might not be able to remove the landmark delineating the property line.
------------------
This is not legal advice and you are not my client. Double check everything with your own attorney and your state's laws.