You could tear it down, but maybe not legally. If you do it and you don't have a right to do it, you could jeopardize a sale by creating a cloudy title issue. If you weren't selling it and you were planning on keeping it for a while and are willing to get into a fight, then I would tear it down - with the knowledge that if you lose the battle you start, you may end up having to pay to replace the fence on your property. Hopefully, this unlikely scenario will not occur.
The problem lies with adverse possession or prescriptive rights. If the fence has been there long enough and the neighbor has followed some or all of the right procedures for stealing your land, he may have a case to prevent you from taking the fence down AND for stealing that portion of your property the fence is on. The same for prescriptive rights, which wont let him steal the property, but will let him get an easement on your property.
For either adverse posession or prescriptive rights, the fence will have had to be there for several years. In my state it is 7 years, and in other states it is 20 years. Check your state to see what the law is and find out how long the fence has been there. Another issue is whether they did this with your (or previous owners of your property, if any) permission. If you or a previous owner DID give him permission then the neighbor CANNOT claim a right to your land. If you DID NOT give permission, then the neighbor MAY be able to claim a right based on other conditions. The logic seems backward, but there is a reason for it (scope is too big for this discussion, but its available everywhere).
Bottom line is if the fence has been there for only a few years or less and if you or a previous owner of your property knew it was on your property and allowed him to do it, then you could tear it down. But just be prepared for the neighbor to take you to court over it - even if you end up winning, it could screw up a sale. A better strategy would be to try to get him on your side.
If you are concerned about tearing down (its a reasonable concern as judges don't really like it when people do things like this sometimes), send him a letter explaining the situation and telling him he has xx days to remove it or you will have to do it yourself. Be prepared for him to get an injunction preventing you from tearing it down. Then you will have to go to court if he gets one. If the judge doesnt award one to him, you can tear it down knowing there wont be any repurcussions