Survey stakes as in wooden stakes, no problem. Unless of course you are still building and your builder is lost without them. Also, if you paid for the survey it may be illegal, but see my comments in last paragraph about that. If you look at your Plat at the county courthouse, you should see IRF (Iron Rod Found), IRS (Iron Rod Set), IPF (Iron Pipe Found), IPS (Iron Pipe Set), or Stone, or tree or some other permanent marker you can find on your property corners. The pipes or rods are usually slightly visible on the surface unless the ground was recently graded, then they could be covered by some dirt (maybe even a foot or so), but you can always find them with a metal detector. Sometimes the stakes are actually taped or strung to the rods/pipes. You should also see LONG/LAT coordinates and then you can use a GPS to find your pipes/rods or whatever. I don't believe that it is legal to remove any permanent marking, but the wood stakes would eventually rot and fall away.
All that said, I am not a surveyor or the like, but I have been doing a good bit of research on my own land and have developed tidbits of knowledge like this, but if you think your neighbor has removed any permanent markings, a new survey should do the trick. The surveyor should be able to find the corners based on records and then set new rods/pipes. Whether or not you can sue your neighbor or he/she can sue you is probably not worth worrying about since a surveys only cost about 300 - 500 and it'll cost you that in aggravation of taking someone to small claims court. Maybe Judge Judy can handle it for you. The RainMan would recommend Judge Whoppner, but I heard he retired a while back.
Hope all goes well for you.