I can understand your frustration. It is very important as a landlord to conduct yourself PROFESSIONALLY and LEGALLY at all times. By letting them treat the house as their own, they have run all over you. If you don't want to do anything "formally", this is the wrong business for you.
If the tenant requests that they are present for the move out inspection, they have the right to be there under law. That means, you schedule a date and time to do the walk through, and if they don't show, oh well, do it on your own.
But please, do yourself a favor and be consistent and be legal. If they don't pay the rent in the time specified in the lease, send them the proper letter (legal action notice, 3 day notice pay rent or quit, whatever your state requires). Phone calls are not a legal way to handle things. If they request to remove a portion of the fence (YOUR property), get it in writing. Document, document, document!