Read your lease carefully. See what it says. (Probably that he can enter for emergencies or to show apartment to prospective tenants after you have given notice to vacate.)
Then tell him he is NOT to enter your apartment without your specific permission absent the kind of emergency specified in the lease, and that if he does it is trespass. Tell him you are also afraid you or a friend may mistake him for a burgular and shoot him with the hunting rifle or shotgun you are thinging of getting, or your friend sometimes brings with him or her.
One quick step is to put a cheap sliding deadbold in (so you can block access when you are inside) so he can't enter when you are there. That's part of the problem solved.