Sounds like harassment from your landlord to me, might want to think about hiring an attorney.
Long story but I am to the point to where I feel unsafe because of my landlord. He comes in whenever he pleases and has gotten verbally abusive. How can I legally end my lease? Talking with him about it isn't an option. He will sue us out of spite.
You need evidence, don't say anything to your neighbour until you acquire it. Find out your rights by talking to a free law advice service.
You should get a lawyer. A little video of his abuse and trespassing attempts would be very helpful in your situation.
What can he sue you for?
Access by landlord to rental property is usually limited:
https://www.nolo.com/legal-encyclopedia/chart-notice-requirements-enter-rental-29033.html
And check out this site:
www.nolo.com/legal-encyclopedia/tenants-rights-privacy-repai