I use Sprout social to keep track of followers, and who is active or not. Really useful tool, but not cheap!

You don't delete followers, you just unfollow them.