Go to follower's list and then just simply hit the delete or unfollow button.