unflag from all users
I'm assuming this question is about D7 ...

Step 1

Start with creating an extremely basic Rules Component with a Rules Action to unflag (using the Flag module) a user, whereas you specify the user to be unflagged as a parameter for that Rules Component. And you pick one of the 3 flags you want to process (eg the "annoying"-flag).

Optionally include whatever Rules Conditions you might need (if any, eg to check if the user is flagged). FYI: this is really like a getting started with Rules ...

Step 2

Use the view of flagged user that you already have and transform that view into a view that uses Views Bulk Operations, and use the technique documented in "How to use Rules to create a custom bulk operation for a VBO view?".

In doing so, you'll then be able to use Rules Components (using the Rules module), in which you can then include the required logic to unflag a single user. I.e the Rules Component from Step 1.

Step 3

Use the checkmark in front of each user in your VBO view as the "button in front of each user to clear that flag" (as in your question), and select the custom VBO action that you created in Step 2.

Step 4

Repeat the previous steps for your additional flags, i.e. the "offensive"-flag and the "fake"-flag.