Count the number of Flagged users
Well, actually it's not too late to implement if you do the following:

1) Add a following count (integer) field as a user field.

2) Go to /user/1/edit and put any number other than 1, inside of this field. (this is just a temporary place holder).

3) With PHPMyAdmin or a DB viewer program, export the following count field as a CSV file.

4) The 1 is where the USER ID goes, and the other random number is the actual following count.

So with views, create a view that shows the user ID and the following flag count number. 

Then you open the exported .csv file on Excel and paste these data in the proper columns.

5) Import back into your Drupal DB and clear flush cache.

*Note: Instead of doing steps 3, 4 and 5, you could just do this with Views Bulk Operations + Rules.

6) Using the Rules module, create a rule that +1's the user following count field, when a user flags the follow flag and -1's it when they unfollow.

7) In views, in your List of Users View, for Filter, simply add the following count field and put less than 5.

8) Use the Field Permissions module to hide the following count field, that way when they edit their profile, the user cannot insert/modify the number.