Flag Module: "Never Flagged" Status
IMO there is a solution without custom coding ... Below is a synopsis (to avoid writing a manual ...) of how to get this to work ...

Step 1:

Create an additional flag, let's label it like "Ever flagged with FlagX" (whereas 'FlagX' reflects the flag for which you want to know if the entity was ever flagged).

Step 2:

Use the Rules module to create a rule like so:


Rules Event: After entity (=node or user) is flagged with FlagX
Rules Action: Flag an entity (= node or user), with parameter the flag "Ever flagged with FlagX"


Here is a sample of such rule (in Rules export format, adapt the machine name of both flags to fit your needs prior to importing it in your own site), for a flag used for flagging an entity of type "user":

{ "rules_set_the_ever_flagged_with_flagx_flag" : {
    "LABEL" : "Set the ever flagged with FlagX flag",
    "PLUGIN" : "reaction rule",
    "OWNER" : "rules",
    "REQUIRES" : [ "flag" ],
    "ON" : { "flag_flagged_flagx" : [] },
    "DO" : [
      { "flag_flaguser" : {
          "flag" : "ever_flagged_with_flagx ",
          "user" : [ "flagged-user" ],
          "flagging_user" : [ "flagging-user" ],
          "permission_check" : "1"
        }
      }
    ]
  }
}


Step 3:

"To know if an entity was ever been flagged before" (as in your question), all you need to do now is to check for the entities that have been flagged with the "Ever flagged with FlagX" flag. Obviously, it should be straight forward to use the Views module to create an appropriate list of those entities (like users) for which the "Ever flagged with FlagX" flag is not yet set ... which would then be the actual answer to your question, i.e. the entities (like users) which were "Never flagged with FlagX".