Limit entity reference field formatter to certain entity types
I think this will work...

There is an isApplicable() method in Drupal\Core\Field\FormatterBase. You can implement it in your FieldFormatter class like this:

public static function isApplicable(FieldDefinitionInterface $field_definition) {
  return ($field_definition-&gt;getFieldStorageDefinition()-&gt;getSetting('target_type') == 'user');
}