Strange behaviour with hook_entity_field_access
AccessResult::allowedIfHasPermission:


  Creates an allowed access result if the permission is present, neutral
  otherwise.


If access is allowed somewhere else, a failed check for the permission doesn't prevent access, because it creates a neutral result.

As you found out you probably need to build a forbidden access result.