Get image file values in $form_state in media entity form
Normally you use a ConstraintValidator for validating entities. In this case core invokes a hook so that you can add your own error messages to the existing file validation process:
function hook_file_validate(Drupal\file\FileInterface $file) {
  $errors = [];

  if (!$file-&gt;getFilename()) {
    $errors[] = t(&quot;The file's name is empty. Please give a name to the file.&quot;);
  }
  if (strlen($file-&gt;getFilename()) &gt; 255) {
    $errors[] = t(&quot;The file's name exceeds the 255 characters limit. Please rename the file and try again.&quot;);
  }

  return $errors;
}

About the caveat for using form alter. You can add custom #upload_validatorsto the image widget. Add settings if necessary:
$element['#upload_validators']['my_custom_file_validator'] = [$my_setting];