Viewing an image file breaks if view mode has responsive image style
This is not a proper answer, but a workaround in case someone else has this issue: detect if a responsive style is used, load it, and apply via theme layer (the style name is hard-coded to be "embed"):

if ($file) {
  $view_mode = \Drupal::entityTypeManager()
     -&gt;getStorage('entity_view_display')
     -&gt;load('file.' . $file-&gt;bundle() . '.embed');
  $content = $view_mode-&gt;get('content');

  // Responsive.
  if (!empty($content['uri']['settings']['responsive_image_style'])) {
    $build = [
      '#theme' =&gt; 'responsive_image',
      '#responsive_image_style_id' =&gt; $content['uri']['settings']['responsive_image_style'],
      '#uri' =&gt; $file-&gt;getFileUri(),
    ];
  }

  // Non responsive.
  else {
    $view_builder = \Drupal::entityTypeManager()-&gt;getViewBuilder($file-&gt;getEntityTypeId());
    $build = $view_builder-&gt;view($file, 'embed');
  }
}