How can I remove all contextual links?
It was a caching issue. This is my final code to cleanly hide all contextual links except in layout building mode:

use Drupal\Core\Entity\Display\EntityViewDisplayInterface;
use Drupal\Core\Entity\EntityInterface;

function mymodule_page_attachments(array &amp;$page) {
  $routeName = \Drupal::routeMatch()-&gt;getRouteName();
  if (!in_array($routeName, ['layout_builder.overrides.node.view', 'layout_builder.defaults.node.view'])) {
    $page['#attached']['library'] = array_diff($page['#attached']['library'], ['contextual/drupal.contextual-links', 'contextual/drupal.contextual-toolbar']);
  }
}

function mymodule_entity_view_alter(array &amp;$build, EntityInterface $entity, EntityViewDisplayInterface $display) {
  $routeName = \Drupal::routeMatch()-&gt;getRouteName();
  if (!in_array($routeName, ['layout_builder.overrides.node.view', 'layout_builder.defaults.node.view'])) {
    unset($build['#contextual_links']);
  }
}


For the caching issue:
The changes above most likely will (after cache clearing) remove all contextual link entries, but leave an empty placeholder like &lt;div data-contextual-id="..." data-contextual-token="..."&gt;&lt;/div&gt; in your rendered HTML. To get rid off those empty divs, go to your browsers Developer console &gt; Application &gt; Session storage and clear all keys starting with Drupal.contextual. After clearing them, the empty placeholder divs are gone too, no more CSS hacking needed.