How to pass current selection to entity browser select view
The Entity Browser doesn't current pass current default value items field in persistent data, but it's easy to add it.

1) Add persistent data using field_widget_form_alter()

/**
 * Implements hook_field_widget_form_alter().
 */
function mymodule_field_widget_form_alter(&amp;$element, FormStateInterface &amp;$form_state, $context) {
  if (!empty($element['entity_browser'])) {
    $default_value =  $element['entity_browser']['#default_value'];
    $ids = [];
    foreach ($default_value as $entity) {
      $ids[] = $entity-&gt;id();
    }
    $element['entity_browser']['#widget_context']['current_ids'] = implode('+', $ids);
  }
}


2) Update your selection so that if blank it shows all:

  /**
   * {@inheritdoc}
   */
  public function getArgument() {
    $argument = NULL;
    $current_request = $this-&gt;view-&gt;getRequest();

    // Check if the widget context is available.
    if ($current_request-&gt;query-&gt;has('uuid')) {
      $uuid = $current_request-&gt;query-&gt;get('uuid');
      if ($storage = $this-&gt;selectionStorage-&gt;get($uuid)) {
        if (!empty($storage['widget_context']['current_ids'])) {
          $argument = $storage['widget_context']['current_ids'];
        }
        else {
          $argument = 'all';
        }
      }
    }
    return $argument;
  }


3) Make sure you have "exclude" and "allow multiple" checked on your selection.



By the way, if you update to the latest dev release of entity_browser, you don't need your custom plugin.  There is a new entity_browser_widget_context default value views plugin that is configurable.

I also added an issue to the entity_browser queue to add this information when in the widget_context.