How to remove a term programmatically from entity reference field using entity browser on translation form?
To remove them from the user interface, you would have to empty the items array of 'current'

$form['field_category_terms']['widget']['current']['items'] = [];

This would however only hide them in the user interface, to actually remove them you also have to NULL target_id

$form['field_category_terms']['widget']['target_id']['#default_value'] = '';

If the users are allowed to add new terms during the translation, you'd also need to make sure the emptying of the terms only are done initially, because if not, they would always appear empty since Entity Browser uses AJAX and change the form when you add a new term. Something like:

  if (!$form_state-&gt;isProcessingInput()) {
    $form['field_category_terms']['widget']['target_id']['#default_value'] = '';
    $form['field_category_terms']['widget']['current']['items'] = [];
  }