How to test an alter hook in your .module with ModuleHandler service?
For alter hooks, there is ModuleHandler::alter(). That is what, for example, the AlterTest class uses.

// Verify alteration of a single argument.
$array_copy = $array;
$array_expected = [
  'foo' =&gt; 'Drupal theme',
];
\Drupal::moduleHandler()-&gt;alter('drupal_alter', $array_copy);
\Drupal::theme()-&gt;alter('drupal_alter', $array_copy);
$this-&gt;assertEqual($array_copy, $array_expected, 'Single array was altered.');
$entity_copy = clone $entity;
$entity_expected = clone $entity;
$entity_expected-&gt;foo = 'Drupal theme';
\Drupal::moduleHandler()-&gt;alter('drupal_alter', $entity_copy);
\Drupal::theme()-&gt;alter('drupal_alter', $entity_copy);
$this-&gt;assertEqual($entity_copy, $entity_expected, 'Single object was altered.');


The Drupal 7 documentation for module_invoke() is more explicit in suggesting to use drupal_alter() for arguments passed by reference.


  All arguments are passed by value. Use drupal_alter() if you need to pass arguments by reference.


If the hook you are testing is also implemented from one of the modules from which your module depends, ModuleHandler::alter() would invoke the hook implemented from those modules (if those modules are installed from the test), but I take that testing an alter hook means also testing it works together the same hook implemented by the module dependencies. (In this way, the module can also test if its alter hook implementation needs to be executed earlier or later, respect other implementations of the same hook.)

if you really need to invoke a single alter hook, and other modules installed from the test implement it, you could use the following method to add in the test class.

function protected invokeAlterHook($module, $type, &amp;$data, &amp;$context1 = NULL, &amp;$context2 = NULL) {
  $function = "$module_$type";
  $function($data, $context1, $context2);
}


If you look at the code in ModuleHandler::alter(), that is simply the way it invokes the hook implementations.

  foreach ($this-&gt;alterFunctions[$cid] as $function) {
    $function($data, $context1, $context2);
  }


ModuleHandler::invoke() doesn't work with parameters expected to be passed as reference because it uses the following code to invoke the hook.

$function = $module . '_' . $hook;
return call_user_func_array($function, $args);


To pass an argument by reference, call_user_func_array() should be called, for example, as call_user_func_array($function, [&amp;$data]).