How to write phpUnit tests for hook_form_alter?
It is technically possible to write a unit test for this. You'd need to manually require your .module file though.

It might be easier to use a Kernel test, as these will automatically load module files, fire hooks, etc.

MyTest extends KernelTestBase {

  /**
   * Modules to enable.
   */
  public static $modules = ['my_module'];

  /**
   * Test the function.
   */
  public function testCancelButton() {
    // The .module file will already be loaded, so you can call your custom function directly.
    $form = [...];
    _cancel_button($form, ...);
    // Now make assertions about expected changes.