How do I override the mail system?
You can achieve this by overriding the service defined from Drupal core.
Altering existing services, providing dynamic services describes the code to use.

In your case, the code should be similar to the following one.

namespace Drupal\my_module;

use Drupal\Core\DependencyInjection\ContainerBuilder;
use Drupal\Core\DependencyInjection\ServiceProviderBase;

// @note: You only need Reference, if you want to change service arguments.
use Symfony\Component\DependencyInjection\Reference;

/**
 * Modifies the mail provider service.
 */
class MyModuleServiceProvider extends ServiceProviderBase {

  /**
   * {@inheritdoc}
   */
  public function alter(ContainerBuilder $container) {
    $definition = $container-&gt;getDefinition('plugin.manager.mail');
    $definition-&gt;setClass('Drupal\MY_MODULE\MyMailProvider');
  }
}