How do I override the mail system? In my application, all mails aren't sent via the Drupal default mail system or SMTP. We have a third-party API which sends all emails. So, whenever the plugin.manager.mail service is called, the class I define in a custom module should be used.

How can I achieve this?
You can achieve this by overriding the service defined from Drupal core.
Altering existing services, providing dynamic services describes the code to use.

In your case, the code should be similar to the following one.

namespace Drupal\my_module;

use Drupal\Core\DependencyInjection\ContainerBuilder;
use Drupal\Core\DependencyInjection\ServiceProviderBase;

// @note: You only need Reference, if you want to change service arguments.
use Symfony\Component\DependencyInjection\Reference;

/**
 * Modifies the mail provider service.
 */
class MyModuleServiceProvider extends ServiceProviderBase {

  /**
   * {@inheritdoc}
   */
  public function alter(ContainerBuilder $container) {
    $definition = $container-&gt;getDefinition('plugin.manager.mail');
    $definition-&gt;setClass('Drupal\MY_MODULE\MyMailProvider');
  }
}