class_docstr: ``TorchDDPDriver`` 通过开启多个进程，让每个进程单独使用一个 gpu 设备来实现分布式训练。
  
  .. note::
  
      您在绝大多数情况下不需要自己使用到该类，通过向 ``Trainer`` 传入正确的参数，您可以方便快速地部署您的分布式训练。
  
      ``TorchDDPDriver`` 目前支持的三种启动方式：
  
          1. 用户自己不进行 ``ddp`` 的任何操作，直接使用我们的 ``Trainer``，这时是由我们自己使用 ``open_subprocesses`` 拉起多个进程，
          然后 ``TorchDDPDriver`` 自己通过调用 ``dist.init_process_group`` 来初始化 ddp 的通信组；（情况 A）
  
          .. code-block::
  
              trainer = Trainer(
                  ...
                  driver='torch',
                  device=[0, 1]
              )
              trainer.run()
  
          通过运行 ``python train.py`` 启动；
  
          2. 用户同样不在 ``Trainer`` 之外初始化 ``ddp``，但是用户自己使用 ``python -m torch.distributed.launch`` 拉起来创建多个进程，这时我们仍旧
          会通过调用 ``dist.init_process_group`` 来初始化 ``ddp`` 的通信组；（情况 B）
  
          .. code-block::
  
              trainer = Trainer(
                  ...
                  driver='torch',
                  device=None, # fastNLP 会忽略传入的 device，并根据 local_rank 自动分配
              )
              trainer.run()
  
          通过运行 ``python -m torch.distributed.launch --nproc_per_node 2 train.py`` 启动；
  
          3. 用户自己在外面初始化 ``DDP``，并且通过 ``python -m torch.distributed.launch`` 拉起，这时无论是多个进程的拉起和 ddp 的通信组的建立
          都由用户自己操作，我们只会在 ``driver.setup`` 的时候对 ``TorchDDPDriver`` 设置一些必要的属性值；（情况 C）
  
          .. code-block::
  
              import torch.distributed as dist
              from torch.nn.parallel import DistributedDataParallel
  
              # 获取当前的进程信息；
              ...
  
              # 初始化 ddp 不同进程间的通信组；
              dist.init_process_group(...)
  
              # 初始化模型使用 DistributedDataParallel 包裹；
              model = Model()
              model = DistributedDataParallel(model, ...)
  
              # 注意此时仍旧不需要您主动地将 datalaoder 的 sampler 替换为 DistributedSampler；
              trainer = Trainer(
                  ...
                  driver='torch',
                  device=None, # fastNLP 会忽略传入的 device，并根据 local_rank 自动分配
              )
              trainer.run()
  
          通过运行 ``python -m torch.distributed.launch --nproc_per_node 2 train.py`` 启动；
  
      注意多机的启动强制要求用户在每一台机器上使用 ``python -m torch.distributed.launch`` 启动；因此我们不会在 ``TorchDDPDriver`` 中保存
      任何当前有多少台机器的信息。
  
  :param model: 传入给 ``Trainer`` 的 ``model`` 参数
  :param parallel_device: 用于分布式训练的 ``gpu`` 设备
  :param is_pull_by_torch_run: 标志当前的脚本的启动是否由 ``python -m torch.distributed.launch`` 启动的
  :param fp16: 是否开启 fp16 训练
  :param torch_kwargs: 
      * *ddp_kwargs* -- 用于在使用 ``TorchDDPDriver`` 时指定 ``DistributedDataParallel`` 初始化时的参数；例如传入
        ``{'find_unused_parameters': True}`` 来解决有参数不参与前向运算导致的报错等
      * *set_grad_to_none* -- 是否在训练过程中在每一次 optimizer 更新后将 grad 置为 ``None``
      * *non_blocking* -- 表示用于 :meth:`torch.Tensor.to` 方法的参数 non_blocking
      * *gradscaler_kwargs* -- 用于 ``fp16=True`` 时，提供给 :class:`torch.amp.cuda.GradScaler` 的参数
  :kwargs:
      * *wo_auto_param_call* (``bool``) -- 是否关闭在训练时调用我们的 ``auto_param_call`` 函数来自动匹配 batch 和前向函数的参数的行为
  
      .. note::
  
          关于该参数的详细说明，请参见 :class:`~fastNLP.core.controllers.Trainer` 中的描述；函数 ``auto_param_call`` 详见 :func:`fastNLP.core.utils.auto_param_call`。
class_docstr_tok: ["``","TorchDDPDriver","``","\u901a\u8fc7\u5f00\u542f\u591a\u4e2a\u8fdb\u7a0b\uff0c\u8ba9\u6bcf\u4e2a\u8fdb\u7a0b\u5355\u72ec\u4f7f\u7528\u4e00\u4e2a","gpu","\u8bbe\u5907\u6765\u5b9e\u73b0\u5206\u5e03\u5f0f\u8bad\u7ec3\u3002","..","note",":",":","\u60a8\u5728\u7edd\u5927\u591a\u6570\u60c5\u51b5\u4e0b\u4e0d\u9700\u8981\u81ea\u5df1\u4f7f\u7528\u5230\u8be5\u7c7b\uff0c\u901a\u8fc7\u5411","``","Trainer","``","\u4f20\u5165\u6b63\u786e\u7684\u53c2\u6570\uff0c\u60a8\u53ef\u4ee5\u65b9\u4fbf\u5feb\u901f\u5730\u90e8\u7f72\u60a8\u7684\u5206\u5e03\u5f0f\u8bad\u7ec3\u3002","``","TorchDDPDriver","``","\u76ee\u524d\u652f\u6301\u7684\u4e09\u79cd\u542f\u52a8\u65b9\u5f0f\uff1a","1",".","\u7528\u6237\u81ea\u5df1\u4e0d\u8fdb\u884c","``","ddp","``","\u7684\u4efb\u4f55\u64cd\u4f5c\uff0c\u76f4\u63a5\u4f7f\u7528\u6211\u4eec\u7684","``","Trainer","``","\uff0c\u8fd9\u65f6\u662f\u7531\u6211\u4eec\u81ea\u5df1\u4f7f\u7528","``","open_subprocesses","``","\u62c9\u8d77\u591a\u4e2a\u8fdb\u7a0b\uff0c","\u7136\u540e","``","TorchDDPDriver","``","\u81ea\u5df1\u901a\u8fc7\u8c03\u7528","``","dist.init_process_group","``","\u6765\u521d\u59cb\u5316","ddp","\u7684\u901a\u4fe1\u7ec4\uff1b\uff08\u60c5\u51b5","A\uff09","..","code-block",":",":","trainer","=","Trainer","(","...","driver='torch","'",",","device=","[","0",",","1","]",")","trainer.run","(",")","\u901a\u8fc7\u8fd0\u884c","``","python","train.py","``","\u542f\u52a8\uff1b","2",".","\u7528\u6237\u540c\u6837\u4e0d\u5728","``","Trainer","``","\u4e4b\u5916\u521d\u59cb\u5316","``","ddp","``","\uff0c\u4f46\u662f\u7528\u6237\u81ea\u5df1\u4f7f\u7528","``","python","-m","torch.distributed.launch","``","\u62c9\u8d77\u6765\u521b\u5efa\u591a\u4e2a\u8fdb\u7a0b\uff0c\u8fd9\u65f6\u6211\u4eec\u4ecd\u65e7","\u4f1a\u901a\u8fc7\u8c03\u7528","``","dist.init_process_group","``","\u6765\u521d\u59cb\u5316","``","ddp","``","\u7684\u901a\u4fe1\u7ec4\uff1b\uff08\u60c5\u51b5","B\uff09","..","code-block",":",":","trainer","=","Trainer","(","...","driver='torch","'",",","device=None",",","#","fastNLP","\u4f1a\u5ffd\u7565\u4f20\u5165\u7684","device\uff0c\u5e76\u6839\u636e","local_rank","\u81ea\u52a8\u5206\u914d",")","trainer.run","(",")","\u901a\u8fc7\u8fd0\u884c","``","python","-m","torch.distributed.launch","--","nproc_per_node","2","train.py","``","\u542f\u52a8\uff1b","3",".","\u7528\u6237\u81ea\u5df1\u5728\u5916\u9762\u521d\u59cb\u5316","``","DDP","``","\uff0c\u5e76\u4e14\u901a\u8fc7","``","python","-m","torch.distributed.launch","``","\u62c9\u8d77\uff0c\u8fd9\u65f6\u65e0\u8bba\u662f\u591a\u4e2a\u8fdb\u7a0b\u7684\u62c9\u8d77\u548c","ddp","\u7684\u901a\u4fe1\u7ec4\u7684\u5efa\u7acb","\u90fd\u7531\u7528\u6237\u81ea\u5df1\u64cd\u4f5c\uff0c\u6211\u4eec\u53ea\u4f1a\u5728","``","driver.setup","``","\u7684\u65f6\u5019\u5bf9","``","TorchDDPDriver","``","\u8bbe\u7f6e\u4e00\u4e9b\u5fc5\u8981\u7684\u5c5e\u6027\u503c\uff1b\uff08\u60c5\u51b5","C\uff09","..","code-block",":",":","import","torch.distributed","as","dist","from","torch.nn.parallel","import","DistributedDataParallel","#","\u83b7\u53d6\u5f53\u524d\u7684\u8fdb\u7a0b\u4fe1\u606f\uff1b","...","#","\u521d\u59cb\u5316","ddp","\u4e0d\u540c\u8fdb\u7a0b\u95f4\u7684\u901a\u4fe1\u7ec4\uff1b","dist.init_process_group","(","...",")","#","\u521d\u59cb\u5316\u6a21\u578b\u4f7f\u7528","DistributedDataParallel","\u5305\u88f9\uff1b","model","=","Model","(",")","model","=","DistributedDataParallel","(","model",",","...",")","#","\u6ce8\u610f\u6b64\u65f6\u4ecd\u65e7\u4e0d\u9700\u8981\u60a8\u4e3b\u52a8\u5730\u5c06","datalaoder","\u7684","sampler","\u66ff\u6362\u4e3a","DistributedSampler\uff1b","trainer","=","Trainer","(","...","driver='torch","'",",","device=None",",","#","fastNLP","\u4f1a\u5ffd\u7565\u4f20\u5165\u7684","device\uff0c\u5e76\u6839\u636e","local_rank","\u81ea\u52a8\u5206\u914d",")","trainer.run","(",")","\u901a\u8fc7\u8fd0\u884c","``","python","-m","torch.distributed.launch","--","nproc_per_node","2","train.py","``","\u542f\u52a8\uff1b","\u6ce8\u610f\u591a\u673a\u7684\u542f\u52a8\u5f3a\u5236\u8981\u6c42\u7528\u6237\u5728\u6bcf\u4e00\u53f0\u673a\u5668\u4e0a\u4f7f\u7528","``","python","-m","torch.distributed.launch","``","\u542f\u52a8\uff1b\u56e0\u6b64\u6211\u4eec\u4e0d\u4f1a\u5728","``","TorchDDPDriver","``","\u4e2d\u4fdd\u5b58","\u4efb\u4f55\u5f53\u524d\u6709\u591a\u5c11\u53f0\u673a\u5668\u7684\u4fe1\u606f\u3002",":","param","model",":","\u4f20\u5165\u7ed9","``","Trainer","``","\u7684","``","model","``","\u53c2\u6570",":","param","parallel_device",":","\u7528\u4e8e\u5206\u5e03\u5f0f\u8bad\u7ec3\u7684","``","gpu","``","\u8bbe\u5907",":","param","is_pull_by_torch_run",":","\u6807\u5fd7\u5f53\u524d\u7684\u811a\u672c\u7684\u542f\u52a8\u662f\u5426\u7531","``","python","-m","torch.distributed.launch","``","\u542f\u52a8\u7684",":","param","fp16",":","\u662f\u5426\u5f00\u542f","fp16","\u8bad\u7ec3",":","param","torch_kwargs",":","*","*","ddp_kwargs","*","--","\u7528\u4e8e\u5728\u4f7f\u7528","``","TorchDDPDriver","``","\u65f6\u6307\u5b9a","``","DistributedDataParallel","``","\u521d\u59cb\u5316\u65f6\u7684\u53c2\u6570\uff1b\u4f8b\u5982\u4f20\u5165","``","{","'find_unused_parameters","'",":","True","}","``","\u6765\u89e3\u51b3\u6709\u53c2\u6570\u4e0d\u53c2\u4e0e\u524d\u5411\u8fd0\u7b97\u5bfc\u81f4\u7684\u62a5\u9519\u7b49","*","*","set_grad_to_none","*","--","\u662f\u5426\u5728\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\u5728\u6bcf\u4e00\u6b21","optimizer","\u66f4\u65b0\u540e\u5c06","grad","\u7f6e\u4e3a","``","None","``","*","*","non_blocking","*","--","\u8868\u793a\u7528\u4e8e",":","meth",":","`","torch.Tensor.to","`","\u65b9\u6cd5\u7684\u53c2\u6570","non_blocking","*","*","gradscaler_kwargs","*","--","\u7528\u4e8e","``","fp16=True","``","\u65f6\uff0c\u63d0\u4f9b\u7ed9",":","class",":","`","torch.amp.cuda.GradScaler","`","\u7684\u53c2\u6570",":","kwargs",":","*","*","wo_auto_param_call","*","(","``","bool","``",")","--","\u662f\u5426\u5173\u95ed\u5728\u8bad\u7ec3\u65f6\u8c03\u7528\u6211\u4eec\u7684","``","auto_param_call","``","\u51fd\u6570\u6765\u81ea\u52a8\u5339\u914d","batch","\u548c\u524d\u5411\u51fd\u6570\u7684\u53c2\u6570\u7684\u884c\u4e3a","..","note",":",":","\u5173\u4e8e\u8be5\u53c2\u6570\u7684\u8be6\u7ec6\u8bf4\u660e\uff0c\u8bf7\u53c2\u89c1",":","class",":","`","~fastNLP.core.controllers.Trainer","`","\u4e2d\u7684\u63cf\u8ff0\uff1b\u51fd\u6570","``","auto_param_call","``","\u8be6\u89c1",":","func",":","`","fastNLP.core.utils.auto_param_call","`","\u3002"]
docstr: :return: 当前使用的 driver 是否是分布式的 driver，对于 ``TorchDDPDriver`` 来说，该函数一定返回 ``True``。
code: def is_distributed(self):
      r"""
      :return: 当前使用的 driver 是否是分布式的 driver，对于 ``TorchDDPDriver`` 来说，该函数一定返回 ``True``。
      """
      return True
  
imports: import os
  import sys
  import __main__
  import socket
  import numpy
  from time import sleep
  from typing import List, Optional, Union, Dict, Tuple, Callable
  from fastNLP.envs.imports import _NEED_IMPORT_TORCH
  from .torch_driver import TorchDriver
  from fastNLP.core.drivers.torch_driver.utils import _DDPWrappingModel, replace_sampler, replace_batch_sampler
  from fastNLP.core.drivers.utils import distributed_open_proc
  from fastNLP.core.utils import auto_param_call, check_user_specific_params
  from fastNLP.core.samplers import ReproducibleSampler, RandomSampler, UnrepeatedSequentialSampler, ReproducibleBatchSampler, re_instantiate_sampler, UnrepeatedSampler, conversion_between_reproducible_and_unrepeated_sampler
  from fastNLP.envs import FASTNLP_DISTRIBUTED_CHECK, FASTNLP_GLOBAL_RANK, FASTNLP_GLOBAL_SEED, FASTNLP_NO_SYNC
  from fastNLP.core.log import logger
  from fastNLP.core.drivers.torch_driver.dist_utils import fastnlp_torch_all_gather, fastnlp_torch_broadcast_object
  from .utils import _check_dataloader_args_for_distributed
label_desc: Use image node_id 21 for calling the TorchDDPDriver obj's underlying member method code with example usage: obj.is_distributed() and returns: True