class_docstr: 实现 ``fairscale`` 功能的 ``Driver`` 。
  
  :param model: 传入给 ``Trainer`` 的 ``model`` 参数。
  :param parallel_device: 用于分布式训练的 ``gpu`` 设备。
  :param is_pull_by_torch_run: 标志当前的脚本的启动是否由 ``python -m torch.distributed.launch`` 启动的。
  :param fp16: 是否开启 fp16 训练。
  :param fairscale_kwargs:
  
      * *oss_kwargs* --
      * *sdp_kwargs* --
      * *fsdp_kwargs* --
      * *ddp_kwargs* --
      * *set_grad_to_none* -- 是否在训练过程中在每一次 optimizer 更新后将 grad 置为 ``None``
      * *non_blocking* -- 表示用于 :meth:`torch.Tensor.to` 方法的参数 non_blocking
      * *gradscaler_kwargs* -- 用于 ``fp16=True`` 时，提供给 :class:`torch.amp.cuda.GradScaler` 的参数
  :kwargs:
      * *wo_auto_param_call* (``bool``) -- 是否关闭在训练时调用我们的 ``auto_param_call`` 函数来自动匹配 batch 和前向函数的参数的行为
  
      .. note::
  
          关于该参数的详细说明，请参见 :class:`~fastNLP.core.controllers.Trainer` 中的描述；函数 ``auto_param_call`` 详见 :func:`fastNLP.core.utils.auto_param_call`。
class_docstr_tok: ["\u5b9e\u73b0","``","fairscale","``","\u529f\u80fd\u7684","``","Driver","``","\u3002",":","param","model",":","\u4f20\u5165\u7ed9","``","Trainer","``","\u7684","``","model","``","\u53c2\u6570\u3002",":","param","parallel_device",":","\u7528\u4e8e\u5206\u5e03\u5f0f\u8bad\u7ec3\u7684","``","gpu","``","\u8bbe\u5907\u3002",":","param","is_pull_by_torch_run",":","\u6807\u5fd7\u5f53\u524d\u7684\u811a\u672c\u7684\u542f\u52a8\u662f\u5426\u7531","``","python","-m","torch.distributed.launch","``","\u542f\u52a8\u7684\u3002",":","param","fp16",":","\u662f\u5426\u5f00\u542f","fp16","\u8bad\u7ec3\u3002",":","param","fairscale_kwargs",":","*","*","oss_kwargs","*","--","*","*","sdp_kwargs","*","--","*","*","fsdp_kwargs","*","--","*","*","ddp_kwargs","*","--","*","*","set_grad_to_none","*","--","\u662f\u5426\u5728\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\u5728\u6bcf\u4e00\u6b21","optimizer","\u66f4\u65b0\u540e\u5c06","grad","\u7f6e\u4e3a","``","None","``","*","*","non_blocking","*","--","\u8868\u793a\u7528\u4e8e",":","meth",":","`","torch.Tensor.to","`","\u65b9\u6cd5\u7684\u53c2\u6570","non_blocking","*","*","gradscaler_kwargs","*","--","\u7528\u4e8e","``","fp16=True","``","\u65f6\uff0c\u63d0\u4f9b\u7ed9",":","class",":","`","torch.amp.cuda.GradScaler","`","\u7684\u53c2\u6570",":","kwargs",":","*","*","wo_auto_param_call","*","(","``","bool","``",")","--","\u662f\u5426\u5173\u95ed\u5728\u8bad\u7ec3\u65f6\u8c03\u7528\u6211\u4eec\u7684","``","auto_param_call","``","\u51fd\u6570\u6765\u81ea\u52a8\u5339\u914d","batch","\u548c\u524d\u5411\u51fd\u6570\u7684\u53c2\u6570\u7684\u884c\u4e3a","..","note",":",":","\u5173\u4e8e\u8be5\u53c2\u6570\u7684\u8be6\u7ec6\u8bf4\u660e\uff0c\u8bf7\u53c2\u89c1",":","class",":","`","~fastNLP.core.controllers.Trainer","`","\u4e2d\u7684\u63cf\u8ff0\uff1b\u51fd\u6570","``","auto_param_call","``","\u8be6\u89c1",":","func",":","`","fastNLP.core.utils.auto_param_call","`","\u3002"]
docstr: 准备分布式环境，该函数主要做以下两件事情：
  
      1. 开启多进程，每个 gpu 设备对应单独的一个进程；
      2. 每个进程将模型迁移到自己对应的 ``gpu`` 设备上；然后使用 ``DistributedDataParallel`` 包裹模型；
docstr_tok: ["\u51c6\u5907\u5206\u5e03\u5f0f\u73af\u5883\uff0c\u8be5\u51fd\u6570\u4e3b\u8981\u505a\u4ee5\u4e0b\u4e24\u4ef6\u4e8b\u60c5\uff1a","1",".","\u5f00\u542f\u591a\u8fdb\u7a0b\uff0c\u6bcf\u4e2a","gpu","\u8bbe\u5907\u5bf9\u5e94\u5355\u72ec\u7684\u4e00\u4e2a\u8fdb\u7a0b\uff1b","2",".","\u6bcf\u4e2a\u8fdb\u7a0b\u5c06\u6a21\u578b\u8fc1\u79fb\u5230\u81ea\u5df1\u5bf9\u5e94\u7684","``","gpu","``","\u8bbe\u5907\u4e0a\uff1b\u7136\u540e\u4f7f\u7528","``","DistributedDataParallel","``","\u5305\u88f9\u6a21\u578b\uff1b"]
code: def setup(self):
      r"""
      准备分布式环境，该函数主要做以下两件事情：
  
          1. 开启多进程，每个 gpu 设备对应单独的一个进程；
          2. 每个进程将模型迁移到自己对应的 ``gpu`` 设备上；然后使用 ``DistributedDataParallel`` 包裹模型；
      """
      if self._has_setup:
          return
      self._has_setup = True
      if self.is_pull_by_torch_run:
          # dist.get_world_size() 只能在 dist.init_process_group 初始化之后进行调用；
          self.world_size = int(os.environ.get("WORLD_SIZE"))
          self.global_rank = int(os.environ.get("RANK"))
          logger.info(
              f"World size: {self.world_size}, Global rank: {self.global_rank}"
          )
  
          if not dist.is_initialized():
              dist.init_process_group(
                  backend="nccl",
                  rank=self.global_rank,
                  world_size=self.world_size,
              )
  
          os.environ["fastnlp_torch_launch_not_ddp"] = "yes"
      else:
          if not dist.is_initialized():
              # 这里主要的问题在于要区分 rank0 和其它 rank 的情况；
              self.world_size = len(self.parallel_device)
              self.open_subprocess()
              self.global_rank = self.local_rank  # rank 一定是通过环境变量去获取的；
              dist.init_process_group(
                  backend="nccl",
                  rank=self.global_rank,
                  world_size=self.world_size,
              )
          # 用户在这个 trainer 前面又初始化了一个 trainer，并且使用的是 TorchDDPDriver；
          else:
              # 如果 `dist.is_initialized() == True`，那么说明 TorchDDPDriver 在之前已经初始化并且已经 setup 过一次，那么我们需要保证现在
              #  使用的（即之后的）TorchDDPDriver 的设置和第一个 TorchDDPDriver 是完全一样的；
              pre_num_processes = int(
                  os.environ[FASTNLP_DISTRIBUTED_CHECK]
              )
              if pre_num_processes != len(self.parallel_device):
                  raise RuntimeError(
                      "Notice you are using `TorchDDPDriver` after one instantiated `TorchDDPDriver`, it is not"
                      "allowed that your second `TorchDDPDriver` has a new setting of parameters "
                      "`num_nodes` and `num_processes`."
                  )
              self.world_size = dist.get_world_size()
              self.global_rank = dist.get_rank()
  
      torch.cuda.set_device(self.model_device)
      if self.fs_type != "fsdp":
          self.model.to(self.model_device)
      self.configure_ddp()
  
      self.barrier()
      # 初始化 self._pids，从而使得每一个进程都能接受到 rank0 的 send 操作；
      self._pids = [
          torch.tensor(0, dtype=torch.int).to(self.data_device)
          for _ in range(dist.get_world_size())
      ]
      dist.all_gather(
          self._pids,
          torch.tensor(os.getpid(), dtype=torch.int).to(
              self.data_device
          ),
      )
      local_world_size = (
          int(os.environ.get("LOCAL_WORLD_SIZE"))
          if "LOCAL_WORLD_SIZE" in os.environ
          else None
      )
      if local_world_size is None:
          local_world_size = torch.tensor(
              int(os.environ.get("LOCAL_RANK")), dtype=torch.int
          ).to(self.data_device)
          dist.all_reduce(local_world_size, op=dist.ReduceOp.MAX)
          local_world_size = local_world_size.tolist() + 1
  
      node_rank = self.global_rank // local_world_size
      self._pids = self._pids[
          node_rank
          * local_world_size : (node_rank + 1)
          * local_world_size
      ]
      self._pids = self.tensor_to_numeric(self._pids)
  
imports: from typing import List, Sequence, Union, Dict, Mapping
  from pathlib import Path
  import os
  import functools
  from fastNLP.envs.imports import _NEED_IMPORT_FAIRSCALE
  from ...log import logger
  from .utils import _DDPWrappingModel
  from .ddp import TorchDDPDriver
  from .torch_driver import TorchDriver
  from .utils import _build_fp16_env
  from ....envs.distributed import all_rank_call_context
  from fastNLP.envs import FASTNLP_DISTRIBUTED_CHECK
  from .utils import optimizer_state_to_device
label_desc: Use image node_id 2 for calling the FairScaleDriver obj's underlying member method code with example usage: obj.setup() without return types