class_docstr: 实现 ``deepspeed`` 分布式训练的 ``Driver``。
  
  .. note::
  
      您在绝大多数情况下不需要自己使用到该类，通过向 ``Trainer`` 传入正确的参数，您可以方便快速地部署您的分布式训练；
  
      ``DeepSpeedDriver`` 目前支持的三种启动方式：
  
          1. 用户自己不进行任何操作，直接使用我们的 ``Trainer``，这时是由我们自己使用 ``open_subprocesses`` 拉起多个进程，
          然后 ``DeepSpeedDriver`` 自己通过调用 ``deepspeed.initialize`` 来初始化模型和通信组；（情况 A）
  
          .. code-block::
  
              trainer = Trainer(
                  ...
                  driver='deepspeed',
                  device=[0, 1]
              )
              trainer.run()
  
          通过运行 ``python train.py`` 启动；
  
          2. 用户同样不在 ``Trainer`` 之外初始化 ``deepspeed``，但是用户自己使用 ``python -m torch.distributed.launch`` 拉起来创建多个进程，这时我们仍旧
          会通过调用 ``model.initialize`` 来初始化 ``ddp`` 的通信组；（情况 B）
  
          .. code-block::
  
              trainer = Trainer(
                  ...
                  driver='deepspeed',
                  device=None, # fastNLP 会忽略传入的 device，并根据 local_rank 自动分配
              )
              trainer.run()
  
          通过运行 ``deepspeed train.py`` 启动；
  
          3. 用户自己在外面初始化 ``deepspeed``，并且通过 ``deepspeed train.py`` 拉起，这时无论是多个进程的拉起和通信组的建立
          都由用户自己操作，我们只会在 ``driver.setup`` 的时候对 ``DeepSpeedDriver`` 设置一些必要的属性值；（情况 C）
  
          .. code-block::
  
              import deepspeed
  
              # 初始化
              model, _, _, _ = deepspeed.initialize(model, ...)
  
              trainer = Trainer(
                  ...
                  driver='deepspeed',
                  device=None, # fastNLP 会忽略传入的 device，并根据 local_rank 自动分配
              )
              trainer.run()
  
          通过运行 ``deepspeed train.py`` 启动。
  
  :param model: 传入给 ``Trainer`` 的 ``model`` 参数。
  :param parallel_device: 用于分布式训练的 ``gpu`` 设备。
  :param is_pull_by_torch_run: 标志当前的脚本的启动是否由 ``python -m torch.distributed.launch`` 启动的。
  :param fp16: 是否开启 fp16 训练。
  :param deepspeed_kwargs: 
      * *strategy* -- 使用 ZeRO 优化的策略，默认为 ``deepspeed``；目前仅支持以下值：
  
          * ``deepspeed`` -- 使用 ZeRO 的第二阶段，等同于 ``deepspeed_stage_2``；
          * ``deepspeed_stage_1`` -- 使用 ZeRO 的第一阶段，仅将 ``optimizer`` 的状态分散到不同设备上；
          * ``deepspeed_stage_2`` -- 使用 ZeRO 的第二阶段，将 ``optimizer`` 和 **梯度** 分散到不同设备上；
          * ``deepspeed_stage_2_offload`` -- 使用 ZeRO 的第二阶段，并且借助 cpu 的内存来进一步节约显存；
          * ``deepspeed_stage_3`` -- 使用 ZeRO 的第三阶段，将 ``optimizer`` 、**梯度** 和 **模型** 分散到不同设备上；
          * ``deepspeed_stage_3_offload`` -- 使用 ZeRO 的第三阶段，并且借助 cpu 的内存来进一步节约显存；
          * ``deepspeed_stage_3_offload_nvme`` -- 使用 ZeRO 的第三阶段，并且借助 NVMe 硬盘来进一步节约显存；
      * *logging_level* -- ``deepspeed`` 库的日志等级，默认为 **logging.ERROR**。
      * *config* -- ``deepspeed`` 的各项设置；**FastNLP** 允许用户传入自己的设置以增强灵活性，但这会使参数
        中的 ``optimizer`` 、``strategy`` 、 ``fp16`` 等失效，即当这个参数存在时，**FastNLP** 会用该参数覆盖
        其它的设置。
  :kwargs:
      * *accumulation_steps* -- 即在 :class:`~fastNLP.core.controllers.Trainer` 传入的 ``accumulation_steps`` 。 deepspeed 会将 ``config`` 的
        ``gradient_accumulation_steps`` 设置为该值。
      * *train_dataloader* -- 即在 :class:`~fastNLP.core.controllers.Trainer` 传入的 ``train_dataloader`` 。 ``deepspeed`` 需要通过它来获取
        数据的 ``batch_size`` 用于设置 ``train_micro_batch_size_per_gpu`` 。如果没有传入的话，则会设置为 **1** 。
      * *wo_auto_param_call* (``bool``) -- 是否关闭在训练时调用我们的 ``auto_param_call`` 函数来自动匹配 batch 和前向函数的参数的行为
  
      .. note::
  
          关于该参数的详细说明，请参见 :class:`~fastNLP.core.controllers.Trainer` 中的描述；函数 ``auto_param_call`` 详见 :func:`fastNLP.core.utils.auto_param_call`。
class_docstr_tok: ["\u5b9e\u73b0","``","deepspeed","``","\u5206\u5e03\u5f0f\u8bad\u7ec3\u7684","``","Driver","``","\u3002","..","note",":",":","\u60a8\u5728\u7edd\u5927\u591a\u6570\u60c5\u51b5\u4e0b\u4e0d\u9700\u8981\u81ea\u5df1\u4f7f\u7528\u5230\u8be5\u7c7b\uff0c\u901a\u8fc7\u5411","``","Trainer","``","\u4f20\u5165\u6b63\u786e\u7684\u53c2\u6570\uff0c\u60a8\u53ef\u4ee5\u65b9\u4fbf\u5feb\u901f\u5730\u90e8\u7f72\u60a8\u7684\u5206\u5e03\u5f0f\u8bad\u7ec3\uff1b","``","DeepSpeedDriver","``","\u76ee\u524d\u652f\u6301\u7684\u4e09\u79cd\u542f\u52a8\u65b9\u5f0f\uff1a","1",".","\u7528\u6237\u81ea\u5df1\u4e0d\u8fdb\u884c\u4efb\u4f55\u64cd\u4f5c\uff0c\u76f4\u63a5\u4f7f\u7528\u6211\u4eec\u7684","``","Trainer","``","\uff0c\u8fd9\u65f6\u662f\u7531\u6211\u4eec\u81ea\u5df1\u4f7f\u7528","``","open_subprocesses","``","\u62c9\u8d77\u591a\u4e2a\u8fdb\u7a0b\uff0c","\u7136\u540e","``","DeepSpeedDriver","``","\u81ea\u5df1\u901a\u8fc7\u8c03\u7528","``","deepspeed.initialize","``","\u6765\u521d\u59cb\u5316\u6a21\u578b\u548c\u901a\u4fe1\u7ec4\uff1b\uff08\u60c5\u51b5","A\uff09","..","code-block",":",":","trainer","=","Trainer","(","...","driver='deepspeed","'",",","device=","[","0",",","1","]",")","trainer.run","(",")","\u901a\u8fc7\u8fd0\u884c","``","python","train.py","``","\u542f\u52a8\uff1b","2",".","\u7528\u6237\u540c\u6837\u4e0d\u5728","``","Trainer","``","\u4e4b\u5916\u521d\u59cb\u5316","``","deepspeed","``","\uff0c\u4f46\u662f\u7528\u6237\u81ea\u5df1\u4f7f\u7528","``","python","-m","torch.distributed.launch","``","\u62c9\u8d77\u6765\u521b\u5efa\u591a\u4e2a\u8fdb\u7a0b\uff0c\u8fd9\u65f6\u6211\u4eec\u4ecd\u65e7","\u4f1a\u901a\u8fc7\u8c03\u7528","``","model.initialize","``","\u6765\u521d\u59cb\u5316","``","ddp","``","\u7684\u901a\u4fe1\u7ec4\uff1b\uff08\u60c5\u51b5","B\uff09","..","code-block",":",":","trainer","=","Trainer","(","...","driver='deepspeed","'",",","device=None",",","#","fastNLP","\u4f1a\u5ffd\u7565\u4f20\u5165\u7684","device\uff0c\u5e76\u6839\u636e","local_rank","\u81ea\u52a8\u5206\u914d",")","trainer.run","(",")","\u901a\u8fc7\u8fd0\u884c","``","deepspeed","train.py","``","\u542f\u52a8\uff1b","3",".","\u7528\u6237\u81ea\u5df1\u5728\u5916\u9762\u521d\u59cb\u5316","``","deepspeed","``","\uff0c\u5e76\u4e14\u901a\u8fc7","``","deepspeed","train.py","``","\u62c9\u8d77\uff0c\u8fd9\u65f6\u65e0\u8bba\u662f\u591a\u4e2a\u8fdb\u7a0b\u7684\u62c9\u8d77\u548c\u901a\u4fe1\u7ec4\u7684\u5efa\u7acb","\u90fd\u7531\u7528\u6237\u81ea\u5df1\u64cd\u4f5c\uff0c\u6211\u4eec\u53ea\u4f1a\u5728","``","driver.setup","``","\u7684\u65f6\u5019\u5bf9","``","DeepSpeedDriver","``","\u8bbe\u7f6e\u4e00\u4e9b\u5fc5\u8981\u7684\u5c5e\u6027\u503c\uff1b\uff08\u60c5\u51b5","C\uff09","..","code-block",":",":","import","deepspeed","#","\u521d\u59cb\u5316","model",",","_",",","_",",","_","=","deepspeed.initialize","(","model",",","...",")","trainer","=","Trainer","(","...","driver='deepspeed","'",",","device=None",",","#","fastNLP","\u4f1a\u5ffd\u7565\u4f20\u5165\u7684","device\uff0c\u5e76\u6839\u636e","local_rank","\u81ea\u52a8\u5206\u914d",")","trainer.run","(",")","\u901a\u8fc7\u8fd0\u884c","``","deepspeed","train.py","``","\u542f\u52a8\u3002",":","param","model",":","\u4f20\u5165\u7ed9","``","Trainer","``","\u7684","``","model","``","\u53c2\u6570\u3002",":","param","parallel_device",":","\u7528\u4e8e\u5206\u5e03\u5f0f\u8bad\u7ec3\u7684","``","gpu","``","\u8bbe\u5907\u3002",":","param","is_pull_by_torch_run",":","\u6807\u5fd7\u5f53\u524d\u7684\u811a\u672c\u7684\u542f\u52a8\u662f\u5426\u7531","``","python","-m","torch.distributed.launch","``","\u542f\u52a8\u7684\u3002",":","param","fp16",":","\u662f\u5426\u5f00\u542f","fp16","\u8bad\u7ec3\u3002",":","param","deepspeed_kwargs",":","*","*","strategy","*","--","\u4f7f\u7528","ZeRO","\u4f18\u5316\u7684\u7b56\u7565\uff0c\u9ed8\u8ba4\u4e3a","``","deepspeed","``","\uff1b\u76ee\u524d\u4ec5\u652f\u6301\u4ee5\u4e0b\u503c\uff1a","*","``","deepspeed","``","--","\u4f7f\u7528","ZeRO","\u7684\u7b2c\u4e8c\u9636\u6bb5\uff0c\u7b49\u540c\u4e8e","``","deepspeed_stage_2","``","\uff1b","*","``","deepspeed_stage_1","``","--","\u4f7f\u7528","ZeRO","\u7684\u7b2c\u4e00\u9636\u6bb5\uff0c\u4ec5\u5c06","``","optimizer","``","\u7684\u72b6\u6001\u5206\u6563\u5230\u4e0d\u540c\u8bbe\u5907\u4e0a\uff1b","*","``","deepspeed_stage_2","``","--","\u4f7f\u7528","ZeRO","\u7684\u7b2c\u4e8c\u9636\u6bb5\uff0c\u5c06","``","optimizer","``","\u548c","*","*","\u68af\u5ea6","*","*","\u5206\u6563\u5230\u4e0d\u540c\u8bbe\u5907\u4e0a\uff1b","*","``","deepspeed_stage_2_offload","``","--","\u4f7f\u7528","ZeRO","\u7684\u7b2c\u4e8c\u9636\u6bb5\uff0c\u5e76\u4e14\u501f\u52a9","cpu","\u7684\u5185\u5b58\u6765\u8fdb\u4e00\u6b65\u8282\u7ea6\u663e\u5b58\uff1b","*","``","deepspeed_stage_3","``","--","\u4f7f\u7528","ZeRO","\u7684\u7b2c\u4e09\u9636\u6bb5\uff0c\u5c06","``","optimizer","``","\u3001","*","*","\u68af\u5ea6","*","*","\u548c","*","*","\u6a21\u578b","*","*","\u5206\u6563\u5230\u4e0d\u540c\u8bbe\u5907\u4e0a\uff1b","*","``","deepspeed_stage_3_offload","``","--","\u4f7f\u7528","ZeRO","\u7684\u7b2c\u4e09\u9636\u6bb5\uff0c\u5e76\u4e14\u501f\u52a9","cpu","\u7684\u5185\u5b58\u6765\u8fdb\u4e00\u6b65\u8282\u7ea6\u663e\u5b58\uff1b","*","``","deepspeed_stage_3_offload_nvme","``","--","\u4f7f\u7528","ZeRO","\u7684\u7b2c\u4e09\u9636\u6bb5\uff0c\u5e76\u4e14\u501f\u52a9","NVMe","\u786c\u76d8\u6765\u8fdb\u4e00\u6b65\u8282\u7ea6\u663e\u5b58\uff1b","*","*","logging_level","*","--","``","deepspeed","``","\u5e93\u7684\u65e5\u5fd7\u7b49\u7ea7\uff0c\u9ed8\u8ba4\u4e3a","*","*","logging.ERROR","*","*","\u3002","*","*","config","*","--","``","deepspeed","``","\u7684\u5404\u9879\u8bbe\u7f6e\uff1b","*","*","FastNLP","*","*","\u5141\u8bb8\u7528\u6237\u4f20\u5165\u81ea\u5df1\u7684\u8bbe\u7f6e\u4ee5\u589e\u5f3a\u7075\u6d3b\u6027\uff0c\u4f46\u8fd9\u4f1a\u4f7f\u53c2\u6570","\u4e2d\u7684","``","optimizer","``","\u3001","``","strategy","``","\u3001","``","fp16","``","\u7b49\u5931\u6548\uff0c\u5373\u5f53\u8fd9\u4e2a\u53c2\u6570\u5b58\u5728\u65f6\uff0c","*","*","FastNLP","*","*","\u4f1a\u7528\u8be5\u53c2\u6570\u8986\u76d6","\u5176\u5b83\u7684\u8bbe\u7f6e\u3002",":","kwargs",":","*","*","accumulation_steps","*","--","\u5373\u5728",":","class",":","`","~fastNLP.core.controllers.Trainer","`","\u4f20\u5165\u7684","``","accumulation_steps","``","\u3002","deepspeed","\u4f1a\u5c06","``","config","``","\u7684","``","gradient_accumulation_steps","``","\u8bbe\u7f6e\u4e3a\u8be5\u503c\u3002","*","*","train_dataloader","*","--","\u5373\u5728",":","class",":","`","~fastNLP.core.controllers.Trainer","`","\u4f20\u5165\u7684","``","train_dataloader","``","\u3002","``","deepspeed","``","\u9700\u8981\u901a\u8fc7\u5b83\u6765\u83b7\u53d6","\u6570\u636e\u7684","``","batch_size","``","\u7528\u4e8e\u8bbe\u7f6e","``","train_micro_batch_size_per_gpu","``","\u3002\u5982\u679c\u6ca1\u6709\u4f20\u5165\u7684\u8bdd\uff0c\u5219\u4f1a\u8bbe\u7f6e\u4e3a","*","*","1","*","*","\u3002","*","*","wo_auto_param_call","*","(","``","bool","``",")","--","\u662f\u5426\u5173\u95ed\u5728\u8bad\u7ec3\u65f6\u8c03\u7528\u6211\u4eec\u7684","``","auto_param_call","``","\u51fd\u6570\u6765\u81ea\u52a8\u5339\u914d","batch","\u548c\u524d\u5411\u51fd\u6570\u7684\u53c2\u6570\u7684\u884c\u4e3a","..","note",":",":","\u5173\u4e8e\u8be5\u53c2\u6570\u7684\u8be6\u7ec6\u8bf4\u660e\uff0c\u8bf7\u53c2\u89c1",":","class",":","`","~fastNLP.core.controllers.Trainer","`","\u4e2d\u7684\u63cf\u8ff0\uff1b\u51fd\u6570","``","auto_param_call","``","\u8be6\u89c1",":","func",":","`","fastNLP.core.utils.auto_param_call","`","\u3002"]
docstr: 进行梯度置零操作；由于 :meth:`DeepSpeedEngine.step` 包含了 :meth:`zero_step` 的功能，因此该接口实际无意义。
docstr_tok: ["\u8fdb\u884c\u68af\u5ea6\u7f6e\u96f6\u64cd\u4f5c\uff1b\u7531\u4e8e",":","meth",":","`","DeepSpeedEngine.step","`","\u5305\u542b\u4e86",":","meth",":","`","zero_step","`","\u7684\u529f\u80fd\uff0c\u56e0\u6b64\u8be5\u63a5\u53e3\u5b9e\u9645\u65e0\u610f\u4e49\u3002"]
code: def zero_grad(self):
      """
      进行梯度置零操作；由于 :meth:`DeepSpeedEngine.step` 包含了 :meth:`zero_step` 的功能，因此该接口实际无意义。
      """
      # DeepSpeedEngine.step 包含了 zero_grad 功能
      pass
  
imports: import os
  import argparse
  import logging
  from pathlib import Path
  from typing import Union, Dict, List
  from .torch_driver import TorchDriver
  from .ddp import TorchDDPDriver
  from .utils import _create_default_config, _DeepSpeedWrappingModel
  from fastNLP.core.utils import nullcontext
  from fastNLP.core.log import logger
  from fastNLP.envs import FASTNLP_DISTRIBUTED_CHECK, FASTNLP_CHECKPOINT_FILENAME
  from fastNLP.envs.imports import _NEED_IMPORT_TORCH, _NEED_IMPORT_DEEPSPEED
label_desc: Use image node_id 6 for calling the DeepSpeedDriver obj's underlying member method code with example usage: obj.zero_grad() without return types