class_docstr: CheckpointIO for Hybrid Parallel Training.
  
  Args:
      dp_group (ProcessGroup): Process group along data parallel dimension.
      pp_group (ProcessGroup): Process group along pipeline parallel dimension.
      tp_group (ProcessGroup): Process group along tensor parallel dimension.
      zero_stage (int): The zero stage of plugin. Should be in [0, 1, 2].
      verbose (bool, optional): Whether to print logging massage when saving/loading has been succesfully executed. Defaults to True.
class_docstr_tok: ["CheckpointIO","for","Hybrid","Parallel","Training",".","Args",":","dp_group","(","ProcessGroup",")",":","Process","group","along","data","parallel","dimension",".","pp_group","(","ProcessGroup",")",":","Process","group","along","pipeline","parallel","dimension",".","tp_group","(","ProcessGroup",")",":","Process","group","along","tensor","parallel","dimension",".","zero_stage","(","int",")",":","The","zero","stage","of","plugin",".","Should","be","in","[","0",",","1",",","2","]",".","verbose","(","bool",",","optional",")",":","Whether","to","print","logging","massage","when","saving\/loading","has","been","succesfully","executed",".","Defaults","to","True","."]
code: def __init__(
      self,
      dp_group: ProcessGroup,
      pp_group: ProcessGroup,
      tp_group: ProcessGroup,
      zero_stage: int,
      verbose: bool = True,
  ) -> None:
      super().__init__()
      self.dp_group = dp_group
      self.pp_group = pp_group
      self.tp_group = tp_group
      self.dp_rank = dist.get_rank(self.dp_group)
      self.tp_rank = dist.get_rank(self.tp_group)
      self.pp_rank = dist.get_rank(self.pp_group)
      self.dp_size = dist.get_world_size(dp_group)
      self.pp_size = dist.get_world_size(pp_group)
      self.tp_size = dist.get_world_size(tp_group)
      self.use_zero = zero_stage > 0
      self.verbose = verbose
      self.coordinator = DistCoordinator()
  
imports: import copy
  from functools import reduce
  import logging
  import os
  from pathlib import Path
  from shutil import rmtree
  from typing import Dict, Iterator, Optional, OrderedDict, Tuple
  import torch
  import torch.distributed
  import torch.nn
  from torch.distributed import ProcessGroup
  from torch.optim.lr_scheduler import _LRScheduler
  from colossalai.cluster import DistCoordinator
  from colossalai.interface import ModelWrapper, OptimizerWrapper
  from .general_checkpoint_io import GeneralCheckpointIO
  from .index_file import CheckpointIndexFile
  from .utils import StateDictSharder, gather_distributed_param, get_model_base_filenames, get_optimizer_base_filenames, is_safetensors_available, load_shard_state_dict, load_state_dict, load_state_dict_into_model, load_states_into_optimizer, save_config_file, save_param_groups, save_state_dict, save_state_dict_shards, search_tp_partition_dim, sharded_optimizer_loading_epilogue
label_desc: Use image node_id 1 to create a new HybridParallelCheckpointIO object from inherited base classes: GeneralCheckpointIO with example: obj = HybridParallelCheckpointIO(dp_group, pp_group, tp_group, zero_stage, verbose)