class_docstr: A ProcessGroupInitializer for sequence parallelism.
  
  Args:
      rank (int): The rank of current process.
      world_size (int): Size of whole communication world.
      config (Config): Running configuration.
      data_parallel_size (int): Size of data parallel.
      pipeline_parallel_size (int): Size of pipeline parallel.
      tensor_parallel_size (int): Size of tensor parallel.
class_docstr_tok: ["A","ProcessGroupInitializer","for","sequence","parallelism",".","Args",":","rank","(","int",")",":","The","rank","of","current","process",".","world_size","(","int",")",":","Size","of","whole","communication","world",".","config","(","Config",")",":","Running","configuration",".","data_parallel_size","(","int",")",":","Size","of","data","parallel",".","pipeline_parallel_size","(","int",")",":","Size","of","pipeline","parallel",".","tensor_parallel_size","(","int",")",":","Size","of","tensor","parallel","."]
docstr: Initialize Sequence parallel process groups and assign local_ranks and groups to each gpu.
  
  Sequence parallelism requires 2 process groups. The first is for model forward where several processes
  exchange partial query, key and value embedding to compute self attention values. The second is for
  all-reduce to synchronize the model parameters.
  
  Returns:
      List[Tuple (local_rank, group_world_size, process_group, ranks_in_group, mode)]:
          A Sequence parallelism's information in list of tuples.
docstr_tok: ["Initialize","Sequence","parallel","process","groups","and","assign","local_ranks","and","groups","to","each","gpu",".","Sequence","parallelism","requires","2","process","groups",".","The","first","is","for","model","forward","where","several","processes","exchange","partial","query",",","key","and","value","embedding","to","compute","self","attention","values",".","The","second","is","for","all-reduce","to","synchronize","the","model","parameters",".","Returns",":","List","[","Tuple","(","local_rank",",","group_world_size",",","process_group",",","ranks_in_group",",","mode",")","]",":","A","Sequence","parallelism","'s","information","in","list","of","tuples","."]
code: def init_dist_group(self):
      """Initialize Sequence parallel process groups and assign local_ranks and groups to each gpu.
  
      Sequence parallelism requires 2 process groups. The first is for model forward where several processes
      exchange partial query, key and value embedding to compute self attention values. The second is for
      all-reduce to synchronize the model parameters.
  
      Returns:
          List[Tuple (local_rank, group_world_size, process_group, ranks_in_group, mode)]:
              A Sequence parallelism's information in list of tuples.
      """
  
      parallel_setting = []
  
      (
          local_rank,
          group_world_size,
          process_group,
          cpu_group,
          ranks_in_group,
          mode,
      ) = self._sequence_initializer.init_dist_group()
      # change mode to sequence
      mode = ParallelMode.SEQUENCE
  
      parallel_setting.append(
          (
              local_rank,
              group_world_size,
              process_group,
              cpu_group,
              ranks_in_group,
              mode,
          )
      )
      parallel_setting.append(
          self._sequence_dp_initializer.init_dist_group()
      )
      return parallel_setting
  
file_path: ColossalAI/colossalai/legacy/context/process_group_initializer/initializer_sequence.py
imports: import torch.distributed
  from colossalai.legacy.registry import DIST_GROUP_INITIALIZER
  from ..parallel_mode import ParallelMode
  from .initializer_tensor import Initializer_Tensor
  from .process_group_initializer import ProcessGroupInitializer
label_desc: Use image node_id 2 for calling the Initializer_Sequence obj's underlying member method code with example usage: obj.init_dist_group() and returns: parallel_setting