class_docstr: CaiInferEngine is a class that handles the pipeline parallel inference.
  
  Args:
      tp_size (int): the size of tensor parallelism.
      pp_size (int): the size of pipeline parallelism.
      model (`nn.Module`): the model not in pipeline style, and will be modified with `ShardFormer`.
      model_policy (`colossalai.shardformer.policies.base_policy.Policy`): the policy to shardformer model.
      micro_batch_size (int): the micro batch size.
      micro_batch_buffer_size (int): the buffer size for micro batch. Normally, it should be the same as the number of pipeline stages.
      max_batch_size (int): the maximum batch size.
      max_input_len (int): the maximum input length.
      max_output_len (int): the maximum output length.
  
  Example:
  
  ```python
  from colossalai.inference import InferEngine
  from colossalai.inference.pipeline.policies import LlamaModelInferPolicy
  import colossalai
  from transformers import LlamaForCausalLM, LlamaTokenizer
  
  colossalai.launch_from_torch(config={})
  
  model = LlamaForCausalLM.from_pretrained("your_path_to_model")
  tokenizer = LlamaTokenizer.from_pretrained("/home/lczyh/share/models/llama-7b-hf")
  # assume the model is infered with 2 pipeline stages
  inferengine = CaiInferEngine(pp_size=2, model=model, model_policy=LlamaModelInferPolicy())
  
  input = ["Introduce a landmark in China ","Introduce a landmark in China "]
  data = tokenizer(input, return_tensors='pt')
  output = inferengine.inference([data.to('cuda').data])
  
  ```
code: def _shardformer(self, model, model_policy, stage_manager, tp_group):
      shardconfig = ShardConfig(
          tensor_parallel_process_group=tp_group,
          pipeline_stage_manager=stage_manager,
          enable_tensor_parallelism=False,
          enable_fused_normalization=False,
          enable_all_optimization=False,
          enable_flash_attention=False,
          enable_jit_fused=False,
          enable_sequence_parallelism=False,
      )
      shardformer = ShardFormer(shard_config=shardconfig)
      shard_model, _ = shardformer.optimize(model, model_policy)
      return shard_model.cuda()
  
file_path: ColossalAI/colossalai/legacy/inference/hybridengine/engine.py
imports: import torch
  import torch.distributed
  import torch.nn
  from transformers.tokenization_utils_base import BatchEncoding
  from colossalai.cluster import ProcessGroupMesh
  from colossalai.pipeline.schedule.generate import GenerateSchedule
  from colossalai.pipeline.stage_manager import PipelineStageManager
  from colossalai.shardformer import ShardConfig, ShardFormer
  from colossalai.shardformer.policies.base_policy import Policy
  from ..pipeline.microbatch_manager import MicroBatchManager
  from ..tensor_parallel.kvcache_manager import MemoryManager
label_desc: Use image node_id 3 for calling the CaiInferEngine obj's underlying member method code with example usage: obj._shardformer(model, model_policy, stage_manager, tp_group) and returns: shard_model