class_docstr:     Trainer for PPO algorithm.
  
  Args:
      strategy (Strategy): the strategy to use for training
      actor (Actor): the actor model in ppo algorithm
      critic (Critic): the critic model in ppo algorithm
      reward_model (RewardModel): the reward model in rlhf algorithm to make reward of sentences
      initial_model (Actor): the initial model in rlhf algorithm to generate reference logics to limit the update of actor
      actor_optim (Optimizer): the optimizer to use for actor model
      critic_optim (Optimizer): the optimizer to use for critic model
      kl_coef (float, defaults to 0.1): the coefficient of kl divergence loss
      train_batch_size (int, defaults to 8): the batch size to use for training
      buffer_limit (int, defaults to 0): the max_size limitation of buffer
      buffer_cpu_offload (bool, defaults to True): whether to offload buffer to cpu
      eps_clip (float, defaults to 0.2): the clip coefficient of policy loss
      vf_coef (float, defaults to 1.0): the coefficient of value loss
      ptx_coef (float, defaults to 0.9): the coefficient of ptx loss
      value_clip (float, defaults to 0.4): the clip coefficient of value loss
      sample_buffer (bool, defaults to False): whether to sample from buffer
      dataloader_pin_memory (bool, defaults to True): whether to pin memory for data loader
      offload_inference_models (bool, defaults to True): whether to offload inference models to cpu during training process
      callbacks (List[Callback], defaults to []): the callbacks to call during training process
      generate_kwargs (dict, optional): the kwargs to use while model generating
code: def _make_experience(self, collect_step: int) -> Experience:
      prompts = self.prompt_dataloader.next()
      if self.offload_inference_models:
          # TODO(ver217): this may be controlled by strategy if they are prepared by strategy
          self.experience_maker.initial_model.to(self.device)
          self.experience_maker.reward_model.to(self.device)
      assert isinstance(
          prompts, dict
      ), f'Unsupported input type "{type(prompts)}"'
      return self.experience_maker.make_experience(
          **prompts, **self.generate_kwargs
      )
  
imports: from typing import Dict, List, Optional
  from coati.experience_buffer import NaiveExperienceBuffer
  from coati.experience_maker import Experience, NaiveExperienceMaker
  from coati.models.base import Actor, Critic, RewardModel, get_base_model
  from coati.models.loss import GPTLMLoss, PolicyLoss, ValueLoss
  from coati.models.utils import calc_action_log_probs
  from torch.optim import Optimizer
  from torch.utils.data import DataLoader, DistributedSampler
  from tqdm import tqdm
  from transformers import PreTrainedTokenizerBase
  from colossalai.utils import get_current_device
  from .base import OnPolicyTrainer
  from .callbacks import Callback
  from .strategies import GeminiStrategy, Strategy
  from .utils import CycledDataLoader, is_rank_0, to_device
label_desc: Use image node_id 3 for calling the PPOTrainer obj's underlying member method code with example usage: obj._make_experience(collect_step) and returns: self