class_docstr:     Trainer for PPO algorithm.
  
  Args:
      strategy (Strategy): the strategy to use for training
      actor (Actor): the actor model in ppo algorithm
      critic (Critic): the critic model in ppo algorithm
      reward_model (RewardModel): the reward model in rlhf algorithm to make reward of sentences
      initial_model (Actor): the initial model in rlhf algorithm to generate reference logics to limit the update of actor
      actor_optim (Optimizer): the optimizer to use for actor model
      critic_optim (Optimizer): the optimizer to use for critic model
      kl_coef (float, defaults to 0.1): the coefficient of kl divergence loss
      train_batch_size (int, defaults to 8): the batch size to use for training
      buffer_limit (int, defaults to 0): the max_size limitation of buffer
      buffer_cpu_offload (bool, defaults to True): whether to offload buffer to cpu
      eps_clip (float, defaults to 0.2): the clip coefficient of policy loss
      vf_coef (float, defaults to 1.0): the coefficient of value loss
      ptx_coef (float, defaults to 0.9): the coefficient of ptx loss
      value_clip (float, defaults to 0.4): the clip coefficient of value loss
      sample_buffer (bool, defaults to False): whether to sample from buffer
      dataloader_pin_memory (bool, defaults to True): whether to pin memory for data loader
      offload_inference_models (bool, defaults to True): whether to offload inference models to cpu during training process
      callbacks (List[Callback], defaults to []): the callbacks to call during training process
      generate_kwargs (dict, optional): the kwargs to use while model generating
docstr: Args:
      prompt_dataloader (DataLoader): the dataloader to use for prompt data
      pretrain_dataloader (DataLoader): the dataloader to use for pretrain data
docstr_tok: ["Args",":","prompt_dataloader","(","DataLoader",")",":","the","dataloader","to","use","for","prompt","data","pretrain_dataloader","(","DataLoader",")",":","the","dataloader","to","use","for","pretrain","data"]
code: def _before_fit(
      self,
      prompt_dataloader: DataLoader,
      pretrain_dataloader: DataLoader,
      log_dir: Optional[str] = None,
      use_wandb: bool = False,
  ):
      """
      Args:
          prompt_dataloader (DataLoader): the dataloader to use for prompt data
          pretrain_dataloader (DataLoader): the dataloader to use for pretrain data
      """
      self.prompt_dataloader = CycledDataLoader(prompt_dataloader)
      self.pretrain_dataloader = CycledDataLoader(pretrain_dataloader)
  
      self.writer = None
      if use_wandb and is_rank_0():
          assert (
              log_dir is not None
          ), "log_dir must be provided when use_wandb is True"
          import wandb
  
          wandb.init(project="Coati-ppo", sync_tensorboard=True)
      if log_dir is not None and is_rank_0():
          import os
          import time
  
          from torch.utils.tensorboard import SummaryWriter
  
          log_dir = os.path.join(log_dir, "ppo")
          log_dir = os.path.join(
              log_dir,
              time.strftime("%Y-%m-%d_%H:%M:%S", time.localtime()),
          )
          self.writer = SummaryWriter(log_dir=log_dir)
  
imports: from typing import Dict, List, Optional
  from coati.experience_buffer import NaiveExperienceBuffer
  from coati.experience_maker import Experience, NaiveExperienceMaker
  from coati.models.base import Actor, Critic, RewardModel, get_base_model
  from coati.models.loss import GPTLMLoss, PolicyLoss, ValueLoss
  from coati.models.utils import calc_action_log_probs
  from torch.optim import Optimizer
  from torch.utils.data import DataLoader, DistributedSampler
  from tqdm import tqdm
  from transformers import PreTrainedTokenizerBase
  from colossalai.utils import get_current_device
  from .base import OnPolicyTrainer
  from .callbacks import Callback
  from .strategies import GeminiStrategy, Strategy
  from .utils import CycledDataLoader, is_rank_0, to_device
label_desc: Use image node_id 2 for calling the PPOTrainer obj's underlying member method code with example usage: obj._before_fit(prompt_dataloader, pretrain_dataloader, log_dir, use_wandb) without return types