class_docstr: Proximal Policy Optimization (PPO). See https://arxiv.org/pdf/1707.06347.pdf
  This is the "clip" version of PPO. It does not include:
  - KL divergence
  - Bootstrapping with a critic model (our approach only works if full trajectories up to terminal state are fed in)
  Optionally, a value network can be trained and used as a baseline for rewards.
code: def update_model(self):
      assert (
          len(self.traj_buffer) == self.update_freq
      ), "trajectory buffer does not have sufficient samples for model_update"
      for _ in range(self.update_epochs):
          # iterate through minibatches of PPO updates in random order
          random_order = torch.randperm(len(self.traj_buffer))
          for i in range(0, len(self.traj_buffer), self.ppo_batch_size):
              idx = random_order[i : i + self.ppo_batch_size]
              training_batch_list = [self.traj_buffer[i] for i in idx]
              self._update_model(training_batch_list)
  
      self.traj_buffer = []
  
imports: import inspect
  import logging
  from dataclasses import field
  from typing import Dict, List, Optional, Union
  import reagent.core.types
  import torch
  import torch.optim
  from reagent.core.configuration import resolve_defaults
  from reagent.gym.policies.policy import Policy
  from reagent.models.base import ModelBase
  from reagent.optimizer.union import Optimizer__Union
  from reagent.training.reagent_lightning_module import ReAgentLightningModule
  from reagent.training.utils import discounted_returns, whiten
label_desc: Use image node_id 6 for calling the PPOTrainer obj's underlying member method code with example usage: obj.update_model() without return types