class_docstr: Proximal Policy Optimization (PPO). See https://arxiv.org/pdf/1707.06347.pdf
  This is the "clip" version of PPO. It does not include:
  - KL divergence
  - Bootstrapping with a critic model (our approach only works if full trajectories up to terminal state are fed in)
  Optionally, a value network can be trained and used as a baseline for rewards.
code: def training_step(
      self,
      training_batch: Union[
          rlt.PolicyGradientInput, Dict[str, torch.Tensor]
      ],
      batch_idx: int,
  ):
      if isinstance(training_batch, dict):
          training_batch = rlt.PolicyGradientInput.from_dict(
              training_batch
          )
  
      self.traj_buffer.append(training_batch)
      if len(self.traj_buffer) == self.update_freq:
          self.update_model()
  
imports: import inspect
  import logging
  from dataclasses import field
  from typing import Dict, List, Optional, Union
  import reagent.core.types
  import torch
  import torch.optim
  from reagent.core.configuration import resolve_defaults
  from reagent.gym.policies.policy import Policy
  from reagent.models.base import ModelBase
  from reagent.optimizer.union import Optimizer__Union
  from reagent.training.reagent_lightning_module import ReAgentLightningModule
  from reagent.training.utils import discounted_returns, whiten
label_desc: Use image node_id 5 for calling the PPOTrainer obj's underlying member method code with example usage: obj.training_step(training_batch, batch_idx) without return types