class_docstr: Proximal Policy Optimization
  
  See https://arxiv.org/abs/1707.06347
  
  Args:
      model (torch.nn.Module): Model to train (including recurrent models)
          state s  |->  (pi(s, _), v(s))
      optimizer (torch.optim.Optimizer): Optimizer used to train the model
      gpu (int): GPU device id if not None nor negative
      gamma (float): Discount factor [0, 1]
      lambd (float): Lambda-return factor [0, 1]
      phi (callable): Feature extractor function
      value_func_coef (float): Weight coefficient for loss of
          value function (0, inf)
      entropy_coef (float): Weight coefficient for entropy bonus [0, inf)
      update_interval (int): Model update interval in step
      minibatch_size (int): Minibatch size
      epochs (int): Training epochs in an update
      clip_eps (float): Epsilon for pessimistic clipping of likelihood ratio
          to update policy
      clip_eps_vf (float): Epsilon for pessimistic clipping of value
          to update value function. If it is ``None``, value function is not
          clipped on updates.
      standardize_advantages (bool): Use standardized advantages on updates
      recurrent (bool): If set to True, `model` is assumed to implement
          `pfrl.nn.Recurrent` and update in a recurrent
          manner.
      max_recurrent_sequence_len (int): Maximum length of consecutive
          sequences of transitions in a minibatch for updating the model.
          This value is used only when `recurrent` is True. A smaller value
          will encourage a minibatch to contain more and shorter sequences.
      act_deterministically (bool): If set to True, choose most probable
          actions in the act method instead of sampling from distributions.
      max_grad_norm (float or None): Maximum L2 norm of the gradient used for
          gradient clipping. If set to None, the gradient is not clipped.
      value_stats_window (int): Window size used to compute statistics
          of value predictions.
      entropy_stats_window (int): Window size used to compute statistics
          of entropy of action distributions.
      value_loss_stats_window (int): Window size used to compute statistics
          of loss values regarding the value function.
      policy_loss_stats_window (int): Window size used to compute statistics
          of loss values regarding the policy.
  
  Statistics:
      average_value: Average of value predictions on non-terminal states.
          It's updated on (batch_)act_and_train.
      average_entropy: Average of entropy of action distributions on
          non-terminal states. It's updated on (batch_)act_and_train.
      average_value_loss: Average of losses regarding the value function.
          It's updated after the model is updated.
      average_policy_loss: Average of losses regarding the policy.
          It's updated after the model is updated.
      n_updates: Number of model updates so far.
      explained_variance: Explained variance computed from the last batch.
code: def _batch_observe_train(
      self, batch_obs, batch_reward, batch_done, batch_reset
  ):
      assert self.training
  
      for i, (
          state,
          action,
          reward,
          next_state,
          done,
          reset,
      ) in enumerate(
          zip(
              self.batch_last_state,
              self.batch_last_action,
              batch_reward,
              batch_obs,
              batch_done,
              batch_reset,
          )
      ):
          if state is not None:
              assert action is not None
              transition = {
                  "state": state,
                  "action": action,
                  "reward": reward,
                  "next_state": next_state,
                  "nonterminal": 0.0 if done else 1.0,
              }
              if self.recurrent:
                  transition[
                      "recurrent_state"
                  ] = get_recurrent_state_at(
                      self.train_prev_recurrent_states, i, detach=True
                  )
                  transition[
                      "next_recurrent_state"
                  ] = get_recurrent_state_at(
                      self.train_recurrent_states, i, detach=True
                  )
              self.batch_last_episode[i].append(transition)
          if done or reset:
              assert self.batch_last_episode[i]
              self.memory.append(self.batch_last_episode[i])
              self.batch_last_episode[i] = []
          self.batch_last_state[i] = None
          self.batch_last_action[i] = None
  
      self.train_prev_recurrent_states = None
  
      if self.recurrent:
          # Reset recurrent states when episodes end
          indices_that_ended = [
              i
              for i, (done, reset) in enumerate(
                  zip(batch_done, batch_reset)
              )
              if done or reset
          ]
          if indices_that_ended:
              self.train_recurrent_states = mask_recurrent_state_at(
                  self.train_recurrent_states, indices_that_ended
              )
  
      self._update_if_dataset_is_ready()
  
imports: import collections
  import itertools
  import random
  import numpy
  import torch
  import torch.nn.functional
  import pfrl
  from pfrl import agent
  from pfrl.utils.batch_states import batch_states
  from pfrl.utils.mode_of_distribution import mode_of_distribution
  from pfrl.utils.recurrent import concatenate_recurrent_states, flatten_sequences_time_first, get_recurrent_state_at, mask_recurrent_state_at, one_step_forward, pack_and_forward
label_desc: Use image node_id 15 for calling the PPO obj's underlying member method code with example usage: obj._batch_observe_train(batch_obs, batch_reward, batch_done, batch_reset) without return types