class_docstr: Proximal Policy Optimization
  
  See https://arxiv.org/abs/1707.06347
  
  Args:
      model (torch.nn.Module): Model to train (including recurrent models)
          state s  |->  (pi(s, _), v(s))
      optimizer (torch.optim.Optimizer): Optimizer used to train the model
      gpu (int): GPU device id if not None nor negative
      gamma (float): Discount factor [0, 1]
      lambd (float): Lambda-return factor [0, 1]
      phi (callable): Feature extractor function
      value_func_coef (float): Weight coefficient for loss of
          value function (0, inf)
      entropy_coef (float): Weight coefficient for entropy bonus [0, inf)
      update_interval (int): Model update interval in step
      minibatch_size (int): Minibatch size
      epochs (int): Training epochs in an update
      clip_eps (float): Epsilon for pessimistic clipping of likelihood ratio
          to update policy
      clip_eps_vf (float): Epsilon for pessimistic clipping of value
          to update value function. If it is ``None``, value function is not
          clipped on updates.
      standardize_advantages (bool): Use standardized advantages on updates
      recurrent (bool): If set to True, `model` is assumed to implement
          `pfrl.nn.Recurrent` and update in a recurrent
          manner.
      max_recurrent_sequence_len (int): Maximum length of consecutive
          sequences of transitions in a minibatch for updating the model.
          This value is used only when `recurrent` is True. A smaller value
          will encourage a minibatch to contain more and shorter sequences.
      act_deterministically (bool): If set to True, choose most probable
          actions in the act method instead of sampling from distributions.
      max_grad_norm (float or None): Maximum L2 norm of the gradient used for
          gradient clipping. If set to None, the gradient is not clipped.
      value_stats_window (int): Window size used to compute statistics
          of value predictions.
      entropy_stats_window (int): Window size used to compute statistics
          of entropy of action distributions.
      value_loss_stats_window (int): Window size used to compute statistics
          of loss values regarding the value function.
      policy_loss_stats_window (int): Window size used to compute statistics
          of loss values regarding the policy.
  
  Statistics:
      average_value: Average of value predictions on non-terminal states.
          It's updated on (batch_)act_and_train.
      average_entropy: Average of entropy of action distributions on
          non-terminal states. It's updated on (batch_)act_and_train.
      average_value_loss: Average of losses regarding the value function.
          It's updated after the model is updated.
      average_policy_loss: Average of losses regarding the policy.
          It's updated after the model is updated.
      n_updates: Number of model updates so far.
      explained_variance: Explained variance computed from the last batch.
code: def get_statistics(self):
      return [
          ("average_value", _mean_or_nan(self.value_record)),
          ("average_entropy", _mean_or_nan(self.entropy_record)),
          ("average_value_loss", _mean_or_nan(self.value_loss_record)),
          (
              "average_policy_loss",
              _mean_or_nan(self.policy_loss_record),
          ),
          ("n_updates", self.n_updates),
          ("explained_variance", self.explained_variance),
      ]
  
imports: import collections
  import itertools
  import random
  import numpy
  import torch
  import torch.nn.functional
  import pfrl
  from pfrl import agent
  from pfrl.utils.batch_states import batch_states
  from pfrl.utils.mode_of_distribution import mode_of_distribution
  from pfrl.utils.recurrent import concatenate_recurrent_states, flatten_sequences_time_first, get_recurrent_state_at, mask_recurrent_state_at, one_step_forward, pack_and_forward
label_desc: Use image node_id 16 for calling the PPO obj's underlying member method code with example usage: obj.get_statistics() and returns: list