class_docstr: Proximal Policy Optimization
  
  See https://arxiv.org/abs/1707.06347
  
  Args:
      model (torch.nn.Module): Model to train (including recurrent models)
          state s  |->  (pi(s, _), v(s))
      optimizer (torch.optim.Optimizer): Optimizer used to train the model
      gpu (int): GPU device id if not None nor negative
      gamma (float): Discount factor [0, 1]
      lambd (float): Lambda-return factor [0, 1]
      phi (callable): Feature extractor function
      value_func_coef (float): Weight coefficient for loss of
          value function (0, inf)
      entropy_coef (float): Weight coefficient for entropy bonus [0, inf)
      update_interval (int): Model update interval in step
      minibatch_size (int): Minibatch size
      epochs (int): Training epochs in an update
      clip_eps (float): Epsilon for pessimistic clipping of likelihood ratio
          to update policy
      clip_eps_vf (float): Epsilon for pessimistic clipping of value
          to update value function. If it is ``None``, value function is not
          clipped on updates.
      standardize_advantages (bool): Use standardized advantages on updates
      recurrent (bool): If set to True, `model` is assumed to implement
          `pfrl.nn.Recurrent` and update in a recurrent
          manner.
      max_recurrent_sequence_len (int): Maximum length of consecutive
          sequences of transitions in a minibatch for updating the model.
          This value is used only when `recurrent` is True. A smaller value
          will encourage a minibatch to contain more and shorter sequences.
      act_deterministically (bool): If set to True, choose most probable
          actions in the act method instead of sampling from distributions.
      max_grad_norm (float or None): Maximum L2 norm of the gradient used for
          gradient clipping. If set to None, the gradient is not clipped.
      value_stats_window (int): Window size used to compute statistics
          of value predictions.
      entropy_stats_window (int): Window size used to compute statistics
          of entropy of action distributions.
      value_loss_stats_window (int): Window size used to compute statistics
          of loss values regarding the value function.
      policy_loss_stats_window (int): Window size used to compute statistics
          of loss values regarding the policy.
  
  Statistics:
      average_value: Average of value predictions on non-terminal states.
          It's updated on (batch_)act_and_train.
      average_entropy: Average of entropy of action distributions on
          non-terminal states. It's updated on (batch_)act_and_train.
      average_value_loss: Average of losses regarding the value function.
          It's updated after the model is updated.
      average_policy_loss: Average of losses regarding the policy.
          It's updated after the model is updated.
      n_updates: Number of model updates so far.
      explained_variance: Explained variance computed from the last batch.
docstr: Update both the policy and the value function.
docstr_tok: ["Update","both","the","policy","and","the","value","function","."]
code: def _update(self, dataset):
      """Update both the policy and the value function."""
  
      device = self.device
  
      if self.obs_normalizer:
          self._update_obs_normalizer(dataset)
  
      assert "state" in dataset[0]
      assert "v_teacher" in dataset[0]
  
      if self.standardize_advantages:
          all_advs = torch.tensor(
              [b["adv"] for b in dataset], device=device
          )
          std_advs, mean_advs = torch.std_mean(all_advs, unbiased=False)
  
      for batch in _yield_minibatches(
          dataset,
          minibatch_size=self.minibatch_size,
          num_epochs=self.epochs,
      ):
          states = self.batch_states(
              [b["state"] for b in batch], self.device, self.phi
          )
          if self.obs_normalizer:
              states = self.obs_normalizer(states, update=False)
          actions = torch.tensor(
              [b["action"] for b in batch], device=device
          )
          distribs, vs_pred = self.model(states)
  
          advs = torch.tensor(
              [b["adv"] for b in batch],
              dtype=torch.float32,
              device=device,
          )
          if self.standardize_advantages:
              advs = (advs - mean_advs) / (std_advs + 1e-8)
  
          log_probs_old = torch.tensor(
              [b["log_prob"] for b in batch],
              dtype=torch.float,
              device=device,
          )
          vs_pred_old = torch.tensor(
              [b["v_pred"] for b in batch],
              dtype=torch.float,
              device=device,
          )
          vs_teacher = torch.tensor(
              [b["v_teacher"] for b in batch],
              dtype=torch.float,
              device=device,
          )
          # Same shape as vs_pred: (batch_size, 1)
          vs_pred_old = vs_pred_old[..., None]
          vs_teacher = vs_teacher[..., None]
  
          self.model.zero_grad()
          loss = self._lossfun(
              distribs.entropy(),
              vs_pred,
              distribs.log_prob(actions),
              vs_pred_old=vs_pred_old,
              log_probs_old=log_probs_old,
              advs=advs,
              vs_teacher=vs_teacher,
          )
          loss.backward()
          if self.max_grad_norm is not None:
              torch.nn.utils.clip_grad_norm_(
                  self.model.parameters(), self.max_grad_norm
              )
          self.optimizer.step()
          self.n_updates += 1
  
imports: import collections
  import itertools
  import random
  import numpy
  import torch
  import torch.nn.functional
  import pfrl
  from pfrl import agent
  from pfrl.utils.batch_states import batch_states
  from pfrl.utils.mode_of_distribution import mode_of_distribution
  from pfrl.utils.recurrent import concatenate_recurrent_states, flatten_sequences_time_first, get_recurrent_state_at, mask_recurrent_state_at, one_step_forward, pack_and_forward
label_desc: Use image node_id 6 for calling the PPO obj's underlying member method code with example usage: obj._update(dataset) without return types