class_docstr: Vanilla Policy Gradient (REINFORCE).
  
  VPG, also known as Reinforce, trains stochastic policy in an on-policy way.
  
  Args:
      env_spec (EnvSpec): Environment specification.
      policy (garage.torch.policies.Policy): Policy.
      value_function (garage.torch.value_functions.ValueFunction): The value
          function.
      sampler (garage.sampler.Sampler): Sampler.
      policy_optimizer (garage.torch.optimizer.OptimizerWrapper): Optimizer
          for policy.
      vf_optimizer (garage.torch.optimizer.OptimizerWrapper): Optimizer for
          value function.
      num_train_per_epoch (int): Number of train_once calls per epoch.
      discount (float): Discount.
      gae_lambda (float): Lambda used for generalized advantage
          estimation.
      center_adv (bool): Whether to rescale the advantages
          so that they have mean 0 and standard deviation 1.
      positive_adv (bool): Whether to shift the advantages
          so that they are always positive. When used in
          conjunction with center_adv the advantages will be
          standardized before shifting.
      policy_ent_coeff (float): The coefficient of the policy entropy.
          Setting it to zero would mean no entropy regularization.
      use_softplus_entropy (bool): Whether to estimate the softmax
          distribution of the entropy to prevent the entropy from being
          negative.
      stop_entropy_gradient (bool): Whether to stop the entropy gradient.
      entropy_method (str): A string from: 'max', 'regularized',
          'no_entropy'. The type of entropy method to use. 'max' adds the
          dense entropy to the reward for each time step. 'regularized' adds
          the mean entropy to the surrogate objective. See
          https://arxiv.org/abs/1805.00909 for more details.
class_docstr_tok: ["Vanilla","Policy","Gradient","(","REINFORCE",")",".","VPG",",","also","known","as","Reinforce",",","trains","stochastic","policy","in","an","on-policy","way",".","Args",":","env_spec","(","EnvSpec",")",":","Environment","specification",".","policy","(","garage.torch.policies.Policy",")",":","Policy",".","value_function","(","garage.torch.value_functions.ValueFunction",")",":","The","value","function",".","sampler","(","garage.sampler.Sampler",")",":","Sampler",".","policy_optimizer","(","garage.torch.optimizer.OptimizerWrapper",")",":","Optimizer","for","policy",".","vf_optimizer","(","garage.torch.optimizer.OptimizerWrapper",")",":","Optimizer","for","value","function",".","num_train_per_epoch","(","int",")",":","Number","of","train_once","calls","per","epoch",".","discount","(","float",")",":","Discount",".","gae_lambda","(","float",")",":","Lambda","used","for","generalized","advantage","estimation",".","center_adv","(","bool",")",":","Whether","to","rescale","the","advantages","so","that","they","have","mean","0","and","standard","deviation","1.","positive_adv","(","bool",")",":","Whether","to","shift","the","advantages","so","that","they","are","always","positive",".","When","used","in","conjunction","with","center_adv","the","advantages","will","be","standardized","before","shifting",".","policy_ent_coeff","(","float",")",":","The","coefficient","of","the","policy","entropy",".","Setting","it","to","zero","would","mean","no","entropy","regularization",".","use_softplus_entropy","(","bool",")",":","Whether","to","estimate","the","softmax","distribution","of","the","entropy","to","prevent","the","entropy","from","being","negative",".","stop_entropy_gradient","(","bool",")",":","Whether","to","stop","the","entropy","gradient",".","entropy_method","(","str",")",":","A","string","from",":","'max","'",",","'regularized","'",",","'no_entropy","'",".","The","type","of","entropy","method","to","use",".","'max","'","adds","the","dense","entropy","to","the","reward","for","each","time","step",".","'regularized","'","adds","the","mean","entropy","to","the","surrogate","objective",".","See","https",":","\/\/arxiv.org\/abs\/1805.00909","for","more","details","."]
docstr: Train the policy.
  
  Args:
      obs (torch.Tensor): Observation from the environment
          with shape :math:`(N, O*)`.
      actions (torch.Tensor): Actions fed to the environment
          with shape :math:`(N, A*)`.
      rewards (torch.Tensor): Acquired rewards
          with shape :math:`(N, )`.
      advantages (torch.Tensor): Advantage value at each step
          with shape :math:`(N, )`.
  
  Returns:
      torch.Tensor: Calculated mean scalar value of policy loss (float).
docstr_tok: ["Train","the","policy",".","Args",":","obs","(","torch.Tensor",")",":","Observation","from","the","environment","with","shape",":","math",":","`","(","N",",","O","*",")","`",".","actions","(","torch.Tensor",")",":","Actions","fed","to","the","environment","with","shape",":","math",":","`","(","N",",","A","*",")","`",".","rewards","(","torch.Tensor",")",":","Acquired","rewards","with","shape",":","math",":","`","(","N",",",")","`",".","advantages","(","torch.Tensor",")",":","Advantage","value","at","each","step","with","shape",":","math",":","`","(","N",",",")","`",".","Returns",":","torch.Tensor",":","Calculated","mean","scalar","value","of","policy","loss","(","float",")","."]
code: def _train_policy(self, obs, actions, rewards, advantages):
      r"""Train the policy.
  
      Args:
          obs (torch.Tensor): Observation from the environment
              with shape :math:`(N, O*)`.
          actions (torch.Tensor): Actions fed to the environment
              with shape :math:`(N, A*)`.
          rewards (torch.Tensor): Acquired rewards
              with shape :math:`(N, )`.
          advantages (torch.Tensor): Advantage value at each step
              with shape :math:`(N, )`.
  
      Returns:
          torch.Tensor: Calculated mean scalar value of policy loss (float).
  
      """
      # pylint: disable=protected-access
      zero_optim_grads(self._policy_optimizer._optimizer)
      loss = self._compute_loss_with_adv(
          obs, actions, rewards, advantages
      )
      loss.backward()
      self._policy_optimizer.step()
  
      return loss
  
imports: import collections
  import copy
  from dowel import tabular
  import numpy
  import torch
  import torch.nn.functional
  from garage import log_performance
  from garage.np import discount_cumsum
  from garage.np.algos import RLAlgorithm
  from garage.torch import compute_advantages, filter_valids
  from garage.torch._functions import np_to_torch, zero_optim_grads
  from garage.torch.optimizers import OptimizerWrapper
label_desc: Use image node_id 7 for calling the VPG obj's underlying member method code with example usage: obj._train_policy(obs, actions, rewards, advantages) and returns: loss