docstr: Policy gradient loss with vtrace importance weighting.
  
  VTraceLoss takes tensors of shape [T, B, ...], where `B` is the
  batch_size. The reason we need to know `B` is for V-trace to properly
  handle episode cut boundaries.
  
  Args:
      behaviour_actions_log_probs: A float32 tensor of shape [T, B].
      target_actions_log_probs: A float32 tensor of shape [T, B].
      policy_entropy: A float32 tensor of shape [T, B].
      dones: A float32 tensor of shape [T, B].
      discount: A float32 scalar.
      rewards: A float32 tensor of shape [T, B].
      values: A float32 tensor of shape [T, B].
      bootstrap_value: A float32 tensor of shape [B].
docstr_tok: ["Policy","gradient","loss","with","vtrace","importance","weighting",".","VTraceLoss","takes","tensors","of","shape","[","T",",","B",",","...","]",",","where","`","B","`","is","the","batch_size",".","The","reason","we","need","to","know","`","B","`","is","for","V-trace","to","properly","handle","episode","cut","boundaries",".","Args",":","behaviour_actions_log_probs",":","A","float32","tensor","of","shape","[","T",",","B","]",".","target_actions_log_probs",":","A","float32","tensor","of","shape","[","T",",","B","]",".","policy_entropy",":","A","float32","tensor","of","shape","[","T",",","B","]",".","dones",":","A","float32","tensor","of","shape","[","T",",","B","]",".","discount",":","A","float32","scalar",".","rewards",":","A","float32","tensor","of","shape","[","T",",","B","]",".","values",":","A","float32","tensor","of","shape","[","T",",","B","]",".","bootstrap_value",":","A","float32","tensor","of","shape","[","B","]","."]
code: def __init__(
      self,
      behaviour_actions_log_probs,
      target_actions_log_probs,
      policy_entropy,
      dones,
      discount,
      rewards,
      values,
      bootstrap_value,
      entropy_coeff=-0.01,
      vf_loss_coeff=0.5,
      clip_rho_threshold=1.0,
      clip_pg_rho_threshold=1.0,
  ):
      """Policy gradient loss with vtrace importance weighting.
  
      VTraceLoss takes tensors of shape [T, B, ...], where `B` is the
      batch_size. The reason we need to know `B` is for V-trace to properly
      handle episode cut boundaries.
  
      Args:
          behaviour_actions_log_probs: A float32 tensor of shape [T, B].
          target_actions_log_probs: A float32 tensor of shape [T, B].
          policy_entropy: A float32 tensor of shape [T, B].
          dones: A float32 tensor of shape [T, B].
          discount: A float32 scalar.
          rewards: A float32 tensor of shape [T, B].
          values: A float32 tensor of shape [T, B].
          bootstrap_value: A float32 tensor of shape [B].
      """
  
      self.vtrace_returns = vtrace.from_importance_weights(
          behaviour_actions_log_probs=behaviour_actions_log_probs,
          target_actions_log_probs=target_actions_log_probs,
          discounts=(~dones).astype("float32") * discount,
          rewards=rewards,
          values=values,
          bootstrap_value=bootstrap_value,
          clip_rho_threshold=clip_rho_threshold,
          clip_pg_rho_threshold=clip_pg_rho_threshold,
      )
  
      # The policy gradients loss
      self.pi_loss = -1.0 * paddle.sum(
          target_actions_log_probs * self.vtrace_returns.pg_advantages
      )
  
      # The baseline loss
      delta = values - self.vtrace_returns.vs
      self.vf_loss = 0.5 * paddle.sum(paddle.square(delta))
  
      # The entropy loss (We want to maximize entropy, so entropy_ceoff < 0)
      self.entropy = paddle.sum(policy_entropy)
  
      # The summed weighted loss
      self.total_loss = (
          self.pi_loss
          + self.vf_loss * vf_loss_coeff
          + self.entropy * entropy_coeff
      )
  
file_path: PARL/parl/algorithms/paddle/impala/impala.py
imports: import parl
  import paddle
  import paddle.nn.functional
  from paddle.distribution import Categorical
  from parl.algorithms.paddle.impala import vtrace
label_desc: Use image node_id 1 to create a new VTraceLoss object from inherited base classes: object with example: obj = VTraceLoss(behaviour_actions_log_probs, target_actions_log_probs, policy_entropy, dones, discount, rewards, values, bootstrap_value, entropy_coeff, vf_loss_coeff, clip_rho_threshold, clip_pg_rho_threshold)