class_docstr: Critic Regularized Regression (CRR) algorithm trainer
  as described in https://arxiv.org/abs/2006.15134
code: def compute_actor_loss(
      self,
      batch_idx,
      action,
      logged_action_probs,
      all_q_values,
      all_action_scores,
  ):
      # Only update actor network after a fixed number of Q updates
      if batch_idx % self.delayed_policy_update != 0:
          # Yielding None prevents the actor network from updating
          actor_loss = None
          return (actor_loss, actor_loss)
  
      # dist is the distribution of actions derived from the actor's outputs (logits)
      dist = pyd.Categorical(logits=all_action_scores)
      # Note: D = dist.probs is equivalent to:
      # e_x = torch.exp(actor_actions)
      # D = e_x / e_x.sum(dim=1, keepdim=True)
      # That is, dist gives a softmax distribution over actor's outputs
  
      # values is the vector of state values in this batch
      values = (all_q_values * dist.probs).sum(dim=1, keepdim=True)
  
      advantages = all_q_values - values
      # Note: the above statement subtracts the "values" column vector from
      # every column of the all_q_values matrix, giving us the advantages
      # of every action in the present state
  
      weight = torch.clamp(
          (
              (1 / self.beta)
              * (advantages * action).sum(dim=1, keepdim=True)
          ).exp(),
          0,
          self.max_weight,
      )
      # Remember: training_batch.action is in the one-hot format
      logged_action_idxs = torch.argmax(action, dim=1, keepdim=True)
  
      # Note: action space is assumed to be discrete with actions
      # belonging to the set {0, 1, ..., action_dim-1}. Therefore,
      # advantages.gather(1, logged_action_idxs) will select, for each data point
      # (row i of the Advantage matrix "advantages"), the element with index
      # action.float_features[i]
  
      # Note: dist.logits already gives log(p), which can be verified by
      # comparing dist.probs and dist.logits.
      # https://pytorch.org/docs/master/distributions.html#multinomial
      # states: logits (Tensor) – event log probabilities
  
      # log_pi_b is the log of the probability assigned by the
      # actor (abbreviated as pi) to the actions of the behavioral (b) policy
      log_pi_b = dist.log_prob(logged_action_idxs.squeeze(1)).unsqueeze(
          1
      )
  
      # entropy regularization
      pi_t = (dist.probs * action).sum(dim=1, keepdim=True)
  
      if self.entropy_coeff > 0:
          pi_b = logged_action_probs.view(pi_t.shape)
          assert torch.min(pi_b) > 0, "Logged action probability <= 0"
          pi_ratio = torch.clip(
              pi_t / pi_b, min=1e-4, max=self.clip_limit
          )
          entropy = (pi_ratio * log_pi_b).mean()
      else:
          # dummy value
          entropy = 0
  
      # Note: the CRR loss for each datapoint (and the magnitude of the corresponding
      # parameter update) is proportional to log_pi_b * weight. Therefore, as mentioned
      # at the top of Section 3.2, the actor on the one hand has incentive to assign
      # larger probabilities to the actions observed in the dataset (so as to reduce
      # the magnitude of log_pi_b), but on the other hand it gives preference to doing
      # this on datapoints where weight is large (i.e., those points on which the
      # Q-value of the observed action is large).
      actor_loss_without_reg = (-log_pi_b * weight.detach()).mean()
      actor_loss = (
          -log_pi_b * weight.detach()
      ).mean() + self.entropy_coeff * entropy
      return actor_loss_without_reg, actor_loss
  
imports: import logging
  from typing import List, Tuple
  import reagent.core.types
  import torch
  import torch.nn.functional
  from reagent.core.configuration import resolve_defaults
  from reagent.core.dataclasses import field
  from reagent.core.parameters import EvaluationParameters, RLParameters
  from reagent.optimizer import Optimizer__Union, SoftUpdate
  from reagent.training.dqn_trainer_base import DQNTrainerBaseLightning
  from torch import distributions
label_desc: Use image node_id 7 for calling the DiscreteCRRTrainer obj's underlying member method code with example usage: obj.compute_actor_loss(batch_idx, action, logged_action_probs, all_q_values, all_action_scores) and returns: actor_loss_without_reg,  actor_loss, actor_loss,  actor_loss