code: def _general_off_policy_corrected_multistep_target(
      r_t,
      pcont_t,
      target_policy_t,
      c_t,
      q_t,
      a_t,
      back_prop=False,
      name=None,
  ):
      """Evaluates targets for various off-policy value correction based algorithms.
  
      `target_policy_t` is the policy that this function aims to evaluate. New
      action-value estimates (target values `T`) must be expressible in this
      recurrent form:
      ```none
      T(x_{t-1}, a_{t-1}) = r_t + γ[ 𝔼_π Q(x_t, .) - c_t Q(x_t, a_t) +
                                                     c_t T(x_t, a_t) ]
      ```
      `T(x_t, a_t)` is an estimate of expected discounted future returns based
      on the current Q value estimates `Q(x_t, a_t)` and rewards `r_t`. The
      evaluated target values can be used as supervised targets for learning the Q
      function itself or as returns for various policy gradient algorithms.
      `Q==T` if convergence is reached. As the formula is recurrent, it will
      evaluate multistep returns for non-zero importance weights `c_t`.
  
      In the usual moving and target network setup `q_t` should be calculated by
      the target network while the `target_policy_t` may be evaluated by either of
      the networks. If `target_policy_t` is evaluated by the current moving network
      the algorithm implemented will have a similar flavour as double DQN.
  
      Depending on the choice of c_t, the algorithm can implement:
      ```none
      Importance Sampling             c_t = π(x_t, a_t) / μ(x_t, a_t),
      Harutyunyan's et al. Q(lambda)  c_t = λ,
      Precup's et al. Tree-Backup     c_t = π(x_t, a_t),
      Munos' et al. Retrace           c_t = λ min(1, π(x_t, a_t) / μ(x_t, a_t)).
      ```
      Please refer to page 3 for more details:
      https://arxiv.org/pdf/1606.02647v1.pdf
  
      Args:
        r_t: 2-D tensor holding rewards received during the transition
          that corresponds to each major index.
          Shape is `[T, B]`.
        pcont_t: 2-D tensor holding pcontinue values received during the
          transition that corresponds to each major index.
          Shape is `[T, B]`.
        target_policy_t:  3-D tensor holding per-action policy probabilities for
          the states encountered just AFTER the transitions that correspond to
          each major index, according to the target policy (i.e. the policy we
          wish to learn). These usually derive from the learning net.
          Shape is `[T, B, num_actions]`.
        c_t: 2-D tensor holding importance weights; see discussion above.
          Shape is `[T, B]`.
        q_t: 3-D tensor holding per-action Q-values for the states
          encountered just AFTER taking the transitions that correspond to each
          major index. Shape is `[T, B, num_actions]`.
        a_t: 2-D tensor holding the indices of actions executed during the
          transition AFTER the transition that corresponds to each major index.
          Shape is `[T, B]`.
        back_prop: whether to backpropagate gradients through time.
        name: name of the op.
  
      Returns:
        Tensor of shape `[T, B, num_actions]` containing Q values.
      """
      # Formula (4) in https://arxiv.org/pdf/1606.02647v1.pdf can be expressed
      # in a recursive form where T is a new target value:
      # T(x_{t-1}, a_{t-1}) = r_t + γ[ 𝔼_π Q(x_t, .) - c_t Q(x_t, a_t) +
      #                                                c_t T(x_t, a_t) ]
      # This recurrent form allows us to express Retrace by using
      # `scan_discounted_sum`.
      # Define:
      #   T_tm1   = T(x_{t-1}, a_{t-1})
      #   T_t     = T(x_t, a_t)
      #   exp_q_t = 𝔼_π Q(x_t,.)
      #   qa_t    = Q(x_t, a_t)
      # Hence:
      #   T_tm1   = r_t + γ * (exp_q_t - c_t * qa_t) + γ * c_t * T_t
      # Define:
      #   current = r_t + γ * (exp_q_t - c_t * qa_t)
      # Thus:
      #   T_tm1 = scan_discounted_sum(current, γ * c_t, reverse=True)
      args = [r_t, pcont_t, target_policy_t, c_t, q_t, a_t]
      with tf.name_scope(
          name, "general_returns_based_off_policy_target", values=args
      ):
          exp_q_t = tf.reduce_sum(target_policy_t * q_t, axis=2)
          qa_t = indexing_ops.batched_index(q_t, a_t)
          current = r_t + pcont_t * (exp_q_t - c_t * qa_t)
          initial_value = qa_t[-1]
          return sequence_ops.scan_discounted_sum(
              current,
              pcont_t * c_t,
              initial_value,
              reverse=True,
              back_prop=back_prop,
          )
  
file_path: trfl/trfl/retrace_ops.py
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  import collections
  from six.moves import zip
  import tensorflow.compat.v1
  from trfl import base_ops
  from trfl import indexing_ops
  from trfl import sequence_ops
label_desc: Use image node_id 2 for calling a global function with example usage: _general_off_policy_corrected_multistep_target(r_t, pcont_t, target_policy_t, c_t, q_t, a_t, back_prop, name) and returns: sequence_ops