class_docstr: The Doubly Robust estimator.
  
  Let s_t, a_t, and r_t be the state, action, and reward at timestep t.
  
  This method trains a Q-model for the evaluation policy \pi_e on behavior
  data generated by \pi_b. Currently, RLlib implements this using
  Fitted-Q Evaluation (FQE). You can also implement your own model
  and pass it in as `q_model_config = {"type": your_model_class, **your_kwargs}`.
  
  For behavior policy \pi_b and evaluation policy \pi_e, define the
  cumulative importance ratio at timestep t as:
  p_t = \sum_{t'=0}^t (\pi_e(a_{t'} | s_{t'}) / \pi_b(a_{t'} | s_{t'})).
  
  Consider an episode with length T. Let V_T = 0.
  For all t in {0, T - 1}, use the following recursive update:
  V_t^DR = (\sum_{a \in A} \pi_e(a | s_t) Q(s_t, a))
      + p_t * (r_t + \gamma * V_{t+1}^DR - Q(s_t, a_t))
  
  This estimator computes the expected return for \pi_e for an episode as:
  V^{\pi_e}(s_0) = V_0^DR
  and returns the mean and standard deviation over episodes.
  
  For more information refer to https://arxiv.org/pdf/1911.06854.pdf
docstr: Initializes a Doubly Robust OPE Estimator.
  
  Args:
      policy: Policy to evaluate.
      gamma: Discount factor of the environment.
      epsilon_greedy: The probability by which we act acording to a fully random
          policy during deployment. With 1-epsilon_greedy we act
          according the target policy.
      normalize_weights: If True, the inverse propensity scores are normalized to
          their sum across the entire dataset. The effect of this is similar to
          weighted importance sampling compared to standard importance sampling.
      q_model_config: Arguments to specify the Q-model. Must specify
          a `type` key pointing to the Q-model class.
          This Q-model is trained in the train() method and is used
          to compute the state-value and Q-value estimates
          for the DoublyRobust estimator.
          It must implement `train`, `estimate_q`, and `estimate_v`.
          TODO (Rohan138): Unify this with RLModule API.
docstr_tok: ["Initializes","a","Doubly","Robust","OPE","Estimator",".","Args",":","policy",":","Policy","to","evaluate",".","gamma",":","Discount","factor","of","the","environment",".","epsilon_greedy",":","The","probability","by","which","we","act","acording","to","a","fully","random","policy","during","deployment",".","With","1-epsilon_greedy","we","act","according","the","target","policy",".","normalize_weights",":","If","True",",","the","inverse","propensity","scores","are","normalized","to","their","sum","across","the","entire","dataset",".","The","effect","of","this","is","similar","to","weighted","importance","sampling","compared","to","standard","importance","sampling",".","q_model_config",":","Arguments","to","specify","the","Q-model",".","Must","specify","a","`","type","`","key","pointing","to","the","Q-model","class",".","This","Q-model","is","trained","in","the","train","(",")","method","and","is","used","to","compute","the","state-value","and","Q-value","estimates","for","the","DoublyRobust","estimator",".","It","must","implement","`","train","`",",","`","estimate_q","`",",","and","`","estimate_v","`",".","TODO","(","Rohan138",")",":","Unify","this","with","RLModule","API","."]
code: def __init__(
      self,
      policy: Policy,
      gamma: float,
      epsilon_greedy: float = 0.0,
      normalize_weights: bool = True,
      q_model_config: Optional[Dict] = None,
  ):
      """Initializes a Doubly Robust OPE Estimator.
  
      Args:
          policy: Policy to evaluate.
          gamma: Discount factor of the environment.
          epsilon_greedy: The probability by which we act acording to a fully random
              policy during deployment. With 1-epsilon_greedy we act
              according the target policy.
          normalize_weights: If True, the inverse propensity scores are normalized to
              their sum across the entire dataset. The effect of this is similar to
              weighted importance sampling compared to standard importance sampling.
          q_model_config: Arguments to specify the Q-model. Must specify
              a `type` key pointing to the Q-model class.
              This Q-model is trained in the train() method and is used
              to compute the state-value and Q-value estimates
              for the DoublyRobust estimator.
              It must implement `train`, `estimate_q`, and `estimate_v`.
              TODO (Rohan138): Unify this with RLModule API.
      """
  
      super().__init__(policy, gamma, epsilon_greedy)
      q_model_config = q_model_config or {}
      q_model_config["gamma"] = gamma
  
      self._model_cls = q_model_config.pop("type", FQETorchModel)
      self._model_configs = q_model_config
      self._normalize_weights = normalize_weights
  
      self.model = self._model_cls(
          policy=policy,
          **q_model_config,
      )
      assert hasattr(
          self.model, "estimate_v"
      ), "self.model must implement `estimate_v`!"
      assert hasattr(
          self.model, "estimate_q"
      ), "self.model must implement `estimate_q`!"
  
imports: import logging
  import numpy
  import math
  import pandas
  from typing import Dict, Any, Optional, List
  from ray.data import Dataset
  from ray.rllib.policy import Policy
  from ray.rllib.policy.sample_batch import SampleBatch, convert_ma_batch_to_sample_batch
  from ray.rllib.utils.annotations import DeveloperAPI, override
  from ray.rllib.utils.typing import SampleBatchType
  from ray.rllib.utils.numpy import convert_to_numpy
  from ray.rllib.offline.estimators.off_policy_estimator import OffPolicyEstimator
  from ray.rllib.offline.estimators.fqe_torch_model import FQETorchModel
  from ray.rllib.offline.offline_evaluator import OfflineEvaluator
  from ray.rllib.offline.offline_evaluation_utils import compute_is_weights, compute_q_and_v_values
label_desc: Use image node_id 1 to create a new DoublyRobust object from inherited base classes: OffPolicyEstimator with example: obj = DoublyRobust(policy, gamma, epsilon_greedy, normalize_weights, q_model_config)