class_docstr: Learning tests for the DirectMethod estimator.
  
  Generates three GridWorldWallPolicy policies and batches with  epsilon = 0.2, 0.5,
  and 0.8 respectively using `get_cliff_walking_wall_policy_and_data`.
  
  Tests that the estimators converge on all eight combinations of evaluation policy
  and behavior batch using `check_estimates`, except random policy-expert batch.
  
  Note: We do not test OPE with the "random" policy (epsilon=0.8)
  and "expert" (epsilon=0.2) batch because of the large policy-data mismatch. The
  expert batch is unlikely to contain the longer trajectories that would be observed
  under the random policy, thus the OPE estimate is flaky and inaccurate.
class_docstr_tok: ["Learning","tests","for","the","DirectMethod","estimator",".","Generates","three","GridWorldWallPolicy","policies","and","batches","with","epsilon","=","0.2",",","0.5",",","and","0.8","respectively","using","`","get_cliff_walking_wall_policy_and_data","`",".","Tests","that","the","estimators","converge","on","all","eight","combinations","of","evaluation","policy","and","behavior","batch","using","`","check_estimates","`",",","except","random","policy-expert","batch",".","Note",":","We","do","not","test","OPE","with","the","``","random","''","policy","(","epsilon=0.8",")","and","``","expert","''","(","epsilon=0.2",")","batch","because","of","the","large","policy-data","mismatch",".","The","expert","batch","is","unlikely","to","contain","the","longer","trajectories","that","would","be","observed","under","the","random","policy",",","thus","the","OPE","estimate","is","flaky","and","inaccurate","."]
code: def setUpClass(cls):
      ray.init()
      # Epsilon-greedy exploration values
      random_eps = 0.8
      mixed_eps = 0.5
      expert_eps = 0.2
      num_episodes = 64
      cls.gamma = 0.99
  
      # Config settings for FQE model
      cls.q_model_config = {
          "n_iters": 800,
          "minibatch_size": 64,
          "polyak_coef": 1.0,
          "model_config": {
              "fcnet_hiddens": [32, 32, 32],
              "activation": "relu",
          },
          "lr": 1e-3,
      }
  
      (
          cls.random_policy,
          cls.random_batch,
          cls.random_reward,
          cls.random_std,
      ) = get_cliff_walking_wall_policy_and_data(
          num_episodes, cls.gamma, random_eps, seed=SEED
      )
      print(
          f"Collected random batch of {cls.random_batch.count} steps "
          f"with return {cls.random_reward} stddev {cls.random_std}"
      )
  
      (
          cls.mixed_policy,
          cls.mixed_batch,
          cls.mixed_reward,
          cls.mixed_std,
      ) = get_cliff_walking_wall_policy_and_data(
          num_episodes, cls.gamma, mixed_eps, seed=SEED
      )
      print(
          f"Collected mixed batch of {cls.mixed_batch.count} steps "
          f"with return {cls.mixed_reward} stddev {cls.mixed_std}"
      )
  
      (
          cls.expert_policy,
          cls.expert_batch,
          cls.expert_reward,
          cls.expert_std,
      ) = get_cliff_walking_wall_policy_and_data(
          num_episodes, cls.gamma, expert_eps, seed=SEED
      )
      print(
          f"Collected expert batch of {cls.expert_batch.count} steps "
          f"with return {cls.expert_reward} stddev {cls.expert_std}"
      )
  
file_path: ray/python/ray/rllib/offline/estimators/tests/test_dm_learning.py
imports: import unittest
  import ray
  from ray.rllib.offline.estimators import DirectMethod
  from ray.rllib.offline.estimators.tests.utils import get_cliff_walking_wall_policy_and_data, check_estimate
label_desc: Use image node_id 1 for calling the TestDMLearning obj's underlying member method code with example usage: obj.setUpClass(cls) without return types