docstr: Comparison between policy gradient and Q-learning-based optimizer
  The input param has two axes, choice1 and choice2.
  
  The value achieved by different combinations of the two choices:
        a     b     c
  1   0.43   0.9   0.45
  
  2   0.9    0.4    0.9
  
  3   0.45   0.9   0.45
  
  In summary, the global minimum is at (choice1=2, choice2=b), but there are local minima
  and maxima which easily hurdle an optimizer from finding the global minimum.
  
  In this setting, Q-learning performs better than policy gradient
docstr_tok: ["Comparison","between","policy","gradient","and","Q-learning-based","optimizer","The","input","param","has","two","axes",",","choice1","and","choice2",".","The","value","achieved","by","different","combinations","of","the","two","choices",":","a","b","c","1","0.43","0.9","0.45","2","0.9","0.4","0.9","3","0.45","0.9","0.45","In","summary",",","the","global","minimum","is","at","(","choice1=2",",","choice2=b",")",",","but","there","are","local","minima","and","maxima","which","easily","hurdle","an","optimizer","from","finding","the","global","minimum",".","In","this","setting",",","Q-learning","performs","better","than","policy","gradient"]
code: def test_policy_gradient_vs_q_learning_discrete(self):
      """
      Comparison between policy gradient and Q-learning-based optimizer
      The input param has two axes, choice1 and choice2.
  
      The value achieved by different combinations of the two choices:
            a     b     c
      1   0.43   0.9   0.45
  
      2   0.9    0.4    0.9
  
      3   0.45   0.9   0.45
  
      In summary, the global minimum is at (choice1=2, choice2=b), but there are local minima
      and maxima which easily hurdle an optimizer from finding the global minimum.
  
      In this setting, Q-learning performs better than policy gradient
      """
      input_param = ng.p.Dict(
          choice1=ng.p.Choice(["1", "2", "3"]),
          choice2=ng.p.Choice(["a", "b", "c"]),
      )
  
      def obj_func(
          sampled_sol: Dict[str, torch.Tensor]
      ) -> torch.Tensor:
          # sampled_sol format:
          #    key = choice_name
          #    val = choice_idx (a tensor of length `batch_size`)
          assert list(sampled_sol.values())[0].dim() == 1
          batch_size = list(sampled_sol.values())[0].shape[0]
          result = torch.zeros(batch_size, 1)
          choice1 = sampled_sol["choice1"]
          choice2 = sampled_sol["choice2"]
          for i in range(batch_size):
              if choice1[i] == 1 and choice2[i] == 1:
                  result[i] = 0.4
              elif choice1[i] == 0 and choice2[i] == 0:
                  result[i] = 0.43
              elif choice1[i] == 1 or choice2[i] == 1:
                  result[i] = 0.9
              else:
                  result[i] = 0.45
          return result
  
      batch_size = 32
      n_generations = 100
      repeat = 10
  
      qlearning_res = self.run_q_learning_optimizer(
          input_param, obj_func, batch_size, n_generations, repeat
      )
      pg_res = self.run_policy_gradient_optimizer(
          input_param, obj_func, batch_size, n_generations, repeat
      )
      print(f"QLearning results over {repeat} repeats: {qlearning_res}")
      print(f"PG results over {repeat} repeats: {pg_res}")
  
      assert (
          np.mean(qlearning_res) < 0.42
      ), "QLearning should end up better than local minimum (0.43)"
      assert np.mean(qlearning_res) < np.mean(
          pg_res
      ), f"In this setting. qlearning should be better than policy gradient over {repeat} repeats"
  
imports: import random
  import unittest
  from collections import defaultdict
  from typing import Dict
  import nevergrad
  import numpy
  import torch
  import torch.nn
  from reagent.lite.optimizer import BayesianByBackpropOptimizer, BayesianMLPEnsemblerOptimizer, BayesianOptimizerBase, GREEDY_TEMP, GumbelSoftmaxOptimizer, NeverGradOptimizer, PolicyGradientOptimizer, QLearningOptimizer, RandomSearchOptimizer, sol_to_tensors
label_desc: Use image node_id 11 for calling the TestComboOptimizer obj's underlying member method code with example usage: obj.test_policy_gradient_vs_q_learning_discrete() and returns: result