class_docstr: Exp3 bandit policy.
  
  This policy works by maintaining a weight for each arm. These weights are used to randomly
  decide which arm to pull. The weights are increased or decreased, depending on the reward. An
  egalitarianism factor $\gamma \in [0, 1]$ is included, to tune the desire to pick an arm
  uniformly at random. That is, if $\gamma = 1$, the arms are picked uniformly at random.
  
  Parameters
  ----------
  gamma
      The egalitarianism factor. Setting this to 0 leads to what is called the EXP3 policy.
  reward_obj
      The reward object used to measure the performance of each arm. This can be a metric, a
      statistic, or a distribution.
  reward_scaler
      A reward scaler used to scale the rewards before they are fed to the reward object. This
      can be useful to scale the rewards to a (0, 1) range for instance.
  burn_in
      The number of steps to use for the burn-in phase. Each arm is given the chance to be pulled
      during the burn-in phase. This is useful to mitigate selection bias.
  seed
      Random number generator seed for reproducibility.
  
  Examples
  --------
  
  >>> import gym
  >>> from river import bandit
  >>> from river import proba
  >>> from river import stats
  
  >>> env = gym.make(
  ...     'river_bandits/CandyCaneContest-v0'
  ... )
  >>> _ = env.reset(seed=42)
  >>> _ = env.action_space.seed(123)
  
  >>> policy = bandit.Exp3(gamma=0.5, seed=42)
  
  >>> metric = stats.Sum()
  >>> while True:
  ...     action = policy.pull(range(env.action_space.n))
  ...     observation, reward, terminated, truncated, info = env.step(action)
  ...     policy.update(action, reward)
  ...     metric.update(reward)
  ...     if terminated or truncated:
  ...         break
  
  >>> metric
  Sum: 799.
  
  References
  ----------
  [^1]: [Auer, P., Cesa-Bianchi, N., Freund, Y. and Schapire, R.E., 2002. The nonstochastic multiarmed bandit problem. SIAM journal on computing, 32(1), pp.48-77.](https://cseweb.ucsd.edu/~yfreund/papers/bandits.pdf)
  [^2]: [Adversarial Bandits and the Exp3 Algorithm — Jeremy Kun](https://jeremykun.com/2013/11/08/adversarial-bandits-and-the-exp3-algorithm/)
class_docstr_tok: ["Exp3","bandit","policy",".","This","policy","works","by","maintaining","a","weight","for","each","arm",".","These","weights","are","used","to","randomly","decide","which","arm","to","pull",".","The","weights","are","increased","or","decreased",",","depending","on","the","reward",".","An","egalitarianism","factor","$","\\gamma","\\in","[","0",",","1","]","$","is","included",",","to","tune","the","desire","to","pick","an","arm","uniformly","at","random",".","That","is",",","if","$","\\gamma","=","1","$",",","the","arms","are","picked","uniformly","at","random",".","Parameters","--","--","--","--","--","gamma","The","egalitarianism","factor",".","Setting","this","to","0","leads","to","what","is","called","the","EXP3","policy",".","reward_obj","The","reward","object","used","to","measure","the","performance","of","each","arm",".","This","can","be","a","metric",",","a","statistic",",","or","a","distribution",".","reward_scaler","A","reward","scaler","used","to","scale","the","rewards","before","they","are","fed","to","the","reward","object",".","This","can","be","useful","to","scale","the","rewards","to","a","(","0",",","1",")","range","for","instance",".","burn_in","The","number","of","steps","to","use","for","the","burn-in","phase",".","Each","arm","is","given","the","chance","to","be","pulled","during","the","burn-in","phase",".","This","is","useful","to","mitigate","selection","bias",".","seed","Random","number","generator","seed","for","reproducibility",".","Examples","--","--","--","--",">",">",">","import","gym",">",">",">","from","river","import","bandit",">",">",">","from","river","import","proba",">",">",">","from","river","import","stats",">",">",">","env","=","gym.make","(","...","'river_bandits\/CandyCaneContest-v0'","...",")",">",">",">","_","=","env.reset","(","seed=42",")",">",">",">","_","=","env.action_space.seed","(","123",")",">",">",">","policy","=","bandit.Exp3","(","gamma=0.5",",","seed=42",")",">",">",">","metric","=","stats.Sum","(",")",">",">",">","while","True",":","...","action","=","policy.pull","(","range","(","env.action_space.n",")",")","...","observation",",","reward",",","terminated",",","truncated",",","info","=","env.step","(","action",")","...","policy.update","(","action",",","reward",")","...","metric.update","(","reward",")","...","if","terminated","or","truncated",":","...","break",">",">",">","metric","Sum",":","799",".","References","--","--","--","--","--","[","^1","]",":","[","Auer",",","P.",",","Cesa-Bianchi",",","N.",",","Freund",",","Y.","and","Schapire",",","R.E.",",","2002",".","The","nonstochastic","multiarmed","bandit","problem",".","SIAM","journal","on","computing",",","32","(","1",")",",","pp.48-77",".","]","(","https",":","\/\/cseweb.ucsd.edu\/~yfreund\/papers\/bandits.pdf",")","[","^2","]",":","[","Adversarial","Bandits","and","the","Exp3","Algorithm","\u2014","Jeremy","Kun","]","(","https",":","\/\/jeremykun.com\/2013\/11\/08\/adversarial-bandits-and-the-exp3-algorithm\/",")"]
code: def _pull(self, arm_ids):
      total = sum(self._weights[arm_id] for arm_id in arm_ids)
      self._probabilities = {
          arm_id: (1 - self.gamma) * (self._weights[arm_id] / total)
          + self.gamma / len(arm_ids)
          for arm_id in arm_ids
      }
      return self._rng.choices(
          arm_ids, weights=self._probabilities.values()
      )[0]
  
file_path: river/river/bandit/exp3.py
imports: from __future__ import annotations
  import collections
  import functools
  import math
  import random
  from river import bandit
label_desc: Use image node_id 2 for calling the Exp3 obj's underlying member method code with example usage: obj._pull(arm_ids) and returns: self