We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
BUG - KellyCoinflip episode should end when the agent reaches the maximum wealth.
# Bug description
The episode does not end when agent reaches the maximum wealth. This is not consistent with the information from the docstring. 

A Monte Carlo approach can also be used to illustrate the problem. The distribution of realized rewards obtained by an agent betting according to the Kelly criterion is incorrect. E.g. the docstring indicates such agent should earn $240 on average, and the bug leads to a much lower mean.

# To reproduce the bug
## Code
```
import gym 

env = gym.make('KellyCoinflip-v0')
env.seed(1)
env.reset()
done=False

while not done:
    action = int(env.wealth * 20)  # bet 20% of the wealth
    observation, reward, done, info = env.step(action)
    env.render()
```

## Output
```
Current wealth:  20.0 ; Rounds left:  299
[...]
Current wealth:  238.47 ; Rounds left:  192
Current wealth:  250.0 ; Rounds left:  191
Current wealth:  200.0 ; Rounds left:  190
[...]
```

## Expected output 
```
Current wealth:  20.0 ; Rounds left:  299
[...]
Current wealth:  238.47 ; Rounds left:  192
Current wealth:  250.0 ; Rounds left:  191
```

# Proposed modifications
I would like to work on a PR with the following modifications:
* fix the bug described above,
* refactor to reduce code duplication,
* rename some variables to follow PEP-8 guidelines. 

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `gym/envs/toy_text/kellycoinflip.py`
Content:
```
1 from scipy.stats import genpareto
2 import numpy as np
3 
4 import gym
5 from gym import spaces
6 from gym.utils import seeding
7 
8 
9 def flip(edge, np_random):
10     return 1 if np_random.uniform() < edge else -1
11 
12 
13 class KellyCoinflipEnv(gym.Env):
14     """The Kelly coinflip game is a simple gambling introduced by Haghani & Dewey 2016's
15     'Rational Decision-Making Under Uncertainty: Observed Betting Patterns on a Biased
16     Coin' (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2856963), to test human
17     decision-making in a setting like that of the stock market: positive expected value
18     but highly stochastic; they found many subjects performed badly, often going broke,
19     even though optimal play would reach the maximum with ~95% probability. In the
20     coinflip game, the player starts with $25.00 to gamble over 300 rounds; each round,
21     they can bet anywhere up to their net worth (in penny increments), and then a coin is
22     flipped; with P=0.6, the player wins twice what they bet, otherwise, they lose it.
23     $250 is the maximum players are allowed to have. At the end of the 300 rounds, they
24     keep whatever they have. The human subjects earned an average of $91; a simple use of
25     the Kelly criterion (https://en.wikipedia.org/wiki/Kelly_criterion), giving a
26     strategy of betting 20% until the cap is hit, would earn $240; a decision tree
27     analysis shows that optimal play earns $246 (https://www.gwern.net/Coin-flip).
28 
29     The game short-circuits when either wealth = $0 (since one can never recover) or
30     wealth = cap (trivial optimal play: one simply bets nothing thereafter).
31 
32     In this implementation, we default to the paper settings of $25, 60% odds, wealth cap
33     of $250, and 300 rounds. To specify the action space in advance, we multiply the
34     wealth cap (in dollars) by 100 (to allow for all penny bets); should one attempt to
35     bet more money than one has, it is rounded down to one's net worth. (Alternately, a
36     mistaken bet could end the episode immediately; it's not clear to me which version
37     would be better.) For a harder version which randomizes the 3 key parameters, see the
38     Generalized Kelly coinflip game."""
39     metadata = {'render.modes': ['human']}
40 
41     def __init__(self, initial_wealth=25.0, edge=0.6, max_wealth=250.0, max_rounds=300):
42 
43         self.action_space = spaces.Discrete(int(max_wealth * 100))  # betting in penny
44         # increments
45         self.observation_space = spaces.Tuple((
46             spaces.Box(0, max_wealth, [1], dtype=np.float32),  # (w,b)
47             spaces.Discrete(max_rounds + 1)))
48         self.reward_range = (0, max_wealth)
49         self.edge = edge
50         self.wealth = initial_wealth
51         self.initial_wealth = initial_wealth
52         self.max_rounds = max_rounds
53         self.max_wealth = max_wealth
54         self.np_random = None
55         self.rounds = None
56         self.seed()
57         self.reset()
58 
59     def seed(self, seed=None):
60         self.np_random, seed = seeding.np_random(seed)
61         return [seed]
62 
63     def step(self, action):
64         bet_in_dollars = min(action/100.0, self.wealth)  # action = desired bet in pennies
65         self.rounds -= 1
66 
67         coinflip = flip(self.edge, self.np_random)
68         self.wealth = min(self.max_wealth, self.wealth + coinflip * bet_in_dollars)
69 
70         done = self.wealth < 0.01 or self.wealth == self.max_wealth or not self.rounds
71         reward = self.wealth if done else 0.0
72 
73         return self._get_obs(), reward, done, {}
74 
75     def _get_obs(self):
76         return np.array([self.wealth]), self.rounds
77 
78     def reset(self):
79         self.rounds = self.max_rounds
80         self.wealth = self.initial_wealth
81         return self._get_obs()
82 
83     def render(self, mode='human'):
84         print("Current wealth: ", self.wealth, "; Rounds left: ", self.rounds)
85 
86 
87 class KellyCoinflipGeneralizedEnv(gym.Env):
88     """The Generalized Kelly coinflip game is an extension by ArthurB & Gwern Branwen
89     which expands the Kelly coinflip game MDP into a POMDP, where the 3 key parameters
90     (edge, maximum wealth, and number of rounds) are unknown random variables drawn
91     from 3 distributions: a Beta(7,3) for the coinflip edge 0-1, a N(300,25) the total
92     number of rounds, and a Pareto(5,200) for the wealth cap. These distributions are
93     chosen to be conjugate & easily updatable, to allow for inference (other choices
94     like the geometric for number of rounds wouldn't make observations informative),
95     and to loosely reflect what a human might expect in the original Kelly coinflip
96     game given that the number of rounds wasn't strictly fixed and they weren't told
97     the wealth cap until they neared it. With these particular distributions, the
98     entire history of the game can be summarized into a few sufficient statistics of
99     rounds-elapsed/wins/losses/max-wealth-ever-reached, from which the Bayes-optimal
100     decision can (in theory) be made; to avoid all agents having to tediously track
101     those sufficient statistics manually in the same way, the observation space is
102     augmented from wealth/rounds-left (rounds-left is deleted because it is a hidden
103     variable) to current-wealth/rounds-elapsed/wins/losses/maximum-observed-wealth.
104     The simple Kelly coinflip game can easily be solved by calculating decision trees,
105     but the Generalized Kelly coinflip game may be intractable (although the analysis
106     for the edge case alone suggests that the Bayes-optimal value may be very close to
107     what one would calculate using a decision tree for any specific case), and
108     represents a good challenge for RL agents."""
109     metadata = {'render.modes': ['human']}
110 
111     def __init__(self, initial_wealth=25.0, edge_prior_alpha=7, edge_prior_beta=3,
112                  max_wealth_alpha=5.0, max_wealth_m=200.0, max_rounds_mean=300.0,
113                  max_rounds_sd=25.0, reseed=True):
114         # store the hyper-parameters for passing back into __init__() during resets so
115         # the same hyper-parameters govern the next game's parameters, as the user
116         # expects:
117         # TODO: this is boilerplate, is there any more elegant way to do this?
118         self.initial_wealth = float(initial_wealth)
119         self.edge_prior_alpha = edge_prior_alpha
120         self.edge_prior_beta = edge_prior_beta
121         self.max_wealth_alpha = max_wealth_alpha
122         self.max_wealth_m = max_wealth_m
123         self.max_rounds_mean = max_rounds_mean
124         self.max_rounds_sd = max_rounds_sd
125 
126         if reseed or not hasattr(self, 'np_random'):
127             self.seed()
128 
129         # draw this game's set of parameters:
130         edge = self.np_random.beta(edge_prior_alpha, edge_prior_beta)
131         max_wealth = round(genpareto.rvs(max_wealth_alpha, max_wealth_m,
132                                          random_state=self.np_random))
133         max_rounds = int(round(self.np_random.normal(max_rounds_mean, max_rounds_sd)))
134 
135         # add an additional global variable which is the sufficient statistic for the
136         # Pareto distribution on wealth cap; alpha doesn't update, but x_m does, and
137         # simply is the highest wealth count we've seen to date:
138         self.max_ever_wealth = float(self.initial_wealth)
139         # for the coinflip edge, it is total wins/losses:
140         self.wins = 0
141         self.losses = 0
142         # for the number of rounds, we need to remember how many rounds we've played:
143         self.rounds_elapsed = 0
144 
145         # the rest proceeds as before:
146         self.action_space = spaces.Discrete(int(max_wealth*100))
147         self.observation_space = spaces.Tuple((
148             spaces.Box(0, max_wealth, shape=[1], dtype=np.float32),  # current wealth
149             spaces.Discrete(max_rounds+1),  # rounds elapsed
150             spaces.Discrete(max_rounds+1),  # wins
151             spaces.Discrete(max_rounds+1),  # losses
152             spaces.Box(0, max_wealth, [1], dtype=np.float32)))  # maximum observed wealth
153         self.reward_range = (0, max_wealth)
154         self.edge = edge
155         self.wealth = self.initial_wealth
156         self.max_rounds = max_rounds
157         self.rounds = self.max_rounds
158         self.max_wealth = max_wealth
159 
160     def seed(self, seed=None):
161         self.np_random, seed = seeding.np_random(seed)
162         return [seed]
163 
164     def step(self, action):
165         bet_in_dollars = min(action/100.0, self.wealth)
166 
167         self.rounds -= 1
168 
169         coinflip = flip(self.edge, self.np_random)
170         self.wealth = min(self.max_wealth, self.wealth + coinflip * bet_in_dollars)
171         self.rounds_elapsed += 1
172 
173         if coinflip:
174             self.max_ever_wealth = max(self.wealth, self.max_ever_wealth)
175             self.wins += 1
176         else:
177             self.losses += 1
178 
179         done = self.wealth < 0.01 or self.wealth == self.max_wealth or not self.rounds
180         reward = self.max_wealth if done else 0.0
181 
182         return self._get_obs(), reward, done, {}
183 
184     def _get_obs(self):
185         return (np.array([float(self.wealth)]), self.rounds_elapsed, self.wins,
186                 self.losses, np.array([float(self.max_ever_wealth)]))
187 
188     def reset(self):
189         # re-init everything to draw new parameters etc, but preserve the RNG for
190         # reproducibility and pass in the same hyper-parameters as originally specified:
191         self.__init__(initial_wealth=self.initial_wealth,
192                       edge_prior_alpha=self.edge_prior_alpha,
193                       edge_prior_beta=self.edge_prior_beta,
194                       max_wealth_alpha=self.max_wealth_alpha,
195                       max_wealth_m=self.max_wealth_m,
196                       max_rounds_mean=self.max_rounds_mean,
197                       max_rounds_sd=self.max_rounds_sd,
198                       reseed=False)
199         return self._get_obs()
200 
201     def render(self, mode='human'):
202         print("Current wealth: ", self.wealth, "; Rounds left: ", self.rounds,
203               "; True edge: ", self.edge, "; True max wealth: ", self.max_wealth,
204               "; True stopping time: ", self.max_rounds, "; Rounds left: ",
205               self.max_rounds - self.rounds_elapsed)
206
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/gym/envs/toy_text/kellycoinflip.py b/gym/envs/toy_text/kellycoinflip.py
--- a/gym/envs/toy_text/kellycoinflip.py
+++ b/gym/envs/toy_text/kellycoinflip.py
@@ -177,7 +177,7 @@
             self.losses += 1
 
         done = self.wealth < 0.01 or self.wealth == self.max_wealth or not self.rounds
-        reward = self.max_wealth if done else 0.0
+        reward = self.wealth if done else 0.0
 
         return self._get_obs(), reward, done, {}

{"golden_diff": "diff --git a/gym/envs/toy_text/kellycoinflip.py b/gym/envs/toy_text/kellycoinflip.py\n--- a/gym/envs/toy_text/kellycoinflip.py\n+++ b/gym/envs/toy_text/kellycoinflip.py\n@@ -177,7 +177,7 @@\n             self.losses += 1\n \n         done = self.wealth < 0.01 or self.wealth == self.max_wealth or not self.rounds\n-        reward = self.max_wealth if done else 0.0\n+        reward = self.wealth if done else 0.0\n \n         return self._get_obs(), reward, done, {}\n", "issue": "BUG - KellyCoinflip episode should end when the agent reaches the maximum wealth.\n# Bug description\r\nThe episode does not end when agent reaches the maximum wealth. This is not consistent with the information from the docstring. \r\n\r\nA Monte Carlo approach can also be used to illustrate the problem. The distribution of realized rewards obtained by an agent betting according to the Kelly criterion is incorrect. E.g. the docstring indicates such agent should earn $240 on average, and the bug leads to a much lower mean.\r\n\r\n# To reproduce the bug\r\n## Code\r\n```\r\nimport gym \r\n\r\nenv = gym.make('KellyCoinflip-v0')\r\nenv.seed(1)\r\nenv.reset()\r\ndone=False\r\n\r\nwhile not done:\r\n    action = int(env.wealth * 20)  # bet 20% of the wealth\r\n    observation, reward, done, info = env.step(action)\r\n    env.render()\r\n```\r\n\r\n## Output\r\n```\r\nCurrent wealth:  20.0 ; Rounds left:  299\r\n[...]\r\nCurrent wealth:  238.47 ; Rounds left:  192\r\nCurrent wealth:  250.0 ; Rounds left:  191\r\nCurrent wealth:  200.0 ; Rounds left:  190\r\n[...]\r\n```\r\n\r\n## Expected output \r\n```\r\nCurrent wealth:  20.0 ; Rounds left:  299\r\n[...]\r\nCurrent wealth:  238.47 ; Rounds left:  192\r\nCurrent wealth:  250.0 ; Rounds left:  191\r\n```\r\n\r\n# Proposed modifications\r\nI would like to work on a PR with the following modifications:\r\n* fix the bug described above,\r\n* refactor to reduce code duplication,\r\n* rename some variables to follow PEP-8 guidelines. \n", "before_files": [{"content": "from scipy.stats import genpareto\nimport numpy as np\n\nimport gym\nfrom gym import spaces\nfrom gym.utils import seeding\n\n\ndef flip(edge, np_random):\n    return 1 if np_random.uniform() < edge else -1\n\n\nclass KellyCoinflipEnv(gym.Env):\n    \"\"\"The Kelly coinflip game is a simple gambling introduced by Haghani & Dewey 2016's\n    'Rational Decision-Making Under Uncertainty: Observed Betting Patterns on a Biased\n    Coin' (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2856963), to test human\n    decision-making in a setting like that of the stock market: positive expected value\n    but highly stochastic; they found many subjects performed badly, often going broke,\n    even though optimal play would reach the maximum with ~95% probability. In the\n    coinflip game, the player starts with $25.00 to gamble over 300 rounds; each round,\n    they can bet anywhere up to their net worth (in penny increments), and then a coin is\n    flipped; with P=0.6, the player wins twice what they bet, otherwise, they lose it.\n    $250 is the maximum players are allowed to have. At the end of the 300 rounds, they\n    keep whatever they have. The human subjects earned an average of $91; a simple use of\n    the Kelly criterion (https://en.wikipedia.org/wiki/Kelly_criterion), giving a\n    strategy of betting 20% until the cap is hit, would earn $240; a decision tree\n    analysis shows that optimal play earns $246 (https://www.gwern.net/Coin-flip).\n\n    The game short-circuits when either wealth = $0 (since one can never recover) or\n    wealth = cap (trivial optimal play: one simply bets nothing thereafter).\n\n    In this implementation, we default to the paper settings of $25, 60% odds, wealth cap\n    of $250, and 300 rounds. To specify the action space in advance, we multiply the\n    wealth cap (in dollars) by 100 (to allow for all penny bets); should one attempt to\n    bet more money than one has, it is rounded down to one's net worth. (Alternately, a\n    mistaken bet could end the episode immediately; it's not clear to me which version\n    would be better.) For a harder version which randomizes the 3 key parameters, see the\n    Generalized Kelly coinflip game.\"\"\"\n    metadata = {'render.modes': ['human']}\n\n    def __init__(self, initial_wealth=25.0, edge=0.6, max_wealth=250.0, max_rounds=300):\n\n        self.action_space = spaces.Discrete(int(max_wealth * 100))  # betting in penny\n        # increments\n        self.observation_space = spaces.Tuple((\n            spaces.Box(0, max_wealth, [1], dtype=np.float32),  # (w,b)\n            spaces.Discrete(max_rounds + 1)))\n        self.reward_range = (0, max_wealth)\n        self.edge = edge\n        self.wealth = initial_wealth\n        self.initial_wealth = initial_wealth\n        self.max_rounds = max_rounds\n        self.max_wealth = max_wealth\n        self.np_random = None\n        self.rounds = None\n        self.seed()\n        self.reset()\n\n    def seed(self, seed=None):\n        self.np_random, seed = seeding.np_random(seed)\n        return [seed]\n\n    def step(self, action):\n        bet_in_dollars = min(action/100.0, self.wealth)  # action = desired bet in pennies\n        self.rounds -= 1\n\n        coinflip = flip(self.edge, self.np_random)\n        self.wealth = min(self.max_wealth, self.wealth + coinflip * bet_in_dollars)\n\n        done = self.wealth < 0.01 or self.wealth == self.max_wealth or not self.rounds\n        reward = self.wealth if done else 0.0\n\n        return self._get_obs(), reward, done, {}\n\n    def _get_obs(self):\n        return np.array([self.wealth]), self.rounds\n\n    def reset(self):\n        self.rounds = self.max_rounds\n        self.wealth = self.initial_wealth\n        return self._get_obs()\n\n    def render(self, mode='human'):\n        print(\"Current wealth: \", self.wealth, \"; Rounds left: \", self.rounds)\n\n\nclass KellyCoinflipGeneralizedEnv(gym.Env):\n    \"\"\"The Generalized Kelly coinflip game is an extension by ArthurB & Gwern Branwen\n    which expands the Kelly coinflip game MDP into a POMDP, where the 3 key parameters\n    (edge, maximum wealth, and number of rounds) are unknown random variables drawn\n    from 3 distributions: a Beta(7,3) for the coinflip edge 0-1, a N(300,25) the total\n    number of rounds, and a Pareto(5,200) for the wealth cap. These distributions are\n    chosen to be conjugate & easily updatable, to allow for inference (other choices\n    like the geometric for number of rounds wouldn't make observations informative),\n    and to loosely reflect what a human might expect in the original Kelly coinflip\n    game given that the number of rounds wasn't strictly fixed and they weren't told\n    the wealth cap until they neared it. With these particular distributions, the\n    entire history of the game can be summarized into a few sufficient statistics of\n    rounds-elapsed/wins/losses/max-wealth-ever-reached, from which the Bayes-optimal\n    decision can (in theory) be made; to avoid all agents having to tediously track\n    those sufficient statistics manually in the same way, the observation space is\n    augmented from wealth/rounds-left (rounds-left is deleted because it is a hidden\n    variable) to current-wealth/rounds-elapsed/wins/losses/maximum-observed-wealth.\n    The simple Kelly coinflip game can easily be solved by calculating decision trees,\n    but the Generalized Kelly coinflip game may be intractable (although the analysis\n    for the edge case alone suggests that the Bayes-optimal value may be very close to\n    what one would calculate using a decision tree for any specific case), and\n    represents a good challenge for RL agents.\"\"\"\n    metadata = {'render.modes': ['human']}\n\n    def __init__(self, initial_wealth=25.0, edge_prior_alpha=7, edge_prior_beta=3,\n                 max_wealth_alpha=5.0, max_wealth_m=200.0, max_rounds_mean=300.0,\n                 max_rounds_sd=25.0, reseed=True):\n        # store the hyper-parameters for passing back into __init__() during resets so\n        # the same hyper-parameters govern the next game's parameters, as the user\n        # expects:\n        # TODO: this is boilerplate, is there any more elegant way to do this?\n        self.initial_wealth = float(initial_wealth)\n        self.edge_prior_alpha = edge_prior_alpha\n        self.edge_prior_beta = edge_prior_beta\n        self.max_wealth_alpha = max_wealth_alpha\n        self.max_wealth_m = max_wealth_m\n        self.max_rounds_mean = max_rounds_mean\n        self.max_rounds_sd = max_rounds_sd\n\n        if reseed or not hasattr(self, 'np_random'):\n            self.seed()\n\n        # draw this game's set of parameters:\n        edge = self.np_random.beta(edge_prior_alpha, edge_prior_beta)\n        max_wealth = round(genpareto.rvs(max_wealth_alpha, max_wealth_m,\n                                         random_state=self.np_random))\n        max_rounds = int(round(self.np_random.normal(max_rounds_mean, max_rounds_sd)))\n\n        # add an additional global variable which is the sufficient statistic for the\n        # Pareto distribution on wealth cap; alpha doesn't update, but x_m does, and\n        # simply is the highest wealth count we've seen to date:\n        self.max_ever_wealth = float(self.initial_wealth)\n        # for the coinflip edge, it is total wins/losses:\n        self.wins = 0\n        self.losses = 0\n        # for the number of rounds, we need to remember how many rounds we've played:\n        self.rounds_elapsed = 0\n\n        # the rest proceeds as before:\n        self.action_space = spaces.Discrete(int(max_wealth*100))\n        self.observation_space = spaces.Tuple((\n            spaces.Box(0, max_wealth, shape=[1], dtype=np.float32),  # current wealth\n            spaces.Discrete(max_rounds+1),  # rounds elapsed\n            spaces.Discrete(max_rounds+1),  # wins\n            spaces.Discrete(max_rounds+1),  # losses\n            spaces.Box(0, max_wealth, [1], dtype=np.float32)))  # maximum observed wealth\n        self.reward_range = (0, max_wealth)\n        self.edge = edge\n        self.wealth = self.initial_wealth\n        self.max_rounds = max_rounds\n        self.rounds = self.max_rounds\n        self.max_wealth = max_wealth\n\n    def seed(self, seed=None):\n        self.np_random, seed = seeding.np_random(seed)\n        return [seed]\n\n    def step(self, action):\n        bet_in_dollars = min(action/100.0, self.wealth)\n\n        self.rounds -= 1\n\n        coinflip = flip(self.edge, self.np_random)\n        self.wealth = min(self.max_wealth, self.wealth + coinflip * bet_in_dollars)\n        self.rounds_elapsed += 1\n\n        if coinflip:\n            self.max_ever_wealth = max(self.wealth, self.max_ever_wealth)\n            self.wins += 1\n        else:\n            self.losses += 1\n\n        done = self.wealth < 0.01 or self.wealth == self.max_wealth or not self.rounds\n        reward = self.max_wealth if done else 0.0\n\n        return self._get_obs(), reward, done, {}\n\n    def _get_obs(self):\n        return (np.array([float(self.wealth)]), self.rounds_elapsed, self.wins,\n                self.losses, np.array([float(self.max_ever_wealth)]))\n\n    def reset(self):\n        # re-init everything to draw new parameters etc, but preserve the RNG for\n        # reproducibility and pass in the same hyper-parameters as originally specified:\n        self.__init__(initial_wealth=self.initial_wealth,\n                      edge_prior_alpha=self.edge_prior_alpha,\n                      edge_prior_beta=self.edge_prior_beta,\n                      max_wealth_alpha=self.max_wealth_alpha,\n                      max_wealth_m=self.max_wealth_m,\n                      max_rounds_mean=self.max_rounds_mean,\n                      max_rounds_sd=self.max_rounds_sd,\n                      reseed=False)\n        return self._get_obs()\n\n    def render(self, mode='human'):\n        print(\"Current wealth: \", self.wealth, \"; Rounds left: \", self.rounds,\n              \"; True edge: \", self.edge, \"; True max wealth: \", self.max_wealth,\n              \"; True stopping time: \", self.max_rounds, \"; Rounds left: \",\n              self.max_rounds - self.rounds_elapsed)\n", "path": "gym/envs/toy_text/kellycoinflip.py"}], "after_files": [{"content": "from scipy.stats import genpareto\nimport numpy as np\n\nimport gym\nfrom gym import spaces\nfrom gym.utils import seeding\n\n\ndef flip(edge, np_random):\n    return 1 if np_random.uniform() < edge else -1\n\n\nclass KellyCoinflipEnv(gym.Env):\n    \"\"\"The Kelly coinflip game is a simple gambling introduced by Haghani & Dewey 2016's\n    'Rational Decision-Making Under Uncertainty: Observed Betting Patterns on a Biased\n    Coin' (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2856963), to test human\n    decision-making in a setting like that of the stock market: positive expected value\n    but highly stochastic; they found many subjects performed badly, often going broke,\n    even though optimal play would reach the maximum with ~95% probability. In the\n    coinflip game, the player starts with $25.00 to gamble over 300 rounds; each round,\n    they can bet anywhere up to their net worth (in penny increments), and then a coin is\n    flipped; with P=0.6, the player wins twice what they bet, otherwise, they lose it.\n    $250 is the maximum players are allowed to have. At the end of the 300 rounds, they\n    keep whatever they have. The human subjects earned an average of $91; a simple use of\n    the Kelly criterion (https://en.wikipedia.org/wiki/Kelly_criterion), giving a\n    strategy of betting 20% until the cap is hit, would earn $240; a decision tree\n    analysis shows that optimal play earns $246 (https://www.gwern.net/Coin-flip).\n\n    The game short-circuits when either wealth = $0 (since one can never recover) or\n    wealth = cap (trivial optimal play: one simply bets nothing thereafter).\n\n    In this implementation, we default to the paper settings of $25, 60% odds, wealth cap\n    of $250, and 300 rounds. To specify the action space in advance, we multiply the\n    wealth cap (in dollars) by 100 (to allow for all penny bets); should one attempt to\n    bet more money than one has, it is rounded down to one's net worth. (Alternately, a\n    mistaken bet could end the episode immediately; it's not clear to me which version\n    would be better.) For a harder version which randomizes the 3 key parameters, see the\n    Generalized Kelly coinflip game.\"\"\"\n    metadata = {'render.modes': ['human']}\n\n    def __init__(self, initial_wealth=25.0, edge=0.6, max_wealth=250.0, max_rounds=300):\n\n        self.action_space = spaces.Discrete(int(max_wealth * 100))  # betting in penny\n        # increments\n        self.observation_space = spaces.Tuple((\n            spaces.Box(0, max_wealth, [1], dtype=np.float32),  # (w,b)\n            spaces.Discrete(max_rounds + 1)))\n        self.reward_range = (0, max_wealth)\n        self.edge = edge\n        self.wealth = initial_wealth\n        self.initial_wealth = initial_wealth\n        self.max_rounds = max_rounds\n        self.max_wealth = max_wealth\n        self.np_random = None\n        self.rounds = None\n        self.seed()\n        self.reset()\n\n    def seed(self, seed=None):\n        self.np_random, seed = seeding.np_random(seed)\n        return [seed]\n\n    def step(self, action):\n        bet_in_dollars = min(action/100.0, self.wealth)  # action = desired bet in pennies\n        self.rounds -= 1\n\n        coinflip = flip(self.edge, self.np_random)\n        self.wealth = min(self.max_wealth, self.wealth + coinflip * bet_in_dollars)\n\n        done = self.wealth < 0.01 or self.wealth == self.max_wealth or not self.rounds\n        reward = self.wealth if done else 0.0\n\n        return self._get_obs(), reward, done, {}\n\n    def _get_obs(self):\n        return np.array([self.wealth]), self.rounds\n\n    def reset(self):\n        self.rounds = self.max_rounds\n        self.wealth = self.initial_wealth\n        return self._get_obs()\n\n    def render(self, mode='human'):\n        print(\"Current wealth: \", self.wealth, \"; Rounds left: \", self.rounds)\n\n\nclass KellyCoinflipGeneralizedEnv(gym.Env):\n    \"\"\"The Generalized Kelly coinflip game is an extension by ArthurB & Gwern Branwen\n    which expands the Kelly coinflip game MDP into a POMDP, where the 3 key parameters\n    (edge, maximum wealth, and number of rounds) are unknown random variables drawn\n    from 3 distributions: a Beta(7,3) for the coinflip edge 0-1, a N(300,25) the total\n    number of rounds, and a Pareto(5,200) for the wealth cap. These distributions are\n    chosen to be conjugate & easily updatable, to allow for inference (other choices\n    like the geometric for number of rounds wouldn't make observations informative),\n    and to loosely reflect what a human might expect in the original Kelly coinflip\n    game given that the number of rounds wasn't strictly fixed and they weren't told\n    the wealth cap until they neared it. With these particular distributions, the\n    entire history of the game can be summarized into a few sufficient statistics of\n    rounds-elapsed/wins/losses/max-wealth-ever-reached, from which the Bayes-optimal\n    decision can (in theory) be made; to avoid all agents having to tediously track\n    those sufficient statistics manually in the same way, the observation space is\n    augmented from wealth/rounds-left (rounds-left is deleted because it is a hidden\n    variable) to current-wealth/rounds-elapsed/wins/losses/maximum-observed-wealth.\n    The simple Kelly coinflip game can easily be solved by calculating decision trees,\n    but the Generalized Kelly coinflip game may be intractable (although the analysis\n    for the edge case alone suggests that the Bayes-optimal value may be very close to\n    what one would calculate using a decision tree for any specific case), and\n    represents a good challenge for RL agents.\"\"\"\n    metadata = {'render.modes': ['human']}\n\n    def __init__(self, initial_wealth=25.0, edge_prior_alpha=7, edge_prior_beta=3,\n                 max_wealth_alpha=5.0, max_wealth_m=200.0, max_rounds_mean=300.0,\n                 max_rounds_sd=25.0, reseed=True):\n        # store the hyper-parameters for passing back into __init__() during resets so\n        # the same hyper-parameters govern the next game's parameters, as the user\n        # expects:\n        # TODO: this is boilerplate, is there any more elegant way to do this?\n        self.initial_wealth = float(initial_wealth)\n        self.edge_prior_alpha = edge_prior_alpha\n        self.edge_prior_beta = edge_prior_beta\n        self.max_wealth_alpha = max_wealth_alpha\n        self.max_wealth_m = max_wealth_m\n        self.max_rounds_mean = max_rounds_mean\n        self.max_rounds_sd = max_rounds_sd\n\n        if reseed or not hasattr(self, 'np_random'):\n            self.seed()\n\n        # draw this game's set of parameters:\n        edge = self.np_random.beta(edge_prior_alpha, edge_prior_beta)\n        max_wealth = round(genpareto.rvs(max_wealth_alpha, max_wealth_m,\n                                         random_state=self.np_random))\n        max_rounds = int(round(self.np_random.normal(max_rounds_mean, max_rounds_sd)))\n\n        # add an additional global variable which is the sufficient statistic for the\n        # Pareto distribution on wealth cap; alpha doesn't update, but x_m does, and\n        # simply is the highest wealth count we've seen to date:\n        self.max_ever_wealth = float(self.initial_wealth)\n        # for the coinflip edge, it is total wins/losses:\n        self.wins = 0\n        self.losses = 0\n        # for the number of rounds, we need to remember how many rounds we've played:\n        self.rounds_elapsed = 0\n\n        # the rest proceeds as before:\n        self.action_space = spaces.Discrete(int(max_wealth*100))\n        self.observation_space = spaces.Tuple((\n            spaces.Box(0, max_wealth, shape=[1], dtype=np.float32),  # current wealth\n            spaces.Discrete(max_rounds+1),  # rounds elapsed\n            spaces.Discrete(max_rounds+1),  # wins\n            spaces.Discrete(max_rounds+1),  # losses\n            spaces.Box(0, max_wealth, [1], dtype=np.float32)))  # maximum observed wealth\n        self.reward_range = (0, max_wealth)\n        self.edge = edge\n        self.wealth = self.initial_wealth\n        self.max_rounds = max_rounds\n        self.rounds = self.max_rounds\n        self.max_wealth = max_wealth\n\n    def seed(self, seed=None):\n        self.np_random, seed = seeding.np_random(seed)\n        return [seed]\n\n    def step(self, action):\n        bet_in_dollars = min(action/100.0, self.wealth)\n\n        self.rounds -= 1\n\n        coinflip = flip(self.edge, self.np_random)\n        self.wealth = min(self.max_wealth, self.wealth + coinflip * bet_in_dollars)\n        self.rounds_elapsed += 1\n\n        if coinflip:\n            self.max_ever_wealth = max(self.wealth, self.max_ever_wealth)\n            self.wins += 1\n        else:\n            self.losses += 1\n\n        done = self.wealth < 0.01 or self.wealth == self.max_wealth or not self.rounds\n        reward = self.wealth if done else 0.0\n\n        return self._get_obs(), reward, done, {}\n\n    def _get_obs(self):\n        return (np.array([float(self.wealth)]), self.rounds_elapsed, self.wins,\n                self.losses, np.array([float(self.max_ever_wealth)]))\n\n    def reset(self):\n        # re-init everything to draw new parameters etc, but preserve the RNG for\n        # reproducibility and pass in the same hyper-parameters as originally specified:\n        self.__init__(initial_wealth=self.initial_wealth,\n                      edge_prior_alpha=self.edge_prior_alpha,\n                      edge_prior_beta=self.edge_prior_beta,\n                      max_wealth_alpha=self.max_wealth_alpha,\n                      max_wealth_m=self.max_wealth_m,\n                      max_rounds_mean=self.max_rounds_mean,\n                      max_rounds_sd=self.max_rounds_sd,\n                      reseed=False)\n        return self._get_obs()\n\n    def render(self, mode='human'):\n        print(\"Current wealth: \", self.wealth, \"; Rounds left: \", self.rounds,\n              \"; True edge: \", self.edge, \"; True max wealth: \", self.max_wealth,\n              \"; True stopping time: \", self.max_rounds, \"; Rounds left: \",\n              self.max_rounds - self.rounds_elapsed)\n", "path": "gym/envs/toy_text/kellycoinflip.py"}]}