# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     name: python3
# ---

# + [markdown] id="view-in-github" colab_type="text"
# <a href="https://colab.research.google.com/github/skycreeper2000/DeepRL-Agents/blob/master/Lesson_1_Multi_Armed_Bandit_with_OpenAi_Gym_ver_3.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>

# + id="B8bZ7U19of0H" colab_type="code" colab={}
#          _____                _____                    _____                    _____                    _____                    _____          
#         /\    \              /\    \                  /\    \                  /\    \                  /\    \                  /\    \         
#        /::\    \            /::\    \                /::\    \                /::\    \                /::\    \                /::\    \        
#       /::::\    \           \:::\    \              /::::\    \              /::::\    \              /::::\    \               \:::\    \       
#      /::::::\    \           \:::\    \            /::::::\    \            /::::::\    \            /::::::\    \               \:::\    \      
#     /:::/\:::\    \           \:::\    \          /:::/\:::\    \          /:::/\:::\    \          /:::/\:::\    \               \:::\    \     
#    /:::/__\:::\    \           \:::\    \        /:::/__\:::\    \        /:::/__\:::\    \        /:::/__\:::\    \               \:::\    \    
#    \:::\   \:::\    \          /::::\    \      /::::\   \:::\    \      /::::\   \:::\    \      /::::\   \:::\    \              /::::\    \   
#  ___\:::\   \:::\    \        /::::::\    \    /::::::\   \:::\    \    /::::::\   \:::\    \    /::::::\   \:::\    \    ____    /::::::\    \  
# /\   \:::\   \:::\    \      /:::/\:::\    \  /:::/\:::\   \:::\    \  /:::/\:::\   \:::\____\  /:::/\:::\   \:::\    \  /\   \  /:::/\:::\    \ 
#/::\   \:::\   \:::\____\    /:::/  \:::\____\/:::/  \:::\   \:::\____\/:::/  \:::\   \:::|    |/:::/  \:::\   \:::\____\/::\   \/:::/  \:::\____\
#\:::\   \:::\   \::/    /   /:::/    \::/    /\::/    \:::\  /:::/    /\::/   |::::\  /:::|____|\::/    \:::\  /:::/    /\:::\  /:::/    \::/    /
# \:::\   \:::\   \/____/   /:::/    / \/____/  \/____/ \:::\/:::/    /  \/____|:::::\/:::/    /  \/____/ \:::\/:::/    /  \:::\/:::/    / \/____/ 
#  \:::\   \:::\    \      /:::/    /                    \::::::/    /         |:::::::::/    /            \::::::/    /    \::::::/    /          
#   \:::\   \:::\____\    /:::/    /                      \::::/    /          |::|\::::/    /              \::::/    /      \::::/____/           
#    \:::\  /:::/    /    \::/    /                       /:::/    /           |::| \::/____/               /:::/    /        \:::\    \           
#     \:::\/:::/    /      \/____/                       /:::/    /            |::|  ~|                    /:::/    /          \:::\    \          
#      \::::::/    /                                    /:::/    /             |::|   |                   /:::/    /            \:::\    \         
#       \::::/    /                                    /:::/    /              \::|   |                  /:::/    /              \:::\____\        
#        \::/    /                                     \::/    /                \:|   |                  \::/    /                \::/    /        
#         \/____/                                       \/____/                  \|___|                   \/____/                  \/____/         

# + [markdown] id="excxXVzFXgDf" colab_type="text"
# # Lesson 1, exercise 1: The Multi Armed Bandit Problem with OpenAi Gym
#
# the purpose of this notebook is:
#
#
#
# 1.   To understand the Gym Environment,
# 2.   Implement Epsilon-Greedy on the bandit problem as discussed in the post
# 3.   **Win at slot machines!!.**
#
# Lets get started!
#
#

# + [markdown] id="oO4T4NXwr9Ac" colab_type="text"
# ~

# + [markdown] id="soq1skOjrr6z" colab_type="text"
# Some tips on using Colaboratory:
#
# 1.   First off click **File**, on the menu above & **Save a copy in drive**, this will copy an instance to your google drive, to allow you to start running it!
#
# 2.   Once the notebook has finished copying, go to your fresh copy and click **"connect"** in the top right hand corner of Colabs. This connects your computer to a power Virtual Machine sitting in Google's cloud.

# + [markdown] id="kAloqi_arsyE" colab_type="text"
# How to run it:
#
#
# **For those new to Colaboratory**, there are two types of "cell blocks" **text** (like the one you are reading right now) and **code**. For code Cell blocks just click the little play button in order to get the code to run. The play button looks like this:
#
# ![alt text](https://image.ibb.co/i4sxHH/Screen_Shot_2018_04_10_at_3_04_50_pm.png)
#
# Or you can simply click the "**Runtime**" menu button up above, and click "**Run all**", sit back & watch colabs go to work.
#
# ~~~~~
#

# + [markdown] id="VBwBIZPkZf_g" colab_type="text"
# First we need to download & install the Gym Library so that it works in Colabs.

# + id="MoRJN4PpXsGH" colab_type="code" colab={}
# !pip install gym > /dev/null 2>&1

# + [markdown] id="on5JDYmWaK-w" colab_type="text"
# **Great!** now for our first bit of code
#

# + [markdown] id="p8fBpXyfbk89" colab_type="text"
# Lets import the Gym class and walk through a basic example of Gym Code

# + id="5hK-jC9ceDuY" colab_type="code" colab={}
import gym

# + [markdown] id="WLePIoe7VJXS" colab_type="text"
# Gym's main purpose is to provide a large collection of "environments" that expose a common interface, using standardized inputs & outputs for Reinforcement Learning model testing purposes. You can find a listing of these environments below, as follows:

# + id="twrKYXVAVIp8" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 54} outputId="630460eb-df95-4152-a47d-2514789638c5"
from gym import envs
print(envs.registry.all())

# + [markdown] id="QA-998XefX85" colab_type="text"
# Unfortunatley, Gym does **not provide a bandit** environment so we need to import it, lets install one with the command below:

# + id="sGDaa_u8fjO3" colab_type="code" colab={}
# !git clone https://github.com/JKCooper2/gym-bandits.git > /dev/null 2>&1
# !pip install /content/gym-bandits/. > /dev/null 2>&1

# + [markdown] id="kogFWTLPhcjx" colab_type="text"
# And import the bandit library too

# + id="ecG0Xn0fhfd9" colab_type="code" colab={}
import gym_bandits

# + [markdown] id="jEkYlxG7cwjC" colab_type="text"
# Unlike in the post, where there was only 2 bandits & we were trying to figure out on average which one paid out the most, this time around we are going to be dealing with **TEN** (10) bandits!!!

# + [markdown] id="nz924RhYdlif" colab_type="text"
# Each bandit will have a payout with a normal distribution (bell curve), but the average payout or, centre of the distribution, will be different for each bandit, like in the image below

# + [markdown] id="0NJTjv1xmi29" colab_type="text"
# ![alt text](https://i.stack.imgur.com/SazYv.png)

# + [markdown] id="rDhrNHQLU2xu" colab_type="text"
# .

# + [markdown] id="2e5lDZlheZuZ" colab_type="text"
#   >** >>>> It will be our goal to try & determine which bandit, out of the 10, pays out the most!!!!!!! <<<<**

# + [markdown] id="HIFfNSFiU4NU" colab_type="text"
# .

# + [markdown] id="9QFSX4fjxrh8" colab_type="text"
# We want to randomly initialise our enviroment, do this by running the code below:

# + id="NWI9R9BiybZl" colab_type="code" colab={}
#For this exercise we will be using a powerful python array library known as *numpy*, so lets import that.
import numpy as np

# + id="txOTaw3jxvhm" colab_type="code" colab={}
#rerun this part of the code if you would like to "reset" or reinitialize your bandit environement
np.random.seed(42) 

# + [markdown] id="6GA0H5L3UNDm" colab_type="text"
# Lets make a a variable called "env" to hold our freshly created 10 arm multi bandit environment

# + id="WJIh1OV_UMMT" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 71} outputId="88bbfae4-d0dc-460e-9721-79f7227ccda3"
#gaussian distribution is just another name for "normal distribution" or bell curve (so many different names for the same thing!)
env = gym.make('BanditTenArmedGaussian-v0')

# + [markdown] id="8E8LePgXTWF6" colab_type="text"
# .

# + [markdown] id="tHDprdS4BI1Z" colab_type="text"
# We are now going to go over a basic example of how OpenAi Gym works.

# + [markdown] id="me9kN1MPWEEf" colab_type="text"
# run the code below & then we will explain what is going on here piece by piece

# + id="63v0tSoHbuwY" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 527} outputId="c3b4abd1-477a-42bc-8ff1-72f47b3ee4e3"
observation = env.reset()

for i_episode in range(5):
  
    print("episode Number is", i_episode)   
    
    action = env.action_space.sample() # sampling the "action" array which in this case only contains 10 "options" because there is 10 bandits
        
    print("action is", action)
        
        
    # here we taking the next "step" in our environment by taking in our action variable randomly selected above
    observation, reward, done, info = env.step(action) 
        
    print("observation space is: ",observation)
    print("reward variable is: ",reward)
    print("done flag is: ",done)
    print("info variable is: ",info)
        
      
            
env.close()

# + [markdown] id="EHSzWhEmh1ha" colab_type="text"
# .

# + [markdown] id="MgJlSiCGbMBP" colab_type="text"
# **Reinforcement Learning** is an **extremely broad machine learning "framework"**, that looks like this:

# + [markdown] id="T3aynRR6ZcOI" colab_type="text"
# .

# + [markdown] id="HkYTLOA3aphJ" colab_type="text"
#
#
# ![alt text](https://keon.io/images/deep-q-learning/rl.png)

# + [markdown] id="HJmgkP_PjNsT" colab_type="text"
# explaining that picture above, The RL framework goes something like this:

# + [markdown] id="PZnKxZw_jWlo" colab_type="text"
#
#
# 1.   You have an **AGENT** (machine learning algorithm), it the image above the agent is a human brain, lol
# 2.   The agent takes an **ACTION**, in the image above available actions are using the joystick, up, down, left, right and the red button. So five actions available (5) in total.
# 3.   a single ACTION is chosen (from our available five from the joystick) and fed to our **ENVIRONMENT**, which in our example above is the Atari game environment
# 4.   at this point our ENVIRONMENT measures how good the action taken was and produces a **REWARD** signal. a Postive number is usually good & negative number is usually bad. 
# 5.   the environment then produces an **OBSERVATION**, again using our example above, think of the OBSERVATION as the next graphical "frame" of the game. In RL, we also call observations **STATES**  
# 6.   The new OBSERVATION & REWARD signal (produced by the old observation-action pair) is then fed back to AGENT for it to decide what move to make next and so on and so forth
#

# + [markdown] id="uH3nZ60oVUi8" colab_type="text"
# However, lets now take the above framework and see how it is implements on our example "toy problem" of the multi armed bandit

# + [markdown] id="xuDzvJasVurw" colab_type="text"
# Taking the code apart piece by piece

# + [markdown] id="HjHOjKpsVymi" colab_type="text"
# we have already created our enviroment with this line of code:
#
#
# ```
# env = gym.make('BanditTenArmedGaussian-v0')
# ```
#
#

# + [markdown] id="rDCrgRjLVndy" colab_type="text"
# but, we need to ask the environment to produce the first **OBSERVATION** (or as we also call it - state) so that we can feed it to our **AGENT** (the RL algorithm) to decide what to do next. 
#

# + [markdown] id="aKrvnNKkX1V6" colab_type="text"
# So next up we get the *first* OBSERVATION by calling the following code:
#
#
#
# ```
# observation = env.reset()
#
# ```
#

# + [markdown] id="oxIQ6ZDEYL4N" colab_type="text"
# Next up, we create a 'for loop' that looped 5 times. In the Multi armed banded scenario an "episode" is just a single "play" of the game. So think of the For loop then as playing the multi armed bandit game 5 times.
#
#
#
# ```
# for i_episode in range(5):
# ```
#
#

# + [markdown] id="CceslAxIZBst" colab_type="text"
# each **ENVIRONMENT** is different. That is to say, each environment gives us different **ACTIONS** that are available, different **OBSERVATIONS** that are available etc

# + [markdown] id="GFzJTTmFaTeu" colab_type="text"
# For the *10 armed bandit problem,* we should have *ten actions available *to us (as there should be 10 different slot machines, each with a different lever to pull)

# + [markdown] id="IXRhRNz6aqxC" colab_type="text"
# We can confirm this is the case by running the code below:

# + id="GMBt0CDuaqGs" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 34} outputId="ed277ee9-1a89-44b8-f5b7-2218bd7eb6d3"
print(env.action_space)

# + [markdown] id="VeGThbjxbje3" colab_type="text"
# So taking this back to our example code previous, the next thing we do is take an action,* by randomly sampling from the* **ACTION SPACE**. Again, dont worry. The ACTION SPACE is just a number assigned to each our bandits. EG [0,1,2,3,4,5,6,7,8,9]. So for our example if we randomly sample the ACTION SPACE and get back the number 8 - all this means is that we will be pulling the "lever" on bandit 8.

# + [markdown] id="uM1J8_n4cbsT" colab_type="text"
# It is also important to note that at this point - we have not implemented any machine learning yet. We are only choosing actions at random. So lets choose one then. This was done with this line of code:
#
#
#
# ```
# action = env.action_space.sample()
# ```
#
#

# + [markdown] id="H8v64MX8dKcU" colab_type="text"
# We have now completed step 2 of our Reinforcement Learning framework discussed earlier. Lets now do step 3, feeding the action into the environment, this was achieved with this line of code
#
#
#
# ```
# observation, reward, done, info = env.step(action) 
# ```
#
#

# + [markdown] id="r_j80CP2d3Va" colab_type="text"
# The above line of code is actually doing a couple of things. We are feeding in our selected action with this line:
#
#
#
# ```
# env.step(action) 
# ```
#
#

# + [markdown] id="AwQ6JLupeHfY" colab_type="text"
# And getting back 4 new variables in return , in this part of the code:
#
#
#
# ```
# observation, reward, done, info = 
# ```
#
#

# + [markdown] id="JeAZp6rMeUfP" colab_type="text"
# For now, we do not have to worry about the DONE and INFO variables. All we care about, in this tutorial is the OBSERVATION & REWARD variables. Also note that in this single line of code we have achieved step 4 & 5 of our RL framework discussed earlier in one go! 

# + [markdown] id="nAC11E-Jfl7a" colab_type="text"
# Where we now breakaway from the RL framework, is that, we are not feeding the **REWARD** & **OBSERVATION** variables back to any **AGENT** (aka step 6 from the framework) to do anything intelligent yet. This is because we yet to create an **AGENT**!!
#
#
#
#

# + [markdown] id="02vg-e-ldZjq" colab_type="text"
# ## The final piece of the puzzle: the Reward-Averaging Sampling learning rule.

# + [markdown] id="Kh-n_ABLdpDv" colab_type="text"
# In the lecture we learned about the epsilon-greedy agent. It is a synthesis of a purely exploratory agent and a completely greedy agent.
#

# + [markdown] id="mR5j1zVTejE3" colab_type="text"
# In the multiarmed bandit problem a purely exploratory agent would sample all the different bandits (or options) available uniformily - building up a distribution of each. However this has the downside such that the agent never gets to use its knowledge of the best option discovered so far. *A purely greedy agent * however would choose a bandit and stick to it's choice for all eternity - it will not try other agents to see if they provide better long term reward & can be thought of as being very "narrow-minded".

# + [markdown] id="3wgKe6vJetd9" colab_type="text"
# To get the best of both worlds, the Epsilon-greedy agent is designed to explore at an Epsilon Chance (let say 10%) whilst the rest of the time (90%) it goes greedy on the best option it had discovered so far.

# + [markdown] id="KZNnBENcev-j" colab_type="text"
# The idea here is that the Greedy mechanism helps the agent exploit the best option it has discovered so far, whilst the small amount of exploration leftover ensures that our agent keeps searching the other options available - to prove that there are not even better options out there.

# + [markdown] id="4_kSLlO5eyzd" colab_type="text"
# The last remaining piece of the puzzle is, how do we assign the idea of "Value", in the epsilon-greedy algorithm, to each of our bandits.

# + [markdown] id="TlMGGjPSe1ce" colab_type="text"
# Given some state S1 and an action A1, we can define a mathematical function to define the long term expected reward for this state action pair (state-action pairs are called "Q values", but we will get to this in a sec)

# + [markdown] id="FhmLHwl2e3l3" colab_type="text"
# So the expected Q value (or average over time) for any one bandit (or choice), can be given by:

# + [markdown] id="xd_4t3zrg8dp" colab_type="text"
# .

# + [markdown] id="_kuNRBdhgzjP" colab_type="text"
# $ Q_{n}(a) = \frac{1}{n+1}(r1 + r2 +r3) .(1) $

# + [markdown] id="szDvwjmpg-gq" colab_type="text"
# .

# + [markdown] id="iHFvuSRghBEk" colab_type="text"
# Where:
#
#
#
# *  **Q** stands for **Quality**, this is an arbitary naming convention
# *  **n** stands for ***the number of times that bandit (or state) has been visited***
# * **r** is reward from visiting a bandit, n many times
#
#
#
#

# + [markdown] id="wXn-k-d1hedE" colab_type="text"
# **also note that there is a +1 on the bottem line. This is here to stop a divide by zero error that will occur if we start with n set to 0.**

# + [markdown] id="K1Te_dcqifyp" colab_type="text"
# .

# + [markdown] id="3JQGXhVSh5ls" colab_type="text"
# So for example, lets say we visit bandit #1, 3 times.
#
# This means that we get three different rewards r1, r2 & r3. Because we visited bandit#1 3 times we also know that n = 3

# + [markdown] id="PfvF4wE2h7Aw" colab_type="text"
# Therefore: 
#
# Q = (1/n)*(r1+r2+r3)

# + [markdown] id="dLQydXfmiB0E" colab_type="text"
# Q = 1/(3+1)*(r1+r2+r3)

# + [markdown] id="TWE1vJDniT0S" colab_type="text"
# This whole process also has another name in life - we also call it , drumroll, "the average". However how the idea of average is used in the context here- is that we are only working out the average **for a single bandit**. 
#
# In order to work out the average for another bandit, we will have to keep track of all of its reward variables & number of times it has been visited seperately

# + [markdown] id="B5ZlYDrqiVYx" colab_type="text"
# In a real life multi armed bandit scenario, we would have to keep track of many reward variables the longer our agent is run for. So in order to reduce the amount of computer memory that is in use, we can use some mathematical trickery to compress the above equation, such that we update our Q value with every visit to the bandit.

# + [markdown] id="YAVtA8g_iorw" colab_type="text"
# the formula then becomes:

# + [markdown] id="GLEdkITbip84" colab_type="text"
# $ Q_{n+1}(a) = Q_{n}(a) + \frac{1}{n+1}(r_{n+1}- Q_{n}(a)) .(4) $

# + [markdown] id="vl4dCDpPjS4x" colab_type="text"
# Where:
#
#
#
# *  $ Q_{n+1}(a) $ is the update to our Q table’s value for a specific bandit, say bandit #1
# *  $ Q_{n}(a) $ is the current value in our Q table for a specific bandit
# * n +1 is the number of times that bandit (or state) has been visited. There is a “+1” here to stop a divide by zero error at the intialization of the algorithm.
# * r+1 is the updated reward from out latest sample of the bandit.
#
#
#
#
#
#
#

# + [markdown] id="WGrkgAJbj8fl" colab_type="text"
# this then is the heart of the epsilon greedy algorithm, or the forumula we use to update our Q table is pseudocode, **you have to use this code in the upcoming exercise to make the agent work.**
#

# + [markdown] id="eFUAx1Ngj9Cz" colab_type="text"
#
#
# ```
# NewQvalue = OldQvalue + ((reward - OldQvalue)/numberOfTimesLeverHasBeenPulledForThisBandit + 1)
# ```
#
#

# + [markdown] id="b-VohcmEjx2p" colab_type="text"
# ## Exercise 1: creating your first (very simple) Agent

# + [markdown] id="YRLF3R_Rj--G" colab_type="text"
# We are now ready to create your very first agent, the epsilon Greedy algorithm

# + [markdown] id="2CsZVD-KkKmx" colab_type="text"
# We will need to keep track of which agent is the best, we do this by creating a big table, with 10 cells, one for each bandit. In computer terms this table is known as an array. We are creating this array to keep track of which bandit is doing the best for every time we play a game.  As before we are using numpy to store our table.

# + id="1YbALrYCktJz" colab_type="code" colab={}
import numpy as np

# + [markdown] id="LuXsMBMTFosy" colab_type="text"
# we also want to randomly initialise our enviroment, do this by running the code below:

# + id="5cc_prTsFyhP" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 34} outputId="8ca883a3-b0a7-41d5-c218-2179f308a892"
#rerun this part of the code if you would like to "reset" or reinitialize your bandit environement
np.random.seed(42) 
env.reset()

# + [markdown] id="i10ZSUORnzus" colab_type="text"
# Let also make a variable that is the total number of bandits operating in our environment, **complete the code below:**

# + id="1ASjY3yNn8BG" colab_type="code" colab={}
numberOfBandits =10 #???? hint we mentioned this number above

# + [markdown] id="hZTZsSATlJNw" colab_type="text"
# we are going to call the array (remember this is just a table) which keeps track of which bandit is the best, a **Q TABLE**. 

# + [markdown] id="EFeQkqeTmW64" colab_type="text"
# Q in this case just stands for quality. The idea being that the number that is the highest in our table, is associated with the action we would like to take with the highest quality. but we are getting a little ahead of ourselves. If you did not get that dont worry. It will all become apparent in a couple of lines of reading!

# + [markdown] id="iMBmGTs8nZ3M" colab_type="text"
# we also want to initialize our table so that all of the values are ones at the start -avoiding a divide by zero error in our algorithm- so lets do that

# + id="Zl4fw0YRnm2a" colab_type="code" colab={}
q_table = np.zeros(numberOfBandits)

# + [markdown] id="Wadn4RHooEz3" colab_type="text"
# Remember, we would like to keep track of the bandit with the *HIGHEST AVERAGE PAYOUT*. In order to do this we need another table to keep track of the number of times each bandit has been "pulled". We also want this table to be fulled with ones, this is to stop a divide by zero error later. **complete the code below:**

# + id="0VfVlrZEoWdi" colab_type="code" colab={}
n_table = np.ones(numberOfBandits)#????

# + [markdown] id="vgYR_56KpJ-_" colab_type="text"
# In the lecture we talked a lot about epsilon. Lets create a new variable called "epsilon" and initialize it to 0.9. **complete the code below:**

# + id="yLi2bsWCpf65" colab_type="code" colab={}
epsilon = 0.9#????

# + [markdown] id="T4cT-7cPtFQj" colab_type="text"
# Below is the pseudo code for the epsilon greedy algorithm. In this implementation we are not going to vary epsilon - it is going to be a fixed number. 
#
# This means that we will keep our exploration rate fixed.

# + [markdown] id="7OaxMbsU37Sc" colab_type="text"
# We will now give you the pseudo code (code recipe) on how to implement the epsilon greedy algorithm. Look below at the "recipe". If you need help, all the ingredients on how to implement each part are below in the pseudo code.

# + [markdown] id="BaKRJx3Ft8US" colab_type="text"
#
#
# ```
# create a for loop, to loop 1000 times
#
#       (IF STATEMENT, inside the loop,) generate a random number between 0 and 1 , if this number is less than 
#       Epsilon enter "exploitation mode" aka use the best bandit we have discovered so far
#       
#             (inside the if statement) get the POSITION (index) in our array of the current max value within our
#             table, this index is the bandit that is giving the best payout so far.
#             
#             (inside the if statement) set your ACTION variable equal to the index we discovered in the last 
#             statement
#             
#       (ELSE, otherwise..) if the number is greater than or equal to Epsilon, go into "exploration mode" and
#       choose a bandit at random 
#       
#           
#             (inside the if statement) generate a random number between 0 and THE_NUMBER_OF_BANDITS
#             
#             (inside the if statement) set your ACTION variable to equal to the random number we just generated.
#             
#         
#             
#       (inside the loop) feed our ACTION variable into our environment by updating it with a step generated 
#       by either of the steps above
#       
#       (inside the loop) now that we have gained some new information from our environment we want to update our 
#       Q_table. We do this using the formula: Q_n+1 = Q_n + (R - Q_n)/n or in simpler english:
#       
#       NewQvalue = OldQvalue + ((reward - OldQvalue)/numberOfTimesLeverHasBeenPulledForThisBandit)
#       
#       Lets think about the intituition of what this forumula is doing. Implement the formula in code.
#       
#       
#       (inside the loop) now that we have updated our Q table, we also need to update the table that is keeping
#       track of how many times each bandit's lever has been pulled. Do this by adding +1 in the position
#       of our currently selected bandit in the N_TABLE array
#       
#       
# (OUTSIDE the loop) once everything is done, we would like to print the Bandit with the highest score! Using
# a print statement, and numpy's argmax function, using our Q table, print the bandit with the highest
# AVERAGE payout
#       
#    
#    
#   
# ```
#
#

# + [markdown] id="4hqS1uXaUgX7" colab_type="text"
# .

# + [markdown] id="FnAfeNYWUBod" colab_type="text"
# **Here are all the pieces required to build the above:**

# + [markdown] id="vBbCYa3QUhrc" colab_type="text"
# How to make a loop that loops a 10 times:

# + id="KKGvYZUmUl1S" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 187} outputId="a5740471-2c03-4e9e-8c4a-96393844f028"
#notice the indentation of the print statement, this indicates that this function (the print statement), is "inside" the loop

for k in range(10):
  print("now in loop iteration number: ",k)

# + [markdown] id="XAU8WC0bUzs-" colab_type="text"
# How to generate a random number between 0 & 1 using numpy:

# + id="E2jW4cMFU9xt" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 34} outputId="73590608-5bd7-41f8-b803-c61ecef90e47"
#rerun this multiple times to generate a different random number
#remember np stands for "numpy" as declared above

randomNumber = np.random.random(1)[0] 

print("random number is ", randomNumber)

# + [markdown] id="n28zjNfQVXj3" colab_type="text"
# How to create an if statement to see if our random number is less than epsilon

# + id="elf3xvsVWcbW" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 34} outputId="e913f102-a03e-4158-ad51-3645d41c4c56"
#epsilon should be set to 0.9 above...

if(randomNumber < epsilon):
  print("doing something less than epsilon")

# + [markdown] id="-ka0VSYXWu8l" colab_type="text"
# How to create an if statement to see if our random number is less than epsilon and do something else if this is not the case

# + id="ab6kK3F0Ws3J" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 34} outputId="02869b0f-0599-461a-d8b1-edafde7abfde"
if(randomNumber < epsilon):
  print("doing something less than epsilon")
else:
  print("doing something more than epsilon")

# + [markdown] id="oYAlpVatXhwL" colab_type="text"
# How to lookup the best bandit's *location* (index) in our Q table, again remember there are 10 values to choose from (10 bandits) & we need to look up the index (location) in our Q_table array of the bandit with the highest score. Numpy's argmax() function allows us to do this.
#

# + id="rXuQ87hkXlpS" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 34} outputId="2fdab7d9-633b-43e3-bbd4-aee2245ff7be"
best_bandit = np.argmax(q_table)

print("best bandit is ",best_bandit)

# + [markdown] id="hxbGaOG_YROy" colab_type="text"
# how to assign the index of our best action so that OpenAi gym can understand it:

# + id="IzD6aQHRYatW" colab_type="code" colab={}
action = best_bandit

# + [markdown] id="BYE8HoMPYmHX" colab_type="text"
# how to generate a random bandit ACTION number between 0 to 9 to take:

# + id="nds0qfLCY3d4" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 34} outputId="7d956607-2a78-44bc-c138-fd2e0fd6598a"
#run this multiple times to see different results
random_number = np.random.randint(numberOfBandits)-1

print(random_number)

# + [markdown] id="4A6eKeMfY7MG" colab_type="text"
# how to update OpenAi gym one time step into the future

# + id="vRPq3EN-ZB08" colab_type="code" colab={}
observation, reward, done, info = env.step(action) 

# + [markdown] id="-PQn_PFOZP_Q" colab_type="text"
# How to directly access a numpy array's value, in this example we are accessing the best bandit's so far payout

# + id="JQ8CQpAlZVX2" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 34} outputId="12d5efa4-b6e8-4d90-efc5-fd0185856d9c"
example_table = np.array([2,3,1,0,5,7,9,8,6,4])

example_table[best_bandit]

# + [markdown] id="YvrXwE05fZMc" colab_type="text"
# To print out all the values of a numpy array for testing purposes, use this code

# + id="eiyUZQggff5e" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 34} outputId="59e461fc-bf56-45dd-fdbc-35c0a5fc1753"
print(example_table[:])

# + [markdown] id="ie_RqC6LdCAQ" colab_type="text"
# after looping 1000 times we would like to present our result - the best bandit out of the 10. Using Numpys argmax function:

# + id="mvp8I-XBdfKv" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 34} outputId="82588db0-42dc-47fc-8cdd-72949e4c9c44"
# think about the result after running this relative to our example table above:

print('and the best bandit is....', np.argmax(example_table))

# + [markdown] id="g9KqH2ScdvwI" colab_type="text"
# ## Code up Epsilon-greedy below

# + [markdown] id="kmH313czeI0u" colab_type="text"
# We now have everything required to code up epsilon greedy. Using our **recipe**, and all the **blocks above,** code up an implementation of epsilon greedy below:

# + id="LAIuudWtc-Sx" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 34} outputId="1618e05f-beb0-4cc0-c6b7-fdd930596d17"
import numpy as np

env.seed(34)

numberofbandits = 10
q_table = np.zeros(numberofbandits)
n_table = np.ones(numberofbandits)

epsilon = 0.9

## YOUR CODE GOES HERE

for i in range(1000):
  rand = np.random.random(1)
  if rand<epsilon:
    best_bandit = np.argmax(q_table)
    action = best_bandit
  else:
    random_number = np.random.randint(numberofbandits)
    action = random_number
  observation, reward, done, info = env.step(action)
  #print(action, observation, reward)
  q_table[action] = q_table[action] + (reward - q_table[action])/(n_table[action]+1) 
  n_table[action]=n_table[action]+1
  #NewQvalue = OldQvalue + ((reward - OldQvalue)/numberOfTimesLeverHasBeenPulledForThisBandit + 1)
  #print(np.round(q_table,2))
print(np.argmax(q_table))

# + id="kYAchRLZzi0w" colab_type="code" colab={}


StarcoderdataJupyterScriptsDedupFiltered