---
layout: slides
title: "On gaining and losing"
category: hidden
type: course
semester: Fall 2020
college: Stevens Institute of Technology
permalink: /thought-and-behavior/exploring-and-exploiting/
---

# On exploring and exploiting.

<img src="https://images.unsplash.com/photo-1525018667593-176858caed6a" width="55%" />

???

---

# The multiarmed bandit

- Multiple options with unknown expected value
- How do we maximize gain?

---

- Bayesian A/B testing
- Clinical trials
- Ordering off a menu

---

# Exploration vs. exploitation

- Exploration: learn new information
- Exploitation: use information we already have

---

# Solution 1: ε-greedy

- Select a value of ε
- One each trial, flip coin weighted ε
- If heads, select random arm
- If tails, select best arm (thus far)

---

class: center, middle, statement, dark

Let's try it!

---

# Sidenote

- Contextual bandits!

---

# Solution 2: UCB

- Set prior over probability of reward for each arm

---

# Solution 2: UCB

<img src="https://miro.medium.com/max/864/1*ZUA3LqodHlQLrIjVqlUAXw.png" />

---

# Solution 2: UCB

- Set prior over probability of reward for each arm
- Set counts for number of pulls and rewards (i.e., wins vs. losses)
- Compute the upper-confidence bound (UCB) for each arm
- Pull arm with highest UCB

---

class: center, middle, statement, dark

Let's try it!

---

# Solution #3: Thompson sampling

- Set prior over probability of reward for each arm
- Set counts for number of pulls and rewards (i.e., wins vs. losses)

???

Should look familiar...

---

# Solution #3: Thompson sampling

- Set prior over probability of reward for each arm
- Set counts for number of pulls and rewards (i.e., wins vs. losses)
- Sample a belief state for each arm
- Pick the best one

???

Should look familiar...

---

class: center, middle, statement, dark

Let's try it!

---

# Minimizing regret

Regret is how much worse we'd expect to do using the strategy we did, in comparison to the optimal strategy. Compare to a Marrian computational-level analysis.

---
class: center, middle
# Fin.

Add slides on exploring vs. exploiting
Add slides on exploring vs. exploiting
