# Homework #5 (Due 10/14/2021, 11:59pm)
## Hierarchical Models and the Theory of Variational Inference

**AM 207: Advanced Scientific Computing**

**Instructor: Weiwei Pan**

**Fall 2021**

**Name: Jiahui Tang**

**Students collaborators: Yujie Cai**

### Instructions:

**Submission Format:** Use this notebook as a template to complete your homework. Please intersperse text blocks (using Markdown cells) amongst `python` code and results -- format your submission for maximum readability. Your assignments will be graded for correctness as well as clarity of exposition and presentation -- a “right” answer by itself without an explanation or is presented with a difficult to follow format will receive no credit.

**Code Check:** Before submitting, you must do a "Restart and Run All" under "Kernel" in the Jupyter or colab menu. Portions of your submission that contains syntactic or run-time errors will not be graded.

**Libraries and packages:** Unless a problems specifically asks you to implement from scratch, you are welcomed to use any `python` library package in the standard Anaconda distribution.

```python
### Import basic libraries
import numpy as np
import pandas as pd
import sklearn as sk
from sklearn.linear_model import LinearRegression
import matplotlib.pyplot as plt
%matplotlib inline

```

```python
# data_df is the dataframe of your data
# estimates_df is a numpy array of cancer rate estimates, one for each county
def scatter_plot_cancer_rates(data_df, estimates=None):
    ax = data_df.plot(kind='scatter', x="pop",y="pct_mortality", alpha=0.1, color="grey")
    bot_kcancer_counties = data_df.sort_values(by='pct_mortality',ascending=True)[:300]
    top_kcancer_counties = data_df.sort_values(by='pct_mortality',ascending=False)[:300]
    top_kcancer_counties.plot(kind='scatter',x="pop",y="pct_mortality",
                              alpha=0.1, color="red", ax=ax, logx=True, label = "highest cancer rate")
    bot_kcancer_counties.plot(kind='scatter',x="pop",y="pct_mortality",
                              alpha=0.1, color="blue", ax=ax, logx=True, label = "lowest cancer rate")
    if estimates is not None:
        ax.plot(data_df['pop'], 5 * estimates, '.', alpha=0.2, color="green", label='mean estimates')
    ax.set_ylim([-0.0001, 0.0003])

```

```python
# data_df is the dataframe of your data
# estimates_df is a numpy array of cancer rate estimates, one for each county
def scatter_plot_cancer_rates_orange(data_df, estimates=None):
    ax = data_df.plot(kind='scatter', x="pop",y="pct_mortality", alpha=0.1, color="grey")
    bot_kcancer_counties = data_df.sort_values(by='pct_mortality',ascending=True)[:300]
    top_kcancer_counties = data_df.sort_values(by='pct_mortality',ascending=False)[:300]
    top_kcancer_counties.plot(kind='scatter',x="pop",y="pct_mortality",
                              alpha=0.1, color="red", ax=ax, logx=True, label = "highest cancer rate")
    bot_kcancer_counties.plot(kind='scatter',x="pop",y="pct_mortality",
                              alpha=0.1, color="blue", ax=ax, logx=True, label = "lowest cancer rate")
    if estimates is not None:
        ax.plot(data_df['pop'], 5 * estimates, '.', alpha=0.2, color="orange", label='mean estimates')
    ax.set_ylim([-0.0001, 0.0003])

```

# Problem Description: Understanding EM and Variational Inference

In this problem, we will draw concrete connections between EM and variational inference by applying both methods to a certain class of latent variable models. You'll need to refer to relevant lecture notes on the derivations of EM and the derivation of the variational inference objective. This is an essay question that requires you to engage with complex derivations at a productive but still high level. No implementation is required.

#### Non-Bayesian Latent Variable Model
Recall the class of latent variable models we studied in lecture:

#### Bayesian Latent Variable Model
A Bayesian version of the same class of models involve adding priors for the model parameters:

1. **(Comparing ELBOs)** For the above type of Bayesian latent variable model, write down the ELBO for variational inference with a mean field variational family. Compare the variational inference ELBO for the Bayesian model to the expectation maximization ELBO for the non-Bayesian model. What are the differences and similarities between these two ELBOs?

  In both EM and variational inference we optimize the ELBO. Compare the update steps in EM to the update steps in Coordinate Ascent Variational Inference, draw a concrete analogy between them.

  ***Hint:*** To make both ELBO's comparable, make sure that both are in terms of $z, y, \theta, \phi$.

**Answer:**

ELBO for **EM** for non-Bayesian model:

$$ELBO^{\text{EM}}(\theta, \phi, q(z))  = \mathbb{E}_{z\sim q(z)}\left[\log\left(\frac{p(y, z|\theta, \phi)}{q(z)}\right)\right] = \mathbb{E}_{z\sim q(z)} \left[\log \left(\frac{P(y|z,\phi)p(z|\theta)}{q(z)}\right)\right]
$$

The ELBO for **Variational Inference** with a mean field variational family, where  $\psi=(\theta,\phi,Z)$:

$$ELBO^{\text{VI}}(\psi) = \mathbb{E}_{\psi \sim q(\psi|\lambda)}\left[\log\left(\frac{p(\psi, Y_1, \ldots, Y_N |a,b))}{q(\psi | \lambda)} \right)\right] $$

Write out in terms of $z, y, \theta, \phi$

$$
ELBO = \mathbb{E}_{z, \phi, \theta \sim q(z, \phi, \theta |\lambda_1, \lambda_2, \lambda_3)} \left[\log\left(\frac{p(y, z, \theta, \phi|a, b)}{q(z, \phi, \theta |\lambda_1, \lambda_2, \lambda_3)}\right)\right]
$$

**The differences and similarities between these two ELBOs:**

**Similarities:**
 1. They all incorporate an auxilary function $q(z)$ in order to take the gradients.
 2. Both take an expectation log ratio between joint distribution over distributions that are not that of the observed variable (auxilary distribution $q$ or variational distribution $q$).

**Differences:**
1. For VI, we introduce an additional parameter $\lambda$ for the variation family $Q$, and it is maximized using expetctation wrt the variation family.
2. In EM, $\phi$ and $\theta$ are known constants, and produce point estimation; In VI, they are random variables with distribution and we derive posterior over them;  Thus in EM, we optimize the ELBO wrt $\theta$ and $\phi$ as point estimates; In variational inference, we optimize over the distribution of $\theta$ and $\phi$
3. The difference between the ELBO and the KL divergence is the log normalizer (i.e. the evidence), which is the quantity that the ELBO bounds.
4. In VI, we take the expectation over the distribution of a hyperparameter. In EM, we take the expectation over the distribution of the latent variable.

**Special Case**:

EM is a special case of ELBO if we use mean-field VI where the variational distributions $q$ are assumed to be point estimations. That is, Let $\theta^{*}$ and $\phi^{*}$ be the unknown location of this point mass:
    $$q_{\theta}(\theta)=\delta\left(\theta-\theta^{*}\right)$$
     $$q_{\phi}(\phi)=\delta\left(\phi-\phi^{*}\right)$$

VI will minimize a KL divergence, and so when we minimumize over the variational family, it is the E-step of EM; and the further when minimizing over $\theta^{*}$ is the M-step of EM.

2. **(Comparing ELBOs and KL-divergences)** Recall that the original objective of variational inference is to minimize a KL-divergence, we rewrote the objective to be that of maximizing the ELBO. Why is directly minimizing the KL-divergence in the original objective difficult (be specific about wherein the difficulty lies)?

  In the derivation of the E-step of EM, we reframed an maximization of the ELBO problem as a minimization of a KL-divergence problem. In this case, why was the KL-divergence easier to minimize and the ELBO harder to maximize (use the instantiation of the E-step for Gaussian Mixture Models in Lecture 7 to help support your answer)?

  In the notes for Lecture 8, we introduce a way to maximize the variational inference ELBO -- through coordinate ascent. In the derivation of the updates for coordinate ascent, there is a place where we reframed an maximization of the ELBO problem as an equivalent minimization of a KL-divergence problem. Write down the exact form of this equivalence (the two expressions are separated in the derivation by a bunch of lines, you'll need to identify both parts that you need). In this case, why was the KL-divergence easier to minimize and the ELBO harder to maximization (use the instantiation of the update for Gaussian Mixture Models in Lecture 8 to help support your answer)?

  Based on this analysis, can you draw some general conclusions about when we'd prefer to minimize the KL-divergence versus when we'd prefer to maximize the ELBO?

**Answer:**

Recall that the original objective of variational inference is to minimize a KL-divergence, we rewrote the objective to be that of maximizing the ELBO. Why is directly minimizing the KL-divergence in the original objective difficult (be specific about wherein the difficulty lies)?

In VI, we are finding a tractable distribution $q$ that best approximate the complex distribution $p$ by minimizing KL divergence.

$$
\begin{aligned}
\lambda^* &= \underset{\lambda}{\text{argmin}}\; D_{\text{KL}}(q(\psi|\lambda) \| p(\psi|Y_1, \ldots, Y_N, a, b))) \\
&= \underset{\lambda}{\text{argmin}}\; \mathbb{E}_{\psi \sim q(\psi|\lambda)}\left[\log\left(\frac{q(\psi | \lambda)}{p(\psi|Y_1, \ldots, Y_N))}\right) \right]
\end{aligned}$$

Write out in terms of $z, y, \theta, \phi$

$$
\underset{\lambda}{\min} D_{\text{KL}}[q(z, \phi, \theta |\lambda_1, \lambda_2, \lambda_3) \| p(\theta, \phi, z|y, a, b)] = \underset{\lambda}{\min} \mathbb{E}_{z, \phi, \theta \sim q(z, \phi, \theta |\lambda_1, \lambda_2, \lambda_3)} \left[\log \left( \frac{q(z, \phi, \theta |\lambda_1, \lambda_2, \lambda_3)}{p(\theta, \phi, z|y, a, b)}\right) \right]
$$

By Bayes' rule, the denominator in the log ratio

$$
p(\theta, \phi, z|y, a, b) = \frac{p(y, z, \theta, \phi| a, b)}{p(y | a, b)} = \frac{p(y, z, \theta, \phi| a, b)}{\int p(y, z, \theta, \phi| a, b) \mathrm{d}(z, \theta,\phi)} \propto p(y, z, \theta, \phi| a, b)
$$

Therefore
\begin{aligned}
D_{\text{KL}}[q(z, \phi, \theta |\lambda_1, \lambda_2, \lambda_3) \| p(\theta, \phi, z|y, a, b)] &= \mathbb{E}_{z, \phi, \theta \sim q(z, \phi, \theta |\lambda_1, \lambda_2, \lambda_3)} \left[\log \left( \frac{q(z, \phi, \theta |\lambda_1, \lambda_2, \lambda_3) p(y | a, b)}{ p(y, z, \theta, \phi| a, b)}\right) \right]\\
    & =  \mathbb{E}_{z, \phi, \theta \sim q(z, \phi, \theta |\lambda_1, \lambda_2, \lambda_3)} \left[\log q(z, \phi, \theta |\lambda_1, \lambda_2, \lambda_3) + \log p(y | a, b) - \log p(y, z, \theta, \phi| a, b) \right]\\
    \end{aligned}

But that $\log p(y| a, b) = \int p(y, z, \theta, \phi| a, b)\mathrm{d}(z, \theta,\phi) $ involve integration that's intractable to integrate over $(z, \theta,\phi)$.

But with ELBO,

\begin{aligned}
ELBO &= \mathbb{E}_{z, \phi, \theta \sim q(z, \phi, \theta |\lambda_1, \lambda_2, \lambda_3)} \left[\log\left(\frac{p(y, z, \theta, \phi|a, b)}{q(z, \phi, \theta |\lambda_1, \lambda_2, \lambda_3)}\right)\right]\\
&= \mathbb{E}_{z, \phi, \theta \sim q(z, \phi, \theta |\lambda_1, \lambda_2, \lambda_3)} \left[\log p(y, z, \theta, \phi|a, b) - \log q(z, \phi, \theta |\lambda_1, \lambda_2, \lambda_3) \right]
\end{aligned}

The term no longer involves $\log p(y| a, b)$, making it easier to compute.
Thus, directly minimizing the KL-divergence in the original objective is difficult, we used maximizing ELBO.The issue is that the gradient taken is with respect to the parameter $\psi$ of the distribution of which we are taking the expectation - i.e. we cannot push the gradient into the expectation.

In the derivation of the E-step of EM, we reframed an maximization of the ELBO problem as a minimization of a KL-divergence problem. In this case, why was the KL-divergence easier to minimize and the ELBO harder to maximize (use the instantiation of the E-step for Gaussian Mixture Models in Lecture 7 to help support your answer)?

**Why KL-divergence is easier to minimize and the ELBO is harder to maximize in EM:**

1. Theoretically, when KL is minimized, we know the approximated distribution is the same as the true distribution.
2. Since we already know what one of the distributions are, we can just equate the other distribution to it and be done. Hence, KL divergence is easier to minimize in this situation
3. Practically, in ELBO, we need to conduct two sets of iterations, one for E-step and another for M step. Also, we only converge to the lower bound, which is not guaranteed to be the true optimized value.
4. Besides, for the E-step of EM, it is hard to maximize the ELBO with respect to the auxiliary $q$ function due to complex integration and multivariate gradients.

To be specific, in E-step,
$$
\underset{q}{\max} ELBO(\theta^*, \phi^*, q(z)) = \underset{q}{\max} \mathbb{E}_{z\sim q(z)}\left[\log\left(\frac{p(y, z|\theta^*, \phi^*)}{q(z)}\right)\right]
$$
(after M step, we get $\theta^*, \phi^*$ as optimized values)

To get that, we will take gradient of ELBO, that is, $\nabla \mathbb{E}_{z\sim q(z)}\left[\log\left(\frac{p(y, z|\theta^*, \phi^*)}{q(z)}\right)\right]$. But taking gradient with respect to the auxiliary $q$ function due to complex integration and multivariate gradients will be very hard.

Therefore, we think about minimize the KL-divergence, as it is easier to minimize KL divergence.

In the notes for Lecture 8, we introduce a way to maximize the variational inference ELBO -- through coordinate ascent. In the derivation of the updates for coordinate ascent, there is a place where we reframed an maximization of the ELBO problem as an equivalent minimization of a KL-divergence problem. Write down the exact form of this equivalence (the two expressions are separated in the derivation by a bunch of lines, you'll need to identify both parts that you need). In this case, why was the KL-divergence easier to minimize and the ELBO harder to maximization (use the instantiation of the update for Gaussian Mixture Models in Lecture 8 to help support your answer)?

**In coordinate ascent, now why we choose minimizing KL-divergence?**

Maximize ELBO is equivalent to minimize the KL-divergence in coordinate ascent.

$\underset{\lambda_i}{\max}\mathbb{E}_{\psi_i \sim q(\psi_i|\lambda_i)}\left[\mathbb{E}_{\psi_{-i} \sim q(\psi_{-i}|\lambda_{-i})} \left[\log \left(\frac{p(\psi,data)}{q(\psi|\lambda)}\right)\right]\right] $
$\equiv \underset{\lambda_i}{\min} D_{\text{KL}}\left[q(\psi_i|\lambda_i)|| z \exp \{\mathbb{E}_{\psi_{-i} \sim q(\psi_{-i}|\lambda_{-i})} [\log p(\psi_i,data|\psi_{-i}]\}\right]$

For the following, we will use the following notation:

\begin{aligned}
\lambda_{-i} &= [\lambda_1\; \ldots\; \lambda_{i-1}\; \lambda_{i+1}\; \ldots\; \lambda_{I}]\\
\psi_{-i} &= [\psi_1\; \ldots\; \psi_{i-1}\; \psi_{i+1}\; \ldots\; \psi_{I}]\\
q(\psi_{-i}|\lambda_{-i}) &= \prod_{j\neq i}q(\psi_{j}|\lambda_{j})\\
\end{aligned}

From lecture 10, we have shown that
    $$\mathbb{E}_{\psi \sim q(\psi|\lambda)}[\ldots] = \mathbb{E}_{\psi_i \sim q(\psi_i|\lambda_i)}\left[\mathbb{E}_{\psi_{-i} \sim q(\psi_{-i}|\lambda_{-i})}[\ldots] \right]$$

and the exact form of the equivalence:

\begin{aligned}
\underset{\lambda_i}{\max} ELBO(\lambda) &= \underset{\lambda_i}{\max} \mathbb{E}_{\psi_i \sim q(\psi_i|\lambda_i)}\left[\mathbb{E}_{\psi_{-i} \sim q(\psi_{-i}|\lambda_{-i})} \left[\log \left(\frac{p(Y_1, \ldots, Y_N, \psi_i | \psi_{-i})}{q(\psi_{i}|\lambda_{i})} \right)\right]\right]\\
&= \underset{\lambda_i}{\min}D_{\text{KL}} \left[ q(\psi_{i}|\lambda_{i})\| \mathcal{Z}\exp\left\{\mathbb{E}_{\psi_{-i} \sim q(\psi_{-i}|\lambda_{-i})} \left[\log \left(p(Y_1, \ldots, Y_N, \psi_i | \psi_{-i}) \right)\right]\right\}\right].
\end{aligned}

As the last set of questions, if we set $q(\psi_{i}|\lambda_{i})$ equal to $\mathcal{Z}\exp\left\{\mathbb{E}_{\psi_{-i} \sim q(\psi_{-i}|\lambda_{-i})} \left[\log \left(p(Y_1, \ldots, Y_N, \psi_i | \psi_{-i}) \right)\right]\right\}$, we can solve the optimization. This will be easier than if we compute the gradient of of $ELBO$. In Gaussian Mixture models, in lec10 derivation, terms with subscript $-i$ will become constant after iterations.

Based on this analysis, can you draw some general conclusions about when we'd prefer to minimize the KL-divergence versus when we'd prefer to maximize the ELBO?

**Conclusion**

1. Look for complexity of mathematics when implementing the right method

2. We use KL divergence when we know one of the distributions exactly - then it is clear that the to minimize the divergence we just have to equate the other distribution to the other distribution

3. When the form of probability $P(A|B)$ is known and not intractible, we can use KL divergence minimization. When $P(A|B)$ is unknown and hard to compute, we can choose to work with ELBO, but this still require the joint of $A$ and $B$ to be tractable.

3. **(The Mean Field Assumption and Coordinate Ascent)**  Describe exactly when and how the mean field assumption is used in the derivation of the coordinate ascent updates.

**Answer:**

We assume that the joint $q(\psi)$ factorizes completely over each dimension of $\psi$, i.e. $q(\psi)= \prod_{i=1}^I q(\psi_i | \lambda_i)$. This is called the ***mean field assumption***. This assumes that each dimension of $q(\psi)$ is independent from the others, which may or may not be valid, but it helps to simplify the expression of $q(\psi)$, making it easier to find the ELBO. Thus it is a design choice to make, and it could also go wrong with this design choice.

The coordinate ascent algorithm maximize an objective function ELBO by iteractively maximizing over one dimension, while holding other dimension constant. That's why it requires us to be able to use mean field assumption and factorize $q(\psi)$ into individual pieces.

To use mean field assumption, we will maximize the $ELBO$ by following steps in the derivation of the coordinate ascent updates.

1. Use the mean-field assumption to break up the expectation $\mathbb{E}_{\psi \sim q(\psi|\lambda)}$ into an iterated expectation $\mathbb{E}_{\psi_i \sim q(\psi_i|\lambda_i)}\mathbb{E}_{\psi_{-i} \sim q(\psi_{-i}|\lambda_{-i})}$
2. We will rewrite the outer expectation $\mathbb{E}_{\psi_i \sim q(\psi_i|\lambda_i)}$ as a negative KL-divergence
3. We will then maximize the negative KL-divergence by setting the two arguments of the divergence equal to each other

4. **(Generalizability of CAVI)** Summarize what kind of derivations/math is needed in order instantiate Coordinate Ascent Variational Inference (CAVI) for a given new model (look at what we did for Gaussian Mixture Models in Lecture 8 and predict what you'd need to do for a new model). Based on this, discuss the potential draw backs of using CAVI for Bayesian inference in general. What do these draw backs imply about the practicality of variational inference as an inference method?

**Answer:**

We see that

$$
D_{\text{KL}} \left[ q(\psi_{i}|\lambda_{i})\| \mathcal{Z}\exp\left\{\mathbb{E}_{\psi_{-i} \sim q(\psi_{-i}|\lambda_{-i})} \left[\log \left(p(Y_1, \ldots, Y_N, \psi_i | \psi_{-i}) \right)\right]\right\}\right]
$$

is minimized when

$$
 q(\psi_{i}|\lambda_{i})\propto \exp\left\{\mathbb{E}_{\psi_{-i} \sim q(\psi_{-i}|\lambda_{-i})} \left[\log \left(p(Y_1, \ldots, Y_N, \psi_i | \psi_{-i}) \right)\right]\right\}.
$$

This is also exactly where the ELBO is maximized.

**Derivations/Math needed:**

1. We need mean field assumption to derive the variational distribution of each variable or parameter independently and separately.
2. For each i, we need to set $q(\psi_{i}|\lambda_{i})$ equal to $\mathcal{Z}\exp\left\{\mathbb{E}_{\psi_{-i} \sim q(\psi_{-i}|\lambda_{-i})} \left[\log \left(p(Y_1, \ldots, Y_N, \psi_i | \psi_{-i}) \right)\right]\right\}$ to solve for ELBO iteractively in CAVI.
3. We need lengthy algebraic manipulation to expand the expectation term
4. Computational wise also expensive, need to iterate through $\lambda_i$

**Drawbacks**
1. It may never converge
2. We will never be able to capture the full complexity of target distribution by sampling from posterior q
3. We are trading off fidelity for computational tracibility.
4. Computational heavy, we need to derive different $q(\psi_{i}|\lambda_{i})$ at each iteration for different models.
5. We need conduct lengthy posterior derivations for each latent variable / paramter which may result in intractable posterior distributions

**Particability**

It is still practical but it may be need some mathematical derivations when iteractively minimize KL. In each step, variety of complex algorithms and sampling methods can be used.

5. **(Generalizability of EM)** Summarize what kind of derivations/math is needed in order instantiate Expectation Maximization (EM) for a given new model (look at what we did for Gaussian Mixture Models in Lecture 9 and predict what you'd need to do for a new model). Based on this, discuss the potential draw backs of using EM for MLE inference in general. What do these draw backs imply about the practicality of EM as an inference method?

**Answer:**

0. **Initialization:** Pick $\theta_0$, $\phi_0$.
1. Repeat $i=1, \ldots, I$ times:

  **E-Step:**
$$q_{\text{new}}(Z_n) = \underset{q}{\mathrm{argmax}}\; ELBO(\theta_{\text{old}}, \phi_{\text{old}}, q) = p(Z_n|Y_n, \theta_{\text{old}}, \phi_{\text{old}})$$

  **M-Step:**
  \begin{aligned}
  \theta_{\text{new}}, \phi_{\text{new}} &= \underset{\theta, \phi}{\mathrm{argmax}}\; ELBO(\theta, \phi, q_{\text{new}})\\
  &= \underset{\theta, \phi}{\mathrm{argmax}}\; \sum_{n=1}^N\mathbb{E}_{Z_n\sim p(Z_n|Y_n, \theta_{\text{old}}, \phi_{\text{old}})}\left[\log \left( p(y_n, Z_n | \phi, \theta\right) \right].
\end{aligned}

For the **E-Step**, we just simply calculate the conditional distribution of each $Z_n|Y_n, \theta_\text{old}, \phi_\text{old}$. It is usually obtained by employing Bayesian Rule. (The $Z_n$ here is of discrete type.)

\begin{align}
p(Z_n|Y_n, \theta_\text{old}, \phi_\text{old}) = \frac{p(y_n|Z_n = k, \theta_\text{old})p(Z_n=k | \phi_{\text{old}})}{\sum_{k=1}^K p(y_n|Z_n = k,\theta_\text{old})p(Z_n=k | \phi_{\text{old}})}
\end{align}

For the **M-Step**, it usually involves expanding the expectation $\mathbb{E}$, splitting into two parts by property of conditional probability and optimizting two parts seperately to get the solution.

\begin{align}
\theta^*, \phi^*
&= \underset{\theta, \phi}{\mathrm{argmax}}\;\sum_{n=1}^N \mathbb{E}_{Z_n\sim p(Z_n|Y_n, \theta_\text{old}, \phi_\text{old})} \left[ \log\left(p(y_n, z_n|\theta, \phi)\right)\right] \\
&= \underset{\theta, \phi}{\mathrm{argmax}} \;\sum_{n=1}^N \mathbb{E}_{Z_n\sim p(Z_n|Y_n, \theta_\text{old}, \phi_\text{old})} \left[ \log p(y_n|z_n, \theta) + \log p(z_n|\phi)\right] \\
&= \underset{\theta, \phi}{\mathrm{argmax}} \;\sum_{n=1}^N \mathbb{E}_{Z_n\sim p(Z_n|Y_n, \theta_\text{old}, \phi_\text{old})} \log p(y_n|z_n, \theta) + \sum_{n=1}^N \mathbb{E}_{Z_n\sim p(Z_n|Y_n, \theta_\text{old}, \phi_\text{old})} \log p(z_n|\phi)
\end{align}

Since we know the $p(Z_n|Y_n, \theta_\text{old}, \phi_\text{old})$ in the previous E-Step, we can plug it in and take gradient to solve for $\theta_\text{new}$ and $\phi_\text{new}$.

For for a given new model, we need to re-derive the equation for $q(z)$ which is the posterior and the update function for M-step which needs taking the gradients in order to instantiate Expectation Maximization (EM). Also, we need to choose $q(z)$ smartly in order to easily re-derive the math.

The potential draw backs are that,
1. It has slow convergence.
2. It makes convergence to the local optima only.
3. It requires both the probabilities, forward and backward (numerical optimization requires only forward probability), to be tractable so we can actually implement it.

Hence, in order to make EM easy to implement for any given latent variable model, we need to ensure that the original distribution has tractable form so that we can update something in the M step. We also should make sure that the posterior is easy to compute so we can use it easily in the E step.

Although EM has drawbacks like those listed above, we can still use it and it is heavily used in fields like Biostatistics. And EM is a pretty straight-forward method to solve a latent variable model.

# Problem Description: Modeling Kidney Cancer Data
In this problem, we will continue to work with the US Kidney Cancer Data set, `kcancer.csv`. This is a dataset of kidney cancer frequencies across the US over 5 years on a per county basis.

**In this homework, we focus on comparing different types of models for this data set.**

## Part I: Empirical Bayes
Let $N$ be the number of counties; let $y_j$ the number of kidney cancer case for the $j$-th county, $n_j$ the population of the $j$-county and $\theta_j$ be the underlying kidney cancer rate for that county. The following is a Bayesian model for our data:

\begin{aligned}
y_j | \theta_j &\sim Poisson(5 \cdot n_j \cdot \theta_j), \quad j = 1, \ldots, N\\
\theta_j &\sim Gamma(\alpha, \beta), \quad j = 1, \ldots, N
\end{aligned}

where $\alpha, \beta$ are hyper-parameters of the model.

1. **(Visualize the raw cancer rates)** Produce a scatter plot of the raw cancer rates (pct mortality) vs the county population size (in log scale). Highlight the top 300 raw cancer rates in red. Highlight the bottom 300 raw cancer rates in blue. What can you say about the counties with the highest and lowest raw cancer rates.

```python
df = pd.read_csv("kcancer.csv")
y = df['dc'].values
n = df['pop'].values

```

```python
#plt.figure(figsize = (10,10))
scatter_plot_cancer_rates(df)
plt.title("raw cancer rates vs the county population size")
plt.show()
```

**Answer:**

The countries with highest and lowest cancer rate are those countries with low populations, located on the left side of the scatterplot. Both the highest rates mortality and the lowest rates mortality counties are not the points have large population.

This could happen due to drawbacks of MLE estimates, it tends to overfit when data is scarce. Countries with less population may only have few data points, thus resulting in overfitting, exaggerated effect and extreme pct_mortality rate.

2. **(Empirical Bayes)** Using Empirical Bayes and moment matching, choose values for the hyperparameters $\alpha, \beta$ based on your data. Use these values of $\alpha$ and $\beta$ to obtain posterior distributions for each county.

***Hint:*** You'll first need to derive the fact that the ***evidence*** for a Poisson-Gamma model has a Negative Binomial distribution.

From the last homework, we first derived $p(\theta_j | \theta_{-j}, y, \alpha, \beta)= Ga(\theta_j; \alpha+y_j, 5n_j+\beta)$

**Part I.** By Bayes' Theorm,
$$
\begin{align}
p(\theta_j | \theta_{-j}, y, \alpha, \beta) &= \frac{p(\theta| y, \alpha, \beta)}{p( \theta_{-j}| y, \alpha, \beta)} \propto p(\theta| y, \alpha, \beta) = \prod_{i=1}^N p(\theta_i|y, \alpha, \beta) \\
\end{align}
$$

By holding $\theta_{-j}, y, \alpha, \beta$ constant, we could see that $p(\theta| y, \alpha, \beta)$ only depends on $$p(\theta_j| y_j, \alpha, \beta) =\frac{ p(\theta_j, y_j|\alpha, \beta)}{p(\alpha, \beta)}\propto p(\theta_j, y_j|\alpha, \beta) = p(y_j|\theta_j, \alpha, \beta)p(\theta_j|\alpha,\beta)$$.

Then, we could compute the joint distribution of $p(\theta_j, y_j|\alpha, \beta)$ as the product of the two distributions.

$$
\begin{align}
p(\theta_j, y_j|\alpha, \beta) &=p(y_j|\theta_j, \alpha, \beta)p(\theta_j|\alpha,\beta)\\ &= Poisson(y_j; 5\cdot n_j \cdot \theta_j)\cdot(\theta_j; \alpha, \beta)\\
&= \frac{(5n_j\theta_j)^{y_j}e^{-5n_j\theta_j}}{y_j!}\cdot\frac{\beta^\alpha\theta_j^{\alpha-1}e^{-\beta\theta_j}}{\Gamma(\alpha)}\\
&\propto \theta_j^{\alpha+y_j-1}e^{-(5n_j+\beta)\theta_j}\\
&= Ga(\theta_j; \alpha+y_j, 5n_j+\beta)
\end{align}
$$
Q.E.D.

Thus, we derived that $p(\theta_j | \theta_{-j}, y, \alpha, \beta)= Ga(\theta_j; \alpha+y_j, 5n_j+\beta)$

Then we prove $p(y_j) = NB\left(r, p\right)$ using the fact: $\Gamma(\alpha) = (\alpha - 1)!$
\begin{aligned}
p(y_j) &= \frac{p(y_j | \theta_j) p(\theta_j)}{p(\theta_j | y_j)}\\
    &= \frac{Poiss(y_j; 5 \cdot n_j \cdot \theta_j) Ga(\theta_j; \alpha, \beta)}{Ga(\alpha + y_j, 5 n_j  + \beta)}\\
&= \frac{\frac{1}{y_j!} Exp[5 \cdot n_j \cdot \theta_j] \frac{\beta^\alpha}{\Gamma(\alpha)} \theta_j^{\alpha - 1} Exp[-\beta\theta_j]}{\frac{\left(5 n_j  + \beta\right)^{\left( \alpha + y_j\right)}}{\Gamma\left(\alpha + y_j\right)} \theta_j^{\alpha + y_j - 1} Exp[-\left(5 n_j  + \beta\right) \theta_j]}\\
&= \frac{\beta^\alpha\Gamma\left(\alpha + y_j\right)  \left(5 \cdot n_j\right)^{y_j} \theta_j^{\alpha + y_j - 1}Exp[-\left(5 \cdot n_j  + \beta\right)\theta_j]}{y_j!\Gamma(\alpha) \left(5 n_j  + \beta\right)^{\left( \alpha + y_j\right)}\theta_j^{\alpha + y_j - 1} Exp[-\left(5 n_j  + \beta\right) \theta_j]}\\
&= \frac{\beta^\alpha\Gamma\left(\alpha + y_j\right)  \left(5 \cdot n_j\right)^{y_j}}{y_j!\Gamma(\alpha) \left(5 n_j  + \beta\right)^{\left( \alpha + y_j\right)}}\\
&= \frac{\Gamma\left(\alpha + y_j\right)}{y_j!\Gamma(\alpha)}\frac{\beta^\alpha}{\left(5 n_j  + \beta\right)^{\alpha}} \frac{\left(5 \cdot n_j\right)^{y_j}}{\left(5 n_j  + \beta\right)^{y_j}}\\
&= \frac{\Gamma\left((\alpha + y_j - 1) + 1\right)}{y_j!\Gamma(\alpha)} \left( \frac{\beta}{5 n_j  + \beta}\right)^\alpha \left( \frac{5 \cdot n_j}{5 n_j  + \beta}\right)^{y_j}\\
&= \frac{(\alpha + y_j - 1)!}{y_j!\Gamma(\alpha)}\left( \frac{\beta}{5 n_j  + \beta}\right)^\alpha \left( \frac{5 \cdot n_j}{5 n_j  + \beta}\right)^{y_j}\\
&= \frac{(\alpha + y_j - 1)!}{y_j!(\alpha - 1)!}\left( \frac{\beta}{5 n_j  + \beta}\right)^\alpha \left( \frac{5 \cdot n_j}{5 n_j  + \beta}\right)^{y_j}\\
&= \binom{\alpha + y_j - 1}{y_j}\left( \frac{\beta}{5 n_j  + \beta}\right)^\alpha \left( \frac{5 \cdot n_j}{5 n_j  + \beta}\right)^{y_j}\\
&= NB\left(\alpha, \frac{5 n_j}{ 5 n_j + \beta}\right)
\end{aligned}

The theoritical mean and variance for negative binomial distribution:
\begin{align}
\mathbb{E}\left[y_j\right] &=  \frac{\alpha5n_j}{\beta} \\
\text{Var}\left[y_j\right] &= \frac{\alpha 5 n_j (5n_j+\beta)}{\beta^2}
\end{align}

Then, the empirical counterparts adjusted by the population of each county:
\begin{aligned}
\widehat{\mathbb{E}}\left[\frac{y_j}{n_j}\right] &= \frac{5\alpha}{\beta}\\
\widehat{\text{Var}}\left[\frac{y_j}{n_j}\right] &= \frac{5 \alpha (5 n_j+\beta)}{n_j\beta^2}  = \frac{5 \alpha(5\bar{n}+\beta)}{\bar{n}\beta^2}
\end{aligned}
For the variance case we shall put population mean instead of $n_j$

Solve and get:
\begin{aligned}
\alpha &= \frac{\bar{pop}\widehat{\mathbb{E}}^2 }{\bar{pop} \widehat{\text{Var}} - \widehat{\mathbb{E}}} \\ \\
\beta &= \frac{5 \bar{pop}  \widehat{ \mathbb{E}} }{\bar{pop}\widehat{\text{Var}} - \widehat{\mathbb{E}}}
\end{aligned}

```python
e_hat = np.mean(y/n)
var_hat = np.var(y/n)
pop_mean = np.mean(n)

alpha = pop_mean*e_hat**2/(pop_mean*var_hat - e_hat)
beta = 5*pop_mean*e_hat/(pop_mean*var_hat - e_hat)

print("empirical cancer rate population adjusted mean: = {}".format(e_hat))
print("empirical cancer rate population adjusted var = {}".format(var_hat))
print("empirical population mean = {}".format(pop_mean))
print('')

print("alpha = {}".format(alpha))
print("beta = {}".format(beta))

```

    empirical cancer rate population adjusted mean: = 5.786552354108626e-05
    empirical cancer rate population adjusted var = 2.527523010812238e-09
    empirical population mean = 160512.41021522647

    alpha = 1.5451734918796929
    beta = 133514.17193888978

Knowing from the derivation that $$\theta_i|y_i \sim Ga(\alpha + y_j, 5 n_j  + \beta)$$
we can sample $\theta$ from $Ga(1.5451734918796929 + y_j, 5 n_j  + 133514.17193888978)$ and compute the mean or mode from the samples:

```python
# update estimate
thetas = []
for i in range(5000):
    thetas.append(np.random.gamma(alpha + df['dc'].values, 1. / (beta + 5 * n)))
thetas = np.array(thetas)
thetas_mean = np.mean(thetas, axis=0)
```

3. **(Posterior Means)** Produce a scatter plot of the raw cancer rates (pct mortality) vs the county population size (in log scale). Highlight the top 300 raw cancer rates in red. Highlight the bottom 300 raw cancer rates in blue. Finally, on the same plot again, scatter plot the posterior mean cancer rate estimates vs the county population size, highlight these means in green.

  Using the scatter plot, explain why using the posterior means (from our model) to estimate cancer rates is preferable to studying the raw rates themselves.

```python
scatter_plot_cancer_rates(df, estimates=thetas_mean)
plt.legend()
plt.show()
```

**Answer:**

Comparing to previous plot, we can observe that the posterior means to estimate cancer rate produce less extreme pct_mortality result. For countries with low population, it no longer has extreme high or low cancer rate, instead, they trend towards population average. It is preferable because the mean estimate mitigates the effect of small population size and overfitting problem.

## Part II: Hierarchical Bayes
Rather than choosing fixed constants for the hyperparameters $\alpha, \beta$, following the Bayesian philosophy, we typically put additional priors on quantities of which we are uncertain. That is, we model the kidney cancer rates using a ***hierarchical model***:

\begin{aligned}
y_j| \theta_j &\sim Poisson(5 \cdot n_j \cdot \theta_j), \quad j = 1, \ldots, N\\
\theta_j | \alpha, \beta &\sim Ga(\alpha, \beta), \quad j = 1, \ldots, N\\
\alpha &\sim Ga(a, b)\\
\beta &\sim Ga(c, d)
\end{aligned}
where $a, b, c, d$ are hyperparameters.

1.  **(Posterior Marginal Means)** Produce a scatter plot of the raw cancer rates (pct mortality) vs the county population size (in log scale). Highlight the top 300 raw cancer rates in red. Highlight the bottom 300 raw cancer rates in blue. Finally, on the same plot again, scatter plot the mean of the posterior marginal distribution over $\theta_j$, i.e. $p(\theta_j|y_1, \ldots, y_N)$, vs the county population size (in log scale), highlight these means in orange.

  You should use `pymc3` to sample from the posterior. Compare `pymc3`'s sampler with your sampler from the previous homework, what is the difference (if any) in the performance of these samplers?

```python
# import relevant package
import pymc3 as pm
from pymc3 import model_to_graphviz
```

```python
# hyperparameters
a, b, c, d = 9, 6, 9, 0.00001
burn_in = 0.1
thinning = 10

y = df['dc'].values
n = df['pop'].values
average_pop = n.mean()

```

```python
#define hierarchical model in pymc3
with pm.Model() as hierarchical_model:
    #priors on alpha
    alpha = pm.Gamma('alpha', alpha=a, beta=b)
    #priors on beta
    beta = pm.Gamma('beta', alpha=c, beta=d)
    #priors on theta
    theta = pm.Gamma('theta', alpha=alpha, beta=beta, shape=len(y))

    #convert rate into number of disease incidents
    mu = theta * 5 * n

    #likelihood
    y_obs = pm.Poisson('y', mu=mu, observed=y)

```

```python
#draw graphical model for the thing we just defined
# model_to_graphviz(hierarchical_model)

```

```python
with hierarchical_model:
    # using default sampler
    trace = pm.sample(1000, tune=1000, target_accept = 0.9)
    print(f'DONE')
```

    <ipython-input-13-995cae4c87d5>:3: FutureWarning: In v4.0, pm.sample will return an `arviz.InferenceData` object instead of a `MultiTrace` by default. You can pass return_inferencedata=True or return_inferencedata=False to be safe and silence this warning.
      trace = pm.sample(1000, tune=1000, target_accept = 0.9)
    Auto-assigning NUTS sampler...
    Initializing NUTS using jitter+adapt_diag...
    Multiprocess sampling (4 chains in 4 jobs)
    NUTS: [theta, beta, alpha]

  <progress value='8000' class='' max='8000' style='width:300px; height:20px; vertical-align: middle;'></progress>
  100.00% [8000/8000 00:34<00:00 Sampling 4 chains, 0 divergences]

    Sampling 4 chains for 1_000 tune and 1_000 draw iterations (4_000 + 4_000 draws total) took 48 seconds.
    The number of effective samples is smaller than 25% for some parameters.

    DONE

```python
N = len(trace['theta'])
theta_trace = trace['theta'][int(burn_in * N)::thinning]
pymc_post_mean = np.mean(theta_trace, axis=0)
```

```python
# make scatter plot
scatter_plot_cancer_rates_orange(df,estimates = pymc_post_mean )
plt.title("pct_mortality vs pop in kcancer dataset")
plt.legend()

```

**Interpretation**:

From this graph, we could observe that mean estimate from the posterior marginal distribution that sampled by hierarchical model using pymc3 now shrink to population mean, with more shrinkage for low population counties.

```python
fig, ax = plt.subplots(1, 2, figsize=(30, 5))
#N = len(trace['theta'].T[county_1])
alpha_trace = trace['alpha'][int(burn_in * N)::thinning]
beta_trace = trace['beta'][int(burn_in * N)::thinning]

ax[0].plot(range(len(alpha_trace)), alpha_trace, color='gray')
ax[0].set_title('Trace Plot for Alphas')
ax[0].set_ylabel('alpha')
ax[1].plot(range(len(beta_trace)), beta_trace, color='gray')
ax[1].set_title('Trace Plot for Betas')
ax[1].set_ylabel('beta')
plt.show()
```

```python
#select the county to visualize the traceplot for theta
#same countries to compare with last homework
county_1 = 0
county_2 = 1

#plot the traceplots for one theta and alpha and beta
fig, ax = plt.subplots(1, 2, figsize=(30, 5))
N = len(trace['theta'].T[county_1])
theta_trace = trace['theta'][int(burn_in * N)::thinning].T[county_1]
ax[0].plot(range(len(theta_trace)), theta_trace, color='gray', alpha=0.5)
ax[0].set_title('trace plot for theta_{}'.format(county_1))

theta_trace = trace['theta'][int(burn_in * N)::thinning].T[county_2]
ax[1].plot(range(len(theta_trace)), theta_trace, color='gray', alpha=0.5)
ax[1].set_title('trace plot for theta_{}'.format(county_2))
```

**Interpretation**:

Comparing to last homework where visual graph show clearly evidence of non-convergence of $\alpha$ and $\beta$ after multiple rounds of Gibbs-MH sampling, this graph shows a considerable level of convergence, and looks better than last time.

I selected two counties that is the same as last time in homework 4 and compare for posterior marginal distribution of $\theta$. It seems they have converged and are fluctuating within a small range too. It seems this sampler perform better than our Gibbs MH sampler implemented. This could be due to a better choice of proposal alpha beta distribution and hyperparameters, and a more complex implementation of sampling method.

2.  **(Hierarchical Bayes vs Empirical Bayes)** Compare the shrinkage of the posterior marginal means of the hierarchical model to the shrinkage of the posterior means from the Bayesian model with empirical Bayes estimates for $\alpha, \beta$. What is the difference in shrinkage between the full hierarchical model and the Bayesian model with empirical Bayes?

```python
scatter_plot_cancer_rates(df, estimates=thetas_mean)
plt.title("pct_mortality vs pop in kcancer dataset (empirical bayes)")
plt.legend()
plt.show()

scatter_plot_cancer_rates_orange(df, estimates = pymc_post_mean)
plt.title("pct_mortality vs pop in kcancer dataset (hierachical bayes)")
plt.legend()
plt.show()
```

**Interpretation**:

Comparing two plots with green shrinkage (empirical bayes) and the orange shrinkage (hierachical bayes) of mean estimates, we could observe a larger shrinkage for the counties with least population in hierachical bayes estimations than empirical bayes estimates.

The hierarchy model pulls together the pct_mortality drastically for counties with small population size, while empirical bayes still shows strips of trends of pct_mortality that are not shrinked to a population mean for counties with small population size. The variations observed in hierachial bayes estimate are that there exists higher variance in the middle parts of population size. However, this doesn't seem to correlated with population size, nor due to the result of overfitting. It could be random variation instead of systematic variation. With hierachical bayes we can safely conclude that the cancer rate is almost the same for all counties regardless of population size, which also makes much more sense in real life.

## Part III: Broader Impact Analysis

Starting in 2020, major machine learning conferences are beginning to ask authors as well as reviewers to explicitly consider the broader impact of new machine learning methods. To properly evaluate the potential good or harm that a piece of technology (AI or not) can do to the general public, we need to be aware that no technology is deployed in ideal conditions or in perfectly neutral contexts. In order to assess the potential broader impact of technology, we need to analyze the social systems/institutions of which these technologies will become a part.

To help you analyze the broader impact of your technology, begin by considering the following questions:

I. Identify the relevant socio-technical systems
  - In what social, political, economic system could the tech be deployed?
  - How would the tech be used in these systems (what role will it take in the decision making processes)?

II. Identify the stakeholders
  - Who are the users?
  - Who are the affected communities (are these the users)?

    ***Hint:*** users are typically decision makers who will use the technology as decision aids (e.g. doctors), whereas affected communities may be folks who are impacted by these decisions but who are not represented in the decision making process (e.g. patients).

III. What types of harm can this tech do?
  - What kinds of failures can this tech have?
  - What kinds of direct harm can these failures cause?
  - What kinds of harm can the socio-technical system cause?

    ***Hint:*** many technical innovations have niche applications, they may sit in a long chain of decision making in a complex system. As such, it may seem, at first glance, that these technologies have no immediate real-life impact. In these cases, it’s helpful to think about the impact of the entire system and then think about how the proposed innovations aid, hamper or change the goals or outcomes of this system.

IV. What types of good can this tech do?
  - What kinds of needs do these users/communities have?
  - What kinds of constraints do these users/communities have?

1. **(Computational Footprint)**  In Homework #4, we considered the broader impact of the this hierarchical model for kidney cancer: we examined under what circumstances a hierachical model for kidney cancer is more preferable to a MLE model or a Bayesian model with hand-picked priors. In this problem, we compare hierarchical Bayes and empirical Bayes.

  In practical terms, what are the real-life advantages and drawbacks of implementing an empirical Bayesian model compared with an hierarchical Bayesian model?

  **Hint:** for example, compare how tractable it is to implement the inference algorithm for each model, compare how easy it is to for the inference method for each model to converge (would you know that your training has converged?). Consider also: how easily could a practitioner diagnose the training process and how could they evaluate the model fit and what kinds of computational resurces would be required to train each model.

**Answers**:

When put this model for kidney cancer and health care resources, we are sampling and estimating $\theta$, which is the underlying kidney cancer rate for that county. The other components of the decision system in which the model will deployed will be citizens living in that county, patients receiving kidney cancer related healthcare or treatments, government, resources department, hospital, doctors etc.

End user will be government, resources department, hospital, doctors etc;
Affected communities will be citizens living in that county, patients receiving kidney cancer related healthcare or treatments.

Whether it is more preferable to use a hiearchical model, an MLE model, a Bayesian model with hand-picked priors, or here, an empirical model depends more on a downstream task.

* Real life advantages:
   * Easy to implement and compute, more tractable to implement the inference algorithm for empirical Bayesian model compared with an hierarchical Bayesian.
   * As we only need to look at evidence moments, it is more easy to understand and interpret to end user and community.
   * We don't need to hand pick prior that incorporate our prior belief if we have no experience in the field. It avoids the risk of a poor choice of prior.
   * It will converge when data size gets larger.

**Drawbacks of implementing an empirical Bayesian model compared with an hierarchical Bayesian model:**
   * It may heavily depend on data quality and data size. Unlike how little poor choice of hierarchical model was, it has less effect on estimatation $\theta$. However, the data quality in empirical model heavily decides our estimation.
   * It may not converge, or the mean estimate we observed may not shrink to a range. There's still overfitting in low population or small size data points. Those data points are unable to borrow statistical power from data points with more populations like in hierarchial models.
   * If not converged or shrink to a population mean, this will leads to difficulty for government and agency to figure out where to fund, which county has higher kidney cancer rate.
   * The needs for evidence and samples will leads to questions about where to put clinical trials, how many trials shall we obtain in terms of data points; Healthcare data points is unlike other data points to collect, it needs a lot of resource and invests to obtain.
   * If we overestimate the cancer rate, it's not serving anything or communities; it's not serving people who needed; The actual death rate is not that high; If we underestimate the cancer rate, we might under devote health care resources to needed people and county;
   * We might also missing out envrionment factors that affect the rate.
   * The algorithm may not always be tractable. If the population is non-conjugate prior, then it would be hard to sample from population to get needed data points for empirical bayesian model.
   * It's hard to diagnose issue within the data after we obtain the estimation.

2. **(Mitigation of Potential Negative Impacts)** Based on your answers for the previous question and your broader impact analysis from HW#4, what information would the model designer/engineer need to disclose to the end-users to mitigate potentially negative impacts?

  **Hint:** consider how your end-user would be able to validate the model (how would they know if the model is working or has ceased to work once it has been deployed), how should the end-user choose between implementing a hiearchical Bayesian model or an empirical Bayesian model, how could the end-user identify and mitigate negative impacts (if they occur).

**Answers**:

The potential downside and negative impacts, as mentioned briefly above, could be that
- If we overestimate the cancer rate, it's not serving anything or communities; it's not serving people who needed; The actual death rate is not that high; If we underestimate the cancer rate, we might under devote health care resources to needed people and county;
- If there exists systematic inequality, the model are helping to exaggerated it. For example, in rural counties they might already have low income, bad infrastructure, and a bad sampling model will help to systematically exaggerating the circumstance.

To mitigate negative impacts, end users should be able to validate the models by convergence check (traceplots etc), cross validation with human experts and experienced professionals. They should also treat end result with care and involve human decison making instead of fully taking computed results.

For patients and other affected communities, end users should give full disclosure of how model is being used and deployed, giving advice per treatment. Besides, other environmental factors should also be reviewed before making decision.

The next thing we would like to consider is that if any of these mistakes or negative impacts happening at random?

- Data inbalance in healthcare is a huge problem (For example, gender data inequality, in many cases there's seldom woman data points, which is regarded as a commonly underpresented group.) The model may exaggerate this problem, and also overfit to one sub-population while be harmful to predictions in other subpopulation.
- Model for these data are qualitatively different. Studies in man doesn't generalize to women.
- For subgroups that are previously undersample, we will likely to remove them, as they look like outliers (Small data inside big data problem).
- Thus, maybe consider using a mixture of several models to compare and cross check for estimation, and also allow small data size points to borrow statistical power from large data size points like what hierarchial model does would be a preferable choice.


```python
%matplotlib inline

import matplotlib
import numpy as np
import matplotlib.pyplot as plt

plt.rcParams["figure.figsize"] = (12, 9)
plt.rcParams["font.size"] = 18
```

# Radioactivity

Learning Objectives:

- Explain how radioactivity was discovered
- Explain the nuclear physical reason for radioactive decay
- List the sources of natural and man made radiation
- Read and understand a decay diagram
- Calculate Q values for various types of decay
- Describe the physics of various types of decay
- State the radioactive decay law
- Derive the radioactive decay law
- Understand how incorporating sources impacts decay calcuations
- Calculate decay with production for simple cases

## Discovery of Radioactivity
- Radioactivity was first discovered in 1896 by  Henri Becquerel, while working on phosphorescent materials.
- These materials glow in the dark after exposure to light, and he thought that the glow produced in cathode ray tubes by X-rays might be connected with phosphorescence.
- He wrapped a photographic plate in black paper and placed various phosphorescent minerals on it.
- All results were negative until he used uranium salts.

Becquerel in the lab

Photographic plate made by Henri Becquerel showing effects of exposure to radioactivity.

**What is nuclear decay?**
A spontaneous process where the protons and neutrons in a given nucleus are rearranged into a lower energy state.
The transition may involve levels of the same nucleus (gamma emission, internal conversion) or levels of a different nucleus (alpha, beta decay).

**Why do nuclei decay?** A nucleus decays when it is **unstable**.  It undergoes decay in order to become more stable (lower energy state).

**Where do the unstable nuclei come from?**

## Sources of Radiation

> A chart of the public's exposure to ionizing radiation (displayed below)
> shows that people generally receive a total annual dose of about 620 millirem.
> Of this total, natural sources of radiation account for about 50 percent,
> while man-made sources account for the remaining 50 percent. -- US NRC

## Natural Sources

### Terrestrial

Uranium, thorium, radium, etc. are all present naturally inside rocks and soil and are part of the four major decay chains. Others are 'primordial.' Primoidal radioactive isotopes include about 20 isotopes that are long lived and not part of any decay chain (K-40, Rubidium-87) Potassium has a quite radioactive isotope, $^{40}K$. (bananas)

Evolution of the earth's radiogenic heat (in the mantle).

### Internal

Mostly $^{40}K$ and $^{14}C$ inside your body.

### Cosmic
Commonly, cosmic radiation includes $^{14}C$, tritium ($^3H$), and others.

CC BY-SA 3.0, Link

Air showers ensuing from very-high-energy cosmic rays can enter Earth’s atmosphere from multiple directions. Credit: Simon Swordy/NASA.

## Man-Made Radiation

Of approximately 3200 known nuclides:
- **266** are stable,
- **65** long-lived radioactive isotopes are found in nature,
- and the remaining **~2900** have been made by humans.

One of the heaviest named elements is Livermorium (Z=116), its most stable isotope $^{293}Lv$ has half-life $t_{1/2} = 60 ms$.

### Types of Decay

```python
# The below IFrame displays Page 121 of your textbook:
# Shultis, J. K. (2016). Fundamentals of Nuclear Science and Engineering Third Edition,
# 3rd Edition. [Vitalsource]. Retrieved from https://bookshelf.vitalsource.com/#/books/9781498769303/

from IPython.display import IFrame
IFrame("https://bookshelf.vitalsource.com/books/9781498769303/pageid/121", width=1000, height=1000)

```

All elements smaller than Z=83 (Bismuth) have at least one stable isotope.
Exceptions: Technetium (Z=43) and Promethium (Z=61).

Once nucleus gets past a certain size, it is unstable
The largest stable nucleus is Pb-208, all isotopes larger than this are unstable

Nuclear conservation laws apply. The following are conserved:
- charge
- number of nucleons (proton + neutron)
- total energy (mass + energy)
- linear momentum (in inertial frame of reference)
- angular momentum (spin)
  - alternatively: leptons (electrons + neutrinos)

```python
# The below IFrame displays Page 100 of your textbook:
# Shultis, J. K. (2016). Fundamentals of Nuclear Science and Engineering Third Edition,
# 3rd Edition. [Vitalsource]. Retrieved from https://bookshelf.vitalsource.com/#/books/9781498769303/
from IPython.display import IFrame
IFrame("https://bookshelf.vitalsource.com/books/9781498769303/pageid/122", width=1000, height=1000)
```

## Energetics of Decay

### Alpha ($\alpha$) Decay

An $\alpha$ particle is emitted.
The daughter is left with 2 fewer neutrons and 2 fewer protons than the parent.

\begin{align}
^{A}_{Z}P \longrightarrow ^{A-4}_{Z-2}D^{2-} + ^4_2\alpha
\end{align}

\begin{align}
\frac{Q}{c^2} = M\left(^{A}_{Z}P\right)-\left[ M\left(^{A-4}_{Z-2}D\right) + M\left(^{4}_{2}He\right)\right]
\end{align}

### Gamma ($\gamma$) Decay

An excited nucleus decays to its ground state by the emission of a gamma photon.

\begin{align}
^{A}_{Z}P^* \longrightarrow ^{A}_{Z}P + \gamma
\end{align}

\begin{align}
Q = E^*
\end{align}

### Negatron ($\beta -$) Decay
A neutron changes into a proton in the nucleus (nuclear weak force). An electron ($\beta -$) and an antineutrino ($\bar{\nu}$) are emitted.

\begin{align}
^{A}_{Z}P \longrightarrow ^{A}_{Z+1}D^{+} + ^0_{-1}e + \bar{\nu}
\end{align}

\begin{align}
\frac{Q}{c^2} = M\left(^{A}_{Z}P\right)- M\left(^{A}_{Z+1}D\right)
\end{align}

### Positron ($\beta +$) Decay

A proton changes into a neutron in the nucleus (nuclear weak force). A positron ($\beta +$) and a neutrino ($\nu$) are emitted.

\begin{align}
^{A}_{Z}P \longrightarrow ^{A}_{Z-1}D^{-} + ^0_{+1}e + \nu
\end{align}

\begin{align}
\frac{Q}{c^2} = M\left(^{A}_{Z}P\right)- \left[ M\left(^{A}_{Z-1}D\right) + 2m_e\right]
\end{align}

### Electron Capture
1. An orbital electron is absorbed by the nucleus,
2. converts a nuclear proton into a neutron and a neutrino ($\nu$),
3. and typically leaves the nucleus in an excited state.

\begin{align}
^{A}_{Z}P + \left(^0_{-1}e\right) \longrightarrow ^{A}_{Z-1}D + \nu
\end{align}

\begin{align}
\frac{Q}{c^2} = M\left(^{A}_{Z}P\right)-  M\left(^{A}_{Z-1}D\right)
\end{align}

### Proton Emission
A proton is ejected from the nucleus.

\begin{align}
^{A}_{Z}P \longrightarrow ^{A-1}_{Z-1}D^{-} + ^1_1p
\end{align}

\begin{align}
\frac{Q}{c^2} = M\left(^{A}_{Z}P\right)-\left[ M\left(^{A-1}_{Z-1}D\right) + M\left(^{1}_{1}H\right)\right]
\end{align}

The decay of a proton rich nucleus A populates excited states of a daughter nucleus B by β+ emission or electron capture (EC). Those excited states that lie below the separation energy for protons (Sp) decay by γ emission towards the groundstate of daughter B. For the higher excited states a competitive decay channel of proton emission to the granddaughter C exists, called β-delayed proton emission.

### Neutron Emission
A neutron is ejected from the nucleus.

\begin{align}
^{A}_{Z}P \longrightarrow ^{A-1}_{Z}P + ^1_0n
\end{align}

\begin{align}
\frac{Q}{c^2} = M\left(^{A}_{Z}P\right)-\left[ M\left(^{A-1}_{Z}P\right) + m_n\right]
\end{align}

### Internal Conversion

The excitation energy of a nucleus is used to eject an orbital electron (typically a K-shell) electron.

\begin{align}
^{A}_{Z}P^* \longrightarrow ^{A-1}_{Z}P^+ + ^0_{-1}e
\end{align}

\begin{align}
Q = E* - BE^K_e
\end{align}

### An Aside on The Nuclear Weak Force

Beta ($\beta \pm$) decay is a consequence of the **weak force**, which is characterized by long decay times (thousands of years). Nucleons are composed of up quarks and down quarks, and **the weak force allows a quark to change type** by the exchange of a W boson and the creation of an electron/antineutrino or positron/neutrino pair. For example, **a neutron, composed of two down quarks and an up quark, decays to a proton composed of a down quark and two up quarks.**

## Radioactive Decay Law

**The Law:** The probability that an unstable parent nucleus will decay spontaneously into one or more particles of lower mass/energy is independent of the past history of the nucleus and is the same for all radionuclides of the same type.

### Decay Constant
Radioactive decay takes place stochastically in a single atom, and quite predictably in a large group of radioactive atoms of the same type.

The probability that any one of the radionuclides in the sample decays in $\Delta t$ is $\frac{\Delta N}{N}$. This **decay probability** $\frac{\Delta N}{N}$ per unit time for a time interval $\Delta t$ should vary smoothly for large $\Delta t$.

The statistically averaged **decay probability per unit time**, in the limit of infinitely small $\Delta t$, approaches a constant $\lambda$.
Thus:

\begin{align}
       \lambda &= \mbox{decay constant}\\
        &\equiv \lim_{\Delta t \to 0} \frac{\left(\Delta N/N\right)}{\Delta t}\\
N(t) &= \mbox{expected number of nuclides in a sample at time t}\\
\implies \mbox{ } -dN &= \mbox{decrease in number of radionuclides}\\
                      &= \lambda N(t) dt\\
\implies \mbox{ } \frac{dN(t)}{dt} &=-\lambda N(t)\\
\end{align}

The above is a differential equation. The solution to this differential equation give a definition of N as a function of t. We can now describe the change in radioactive nuclides over time as:

\begin{align}
        \frac{dN}{dt} &= -\lambda N \\
       \Rightarrow N_i(t) &= N_i(0)e^{-\lambda t}\\
\end{align}

where

\begin{align}
        N_i(t) &= \mbox{number of isotopes i adjusted for decay}\\
        N_i(0)&= \mbox{initial condition}\\
        \end{align}

```python
import math
def n_decay(t, n_initial=100, lam=1):
    """This function describes the decay of an isotope"""
    return n_initial*math.exp(-lam*t)

# This code plots the decay of an isotope
import numpy as np
y = np.arange(6.0)
x = np.arange(6.0)
for t in range(0,6):
    x[t] = t
    y[t] = n_decay(t)

# creates a figure and axes with matplotlib
fig, ax = plt.subplots()
scatter = plt.scatter(x, y, color='blue', s=y*20, alpha=0.4)
ax.plot(x, y, color='red')

# adds labels to the plot
ax.set_ylabel('N_i(t)')
ax.set_xlabel('Time')
ax.set_title('N_i')

# adds tooltips
import mpld3
labels = ['{0}% remaining'.format(i) for i in y]
tooltip = mpld3.plugins.PointLabelTooltip(scatter, labels=labels)
mpld3.plugins.connect(fig, tooltip)

mpld3.display()
```

## Half Life

All dynamic processes which decay (or grow) exponentially can be characterized by their half life (or doubling time). In the case of radioactive decay

\begin{align}
        \tau_{1/2}&= \mbox{half-life}\\
               &=\frac{ln(2)}{\lambda}  \\
        t &= \mbox{time elapsed [s]}\\
        \tau_{1/2} &= \mbox{half-life [s]} \\
\end{align}

```python
# This code converts decay constant to half life

def half_life(lam):
    return math.log(2)/lam
lam = 1.0

# This code plots the decay of an isotope for various half lives
import numpy as np
y = np.arange(8.0)
x = np.arange(8.0)

lives = np.arange(half_life(lam), 9*half_life(lam), half_life(lam))
for i, t in enumerate(lives):
    x[i] = float(t)
    y[i] = n_decay(t)

# creates a figure and axes with matplotlib
fig, ax = plt.subplots()
scatter = plt.scatter(x, y, color='blue', s=y*20, alpha=0.4)
ax.plot(x, y, color='red')

# adds labels to the plot
ax.set_ylabel('N_i(t)')
ax.set_xlabel('Time')
ax.set_title('N_i')

# adds tooltips
import mpld3
labels = ['{0}% remaining'.format(i) for i in y]
tooltip = mpld3.plugins.PointLabelTooltip(scatter, labels=labels)
mpld3.plugins.connect(fig, tooltip)

mpld3.display()
```

After n half lives, only $\frac{1}{2^n}$ of the original sample remain.
\begin{align}
N(n\tau_{1/2})= \frac{1}{2^n}N_0
\end{align}

In a sample, $N_0$ has has been reduced to a fraction of $N_0$, $\epsilon$. How many half lives have passed?

\begin{align}
n = \frac{-ln\epsilon}{ln2}
\end{align}

Finally, the radioactive decay law can be expressed using the half-life:

\begin{align}
N(t) = N_0\left(\frac{1}{2}\right)^{t/\tau_{1/2}}
\end{align}

## Decay by competing processes

\begin{align}
\frac{dN(t)}{dt} &= -\lambda_1N(t) - \lambda_2N(t) - \cdots \lambda_nN(t)\\
 &= -\sum_{i=1}^n \lambda_iN(t)\\
 &\equiv-\lambda N(t)
\end{align}

A nuclide will decay by the $i^{th}$ mode with probability $f_i$.

\begin{align}
f_i &= \frac{\mbox{decay rate by ith mode}}{\mbox{decay rate by all modes}}\\
    &=\frac{\lambda_i}{\lambda}
\end{align}

## Decay with Production

In reactors, isotopes decay into one another and still others are born from fission. Thus, if there is production, we can rewrite the standard decay differential equation as:

\begin{align}
\frac{dN(t)}{dt} &= -\mbox{rate of decay} + \mbox{rate of production}\\
\implies N(t) &= N_0 e^{-\lambda t} + \int_0^t dt'Q(t')e^{-\lambda (t-t')}\\
\end{align}

If the production rate is constant, this simplifies:

\begin{align}
N(t) &= N_0 e^{-\lambda t} + \frac{Q_0}{\lambda}\left[1-e^{-\lambda t}\right]\\
\end{align}

### Fuel Depletion

Decays, fissions, and absorptions compete throughout the life of the reactor.

#### Aside: Reaction Rates

In a reactor, this Q can be characterized via reaction rates.

- The microscopic cross section $\sigma_{i,j}$ is just the likelihood of the event per unit area.
- The macroscopic cross section $\Sigma_{i,j}$is just the likelihood of the event per unit area of a certain density of target isotopes.
- The reaction rate is the macroscopic cross section times the flux of incident neutrons.

\begin{align}
R_{i,j}(\vec{r}) &= N_j(\vec{r})\int dE \phi(\vec{r},E)\sigma_{i,j}(E)\\
R_{i,j}(\vec{r}) &= \mbox{reactions of type i involving isotope j } [reactions/cm^2s]\\
N_j(\vec{r}) &= \mbox{number of nuclei participating in the reactions }\\
E &= \mbox{energy}\\
\phi(\vec{r},E)&= \mbox{flux of neutrons with energy E at position i}\\
\sigma_{i,j}(E)&= \mbox{cross section}\\
\end{align}

We said this can be written more simply as $R_x = \sigma_x I N$, where I is intensity of the neutron flux. In the notation of the above equation, we can describe the production of an isotope by neutron absorption by another isotope as :

\begin{align}
\mbox{isotope i production via neutron absorption in m} = f_{im}\sigma_{am}N_m  \phi
\end{align}

### Total composition evolution

\begin{align}
\frac{dN_i}{dt} &= \sum_{m=1}^{M}l_{im}\lambda_mN_m + \phi\sum_{m=1}^{M}f_{im}\sigma_mN_m - (\lambda_i + \phi\sigma_i + r_i - c_i)N_i + F_i\Big|_{i\in [1,M]}\\
\end{align}
\begin{align}
N_i &= \mbox{atom density of nuclide i}\\
M &= \mbox{number of nuclides}\\
l_{im} &= \mbox{fraction of decays of nuclide m that result in formation of nuclide i}\\
\lambda_i &= \mbox{radioactive decay constant of nuclide i}\\
\phi &= \mbox{neutron flux, averaged over position and energy}\\
f_{im} &= \mbox{fraction of neutron absorption by nuclide m leading to the formation of nuclide i}\\
\sigma_m &= \mbox{average neutron absorption cross section of nuclide m}\\
r_i &= \mbox{continuous removal rate of nuclide i from the system}\\
c_i &= \mbox{continuous feed rate of nuclide i}\\
F_i &= \mbox{production rate of nuclide i directly from fission}\\
\end{align}

### Example: $^{135}Xe$

**Discussion: What is interesting about Xenon?**

$^{135}Xe$ is produced directly by fission and from the decay of iodine.

\begin{align}
\frac{dN_{xe}}{dt} &= \sum_{m=1}^{M}l_{Xem}\lambda_mN_m + \phi\sum_{m=1}^{M}f_{Xem}\sigma_mN_m - (\lambda_{Xe} + \phi\sigma_{Xe} + r_{Xe} - c_{Xe})N_{Xe} + F_{Xe}\\
 &= -\lambda_{Xe}N_{Xe}  - \sigma_{aXe}\phi N_{Xe} + \lambda_IN_I + F_{Xe}\\
 &= -\lambda_{Xe}N_{Xe}  - \sigma_{aXe}\phi N_{Xe} + \lambda_IN_I + \gamma_{Xe}\Sigma_f\phi\\
 \gamma_{Xe} &= 0.003\\
 \gamma_{I} &= 0.061\\
\end{align}

### Example: $^{239}Pu$

\begin{align}
\frac{dN_{Pu}}{dt} &= \sum_{m=1}^{M}l_{Pum}\lambda_mN_m + \phi\sum_{m=1}^{M}f_{Pum}\sigma_mN_m - (\lambda_{Pu} + \phi\sigma_{Pu} + r_{Pu} - c_{Pu})N_{Pu} + F_{Pu}\\
\end{align}

Let's formulate this equation together.

$$\mathrm{^{238}_{\ 92}U \ + \ ^{1}_{0}n \ \longrightarrow \ ^{239}_{\ 92}U \ \xrightarrow [23.5\ min]{\beta^-} \ ^{239}_{\ 93}Np \ \xrightarrow [2.3565\ d]{\beta^-} \ ^{239}_{\ 94}Pu}$$

- Decay of what nuclides result in the formation of $^{239}Pu$?
- Does $^{239}Pu$ decay?
- Is there a nuclide that becomes  $^{239}Pu$ after it absorbs a neutron?
- Does  $^{239}Pu$ ever absorb neutrons?
- Is  $^{239}Pu$ ever produced directly from fission?

## Burnable Poisons

- Gadolinia ($Gd_2O_3$) or erbia ($Er_2O_3$) common
- Natural Erbium consists of Er166, Er167, Er168  and Er170 primarily.  Er167 has large thermal  cross section.
- Gd is an early life burnable poison, typically gone by 10‐20 GWd
- Boron also used widely.
- Can be mixed with the fuel or a coating on the pellet.

\begin{align}
\frac{dN^P(t)}{dt} &= -g(t)\sigma_{aP}N^P(t)\phi\\
g(t) &= \frac{\mbox{average flux inside BP}}{\mbox{average flux in core}}\\
\sigma_{aP} &=\mbox{neutron absorption cross section of the BP}\\
N^P(t) &= \mbox{number of atoms of the BP at time t}
\end{align}

```python

```


```
# This mounts your Google Drive to the Colab VM.
from google.colab import drive
drive.mount('/content/drive')

# TODO: Enter the foldername in your Drive where you have saved the unzipped
# assignment folder, e.g. 'cs231n/assignments/assignment1/'
FOLDERNAME = None
assert FOLDERNAME is not None, "[!] Enter the foldername."

# Now that we've mounted your Drive, this ensures that
# the Python interpreter of the Colab VM can load
# python files from within it.
import sys
sys.path.append('/content/drive/My Drive/{}'.format(FOLDERNAME))

# This downloads the CIFAR-10 dataset to your Drive
# if it doesn't already exist.
%cd /content/drive/My\ Drive/$FOLDERNAME/cs231n/datasets/
!bash get_datasets.sh
%cd /content/drive/My\ Drive/$FOLDERNAME
```

# Batch Normalization
One way to make deep networks easier to train is to use more sophisticated optimization procedures such as SGD+momentum, RMSProp, or Adam. Another strategy is to change the architecture of the network to make it easier to train. One idea along these lines is batch normalization, proposed by [1] in 2015.

To understand the goal of batch normalization, it is important to first recognize that machine learning methods tend to perform better with input data consisting of uncorrelated features with zero mean and unit variance. When training a neural network, we can preprocess the data before feeding it to the network to explicitly decorrelate its features. This will ensure that the first layer of the network sees data that follows a nice distribution. However, even if we preprocess the input data, the activations at deeper layers of the network will likely no longer be decorrelated and will no longer have zero mean or unit variance, since they are output from earlier layers in the network. Even worse, during the training process the distribution of features at each layer of the network will shift as the weights of each layer are updated.

The authors of [1] hypothesize that the shifting distribution of features inside deep neural networks may make training deep networks more difficult. To overcome this problem, they propose to insert into the network layers that normalize batches. At training time, such a layer uses a minibatch of data to estimate the mean and standard deviation of each feature. These estimated means and standard deviations are then used to center and normalize the features of the minibatch. A running average of these means and standard deviations is kept during training, and at test time these running averages are used to center and normalize features.

It is possible that this normalization strategy could reduce the representational power of the network, since it may sometimes be optimal for certain layers to have features that are not zero-mean or unit variance. To this end, the batch normalization layer includes learnable shift and scale parameters for each feature dimension.

[1] [Sergey Ioffe and Christian Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing
Internal Covariate Shift", ICML 2015.](https://arxiv.org/abs/1502.03167)

```
# Setup cell.
import time
import numpy as np
import matplotlib.pyplot as plt
from cs231n.classifiers.fc_net import *
from cs231n.data_utils import get_CIFAR10_data
from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array
from cs231n.solver import Solver

%matplotlib inline
plt.rcParams["figure.figsize"] = (10.0, 8.0)  # Set default size of plots.
plt.rcParams["image.interpolation"] = "nearest"
plt.rcParams["image.cmap"] = "gray"

%load_ext autoreload
%autoreload 2

def rel_error(x, y):
    """Returns relative error."""
    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))

def print_mean_std(x,axis=0):
    print(f"  means: {x.mean(axis=axis)}")
    print(f"  stds:  {x.std(axis=axis)}\n")
```

```
# Load the (preprocessed) CIFAR-10 data.
data = get_CIFAR10_data()
for k, v in list(data.items()):
    print(f"{k}: {v.shape}")
```

# Batch Normalization: Forward Pass
In the file `cs231n/layers.py`, implement the batch normalization forward pass in the function `batchnorm_forward`. Once you have done so, run the following to test your implementation.

Referencing the paper linked to above in [1] may be helpful!

```
# Check the training-time forward pass by checking means and variances
# of features both before and after batch normalization

# Simulate the forward pass for a two-layer network.
np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before batch normalization:')
print_mean_std(a,axis=0)

gamma = np.ones((D3,))
beta = np.zeros((D3,))

# Means should be close to zero and stds close to one.
print('After batch normalization (gamma=1, beta=0)')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)

gamma = np.asarray([1.0, 2.0, 3.0])
beta = np.asarray([11.0, 12.0, 13.0])

# Now means should be close to beta and stds close to gamma.
print('After batch normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)
```

```
# Check the test-time forward pass by running the training-time
# forward pass many times to warm up the running averages, and then
# checking the means and variances of activations after a test-time
# forward pass.

np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)

bn_param = {'mode': 'train'}
gamma = np.ones(D3)
beta = np.zeros(D3)

for t in range(50):
  X = np.random.randn(N, D1)
  a = np.maximum(0, X.dot(W1)).dot(W2)
  batchnorm_forward(a, gamma, beta, bn_param)

bn_param['mode'] = 'test'
X = np.random.randn(N, D1)
a = np.maximum(0, X.dot(W1)).dot(W2)
a_norm, _ = batchnorm_forward(a, gamma, beta, bn_param)

# Means should be close to zero and stds close to one, but will be
# noisier than training-time forward passes.
print('After batch normalization (test-time):')
print_mean_std(a_norm,axis=0)
```

# Batch Normalization: Backward Pass
Now implement the backward pass for batch normalization in the function `batchnorm_backward`.

To derive the backward pass you should write out the computation graph for batch normalization and backprop through each of the intermediate nodes. Some intermediates may have multiple outgoing branches; make sure to sum gradients across these branches in the backward pass.

Once you have finished, run the following to numerically check your backward pass.

```
# Gradient check batchnorm backward pass.
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
fx = lambda x: batchnorm_forward(x, gamma, beta, bn_param)[0]
fg = lambda a: batchnorm_forward(x, a, beta, bn_param)[0]
fb = lambda b: batchnorm_forward(x, gamma, b, bn_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = batchnorm_forward(x, gamma, beta, bn_param)
dx, dgamma, dbeta = batchnorm_backward(dout, cache)

# You should expect to see relative errors between 1e-13 and 1e-8.
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

# Batch Normalization: Alternative Backward Pass
In class we talked about two different implementations for the sigmoid backward pass. One strategy is to write out a computation graph composed of simple operations and backprop through all intermediate values. Another strategy is to work out the derivatives on paper. For example, you can derive a very simple formula for the sigmoid function's backward pass by simplifying gradients on paper.

Surprisingly, it turns out that you can do a similar simplification for the batch normalization backward pass too!

In the forward pass, given a set of inputs $X=\begin{bmatrix}x_1\\x_2\\...\\x_N\end{bmatrix}$,

we first calculate the mean $\mu$ and variance $v$.
With $\mu$ and $v$ calculated, we can calculate the standard deviation $\sigma$  and normalized data $Y$.
The equations and graph illustration below describe the computation ($y_i$ is the i-th element of the vector $Y$).

\begin{align}
& \mu=\frac{1}{N}\sum_{k=1}^N x_k  &  v=\frac{1}{N}\sum_{k=1}^N (x_k-\mu)^2 \\
& \sigma=\sqrt{v+\epsilon}         &  y_i=\frac{x_i-\mu}{\sigma}
\end{align}

The meat of our problem during backpropagation is to compute $\frac{\partial L}{\partial X}$, given the upstream gradient we receive, $\frac{\partial L}{\partial Y}.$ To do this, recall the chain rule in calculus gives us $\frac{\partial L}{\partial X} = \frac{\partial L}{\partial Y} \cdot \frac{\partial Y}{\partial X}$.

The unknown/hard part is $\frac{\partial Y}{\partial X}$. We can find this by first deriving step-by-step our local gradients at
$\frac{\partial v}{\partial X}$, $\frac{\partial \mu}{\partial X}$,
$\frac{\partial \sigma}{\partial v}$,
$\frac{\partial Y}{\partial \sigma}$, and $\frac{\partial Y}{\partial \mu}$,
and then use the chain rule to compose these gradients (which appear in the form of vectors!) appropriately to compute $\frac{\partial Y}{\partial X}$.

If it's challenging to directly reason about the gradients over $X$ and $Y$ which require matrix multiplication, try reasoning about the gradients in terms of individual elements $x_i$ and $y_i$ first: in that case, you will need to come up with the derivations for $\frac{\partial L}{\partial x_i}$, by relying on the Chain Rule to first calculate the intermediate $\frac{\partial \mu}{\partial x_i}, \frac{\partial v}{\partial x_i}, \frac{\partial \sigma}{\partial x_i},$ then assemble these pieces to calculate $\frac{\partial y_i}{\partial x_i}$.

You should make sure each of the intermediary gradient derivations are all as simplified as possible, for ease of implementation.

After doing so, implement the simplified batch normalization backward pass in the function `batchnorm_backward_alt` and compare the two implementations by running the following. Your two implementations should compute nearly identical results, but the alternative implementation should be a bit faster.

```
np.random.seed(231)
N, D = 100, 500
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
out, cache = batchnorm_forward(x, gamma, beta, bn_param)

t1 = time.time()
dx1, dgamma1, dbeta1 = batchnorm_backward(dout, cache)
t2 = time.time()
dx2, dgamma2, dbeta2 = batchnorm_backward_alt(dout, cache)
t3 = time.time()

print('dx difference: ', rel_error(dx1, dx2))
print('dgamma difference: ', rel_error(dgamma1, dgamma2))
print('dbeta difference: ', rel_error(dbeta1, dbeta2))
print('speedup: %.2fx' % ((t2 - t1) / (t3 - t2)))
```

# Fully Connected Networks with Batch Normalization
Now that you have a working implementation for batch normalization, go back to your `FullyConnectedNet` in the file `cs231n/classifiers/fc_net.py`. Modify your implementation to add batch normalization.

Concretely, when the `normalization` flag is set to `"batchnorm"` in the constructor, you should insert a batch normalization layer before each ReLU nonlinearity. The outputs from the last layer of the network should not be normalized. Once you are done, run the following to gradient-check your implementation.

**Hint:** You might find it useful to define an additional helper layer similar to those in the file `cs231n/layer_utils.py`.

```
np.random.seed(231)
N, D, H1, H2, C = 2, 15, 20, 30, 10
X = np.random.randn(N, D)
y = np.random.randint(C, size=(N,))

# You should expect losses between 1e-4~1e-10 for W,
# losses between 1e-08~1e-10 for b,
# and losses between 1e-08~1e-09 for beta and gammas.
for reg in [0, 3.14]:
  print('Running check with reg = ', reg)
  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,
                            reg=reg, weight_scale=5e-2, dtype=np.float64,
                            normalization='batchnorm')

  loss, grads = model.loss(X, y)
  print('Initial loss: ', loss)

  for name in sorted(grads):
    f = lambda _: model.loss(X, y)[0]
    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)
    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))
  if reg == 0: print()
```

# Batch Normalization for Deep Networks
Run the following to train a six-layer network on a subset of 1000 training examples both with and without batch normalization.

```
np.random.seed(231)

# Try training a very deep net with batchnorm.
hidden_dims = [100, 100, 100, 100, 100]

num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

weight_scale = 2e-2
bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

print('Solver with batch norm:')
bn_solver = Solver(bn_model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True,print_every=20)
bn_solver.train()

print('\nSolver without batch norm:')
solver = Solver(model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True, print_every=20)
solver.train()
```

Run the following to visualize the results from two networks trained above. You should find that using batch normalization helps the network to converge much faster.

```
def plot_training_history(title, label, baseline, bn_solvers, plot_fn, bl_marker='.', bn_marker='.', labels=None):
    """utility function for plotting training history"""
    plt.title(title)
    plt.xlabel(label)
    bn_plots = [plot_fn(bn_solver) for bn_solver in bn_solvers]
    bl_plot = plot_fn(baseline)
    num_bn = len(bn_plots)
    for i in range(num_bn):
        label='with_norm'
        if labels is not None:
            label += str(labels[i])
        plt.plot(bn_plots[i], bn_marker, label=label)
    label='baseline'
    if labels is not None:
        label += str(labels[0])
    plt.plot(bl_plot, bl_marker, label=label)
    plt.legend(loc='lower center', ncol=num_bn+1)

plt.subplot(3, 1, 1)
plot_training_history('Training loss','Iteration', solver, [bn_solver], \
                      lambda x: x.loss_history, bl_marker='o', bn_marker='o')
plt.subplot(3, 1, 2)
plot_training_history('Training accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.train_acc_history, bl_marker='-o', bn_marker='-o')
plt.subplot(3, 1, 3)
plot_training_history('Validation accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.val_acc_history, bl_marker='-o', bn_marker='-o')

plt.gcf().set_size_inches(15, 15)
plt.show()
```

# Batch Normalization and Initialization
We will now run a small experiment to study the interaction of batch normalization and weight initialization.

The first cell will train eight-layer networks both with and without batch normalization using different scales for weight initialization. The second layer will plot training accuracy, validation set accuracy, and training loss as a function of the weight initialization scale.

```
np.random.seed(231)

# Try training a very deep net with batchnorm.
hidden_dims = [50, 50, 50, 50, 50, 50, 50]
num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

bn_solvers_ws = {}
solvers_ws = {}
weight_scales = np.logspace(-4, 0, num=20)
for i, weight_scale in enumerate(weight_scales):
    print('Running weight scale %d / %d' % (i + 1, len(weight_scales)))
    bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

    bn_solver = Solver(bn_model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
    bn_solver.train()
    bn_solvers_ws[weight_scale] = bn_solver

    solver = Solver(model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
    solver.train()
    solvers_ws[weight_scale] = solver
```

```
# Plot results of weight scale experiment.
best_train_accs, bn_best_train_accs = [], []
best_val_accs, bn_best_val_accs = [], []
final_train_loss, bn_final_train_loss = [], []

for ws in weight_scales:
  best_train_accs.append(max(solvers_ws[ws].train_acc_history))
  bn_best_train_accs.append(max(bn_solvers_ws[ws].train_acc_history))

  best_val_accs.append(max(solvers_ws[ws].val_acc_history))
  bn_best_val_accs.append(max(bn_solvers_ws[ws].val_acc_history))

  final_train_loss.append(np.mean(solvers_ws[ws].loss_history[-100:]))
  bn_final_train_loss.append(np.mean(bn_solvers_ws[ws].loss_history[-100:]))

plt.subplot(3, 1, 1)
plt.title('Best val accuracy vs. weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best val accuracy')
plt.semilogx(weight_scales, best_val_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_val_accs, '-o', label='batchnorm')
plt.legend(ncol=2, loc='lower right')

plt.subplot(3, 1, 2)
plt.title('Best train accuracy vs. weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best training accuracy')
plt.semilogx(weight_scales, best_train_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_train_accs, '-o', label='batchnorm')
plt.legend()

plt.subplot(3, 1, 3)
plt.title('Final training loss vs. weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Final training loss')
plt.semilogx(weight_scales, final_train_loss, '-o', label='baseline')
plt.semilogx(weight_scales, bn_final_train_loss, '-o', label='batchnorm')
plt.legend()
plt.gca().set_ylim(1.0, 3.5)

plt.gcf().set_size_inches(15, 15)
plt.show()
```

## Inline Question 1:
Describe the results of this experiment. How does the weight initialization scale affect models with/without batch normalization differently, and why?

## Answer:
[FILL THIS IN]

# Batch Normalization and Batch Size
We will now run a small experiment to study the interaction of batch normalization and batch size.

The first cell will train 6-layer networks both with and without batch normalization using different batch sizes. The second layer will plot training accuracy and validation set accuracy over time.

```
def run_batchsize_experiments(normalization_mode):
    np.random.seed(231)

    # Try training a very deep net with batchnorm.
    hidden_dims = [100, 100, 100, 100, 100]
    num_train = 1000
    small_data = {
      'X_train': data['X_train'][:num_train],
      'y_train': data['y_train'][:num_train],
      'X_val': data['X_val'],
      'y_val': data['y_val'],
    }
    n_epochs=10
    weight_scale = 2e-2
    batch_sizes = [5,10,50]
    lr = 10**(-3.5)
    solver_bsize = batch_sizes[0]

    print('No normalization: batch size = ',solver_bsize)
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)
    solver = Solver(model, small_data,
                    num_epochs=n_epochs, batch_size=solver_bsize,
                    update_rule='adam',
                    optim_config={
                      'learning_rate': lr,
                    },
                    verbose=False)
    solver.train()

    bn_solvers = []
    for i in range(len(batch_sizes)):
        b_size=batch_sizes[i]
        print('Normalization: batch size = ',b_size)
        bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=normalization_mode)
        bn_solver = Solver(bn_model, small_data,
                        num_epochs=n_epochs, batch_size=b_size,
                        update_rule='adam',
                        optim_config={
                          'learning_rate': lr,
                        },
                        verbose=False)
        bn_solver.train()
        bn_solvers.append(bn_solver)

    return bn_solvers, solver, batch_sizes

batch_sizes = [5,10,50]
bn_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('batchnorm')
```

```
plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 2:
Describe the results of this experiment. What does this imply about the relationship between batch normalization and batch size? Why is this relationship observed?

## Answer:
[FILL THIS IN]

# Layer Normalization
Batch normalization has proved to be effective in making networks easier to train, but the dependency on batch size makes it less useful in complex networks which have a cap on the input batch size due to hardware limitations.

Several alternatives to batch normalization have been proposed to mitigate this problem; one such technique is Layer Normalization [2]. Instead of normalizing over the batch, we normalize over the features. In other words, when using Layer Normalization, each feature vector corresponding to a single datapoint is normalized based on the sum of all terms within that feature vector.

[2] [Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. "Layer Normalization." stat 1050 (2016): 21.](https://arxiv.org/pdf/1607.06450.pdf)

## Inline Question 3:
Which of these data preprocessing steps is analogous to batch normalization, and which is analogous to layer normalization?

1. Scaling each image in the dataset, so that the RGB channels for each row of pixels within an image sums up to 1.
2. Scaling each image in the dataset, so that the RGB channels for all pixels within an image sums up to 1.
3. Subtracting the mean image of the dataset from each image in the dataset.
4. Setting all RGB values to either 0 or 1 depending on a given threshold.

## Answer:
[FILL THIS IN]

# Layer Normalization: Implementation

Now you'll implement layer normalization. This step should be relatively straightforward, as conceptually the implementation is almost identical to that of batch normalization. One significant difference though is that for layer normalization, we do not keep track of the moving moments, and the testing phase is identical to the training phase, where the mean and variance are directly calculated per datapoint.

Here's what you need to do:

* In `cs231n/layers.py`, implement the forward pass for layer normalization in the function `layernorm_forward`.

Run the cell below to check your results.
* In `cs231n/layers.py`, implement the backward pass for layer normalization in the function `layernorm_backward`.

Run the second cell below to check your results.
* Modify `cs231n/classifiers/fc_net.py` to add layer normalization to the `FullyConnectedNet`. When the `normalization` flag is set to `"layernorm"` in the constructor, you should insert a layer normalization layer before each ReLU nonlinearity.

Run the third cell below to run the batch size experiment on layer normalization.

```
# Check the training-time forward pass by checking means and variances
# of features both before and after layer normalization.

# Simulate the forward pass for a two-layer network.
np.random.seed(231)
N, D1, D2, D3 =4, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before layer normalization:')
print_mean_std(a,axis=1)

gamma = np.ones(D3)
beta = np.zeros(D3)

# Means should be close to zero and stds close to one.
print('After layer normalization (gamma=1, beta=0)')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)

gamma = np.asarray([3.0,3.0,3.0])
beta = np.asarray([5.0,5.0,5.0])

# Now means should be close to beta and stds close to gamma.
print('After layer normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)
```

```
# Gradient check batchnorm backward pass.
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

ln_param = {}
fx = lambda x: layernorm_forward(x, gamma, beta, ln_param)[0]
fg = lambda a: layernorm_forward(x, a, beta, ln_param)[0]
fb = lambda b: layernorm_forward(x, gamma, b, ln_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = layernorm_forward(x, gamma, beta, ln_param)
dx, dgamma, dbeta = layernorm_backward(dout, cache)

# You should expect to see relative errors between 1e-12 and 1e-8.
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

# Layer Normalization and Batch Size

We will now run the previous batch size experiment with layer normalization instead of batch normalization. Compared to the previous experiment, you should see a markedly smaller influence of batch size on the training history!

```
ln_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('layernorm')

plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 4:
When is layer normalization likely to not work well, and why?

1. Using it in a very deep network
2. Having a very small dimension of features
3. Having a high regularization term

## Answer:
[FILL THIS IN]