```python
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

np.random.seed(1789)

from IPython.core.display import HTML
def css_styling():
    styles = open("styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

# Statistical Data Modeling

Pandas, NumPy and SciPy provide the core functionality for building statistical models of our data. We use models to:

- Concisely **describe** the components of our data
- Provide **inference** about underlying parameters that may have generated the data
- Make **predictions** about unobserved data, or expected future observations.

This section of the tutorial illustrates how to use Python to build statistical models of low to moderate difficulty from scratch, and use them to extract estimates and associated measures of uncertainty.

Estimation
==========

An recurring statistical problem is finding estimates of the relevant parameters that correspond to the distribution that best represents our data.

In **parametric** inference, we specify *a priori* a suitable distribution, then choose the parameters that best fit the data.

* e.g. the mean $\mu$ and the variance $\sigma^2$ in the case of the normal distribution

```python
x = np.array([ 1.00201077,  1.58251956,  0.94515919,  6.48778002,  1.47764604,
        5.18847071,  4.21988095,  2.85971522,  3.40044437,  3.74907745,
        1.18065796,  3.74748775,  3.27328568,  3.19374927,  8.0726155 ,
        0.90326139,  2.34460034,  2.14199217,  3.27446744,  3.58872357,
        1.20611533,  2.16594393,  5.56610242,  4.66479977,  2.3573932 ])
_ = plt.hist(x, bins=7)
```

### Fitting data to probability distributions

We start with the problem of finding values for the parameters that provide the best fit between the model and the data, called point estimates. First, we need to define what we mean by ‘best fit’. There are two commonly used criteria:

* **Method of moments** chooses the parameters so that the sample moments (typically the sample mean and variance) match the theoretical moments of our chosen distribution.
* **Maximum likelihood** chooses the parameters to maximize the likelihood, which measures how likely it is to observe our given sample.

### Discrete Random Variables

$$X = \{0,1\}$$

$$Y = \{\ldots,-2,-1,0,1,2,\ldots\}$$

**Probability Mass Function**:

For discrete $X$,

$$Pr(X=x) = f(x|\theta)$$

***e.g. Poisson distribution***

The Poisson distribution models unbounded counts:

$$Pr(X=x)=\frac{e^{-\lambda}\lambda^x}{x!}$$

* $X=\{0,1,2,\ldots\}$
* $\lambda > 0$

$$E(X) = \text{Var}(X) = \lambda$$

### Continuous Random Variables

$$X \in [0,1]$$

$$Y \in (-\infty, \infty)$$

**Probability Density Function**:

For continuous $X$,

$$Pr(x \le X \le x + dx) = f(x|\theta)dx \, \text{ as } \, dx \rightarrow 0$$

***e.g. normal distribution***

$$f(x) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left[-\frac{(x-\mu)^2}{2\sigma^2}\right]$$

* $X \in \mathbf{R}$
* $\mu \in \mathbf{R}$
* $\sigma>0$

$$\begin{align}E(X) &= \mu \cr
\text{Var}(X) &= \sigma^2 \end{align}$$

### Example: Nashville Precipitation

The dataset `nashville_precip.txt` contains [NOAA precipitation data for Nashville measured since 1871](http://bit.ly/nasvhville_precip_data).

The gamma distribution is often a good fit to aggregated rainfall data, and will be our candidate distribution in this case.

```python
precip = pd.read_table("../data/nashville_precip.txt", index_col=0, na_values='NA', delim_whitespace=True)
precip.head()
```

      Jan
      Feb
      Mar
      Apr
      May
      Jun
      Jul
      Aug
      Sep
      Oct
      Nov
      Dec

      Year

      1871
      2.76
      4.58
      5.01
      4.13
      3.30
      2.98
      1.58
      2.36
      0.95
      1.31
      2.13
      1.65

      1872
      2.32
      2.11
      3.14
      5.91
      3.09
      5.17
      6.10
      1.65
      4.50
      1.58
      2.25
      2.38

      1873
      2.96
      7.14
      4.11
      3.59
      6.31
      4.20
      4.63
      2.36
      1.81
      4.28
      4.36
      5.94

      1874
      5.22
      9.23
      5.36
      11.84
      1.49
      2.87
      2.65
      3.52
      3.12
      2.63
      6.12
      4.19

      1875
      6.15
      3.06
      8.14
      4.22
      1.73
      5.63
      8.12
      1.60
      3.79
      1.25
      5.46
      4.30

```python
_ = precip.hist(sharex=True, sharey=True, grid=False)
plt.tight_layout()
```

The first step is recognizing what sort of distribution to fit our data to. A couple of observations:

1. The data are skewed, with a longer tail to the right than to the left
2. The data are positive-valued, since they are measuring rainfall
3. The data are continuous

There are a few possible choices, but one suitable alternative is the **gamma distribution**:

$$x \sim \text{Gamma}(\alpha, \beta) = \frac{\beta^{\alpha}x^{\alpha-1}e^{-\beta x}}{\Gamma(\alpha)}$$

The ***method of moments*** simply assigns the empirical mean and variance to their theoretical counterparts, so that we can solve for the parameters.

So, for the gamma distribution, the mean and variance are:

$$ \hat{\mu} = \bar{X} = \alpha \beta $$
$$ \hat{\sigma}^2 = S^2 = \alpha \beta^2 $$

So, if we solve for these parameters, we can use a gamma distribution to describe our data:

$$ \alpha = \frac{\bar{X}^2}{S^2}, \, \beta = \frac{S^2}{\bar{X}} $$

Let's deal with the missing value in the October data. Given what we are trying to do, it is most sensible to fill in the missing value with the average of the available values. We will learn more sophisticated methods for handling missing data later in the course.

```python
precip.fillna(value={'Oct': precip.Oct.mean()}, inplace=True)
```

      Jan
      Feb
      Mar
      Apr
      May
      Jun
      Jul
      Aug
      Sep
      Oct
      Nov
      Dec

      Year

      1871
      2.76
      4.58
      5.01
      4.13
      3.30
      2.98
      1.58
      2.36
      0.95
      1.31
      2.13
      1.65

      1872
      2.32
      2.11
      3.14
      5.91
      3.09
      5.17
      6.10
      1.65
      4.50
      1.58
      2.25
      2.38

      1873
      2.96
      7.14
      4.11
      3.59
      6.31
      4.20
      4.63
      2.36
      1.81
      4.28
      4.36
      5.94

      1874
      5.22
      9.23
      5.36
      11.84
      1.49
      2.87
      2.65
      3.52
      3.12
      2.63
      6.12
      4.19

      1875
      6.15
      3.06
      8.14
      4.22
      1.73
      5.63
      8.12
      1.60
      3.79
      1.25
      5.46
      4.30

      1876
      6.41
      2.22
      5.28
      3.62
      3.40
      5.65
      7.15
      5.77
      2.52
      2.68
      1.26
      0.95

      1877
      4.05
      1.06
      4.98
      9.47
      1.25
      6.02
      3.25
      4.16
      5.40
      2.61
      4.93
      2.49

      1878
      3.34
      2.10
      3.48
      6.88
      2.33
      3.28
      9.43
      5.02
      1.28
      2.17
      3.20
      6.04

      1879
      6.32
      3.13
      3.81
      2.88
      2.88
      2.50
      8.47
      4.62
      5.18
      2.90
      5.85
      9.15

      1880
      3.74
      12.37
      8.16
      5.26
      4.13
      3.97
      5.69
      2.22
      5.39
      7.24
      5.77
      3.32

      1881
      3.54
      5.48
      2.79
      5.12
      3.67
      3.70
      0.86
      1.81
      6.57
      4.80
      4.89
      4.85

      1882
      14.51
      8.61
      9.38
      3.59
      7.38
      2.54
      4.06
      5.54
      1.61
      1.11
      3.60
      1.52

      1883
      3.76
      7.90
      3.98
      9.12
      4.82
      3.82
      4.94
      4.47
      2.23
      5.27
      3.11
      4.97

      1884
      7.20
      8.18
      8.89
      3.51
      3.58
      6.53
      3.18
      2.81
      2.36
      2.43
      1.57
      3.78

      1885
      6.29
      2.00
      2.33
      3.75
      4.36
      3.72
      5.26
      1.02
      5.60
      2.99
      2.73
      2.90

      1886
      5.18
      3.82
      4.76
      2.36
      2.10
      7.69
      1.90
      5.50
      3.68
      0.51
      5.76
      1.48

      1887
      5.13
      8.47
      3.36
      2.67
      3.43
      2.31
      3.77
      2.89
      6.85
      1.92
      2.29
      5.31

      1888
      6.29
      3.78
      6.46
      4.18
      2.97
      4.68
      2.36
      7.03
      3.82
      2.82
      4.33
      1.77

      1889
      3.83
      1.84
      2.47
      2.83
      5.30
      5.33
      2.74
      1.57
      6.81
      1.54
      6.88
      1.17

      1890
      8.10
      10.95
      8.64
      3.84
      4.16
      2.23
      0.46
      6.59
      5.86
      3.01
      2.01
      4.12

      1891
      6.15
      6.96
      10.31
      2.24
      2.39
      6.50
      1.49
      3.72
      1.25
      0.84
      6.71
      4.26

      1892
      2.81
      2.73
      4.10
      7.45
      4.03
      5.01
      5.13
      3.39
      4.78
      0.25
      3.91
      6.43

      1893
      1.27
      4.88
      3.37
      4.11
      7.31
      4.74
      2.12
      1.92
      6.43
      3.68
      2.97
      3.50

      1894
      4.28
      8.65
      2.69
      4.05
      2.53
      3.55
      5.45
      2.43
      3.07
      0.53
      1.92
      2.81

      1895
      5.71
      0.98
      5.09
      3.07
      2.05
      2.90
      7.14
      1.40
      6.69
      1.57
      2.14
      4.09

      1896
      1.37
      3.65
      6.45
      2.92
      4.05
      1.82
      7.33
      1.40
      2.74
      0.98
      5.71
      1.79

      1897
      3.13
      3.84
      8.49
      5.79
      1.22
      1.82
      8.53
      2.34
      0.19
      0.92
      2.83
      4.93

      1898
      9.46
      0.63
      5.36
      3.16
      1.80
      4.97
      4.50
      6.56
      4.87
      3.21
      3.09
      2.41

      1899
      5.59
      5.19
      7.81
      3.25
      3.36
      0.75
      6.44
      2.53
      1.50
      1.83
      1.55
      4.64

      1900
      2.61
      3.80
      2.20
      4.04
      1.86
      10.35
      2.87
      1.24
      4.55
      3.93
      8.87
      2.22

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      1982
      6.50
      4.80
      3.00
      4.36
      4.19
      2.28
      5.47
      3.46
      3.23
      1.91
      3.87
      6.36

      1983
      2.56
      2.93
      3.44
      6.80
      11.04
      3.93
      1.71
      1.36
      0.45
      2.77
      6.98
      7.75

      1984
      1.79
      2.38
      5.14
      8.41
      9.68
      4.49
      6.63
      2.42
      0.97
      6.00
      6.20
      2.38

      1985
      3.02
      3.30
      2.70
      2.91
      2.65
      1.53
      2.00
      3.91
      2.52
      1.59
      3.81
      0.98

      1986
      0.19
      3.59
      2.29
      0.52
      3.36
      2.38
      0.77
      3.38
      2.19
      2.19
      7.43
      3.31

      1987
      1.61
      4.87
      1.18
      1.03
      4.41
      2.82
      2.56
      0.73
      1.95
      0.21
      3.40
      5.46

      1988
      3.73
      2.02
      2.18
      2.09
      1.86
      0.45
      3.26
      2.39
      2.45
      1.54
      5.49
      3.95

      1989
      4.52
      9.36
      5.31
      2.68
      4.61
      7.87
      3.18
      3.67
      6.30
      3.62
      3.94
      1.97

      1990
      2.76
      4.73
      3.26
      1.60
      2.80
      2.37
      4.86
      3.12
      2.13
      4.41
      4.29
      10.76

      1991
      2.92
      5.44
      4.25
      3.35
      5.63
      1.25
      2.82
      1.79
      5.47
      3.88
      2.87
      7.27

      1992
      2.97
      2.60
      4.50
      0.77
      3.12
      4.31
      5.89
      3.25
      3.45
      1.62
      4.48
      2.88

      1993
      2.76
      3.33
      5.50
      3.33
      4.50
      5.31
      3.64
      1.76
      2.90
      2.20
      2.53
      6.62

      1994
      4.36
      6.18
      7.56
      5.72
      3.76
      8.08
      4.82
      5.05
      4.20
      3.31
      4.04
      2.69

      1995
      5.61
      1.81
      3.87
      3.95
      7.66
      3.69
      1.95
      3.40
      5.00
      5.60
      3.98
      2.32

      1996
      3.82
      2.46
      5.15
      3.68
      4.48
      3.68
      5.45
      1.09
      4.88
      3.16
      6.00
      4.77

      1997
      4.19
      3.10
      9.64
      2.42
      4.92
      6.66
      3.26
      3.52
      5.75
      2.71
      6.59
      2.19

      1998
      3.68
      4.11
      3.13
      6.31
      4.46
      11.95
      4.63
      2.93
      1.39
      1.59
      1.30
      6.53

      1999
      9.28
      2.33
      4.27
      2.29
      4.35
      3.56
      3.19
      3.05
      1.97
      2.04
      2.99
      2.50

      2000
      3.52
      3.75
      3.34
      6.23
      7.66
      1.74
      2.25
      1.95
      1.90
      0.26
      6.39
      3.44

      2001
      3.21
      8.54
      2.73
      2.42
      5.54
      4.47
      2.77
      4.07
      1.79
      4.61
      5.09
      3.32

      2002
      4.93
      1.99
      9.40
      4.31
      3.98
      3.76
      5.64
      3.13
      6.29
      4.48
      2.91
      5.81

      2003
      1.59
      8.47
      2.30
      4.69
      10.73
      7.08
      2.87
      3.88
      8.70
      1.80
      4.17
      3.19

      2004
      3.60
      5.77
      4.81
      6.69
      6.90
      3.39
      3.19
      4.24
      4.55
      4.90
      5.21
      5.93

      2005
      4.42
      3.84
      3.90
      6.93
      1.03
      2.70
      2.39
      6.89
      1.44
      0.02
      3.29
      2.46

      2006
      6.57
      2.69
      2.90
      4.14
      4.95
      2.19
      2.64
      5.20
      4.00
      2.98
      4.05
      3.41

      2007
      3.32
      1.84
      2.26
      2.75
      3.30
      2.37
      1.47
      1.38
      1.99
      4.95
      6.20
      3.83

      2008
      4.76
      2.53
      5.56
      7.20
      5.54
      2.21
      4.32
      1.67
      0.88
      5.03
      1.75
      6.72

      2009
      4.59
      2.85
      2.92
      4.13
      8.45
      4.53
      6.03
      2.14
      11.08
      6.49
      0.67
      3.99

      2010
      4.13
      2.77
      3.52
      3.48
      16.43
      4.96
      5.86
      6.99
      1.17
      2.49
      5.41
      1.87

      2011
      2.31
      5.54
      4.59
      7.51
      4.38
      5.04
      3.46
      1.78
      6.20
      0.93
      6.15
      4.25

141 rows × 12 columns

Now, let's calculate the sample moments of interest, the means and variances by month:

```python
precip_mean = precip.mean()
precip_mean
```

    Jan    4.523688
    Feb    4.097801
    Mar    4.977589
    Apr    4.204468
    May    4.325674
    Jun    3.873475
    Jul    3.895461
    Aug    3.367305
    Sep    3.377660
    Oct    2.610500
    Nov    3.685887
    Dec    4.176241
    dtype: float64

```python
precip_var = precip.var()
precip_var
```

    Jan    6.928862
    Feb    5.516660
    Mar    5.365444
    Apr    4.117096
    May    5.306409
    Jun    5.033206
    Jul    3.777012
    Aug    3.779876
    Sep    4.940099
    Oct    2.741659
    Nov    3.679274
    Dec    5.418022
    dtype: float64

We then use these moments to estimate $\alpha$ and $\beta$ for each month:

```python
alpha_mom = precip_mean ** 2 / precip_var
beta_mom = precip_var / precip_mean
```

```python
alpha_mom, beta_mom
```

    (Jan    2.953407
     Feb    3.043866
     Mar    4.617770
     Apr    4.293694
     May    3.526199
     Jun    2.980965
     Jul    4.017624
     Aug    2.999766
     Sep    2.309383
     Oct    2.485616
     Nov    3.692511
     Dec    3.219070
     dtype: float64, Jan    1.531684
     Feb    1.346249
     Mar    1.077920
     Apr    0.979219
     May    1.226724
     Jun    1.299403
     Jul    0.969593
     Aug    1.122522
     Sep    1.462581
     Oct    1.050243
     Nov    0.998206
     Dec    1.297344
     dtype: float64)

We can use the `gamma.pdf` function in `scipy.stats.distributions` to plot the ditribtuions implied by the calculated alphas and betas. For example, here is January:

```python
from scipy.stats.distributions import gamma

precip.Jan.hist(normed=True, bins=20)
plt.plot(np.linspace(0, 10), gamma.pdf(np.linspace(0, 10), alpha_mom[0], beta_mom[0]))
```

Looping over all months, we can create a grid of plots for the distribution of rainfall, using the gamma distribution:

```python
axs = precip.hist(normed=True, figsize=(12, 8), sharex=True, sharey=True, bins=15, grid=False)

for ax in axs.ravel():

    # Get month
    m = ax.get_title()

    # Plot fitted distribution
    x = np.linspace(*ax.get_xlim())
    ax.plot(x, gamma.pdf(x, alpha_mom[m], beta_mom[m]))

    # Annotate with parameter estimates
    label = 'alpha = {0:.2f}\nbeta = {1:.2f}'.format(alpha_mom[m], beta_mom[m])
    ax.annotate(label, xy=(10, 0.2))

plt.tight_layout()
```

Maximum Likelihood
==================

**Maximum likelihood** (ML) fitting is usually more work than the method of moments, but it is preferred as the resulting estimator is known to have good theoretical properties.

There is a ton of theory regarding ML. We will restrict ourselves to the mechanics here.

Say we have some data $y = y_1,y_2,\ldots,y_n$ that is distributed according to some distribution:

$$Pr(Y_i=y_i | \theta)$$

Here, for example, is a **Poisson distribution** that describes the distribution of some discrete variables, typically *counts*:

```python
y = np.random.poisson(5, size=100)
plt.hist(y, bins=12, normed=True)
plt.xlabel('y'); plt.ylabel('Pr(y)')
```

The product $\prod_{i=1}^n Pr(y_i | \theta)$ gives us a measure of how **likely** it is to observe values $y_1,\ldots,y_n$ given the parameters $\theta$.

Maximum likelihood fitting consists of choosing the appropriate function $l= Pr(Y|\theta)$ to maximize for a given set of observations. We call this function the *likelihood function*, because it is a measure of how likely the observations are if the model is true.

> Given these data, how likely is this model?

In the above model, the data were drawn from a Poisson distribution with parameter $\lambda =5$.

$$L(y|\lambda=5) = \frac{e^{-5} 5^y}{y!}$$

So, for any given value of $y$, we can calculate its likelihood:

```python
poisson_like = lambda x, lam: np.exp(-lam) * (lam**x) / (np.arange(x)+1).prod()

lam = 6
value = 10
poisson_like(value, lam)
```

    0.041303093412337726

```python
np.sum(poisson_like(yi, lam) for yi in y)
```

    11.499056250911673

```python
lam = 8
np.sum(poisson_like(yi, lam) for yi in y)
```

    7.8028592304816868

We can plot the likelihood function for any value of the parameter(s):

```python
lambdas = np.linspace(0,15)
x = 5
plt.plot(lambdas, [poisson_like(x, l) for l in lambdas])
plt.xlabel('$\lambda$')
plt.ylabel('L($\lambda$|x={0})'.format(x))
```

How is the likelihood function different than the probability distribution function (PDF)? The likelihood is a function of the parameter(s) *given the data*, whereas the PDF returns the probability of data given a particular parameter value. Here is the PDF of the Poisson for $\lambda=5$.

```python
lam = 5
xvals = np.arange(15)
plt.bar(xvals, [poisson_like(x, lam) for x in xvals], width=0.2)
plt.xlabel('x')
plt.ylabel('Pr(X|$\lambda$=5)')
```

*Why are we interested in the likelihood function?*

A reasonable estimate of the true, unknown value for the parameter is one which **maximizes the likelihood function**. So, inference is reduced to an optimization problem.

Going back to the rainfall data, if we are using a gamma distribution we need to maximize:

$$\begin{align}l(\alpha,\beta) &= \sum_{i=1}^n \log[\beta^{\alpha} x^{\alpha-1} e^{-x/\beta}\Gamma(\alpha)^{-1}] \cr
&= n[(\alpha-1)\overline{\log(x)} - \bar{x}\beta + \alpha\log(\beta) - \log\Gamma(\alpha)]\end{align}$$

*N.B.: Its usually easier to work in the log scale*

where $n = 2012 − 1871 = 141$ and the bar indicates an average over all *i*. We choose $\alpha$ and $\beta$ to maximize $l(\alpha,\beta)$.

Notice $l$ is infinite if any $x$ is zero. We do not have any zeros, but we do have an NA value for one of the October data, which we dealt with above.

### Finding the MLE

To find the maximum of any function, we typically take the *derivative* with respect to the variable to be maximized, set it to zero and solve for that variable.

$$\frac{\partial l(\alpha,\beta)}{\partial \beta} = n\left(\frac{\alpha}{\beta} - \bar{x}\right) = 0$$

Which can be solved as $\beta = \alpha/\bar{x}$. However, plugging this into the derivative with respect to $\alpha$ yields:

$$\frac{\partial l(\alpha,\beta)}{\partial \alpha} = \log(\alpha) + \overline{\log(x)} - \log(\bar{x}) - \frac{\Gamma(\alpha)'}{\Gamma(\alpha)} = 0$$

This has no closed form solution. We must use ***numerical optimization***!

Numerical optimization alogarithms take an initial "guess" at the solution, and **iteratively** improve the guess until it gets "close enough" to the answer.

Here, we will use *Newton-Raphson* method, which is a **root-finding algorithm**:

$$x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)}$$

which is available to us via SciPy:

```python
from scipy.optimize import newton
```

Here is a graphical example of how Newton-Raphson converges on a solution, using an arbitrary function:

```python
%run newton_raphson_plot.py
```

To apply the Newton-Raphson algorithm, we need a function that returns a vector containing the **first and second derivatives** of the function with respect to the variable of interest. The second derivative of the gamma distribution with respect to $\alpha$ is:

$$\frac{\partial^2 l(\alpha,\beta)}{\partial \alpha^2} = \frac{1}{\alpha} - \frac{\partial}{\partial \alpha} \left[ \frac{\Gamma(\alpha)'}{\Gamma(\alpha)} \right]$$

```python
from scipy.special import psi, polygamma

dlgamma = lambda a, log_mean, mean_log: np.log(a) - psi(a) - log_mean + mean_log
dl2gamma = lambda a, *args: 1./a - polygamma(1, a)
```

where `log_mean` and `mean_log` are $\log{\bar{x}}$ and $\overline{\log(x)}$, respectively. `psi` and `polygamma` are complex functions of the Gamma function that result when you take first and second derivatives of that function.

```python
# Calculate statistics
log_mean = precip.mean().apply(np.log)
mean_log = precip.apply(np.log).mean()
```

Time to optimize!

```python
# Alpha MLE for December
alpha_mle = newton(dlgamma, 2, dl2gamma, args=(log_mean[-1], mean_log[-1]))
alpha_mle
```

    3.5189679152399647

And now plug this back into the solution for beta:

$$ \beta  = \frac{\alpha}{\bar{X}} $$

```python
beta_mle = alpha_mle/precip.mean()[-1]
beta_mle
```

    0.84261607548413797

We can compare the fit of the estimates derived from MLE to those from the method of moments:

```python
dec = precip.Dec
dec.hist(normed=True, bins=10, grid=False)
x = np.linspace(0, dec.max())
plt.plot(x, gamma.pdf(x, alpha_mom[-1], beta_mom[-1]), 'm-', label='Moment estimator')
plt.plot(x, gamma.pdf(x, alpha_mle, beta_mle), 'r--', label='ML estimator')
plt.legend()
```

For some common distributions, SciPy includes methods for fitting via MLE:

```python
from scipy.stats import gamma

gamma.fit(precip.Dec)
```

    (2.2427517753152308, 0.65494604470188622, 1.570073932063466)

This fit is not directly comparable to our estimates, however, because SciPy's `gamma.fit` method fits an odd 3-parameter version of the gamma distribution.

### Model checking

An informal way of checking the fit of our parametric model is to compare the observed quantiles of the data to those of the theoretical model we are fitting it to. If the model is a good fit, the points should fall on a 45-degree reference line. This is called a **probability plot**.

SciPy includes a `probplot` function that generates probability plots based on the data and a specified distribution.

```python
from scipy.stats import probplot

probplot(precip.Dec, dist=gamma(3.51, scale=0.84), plot=plt);
```

### Example: truncated distribution

Suppose that we observe $Y$ truncated below at $a$ (where $a$ is known). If $X$ is the distribution of our observation, then:

$$ P(X \le x) = P(Y \le x|Y \gt a) = \frac{P(a \lt Y \le x)}{P(Y \gt a)}$$

(so, $Y$ is the original variable and $X$ is the truncated variable)

Then X has the density:

$$f_X(x) = \frac{f_Y (x)}{1−F_Y (a)} \, \text{for} \, x \gt a$$

Suppose $Y \sim N(\mu, \sigma^2)$ and $x_1,\ldots,x_n$ are independent observations of $X$. We can use maximum likelihood to find $\mu$ and $\sigma$.

First, we can simulate a truncated distribution using a `while` statement to eliminate samples that are outside the support of the truncated distribution.

```python
x = np.random.normal(size=10000)

# Truncation point
a = -1

# Resample until all points meet criterion
x_small = x < a
while x_small.sum():
    x[x_small] = np.random.normal(size=x_small.sum())
    x_small = x < a

_ = plt.hist(x, bins=100)
```

We can construct a log likelihood for this function using the conditional form:

$$f_X(x) = \frac{f_Y (x)}{1−F_Y (a)} \, \text{for} \, x \gt a$$

The denominator normalizes the truncated distribution so that it integrates to one.

```python
from scipy.stats.distributions import norm

trunc_norm = lambda theta, a, x: -(np.log(norm.pdf(x, theta[0], theta[1])) -
                                      np.log(1 - norm.cdf(a, theta[0], theta[1]))).sum()
```

For this example, we will use an optimization algorithm, the **Nelder-Mead simplex algorithm**. It has a couple of advantages:

- it does not require derivatives
- it can optimize (minimize) a vector of parameters

SciPy implements this algorithm in its `fmin` function:

```python
from scipy.optimize import fmin

fmin(trunc_norm, np.array([1,2]), args=(-1, x))
```

    Optimization terminated successfully.
             Current function value: 11084.885149
             Iterations: 47
             Function evaluations: 88

    array([ 0.00531007,  1.00652051])

In general, simulating data is a terrific way of testing your model before using it with real data.

## Kernel density estimates

In some instances, we may not be interested in the parameters of a particular distribution of data, but just a smoothed representation of the data at hand. In this case, we can estimate the disribution *non-parametrically* (i.e. making no assumptions about the form of the underlying distribution) using kernel density estimation.

```python
# Some random data
y = np.random.normal(10, size=15)
y
```

    array([  9.68444175,   6.86641455,   8.90236824,   8.48662651,
             9.4599326 ,   9.83248454,   9.45367613,  12.95585035,
            10.58747989,  10.94829904,  10.48694903,   9.13200438,
             9.73979452,  10.14852737,   8.62582257])

The kernel estimator is a sum of symmetric densities centered at each observation. The selected kernel function determines the shape of each component while the **bandwidth** determines their spread. For example, if we use a Gaussian kernel function, the variance acts as the bandwidth.

```python
x = np.linspace(7, 13, 100)
# Smoothing parameter
s = 0.3
# Calculate the kernels
kernels = np.transpose([norm.pdf(x, yi, s) for yi in y])
plt.plot(x, kernels, 'k:')
plt.plot(x, kernels.sum(1))
plt.plot(y, np.zeros(len(y)), 'ro', ms=10)
```

SciPy implements a Gaussian KDE that automatically chooses an appropriate bandwidth. Let's create a bi-modal distribution of data that is not easily summarized by a parametric distribution:

```python
# Create a bi-modal distribution with a mixture of Normals.
x1 = np.random.normal(0, 2, 50)
x2 = np.random.normal(5, 1, 50)

# Append by row
x = np.r_[x1, x2]
```

```python
plt.hist(x, bins=10, normed=True)
```

```python
from scipy.stats import kde

density = kde.gaussian_kde(x)
xgrid = np.linspace(x.min(), x.max(), 100)
plt.hist(x, bins=8, normed=True)
plt.plot(xgrid, density(xgrid), 'r-')
```

### Exercise: Comparative Chopstick Effectiveness

A few researchers set out to determine what the optimal length for chopsticks is. The dataset `chopstick-effectiveness.csv` includes measurements of "Food Pinching Efficiency" across a range of chopstick lengths for 31 individuals.

Use the method of moments or MLE to calculate the mean and variance of food pinching efficiency for each chopstick length. This means you need to select an appropriate distributional form for this data.

```python
# Write your answer here
```


#### Procesamiento Digital de Señales

# Trabajo Práctico Nº0
#### Yanina Corsaro

# Introducción
Jupyter Notebook es una herramienta para la confección de reportes técnicos, dado que permite la interacción en el mismo ambiente de:
1. un procesador de texto elemental (formato Markdown) que permite resaltar texto, en forma de *itálica* o **negrita** de manera muy legible (haciendo doble click en este texto podrás ver el código fuente estilo Markdown). Cuenta con estilos predefinidos:

# Título 1
## Título 2
### Título 3

y también la capacidad de incluir enlaces a otras páginas, como por ejemplo [esta página](https://medium.com/ibm-data-science-experience/markdown-for-jupyter-notebooks-cheatsheet-386c05aeebed) donde encontrarás más funcionalidades del lenguaje **Markdown**

2. capacidad para incluir lenguaje matemático estilo LaTex, tanto de forma presentada

\begin{equation}
T(z) = \frac{Y(z)}{X(z)} = \frac{ b_2 \, z^{-2} + b_1 \,  z^{-1} + b_0 }
{a_2 \, z^{-2} + a_1 \,  z^{-1} + a_0}
\end{equation}

como *inline* en el propio párrafo $y[k] = \frac{1}{a_0} \left( \sum_{m=0}^{M} b_m \; x[k-m] - \sum_{n=1}^{N} a_n \; y[k-n] \right) $

3. La posibilidad de incluir scripts en Python, como los que usaremos para las simulaciones en los TPs de la materia. En este caso usaremos el *testbench0.py* como ejemplo. Una vez que lo probamos y estamos seguros que funciona de forma esperada en *Spyder*, podemos incluir los resultados de la simulación de manera casi transparente. Solo tenemos que agregar una celda de código donde incluimos el código, y los resultados directamente quedan incluidos en este documento.

```python
# Módulos para Jupyter
import warnings
warnings.filterwarnings('ignore')

import numpy as np
import matplotlib as mpl
#%%  Inicialización de librerías
# Setup inline graphics: Esto lo hacemos para que el tamaño de la salida,
# sea un poco más adecuada al tamaño del documento
mpl.rcParams['figure.figsize'] = (10,10)

import matplotlib.pyplot as plt
import pdsmodulos as pds

#%% Esto tiene que ver con cuestiones de presentación de los gráficos,
# NO ES IMPORTANTE
fig_sz_x = 14
fig_sz_y = 13
fig_dpi = 80 # dpi

fig_font_family = 'Ubuntu'
fig_font_size = 16

plt.rcParams.update({'font.size':fig_font_size})
plt.rcParams.update({'font.family':fig_font_family})

##############################################
#%% A partir de aquí comienza lo IMPORTANTE #
#############################################

def my_testbench( sig_type ):

    # Datos generales de la simulación
    fs = 1000.0 # frecuencia de muestreo (Hz)
    N = 1000   # cantidad de muestras

    ts = 1/fs # tiempo de muestreo
    df = fs/N # resolución espectral

    # grilla de sampleo temporal
    tt = np.linspace(0, (N-1)*ts, N).flatten()

    # grilla de sampleo frecuencial
    ff = np.linspace(0, (N-1)*df, N).flatten()

    # Concatenación de matrices:
    # guardaremos las señales creadas al ir poblando la siguiente matriz vacía
    x = np.array([], dtype=np.float).reshape(N,0)
    ii = 0

    # estructuras de control de flujo
    if sig_type['tipo'] == 'senoidal':

        # calculo cada senoidal de acuerdo a sus parámetros
        for this_freq in sig_type['frecuencia']:
            # prestar atención que las tuplas dentro de los diccionarios también pueden direccionarse mediante "ii"
            aux = sig_type['amplitud'][ii] * np.sin( 2*np.pi*this_freq*tt + sig_type['fase'][ii] )
            # para concatenar horizontalmente es necesario cuidar que tengan iguales FILAS
            x = np.hstack([x, aux.reshape(N,1)] )
            ii += 1

    elif sig_type['tipo'] == 'ruido':

        # calculo cada señal de ruido incorrelado (blanco), Gausiano de acuerdo a sus parámetros
        # de varianza
        for this_var in sig_type['varianza']:
            aux = np.sqrt(this_var) * np.random.randn(N,1)
            # para concatenar horizontalmente es necesario cuidar que tengan iguales FILAS
            x = np.hstack([x, aux] )

        # Podemos agregar algún dato extra a la descripción de forma programática
        # {0:.3f} significa 0: primer argunmento de format
        # .3f formato flotante, con 3 decimales
        # $ ... $ indicamos que incluiremos sintaxis LaTex: $\hat{{\sigma}}^2$
        sig_props['descripcion'] = [ sig_props['descripcion'][ii] + ' - $\hat{{\sigma}}^2$ :{0:.3f}'.format( np.var(x[:,ii]))  for ii in range(0,len(sig_props['descripcion'])) ]

    else:

        print("Tipo de señal no implementado.")
        return

    #%% Presentación gráfica de los resultados

    plt.figure(1)
    line_hdls = plt.plot(tt, x)
    plt.title('Señal: ' + sig_type['tipo'] )
    plt.xlabel('tiempo [segundos]')
    plt.ylabel('Amplitud [V]')
    #    plt.grid(which='both', axis='both')

    # presentar una leyenda para cada tipo de señal
    axes_hdl = plt.gca()

    # este tipo de sintaxis es *MUY* de Python
    axes_hdl.legend(line_hdls, sig_type['descripcion'], loc='upper right'  )

    plt.show()

```

Dado que nuestro *testbench* ha sido desarrollado de manera funcional, llamando a la función *my_testbench()* con diferentes parámetros, podemos lograr funcionalidades diferentes, como mostramos a continuación primero con una senoidal:

```python
sig_props = { 'tipo': 'senoidal',
              'frecuencia': (3, 10, 20), # Uso de tuplas para las frecuencias
              'amplitud':   (1, 1,  1),
              'fase':       (0, 0,  0)
             }
sig_props2 = { 'tipo': 'senoidal',
              'frecuencia': (3, 10, 20), # Uso de tuplas para las frecuencias
              'amplitud':   (1, 1,  1),
              'fase':       (10, 5,  0)
             }
# Como también puedo agregar un campo descripción de manera programática
# este tipo de sintaxis es *MUY* de Python
sig_props['descripcion'] = [ str(a_freq) + ' Hz' for a_freq in sig_props['frecuencia'] ]
sig_props2['descripcion'] = [ str(a_freq) + ' Hz' for a_freq in sig_props['frecuencia'] ]
# Invocamos a nuestro testbench exclusivamente:
my_testbench( sig_props )
my_testbench( sig_props2 )
```

Y ahora con una señal aleatoria, en este caso ruido blanco Gaussiano incorrelado de varianza $\sigma^2$:

```python
# Usar CTRL+1 para comentar o descomentar el bloque de abajo.
sig_props = { 'tipo': 'ruido',
              'varianza': (1, 1, 1) # Uso de tuplas para las frecuencias
             }
sig_props['descripcion'] = [ '$\sigma^2$ = ' + str(a_var) for a_var in sig_props['varianza'] ]

# Invocamos a nuestro testbench exclusivamente:
my_testbench( sig_props )

```

Como puede verse en la figura anterior, al samplear una distribución estadística de media nula y varianza $\sigma^2=1$, obtenemos realizaciones cuyo parámetro $\sigma^2$ estimado, es decir $\hat\sigma^2$, tienen una desviación respecto al verdadero valor (sesgo). Nos ocuparemos de estudiar el sesgo y la varianza de algunos estimadores cuando veamos **Estimación Espectral**.

# Una vez terminado ...
Una vez que hayas termiando con la confección del documento, podemos utilizar una ventaja muy importante de este tipo de documentos que es la posibilidad de compartirlos *online* mediante la [página de nbviewer](http://nbviewer.jupyter.org/). Para ello es necesario que tu notebook y todos los recursos asociados estén alojados en un repositorio de [Github](https://github.com/). Como ejemplo, podés ver este mismo documento disponible [online](http://nbviewer.jupyter.org/github/marianux/pdstestbench/blob/master/notebook0.ipynb).

# Título de prueba

Agregué mi nombre y corrí el código para que aparezcan las imágenes. También agregue otra señal cambiando arbitrariamente la fase.


Probabilistic Programming
=====
and Bayesian Methods for Hackers
========

##### Version 0.1

`Original content created by Cam Davidson-Pilon`

`Ported to Python 3 and PyMC3 by Max Margenot (@clean_utensils) and Thomas Wiecki (@twiecki) at Quantopian (@quantopian)`
___

Welcome to *Bayesian Methods for Hackers*. The full Github repository is available at [github/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers](https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers). The other chapters can be found on the project's [homepage](https://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/). We hope you enjoy the book, and we encourage any contributions!

Chapter 1
======
***

The Philosophy of Bayesian Inference
------

> You are a skilled programmer, but bugs still slip into your code. After a particularly difficult implementation of an algorithm, you decide to test your code on a trivial example. It passes. You test the code on a harder problem. It passes once again. And it passes the next, *even more difficult*, test too! You are starting to believe that there may be no bugs in this code...

If you think this way, then congratulations, you already are thinking Bayesian! **Bayesian inference is simply updating your beliefs after considering new evidence.** A Bayesian can rarely be certain about a result, but he or she can be very confident. Just like in the example above, **we can never be 100% sure that our code is bug-free unless we test it on every possible problem**; something rarely possible in practice. Instead, we can test it on a large number of problems, and if it succeeds we can feel more *confident* about our code, but still not certain.  Bayesian inference works identically: we update our beliefs about an outcome; rarely can we be absolutely sure unless we rule out all other alternatives.

### The Bayesian state of mind

Bayesian inference differs from more traditional statistical inference by **preserving *uncertainty***. At first, this sounds like a bad statistical technique. Isn't statistics all about deriving *certainty* from randomness? To reconcile this, we need to start thinking like Bayesians.

The Bayesian world-view interprets probability as measure of *believability in an event*, that is, how confident we are in an event occurring. In fact, we will see in a moment that this is the natural interpretation of probability.

For this to be clearer, we consider an alternative interpretation of probability: ***Frequentist*, known as the more *classical* version of statistics, assume that probability is the long-run frequency of events (hence the bestowed title)**. For example, the *probability of plane accidents* under a frequentist philosophy is interpreted as the *long-term frequency of plane accidents*. This makes logical sense for many probabilities of events, but becomes more difficult to understand when events have no long-term frequency of occurrences. Consider: we often assign probabilities to outcomes of presidential elections, but the election itself only happens once! Frequentists get around this by invoking alternative realities and saying across all these realities, the frequency of occurrences defines the probability.

Bayesians, on the other hand, have a more intuitive approach. Bayesians interpret a probability as measure of ***belief***, or **confidence**, of an event occurring. Simply, a probability is a summary of an opinion. An individual who assigns a belief of 0 to an event has no confidence that the event will occur; conversely, assigning a belief of 1 implies that the individual is absolutely certain of an event occurring. Beliefs between 0 and 1 allow for weightings of other outcomes. This definition agrees with the probability of a plane accident example, for having observed the frequency of plane accidents, an individual's belief should be equal to that frequency, excluding any outside information. Similarly, under this definition of probability being equal to beliefs, it is meaningful to speak about probabilities (beliefs) of presidential election outcomes: how confident are you candidate *A* will win?

Notice in the paragraph above, **I assigned the belief (probability) measure to an *individual*, not to Nature**. This is very interesting, as this definition leaves room for conflicting beliefs between individuals. Again, this is appropriate for what naturally occurs: different individuals have different beliefs of events occurring, because they possess different *information* about the world. The existence of different beliefs does not imply that anyone is wrong. Consider the following examples demonstrating the relationship between individual beliefs and probabilities:

- I flip a coin, and we both guess the result. We would both agree, assuming the coin is fair, that the probability of Heads is 1/2. Assume, then, that I peek at the coin. Now I know for certain what the result is: I assign probability 1.0 to either Heads or Tails (whichever it is). Now what is *your* belief that the coin is Heads? My knowledge of the outcome has not changed the coin's results. Thus we assign different probabilities to the result.

-  Your code either has a bug in it or not, but we do not know for certain which is true, though we have a belief about the presence or absence of a bug.

-  A medical patient is exhibiting symptoms $x$, $y$ and $z$. There are a number of diseases that could be causing all of them, but only a single disease is present. A doctor has beliefs about which disease, but a second doctor may have slightly different beliefs.

This philosophy of treating beliefs as probability is natural to humans. We employ it constantly as we interact with the world and only see partial truths, but gather evidence to form beliefs. Alternatively, you have to be *trained* to think like a frequentist.

To align ourselves with traditional probability notation, **we denote our belief about event $A$ as $P(A)$. We call this quantity the *prior probability***.

John Maynard Keynes, a great economist and thinker, said "When the facts change, I change my mind. What do you do, sir?" This quote reflects the way a Bayesian updates his or her beliefs after seeing evidence. Even — especially — if the evidence is counter to what was initially believed, the evidence cannot be ignored. We denote our updated belief as $P(A |X )$, interpreted as the probability of $A$ given the evidence $X$. We call the updated belief the *posterior probability* so as to contrast it with the prior probability. For example, consider the posterior probabilities (read: posterior beliefs) of the above examples, after observing some evidence $X$:

1\. $P(A): \;\;$ the coin has a 50 percent chance of being Heads. $P(A | X):\;\;$ You look at the coin, observe a Heads has landed, denote this information $X$, and trivially assign probability 1.0 to Heads and 0.0 to Tails.

2\.   $P(A): \;\;$  This big, complex code likely has a bug in it. $P(A | X): \;\;$ The code passed all $X$ tests; there still might be a bug, but its presence is less likely now.

3\.  $P(A):\;\;$ The patient could have any number of diseases. $P(A | X):\;\;$ Performing a blood test generated evidence $X$, ruling out some of the possible diseases from consideration.

**It's clear that in each example we did not completely discard the prior belief after seeing new evidence $X$, but we *re-weighted the prior* to incorporate the new evidence** (i.e. we put more weight, or confidence, on some beliefs versus others).

By introducing prior uncertainty about events, we are already admitting that any guess we make is potentially very wrong. After observing data, evidence, or other information, we update our beliefs, and our guess becomes *less wrong*. This is the alternative side of the prediction coin, where typically we try to be *more right*.

### Bayesian Inference in Practice

 If frequentist and Bayesian inference were programming functions, with inputs being statistical problems, then the two would be different in what they return to the user. The frequentist inference function would return a number, representing an estimate (typically a summary statistic like the sample average etc.), whereas the Bayesian function would return *probabilities*.

For example, in our debugging problem above, calling the frequentist function with the argument "My code passed all $X$ tests; is my code bug-free?" would return a *YES*. On the other hand, asking our Bayesian function "Often my code has bugs. My code passed all $X$ tests; is my code bug-free?" would return something very different: probabilities of *YES* and *NO*. The function might return:

>    *YES*, with probability 0.8; *NO*, with probability 0.2

This is very different from the answer the frequentist function returned. Notice that the Bayesian function accepted an additional argument:  *"Often my code has bugs"*. This parameter is the *prior*. By including the prior parameter, we are telling the Bayesian function to include our belief about the situation. Technically this parameter in the Bayesian function is optional, but we will see excluding it has its own consequences.

#### Incorporating evidence

As we acquire more and more instances of evidence, our prior belief is *washed out* by the new evidence. This is to be expected. For example, if your prior belief is something ridiculous, like "I expect the sun to explode today", and each day you are proved wrong, you would hope that any inference would correct you, or at least align your beliefs better. Bayesian inference will correct this belief.

Denote $N$ as the number of instances of evidence we possess. As we gather an *infinite* amount of evidence, say as $N \rightarrow \infty$, our Bayesian results (often) align with frequentist results. Hence for large $N$, statistical inference is more or less objective. On the other hand, **for small $N$, inference is much more *unstable*: frequentist estimates have more variance and larger confidence intervals. This is where Bayesian analysis excels. By introducing a prior, and returning probabilities (instead of a scalar estimate), we *preserve the uncertainty*** that reflects the instability of statistical inference of a small $N$ dataset.

One may think that for large $N$, one can be indifferent between the two techniques since they offer similar inference, and might lean towards the computationally-simpler, frequentist methods. An individual in this position should consider the following quote by Andrew Gelman (2005)[1], before making such a decision:

> Sample sizes are never large. If $N$ is too small to get a sufficiently-precise estimate, you need to get more data (or make more assumptions). But once $N$ is "large enough," you can start subdividing the data to learn more (for example, in a public opinion poll, once you have a good estimate for the entire country, you can estimate among men and women, northerners and southerners, different age groups, etc.). $N$ is never enough because if it were "enough" you'd already be on to the next problem for which you need more data.

### Are frequentist methods incorrect then?

**No.**

Frequentist methods are still useful or state-of-the-art in many areas. Tools such as least squares linear regression, LASSO regression, and expectation-maximization algorithms are all powerful and fast. Bayesian methods complement these techniques by solving problems that these approaches cannot, or by illuminating the underlying system with more flexible modeling.

#### A note on *Big Data*
Paradoxically, big data's predictive analytic problems are actually solved by relatively simple algorithms [2][4]. Thus we can argue that big data's prediction difficulty does not lie in the algorithm used, but instead on the computational difficulties of storage and execution on big data. (One should also consider Gelman's quote from above and ask "Do I really have big data?")

The much more difficult analytic problems involve *medium data* and, especially troublesome, *really small data*. Using a similar argument as  Gelman's above, if big data problems are *big enough* to be readily solved, then we should be more interested in the *not-quite-big enough* datasets.

### Our Bayesian framework

We are interested in beliefs, which can be interpreted as probabilities by thinking Bayesian. We have a *prior* belief in event $A$, beliefs formed by previous information, e.g., our prior belief about bugs being in our code before performing tests.

Secondly, we observe our evidence. To continue our buggy-code example: if our code passes $X$ tests, we want to update our belief to incorporate this. We call this new belief the *posterior* probability. Updating our belief is done via the following equation, known as Bayes' Theorem, after its discoverer Thomas Bayes:

\begin{align}
 P( A | X ) = & \frac{ P(X | A) P(A) } {P(X) } \\\\[5pt]
& \propto P(X | A) P(A)\;\; (\propto \text{is proportional to })
\end{align}

The above formula is not unique to Bayesian inference: it is a mathematical fact with uses outside Bayesian inference. Bayesian inference merely uses it to connect prior probabilities $P(A)$ with an updated posterior probabilities $P(A | X )$.

##### Example: Mandatory coin-flip example

Every statistics text must contain a coin-flipping example, I'll use it here to get it out of the way. Suppose, naively, that you are unsure about the probability of heads in a coin flip (spoiler alert: it's 50%). You believe there is some true underlying ratio, call it $p$, but have no prior opinion on what $p$ might be.

We begin to flip a coin, and record the observations: either $H$ or $T$. This is our observed data. An interesting question to ask is how our inference changes as we observe more and more data? More specifically, what do our posterior probabilities look like when we have little data, versus when we have lots of data.

Below we plot a sequence of updating posterior probabilities as we observe increasing amounts of data (coin flips).

```python
"""
The book uses a custom matplotlibrc file, which provides the unique styles for
matplotlib plots. If executing this book, and you wish to use the book's
styling, provided are two options:
    1. Overwrite your own matplotlibrc file with the rc-file provided in the
       book's styles/ dir. See http://matplotlib.org/users/customizing.html
    2. Also in the styles is  bmh_matplotlibrc.json file. This can be used to
       update the styles in only this notebook. Try running the following code:

        import json
        s = json.load(open("../styles/bmh_matplotlibrc.json"))
        matplotlib.rcParams.update(s)

"""

# The code below can be passed over, as it is currently not important, plus it
# uses advanced topics we have not covered yet. LOOK AT PICTURE, MICHAEL!
%matplotlib inline
from IPython.core.pylabtools import figsize
import numpy as np
from matplotlib import pyplot as plt
figsize(11, 9)

import scipy.stats as stats

dist = stats.beta
n_trials = [0, 1, 2, 3, 4, 5, 8, 15, 50, 500]
data = stats.bernoulli.rvs(0.5, size=n_trials[-1])
x = np.linspace(0, 1, 100)

# For the already prepared, I'm using Binomial's conj. prior.
for k, N in enumerate(n_trials):
    sx = plt.subplot(len(n_trials)/2, 2, k+1)
    plt.xlabel("$p$, probability of heads") \
        if k in [0, len(n_trials)-1] else None
    plt.setp(sx.get_yticklabels(), visible=False)
    heads = data[:N].sum()
    y = dist.pdf(x, 1 + heads, 1 + N - heads)
    plt.plot(x, y, label="observe %d tosses,\n %d heads" % (N, heads))
    plt.fill_between(x, 0, y, color="#348ABD", alpha=0.4)
    plt.vlines(0.5, 0, 4, color="k", linestyles="--", lw=1)

    leg = plt.legend()
    leg.get_frame().set_alpha(0.4)
    plt.autoscale(tight=True)

plt.suptitle("Bayesian updating of posterior probabilities",
             y=1.02,
             fontsize=14)

plt.tight_layout()
```

The posterior probabilities are represented by the curves, and **our uncertainty is proportional to the width of the curve**. As the plot above shows, as we start to observe data our posterior probabilities start to shift and move around. Eventually, as we observe more and more data (coin-flips), our probabilities will tighten closer and closer around the true value of $p=0.5$ (marked by a dashed line).

Notice that the plots are not always *peaked* at 0.5. There is no reason it should be: recall we assumed we did not have a prior opinion of what $p$ is. In fact, if we observe quite extreme data, say 8 flips and only 1 observed heads, our distribution would look very biased *away* from lumping around 0.5 (with no prior opinion, how confident would you feel betting on a fair coin after observing 8 tails and 1 head?). As more data accumulates, we would see more and more probability being assigned at $p=0.5$, though never all of it.

The next example is a simple demonstration of the mathematics of Bayesian inference.

##### Example: Bug, or just sweet, unintended feature?

Let $A$ denote the event that our code has **no bugs** in it. Let $X$ denote the event that the code passes all debugging tests. For now, we will leave the prior probability of no bugs as a variable, i.e. $P(A) = p$.

We are interested in $P(A|X)$, i.e. the probability of no bugs, given our debugging tests $X$. To use the formula above, we need to compute some quantities.

What is $P(X | A)$, i.e., the probability that the code passes $X$ tests *given* there are no bugs? Well, it is equal to 1, for a code with no bugs will pass all tests.

$P(X)$ is a little bit trickier: The event $X$ can be divided into two possibilities, event $X$ occurring even though our code *indeed has* bugs (denoted $\sim A\;$, spoken *not $A$*), or event $X$ without bugs ($A$). $P(X)$ can be represented as:

\begin{align}
P(X ) & = P(X \text{ and } A) + P(X \text{ and } \sim A) \\\\[5pt]
 & = P(X|A)P(A) + P(X | \sim A)P(\sim A)\\\\[5pt]
& = P(X|A)p + P(X | \sim A)(1-p)
\end{align}

We have already computed $P(X|A)$ above. On the other hand, $P(X | \sim A)$ is subjective: our code can pass tests but still have a bug in it, though the probability there is a bug present is reduced. Note this is dependent on the number of tests performed, the degree of complication in the tests, etc. Let's be conservative and assign $P(X|\sim A) = 0.5$. Then

\begin{align}
P(A | X) & = \frac{1\cdot p}{ 1\cdot p +0.5 (1-p) } \\\\
& = \frac{ 2 p}{1+p}
\end{align}
This is the posterior probability. What does it look like as a function of our prior, $p \in [0,1]$?

```python
figsize(12.5, 4)
p = np.linspace(0, 1, 50)
plt.plot(p, 2*p/(1+p), color="#348ABD", lw=3)
#plt.fill_between(p, 2*p/(1+p), alpha=.5, facecolor=["#A60628"])
plt.scatter(0.2, 2*(0.2)/1.2, s=140, c="#348ABD")
plt.xlim(0, 1)
plt.ylim(0, 1)
plt.xlabel("Prior, $P(A) = p$")
plt.ylabel("Posterior, $P(A|X)$, with $P(A) = p$")
plt.title("Are there bugs in my code?");
```

We can see the biggest gains if we observe the $X$ tests passed when the prior probability, $p$, is low. Let's settle on a specific value for the prior. I'm a strong programmer (I think), so I'm going to give myself a realistic prior of 0.20, that is, there is a 20% chance that I write code bug-free. To be more realistic, this prior should be a function of how complicated and large the code is, but let's pin it at 0.20. Then my updated belief that my code is bug-free is 0.33.

Recall that the prior is a probability: $p$ is the prior probability that there *are no bugs*, so $1-p$ is the prior probability that there *are bugs*.

Similarly, our posterior is also a probability, with $P(A | X)$ the probability there is no bug *given we saw all tests pass*, hence $1-P(A|X)$ is the probability there is a bug *given all tests passed*. What does our posterior probability look like? Below is a chart of both the prior and the posterior probabilities.

```python
figsize(12.5, 4)
colours = ["#348ABD", "#A60628"]

prior = [0.20, 0.80]
posterior = [1./3, 2./3]
plt.bar([0, .7], prior, alpha=0.70, width=0.25,
        color=colours[0], label="prior distribution",
        lw="3", edgecolor=colours[0])

plt.bar([0+0.25, .7+0.25], posterior, alpha=0.7,
        width=0.25, color=colours[1],
        label="posterior distribution",
        lw="3", edgecolor=colours[1])

plt.xticks([0.20, .95], ["Bugs Absent", "Bugs Present"])
plt.title("Prior and Posterior probability of bugs present")
plt.ylabel("Probability")
plt.legend(loc="upper left");
```

Notice that after we observed $X$ occur, the probability of bugs being absent increased. By increasing the number of tests, we can approach confidence (probability 1) that there are no bugs present.

This was a very simple example of Bayesian inference and Bayes rule. Unfortunately, the mathematics necessary to perform more complicated Bayesian inference only becomes more difficult, except for artificially constructed cases. We will later see that this type of mathematical analysis is actually unnecessary. First we must broaden our modeling tools. The next section deals with *probability distributions*. If you are already familiar, feel free to skip (or at least skim), but for the less familiar the next section is essential.

_______

## Probability Distributions

**Let's quickly recall what a probability distribution is:** Let $Z$ be some random variable. Then associated with $Z$ is a *probability **distribution** function* that assigns probabilities to the different outcomes $Z$ can take. Graphically, a probability distribution is a curve where the probability of an outcome is proportional to the height of the curve. You can see examples in the first figure of this chapter.

We can divide random variables into three classifications:

-   **$Z$ is discrete**: Discrete random variables may only assume values on a specified list. Things like populations, movie ratings, and number of votes are all discrete random variables. Discrete random variables become more clear when we contrast them with...

-   **$Z$ is continuous**: Continuous random variable can take on arbitrarily exact values. For example, temperature, speed, time, color are all modeled as continuous variables because you can progressively make the values more and more precise.

- **$Z$ is mixed**: Mixed random variables assign probabilities to both discrete and continuous random variables, i.e. it is a combination of the above two categories.

### Discrete Case
If $Z$ is discrete, then its distribution is called a *probability **mass** function*, which measures the probability $Z$ takes on the value $k$, denoted $P(Z=k)$. Note that the probability mass function completely describes the random variable $Z$, that is, if we know the mass function, we know how $Z$ should behave. There are popular probability mass functions that consistently appear: we will introduce them as needed, but let's introduce the first very useful probability mass function. We say $Z$ is *Poisson*-distributed if:

$$P(Z = k) =\frac{ \lambda^k e^{-\lambda} }{k!}, \; \; k=0,1,2, \dots $$

$\lambda$ is called a parameter of the distribution, and it controls the distribution's shape. For the Poisson distribution, $\lambda$ can be any positive number. By increasing $\lambda$, we add more probability to larger values, and conversely by decreasing $\lambda$ we add more probability to smaller values. One can describe $\lambda$ as the *intensity* of the Poisson distribution.

Unlike $\lambda$, which can be any positive number, the value $k$ in the above formula must be a non-negative integer, i.e., $k$ must take on values 0,1,2, and so on. This is very important, because if you wanted to model a population you could not make sense of populations with 4.25 or 5.612 members.

If a random variable $Z$ has a Poisson mass distribution, we denote this by writing

$$Z \sim \text{Poi}(\lambda) $$

One useful property of the Poisson distribution is that its expected value is equal to its parameter, i.e.:

$$E\large[ \;Z\; | \; \lambda \;\large] = \lambda $$

**We will use this property often, so it's useful to remember**. Below, we plot the probability mass distribution for different $\lambda$ values. The first thing to notice is that by increasing $\lambda$, we add more probability of larger values occurring. Second, notice that although the graph ends at 15, the distributions do not. They assign positive probability to every non-negative integer.

```python
figsize(12.5, 4)

import scipy.stats as stats
a = np.arange(16)
poi = stats.poisson
lambda_ = [1.5, 4.25]
colours = ["#348ABD", "#A60628"]

plt.bar(a, poi.pmf(a, lambda_[0]), color=colours[0],
        label="$\lambda = %.1f$" % lambda_[0], alpha=0.60,
        edgecolor=colours[0], lw="3")

plt.bar(a, poi.pmf(a, lambda_[1]), color=colours[1],
        label="$\lambda = %.1f$" % lambda_[1], alpha=0.60,
        edgecolor=colours[1], lw="3")

plt.xticks(a + 0.4, a)
plt.legend()
plt.ylabel("probability of $k$")
plt.xlabel("$k$")
plt.title("Probability mass function of a Poisson random variable; differing \
$\lambda$ values");
```

### Continuous Case
Instead of a probability mass function, a continuous random variable has a *probability **density** function*. This might seem like unnecessary nomenclature, but the density function and the mass function are very different creatures. An example of continuous random variable is a random variable with *exponential density*. The density function for an exponential random variable looks like this:

$$f_Z(z | \lambda) = \lambda e^{-\lambda z }, \;\; z\ge 0$$

Like a Poisson random variable, an exponential random variable can take on only non-negative values. But unlike a Poisson variable, the exponential can take on *any* non-negative values, including non-integral values such as 4.25 or 5.612401. This property makes it a poor choice for count data, which must be an integer, but a great choice for time data, temperature data (measured in Kelvins, of course), or any other precise *and positive* variable. The graph below shows two probability density functions with different $\lambda$ values.

When a random variable $Z$ has an exponential distribution with parameter $\lambda$, we say *$Z$ is exponential* and write

$$Z \sim \text{Exp}(\lambda)$$

Given a specific $\lambda$, the expected value of an exponential random variable is equal to the inverse of $\lambda$, that is:

$$E[\; Z \;|\; \lambda \;] = \frac{1}{\lambda}$$

```python
a = np.linspace(0, 4, 100)
expo = stats.expon
lambda_ = [0.5, 1]

for l, c in zip(lambda_, colours):
    plt.plot(a, expo.pdf(a, scale=1./l), lw=3,
             color=c, label="$\lambda = %.1f$" % l)
    plt.fill_between(a, expo.pdf(a, scale=1./l), color=c, alpha=.33)

plt.legend()
plt.ylabel("PDF at $z$")
plt.xlabel("$z$")
plt.ylim(0,1.2)
plt.title("Probability density function of an Exponential random variable;\
 differing $\lambda$");
```

### But what is $\lambda \;$?

**This question is what motivates statistics**. In the real world, $\lambda$ is hidden from us. We see only $Z$, and must go backwards to try and determine $\lambda$. The problem is difficult because there is no one-to-one mapping from $Z$ to $\lambda$. Many different methods have been created to solve the problem of estimating $\lambda$, but since $\lambda$ is never actually observed, no one can say for certain which method is best!

Bayesian inference is concerned with *beliefs* about what $\lambda$ might be. Rather than try to guess $\lambda$ exactly, we can only talk about what $\lambda$ is likely to be by assigning a probability distribution to $\lambda$.

This might seem odd at first. After all, $\lambda$ is fixed; it is not (necessarily) random! How can we assign probabilities to values of a non-random variable? Ah, we have fallen for our old, frequentist way of thinking. Recall that under Bayesian philosophy, we *can* assign probabilities if we interpret them as beliefs. And it is entirely acceptable to have *beliefs* about the parameter $\lambda$.

##### Example: Inferring behaviour from text-message data

Let's try to model a more interesting example, one that concerns the rate at which a user sends and receives text messages:

>  You are given a series of daily text-message counts from a user of your system. The data, plotted over time, appears in the chart below. You are curious to know if the user's text-messaging habits have changed over time, either gradually or suddenly. How can you model this? (This is in fact my own text-message data. Judge my popularity as you wish.)

```python
figsize(12.5, 3.5)
count_data = np.loadtxt("data/txtdata.csv")
n_count_data = len(count_data)
plt.bar(np.arange(n_count_data), count_data, color="#348ABD")
plt.xlabel("Time (days)")
plt.ylabel("count of text-msgs received")
plt.title("Did the user's texting habits change over time?")
plt.xlim(0, n_count_data);
```

Before we start modeling, see what you can figure out just by looking at the chart above. Would you say there was a change in behaviour during this time period?

How can we start to model this? Well, as we have conveniently already seen, a Poisson random variable is a very appropriate model for this type of *count* data. Denoting day $i$'s text-message count by $C_i$,

$$ C_i \sim \text{Poisson}(\lambda)  $$

We are not sure what the value of the $\lambda$ parameter really is, however. Looking at the chart above, it appears that the rate might become higher late in the observation period, which is equivalent to saying that $\lambda$ increases at some point during the observations. (Recall that a higher value of $\lambda$ assigns more probability to larger outcomes. That is, there is a higher probability of many text messages having been sent on a given day.)

How can we represent this observation mathematically? Let's assume that on some day during the observation period (call it $\tau$), the parameter $\lambda$ suddenly jumps to a higher value. So we really have two $\lambda$ parameters: one for the period before $\tau$, and one for the rest of the observation period. In the literature, a sudden transition like this would be called a *switchpoint*:

$$
\lambda =
\begin{cases}
\lambda_1  & \text{if } t \lt \tau \cr
\lambda_2 & \text{if } t \ge \tau
\end{cases}
$$

If, in reality, no sudden change occurred and indeed $\lambda_1 = \lambda_2$, then the $\lambda$s posterior distributions should look about equal.

We are interested in inferring the unknown $\lambda$s. To use Bayesian inference, we need to assign prior probabilities to the different possible values of $\lambda$. What would be good prior probability distributions for $\lambda_1$ and $\lambda_2$? Recall that $\lambda$ can be any positive number. As we saw earlier, the *exponential* distribution provides a continuous density function for positive numbers, so it might be a good choice for modeling $\lambda_i$. But recall that the exponential distribution takes a parameter of its own, so we'll need to include that parameter in our model. Let's call that parameter $\alpha$.

\begin{align}
&\lambda_1 \sim \text{Exp}( \alpha ) \\\
&\lambda_2 \sim \text{Exp}( \alpha )
\end{align}

$\alpha$ is called a *hyper-parameter* or *parent variable*. In literal terms, it is a parameter that influences other parameters. Our initial guess at $\alpha$ does not influence the model too strongly, so we have some flexibility in our choice.  **A good rule of thumb is to set the exponential parameter equal to the inverse of the average of the count data**. Since we're modeling $\lambda$ using an exponential distribution, we can use the expected value identity shown earlier to get:

$$\frac{1}{N}\sum_{i=0}^N \;C_i \approx E[\; \lambda \; |\; \alpha ] = \frac{1}{\alpha}$$

An alternative, and something I encourage the reader to try, would be to have two priors: one for each $\lambda_i$. Creating two exponential distributions with different $\alpha$ values reflects our prior belief that the rate changed at some point during the observations.

What about $\tau$? Because of the noisiness of the data, it's difficult to pick out a priori when $\tau$ might have occurred. Instead, we can assign a *uniform prior belief* to every possible day. This is equivalent to saying

\begin{align}
& \tau \sim \text{DiscreteUniform(1,70) }\\\\
& \Rightarrow P( \tau = k ) = \frac{1}{70}
\end{align}

So after all this, what does our overall prior distribution for the unknown variables look like? Frankly, *it doesn't matter*. What we should understand is that it's an ugly, complicated mess involving symbols only a mathematician could love. And things will only get uglier the more complicated our models become. Regardless, all we really care about is the posterior distribution.

We next turn to PyMC3, a Python library for performing Bayesian analysis that is undaunted by the mathematical monster we have created.

Introducing our first hammer: PyMC3
-----

PyMC3 is a Python library for programming Bayesian analysis [3]. It is a fast, well-maintained library. The only unfortunate part is that its documentation is lacking in certain areas, especially those that bridge the gap between beginner and hacker. One of this book's main goals is to solve that problem, and also to demonstrate why PyMC3 is so cool.

We will model the problem above using PyMC3. This type of programming is called *probabilistic programming*, an unfortunate misnomer that invokes ideas of randomly-generated code and has likely confused and frightened users away from this field. The code is not random; it is probabilistic in the sense that we create probability models using programming variables as the model's components. Model components are first-class primitives within the PyMC3 framework.

B. Cronin [5] has a very motivating description of probabilistic programming:

>   Another way of thinking about this: unlike a traditional program, which only runs in the forward directions, a probabilistic program is run in both the forward and backward direction. It runs forward to compute the consequences of the assumptions it contains about the world (i.e., the model space it represents), but it also runs backward from the data to constrain the possible explanations. In practice, many probabilistic programming systems will cleverly interleave these forward and backward operations to efficiently home in on the best explanations.

Because of the confusion engendered by the term *probabilistic programming*, I'll refrain from using it. Instead, I'll simply say *programming*, since that's what it really is.

PyMC3 code is easy to read. The only novel thing should be the syntax. Simply remember that we are representing the model's components ($\tau, \lambda_1, \lambda_2$ ) as variables.

```python
import pymc3 as pm
import theano.tensor as tt

with pm.Model() as model:
    alpha = 1.0/count_data.mean()  # Recall count_data is the
                                   # variable that holds our txt counts
    lambda_1 = pm.Exponential("lambda_1", alpha)
    lambda_2 = pm.Exponential("lambda_2", alpha)

    tau = pm.DiscreteUniform("tau", lower=0, upper=n_count_data - 1)
```

In the code above, we create the PyMC3 variables corresponding to $\lambda_1$ and $\lambda_2$. We assign them to PyMC3's *stochastic variables*, so-called because they are treated by the back end as random number generators.

```python
with model:
    idx = np.arange(n_count_data) # Index
    lambda_ = pm.math.switch(tau > idx, lambda_1, lambda_2)
```

This code creates a new function `lambda_`, but really we can think of it as a random variable: the random variable $\lambda$ from above. The `switch()` function assigns `lambda_1` or `lambda_2` as the value of `lambda_`, depending on what side of `tau` we are on. The values of `lambda_` up until `tau` are `lambda_1` and the values afterwards are `lambda_2`.

Note that because `lambda_1`, `lambda_2` and `tau` are random, `lambda_` will be random. We are **not** fixing any variables yet.

```python
with model:
    observation = pm.Poisson("obs", lambda_, observed=count_data)
```

The variable `observation` combines our data, `count_data`, with our proposed data-generation scheme, given by the variable `lambda_`, through the `observed` keyword.

The code below will be explained in Chapter 3, but I show it here so you can see where our results come from. One can think of it as a *learning* step. The machinery being employed is called *Markov Chain Monte Carlo* (MCMC), which I also delay explaining until Chapter 3. This technique returns thousands of random variables from the posterior distributions of $\lambda_1, \lambda_2$ and $\tau$. We can plot a histogram of the random variables to see what the posterior distributions look like. Below, we collect the samples (called *traces* in the MCMC literature) into histograms.

```python
### Mysterious code to be explained in Chapter 3.
with model:
    step = pm.Metropolis()
    trace = pm.sample(10000, tune=5000,step=step)
```

    Multiprocess sampling (4 chains in 4 jobs)
    CompoundStep
    >Metropolis: [tau]
    >Metropolis: [lambda_2]
    >Metropolis: [lambda_1]
    Sampling 4 chains: 100%|██████████| 60000/60000 [00:13<00:00, 4577.12draws/s]
    The number of effective samples is smaller than 25% for some parameters.

```python
lambda_1_samples = trace['lambda_1']
lambda_2_samples = trace['lambda_2']
tau_samples = trace['tau']
```

```python
figsize(12.5, 10)
#histogram of the samples:

ax = plt.subplot(311)
ax.set_autoscaley_on(False)

plt.hist(lambda_1_samples, histtype='stepfilled', bins=30, alpha=0.85,
         label="posterior of $\lambda_1$", color="#A60628", density=True)
plt.legend(loc="upper left")
plt.title(r"""Posterior distributions of the variables
    $\lambda_1,\;\lambda_2,\;\tau$""")
plt.xlim([15, 30])
plt.xlabel("$\lambda_1$ value")

ax = plt.subplot(312)
ax.set_autoscaley_on(False)
plt.hist(lambda_2_samples, histtype='stepfilled', bins=30, alpha=0.85,
         label="posterior of $\lambda_2$", color="#7A68A6", density=True)
plt.legend(loc="upper left")
plt.xlim([15, 30])
plt.xlabel("$\lambda_2$ value")

plt.subplot(313)
w = 1.0 / tau_samples.shape[0] * np.ones_like(tau_samples)
plt.hist(tau_samples, bins=n_count_data, alpha=1,
         label=r"posterior of $\tau$",
         color="#467821", weights=w, rwidth=2.)
plt.xticks(np.arange(n_count_data))

plt.legend(loc="upper left")
plt.ylim([0, .75])
plt.xlim([35, len(count_data)-20])
plt.xlabel(r"$\tau$ (in days)")
plt.ylabel("probability");
```

### Interpretation

Recall that Bayesian methodology returns a *distribution*. Hence we now have distributions to describe the unknown $\lambda$s and $\tau$. What have we gained? Immediately, we can see the uncertainty in our estimates: the wider the distribution, the less certain our posterior belief should be. We can also see what the plausible values for the parameters are: $\lambda_1$ is around 18 and $\lambda_2$ is around 23. The posterior distributions of the two $\lambda$s are clearly distinct, indicating that it is indeed likely that there was a change in the user's text-message behaviour.

What other observations can you make? If you look at the original data again, do these results seem reasonable?

**Notice also that the posterior distributions for the $\lambda$s do not look like exponential distributions, even though our priors for these variables were exponential**. In fact, the posterior distributions are not really of any form that we recognize from the original model. But that's OK! This is one of the benefits of taking a computational point of view. If we had instead done this analysis using mathematical approaches, we would have been stuck with an analytically intractable (and messy) distribution. Our use of a computational approach makes us indifferent to mathematical tractability.

Our analysis also returned a distribution for $\tau$. Its posterior distribution looks a little different from the other two because it is a discrete random variable, so it doesn't assign probabilities to intervals. We can see that near day 45, there was a 50% chance that the user's behaviour changed. Had no change occurred, or had the change been gradual over time, the posterior distribution of $\tau$ would have been more spread out, reflecting that many days were plausible candidates for $\tau$. By contrast, in the actual results we see that only three or four days make any sense as potential transition points.

### Why would I want samples from the posterior, anyways?

We will deal with this question for the remainder of the book, and it is an understatement to say that it will lead us to some amazing results. For now, let's end this chapter with one more example.

We'll use the posterior samples to answer the following question: what is the expected number of texts at day $t, \; 0 \le t \le 70$ ? Recall that the expected value of a Poisson variable is equal to its parameter $\lambda$. Therefore, the question is equivalent to *what is the expected value of $\lambda$ at time $t$*?

In the code below, let $i$ index samples from the posterior distributions. Given a day $t$, we average over all possible $\lambda_i$ for that day $t$, using $\lambda_i = \lambda_{1,i}$ if $t \lt \tau_i$ (that is, if the behaviour change has not yet occurred), else we use $\lambda_i = \lambda_{2,i}$.

```python
figsize(12.5, 5)
# tau_samples, lambda_1_samples, lambda_2_samples contain
# N samples from the corresponding posterior distribution
N = tau_samples.shape[0]
expected_texts_per_day = np.zeros(n_count_data)
for day in range(0, n_count_data):
    # ix is a bool index of all tau samples corresponding to
    # the switchpoint occurring prior to value of 'day'
    ix = day < tau_samples
    # Each posterior sample corresponds to a value for tau.
    # for each day, that value of tau indicates whether we're "before"
    # (in the lambda1 "regime") or
    #  "after" (in the lambda2 "regime") the switchpoint.
    # by taking the posterior sample of lambda1/2 accordingly, we can average
    # over all samples to get an expected value for lambda on that day.
    # As explained, the "message count" random variable is Poisson distributed,
    # and therefore lambda (the poisson parameter) is the expected value of
    # "message count".
    expected_texts_per_day[day] = (lambda_1_samples[ix].sum()
                                   + lambda_2_samples[~ix].sum()) / N

plt.plot(range(n_count_data), expected_texts_per_day, lw=4, color="#E24A33",
         label="expected number of text-messages received")
plt.xlim(0, n_count_data)
plt.xlabel("Day")
plt.ylabel("Expected # text-messages")
plt.title("Expected number of text-messages received")
plt.ylim(0, 60)
plt.bar(np.arange(len(count_data)), count_data, color="#348ABD", alpha=0.65,
        label="observed texts per day")

plt.legend(loc="upper left");
```

Our analysis shows strong support for believing the user's behavior did change ($\lambda_1$ would have been close in value to $\lambda_2$ had this not been true), and that the change was sudden rather than gradual (as demonstrated by $\tau$'s strongly peaked posterior distribution). We can speculate what might have caused this: a cheaper text-message rate, a recent weather-to-text subscription, or perhaps a new relationship. (In fact, the 45th day corresponds to Christmas, and I moved away to Toronto the next month, leaving a girlfriend behind.)

##### Exercises

1\.  Using `lambda_1_samples` and `lambda_2_samples`, what is the mean of the posterior distributions of $\lambda_1$ and $\lambda_2$?

```python
#type your code here.
lambda_1_samples.mean(), lambda_2_samples.mean()
```

    (17.760179819690123, 22.710926545663497)

2\.  What is the expected percentage increase in text-message rates? `hint:` compute the mean of `lambda_1_samples/lambda_2_samples`. Note that this quantity is very different from `lambda_1_samples.mean()/lambda_2_samples.mean()`.

```python
#type your code here.
(lambda_1_samples/lambda_2_samples).mean()
```

    0.7832074809697906

```python
lambda_1_samples.mean()/lambda_2_samples.mean()
```

    0.7820103589336522

PS to myself: Note that the results returned are different !

3\. What is the mean of $\lambda_1$ **given** that we know $\tau$ is less than 45.  That is, suppose we have been given new information that the change in behaviour occurred prior to day 45. What is the expected value of $\lambda_1$ now? (You do not need to redo the PyMC3 part. Just consider all instances where `tau_samples < 45`.)

```python
#type your code here.
idx = tau_samples < 45
lambda_1_samples[idx].mean()
```

    17.757353124712516

### References

-  [1] Gelman, Andrew. N.p.. Web. 22 Jan 2013. [N is never large enough](http://andrewgelman.com/2005/07/31/n_is_never_larg).
-  [2] Norvig, Peter. 2009. [The Unreasonable Effectiveness of Data](http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35179.pdf).
- [3] Salvatier, J, Wiecki TV, and Fonnesbeck C. (2016) Probabilistic programming in Python using PyMC3. *PeerJ Computer Science* 2:e55 <https://doi.org/10.7717/peerj-cs.55>
- [4] Jimmy Lin and Alek Kolcz. Large-Scale Machine Learning at Twitter. Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data (SIGMOD 2012), pages 793-804, May 2012, Scottsdale, Arizona.
- [5] Cronin, Beau. "Why Probabilistic Programming Matters." 24 Mar 2013. Google, Online Posting to Google . Web. 24 Mar. 2013. <https://plus.google.com/u/0/107971134877020469960/posts/KpeRdJKR6Z1>.

```python
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunss.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-weight: bold;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsx.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-style: oblique;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsi.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-weight: bold;
        font-style: oblique;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunso.otf');
    }
    div.cell{
        width:800px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: Helvetica, serif;
    }
    h4{
        margin-top:12px;
        margin-bottom: 3px;
       }
    div.text_cell_render{
        font-family: Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 145%;
        font-size: 130%;
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro", source-code-pro,Consolas, monospace;
    }
    .prompt{
        display: None;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 22pt;
        color: #4057A1;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>


```python
%run common-imports.ipynb
```

<!-- Many of the styles here are inspired by:
    https://towardsdatascience.com/10-practical-tips-you-need-to-know-to-personalize-jupyter-notebook-fbd202777e20

    On the author's local machine, these exist in the custom.css file. However, in order to keep uniform look and feel,
    and at the request of participants, I have added it to this common import-file here.

    -->

<link href="https://fonts.googleapis.com/css?family=Lora:400,700|Montserrat:300" rel="stylesheet">

<link href="https://fonts.googleapis.com/css2?family=Crimson+Pro&family=Literata&display=swap" rel="stylesheet">
<style>

#ipython_notebook::before{
 content:"Workshop: Comprehensive Introduction to Data Science";
        color: white;
        font-weight: bold;
        text-transform: uppercase;
        font-family: 'Lora',serif;
        font-size:20pt;
        margin-bottom:15px;

}
body > #header {
    background: maroon;
        opacity: 0.7;
}

.navbar-default .navbar-nav > li > a, #kernel_indicator {
    color: white;
    transition: all 0.25s;
    font-size:12pt;
    font-family: sans;
    /*font-weight:bold;*/
}
.navbar-default {
    padding-left:100px;
    background: none;
    border: none;
}

body > menubar-container {
    background-color: wheat;
}
#ipython_notebook img{
    display:block;

    background: url("https://www.supportvectors.com/wp-content/uploads/2016/03/logo-poster-smaller.png") no-repeat;
    background-size: contain;

    padding-left: 600px;
    padding-right: 100px;

    -moz-box-sizing: border-box;
    box-sizing: border-box;
}

body {
 #font-family:  'Literata', serif;
    font-family:'Lora', san-serif;
    text-align: justify;
    font-weight: 400;
    font-size: 14pt;
}

iframe{
    width:100%;
    min-height:600px;
}

h1, h2, h3, h4, h5, h6 {
# font-family: 'Montserrat', sans-serif;
 font-family:'Lora', serif;
 font-weight: 300;
 text-transform: uppercase;
 color: #EC7063 ;
}

h2 {
    color: #000080;
}

.checkpoint_status, .autosave_status {
    color:wheat;
}

#notebook_name {
    font-weight: 1000;
    font-size:20pt;
    text-variant:uppercase;
    color: wheat;
    margin-right:20px;
}
#notebook_name:hover {
background-color: salmon;
}

.dataframe { /* dataframe atau table */
    background: white;
    box-shadow: 0px 1px 2px #bbb;
}
.dataframe thead th, .dataframe tbody td {
    text-align: center;
    padding: 1em;
}

.checkpoint_status, .autosave_status {
    color:wheat;
}

.output {
    align-items: center;
}

div.cell {
    transition: all 0.25s;
    border: none;
    position: relative;
    top: 0;
}
div.cell.selected, div.cell.selected.jupyter-soft-selected {
    border: none;
    background: transparent;
    box-shadow: 0 6px 18px #aaa;
    z-index: 10;
    top: -10px;
}
.CodeMirror pre, .CodeMirror-dialog, .CodeMirror-dialog .CodeMirror-search-field, .terminal-app .terminal {
    font-family: 'Source Code Pro Medium' , serif;
    font-weight: 500;
    font-size: 12pt;
}

</style>

#  Residuals variance ($\sigma_\epsilon$), RSS and $R^2$

## Q:
''*What is the the relationship between variance of the response and $R^2$?*''

## Answer

We assume that this question is asked in the context of (ordinary) linear regression models, which we are currently studying.

First, let us clarify our notation. We will treat $y_i$ to be the response value of a datum used to train a linear regression model. Once linear regression has learned an estimator $\hat y = f(\mathbf{x})$ from the training data, the estimation it provides for a given input $\mathbf x_i$ is $y_i$. For a given dataset on $n$ instances of datum, $\mathscr{D}$, one can then compute the residual error of each prediction as:

\begin{equation} \epsilon_i = y_i - \hat y_i \end{equation}

Then a measure of error as the residual errors summed and squared is:

$$ \begin{aligned}
RSS &= \sum_{i=1}^n \epsilon_i^2 \\
    &= \sum_{i=1}^n (y_i -\hat y_i)^2 \\
\end{aligned}
$$

There is a close relationship between the variance of the **residuals** $\sigma_\epsilon^2$, rather than the variance of the response variable itself. Let us explore this systematically.

Once a linear regession model has been fit to the data, in general, the mean of the residuals $\mu_\epsilon$ will be close to zero, i.e. $\mathbf{\mu_\epsilon \approx 0}$. With this in hand, let us consider the variance of the residual errors over the training dataset $\mathscr{D}$ of $n$ sample instances.Then:

$$
\begin{aligned}\sigma^2 &= \frac{1}{n-1}\sum_1^n (\epsilon_i - \mu_\epsilon)^2 \\
& = \frac{1}{n-1}\sum_1^n \epsilon_i^2 \text{  since } \mu_\epsilon \approx 0 \\
\end{aligned}
$$

Now consider the definition of TSS (total residual errors summed and squared) of the null-hypothesis $\phi$, where $\epsilon_{i,\phi} = (y_i -\hat y_i^\phi), \text{ here } y_i^\phi \text { is the prediction of the null hypothesis and thus always equal to } \bar y$. Thus $\epsilon_{i,\phi} = (y_i - \bar y)$.

$$\begin{aligned}
TSS &= \sum_{i=1}^n \epsilon_\phi^2 \\
&=(n-1)\sigma_\phi^2 \\
\end{aligned}
$$

Likewise, the RSS of a given hypothesis $h$, with errors $\epsilon_{i, h} = (y_i - \hat y_{i,h})$, is:

$$\begin{aligned}
RSS &= \sum_{i=1}^n \epsilon_{i,h}^2 \\
&=(n-1)\sigma_h^2 \\
\end{aligned}
$$

The coefficient of determination, $R^2$, is defined as:

\begin{equation}
R^2 = \frac{TSS - RSS}{TSS}
\end{equation}

Substituting the values of TSS and RSS from the prior equations in terms of their variance of residuals, we get, therefore:

\begin{equation}
R^2 = \frac{\sigma_\phi^2 - \sigma_h^2}{\sigma_\phi^2}
\end{equation}

Thus, we have arrived at the relationship between RSS, $R^2$ and the variance of the residual errors.

```python

```


```python
%run ../../common/import_all.py

from common.setup_notebook import set_css_style, setup_matplotlib, config_ipython
config_ipython()
setup_matplotlib()
set_css_style()
```

# The Maximum Likelihood, Maximum a Posteriori and Expectation-maximisation estimation methods

## The likelihood

Imagine you have a statistical model, that is, a mathematical description of your data which depends on some parameters $\theta$. The *likelihood function*, usually indicated as $\mathcal{L}$, is a function of these parameters and represents the probability of observing evidence (observed data) $E$ given said parameters:

$$
\mathcal{L} = P(E \ | \  \theta)
$$

Because it is a function of the parameters given the outcome, you write

$$
\mathcal{L}(\theta \  | \ E) = P(E \ | \  \theta)
$$

The difference between *probability* and *likelihood* is quite subtle in that in common language they are be casually swapped, but they represent different things. The probability mesaures the outcomes observed as a function of the parameters $\theta$ of the underlying model. But in reality $\theta$ are unknown and in fact, we go through the reverse process: estimating the parameters given the evidence we observe. For this, we use the likelihood, which is defined as above because we maximise it in such a way to respond to the equality above. This is exactly what the ML estimation does, as per below.

Bear in mind that the likelihood is a function of $\theta$.

## The MLE method

The Maximum Likelihood Estimation (MLE) is a procedure to find the parameters of a statistical model via the maximisation of the likelihood so as to maximise the agreement between the model and the observed data.

The maximisation of the likelihood is usually performed via the maximisation of its logarithm as it is much more convenient; the logarithm is a monotonic function so the procedure is legit.

### Example: a Bernoulli distribution

The likelihood function for a [Bernoulli distribution](../distributions-measures/famous-distributions.ipynb#Bernoulli)  ($x_i \in {0, 1}$) is, for parameter $p$:

\begin{align}
\mathcal{L}(x_1, x_2, \ldots, x_n \ | \ p) &= P(X_1 = x_1, X_2 = x_2, \ldots, X_n = x_n \ | \ p) \\
&= p^{x_1}(1-p)^{1-x_1} \cdot \ldots \cdot p^{x_n}(1-p)^{1-x_n} \\
&= p^{\sum_i x_i}(1-p)^{\sum_i(1-x_i)} \\
&=  p^{\sum_i x_i}(1-p)^{n -\sum_i x_i}
\end{align}

so that if we take the logarithm, we get

$$
\log \mathcal{L} = \sum_i x_i \log p + \Big(n - \sum_i x_i\Big) \log (1-p) \ .
$$

To maximise it, we compute and nullify the first derivative

$$
\frac{d \log \mathcal{L}}{d p} = \frac{\sum_i x_i}{p} - \frac{n - \sum_i x_i}{1-p} = 0
$$

which leads to

$$
\sum_i x_i - p \sum_i x_i = np - p \sum_i x_i
$$

and finally to

$$
p = \frac{\sum_i x_i}{n}
$$

### Example: estimating the best mean of some data

This example is reported from [[here]](#2). Let us assume we know the weights of women are normally distributed with a mean $\mu$ and standard deviation $\sigma$. A random sample of $10$ women is (in pounds):

$$
115, 122, 130, 124, 149, 160, 152, 138, 149, 180
$$

We want to estimate $\mu$. We know

$$
P(x_i ; \mu, \sigma) = \frac{1}{\sqrt{2 \pi \sigma^2}} e^{- \frac{(x_i - \mu)^2}{2 \sigma^2}}
$$

The likelihood is (note that the $X_i$ are independent)

\begin{align}
\mathcal{L}(x_i | \mu, \sigma) &= P(X_1=x_1, \ldots, X_n=x_n)  \\
&= \Pi_i p(x_i; \mu \sigma) \\
&= \sigma^n (2 \pi)^{-n/2} e^{- \frac{1}{2 \sigma^2} \sum_i (x_i - \mu)^2}
\end{align}

Now, again it is easier to work with the logarithm:

$$
\log \mathcal{L} = -n \log \sigma \frac{n}{2} \log 2 \pi - \frac{1}{2 \sigma^2} \sum_i (x_i - \mu)^2
$$

so that

$$
\frac{d \log \mathcal{L}}{d \mu} = -\frac{1}{2 \sigma^2} 2 \sum_i (x_i - \mu) = 0
$$

$$
\sum_i x_i - n \mu = 0
$$

$$
\mu = \frac{\sum_i x_i}{n}
$$

and so the maximum likelihood estimate for a given sample is 142.2 and we can could do the same to estimate $\sigma$, obtaining (can be proven through second derivative that it is a maximum)

$$
\sigma^2 = \frac{\sum_i (x_i - \mu)^2}{n}
$$

## The MAP method

This Maximim a Posteriori (MAP) estimation method uses the mode of the posterior to estimate the unknown population.

From Bayes' theorem, the posterior is expressed as

$$
P(\theta \ | \ x) = \frac{P(x \ | \ \theta) P(\theta)}{\int d \theta' P(x \ | \ \theta') P(\theta')} \ ,
$$

with $\theta$ being the parameters of the statistical model and $x$ the observed data. The MAP method estimates $\theta$ as the one which maximises the posterior; note that the denominator is just a normalisation factor:

$$
\hat{\theta}_{MAP}(x) = arg \max_\theta P(\theta \  | \  x) = arg \max_\theta P(x \ | \ \theta) P(\theta) \ .
$$

This means exactly taking the mode of the posterior distribution.

In the case of a uniform prior, the MAP estimation is equal to the ML estimation as we get to maximise the likelihood because the prior becomes just a factor. For the computation, conjugate priors are particularly handy.

As in the case of the MLE, what we really do is maximising the logarithm of the posterior rather than the posterior itself, so we do

$$
\hat \theta_{MAP}(x) = arg \max_{\theta} \log P(\theta \ | \ x) = arg \max_{\theta} [\log P(x \ | \ \theta) + \log P(\theta)] \ .
$$

## MAP and ML

In the last equation, if we only had the first term to maximise, we would be doing a ML estimation. The second term is the one accounting for the presence of a prior: this is why the MAP method is considered as a regularised ML as prior knowledge is factored in the computation.

While the ML method can be seen as responding to a frequentist approach, the MAP method responds to a Bayesian approach.

## The Expectation-Maximisation algorithm

The EM algorithm can be used to find the solution of MLE or MAP when some data is missing, meaning there are some latent variables not observed.

Let's say that for the random variable $x$ we have the $n$ observations

$$
x_1, \ldots, x_n \ ,
$$

which depend on parameters $\theta$, and that the goal is to find the parameter $\theta$ that maximises the likelihood which is of the form

$$
L = \sum_z P_\theta(x, z) \ ,
$$

meaning it is a sum over the latent variables $z$; this makes the problem difficult to solve analytically.

The EM algorithm updates the parameters in steps, which means it risks obtaining a local rather than a global maximum.

### The E step

In the E phase (time $t$), the expected value of $L$ is computed with respect to the conditional distribution of $z$ given $x$ under the current estimate of parameters $\theta$:

$$
\bar L(\theta | \theta^t) = \mathbb E_{z | x, \theta^t} [\log L (\theta, x)]
$$

This means that the log-likelihood is evaluated using the current state of the parameters.

### The M step

In the M phase (time $t+1$), we find the parameters which maximises the log-likelihood found in the E step:

$$
\theta^{t+1} = arg \max_{\theta} \bar L(\theta | \theta^t)
$$

## References

1.  [Cross Validated on the difference between Probability and Likelihood](https://stats.stackexchange.com/questions/2641/what-is-the-difference-between-likelihood-and-probability)
2.  Some examples in [this course](https://onlinecourses.science.psu.edu/stat414/node/191) from Penn State
3. [An assignement on the method from Carnegie Mellon](http://www.cs.cmu.edu/~aarti/Class/10601/homeworks/hw2Solutions.pdf)

```python

```