```python
import numpy as np
import pyemma as pm
import mdshare
import matplotlib.pyplot as plt
%matplotlib inline
```

# Information, entropy and caliber

**warning inconsistent nomenclature ahead (sorry, no time to tidy up)**

## Information:

1 Bit = reduce uncertainty by factor of 2 (Shannon definition).

$$
h(x)=\log_{2} \frac{1}{P(x)}
$$

if we observe $x$ when the probability is $P(x)$ then we reduce our uncertainty by $2^{h(x)}$. e.g., if weather is 75% *rain*, 25% *sun*, and we observe *sun* our uncertainty has been reduced by a factor of $4$ or $\log_{2}(4) = 2$ bits. If we observe *rain*, intuitively we have learned *less*, our uncertainty has been reduced by a smaller amount ($4/3$) or $\log_{2}(4/3) = 0.42$ bits. **The less likely something is, the more suprising it is when we observe it/the more information we gained**.

## Entropy

Entropy is the **average information** content of a distribution:

$$
H(X) \equiv \sum_{x \in \mathcal{A}_{X}} P(x) \log \frac{1}{P(x)}
$$

### Joint entropy
Many extensions, e.g., joint entropy = average information gained when observing $x, y$ from a joint distribution:

$$
H(X, Y)=\sum_{x y \in \mathcal{A}_{X} \mathcal{A}_{Y}} P(x, y) \log \frac{1}{P(x, y)}
$$

### Conditional entropy

Average information from $Y$ conditional on a specific value of $X$ occuring, then averaged over all possible values of $X$:

$$
\begin{aligned}
\mathrm{H}(Y \mid X) & \equiv \sum_{x \in \mathcal{X}} p(x) \mathrm{H}(Y \mid X=x) \\
& [\mathrm{some\ algebra}]\\
&=\sum_{x \in \mathcal{X}, y \in \mathcal{Y}} p(x, y) \log \frac{p(x)}{p(x, y)}
\end{aligned}
$$

Many interesting properties of conditional entropy used in derivations below (not shown, please use [Wikipedia](https://en.wikipedia.org/wiki/Conditional_entropy))

## Entropy *rate*

### Definition
This is taken from [](https://homepages.cwi.nl/~schaffne/courses/infcom/2014/reports/EntropyRate_Mulder_Peters.pdf)

We can talk about the entropy of a stochastic process, $X_{1}, X_{2}, ..., X_{n} = \{X_{i}\}$.

If $X_{i}$ are identically and *indepedently* distributed then $H(\{X_{i}\}) = n \times H(X_{1})$ (use formula for joint entropy and the fact that for i.i.d., we have $P(x, y) = P(x)P(y)$).

However, in general (e.g., Markov process) $X_{i}$ are not independent so we define **entropy rate**:

$$
H\left(\left\{X_{i}\right\}\right) \triangleq \lim _{n \rightarrow \infty} \frac{H\left(X_{1}, X_{2}, \ldots, X_{n}\right)}{n}
$$

### Conditional entropy rate

We can also define the **conditional entropy rate**:

$$
\mathrm{H}^{\prime}\left(\left\{X_{i}\right\}\right) \triangleq \lim _{n \rightarrow \infty} \mathrm{H}\left(X_{n} \mid X_{n-1}, \ldots, X_{1}\right)
$$

## Entropy rate and conditional entropy rate are the same

For *stationary* stochastic process we have:

$$
\mathrm{H}\left(\left\{X_{i}\right\}\right)=\lim _{n \rightarrow \infty} \frac{\mathrm{H}\left(\mathrm{X}_{1}, \ldots, \mathrm{X}_{\mathrm{n}}\right)}{\mathrm{n}}=\lim _{n \rightarrow \infty} \mathrm{H}\left(\mathrm{X}_{\mathrm{n}} \mid \mathrm{X}_{\mathrm{n}-1}, \ldots, \mathrm{X}_{1}\right)=\mathrm{H}^{\prime}\left(\left\{\mathrm{X}_{\mathrm{i}}\right\}\right)
$$

Proof: [see here](https://homepages.cwi.nl/~schaffne/courses/infcom/2014/reports/EntropyRate_Mulder_Peters.pdf)

### Some facts:

1. Both entropy rates **do not increase with $n$**. i.e., as we observe more of a stochastic process, the average information of the whole sequence **or** the information conditional on past sequence (which are the same in the limit of large $n$) decreases (or stays the same).
2. For a finite $n$, the conditional entropy rate is always smaller (conditioning never increases the entropy as you're restricting the outcomes).

## Entropy of Markov process

For a stationary Markov process we have:

$$
\begin{aligned}
\mathrm{H}\left(\left\{\mathbf{Z}_{i}\right\}\right) &=\mathrm{H}^{\prime}\left(\left\{Z_{i}\right\}\right) \\
&=\lim _{n \rightarrow \infty} \mathrm{H}\left(Z_{n} \mid Z_{n-1}, \ldots, Z_{1}\right) \\
&=\lim _{n \rightarrow \infty} \mathrm{H}\left(Z_{n} \mid Z_{n-1}\right)\quad \mathrm{(Markov\ property)} \\
&=\mathrm{H}\left(Z_{2} \mid Z_{1}\right)\quad \mathrm{(Stationary\  process)} \\
&=-\sum_{i=1}^{|Z|} \mu_{i}\left(\sum_{j=1}^{|Z|} P_{i j} \log P_{i j}\right)
\end{aligned}
$$

i.e., Entropy rate of MP:

1. **First (inner) summation**: consider single state, $i$, the distribution of transitions out of that state has some entropy, $h_i$
2. **Second (outer) summation**: Now we want average entropy over all states, so use stationary distribution $\sum \mu_i h_i$

## Cross-entropy
(original definition is in terms of message lengths - we'll use a different motivation)

Consider estimating probability of an event/quantity etc., $q_i$.  We observe it happening $Np_i$ times (i.e., empirical probability is $p_i$).  The likelihood of the parameter $q_i$ is:

$$
\mathcal{L} = \prod_{i} q_{i}^{N p_{i}}
$$

now take logs and divide by $N$ (yes, notation overloaded here)

$$
\frac{1}{N} \log \prod_{i} q_{i}^{N p_{i}}=\sum_{i} p_{i} \log q_{i}=-H(p, q)
$$

this is the cross-entropy. It measures the information content of a predicted distribution i.e., $q$ given the true distributon. If your guess about a distribution is not good, then you will have a high probability $p_i$ of of observing high information events $q_i$ which doesn't make sense - high information events have small probabilities!

It's always going to be greater than the entropy because this is actual average infomormation content.

### KL divergence
The difference between the true average information content ($H(p)$) of the ditribution $p$, and the average information content ($H(p, q)$) from your best guess $q$ is the KL divergence:

$$
\begin{align}
D_{\mathrm{KL}}(p \| q) &= H(p) - H(p, q) \\
& =\sum_{i} p_i \log \frac{p_i}{q_i}
\end{align}
$$

### KL divergence of MP = - Caliber

Now we get to formula:
$$
\mathcal{D}=\sum_{i, j} \pi_{i} p_{i j} \ln \left(\frac{p_{i j}}{p_{i j}^{*}}\right)
$$

This just the entropy of an MP with the cross entropy subtracted off.

#  Example - 2 state

$$
P=\left[\begin{array}{cc}
1-\alpha & \alpha \\
\beta & 1-\beta
\end{array}\right]
$$

For a two state process we don't really need MaxCal approach as we can always choose $\alpha$ and $\beta$ to match a given $\mu$ but lets do it using the Hongbin approach.

**Start with Markov process 1, (P1) with stationary distribution $\pi_{1}$**

```python

import scipy.stats as sp

alpha1 = 0.1
beta1 = 0.2

P1 = np.array([[1-alpha1, alpha1], [beta1, 1-beta1]])
evals1, evecs1 = np.linalg.eig(P1.T)
pi1 = evecs1[:, evals1==1].flatten()
pi1 = pi1/np.sum(pi1)
print('Transition matrix:\n{}\n'.format(P1))
print('Stationary distribution:\n{}\n'.format(pi1))
print('Satisfies DB?\n{}'.format(np.allclose(pi1[0]*P1[0, 1], pi1[1]*P1[1, 0])))
```

    Transition matrix:
    [[0.9 0.1]
     [0.2 0.8]]

    Stationary distribution:
    [0.66666667 0.33333333]

    Satisfies DB?
    True

**Now consider another MP (P2) where we only know $\pi_2$ (but really we know P2 as well so that we can compare exact results with MaxCal approach)**

```python
alpha2 = alpha1/2
beta2 = 0.2

P2 = np.array([[1-alpha2, alpha2], [beta2, 1-beta2]])
evals2, evecs2 = np.linalg.eig(P2.T)
pi2 = evecs2[:, evals2==1].flatten()
pi2 = pi2/np.sum(pi2)
print('Transition matrix:\n{}\n'.format(P2))
print('Stationary distribution:\n{}\n'.format(pi2))
print('Satisfies DB?\n{}'.format(np.allclose(pi2[0]*P2[0, 1], pi2[1]*P2[1, 0])))
```

    Transition matrix:
    [[0.95 0.05]
     [0.2  0.8 ]]

    Stationary distribution:
    [0.8 0.2]

    Satisfies DB?
    True

**Question: can we calculate P2 given $\pi_2$ and P1?**

Answer: kind of -  use iterative scheme in equations 13, 14

```python
def update_pij(pi, p_star, w):
    n = pi.shape[0]
    p = np.empty((n, n))
    for i in range(n):
        for j in range(n):
            top = pi[j]*p_star[j, i]*w[j]
            bottom = pi[i]*p_star[i, j]*w[i]
            p[i, j] = p_star[i, j]*np.sqrt(top/bottom)*w[i]
    return p

def update_w(w, p):
    n = w.shape[0]
    w_new = np.empty(n)
    for i in range(n):
        w_new[i] = w[i]/np.sum(p[i, :])
    return w_new
```

```python
max_iter = 100

n = P1.shape[0]
w = np.ones(n)
alpha_ests = []
for i in range(max_iter):
    P2_est = update_pij(pi2, P1, w)
    w = update_w(w, P2_est)

evals, evecs = np.linalg.eig(P2_est.T)

pi2_est = evecs[:, np.argmin(np.abs(evals-1))]
pi2_est = pi2_est/np.sum(pi2_est)

```

```python
pi2_est
```

    array([0.8, 0.2])

```python
P2_est
```

    array([[0.93147581, 0.06852419],
           [0.27409676, 0.72590324]])

**Why the difference?**

Stationary distribution of 2 x 2 is (some algebra):

$$
\pi = \left[\begin{array}{cc}
\frac{\beta}{\alpha+\beta} & \frac{\alpha}{\alpha+\beta}
\end{array}\right]
$$

Fixing the stationary distribution at $0.8, 0.2$ to solve for $\alpha$ and $\beta$

$$
\left[\begin{array}{cc}
4 & -1 \\
4 & -1
\end{array}\right]
\left[\begin{array}{c}
\alpha \\
\beta
\end{array}\right] =
\left[\begin{array}{c}
0 \\
0
\end{array}\right]
$$

so all solutions have $4\alpha = \beta$. For stochastic matrices we have $0 < \alpha, \beta < 1$ then:

```python
alphas = np.linspace(0.01, 0.249, 100)
betas = alphas*4

plt.plot(alphas, betas)
```

Now let's calculate the entropy of the all the potential solutions on this line.

```python
def entropy_mp(alpha, beta):
    P = np.array([[1-alpha, alpha], [beta, 1-beta]])
    evals, evecs = np.linalg.eig(P.T)
    pi = evecs[:, np.argmin(np.abs(evals-1))]
    pi = pi/np.sum(pi)
    ent = np.dot(sp.entropy(P.T), pi)
    return ent

def kl_divergence(alpha, beta, alpha_star, beta_star):
    P = np.array([[1-alpha, alpha], [beta, 1-beta]])
    P_star = np.array([[1-alpha_star, alpha_star], [beta_star, 1-beta_star]])
    evals, evecs = np.linalg.eig(P.T)
    pi = evecs[:, np.argmin(np.abs(evals-1))]
    pi = pi/np.sum(pi)
    n = P.shape[0]
    ent = 0
    for i in range(n):
        for j in range(n):
             ent -= pi[i]*(P[i, j]*np.log2(P[i, j])-P[i, j]*np.log2(P_star[i, j]))

    return ent
```

```python
entropies =  np.array([entropy_mp(float(x), float(y)) for x, y in zip(alphas, betas)])
kl_divs = np.array([kl_divergence(float(x), float(y), alpha1, beta1) for x, y in zip(alphas, betas)])
```

```python
plt.plot(alphas, betas, label=r'$\beta = 4\alpha$')
plt.plot(alphas, entropies,  label='Entropy')
plt.plot(alphas, kl_divs,  label='KL divergence')
plt.scatter(P2_est[0, 1], P2_est[1, 0], marker='x', s=100, label='Estimated MaxCal', color='k')
plt.scatter(alphas[np.argmax(entropies)], betas[np.argmax(entropies)], label='True MaxEnt')
plt.scatter(alphas[np.argmax(kl_divs)], betas[np.argmax(kl_divs)], label='True MaxCal')
plt.legend(bbox_to_anchor=(1, 1))
```

```python
print('MaxEnt alpha: ', alphas[np.argmax(entropies)])
print('MaxEnt  beta: ', betas[np.argmax(entropies)])

```

    MaxEnt alpha:  0.20071717171717174
    MaxEnt  beta:  0.802868686868687

```python
print('MaxCal alpha: ', alphas[np.argmax(kl_divs)])
print('MaxCal  beta: ', betas[np.argmax(kl_divs)])

```

    MaxCal alpha:  0.06793939393939394
    MaxCal  beta:  0.27175757575757575

```python
P2_est
```

    array([[0.93147581, 0.06852419],
           [0.27409676, 0.72590324]])


# PHY321: Conservative Forces, Momentum and Angular Momentum conservation
**[Morten Hjorth-Jensen](http://mhjgit.github.io/info/doc/web/)**, Department of Physics and Astronomy and Facility for Rare Ion Beams (FRIB), Michigan State University, USA and Department of Physics, University of Oslo, Norway

Date: **Feb 11, 2022**

Copyright 1999-2022, [Morten Hjorth-Jensen](http://mhjgit.github.io/info/doc/web/). Released under CC Attribution-NonCommercial 4.0 license

## Aims and Overarching Motivation

### Monday 2/7

Short repetition from last week. Discussion of momentum and angular momentum conservation.
Reading suggestion: Taylor sections 3.4 and 3.5 and chapter 4.

### Wednesday 2/9

Examples of application of conservations laws (see chapter 4 of Taylor) and exercises (see next slide).

### Friday 2/11

Conservative forces and discussion of homework exercises. Problem solving. Deadline fourth homework.

**Reading suggestions.**

Taylor chapter 4 is the essential reading.
See also chapter 7 of Malthe-Sørenssen (in particular section 7.5) for exercise 6 in homework 4

## Exercises for Wednesday after lecture part

**Example 1.**

We study a classical electron which moves in the $x$-direction along a surface. The force from the surface is

$$
\boldsymbol{F}(x)=-F_0\sin{(\frac{2\pi x}{b})}\boldsymbol{i}.
$$

Show that the force is conservative.

**Example 2.**

Show that the force

$$
\boldsymbol{F}(\boldsymbol{r})=\gamma \frac{\boldsymbol{r}}{r^3},
$$

is a conservative force. Here $\gamma$ is a constant and $r=\sqrt{x^2+y^2+z^2}$ and $\boldsymbol{r}=x\boldsymbol{i}+y\boldsymbol{j}+z\boldsymbol{k}$.

## One Figure to Rule All Forces (thx to Julie)

Figure 1:

## What is a Conservative Force?

A conservative force is a force whose property is that the total work
done in moving an object between two points is independent of the
taken path. This means that the work on an object under the influence
of a conservative force, is independent on the path of the object. It
depends only on the spatial degrees of freedom and it is possible to
assign a numerical value for the potential at any point. It leads to
conservation of energy. The gravitational force is an example of a
conservative force.

## Two important conditions

First, a conservative force depends only on the spatial degrees of freedom. This is a necessary condition for obtaining a path integral which is independent of path.
The  important condition for the final work to be independent of the path is that the **curl** of the force is zero, that

$$
\boldsymbol{\nabla} \times \boldsymbol{F}=0
$$

## The total Momentum

The total momentum $\boldsymbol{P}$ is defined as the sum of the individual momenta, meaning that we can rewrite

$$
\boldsymbol{F}_1^{\mathrm{net}}+\boldsymbol{F}_2^{\mathrm{net}}=\frac{d\boldsymbol{p}_1}{dt}+\frac{d\boldsymbol{p}_2}{dt}=\frac{d\boldsymbol{P}}{dt},
$$

that is the derivate with respect to time of the total momentum. If we now
write the net forces as sums of the external plus internal forces
between the objects we have

$$
\frac{d\boldsymbol{P}}{dt}=\boldsymbol{F}_1^{\mathrm{ext}}+\boldsymbol{F}_{12}+\boldsymbol{F}_2^{\mathrm{ext}}+\boldsymbol{F}_{21}=\boldsymbol{F}_1^{\mathrm{ext}}+\boldsymbol{F}_2^{\mathrm{ext}}.
$$

The derivative of the total momentum is just **the sum of the external
forces**. If we assume that the external forces are zero and that only
internal (here two-body forces) are at play, we obtain the important
result that the derivative of the total momentum is zero. This means
again that the total momentum is a constant of the motion and
conserved quantity. This is a very important result that we will use
in many applications to come.

## Newton's Second Law

Let us now general to several objects $N$ and let us also assume that there are no external forces. We will label such a system as **an isolated system**.

Newton's second law, $\boldsymbol{F}=m\boldsymbol{a}$, can be written for a particle $i$ as

$$
\begin{equation}
\boldsymbol{F}_i=\sum_{j\ne i}^N \boldsymbol{F}_{ij}=m_i\boldsymbol{a}_i,
\label{_auto1} \tag{1}
\end{equation}
$$

where $\boldsymbol{F}_i$ (a single subscript) denotes the net force acting on $i$ from the other objects/particles.
Because the mass of $i$ is fixed and we assume it does not change with time, one can see that

$$
\begin{equation}
\boldsymbol{F}_i=\frac{d}{dt}m_i\boldsymbol{v}_i=\sum_{j\ne i}^N\boldsymbol{F}_{ij}.
\label{_auto2} \tag{2}
\end{equation}
$$

## Summing over all Objects/Particles

Now, one can sum over all the objects/particles and obtain

$$
\frac{d}{dt}\sum_i m_iv_i=\sum_{ij, i\ne j}^N\boldsymbol{F}_{ij}=0.
$$

How did we arrive at the last step?  We rewrote the double sum as

$$
\sum_{ij, i\ne j}^N\boldsymbol{F}_{ij}=\sum_i^N\sum_{j>i}\left(\boldsymbol{F}_{ij}+\boldsymbol{F}_{ji}\right),
$$

and using Newton's third law which states that
$\boldsymbol{F}_{ij}=-\boldsymbol{F}_{ji}$, we obtain that the net sum over all  the two-particle
forces is zero when we only consider so-called **internal forces**.
Stated differently, the last step made use of the fact that for every
term $ij$, there is an equivalent term $ji$ with opposite
force. Because the momentum is defined as $m\boldsymbol{v}$, for a system of
particles, we have thus

$$
\begin{equation}
\frac{d}{dt}\sum_im_i\boldsymbol{v}_i=0,~~{\rm for~isolated~particles}.
\label{_auto3} \tag{3}
\end{equation}
$$

## Conservation of total Momentum

By "isolated" one means that the only force acting on any particle $i$
are those originating from other particles in the sum, i.e. "no
external" forces. Thus, Newton's third law leads to the conservation
of total momentum,

$$
\boldsymbol{P}=\sum_i m_i\boldsymbol{v}_i,
$$

and we have

$$
\frac{d}{dt}\boldsymbol{P}=0.
$$

## Conservation of Angular Momentum

The angular momentum is defined as

$$
\begin{equation}
\boldsymbol{L}=\boldsymbol{r}\times\boldsymbol{p}=m\boldsymbol{r}\times\boldsymbol{v}.
\label{_auto4} \tag{4}
\end{equation}
$$

It means that the angular momentum is perpendicular to the plane defined by position $\boldsymbol{r}$ and the momentum $\boldsymbol{p}$ via $\boldsymbol{r}\times \boldsymbol{p}$.

## Rate of Change of Angular Momentum

The rate of change of the angular momentum is

$$
\frac{d\boldsymbol{L}}{dt}=m\boldsymbol{v}\times\boldsymbol{v}+m\boldsymbol{r}\times\dot{\boldsymbol{v}}=\boldsymbol{r}\times{\boldsymbol{F}}
$$

The first term is zero because $\boldsymbol{v}$ is parallel to itself, and the
second term defines the so-called torque. If $\boldsymbol{F}$ is parallel to $\boldsymbol{r}$ then the torque is zero and we say that angular momentum is conserved.

If the force is not radial, $\boldsymbol{r}\times\boldsymbol{F}\ne 0$ as above, and angular momentum is no longer conserved,

$$
\begin{equation}
\frac{d\boldsymbol{L}}{dt}=\boldsymbol{r}\times\boldsymbol{F}\equiv\boldsymbol{\tau},
\label{_auto5} \tag{5}
\end{equation}
$$

where $\boldsymbol{\tau}$ is the torque.

## The Torque, Example 1 (hw 4, exercise 4)

Let us assume we have an initial position $\boldsymbol{r}_0=x_0\boldsymbol{e}_1+y_0\boldsymbol{e}_2$ at a time $t_0=0$.
We add now a force in the positive $x$-direction

$$
\boldsymbol{F}=F_x\boldsymbol{e}_1=\frac{d\boldsymbol{p}}{dt},
$$

where we used the force as defined by the time derivative of the momentum.

We can use this force (and its pertinent acceleration) to find the velocity via the relation

$$
\boldsymbol{v}(t)=\boldsymbol{v}_0+\int_{t_0}^t\boldsymbol{a}dt',
$$

and with $\boldsymbol{v}_0=0$ we have

$$
\boldsymbol{v}(t)=\int_{t_0}^t\frac{\boldsymbol{F}}{m}dt',
$$

where $m$ is the mass of the object.

## The Torque, Example 1 (hw 4, exercise 4)

Since the force acts only in the $x$-direction, we have after integration

$$
\boldsymbol{v}(t)=\frac{\boldsymbol{F}}{m}t=\frac{F_x}{m}t\boldsymbol{e}_1=v_x(t)\boldsymbol{e}_1.
$$

The momentum is in turn given by $\boldsymbol{p}=p_x\boldsymbol{e}_1=mv_x\boldsymbol{e}_1=F_xt\boldsymbol{e}_1$.

Integrating over time again we find the final position as (note the force depends only on the $x$-direction)

$$
\boldsymbol{r}(t)=(x_0+\frac{1}{2}\frac{F_x}{m}t^2) \boldsymbol{e}_1+y_0\boldsymbol{e}_2.
$$

There is no change in the position in the $y$-direction since the force acts only in the $x$-direction.

## The Torque, Example 1 (hw 4, exercise 4)

We can now compute the angular momentum given by

$$
\boldsymbol{l}=\boldsymbol{r}\times\boldsymbol{p}=\left[(x_0+\frac{1}{2}\frac{F_x}{m}t^2) \boldsymbol{e}_1+y_0\boldsymbol{e}_2\right]\times F_xt\boldsymbol{e}_1.
$$

Computing the cross product we find

$$
\boldsymbol{l}=-y_0F_xt\boldsymbol{e}_3=-y_0F_xt\boldsymbol{e}_z.
$$

The torque is the time derivative of the angular momentum and we have

$$
\boldsymbol{\tau}=-y_0F_x\boldsymbol{e}_3=-y_0F_x\boldsymbol{e}_z.
$$

The torque is non-zero and angular momentum is not conserved.

## System of Isolated Particles

For a system of isolated particles, one can write

$$
\begin{eqnarray}
\frac{d}{dt}\sum_i\boldsymbol{L}_i&=&\sum_{i\ne j}\boldsymbol{r}_i\times \boldsymbol{F}_{ij}\\
\nonumber
&=&\frac{1}{2}\sum_{i\ne j} \boldsymbol{r}_i\times \boldsymbol{F}_{ij}+\boldsymbol{r}_j\times\boldsymbol{F}_{ji}\\
\nonumber
&=&\frac{1}{2}\sum_{i\ne j} (\boldsymbol{r}_i-\boldsymbol{r}_j)\times\boldsymbol{F}_{ij}=0,
\end{eqnarray}
$$

where the last step used Newton's third law,
$\boldsymbol{F}_{ij}=-\boldsymbol{F}_{ji}$. If the forces between the particles are
radial, i.e. $\boldsymbol{F}_{ij} ~||~ (\boldsymbol{r}_i-\boldsymbol{r}_j)$, then each term in
the sum is zero and the net angular momentum is fixed. Otherwise, you
could imagine an isolated system that would start spinning
spontaneously.

## Work, Energy, Momentum and Conservation laws

Energy conservation is most convenient as a strategy for addressing
problems where time does not appear. For example, a particle goes
from position $x_0$ with speed $v_0$, to position $x_f$; what is its
new speed? However, it can also be applied to problems where time
does appear, such as in solving for the trajectory $x(t)$, or
equivalently $t(x)$.

## Energy Conservation
Energy is conserved in the case where the potential energy, $V(\boldsymbol{r})$, depends only on position, and not on time. The force is determined by $V$,

$$
\begin{equation}
\boldsymbol{F}(\boldsymbol{r})=-\boldsymbol{\nabla} V(\boldsymbol{r}).
\label{_auto6} \tag{6}
\end{equation}
$$

## Conservative forces

We say a force is conservative if it satisfies the following conditions:
1. The force $\boldsymbol{F}$ acting on an object only depends on the position $\boldsymbol{r}$, that is $\boldsymbol{F}=\boldsymbol{F}(\boldsymbol{r})$.

2. For any two points $\boldsymbol{r}_1$ and $\boldsymbol{r}_2$, the work done by the force $\boldsymbol{F}$ on the displacement between these two points is independent of the path taken.

3. Finally, the **curl** of the force is zero $\boldsymbol{\nabla}\times\boldsymbol{F}=0$.

## Forces and Potentials

The energy $E$ of a given system is defined as the sum of kinetic and potential energies,

$$
E=K+V(\boldsymbol{r}).
$$

We define the potential energy at a point $\boldsymbol{r}$ as the negative work done from a starting point $\boldsymbol{r}_0$ to a final point $\boldsymbol{r}$

$$
V(\boldsymbol{r})=-W(\boldsymbol{r}_0\rightarrow\boldsymbol{r})= -\int_{\boldsymbol{r}_0}^{\boldsymbol{r}}d\boldsymbol{r}'\boldsymbol{F}(\boldsymbol{r}').
$$

If the potential depends on the path taken between these two points there is no unique potential.

## Example (relevant for homework 5)

We study a classical electron which moves in the $x$-direction along a surface. The force from the surface is

$$
\boldsymbol{F}(x)=-F_0\sin{(\frac{2\pi x}{b})}\boldsymbol{e}_1.
$$

The constant $b$ represents the distance between atoms at the surface of the material, $F_0$ is a constant and $x$ is the position of the electron.

This is indeed a conservative force since it depends only on position
and its **curl** is zero, that is $-\boldsymbol{\nabla}\times \boldsymbol{F}=0$. This means that energy is conserved and the
integral over the work done by the force is independent of the path
taken. We will come back to this in more detail next week.

## Example Continues

Using the work-energy theorem we can find the work $W$ done when
moving an electron from a position $x_0$ to a final position $x$
through the integral

$$
W=-\int_{x_0}^x \boldsymbol{F}(x')dx' =  \int_{x_0}^x F_0\sin{(\frac{2\pi x'}{b})} dx',
$$

which results in

$$
W=\frac{F_0b}{2\pi}\left[\cos{(\frac{2\pi x}{b})}-\cos{(\frac{2\pi x_0}{b})}\right].
$$

Since this is related to the change in kinetic energy we have, with $v_0$ being the initial velocity at a  time $t_0$,

$$
v  = \pm\sqrt{\frac{2}{m}\frac{F_0b}{2\pi}\left[\cos{(\frac{2\pi x}{b})}-\cos{(\frac{2\pi x_0}{b})}\right]+v_0^2}.
$$

## The potential energy from this example

The potential energy, due to energy conservation is

$$
V(x)=V(x_0)+\frac{1}{2}mv_0^2-\frac{1}{2}mv^2,
$$

with $v$ given by the velocity from above.

We can now, in order to find a more explicit expression for the
potential energy at a given value $x$, define a zero level value for
the potential. The potential is defined, using the work-energy
theorem, as

$$
V(x)=V(x_0)+\int_{x_0}^x (-F(x'))dx',
$$

and if you recall the definition of the indefinite integral, we can rewrite this as

$$
V(x)=\int (-F(x'))dx'+C,
$$

where $C$ is an undefined constant. The force is defined as the
gradient of the potential, and in that case the undefined constant
vanishes. The constant does not affect the force we derive from the
potential.

We have then

$$
V(x)=V(x_0)-\int_{x_0}^x \boldsymbol{F}(x')dx',
$$

which results in

$$
V(x)=\frac{F_0b}{2\pi}\left[\cos{(\frac{2\pi x}{b})}-\cos{(\frac{2\pi x_0}{b})}\right]+V(x_0).
$$

We can now define

$$
\frac{F_0b}{2\pi}\cos{(\frac{2\pi x_0}{b})}=V(x_0),
$$

which gives

$$
V(x)=\frac{F_0b}{2\pi}\left[\cos{(\frac{2\pi x}{b})}\right].
$$

## Force and Potential

We have defined work as the energy resulting from a net force acting
on an object (or sseveral objects), that is

$$
W(\boldsymbol{r}\rightarrow \boldsymbol{r}+d\boldsymbol{r})= \boldsymbol{F}(\boldsymbol{r})d\boldsymbol{r}.
$$

If we write out this for each component we have

$$
W(\boldsymbol{r}\rightarrow \boldsymbol{r}+d\boldsymbol{r})=\boldsymbol{F}(\boldsymbol{r})d\boldsymbol{r}=F_xdx+F_ydy+F_zdz.
$$

The work done from an initial position to a final one defines also the difference in potential energies

$$
W(\boldsymbol{r}\rightarrow \boldsymbol{r}+d\boldsymbol{r})=-\left[V(\boldsymbol{r}+d\boldsymbol{r})-V(\boldsymbol{r})\right].
$$

## Getting to $\boldsymbol{F}(\boldsymbol{r})=-\boldsymbol{\nabla} V(\boldsymbol{r})$

We can write out the differences in potential energies as

$$
V(\boldsymbol{r}+d\boldsymbol{r})-V(\boldsymbol{r})=V(x+dx,y+dy,z+dz)-V(x,y,z)=dV,
$$

and using the expression the differential of a multi-variable function $f(x,y,z)$

$$
df=\frac{\partial f}{\partial x}dx+\frac{\partial f}{\partial y}dy+\frac{\partial f}{\partial z}dz,
$$

we can write the expression for the work done as

$$
W(\boldsymbol{r}\rightarrow \boldsymbol{r}+d\boldsymbol{r})=-dV=-\left[\frac{\partial V}{\partial x}dx+\frac{\partial V}{\partial y}dy+\frac{\partial V}{\partial z}dz \right].
$$

## Final expression

Comparing the last equation with

$$
W(\boldsymbol{r}\rightarrow \boldsymbol{r}+d\boldsymbol{r})=F_xdx+F_ydy+F_zdz,
$$

we have

$$
F_xdx+F_ydy+F_zdz=-\left[\frac{\partial V}{\partial x}dx+\frac{\partial V}{\partial y}dy+\frac{\partial V}{\partial z}dz \right],
$$

leading to

$$
F_x=-\frac{\partial V}{\partial x},
$$

and

$$
F_y=-\frac{\partial V}{\partial y},
$$

and

$$
F_z=-\frac{\partial V}{\partial z},
$$

or just

$$
\boldsymbol{F}=-\frac{\partial V}{\partial x}\boldsymbol{e}_1-\frac{\partial V}{\partial y}\boldsymbol{e}_2-\frac{\partial V}{\partial z}\boldsymbol{e}_3=-\boldsymbol{\nabla}V(\boldsymbol{r}).
$$

And this connection is the one we wanted to show.

## Net Energy

The net energy, $E=V+K$ where $K$ is the kinetic energy, is then conserved,

$$
\begin{eqnarray}
\frac{d}{dt}(K+V)&=&\frac{d}{dt}\left(\frac{m}{2}(v_x^2+v_y^2+v_z^2)+V(\boldsymbol{r})\right)\\
\nonumber
&=&m\left(v_x\frac{dv_x}{dt}+v_y\frac{dv_y}{dt}+v_z\frac{dv_z}{dt}\right)
+\partial_xV\frac{dx}{dt}+\partial_yV\frac{dy}{dt}+\partial_zV\frac{dz}{dt}\\
\nonumber
&=&v_xF_x+v_yF_y+v_zF_z-F_xv_x-F_yv_y-F_zv_z=0.
\end{eqnarray}
$$

## In Vector Notation

The same proof can be written more compactly with vector notation,

$$
\begin{eqnarray}
\frac{d}{dt}\left(\frac{m}{2}v^2+V(\boldsymbol{r})\right)
&=&m\boldsymbol{v}\cdot\dot{\boldsymbol{v}}+\boldsymbol{\nabla} V(\boldsymbol{r})\cdot\dot{\boldsymbol{r}}\\
\nonumber
&=&\boldsymbol{v}\cdot\boldsymbol{F}-\boldsymbol{F}\cdot\boldsymbol{v}=0.
\end{eqnarray}
$$

Inverting the expression for kinetic energy,

$$
\begin{equation}
v=\sqrt{2K/m}=\sqrt{2(E-V)/m},
\label{_auto7} \tag{7}
\end{equation}
$$

allows one to solve for the one-dimensional trajectory $x(t)$, by finding $t(x)$,

$$
\begin{equation}
t=\int_{x_0}^x \frac{dx'}{v(x')}=\int_{x_0}^x\frac{dx'}{\sqrt{2(E-V(x'))/m}}.
\label{_auto8} \tag{8}
\end{equation}
$$

Note this would be much more difficult in higher dimensions, because
you would have to determine which points, $x,y,z$, the particles might
reach in the trajectory, whereas in one dimension you can typically
tell by simply seeing whether the kinetic energy is positive at every
point between the old position and the new position.

## The Torque, Example 2

One can write the torque about a given axis, which we will denote as $\hat{z}$, in polar coordinates, where

$$
\begin{eqnarray}
x&=&r\sin\theta\cos\phi,~~y=r\sin\theta\sin\phi,~~z=r\cos\theta,
\end{eqnarray}
$$

to find the $z$ component of the torque,

$$
\begin{eqnarray}
\tau_z&=&xF_y-yF_x\\
\nonumber
&=&-r\sin\theta\left\{\cos\phi \partial_y-\sin\phi \partial_x\right\}V(x,y,z).
\end{eqnarray}
$$

## Chain Rule and Partial Derivatives

One can use the chain rule to write the partial derivative w.r.t. $\phi$ (keeping $r$ and $\theta$ fixed),

$$
\begin{eqnarray}
\partial_\phi&=&\frac{\partial x}{\partial\phi}\partial_x+\frac{\partial_y}{\partial\phi}\partial_y
+\frac{\partial z}{\partial\phi}\partial_z\\
\nonumber
&=&-r\sin\theta\sin\phi\partial_x+\sin\theta\cos\phi\partial_y.
\end{eqnarray}
$$

Combining the two equations,

$$
\begin{eqnarray}
\tau_z&=&-\partial_\phi V(r,\theta,\phi).
\end{eqnarray}
$$

Thus, if the potential is independent of the azimuthal angle $\phi$,
there is no torque about the $z$ axis and $L_z$ is conserved.

## The Earth-Sun system

We will now venture into a study of a system which is energy
conserving. The aim is to see if we (since it is not possible to solve
the general equations analytically) we can develop stable numerical
algorithms whose results we can trust!

We solve the equations of motion numerically. We will also compute
quantities like the energy numerically.

We start with a simpler case first, the Earth-Sun system  in two dimensions only.  The gravitational force $F_G$ on the earth from the sun is

$$
\boldsymbol{F}_G=-\frac{GM_{\odot}M_E}{r^3}\boldsymbol{r},
$$

where $G$ is the gravitational constant,

$$
M_E=6\times 10^{24}\mathrm{Kg},
$$

the mass of Earth,

$$
M_{\odot}=2\times 10^{30}\mathrm{Kg},
$$

the mass of the Sun and

$$
r=1.5\times 10^{11}\mathrm{m},
$$

is the distance between Earth and the Sun. The latter defines what we call an astronomical unit **AU**.

## The Earth-Sun system, Newton's Laws

From Newton's second law we have then for the $x$ direction

$$
\frac{d^2x}{dt^2}=-\frac{F_{x}}{M_E},
$$

and

$$
\frac{d^2y}{dt^2}=-\frac{F_{y}}{M_E},
$$

for the $y$ direction.

Here we will use  that  $x=r\cos{(\theta)}$, $y=r\sin{(\theta)}$ and

$$
r = \sqrt{x^2+y^2}.
$$

We can rewrite

$$
F_{x}=-\frac{GM_{\odot}M_E}{r^2}\cos{(\theta)}=-\frac{GM_{\odot}M_E}{r^3}x,
$$

and

$$
F_{y}=-\frac{GM_{\odot}M_E}{r^2}\sin{(\theta)}=-\frac{GM_{\odot}M_E}{r^3}y,
$$

for the $y$ direction.

## The Earth-Sun system, rewriting the Equations

We can rewrite these two equations

$$
F_{x}=-\frac{GM_{\odot}M_E}{r^2}\cos{(\theta)}=-\frac{GM_{\odot}M_E}{r^3}x,
$$

and

$$
F_{y}=-\frac{GM_{\odot}M_E}{r^2}\sin{(\theta)}=-\frac{GM_{\odot}M_E}{r^3}y,
$$

as four first-order coupled differential equations

$$
\frac{dv_x}{dt}=-\frac{GM_{\odot}}{r^3}x,
$$

$$
\frac{dx}{dt}=v_x,
$$

$$
\frac{dv_y}{dt}=-\frac{GM_{\odot}}{r^3}y,
$$

$$
\frac{dy}{dt}=v_y.
$$

## Building a code for the solar system, final coupled equations

The four coupled differential equations

$$
\frac{dv_x}{dt}=-\frac{GM_{\odot}}{r^3}x,
$$

$$
\frac{dx}{dt}=v_x,
$$

$$
\frac{dv_y}{dt}=-\frac{GM_{\odot}}{r^3}y,
$$

$$
\frac{dy}{dt}=v_y,
$$

can be turned into dimensionless equations or we can introduce astronomical units with $1$ AU = $1.5\times 10^{11}$.

Using the equations from circular motion (with $r =1\mathrm{AU}$)

$$
\frac{M_E v^2}{r} = F = \frac{GM_{\odot}M_E}{r^2},
$$

we have

$$
GM_{\odot}=v^2r,
$$

and using that the velocity of Earth (assuming circular motion) is
$v = 2\pi r/\mathrm{yr}=2\pi\mathrm{AU}/\mathrm{yr}$, we have

$$
GM_{\odot}= v^2r = 4\pi^2 \frac{(\mathrm{AU})^3}{\mathrm{yr}^2}.
$$

## Building a code for the solar system, discretized equations

The four coupled differential equations can then be discretized using Euler's method as (with step length $h$)

$$
v_{x,i+1}=v_{x,i}-h\frac{4\pi^2}{r_i^3}x_i,
$$

$$
x_{i+1}=x_i+hv_{x,i},
$$

$$
v_{y,i+1}=v_{y,i}-h\frac{4\pi^2}{r_i^3}y_i,
$$

$$
y_{i+1}=y_i+hv_{y,i},
$$

## Code Example with Euler's Method

The code here implements Euler's method for the Earth-Sun system using a more compact way of representing the vectors. Alternatively, you could have spelled out all the variables $v_x$, $v_y$, $x$ and $y$ as one-dimensional arrays.

```
%matplotlib inline

# Common imports
import numpy as np
import pandas as pd
from math import *
import matplotlib.pyplot as plt
import os

# Where to save the figures and data files
PROJECT_ROOT_DIR = "Results"
FIGURE_ID = "Results/FigureFiles"
DATA_ID = "DataFiles/"

if not os.path.exists(PROJECT_ROOT_DIR):
    os.mkdir(PROJECT_ROOT_DIR)

if not os.path.exists(FIGURE_ID):
    os.makedirs(FIGURE_ID)

if not os.path.exists(DATA_ID):
    os.makedirs(DATA_ID)

def image_path(fig_id):
    return os.path.join(FIGURE_ID, fig_id)

def data_path(dat_id):
    return os.path.join(DATA_ID, dat_id)

def save_fig(fig_id):
    plt.savefig(image_path(fig_id) + ".png", format='png')

DeltaT = 0.001
#set up arrays
tfinal = 10 # in years
n = ceil(tfinal/DeltaT)
# set up arrays for t, a, v, and x
t = np.zeros(n)
v = np.zeros((n,2))
r = np.zeros((n,2))
# Initial conditions as compact 2-dimensional arrays
r0 = np.array([1.0,0.0])
v0 = np.array([0.0,2*pi])
r[0] = r0
v[0] = v0
Fourpi2 = 4*pi*pi
# Start integrating using Euler's method
for i in range(n-1):
    # Set up the acceleration
    # Here you could have defined your own function for this
    rabs = sqrt(sum(r[i]*r[i]))
    a =  -Fourpi2*r[i]/(rabs**3)
    # update velocity, time and position using Euler's forward method
    v[i+1] = v[i] + DeltaT*a
    r[i+1] = r[i] + DeltaT*v[i]
    t[i+1] = t[i] + DeltaT
# Plot position as function of time
fig, ax = plt.subplots()
#ax.set_xlim(0, tfinal)
ax.set_ylabel('y[AU]')
ax.set_xlabel('x[AU]')
ax.plot(r[:,0], r[:,1])
fig.tight_layout()
save_fig("EarthSunEuler")
plt.show()
```

## Problems with Euler's Method

We notice here that Euler's method doesn't give a stable orbit. It
means that we cannot trust Euler's method. In a deeper way, as we will
see in homework 5, Euler's method does not conserve energy. It is an
example of an integrator which is not
[symplectic](https://en.wikipedia.org/wiki/Symplectic_integrator).

Here we present thus two methods, which with simple changes allow us to avoid these pitfalls. The simplest possible extension is the so-called Euler-Cromer method.
The changes we need to make to our code are indeed marginal here.
We need simply to replace

```
    r[i+1] = r[i] + DeltaT*v[i]
```

in the above code with the velocity at the new time $t_{i+1}$

```
    r[i+1] = r[i] + DeltaT*v[i+1]
```

By this simple caveat we get stable orbits.
Below we derive the Euler-Cromer method as well as one of the most utlized algorithms for sovling the above type of problems, the so-called Velocity-Verlet method.

## Deriving the Euler-Cromer Method

Let us repeat Euler's method.
We have a differential equation

$$
\begin{equation}
y'(t_i)=f(t_i,y_i)
\label{_auto9} \tag{9}
\end{equation}
$$

and if we truncate at the first derivative, we have from the Taylor expansion

$$
\begin{equation}
y_{i+1}=y(t_i) + (\Delta t) f(t_i,y_i) + O(\Delta t^2), \label{eq:euler} \tag{10}
\end{equation}
$$

which when complemented with $t_{i+1}=t_i+\Delta t$ forms
the algorithm for the well-known Euler method.
Note that at every step we make an approximation error
of the order of $O(\Delta t^2)$, however the total error is the sum over all
steps $N=(b-a)/(\Delta t)$ for $t\in [a,b]$, yielding thus a global error which goes like
$NO(\Delta t^2)\approx O(\Delta t)$.

To make Euler's method more precise we can obviously
decrease $\Delta t$ (increase $N$), but this can lead to loss of numerical precision.
Euler's method is not recommended for precision calculation,
although it is handy to use in order to get a first
view on how a solution may look like.

Euler's method is asymmetric in time, since it uses information about the derivative at the beginning
of the time interval. This means that we evaluate the position at $y_1$ using the velocity
at $v_0$. A simple variation is to determine $x_{n+1}$ using the velocity at
$v_{n+1}$, that is (in a slightly more generalized form)

$$
\begin{equation}
y_{n+1}=y_{n}+ v_{n+1}+O(\Delta t^2)
\label{_auto10} \tag{11}
\end{equation}
$$

and

$$
\begin{equation}
v_{n+1}=v_{n}+(\Delta t) a_{n}+O(\Delta t^2).
\label{_auto11} \tag{12}
\end{equation}
$$

The acceleration $a_n$ is a function of $a_n(y_n, v_n, t_n)$ and needs to be evaluated
as well. This is the Euler-Cromer method.

**Exercise**: go back to the above code with Euler's method and add the Euler-Cromer method.

## Deriving the Velocity-Verlet Method

Let us stay with $x$ (position) and $v$ (velocity) as the quantities we are interested in.

We have the Taylor expansion for the position given by

$$
x_{i+1} = x_i+(\Delta t)v_i+\frac{(\Delta t)^2}{2}a_i+O((\Delta t)^3).
$$

The corresponding expansion for the velocity is

$$
v_{i+1} = v_i+(\Delta t)a_i+\frac{(\Delta t)^2}{2}v^{(2)}_i+O((\Delta t)^3).
$$

Via Newton's second law we have normally an analytical expression for the derivative of the velocity, namely

$$
a_i= \frac{d^2 x}{dt^2}\vert_{i}=\frac{d v}{dt}\vert_{i}= \frac{F(x_i,v_i,t_i)}{m}.
$$

If we add to this the corresponding expansion for the derivative of the velocity

$$
v^{(1)}_{i+1} = a_{i+1}= a_i+(\Delta t)v^{(2)}_i+O((\Delta t)^2)=a_i+(\Delta t)v^{(2)}_i+O((\Delta t)^2),
$$

and retain only terms up to the second derivative of the velocity since our error goes as $O(h^3)$, we have

$$
(\Delta t)v^{(2)}_i\approx a_{i+1}-a_i.
$$

We can then rewrite the Taylor expansion for the velocity as

$$
v_{i+1} = v_i+\frac{(\Delta t)}{2}\left( a_{i+1}+a_{i}\right)+O((\Delta t)^3).
$$

## The velocity Verlet method

Our final equations for the position and the velocity become then

$$
x_{i+1} = x_i+(\Delta t)v_i+\frac{(\Delta t)^2}{2}a_{i}+O((\Delta t)^3),
$$

and

$$
v_{i+1} = v_i+\frac{(\Delta t)}{2}\left(a_{i+1}+a_{i}\right)+O((\Delta t)^3).
$$

Note well that the term $a_{i+1}$ depends on the position at $x_{i+1}$. This means that you need to calculate
the position at the updated time $t_{i+1}$ before the computing the next velocity.  Note also that the derivative of the velocity at the time
$t_i$ used in the updating of the position can be reused in the calculation of the velocity update as well.

## Adding the Velocity-Verlet Method

We can now easily add the Verlet method to our original code as

```
DeltaT = 0.01
#set up arrays
tfinal = 10 # in years
n = ceil(tfinal/DeltaT)
# set up arrays for t, a, v, and x
t = np.zeros(n)
v = np.zeros((n,2))
r = np.zeros((n,2))
# Initial conditions as compact 2-dimensional arrays
r0 = np.array([1.0,0.0])
v0 = np.array([0.0,2*pi])
r[0] = r0
v[0] = v0
Fourpi2 = 4*pi*pi
# Start integrating using the Velocity-Verlet  method
for i in range(n-1):
    # Set up forces, air resistance FD, note now that we need the norm of the vecto
    # Here you could have defined your own function for this
    rabs = sqrt(sum(r[i]*r[i]))
    a =  -Fourpi2*r[i]/(rabs**3)
    # update velocity, time and position using the Velocity-Verlet method
    r[i+1] = r[i] + DeltaT*v[i]+0.5*(DeltaT**2)*a
    rabs = sqrt(sum(r[i+1]*r[i+1]))
    anew = -4*(pi**2)*r[i+1]/(rabs**3)
    v[i+1] = v[i] + 0.5*DeltaT*(a+anew)
    t[i+1] = t[i] + DeltaT
# Plot position as function of time
fig, ax = plt.subplots()
ax.set_ylabel('y[AU]')
ax.set_xlabel('x[AU]')
ax.plot(r[:,0], r[:,1])
fig.tight_layout()
save_fig("EarthSunVV")
plt.show()
```

You can easily generalize the calculation of the forces by defining a function
which takes in as input the various variables. We leave this as a challenge to you.

## Additional Material: Link between Line Integrals and Conservative forces

The concept of line integrals plays an important role in our discussion of energy conservation,
our definition of potentials and conservative forces.

Let us remind ourselves of some the basic elements (most of you may
have seen this in a calculus course under the general topic of vector
fields).

We define a path integration $C$, that is we integrate
from a point $\boldsymbol{r}_1$ to a point $\boldsymbol{r}_2$.
Let us assume that the path $C$ is represented by an arc length $s$. In three dimension we have the following representation of $C$

$$
\boldsymbol{r}(s)=x(s)\boldsymbol{e}_1+y(s)\boldsymbol{e}_2+z(s)\boldsymbol{e}_3,
$$

then our integral of a function $f(x,y,z)$ along the path $C$ is defined as

$$
\int_Cf(x,y,z)ds=\int_a^bf\left(x(s),y(s),z(s)\right)ds,
$$

where the initial and final points are $a$ and $b$, respectively.

## Exactness and Independence of Path

With the definition of a line integral, we can in tunrn set up the
theorem of independence of integration path.

Let us define
$f(x,y,z)$, $g(x,y,z)$ and $h(x,y,z)$ to be functions which are
defined and continuous in a domain $D$ in space. Then a line integral
like the above is said to be independent of path in $D$, if for every
pair of endpoints $a$ and $b$ in $D$ the value of the integral is the
same for all paths $C$ in $D$ starting from a point $a$ and ending in
a point $b$. The integral depends thus only on the integration limits
and not on the path.

## Differential Forms

An expression of the form

$$
fdx+gdy+hdz,
$$

where $f$, $g$ and $h$ are functions defined in $D$, is a called a first-order differential form
in three variables.
The form is said to be exact if it is the differential

$$
du= \frac{\partial u}{\partial x}dx+\frac{\partial u}{\partial y}dy+\frac{\partial u}{\partial z}dz,
$$

of a differentiable function $u(x,y,z)$ everywhere in $D$, that is

$$
du=fdx+gdy+hdz.
$$

It is said to be exact if and only if we can then set

$$
f=\frac{\partial u}{\partial x},
$$

and

$$
g=\frac{\partial u}{\partial y},
$$

and

$$
h=\frac{\partial u}{\partial z},
$$

everywhere in the domain $D$.

## In Vector Language

In vector language the above means that the differential form

$$
fdx+gdy+hdz,
$$

is exact in $D$ if and only if the vector function (it could be a force, or velocity, acceleration or other vectors we encounter in this course)

$$
\boldsymbol{F}=f\boldsymbol{e}_1+g\boldsymbol{e}_2+h\boldsymbol{e}_3,
$$

is the gradient of a function $u(x,y,z)$

$$
\boldsymbol{v}=\boldsymbol{\nabla}u=\frac{\partial u}{\partial x}\boldsymbol{e}_1+\frac{\partial u}{\partial y}\boldsymbol{e}_2+\frac{\partial u}{\partial z}\boldsymbol{e}_3.
$$

## Path Independence Theorem

If this is the case, we can state the path independence theorem which
states that with functions $f(x,y,z)$, $g(x,y,z)$ and $h(x,y,z)$ that fulfill the above
exactness conditions,  the line integral

$$
\int_C\left(fdx+gdy+hdz\right),
$$

is independent of path in $D$ if and only if the differential form under the integral sign is exact in $D$.

This is the path independence theorem.

We will not give a proof of the theorem. You can find this in any vector analysis chapter in a mathematics textbook.

We note however that the path integral from a point $p$ to a final point $q$ is given by

$$
\int_p^q\left(fdx+gdy+hdz\right)=\int_p^q\left(\frac{\partial u}{\partial x}dx+\frac{\partial u}{\partial y}dy+\frac{\partial u}{\partial z}dz\right)=\int_p^qdu.
$$

Assume now that we have a dependence on a variable $s$ for $x$, $y$ and $z$. We have then

$$
\int_p^qdu=\int_{s_1}^{s_2}\frac{du}{ds}ds = u(x(s),y(s),z(s))\vert_{s=s_1}^{s=s_2}=u(q)-u(p).
$$

This last equation

$$
\int_p^q\left(fdx+gdy+hdz\right)=u(q)-u(p),
$$

is the analogue of the usual formula

$$
\int_a^bf(x)dx=F(x)\vert_a^b=F(b)-F(a),
$$

with $F'(x)=f(x)$.

## Work-Energy Theorem again

We remember that a the work done by a force
$\boldsymbol{F}=f\boldsymbol{e}_1+g\boldsymbol{e}_2+h\boldsymbol{e}_3$ on a displacemnt $d\boldsymbol{r}$

$$
W=\int_C\boldsymbol{F}d\boldsymbol{r}=\int_C(fdx+gdy+hdz).
$$

From the path independence theorem, we know that this has to result in
the difference between the two endpoints only. This is exact if and
only if the force is the force $\boldsymbol{F}$ is the gradient of a scalar
function $u$. We call this scalar function, which depends only the
positions $x,y,z$ for the potential energy $V(x,y,z)=V(\boldsymbol{r})$.

We have thus

$$
\boldsymbol{F}(\boldsymbol{r})\propto \boldsymbol{\nabla}V(\boldsymbol{r}),
$$

and we define  this as

$$
\boldsymbol{F}(\boldsymbol{r})= -\boldsymbol{\nabla}V(\boldsymbol{r}).
$$

Such a force is called **a conservative force**. The above expression can be used to demonstrate
energy conservation.

## Additional Theorem

Finally we can define the criterion for exactness and independence of
path. This theorem states that if $f(x,y,z)$, $g(x,y,z)$ and
$h(x,y,z)$ are continuous functions with continuous first partial derivatives in the domain $D$,
then the line integral

$$
\int_C\left(fdx+gdy+hdz\right),
$$

is independent of path in $D$ when

$$
\frac{\partial h}{\partial y}=\frac{\partial g}{\partial z},
$$

and

$$
\frac{\partial f}{\partial z}=\frac{\partial h}{\partial x},
$$

and

$$
\frac{\partial g}{\partial x}=\frac{\partial f}{\partial y}.
$$

This leads to the **curl** of $\boldsymbol{F}$ being zero

$$
\boldsymbol{\nabla}\times\boldsymbol{F}=\boldsymbol{\nabla}\times\left(-\boldsymbol{\nabla}V(\boldsymbol{r})\right)=0!
$$

## Summarizing

A conservative force $\boldsymbol{F}$ is a defined as the partial derivative of a scalar potential which depends only on the position,

$$
\boldsymbol{F}(\boldsymbol{r})= -\boldsymbol{\nabla}V(\boldsymbol{r}).
$$

This leads to conservation of energy and a path independent line integral as long as the curl of the force is zero, that is

$$
\boldsymbol{\nabla}\times\boldsymbol{F}=\boldsymbol{\nabla}\times\left(-\boldsymbol{\nabla}V(\boldsymbol{r})\right)=0.
$$


# Lambda School Data Science Module 143

## Introduction to Bayesian Inference

!['Detector! What would the Bayesian statistician say if I asked him whether the--' [roll] 'I AM A NEUTRINO DETECTOR, NOT A LABYRINTH GUARD. SERIOUSLY, DID YOUR BRAIN FALL OUT?' [roll] '... yes.'](https://imgs.xkcd.com/comics/frequentists_vs_bayesians.png)

*[XKCD 1132](https://www.xkcd.com/1132/)*

## Prepare - Bayes' Theorem and the Bayesian mindset

Bayes' theorem possesses a near-mythical quality - a bit of math that somehow magically evaluates a situation. But this mythicalness has more to do with its reputation and advanced applications than the actual core of it - deriving it is actually remarkably straightforward.

### The Law of Total Probability

By definition, the total probability of all outcomes (events) if some variable (event space) $A$ is 1. That is:

$$P(A) = \sum_n P(A_n) = 1$$

The law of total probability takes this further, considering two variables ($A$ and $B$) and relating their marginal probabilities (their likelihoods considered independently, without reference to one another) and their conditional probabilities (their likelihoods considered jointly). A marginal probability is simply notated as e.g. $P(A)$, while a conditional probability is notated $P(A|B)$, which reads "probability of $A$ *given* $B$".

The law of total probability states:

$$P(A) = \sum_n P(A | B_n) P(B_n)$$

In words - the total probability of $A$ is equal to the sum of the conditional probability of $A$ on any given event $B_n$ times the probability of that event $B_n$, and summed over all possible events in $B$.

### The Law of Conditional Probability

What's the probability of something conditioned on something else? To determine this we have to go back to set theory and think about the intersection of sets:

The formula for actual calculation:

$$P(A|B) = \frac{P(A \cap B)}{P(B)}$$

Think of the overall rectangle as the whole probability space, $A$ as the left circle, $B$ as the right circle, and their intersection as the red area. Try to visualize the ratio being described in the above formula, and how it is different from just the $P(A)$ (not conditioned on $B$).

We can see how this relates back to the law of total probability - multiply both sides by $P(B)$ and you get $P(A|B)P(B) = P(A \cap B)$ - replaced back into the law of total probability we get $P(A) = \sum_n P(A \cap B_n)$.

This may not seem like an improvement at first, but try to relate it back to the above picture - if you think of sets as physical objects, we're saying that the total probability of $A$ given $B$ is all the little pieces of it intersected with $B$, added together. The conditional probability is then just that again, but divided by the probability of $B$ itself happening in the first place.

### Bayes Theorem

Here is is, the seemingly magic tool:

$$P(A|B) = \frac{P(B|A)P(A)}{P(B)}$$

In words - the probability of $A$ conditioned on $B$ is the probability of $B$ conditioned on $A$, times the probability of $A$ and divided by the probability of $B$. These unconditioned probabilities are referred to as "prior beliefs", and the conditioned probabilities as "updated."

Why is this important? Scroll back up to the XKCD example - the Bayesian statistician draws a less absurd conclusion because their prior belief in the likelihood that the sun will go nova is extremely low. So, even when updated based on evidence from a detector that is $35/36 = 0.972$ accurate, the prior belief doesn't shift enough to change their overall opinion.

There's many examples of Bayes' theorem - one less absurd example is to apply to [breathalyzer tests](https://www.bayestheorem.net/breathalyzer-example/). You may think that a breathalyzer test that is 100% accurate for true positives (detecting somebody who is drunk) is pretty good, but what if it also has 8% false positives (indicating somebody is drunk when they're not)? And furthermore, the rate of drunk driving (and thus our prior belief)  is 1/1000.

What is the likelihood somebody really is drunk if they test positive? Some may guess it's 92% - the difference between the true positives and the false positives. But we have a prior belief of the background/true rate of drunk driving. Sounds like a job for Bayes' theorem!

$$
\begin{aligned}
P(Drunk | Positive) &= \frac{P(Positive | Drunk)P(Drunk)}{P(Positive)} \\
&= \frac{1 \times 0.001}{0.08} \\
&= 0.0125
\end{aligned}
$$

In other words, the likelihood that somebody is drunk given they tested positive with a breathalyzer in this situation is only 1.25% - probably much lower than you'd guess. This is why, in practice, it's important to have a repeated test to confirm (the probability of two false positives in a row is $0.08 * 0.08 = 0.0064$, much lower), and Bayes' theorem has been relevant in court cases where proper consideration of evidence was important.

## Derive Baye's Rule

\begin{align}
P(A|B) &= \frac{P(A \cap B)}{P(B)}\\
\Rightarrow P(A|B)P(B) &= P(A \cap B)\\
P(B|A) &= \frac{P(B \cap A)}{P(A)}\\
\Rightarrow P(B|A)P(A) &= P(B \cap A)\\
\Rightarrow P(A|B)P(B) &= P(B|A)P(A) \\
P(A \cap B) &= P(B \cap A)\\
P(A|B) &= \frac{P(B|A) \times P(A)}{P(B)}
\end{align}

## Live Lecture - Deriving Bayes' Theorem, Calculating Bayesian Confidence

Notice that $P(A|B)$ appears in the above laws - in Bayesian terms, this is the belief in $A$ updated for the evidence $B$. So all we need to do is solve for this term to derive Bayes' theorem. Let's do it together!

```python
# Activity 2 - Use SciPy to calculate Bayesian confidence intervals
# https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bayes_mvs.html#scipy.stats.bayes_mvs

from scipy import stats
import numpy as np

np.random.seed(seed=42)

coinflips = np.random.binomial(n=1, p=.5, size=100)
print(coinflips)
```

    [0 1 1 1 0 0 0 1 1 1 0 1 1 0 0 0 0 1 0 0 1 0 0 0 0 1 0 1 1 0 1 0 0 1 1 1 0
     0 1 0 0 0 0 1 0 1 0 1 1 0 1 1 1 1 1 1 0 0 0 0 0 0 1 0 0 1 0 1 0 1 1 0 0 1
     1 1 1 0 0 0 1 1 0 0 0 0 1 1 1 0 0 1 1 1 1 0 1 0 0 0]

```python
def confidence_interval(data, confidence=.95):
  n = len(data)
  mean = sum(data)/n
  data = np.array(data)
  stderr = stats.sem(data)
  interval = stderr * stats.t.ppf((1 + confidence) / 2.0, n-1)
  return (mean , mean-interval, mean+interval)
```

```python
confidence_interval(coinflips)
```

    (0.47, 0.3704689875017368, 0.5695310124982632)

```python
help(stats.bayes_mvs)
```

    Help on function bayes_mvs in module scipy.stats.morestats:

    bayes_mvs(data, alpha=0.9)
        Bayesian confidence intervals for the mean, var, and std.

        Parameters
        ----------
        data : array_like
            Input data, if multi-dimensional it is flattened to 1-D by `bayes_mvs`.
            Requires 2 or more data points.
        alpha : float, optional
            Probability that the returned confidence interval contains
            the true parameter.

        Returns
        -------
        mean_cntr, var_cntr, std_cntr : tuple
            The three results are for the mean, variance and standard deviation,
            respectively.  Each result is a tuple of the form::

                (center, (lower, upper))

            with `center` the mean of the conditional pdf of the value given the
            data, and `(lower, upper)` a confidence interval, centered on the
            median, containing the estimate to a probability ``alpha``.

        See Also
        --------
        mvsdist

        Notes
        -----
        Each tuple of mean, variance, and standard deviation estimates represent
        the (center, (lower, upper)) with center the mean of the conditional pdf
        of the value given the data and (lower, upper) is a confidence interval
        centered on the median, containing the estimate to a probability
        ``alpha``.

        Converts data to 1-D and assumes all data has the same mean and variance.
        Uses Jeffrey's prior for variance and std.

        Equivalent to ``tuple((x.mean(), x.interval(alpha)) for x in mvsdist(dat))``

        References
        ----------
        T.E. Oliphant, "A Bayesian perspective on estimating mean, variance, and
        standard-deviation from data", https://scholarsarchive.byu.edu/facpub/278,
        2006.

        Examples
        --------
        First a basic example to demonstrate the outputs:

        >>> from scipy import stats
        >>> data = [6, 9, 12, 7, 8, 8, 13]
        >>> mean, var, std = stats.bayes_mvs(data)
        >>> mean
        Mean(statistic=9.0, minmax=(7.103650222612533, 10.896349777387467))
        >>> var
        Variance(statistic=10.0, minmax=(3.176724206..., 24.45910382...))
        >>> std
        Std_dev(statistic=2.9724954732045084, minmax=(1.7823367265645143, 4.945614605014631))

        Now we generate some normally distributed random data, and get estimates of
        mean and standard deviation with 95% confidence intervals for those
        estimates:

        >>> n_samples = 100000
        >>> data = stats.norm.rvs(size=n_samples)
        >>> res_mean, res_var, res_std = stats.bayes_mvs(data, alpha=0.95)

        >>> import matplotlib.pyplot as plt
        >>> fig = plt.figure()
        >>> ax = fig.add_subplot(111)
        >>> ax.hist(data, bins=100, density=True, label='Histogram of data')
        >>> ax.vlines(res_mean.statistic, 0, 0.5, colors='r', label='Estimated mean')
        >>> ax.axvspan(res_mean.minmax[0],res_mean.minmax[1], facecolor='r',
        ...            alpha=0.2, label=r'Estimated mean (95% limits)')
        >>> ax.vlines(res_std.statistic, 0, 0.5, colors='g', label='Estimated scale')
        >>> ax.axvspan(res_std.minmax[0],res_std.minmax[1], facecolor='g', alpha=0.2,
        ...            label=r'Estimated scale (95% limits)')

        >>> ax.legend(fontsize=10)
        >>> ax.set_xlim([-4, 4])
        >>> ax.set_ylim([0, 0.5])
        >>> plt.show()

```python
#0.37046898750173674
#0.3704689875017368

stats.bayes_mvs(coinflips, alpha=.95)
```

    (Mean(statistic=0.47, minmax=(0.37046898750173674, 0.5695310124982632)),
     Variance(statistic=0.25680412371134015, minmax=(0.1939698977025208, 0.3395533426586547)),
     Std_dev(statistic=0.5054540733507159, minmax=(0.44042013771229943, 0.5827120581030176)))

```python
coinflips_mean_dist, _, _ = stats.mvsdist(coinflips)
coinflips_mean_dist
```

    <scipy.stats._distn_infrastructure.rv_frozen at 0x2afcb4869b0>

```python
help(coinflips_mean_dist)
```

    Help on rv_frozen in module scipy.stats._distn_infrastructure object:

    class rv_frozen(builtins.object)
     |  rv_frozen(dist, *args, **kwds)
     |
     |  # Frozen RV class
     |
     |  Methods defined here:
     |
     |  __init__(self, dist, *args, **kwds)
     |      Initialize self.  See help(type(self)) for accurate signature.
     |
     |  cdf(self, x)
     |
     |  entropy(self)
     |
     |  expect(self, func=None, lb=None, ub=None, conditional=False, **kwds)
     |
     |  interval(self, alpha)
     |
     |  isf(self, q)
     |
     |  logcdf(self, x)
     |
     |  logpdf(self, x)
     |
     |  logpmf(self, k)
     |
     |  logsf(self, x)
     |
     |  mean(self)
     |
     |  median(self)
     |
     |  moment(self, n)
     |
     |  pdf(self, x)
     |
     |  pmf(self, k)
     |
     |  ppf(self, q)
     |
     |  rvs(self, size=None, random_state=None)
     |
     |  sf(self, x)
     |
     |  stats(self, moments='mv')
     |
     |  std(self)
     |
     |  var(self)
     |
     |  ----------------------------------------------------------------------
     |  Data descriptors defined here:
     |
     |  __dict__
     |      dictionary for instance variables (if defined)
     |
     |  __weakref__
     |      list of weak references to the object (if defined)
     |
     |  random_state

```python
coinflips_mean_dist.rvs(100)
```

    array([0.47447628, 0.51541425, 0.54722018, 0.4589882 , 0.51501386,
           0.53819192, 0.43382292, 0.53546659, 0.47026173, 0.44967562,
           0.4621107 , 0.42691904, 0.37324325, 0.47531437, 0.46052277,
           0.48711257, 0.52456771, 0.43332181, 0.49545882, 0.44671454,
           0.47520117, 0.47047251, 0.41828918, 0.50159477, 0.42965501,
           0.45273383, 0.48045849, 0.45342529, 0.48238344, 0.53966291,
           0.48230241, 0.48073422, 0.48553525, 0.47962228, 0.41274185,
           0.42892633, 0.5170948 , 0.42678096, 0.42249309, 0.51499109,
           0.47059199, 0.39903942, 0.41790336, 0.46406817, 0.42232382,
           0.42163269, 0.47848227, 0.48232842, 0.4731858 , 0.51077244,
           0.3957508 , 0.48504646, 0.49014295, 0.53252732, 0.45495376,
           0.47883978, 0.60393033, 0.4492549 , 0.44797902, 0.54782121,
           0.43380002, 0.5760073 , 0.36941266, 0.44467418, 0.4939245 ,
           0.45278835, 0.55635162, 0.48695459, 0.39080983, 0.45948606,
           0.2941779 , 0.35950718, 0.44805696, 0.4725126 , 0.42218381,
           0.45985418, 0.47545393, 0.44317753, 0.46267013, 0.4458753 ,
           0.44204707, 0.51334913, 0.50914181, 0.49923748, 0.46895674,
           0.43892798, 0.45984946, 0.44984632, 0.53560791, 0.45865723,
           0.48646824, 0.55937503, 0.41464303, 0.50701457, 0.46934196,
           0.37681534, 0.42748113, 0.49812825, 0.48278895, 0.4964763 ])

```python
import pandas as pd
pd.DataFrame(coinflips).describe()
```

      0

      count
      100.000000

      mean
      0.470000

      std
      0.501614

      min
      0.000000

      25%
      0.000000

      50%
      0.000000

      75%
      1.000000

      max
      1.000000

## Assignment - Code it up!

Most of the above was pure math - now write Python code to reproduce the results! This is purposefully open ended - you'll have to think about how you should represent probabilities and events. You can and should look things up, and as a stretch goal - refactor your code into helpful reusable functions!

Specific goals/targets:

1. Write a function `def prob_drunk_given_positive(prob_drunk_prior, prob_positive, prob_positive_drunk)` that reproduces the example from lecture, and use it to calculate and visualize a range of situations
2. Explore `scipy.stats.bayes_mvs` - read its documentation, and experiment with it on data you've tested in other ways earlier this week
3. Create a visualization comparing the results of a Bayesian approach to a traditional/frequentist approach
4. In your own words, summarize the difference between Bayesian and Frequentist statistics

If you're unsure where to start, check out [this blog post of Bayes theorem with Python](https://dataconomy.com/2015/02/introduction-to-bayes-theorem-with-python/) - you could and should create something similar!

Stretch goals:

- Apply a Bayesian technique to a problem you previously worked (in an assignment or project work) on from a frequentist (standard) perspective
- Check out [PyMC3](https://docs.pymc.io/) (note this goes beyond hypothesis tests into modeling) - read the guides and work through some examples
- Take PyMC3 further - see if you can build something with it!

```python
import pandas as pd
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
import scipy.stats as stats
```

## Problem 1

```python
def prob_drunk_given_positive(prob_positive_drunk, prob_drunk_prior, prob_positive):
  test1 = (prob_positive_drunk * prob_drunk_prior) / prob_positive
  test2 = (prob_positive_drunk * prob_drunk_prior) / (prob_positive)**2
  result1 = 'Probability a person is drunk, given one failed breathalyzer test:', test1
  result2 = 'Probability a person is drunk, given two failed breathalyzer tests:', test2
  return result1, result2

prob_drunk_given_positive(1, 0.001, 0.08)
```

    (('Probability a person is drunk, given one failed breathalyzer test:',
      0.0125),
     ('Probability a person is drunk, given two failed breathalyzer tests:',
      0.15625))

## Problem 2

```python
url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/voting-records/house-votes-84.data'
columns = ['class_name', 'handicapped_infants', 'water_project_cost_sharing', 'adoption_of_the_budget_resolution',
          'physician_fee_freeze', 'el_salvador_aid', 'religious_groups_in_schools', 'anti_satellite_test_ban',
          'aid_to_nicaraguan_contras', 'mx_missile', 'immigration', 'synfuels_corporation_cutback',
          'education_spending', 'superfund_right_to_sue', 'crime', 'duty_free_exports',
          'export_administration_act_south_africa']

df = pd.read_csv(url, header=None, names=columns, na_values='?').set_index('class_name')
df.head()
```

      handicapped_infants
      water_project_cost_sharing
      adoption_of_the_budget_resolution
      physician_fee_freeze
      el_salvador_aid
      religious_groups_in_schools
      anti_satellite_test_ban
      aid_to_nicaraguan_contras
      mx_missile
      immigration
      synfuels_corporation_cutback
      education_spending
      superfund_right_to_sue
      crime
      duty_free_exports
      export_administration_act_south_africa

      class_name

      republican
      n
      y
      n
      y
      y
      y
      n
      n
      n
      y
      NaN
      y
      y
      y
      n
      y

      republican
      n
      y
      n
      y
      y
      y
      n
      n
      n
      n
      n
      y
      y
      y
      n
      NaN

      democrat
      NaN
      y
      y
      NaN
      y
      y
      n
      n
      n
      n
      y
      n
      y
      y
      n
      n

      democrat
      n
      y
      y
      n
      NaN
      y
      n
      n
      n
      n
      y
      n
      y
      n
      n
      y

      democrat
      y
      y
      y
      n
      y
      y
      n
      n
      n
      n
      y
      NaN
      y
      y
      y
      y

```python
# Removed nan values because bayes_mvs doesn't have an omit param
df = df.dropna()
```

```python
df = df.replace({'y': 1, 'n': 0})
```

```python
rep = df.loc['republican']
dem = df.loc['democrat']
```

```python
rep.head()
```

      handicapped_infants
      water_project_cost_sharing
      adoption_of_the_budget_resolution
      physician_fee_freeze
      el_salvador_aid
      religious_groups_in_schools
      anti_satellite_test_ban
      aid_to_nicaraguan_contras
      mx_missile
      immigration
      synfuels_corporation_cutback
      education_spending
      superfund_right_to_sue
      crime
      duty_free_exports
      export_administration_act_south_africa

      class_name

      republican
      0
      1
      0
      1
      1
      1
      0
      0
      0
      0
      0
      1
      1
      1
      0
      1

      republican
      1
      0
      0
      1
      1
      0
      1
      1
      1
      0
      0
      1
      1
      1
      0
      1

      republican
      0
      1
      0
      1
      1
      1
      0
      0
      0
      0
      0
      1
      1
      1
      0
      0

      republican
      0
      1
      0
      1
      1
      1
      0
      0
      0
      0
      0
      1
      1
      1
      0
      1

      republican
      0
      1
      0
      1
      1
      1
      0
      0
      0
      0
      0
      1
      1
      1
      0
      0

```python
dem.head()
```

      handicapped_infants
      water_project_cost_sharing
      adoption_of_the_budget_resolution
      physician_fee_freeze
      el_salvador_aid
      religious_groups_in_schools
      anti_satellite_test_ban
      aid_to_nicaraguan_contras
      mx_missile
      immigration
      synfuels_corporation_cutback
      education_spending
      superfund_right_to_sue
      crime
      duty_free_exports
      export_administration_act_south_africa

      class_name

      democrat
      0
      1
      1
      0
      1
      1
      0
      0
      0
      0
      0
      0
      1
      1
      1
      1

      democrat
      1
      1
      1
      0
      0
      0
      1
      1
      1
      0
      1
      0
      0
      0
      1
      1

      democrat
      1
      1
      1
      0
      0
      0
      1
      1
      1
      0
      0
      0
      0
      0
      1
      1

      democrat
      1
      0
      1
      0
      0
      0
      1
      1
      1
      1
      0
      0
      0
      0
      1
      1

      democrat
      1
      0
      1
      0
      0
      0
      1
      1
      1
      0
      1
      0
      0
      0
      1
      1

```python
def confidence_interval(data, confidence=0.95):
  '''
  Calculate a confidence interval around a sample mean for given data.
  using t-distribution and two tailed test, default 95% confidence.

  Arguments:
  data = iterable (list or np array) of sample observations
  confidence - level of confidence for interval
  '''

  data = np.array(data)
  mean = np.mean(data)
  n = len(data)
  stderr = stats.sem(data)
  interval = stderr * stats.t.ppf((1 + confidence) / 2., n-1)
  return (mean, mean - interval, mean + interval)

def report_confidence_interval(confidence_interval):
  '''
  Return a string with a pretty report of a confidence interval

  Arguments:
    confidence_interval - a tuple of (mean, lower bount, upper bound)

   Returns:
    None, but prints to screen the report
    '''
#   print('Mean: {}'.format(confidence_interval[0]))
#   print('Lower bound: {}'.format(confidence_interval[1]))
#   print('Upper bound: {}'.format(confidence_interval[2]))
  s = 'our mean lies in the interval [{:.2}, {:.2}]'.format(
    confidence_interval[1], confidence_interval[2])
  return s
```

```python
stats.bayes_mvs(rep['handicapped_infants'], alpha=0.95)
```

    (Mean(statistic=0.21296296296296297, minmax=(0.13450349074958223, 0.2914224351763437)),
     Variance(statistic=0.1723985890652557, minmax=(0.13163384272877396, 0.22552107883595443)),
     Std_dev(statistic=0.4142216885759803, minmax=(0.3628137851967231, 0.4748905967019713)))

```python
confidence_interval(rep['handicapped_infants'])
```

    (0.21296296296296297, 0.13450349074958223, 0.2914224351763437)

```python
rep.describe().T
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      handicapped_infants
      108.0
      0.212963
      0.411310
      0.0
      0.0
      0.0
      0.0
      1.0

      water_project_cost_sharing
      108.0
      0.472222
      0.501555
      0.0
      0.0
      0.0
      1.0
      1.0

      adoption_of_the_budget_resolution
      108.0
      0.157407
      0.365882
      0.0
      0.0
      0.0
      0.0
      1.0

      physician_fee_freeze
      108.0
      0.990741
      0.096225
      0.0
      1.0
      1.0
      1.0
      1.0

      el_salvador_aid
      108.0
      0.953704
      0.211106
      0.0
      1.0
      1.0
      1.0
      1.0

      religious_groups_in_schools
      108.0
      0.870370
      0.337461
      0.0
      1.0
      1.0
      1.0
      1.0

      anti_satellite_test_ban
      108.0
      0.268519
      0.445255
      0.0
      0.0
      0.0
      1.0
      1.0

      aid_to_nicaraguan_contras
      108.0
      0.148148
      0.356903
      0.0
      0.0
      0.0
      0.0
      1.0

      mx_missile
      108.0
      0.138889
      0.347443
      0.0
      0.0
      0.0
      0.0
      1.0

      immigration
      108.0
      0.574074
      0.496788
      0.0
      0.0
      1.0
      1.0
      1.0

      synfuels_corporation_cutback
      108.0
      0.157407
      0.365882
      0.0
      0.0
      0.0
      0.0
      1.0

      education_spending
      108.0
      0.851852
      0.356903
      0.0
      1.0
      1.0
      1.0
      1.0

      superfund_right_to_sue
      108.0
      0.842593
      0.365882
      0.0
      1.0
      1.0
      1.0
      1.0

      crime
      108.0
      0.981481
      0.135445
      0.0
      1.0
      1.0
      1.0
      1.0

      duty_free_exports
      108.0
      0.111111
      0.315735
      0.0
      0.0
      0.0
      0.0
      1.0

      export_administration_act_south_africa
      108.0
      0.666667
      0.473602
      0.0
      0.0
      1.0
      1.0
      1.0

```python
stats.bayes_mvs(dem['handicapped_infants'], alpha=0.95)
```

    (Mean(statistic=0.5887096774193549, minmax=(0.5008854514528095, 0.6765339033859002)),
     Variance(statistic=0.24813383097840572, minmax=(0.1929709352919263, 0.3187452362753357)),
     Std_dev(statistic=0.4971022146015008, minmax=(0.4392845721077925, 0.5645752706905747)))

### Tuple unpacking: this will allow me to add the bayesian mean, upper bound, and lower bound to the comparison table for problem 3.

```python
a, b, c = stats.bayes_mvs(dem['handicapped_infants'], alpha=0.95)
```

```python
print(a)
print(b)
print(c)
```

    Mean(statistic=0.5887096774193549, minmax=(0.5008854514528095, 0.6765339033859002))
    Variance(statistic=0.24813383097840572, minmax=(0.1929709352919263, 0.3187452362753357))
    Std_dev(statistic=0.4971022146015008, minmax=(0.4392845721077925, 0.5645752706905747))

```python
d, e = a
```

```python
print(d)
print(e)
```

    0.5887096774193549
    (0.5008854514528095, 0.6765339033859002)

```python
f,g = e
print(f)
print(g)
```

    0.5008854514528095
    0.6765339033859002

```python
stats.ttest_1samp(dem['handicapped_infants'], 0.588710)
```

    Ttest_1sampResult(statistic=-7.270529297663421e-06, pvalue=0.9999942107355586)

```python
confidence_interval(dem['handicapped_infants'])
```

    (0.5887096774193549, 0.5008854514528094, 0.6765339033859004)

```python
dem.describe().T
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      handicapped_infants
      124.0
      0.588710
      0.494064
      0.0
      0.0
      1.0
      1.0
      1.0

      water_project_cost_sharing
      124.0
      0.451613
      0.499672
      0.0
      0.0
      0.0
      1.0
      1.0

      adoption_of_the_budget_resolution
      124.0
      0.854839
      0.353692
      0.0
      1.0
      1.0
      1.0
      1.0

      physician_fee_freeze
      124.0
      0.048387
      0.215453
      0.0
      0.0
      0.0
      0.0
      1.0

      el_salvador_aid
      124.0
      0.201613
      0.402832
      0.0
      0.0
      0.0
      0.0
      1.0

      religious_groups_in_schools
      124.0
      0.443548
      0.498818
      0.0
      0.0
      0.0
      1.0
      1.0

      anti_satellite_test_ban
      124.0
      0.766129
      0.425008
      0.0
      1.0
      1.0
      1.0
      1.0

      aid_to_nicaraguan_contras
      124.0
      0.830645
      0.376587
      0.0
      1.0
      1.0
      1.0
      1.0

      mx_missile
      124.0
      0.790323
      0.408730
      0.0
      1.0
      1.0
      1.0
      1.0

      immigration
      124.0
      0.532258
      0.500983
      0.0
      0.0
      1.0
      1.0
      1.0

      synfuels_corporation_cutback
      124.0
      0.508065
      0.501963
      0.0
      0.0
      1.0
      1.0
      1.0

      education_spending
      124.0
      0.129032
      0.336596
      0.0
      0.0
      0.0
      0.0
      1.0

      superfund_right_to_sue
      124.0
      0.290323
      0.455753
      0.0
      0.0
      0.0
      1.0
      1.0

      crime
      124.0
      0.346774
      0.477874
      0.0
      0.0
      0.0
      1.0
      1.0

      duty_free_exports
      124.0
      0.596774
      0.492535
      0.0
      0.0
      1.0
      1.0
      1.0

      export_administration_act_south_africa
      124.0
      0.943548
      0.231728
      0.0
      1.0
      1.0
      1.0
      1.0

## Problem 3

```python
table = pd.DataFrame()

def comparison_table(rep_df, dem_df):
  confidence = 0.95
  for issue in rep_df.describe():
    table.loc[issue, 'dem_mean'] = dem_df[issue].mean()
    table.loc[issue, 'rep_mean'] = rep_df[issue].mean()
    table.loc[issue, 'dem_interval'] = stats.sem(dem_df[issue])*stats.t.ppf((1+confidence)/2, dem_df[issue].size-1)
    table.loc[issue, 'rep_interval'] = stats.sem(rep_df[issue])*stats.t.ppf((1+confidence)/2, rep_df[issue].size-1)
    table.loc[issue, 'dem_ub_ci'] = table.loc[issue, 'dem_mean'] + table.loc[issue, 'dem_interval']
    table.loc[issue, 'rep_ub_ci'] = table.loc[issue, 'rep_mean'] + table.loc[issue, 'rep_interval']
    table.loc[issue, 'dem_lb_ci'] = table.loc[issue, 'dem_mean'] - table.loc[issue, 'dem_interval']
    table.loc[issue, 'rep_lb_ci'] = table.loc[issue, 'rep_mean'] - table.loc[issue, 'rep_interval']
    dem_bayes = stats.bayes_mvs(dem_df[issue], alpha=0.95)
    dem_a, dem_b, dem_c = dem_bayes
    dem_d, dem_e = dem_a
    dem_f, dem_g = dem_e
    table.loc[issue, 'dem_bayes_mean'] = dem_d
    table.loc[issue, 'dem_bayes_ub'] = dem_g
    table.loc[issue, 'dem_bayes_lb'] = dem_f
    rep_bayes = stats.bayes_mvs(rep_df[issue], alpha=0.95)
    rep_a, rep_b, rep_c = rep_bayes
    rep_d, rep_e = rep_a
    rep_f, rep_g = rep_e
    table.loc[issue, 'rep_bayes_mean'] = rep_d
    table.loc[issue, 'rep_bayes_ub'] = rep_g
    table.loc[issue, 'rep_bayes_lb'] = rep_f
  return table

comparison_table(rep, dem)
```

      dem_mean
      rep_mean
      dem_interval
      rep_interval
      dem_ub_ci
      rep_ub_ci
      dem_lb_ci
      rep_lb_ci
      dem_bayes_mean
      dem_bayes_ub
      dem_bayes_lb
      rep_bayes_mean
      rep_bayes_ub
      rep_bayes_lb

      handicapped_infants
      0.588710
      0.212963
      0.087824
      0.078459
      0.676534
      0.291422
      0.500885
      0.134503
      0.588710
      0.676534
      0.500885
      0.212963
      0.291422
      0.134503

      water_project_cost_sharing
      0.451613
      0.472222
      0.088821
      0.095674
      0.540434
      0.567896
      0.362792
      0.376548
      0.451613
      0.540434
      0.362792
      0.472222
      0.567896
      0.376548

      adoption_of_the_budget_resolution
      0.854839
      0.157407
      0.062872
      0.069794
      0.917711
      0.227201
      0.791967
      0.087614
      0.854839
      0.917711
      0.791967
      0.157407
      0.227201
      0.087614

      physician_fee_freeze
      0.048387
      0.990741
      0.038299
      0.018355
      0.086686
      1.009096
      0.010088
      0.972385
      0.048387
      0.086686
      0.010088
      0.990741
      1.009096
      0.972385

      el_salvador_aid
      0.201613
      0.953704
      0.071607
      0.040269
      0.273220
      0.993973
      0.130006
      0.913434
      0.201613
      0.273220
      0.130006
      0.953704
      0.993973
      0.913434

      religious_groups_in_schools
      0.443548
      0.870370
      0.088669
      0.064372
      0.532218
      0.934743
      0.354879
      0.805998
      0.443548
      0.532218
      0.354879
      0.870370
      0.934743
      0.805998

      anti_satellite_test_ban
      0.766129
      0.268519
      0.075549
      0.084935
      0.841678
      0.353453
      0.690580
      0.183584
      0.766129
      0.841678
      0.690580
      0.268519
      0.353453
      0.183584

      aid_to_nicaraguan_contras
      0.830645
      0.148148
      0.066942
      0.068081
      0.897587
      0.216229
      0.763704
      0.080067
      0.830645
      0.897587
      0.763704
      0.148148
      0.216229
      0.080067

      mx_missile
      0.790323
      0.138889
      0.072655
      0.066276
      0.862978
      0.205165
      0.717667
      0.072612
      0.790323
      0.862978
      0.717667
      0.138889
      0.205165
      0.072612

      immigration
      0.532258
      0.574074
      0.089054
      0.094765
      0.621312
      0.668839
      0.443204
      0.479309
      0.532258
      0.621312
      0.443204
      0.574074
      0.668839
      0.479309

      synfuels_corporation_cutback
      0.508065
      0.157407
      0.089228
      0.069794
      0.597293
      0.227201
      0.418836
      0.087614
      0.508065
      0.597293
      0.418836
      0.157407
      0.227201
      0.087614

      education_spending
      0.129032
      0.851852
      0.059833
      0.068081
      0.188865
      0.919933
      0.069199
      0.783771
      0.129032
      0.188865
      0.069199
      0.851852
      0.919933
      0.783771

      superfund_right_to_sue
      0.290323
      0.842593
      0.081014
      0.069794
      0.371337
      0.912386
      0.209309
      0.772799
      0.290323
      0.371337
      0.209309
      0.842593
      0.912386
      0.772799

      crime
      0.346774
      0.981481
      0.084946
      0.025837
      0.431721
      1.007318
      0.261828
      0.955645
      0.346774
      0.431721
      0.261828
      0.981481
      1.007318
      0.955645

      duty_free_exports
      0.596774
      0.111111
      0.087553
      0.060228
      0.684327
      0.171339
      0.509222
      0.050883
      0.596774
      0.684327
      0.509222
      0.111111
      0.171339
      0.050883

      export_administration_act_south_africa
      0.943548
      0.666667
      0.041192
      0.090342
      0.984740
      0.757009
      0.902357
      0.576325
      0.943548
      0.984740
      0.902357
      0.666667
      0.757009
      0.576325

```python
table = table.reset_index()
table.head()
```

      index
      dem_mean
      rep_mean
      dem_interval
      rep_interval
      dem_ub_ci
      rep_ub_ci
      dem_lb_ci
      rep_lb_ci
      dem_bayes_mean
      dem_bayes_ub
      dem_bayes_lb
      rep_bayes_mean
      rep_bayes_ub
      rep_bayes_lb

      0
      handicapped_infants
      0.588710
      0.212963
      0.087824
      0.078459
      0.676534
      0.291422
      0.500885
      0.134503
      0.588710
      0.676534
      0.500885
      0.212963
      0.291422
      0.134503

      1
      water_project_cost_sharing
      0.451613
      0.472222
      0.088821
      0.095674
      0.540434
      0.567896
      0.362792
      0.376548
      0.451613
      0.540434
      0.362792
      0.472222
      0.567896
      0.376548

      2
      adoption_of_the_budget_resolution
      0.854839
      0.157407
      0.062872
      0.069794
      0.917711
      0.227201
      0.791967
      0.087614
      0.854839
      0.917711
      0.791967
      0.157407
      0.227201
      0.087614

      3
      physician_fee_freeze
      0.048387
      0.990741
      0.038299
      0.018355
      0.086686
      1.009096
      0.010088
      0.972385
      0.048387
      0.086686
      0.010088
      0.990741
      1.009096
      0.972385

      4
      el_salvador_aid
      0.201613
      0.953704
      0.071607
      0.040269
      0.273220
      0.993973
      0.130006
      0.913434
      0.201613
      0.273220
      0.130006
      0.953704
      0.993973
      0.913434

```python
rep_freq = table[['index', 'rep_mean', 'rep_lb_ci', 'rep_ub_ci']]
dem_freq = table[['index', 'dem_mean', 'dem_lb_ci', 'dem_ub_ci']]
rep_bayes = table[['index', 'rep_bayes_mean', 'rep_bayes_lb', 'rep_bayes_ub']]
dem_bayes = table[['index', 'dem_bayes_mean', 'dem_bayes_lb', 'dem_bayes_ub']]
```

```python
# Plot frequentist approach
dem_means, dem_std = dem_freq['dem_mean'], (dem_freq['dem_ub_ci'] - dem_freq['dem_mean']);
rep_means, rep_std = rep_freq['rep_mean'], (rep_freq['rep_ub_ci'] - rep_freq['rep_mean']);

ind = np.arange(len(dem_freq))
width = 0.3

# create plot
fig, ax = plt.subplots(figsize=(25, 9))

dem_rects = ax.bar(ind - width/2, dem_means, width, yerr=dem_std,
                      color='blue', label='House Democrats');
rep_rects = ax.bar(ind + width/2, rep_means, width, yerr=rep_std,
                      color='red', label='House Republicans');

# labeling
ax.set_title('Distribution of House of Representatives Voting in 1984', fontsize=18);
ax.set_ylabel('Probability to Vote Yes', fontsize=14);
ax.set_xticks(ind);
ax.set_xticklabels(('handicapped_infants', 'water_project_cost_sharing', 'adoption_of_the_budget_resolution',
          'physician_fee_freeze', 'el_salvador_aid', 'religious_groups_in_schools', 'anti_satellite_test_ban',
          'aid_to_nicaraguan_contras', 'mx_missile', 'immigration', 'synfuels_corporation_cutback',
          'education_spending', 'superfund_right_to_sue', 'crime', 'duty_free_exports',
          'export_administration_act_south_africa'));
ax.set_xticklabels(ax.get_xticklabels(), rotation=60, horizontalalignment='right', fontsize=14);
ax.legend();

```

```python
# Plot frequentist approach
dem_means, dem_std = dem_bayes['dem_bayes_mean'], (dem_bayes['dem_bayes_ub'] - dem_bayes['dem_bayes_mean']);
rep_means, rep_std = rep_bayes['rep_bayes_mean'], (rep_bayes['rep_bayes_ub'] - rep_bayes['rep_bayes_mean']);

ind = np.arange(len(dem_bayes))
width = 0.3

# create plot
fig, ax = plt.subplots(figsize=(25, 9))

dem_rects = ax.bar(ind - width/2, dem_means, width, yerr=dem_std,
                      color='blue', label='House Democrats');
rep_rects = ax.bar(ind + width/2, rep_means, width, yerr=rep_std,
                      color='red', label='House Republicans');

# labeling
ax.set_title('Distribution of House of Representatives Voting in 1984', fontsize=18);
ax.set_ylabel('Probability to Vote Yes', fontsize=14);
ax.set_xticks(ind);
ax.set_xticklabels(('handicapped_infants', 'water_project_cost_sharing', 'adoption_of_the_budget_resolution',
          'physician_fee_freeze', 'el_salvador_aid', 'religious_groups_in_schools', 'anti_satellite_test_ban',
          'aid_to_nicaraguan_contras', 'mx_missile', 'immigration', 'synfuels_corporation_cutback',
          'education_spending', 'superfund_right_to_sue', 'crime', 'duty_free_exports',
          'export_administration_act_south_africa'));
ax.set_xticklabels(ax.get_xticklabels(), rotation=60, horizontalalignment='right', fontsize=14);
ax.legend();

```

## Problem 4

The results from the frequentist approach and the Bayesian approach were very similar, but had slight differences. Those differences could be enough to make the frequentist approach less reliable, depending on the data set. For data sets that require high precision, you should use the Baeysian approach.

## Resources

- [Worked example of Bayes rule calculation](https://en.wikipedia.org/wiki/Bayes'_theorem#Examples) (helpful as it fully breaks out the denominator)
- [Source code for mvsdist in scipy](https://github.com/scipy/scipy/blob/90534919e139d2a81c24bf08341734ff41a3db12/scipy/stats/morestats.py#L139)