###### Content under Creative Commons Attribution license CC-BY 4.0, code under BSD 3-Clause License © 2019 by D. Koehn, notebook style sheet by L.A. Barba, N.C. Clementi

```python
# Execute this cell to load the notebook's style sheet, then ignore it
from IPython.core.display import HTML
css_file = '../style/custom.css'
HTML(open(css_file, "r").read())
```

<link href="https://fonts.googleapis.com/css?family=Merriweather:300,300i,400,400i,700,700i,900,900i" rel='stylesheet' >
<link href="https://fonts.googleapis.com/css?family=Source+Sans+Pro:300,300i,400,400i,700,700i" rel='stylesheet' >
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' >
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 800px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.5em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Source Sans Pro', sans-serif;
    line-height: 140%;
    font-size: 110%;
    width:680px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Merriweather', serif;
    font-style:regular;
    font-weight: bold;
    font-size: 250%;
    line-height: 100%;
    color: #004065;
    margin-bottom: 1em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Merriweather', serif;
    font-weight: bold;
    font-size: 180%;
    line-height: 100%;
    color: #0096d6;
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Merriweather', serif;
	font-size: 150%;
    margin-top:12px;
    margin-bottom: 3px;
    font-style: regular;
    color: #008367;
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Merriweather', serif;
    font-weight: 300;
    font-size: 100%;
    line-height: 120%;
    text-align: left;
    width:500px;
    margin-top: 1em;
    margin-bottom: 2em;
    margin-left: 80pt;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Source Sans Pro', sans-serif;
    font-weight: regular;
    font-size: 130%;
    color: #e31937;
    font-style: italic;
    margin-bottom: .5em;
    margin-top: 1em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'Source Code Pro', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

# Exercise: How to sail without wind

Imagine, the Bsc-students of the "Differential Equations in the Earth System" course are organizing a sailing trip in the Kiel Bay area and baltic sea. Unfortunately, the strong wind gusts predicted by the meteorologists, become not even a small breeze. Sometimes even physicists are not able to predict the future. We will learn why in the next lecture.

Fortunately, the oceanographers can deliver sea current data of the specified area. So how can the students sail without wind and stay on course? By letting their thoughts and boat drift and solving the simplest, uncoupled ordinary differential equation, I can imagine.

## Governing equations

The velocity vector field ${\bf{V}} = (v_x,v_y)^T$ is componentwise related to the spatial coordinates ${\bf{x}} = (x,y)^T$ by

\begin{equation}
v_x = \frac{dx}{dt},\; v_y = \frac{dy}{dt}
\end{equation}

To estimate the drift or **streamline** of our boat in the velocity vector field $\bf{V}$, starting from an initial position ${\bf{x_0}} = (x_0,y_0)^T$, we have to solve the uncoupled ordinary differential equations using the finite difference method introduced at the beginning of this class.

Approximating the temporal derivatives in eqs. (1) using the **backward FD operator**

\begin{equation}
\frac{df}{dt} \approx \frac{f(t)-f(t-dt)}{dt} \notag
\end{equation}

with the time sample interval $dt$ leads to

\begin{equation}
\begin{split}
v_x &= \frac{x(t)-x(t-dt)}{dt}\\
v_y &= \frac{y(t)-y(t-dt)}{dt}\\
\end{split}
\notag
\end{equation}

After solving for $x(t), y(t)$, we get the **explicit time integration scheme**:

\begin{equation}
\begin{split}
x(t) &= x(t-dt) + dt\; v_x\\
y(t) &= y(t-dt) + dt\; v_y\\
\end{split}
\notag
\end{equation}

and by introducing a temporal dicretization $t^n = n * dt$ with $n \in [0,1,...,nt]$, where $nt$ denotes the maximum time steps, the final FD code becomes:

\begin{equation}
\begin{split}
x^n &= x^{n-1} + dt\; v_x^{n-1}\\
y^n &= y^{n-1} + dt\; v_y^{n-1}\\
\end{split}
\end{equation}

These equations simply state, that we can extrapolate the next position of our boat $(x^{(n)},y^{(n)})^T$ in the velocity vector field based on the position at a previous time step $(x^{(n-1)},y^{(n-1)})^T$, the velocity field at this previous position $(v_x^{(n-1)},v_y^{(n-1)})^T$ and a predefined time step $dt$. Before implementing the FD scheme in Python, let 's try to find a simple velocity vector field ...

## Boring velocity vector field

We should start with a simple, boring velocity vector field, where we can easily predict the drift of the boat. Let's take - this:

\begin{equation}
{\bf{V}} = (y,-x)^T \notag
\end{equation}

and visualize it with Matplotlib using a `Streamplot`. First, we load all required libraries ...

```python
# Import Libraries
# ----------------
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from pylab import rcParams

# Ignore Warning Messages
# -----------------------
import warnings
warnings.filterwarnings("ignore")
```

... and define the coordinates for the `Streamplot`:

```python
dh = 50.
x1 = -1000.
x2 = 1000.
X, Y = np.meshgrid(np.arange(x1, x2, dh), np.arange(x1, x2, dh))
```

For more flexibility and avoid code redundances later on, we write a short function, which evaluates the velocity components $(v_x,v_y)^T$ at a given position $(x,y)^T$

```python
# compute velocity components V = (vx,vy)^T at position x,y
def vel_xy(x,y):

    vx = y / 1000.
    vy = -x / 1000.

    return vx, vy
```

After these preparations, we can plot the velocity vector field

```python
# Define figure size
rcParams['figure.figsize'] = 8, 8

fig1, ax1 = plt.subplots()

# Define vector field components for coordinates X,Y
VX,VY = vel_xy(X,Y)

ax1.set_title(r'Plot of velocity vector field $V=(y,-x)^T$')
plt.axis('equal')
Q = ax1.streamplot(X,Y,VX,VY)

plt.xlabel('x [m]')
plt.ylabel('y [m]')

plt.savefig('Plot_vector_field_V_boring.pdf', bbox_inches='tight', format='pdf')
plt.show()
```

So the velocity vector field ${\bf{V}} = (y,-x)^T$ is simply a large vortex with zero velocity at the origin and linear increasing velocities with distance.

### Sailing in the boring vector field $V =(y,-x)^T$

Next, we want to predict our sailing course in this large vortex. Eventhough it is unrealistic, we assume, that such a large vortex exists in the [Kiel Fjord](https://en.wikipedia.org/wiki/Kieler_F%C3%B6rde#/media/File:Kiel_Luftaufnahme.JPG), maybe related to some suspicous, top secret activity in the Kiel military harbor.

##### Exercise 1

Complete the following Python code `sailing_boring`, to predict the sailing course in the boring velocity vector field $V =(y,-x)^T$. Most of the code is already implemented, you only have to add the FD solution of the uncoupled, ordinary differential equations (2):

```python
def sailing_boring(tmax, dt, x0, y0):

    # Compute number of time steps based on tmax and dt
    nt = (int)(tmax/dt)

    # vectors for storage of x, y positions
    x  = np.zeros(nt + 1)
    y  = np.zeros(nt + 1)

    # define initial position
    x[0] = x0
    y[0] = y0

    # start time stepping over time samples n
    for n in range(1,nt + 1):

        # compute velocity components at current position
        vx, vy = vel_xy(x[n-1],y[n-1])

        # compute new position using FD approximation of time derivative
        # ADD FD SOLUTION OF THE UNCOUPLED, ORDINARY DIFFERENTIAL EQUATIONS (2) HERE!
        x[n] =
        y[n] =

    # Define figure size
    rcParams['figure.figsize'] = 8, 8

    fig1, ax1 = plt.subplots()

    # Define vector field components for Streamplot
    VX,VY = vel_xy(X,Y)

    ax1.set_title(r'Streamplot of vector field $V=(y,-x)^T$')
    plt.axis('equal')
    Q = ax1.streamplot(X,Y,VX,VY)
    plt.plot(x,y,'r-',linewidth=3)

    # mark initial and final position
    plt.plot(x[0],y[0],'ro')
    plt.plot(x[nt],y[nt],'go')

    plt.xlabel('x [m]')
    plt.ylabel('y [m]')

    plt.savefig('sailing_boring.pdf', bbox_inches='tight', format='pdf')
    plt.show()
```

##### Exercise 2

After completing the FD code `sailing_boring`, we can define some basic modelling parameters. How long do you want to sail, defined by the parameter $tmax [s]$. What time step $dt$ do you want to use? $dt=1.\;s$ should work for the first test of your FD code. To solve the problem you also have to define the initial position of your boat. Let's assume that ${\bf{x_{0}}}=(-900,0)^T$ is the location of some jetty on the western shore of the Kiel Fjord.

By executing the cell below (`SHIFT+ENTER`), the FD code `sailing_boring` should compute the course of the boat and plot it as red line on top of the `Streamplot`. Inital and final position are defined by a red and green dot, respectively.

What course would you expect, based on the `Streamplot`? Is it confirmed by your FD code solution? If not, there might be an error in your FD implementation.

```python
# How long do you want to sail [s] ?
tmax = 1000

# Define time step dt
dt = 1.

# Define initial position
x0 = -900.
y0 = 0.

# Sail for tmax s in the boring vector field
sailing_boring(tmax, dt, x0, y0)
```

##### Exercise 3

At this point you might get an idea why the code is called `sailing_boring`. We start at the western shore of the Kiel Fjord, follow an enclosed streamline to the eastern shore and travel back to the initial position of the jetty - it 's a boring Kiel harbor tour.

How long will the boring tour actually take? Vary $tmax$ until the green dot of the final position coincides with the red dot of the initial position.

You also might think: why should I invest so much computation time into this boring tour.
Copy the cell above, below this text box and increase the time step $dt$ to 20 s. How does the new FD solution differ from the one above with $dt=1\; s$? Give a possible explanation.

### Sailing in the more exciting vector field $V=(cos((x+y)/500),sin((x-y)/500))^T$

Time to sail in a more complex and exciting velocity vector field, like this one:

\begin{equation}
V=(cos((x+y)/500),sin((x-y)/500))^T \notag
\end{equation}

As in the case of the boring vector field, we define a function to compute the velocity components for a given ${\bf{x}} = (x,y)^T$:

```python
# define new vector field
def vel_xy_1(x,y):

    vx = np.cos((x+y)/500)
    vy = np.sin((x-y)/500)

    return vx, vy
```

For the visualization of this more complex vector field, I recommend to use a `Quiver` instead of the `Streamplot`

```python
# Define figure size
rcParams['figure.figsize'] = 8, 8

fig1, ax1 = plt.subplots()

# Define vector field components for coordinates X,Y
VX,VY = vel_xy_1(X,Y)

ax1.set_title(r'Plot of vector field $V=(cos((x+y)/500),sin((x-y)/500))^T$')
plt.axis('equal')
Q = ax1.quiver(X,Y,VX,VY)
plt.plot(392,392,'ro')

plt.xlabel('x [m]')
plt.ylabel('y [m]')

#plt.savefig('Plot_vector_field_V_exciting.pdf', bbox_inches='tight', format='pdf')
plt.show()
```

##### Exercise 4

Now, this velocity vector field looks more exciting, than the previous one. The red dot at ${\bf{x_{island}}}=(392,392)^T$ marks the location of an island you want to reach. To compute the course, we can recycle most parts of the `sailing_boring` code.

- Rename the code below from `sailing_boring` to `sailing_exciting`
- Add the FD solution of the uncoupled, ordinary differential equations (2) to the code
- Replace in the new `sailing_exciting` code the function calls of the boring velocity field `vel_xy` by the new exciting velocity field `vel_xy_1`
- Replace in `sailing_exciting` the `Streamplot` by a `Quiver` plot.
- Mark the position of the island by a red dot by inserting
```python
plt.plot(392,392,'ro')
```
below the `Quiver` plot in `sailing_exciting`

```python
def sailing_boring(tmax, dt, x0, y0):

    # Compute number of time steps
    nt = (int)(tmax/dt)

    # vectors for storage of x, y positions
    x  = np.zeros(nt + 1)
    y  = np.zeros(nt + 1)

    # define initial position
    x[0] = x0
    y[0] = y0

    # start time stepping
    for n in range(1,nt + 1):

        # compute velocity components at current position
        vx, vy = vel_xy(x[n-1],y[n-1])

        # compute new position using FD approximation of time derivative
        # ADD FD SOLUTION OF THE UNCOUPLED, ORDINARY DIFFERENTIAL EQUATIONS (2) HERE!
        x[n] =
        y[n] =

    # Define figure size
    rcParams['figure.figsize'] = 8, 8

    fig1, ax1 = plt.subplots()

    # Define vector field components for quiver plot
    VX,VY = vel_xy(X,Y)

    ax1.set_title(r'Plot of vector field $V=(cos((x+y)/500),sin((x-y)/500))^T$')
    plt.axis('equal')
    Q = ax1.streamplot(X,Y,VX,VY)
    plt.plot(x,y,'r-',linewidth=3)

    # mark initial and final position
    plt.plot(x[0],y[0],'ro')
    plt.plot(x[nt],y[nt],'go')

    print(x[nt],y[nt])

    plt.xlabel('x [m]')
    plt.ylabel('y [m]')

    plt.savefig('sailing_exciting.pdf', bbox_inches='tight', format='pdf')
    plt.show()
```

##### Exercise 5

Time to sail to the island. To make the problem more interesting, you have to find a course to the island from the north, south, east and west boundaries. In the four cells below the x0 and y0 coordinates of the given boundary is already defined. You only have to add and change the missing coordinate vector component until you reach the island. You might also have to modify $tmax$.

**Approach from the northern boundary**

```python
# How long do you want to sail [s] ?
tmax = 1000

# Define time step dt
dt = 2.

# DEFINE INTIAL POSITION AT NORTHERN BOUNDARY HERE!
x0 =
y0 = 950.

# Sail for tmax s in the boring vector field
sailing_exciting(tmax, dt, x0, y0)
```

**Approach from the southern boundary**

```python
# How long do you want to sail [s] ?
tmax = 1000

# Define time step dt
dt = 2.

# DEFINE INTIAL POSITION AT SOUTHERN BOUNDARY HERE!
x0 =
y0 = -980.

# Sail for tmax s in the boring vector field
sailing_exciting(tmax, dt, x0, y0)
```

**Approach from the western boundary**

```python
# How long do you want to sail [s] ?
tmax = 1000

# Define time step dt
dt = 2.

# DEFINE INTIAL POSITION AT WESTERN BOUNDARY HERE!
x0 = -950.
y0 =

# Sail for tmax s in the boring vector field
sailing_exciting(tmax, dt, x0, y0)
```

**Approach from the eastern boundary**

```python
# How long do you want to sail [s] ?
tmax = 1000

# Define time step dt
dt = 2.

# DEFINE INTIAL POSITION AT EASTERN BOUNDARY HERE!
x0 = 990.
y0 =

# Sail for tmax s in the boring vector field
sailing_exciting(tmax, dt, x0, y0)
```

##### Bonus Exercise

How do you reach the blue island in the vector plot below?

```python
# Define figure size
rcParams['figure.figsize'] = 8, 8

fig1, ax1 = plt.subplots()

# Define vector field components for coordinates X,Y
VX,VY = vel_xy_1(X,Y)

ax1.set_title(r'Plot of vector field $V=(cos((x+y)/500),sin((x-y)/500))^T$')
plt.axis('equal')
Q = ax1.quiver(X,Y,VX,VY)
plt.plot(-392,-392,'bo')

plt.xlabel('x [m]')
plt.ylabel('y [m]')

plt.show()
```

## What we learned

- How to solve a simple system of ordinary differential equations by an explicit time integration scheme

- The long-term impact of small inaccuracies in time integration schemes by choosing a too large time step $dt$

- The solution to a problem is not only defined by a differential equation, but also by an initial condition

- How to sail without wind, by using flow data and numerical solutions of ordinary differential equations


***
## Title

Programming for Data Analysis - Assignment 2020.
Submitted by ***Jack Caffrey***
***

## Introduction

The following assignment was undertaken as part of the Higher Diploma in Science - Data Analytics course through the Galway-Mayo Institute of Technology, for the module Programming for Data Analysis.

***Please note the following:***
1. The assignment critera will be outlined in the below ***Problem Statement*** section of the assignment.
2. All reference will be indicated using [#]. These references will be listed in the ***References*** section of this assignment.
3. Markdown formatting references are as follows.
     * Basic Syntax. Matt Cone. https://www.markdownguide.org/basic-syntax/
     * Extended Syntax. Matt Cone. https://www.markdownguide.org/extended-syntax/
     * Motivating Examples. Jupyter Team. https://jupyternotebook.readthedocs.io/en/stable/examples/Notebook/Typesetting%20Equations.html

### Problem Statement

Using a Jupyter notebook explain the use on the ***numpy.random*** package in Python.
This explanation must include the following:

1. The **purpose** of the package.
2. The use of the **"Simple random data"** and **"Permutations"** functions.
3. The use and purpose of at least five **"Distributions"** functions.
4. Explain the use of **"seeds"** in generating pseudorandom numbers.

***Note:***
The problem statement was defined using the critera outlined in the Programming for Data Analysis Assignment [1]

***

## 1. History of NumPy.

Before exploring the purpose of the ***numpy.random*** package it is important to provide a brief history of the development of the ***numPy*** package as a whole.

The ***numPy*** package developed from the Python programmming language extensions ***Numeric*** and ***Numarray*** [2]. Numeric was largely developed by Jim Hugunin a software programmer in 1995, with contributions from many people including Jim Fulton, David Ascher, Paul DuBois, Konard Hinsen [3].
Numeric was originally developed with maximum performance as it's main aim. A consicence of solely focusing efforts on maximum performance, resulted in design choices that meant Numeric was not extremely efficient for very large data sets [4]. (A data set is "*a collection of related sets of information that is composed of separate elements but can be manipulated as a unit by a computer*") [5].

As a result of this poor efficiency when dealing with large data sets the Numarray package was developed. Numarray was developed to have faster operating speeds for larger data sets when compared to Numeric, but this lead to Numarray having slower operating speeds for smaller data sets versus Numeric. This lead to both packages being used simultaneoulsy depending on the desired output [6].

At the beginning of 2005, an American Data Scientist and Businessman Travis Oliphant [7] wanted to develop a single array package (An array "*is a data structure consisting of a collection of elements, each identified by at least one array index or key.*") [8]. The creation of this single package was called *SciPy core*, with the intention to implement this as part of the bigger scientific package *SciPy*. This approach lead to confusion of how the package operated, resulting in the name change to *numerix*. The *numerix* name was already trademarked by another organisation, as a result of this trademarking another name change was required and the package ***NumPy*** was born. [9].

### What is NumPy?

Numpy is described as "the fundamental package for scientific computing in Python". It is through this Python library that is possible to complete fast operations on arrays including but not limited to the following:
1. Mathematical.
2. Logical.
3. Shape Manipulation.
4. Sorting.
5. Random Simulation. [10]

Note: For the purpose of the assignment random simulation (sampling) will be the main area of focus and the ***numpy.random*** package.

### Random Sampling & the *numpy.random* Packcage

Random sampling is a method used to select a sample of data from a larger data population.
In random sampling each sample of the data population has an equal chance of being selected. This selection is meant to be a netural and unbiased portayal of the larger data population it is selected from. If the selection does not represent a netural and unbiased portayal of the larger data population this is known as "*sampling error*" [11].

Random sampling is the best method of selecting samples from the data population you are interested in. The sample selected should portray the data population you are investigating and remove sampling bias. [12].

A package used by Python to generator these random values is ***numpy.random***. This ***numpy.random*** package is used to supplement the Python ***random*** function, with functions for efficiently generating whole arrays of sample values from many kinds of probability distributions [13].

In order to generate the required psuedorandom numbers (samples) from a population a combination of a BitGenerator and a Generator is used.
* **BitGenerator** - is used to create sequences.
* **Generator**  - uses the created sequences to sample from the required statistical distribution [14].
* **Psuedorandom Numbers** - "A set of values or elements that is statistically random, but it is derived from a known starting point and is typically repeated over and over. The algorithm can repeat the sequence, therefore the numbers are not entirely random".[15].

Today by default, Generator uses bits provided by PCG64 (Permuted Congruential Generator (64-bit)) and has replaced the use of MT19937 (Mersenne Twister 19937).

MT19937 is a legacy psuedorandom number generator. **RandomState** is used to provide access to the generator.It is best practice to only use this class when it is essential to have randoms that are identical to ones produced using previous version on Numpy,  as it is not possible to reproduce the exact random values required using Generator for normal distrubitions or any other distribution.

PCG64 provides bits to the **Generator** which has more efficent statistical properties when compared to **RandomState**.

### PCG64 vs MT19337 Quick Comparison

All comparison information is provided from https://numpy.org/doc/stable/reference/random/new-or-different.html#new-or-different

| Feature  | Older Equivalent | Notes |
|:--------:|:----------------:|:-----|
|Generator |  RandomState     |Generator requires a stream source, called a BitGenerator A number of these are provided. RandomState uses the Mersenne Twister MT19937 by default, but can also be instantiated with any BitGenerator.       |
|random    |random_sample, |Access the values in a BitGenerator, convert them to float64 in the interval [0.0., `` 1.0)``. In addition to the size kwarg, now supports dtype='d' or dtype='f', and an out kwarg to fill a user- supplied array.|
|integers  |randint, random_integers |  Many other distributions are also supported.Use the endpoint kwarg to adjust the inclusion or exclution of the high interval endpoint     |

For a more detailed comparison please see: https://numpy.org/doc/stable/reference/random/new-or-different.html#new-or-different

The **Seed** plays a vital role which enables both PCG64 and MT19337.

MT19337 uses a random ***Seed*** to begin the pseudorandom number generator. Values can be any integer between 0 and 624  including the number 1. If no value is provided the BitGenerator will take values from the windows analogue or the clock[16].

PCG64 supports the advance method to support the pseudorandom number generator ***Seed***. This is represented by 2 128-bit unassigned integers. The first is the state of the PRNG (psuedorandom number generator) which is advanced by a LCG (Linear Congruential Generator). The second is a fixed odd increment used in LCG. [17].
***

### Simple Random Data Functions
***

Simple Random Data is a population of values where each value of the population has an equal probability of being selected.This random sample is meant to be an unbiased representation of the population.[18].

The following are Simple Random Data Functions used by the numpy.random package. Examples are generated per numpy.org documentation [20]:
   * numpy.random.Generator.integers
   * numpy.random.Generator.random
   * numpy.random.Generator.choice
   * numpy.random.Generator.bytes

#### 1. Random Number Generator Interges

Makes use of low, high, and size parameters and optional parameters dtype, and endpoint to produce an array of random numbers from a distrubution or population of data.

Note: To return low inclusive to high inclusive random integers set endpoint=True.

```python
# numpy.random.Generator.integers

import numpy as np #importing numpy to access the function.

x = np.random.default_rng()

x.integers(5, size=10, endpoint=True) # produces a random order of values up to put not including 2 for a total of 10 values
```

    array([3, 0, 2, 5, 1, 0, 1, 1, 0, 3], dtype=int64)

#### 2. Random Number Generator Random

Makes use of size parameters in the form int or tuple of ints and optional parameters dtype and out to produce an array of random floats of a set shape or size
Note: if size = None then a single float is returned.

```python
# numpy.random.Generator.random

import numpy as np  #importing numpy to access the function.

rng = np.random.default_rng()

2 * rng.random((2,2))  - 5 # Generators a random 2 by 2 array for random floats between -5 and -3
```

    array([[-3.39989006, -4.42317879],
           [-3.06481884, -4.61347249]])

#### 3. Random Number Generator Choice

Makes use of the parameters a, and optional parameters size, replace, p, axis and shuffle to generate a random sample from a given 1-D array.

```python
# Random Number Generator Choice

import numpy as np #importing numpy to access the function.

rng.choice(4, 10, p=[0.4, 0.1,0.2,0.3,]) # Generates a random sample up to but not including 4 returning 10 values in a 1-D array, with the probabilities asssociated with each entry in a defined.
```

    array([3, 3, 3, 2, 2, 2, 0, 2, 3, 0], dtype=int64)

#### 4. numpy.random.Generator.bytes

Makes use of length parameters to return random bytes.

```python
# numpy.random.Generator.bytes

import numpy as np #importing numpy to access the function.

np.random.default_rng().bytes(15)#returns random bytes of () determined length.
```

    b'\xca\xe5Zi\xad\xee\xf2\xebD\xb11\xb0\xfc\x1ak'

***
### Permutation Functions
***

A Permutation function is an ordered arrangment of values from a population without any value being repeated.[19]

The following are Permutation Functions used by the numpy.random package. Examples are generated per numpy.org documentation [20]:
   * numpy.random.Generator.shuffle
   * numpy.random.Generator.permutation

### 1. numpy.random.Generator.shuffle

Makes use of parameter "x" and optional axis parameters to shuffle the contents of sub-arrays.

Note: By using the shuffle method changes will be made to the original array.

```python
import numpy as np #importing the numpy package

rng = np.random.default_rng() # defining rng
arr = np.arange(9).reshape((3, 3)) #reshapes array to 3x3 array.
rng.shuffle(arr) #Shuffles values with the array
arr #displays the shuffled array
```

    array([[6, 7, 8],
           [3, 4, 5],
           [0, 1, 2]])

### 2. numpy.random.Generator.permutation

Makes use of parameter "x" and optional axis parameters to generate a permuted sequence or array sequence.

Note: By using the permutation method a re-arranged array is returned but the original array remains un-changed.

```python
import numpy as np

rng = np.random.default_rng()

arr = np.arange(9).reshape((3, 3)) #reshapes array to 3x3 array.
rng.permutation(arr) #displays the re-arranged array

```

    array([[6, 7, 8],
           [0, 1, 2],
           [3, 4, 5]])

### Distribution Functions
***

1. Unifrom Distribution
2. Pareto Distribution
3. Normal Distribution
4. Exponential Distribution
5. Poisson Distribution

***
### Uniform Distrubtion

\begin{align}
\ p(x) & = (\frac{1}{b-a})
\end{align}

#### 1. Uniform Distribution

Unifrom distrubtion is used to describe the probability where every evant has an equal chance of occuring.

It has three parameters:

'a' lower bound - default 0.0.
'b' upper bound - default 0.1.
'size' The shape of the returned array. [21]

The unifrom distrubtion has no mode and no skweness. Due to this the mean and median correspond.

Unifrom Distrubtion can take two forms:
 * 1 Discrete Random Variables - are variables that can only take a countable number of values. [22]
 * 2 Continuous Random Variables - are variables where the data can take infintely many values [23]

```python
# Example ref: https://numpy.org/doc/stable/reference/random/generated/numpy.random.Generator.pareto.html#numpy.random.Generator.pareto
import numpy as np
import matplotlib.pyplot as plt

s = np.random.default_rng().uniform(-1,0,1000)

np.all(s >= -1) # Verify all values are within the given interval
np.all(s < 0)

count, bins, ignored = plt.hist(s, 15, density=True)
plt.plot(bins, np.ones_like(bins), linewidth=2, color='r')
plt.show()
```

***
### Pareto Distribution

\begin{align}
\ p(x) & = \frac{am^a}{x^{a+1}}
\end{align}

#### 2. Pareto Distribution

Pareto Distrubtion is a standard quality mechanism used to help identify the most frequent occurance of any factors you can count and categotize. [24]

It has two parameters:

'a' - shape parameter.
'size' - The shape of the returned array.

Pareto's law the 80-20 Distribution works from the principle the 20% factors causes 80% outcome. [25]

```python
# Example ref: https://numpy.org/doc/stable/reference/random/generated/numpy.random.Generator.pareto.html#numpy.random.Generator.pareto
import numpy as np
import matplotlib.pyplot as plt

a, m = 3., 2.  # shape and mode
s = (np.random.default_rng().pareto(a, 1000) + 1) * m

count, bins, _ = plt.hist(s, 100, density=True) # Formatting the graph
fit = a*m**a / bins**(a+1)
plt.plot(bins, max(count)*fit/max(fit), linewidth=2, color='r')
plt.show()
```

***
### Normal Distribution

\begin{align}
\ p(x) = \frac{1}{\sqrt{2\pi}{\sigma}}e^{-{\frac{(x -\mu)^2}{2\sigma^2}}} \\
\end{align}

#### 3. Normal Distribution

The normal distribution is a probability function that explains how the values contained in a data set or population are distributed. Extreme values at both tails of the distribution are unlikely. [26]

It has three parameters:

'loc' - Mean where the peak of the bell exists.
'scale' - Standard Deviation how the flat graph distrbution should be.
'size' The shape of the returned array.

The normal Distribution is identified as one of the most if not the most important probability distribution in statistics.

```python
# Example https://numpy.org/doc/stable/reference/random/generated/numpy.random.Generator.normal.html#numpy.random.Generator.normal
import numpy as np
import matplotlib.pyplot as plt

mu, sigma = 0, 0.1 # mean and standard deviation
s = np.random.default_rng().normal(mu, sigma, 1000)

abs(mu - np.mean(s))
abs(sigma - np.std(s, ddof=1))

count, bins, ignored = plt.hist(s, 30, density=True)
plt.plot(bins, 1/(sigma * np.sqrt(2 * np.pi)) *np.exp( - (bins - mu)**2 / (2 * sigma**2) ),linewidth=2, color='r')
plt.show()

```

***
### Exponential Distribution

\begin{align}
\ f(x;\frac{1}{\beta}) = \frac{1}{\beta}exp(-\frac{x}{\beta}) \\
\end{align}

#### 4.Exponential Distribution

The Exponential Distribution is a continous probability distribution used to model the time required to wait before a given event will occur. [27]

It has two parameters:

'scale'- inverse of rate - defaults to 1.0.
'size' - The shape of the returned array.

```python
# Example ref: https://www.w3schools.com/python/numpy_random_exponential.asp

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

sns.distplot(np.random.exponential(size=1000), hist=True,color='b')

plt.show()
```

***
### Poisson Distribution

\begin{align}
\ f(k; {\lambda}) = \frac{\lambda^ke^{-\lambda}}{k!}\\
\end{align}

#### 5. Poisson Distribution

The Poisson Distribution is a discrete probability distribution that estimates how many times an event can happen in a pre determined timeframe. [28]

It has two parameters:
'lam' - rate or known number of occurences.
'size' - The shape of the returned array.

```python
# Example ref: https://numpy.org/doc/stable/reference/random/generated/numpy.random.Generator.poisson.html#numpy.random.Generator.poisson

import numpy as np #
import matplotlib.pyplot as plt

rng = np.random.default_rng()
s = rng.poisson(5, 10000)

count, bins, ignored = plt.hist(s, 14, density=True)
plt.show()
```

***
### References
[1] ProgDA_Assignment. GMIT.
[2] People / Jim Hugunin. Peoplepill. https://peoplepill.com/people/jim-hugunin/
[3] The birth of Numeric.  SciPy History_of_SciPy. https://scipy.github.io/old-wiki/pages/History_of_SciPy
[4] Python Numeric. History. http://people.csail.mit.edu/jrennie/python/numeric/
[5] Definitions from Oxford Languages. Dictionary. https://www.google.com/search?client=firefox-b-d&q=what+is+a+data+set
[6] NumPy. History. https://en.wikipedia.org/wiki/NumPy
[7] People / Travis Oliphant. Peoplepill. https://peoplepill.com/people/travis-oliphant/
[8] Array data structure. Wikipedia. https://en.wikipedia.org/wiki/Array_data_structure
[9] The reunion, aka the birth of NumPy. SciPy History_of_SciPy. https://scipy.github.io/old-wiki/pages/History_of_SciPy
[10] What is NumPy?. Numpy. https://numpy.org/doc/stable/user/whatisnumpy.html#
[11] The Economic Times .Definition of 'Random Sampling'. https://economictimes.indiatimes.com/definition/Random-Sampling
[12] Saul Mcleod.Random Sampling. https://www.simplypsychology.org/sampling.html#:~:text=Random%20samples%20are%20the%20best,time%2C%20effort%20and%20money [13]lmiguelvargasf. Differences between numpy.random and random.random in Python.https://stackoverflow.com/questions/7029993/differences-between-numpy-random-and-random-random-in-python#:~:text=From%20Python%20for%20Data%20Analysis,many%20kinds%20of%20probability%20distributions
[14] NumPy .Random sampling (numpy.random).https://numpy.org/doc/stable/reference/random/index.html
[15] encyclopedia.pseudo-random numbers.https://www.pcmag.com/encyclopedia/term/pseudo-random-numbers
[16] Numpy.Parameters. https://numpy.org/doc/stable/reference/random/legacy
[17] Numpy. State and Seeding. https://numpy.org/doc/stable/reference/random/bit_generators/pcg64.html#numpy.random.PCG64
[18] Adam Hayes.Simple Random Sample. https://www.investopedia.com/terms/s/simple-random-sample.asp#:~:text=A%20simple%20random%20sample%20is,equal%20probability%20of%20being%20chosen.&text=In%20this%20case%2C%20the%20population,equal%20chance%20of%20being%20chosen.
[19] Permutations function. Minitab® 18 Support. https://support.minitab.com/en-us/minitab/18/help-and-how-to/calculations-data-generation-and-matrices/calculator/calculator-functions/arithmetic-calculator-functions/permutations-function/#:~:text=A%20permutation%20is%20an%20ordered,from%20a%20group%20without%20repetitions.&text=Use%20the%20Permutation%20function%20to,possible%20outcomes%20(binomial%20experiment)
[20] Numpy, Random Generator, https://numpy.org/doc/stable/reference/random/generator.html
[21] w3schools, Uniform Distribution, https://www.w3schools.com/python/numpy_random_uniform.asp
[22] Revision maths, Discrete Random Variables, https://revisionmaths.com/advanced-level-maths-revision/statistics/discrete-random-variables
[23] Revision maths, Continuous Random Variables, https://revisionmaths.com/advanced-level-maths-revision/statistics/continuous-random-variables
[24] Minitab, When to Use a Pareto Chart, https://blog.minitab.com/blog/understanding-statistics/when-to-use-a-pareto-chart<br>
[25] w3schools ,Pareto Distribution,https://www.w3schools.com/python/numpy_random_pareto.asp
[26] Jim Frost, Statistics By Jim, https://statisticsbyjim.com/basics/normal-distribution/
[27] Marco Taboga, PHD, Exponential distribution, https://www.statlect.com/probability-distributions/exponential-distribution <br>
[28] w3schools, Poisson Distribution, https://www.w3schools.com/python/numpy_random_poisson.asp <br>
***
## End

Assignment-2020 Programming for Data Analysis.ipynb


```python
import sys
!{sys.executable} -m pip install numpy
!{sys.executable} -m pip install matplotlib
!{sys.executable} -m pip install pandas
!{sys.executable} -m pip install scipy
```

    Requirement already satisfied: numpy in /usr/local/Cellar/jupyterlab/3.0.10/libexec/lib/python3.9/site-packages (1.20.1)
    WARNING: You are using pip version 21.0.1; however, version 22.0.2 is available.
    You should consider upgrading via the '/usr/local/Cellar/jupyterlab/3.0.10/libexec/bin/python3 -m pip install --upgrade pip' command.
    Requirement already satisfied: matplotlib in /usr/local/Cellar/jupyterlab/3.0.10/libexec/lib/python3.9/site-packages (3.3.4)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /usr/local/Cellar/jupyterlab/3.0.10/libexec/lib/python3.9/site-packages (from matplotlib) (2.4.7)
    Requirement already satisfied: numpy>=1.15 in /usr/local/Cellar/jupyterlab/3.0.10/libexec/lib/python3.9/site-packages (from matplotlib) (1.20.1)
    Requirement already satisfied: pillow>=6.2.0 in /usr/local/Cellar/jupyterlab/3.0.10/libexec/lib/python3.9/site-packages (from matplotlib) (8.1.2)
    Requirement already satisfied: python-dateutil>=2.1 in /usr/local/Cellar/jupyterlab/3.0.10/libexec/lib/python3.9/site-packages (from matplotlib) (2.8.1)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/Cellar/jupyterlab/3.0.10/libexec/lib/python3.9/site-packages (from matplotlib) (1.3.1)
    Requirement already satisfied: cycler>=0.10 in /usr/local/Cellar/jupyterlab/3.0.10/libexec/lib/python3.9/site-packages (from matplotlib) (0.10.0)
    Requirement already satisfied: six in /usr/local/Cellar/jupyterlab/3.0.10/libexec/lib/python3.9/site-packages (from cycler>=0.10->matplotlib) (1.15.0)
    WARNING: You are using pip version 21.0.1; however, version 22.0.2 is available.
    You should consider upgrading via the '/usr/local/Cellar/jupyterlab/3.0.10/libexec/bin/python3 -m pip install --upgrade pip' command.
    Requirement already satisfied: pandas in /usr/local/Cellar/jupyterlab/3.0.10/libexec/lib/python3.9/site-packages (1.3.4)
    Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/Cellar/jupyterlab/3.0.10/libexec/lib/python3.9/site-packages (from pandas) (2.8.1)
    Requirement already satisfied: pytz>=2017.3 in /usr/local/Cellar/jupyterlab/3.0.10/libexec/lib/python3.9/site-packages (from pandas) (2021.1)
    Requirement already satisfied: numpy>=1.17.3 in /usr/local/Cellar/jupyterlab/3.0.10/libexec/lib/python3.9/site-packages (from pandas) (1.20.1)
    Requirement already satisfied: six>=1.5 in /usr/local/Cellar/jupyterlab/3.0.10/libexec/lib/python3.9/site-packages (from python-dateutil>=2.7.3->pandas) (1.15.0)
    WARNING: You are using pip version 21.0.1; however, version 22.0.2 is available.
    You should consider upgrading via the '/usr/local/Cellar/jupyterlab/3.0.10/libexec/bin/python3 -m pip install --upgrade pip' command.
    Requirement already satisfied: scipy in /usr/local/Cellar/jupyterlab/3.0.10/libexec/lib/python3.9/site-packages (1.7.3)
    Requirement already satisfied: numpy<1.23.0,>=1.16.5 in /usr/local/Cellar/jupyterlab/3.0.10/libexec/lib/python3.9/site-packages (from scipy) (1.20.1)
    WARNING: You are using pip version 21.0.1; however, version 22.0.2 is available.
    You should consider upgrading via the '/usr/local/Cellar/jupyterlab/3.0.10/libexec/bin/python3 -m pip install --upgrade pip' command.

# *Metody obliczeniowe/Metody numeryczne*
## *Laboratorium 2*
## *Rozwiązywanie równań nieliniowych*

## Spis treści

* [Wstęp](#wstep)
* [Przykład 1. Rozwiązywanie równań nieliniowych](#rrn)
* [Przykład 2. Krótko o pandas](#pandas)
* [Ćwiczenie 1](#cw1)
* [Metoda bisekcji](#bisect)
* [Ćwiczenie 2](#cw2)
* [Regula falsi](#rfalsi)
* [Ćwiczenie 3](#cw3)
* [Metoda siecznych](#secant)
* [Ćwiczenie 4](#cw4)
* [Metoda Newtona](#newton)
* [Ćwiczenie 5](#cw5)
* [Metoda Newtona dla układów równań nieliniowych](#newton_nles)
* [Przykład 3. Wyznaczanie pochodnych cząstkowych](#pcz)
* [Ćwiczenie 6](#cw6)
* [Ćwiczenie 7](#cw7)
* [Ćwiczenie 8](#cw8)
* [Ćwiczenie 9](#cw9)
* [Ćwiczenie 10](#cw10)

## Wstęp

Równaniem nieliniowym nazywa się równanie w którym jedna lub więcej zmiennych występuje nieliniowo, np. funkcje trygonometryczne lub wielomian trzeciego stopnia. Rozwiązywanie takich równań(funkcji) polega na znajdowaniu takiego x dla którego f(x) = 0. Na tym laboratorium będą omawianie iteracyjne metody rozwiązywania takich równań, czyli takie które z każdą kolejną iteracją (krokiem), są coraz bliżej rozwiązania. Pierwszym krokiem przy iteracyjnym rozwiązywaniu zadań, jest upewnienie się że dane równanie posiada miejsce zerowe. Następnym krokiem jest wybór odpowiedniego algorytmu. Kolejnym krokiem jest wybór warunku stopu, czyli określenie jakie przybliżenie jest dla nas zadowalające i można przerwać iterację. Istnieją na to trzy sposoby:

* grube stopowanie na osi OY:

\begin{equation}
    |f(x_n)| < tol_{1}
\end{equation}

* grube stopowanie na osi OX:

\begin{equation}
|x_n - x_{n-1}| < tol_{2}
\end{equation}

* drugi sposób i sprawdzanie zmiany pomiędzy dwoma kolejnymi przybliżeniami:

\begin{equation}
|x_n - x_{n-1}| < tol_{2} \wedge |x_{n} - x_{n-1}| \leq |x_{n-1} - x_{n-2}|
\end{equation}

Pierwszy z nich i zarazem najbardziej naiwny, sprawdza odległość pomiędzy wartością funkcji w wyznaczonym przybliżeniu a dokładnością, która jest bardzo bliska zeru. Drugi sposób porównuje odległość pomiędzy dwoma kolejnymi przybliżeniami pierwiastka, natomiast trzeci jest  rozszerzeniem drugiego warunku - dodatkowo sprawdzana jest różnica pomiędzy dwoma ostatnimi przybliżeniami. Trzeci warunek może być stosowany tylko dla metod rekurencyjnych. W ćwiczeniach będzie stosowany pierwszy warunek, gdyż można go zastosować do wszystkich omawianych algorytmów.

## Przykład 1. Rozwiązywanie równań nieliniowych

Do rozwiązywania równań nieliniowych (m.in.) służą pakiety SciPy oraz SymPy. Pierwszy z nich służy do obliczeń numerycznych, drugi natomiast do symbolicznych. Najpopularniejszą funkcją z pierwszego pakietu która rozwiązuje równania jest funkcja fsolve. Jako argumenty funkcja ta przyyjmuje funkcję w postaci wielomianiu, oraz tablicę pierwszych przybliżeń. Metodę tą można stosować zarówno dla pojedynczych równań jak ich układów. Funkcja ta jest wrapperem dla zdefiniowanych w pakiecie MINPACK algorytmów hybrd oraz hybrj. Poniżej przedstawiono przykład jej użycia.

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from numpy import sign
from scipy.optimize import fsolve

# rozwiązywanie pojedyńczego równania kwadratowego x^2 + x - 5 = 0,
# posiadającego dwa pierwiastki
fun = lambda x: x ** 2 + x - 5

# tablica pierwszych przybliżeń
xguess = [-4, 2]

roots = fsolve(fun, xguess)
y = fun(roots)

print(f"x = {roots} \ny = {y}")

# sprawdzenie czy wartości y obliczonych pierwiastków są bliskie 0
print(np.isclose(y, [0.0, 0.0]))
```

    x = [-2.79128785  1.79128785]
    y = [0.0000000e+00 8.8817842e-16]
    [ True  True]

## Przykład 2. Krótko o pandas

Pandas to biblioteka, powszechnie używana w szeroko rozumianym data science. Udostępnia ona zestaw narzędzi do analizy i obróbki danych, z których najbardziej znanym jest klasa DataFrame - dwuwymiarowa tablica do przechowywania danych.

* klasa Series

można porównać ją do kolumny w tablicy

```python
import pandas as pd

pdk = [2, 1, 3, 7]

# tworzenie z tablicy, indeks domyślny (czyli numeracja rzędów od 0)
s = pd.Series(pdk)

print(s)

# 0    2
# 1    1
# 2    3
# 3    7
# dtype: int64

d = {'jeden': 1, 'dwa' : 2}

# tworzenie ze słownika, indeks jako klucz
s = pd.Series(d)

print(s)

# jeden    1
# dwa      2
# dtype: int64
```

* klasa DataFrame

można nazwać ją połączeniem Excela z kolumną z bazy danych

```python
import pandas as pd

dane = {'Imie': ['Andrzej', 'Karol', 'Marta'],
        'Rok urodzenia': [1998, 2001, 1995]}

# tworzenie obiektu DataFrame z indeksem domyślnym
df = pd.DataFrame(dane)

print(df)

#       Imie  Rok urodzenia
# 0  Andrzej           1998
# 1    Karol           2001
# 2    Marta           1995

# odwoływanie się do kolumny odbywa się za pomocą jej nazwy
print(df['Imie'])

# 0    Andrzej
# 1      Karol
# 2      Marta
# Name: Imie, dtype: object

# odwoływanie się do wiersza za pomocą indeksu odbywa się za pomocą funkcji loc
print(df.loc[1])

# Imie             Karol
# Rok urodzenia     2001
# Name: 1, dtype: object

# można też odwoływać się do wielu wierszy jednocześnie
print(df.loc[[0, 2]])

#       Imie  Rok urodzenia
# 0  Andrzej           1998
# 2    Marta           1995

# do wstępnej analizy zestawu danych przydatne są funkcje head, tail oraz info
# które służą do wyświetlania kolejno 5 pierwszych, 5 ostatnich rekordów oraz informacji o danych

df.head()
df.tail()
df.info()

```

## Ćwiczenie 1

Napisz funkcję $fun1$ będącą implementacją wzoru
\begin{equation}
f(x) = x^3+x^2-3x-3
\tag{1}
\end{equation}

Do rozwiązania tego punktu można użyć operatora potęgowania **\*\***, a sam wzór zdefiniować jako zwyczajną funkcję lub wyrażenie lambda.

```python
# Przedział zawierający zgrubne położenie miejsc zerowych
x = np.arange(-3, 3, 0.01)

# Zdefiniowanie funkcji
# TODO
# fun1 = ...
# lub
# def fun1(x): ...

# Wartości funkcji na przedziale x
y = fun1(x)

# Wykres pozwalający na dokładniejsze oszacowanie miejsc zerowych
plt.plot(x, y)
plt.grid(True)
plt.show()
```

## Metoda bisekcji

Metoda ta polega na iteracyjnym wyznaczaniu coraz mniejszych przedziałów, co do których mamy pewność, że zawierają zero funkcji. Załóżmy, że dana jest funkcja $f(x)$, która jest ciągła w przedziale $[a,b]$ i przyjmuje na jego brzegach wartości o przeciwnych znakach ($f(a)f(b)<0$). W takim przypadku przedział $[a,b]$ musi zawierać pierwiastek funkcji. Za przybliżone rozwiązanie możemy uznać punkt leżący w środku przedziału $[a,b]$, czyli:

\begin{equation}
c = \frac{a+b}{2}
\tag{2}
\end{equation}

W przypadku gdy $f(c)=0$ miejsce zerowe zostało odnalezione. Z reguły jednak $f(c) \neq 0$ i poszukiwania trzeba kontynuować. Zero będzie znajdować się albo w przedziale $[a,c]$, albo w przedziale $[c,b]$. Aby wybrać właściwy przedział wystarczy sprawdzić wartość funkcji w punkcie $c$. Jeśli $f(a)f(c)<0$ kontynuujemy proces w przedziale $[a,c]$, jeśli $f(c)f(b)<0$ oznacza to, że w następnej iteracji wartość $c$ będzie trzeba podstawić w miejsce $a$. Iteracje można przerywać jeśli odnaleziona wartość $f(c)< tol$, gdzie $tol$ jest zadaną przez nas wartością.

Metoda bisekcji jest wolno zbieżna (zbieżność jest liniowa), gdyż w ogóle nie korzysta z informacji jakie daje kształt funkcji $f(x)$. Jej niewątpliwą zaletą jest natomiast to, że jest pewna. Bisekcja sprawuje się dobrze, tam gdzie inne (szybsze) metody mają problemy. W obliczeniach numerycznych jest rzadko używana osobno. Za to często stosuje się ją do początkowego przybliżenia przedziału zawierającego miejsce zerowe funkcji, aby potem skorzystać z szybszych metod.

Pseudo kod dla metody bisekcji:

 Wejście:

 fun ← funkcja, dla której szukamy miejsca zerowego

 a, b ← granice przedziału w którym znajduje się pierwiastek

 tol ← żądana dokładność

 i_max ← maksymalna ilość iteracji

 Wyjście:

 i → liczba iteracji

 c → znalezione przybliżenie pierwiastka

 fc → wartość funkcji w punkcie c

 Algorytm:

 fa ← fun(a)

 fb ← fun(b)

 if   sign(fa) == sign(fb)   then

          return  error

         end if

  for  i ← 1  to  i_max  do

 c ← (a + b) / 2

 fc ← fun(c)

 e ← (b - a) / 2

  if   |fc| < dok   then

   return i, c fc

  end if

 if   sign(fc) != sign(fa)   then

   b ← c

   fb ← fc

 else

   a ← c

   fa ← fc

  end if

  end for

## Ćwiczenie 2

Uzupełnij kod funkcji implementującej metodę bisekcji. Znajduje się on w poniższej komórce.

Jej wywołanie wygląda następująco:
```python
[c,  yc,  df] = bisect(fun,  a,  b,  i_max,  dok)
```

gdzie jej parametry to:

*  fun ← funkcja, dla której szukamy miejsca zerowego
*  a, b ← granice przedziału w którym znajduje się pierwiastek
*  dok ← żądana dokładność
*  i_max ← maksymalna ilość iteracji

a zwracane wartości to:

*  c → znalezione przybliżenie pierwiastka
*  yc → wartość funkcji dla znalezionego przybliżenia pierwiastka
*  df → obiekt DataFrame z danymi z każdej iteracji

```python
def bisect(fun, a, b, i_max, dok):
    """
    :param fun: funkcja
    :param a: początek przedziału
    :param b: koniec przedziału
    :param i_max: maksymalna liczba iteracji
    :param dok: dokładność
    :return: (przybliżona wartość pierwiastka, wartość funkcji od przybliżenia, tablica z danymi z każdej iteracji)
    """

    if i_max < 1:
        raise ValueError("i_max musi być większe od 0")

    # Wartości funkcji na krańcach zadanego przedziału
    ya = fun(a)
    yb = fun(b)

    # Sprawdzenie czy podano poprawne wartości krańcowe
    if ya * yb > 0:
        raise ValueError("Wartości funkcji nie posiadają różnych znaków na początku i końcu przedziału")

    df = pd.DataFrame(columns=["Krok", "a", "b", "c", "yc", "błąd"])

    for i in range(i_max):
        # TODO
        # c = ...
        yc = fun(c)
        err = (b - a) / 2

        df = df.append(pd.Series(data={"Krok": i + 1, "a": a, "b": b, "c": c, "yc": yc, "błąd": err}),
                       ignore_index=True)

        # Sprawdzenie zbieżności
        if abs(yc) < dok:
            print(f"Metoda bisekcji zbieżna po {i + 1} iteracjach")
            break

        if sign(ya) == sign(yc):
            a = c
            ya = yc
        else:
            b = c
            yb = yc

    else:
        #
        print("Funkcja nie mogła znaleźć przybliżenia z zadaną dokładnością w podanej liczbie iteracji")

    df = df.astype({"Krok": "int64"})
    df = df.set_index("Krok")

    return c, yc, df

a, b , c = bisect(lambda x: x*x + x -3, 3, 1, 100, 0.1)
c
```

    Metoda bisekcji zbieżna po 5 iteracjach

      a
      b
      c
      yc
      błąd

      Krok

      1
      3.00
      1.00
      2.00
      3.00
      -1.00

      2
      2.00
      1.00
      1.50
      0.75
      -0.50

      3
      1.50
      1.00
      1.25
      -0.19
      -0.25

      4
      1.50
      1.25
      1.38
      0.27
      -0.12

      5
      1.38
      1.25
      1.31
      0.04
      -0.06

## Regula falsi

Pierwszym sposobem na to, aby wykorzystać informacje o kształcie funkcji do przyspieszenia zbieżności poszukiwania zera jest aproksymacja funkcji $f(x)$ za pomocą linii prostej. W metodzie regula falsi, podobnie jak metodzie połowienia zaczynamy od przedziału $[a,b]$, na którym spełniony jest warunek $f(a)f(b)<0$. Przybliżoną wartością pierwiastka w każdym następnym kroku  będzie miejsce przecięcia osi OX i linii przechodzącej przez punkty $(a,f(a))$ i $(b,f(b))$, tzn:

\begin{equation}
c = b - \frac{b-a}{f(b)-f(a)}f(b)
    \tag{3}
\end{equation}

Jeśli zero znajduje się w przedziale $[a,c]$ zmienną $a$ pozostawiamy bez zmian i podstawiamy $b = c$; w przeciwnym przypadku przypisujemy $a=c$, z kolei $b$ pozostaje takie samo.

Pseudo kod dla metody regula falsi:

 Wejście:

 fun ← funkcja, dla której szukamy miejsca zerowego

 a, b ← granice przedziału w którym znajduje się pierwiastek

 dok ← żądana dokładność

 i_max ← maksymalna ilość iteracji

 Wyjście:

 i → liczba iteracji

 c → znalezione przybliżenie pierwiastka

 fc → wartość funkcji w punkcie c

 Algorytm:

 fa ← fun(a)

 fb ← fun(b)

 if   sign(fa) == sign(fb)   then

          return  error

         end if

  for  i ← 1  to  i_max  do

 c ←  b - (b - a) / (fb - fa) * fb

 fc ← fun(c)

  if  |fc| < dok  then

   return i, c fc

  end if

 if   sign(fc) != sign(fa)   then

   b ← c

   fb ← fc

 else

   a ← c

   fa ← fc

  end if

  end for

## Ćwiczenie 3

Uzupełnij kod funkcji implementującej metodę regula falsi.

```python
def rfalsi(fun, a, b, i_max, dok):
    """
    :param fun: funkcja
    :param a: początek przedziału
    :param b: koniec przedziału
    :param i_max: maksymalna liczba iteracji
    :param dok: dokładność
    :return: (przybliżona wartość pierwiastka, wartość funkcji od przybliżenia, tablica z danymi z każdej iteracji)
    """

    if i_max < 1:
        raise ValueError("i_max musi być większe od 0")

    # Wartości funkcji na krańcach zadanego przedziału
    ya = fun(a)
    yb = fun(b)

    # Sprawdzenie czy podano poprawne wartości krańcowe
    if ya * yb > 0:
        raise ValueError("Wartości funkcji nie posiadają różnych znaków na początku i końcu przedziału")

    df = pd.DataFrame(columns=["Krok", "a", "b", "c", "yc"])

    for i in range(i_max):
        # TODO
        # c = ...
        yc = fun(c)

        df = df.append(pd.Series(data={"Krok": i + 1, "a": a, "b": b, "c": c, "yc": yc}),
                       ignore_index=True)

        # Sprawdzenie zbieżności
        if abs(yc) < dok:
            print(f"Regula falsi zbieżna po {i + 1} krokach")
            break

        if sign(ya) == sign(yc):
            a = c
            ya = yc
        else:
            b = c
            yb = yc

    else:
        print("Funkcja nie mogła znaleźć przybliżenia z zadaną dokładnością w podanej liczbie iteracji")

    df = df.astype({"Krok": "int64"})
    df = df.set_index("Krok")

    return c, yc, df

a, b , c = rfalsi(lambda x: x*x + x - 5, -6, 1, 100, 0.1)
c
```

    Regula falsi zbieżna po 8 krokach

      a
      b
      c
      yc

      Krok

      1
      -6.0
      1.000000
      0.250000
      -4.687500

      2
      -6.0
      0.250000
      -0.736842
      -5.193906

      3
      -6.0
      -0.736842
      -1.642202
      -3.945375

      4
      -6.0
      -1.642202
      -2.236188
      -2.235652

      5
      -6.0
      -2.236188
      -2.545142
      -1.067393

      6
      -6.0
      -2.545142
      -2.686610
      -0.468737

      7
      -6.0
      -2.686610
      -2.747591
      -0.198335

      8
      -6.0
      -2.747591
      -2.773190
      -0.082605

## Metoda siecznych

Podobnie jak w metodzie regula falsi, tak i w tej funkcja $f(x)$ jest aproksymowana linią prostą. Jednak metoda siecznych nie wymaga, aby w każdej iteracji sprawdzać przedział, w którym znajduje się pierwiastek. Aby rozpocząć algorytm wymagane są dwa przybliżenia pierwiastka ($x_0,x_{1}$) i nie musi być spełniony warunek $f(x_0)f(x_{1})<0$. Kolejne aproksymowane wartości pierwiastka znajdujemy (podobnie jak w powyższej metodzie) jako miejsce przecięcia się osi OX i linii przechodzącej przez dwie ostatnie aproksymacje pierwiastka $(x_{k-1},f(x_{k-1}))$ i $(x_{k},f(x_{k}))$.

\begin{equation}
x_{k+1} = x_k - \frac{x_k-x_{k-1}}{f(x_k)-f(x_{k-1})}f(x_k)
    \tag{4}
\end{equation}

Pseudo kod dla metody siecznych:

 Wejście:

 fun ← funkcja, dla której szukamy miejsca zerowego

 xn0, xn1 ← dwa pierwsze przybliżenia pierwiastka

 dok ← żądana dokładność

 i_max ← maksymalna ilość iteracji

 Wyjście:

 i → liczba iteracji

 xn2 → znalezione przybliżenie pierwiastka

 fn2 → wartość funkcji w punkcie xn2

 Algorytm:

  for  i ← 1  to  i_max  do

 fn0 ← fun(xn0)

 fn1 ← fun(xn1)

 xn2 ← xn1 - (xn1 - xn0) / (fn1 - fn0) * fn1

 fn2 ← fun(xn2)

 xn0 ← xn1

 xn1 ← xn2

 if   |fn2| < dok   then

   return i, xn2, fn2

  end if

  end for

## Ćwiczenie 4

Uzupełnij kod funkcji implementującej metodę siecznych.

```python
def secant(fun, xn0, xn1, i_max, dok):
    """
    :param fun: funkcja
    :param xn0: pierwsze przybliżenie pierwiastka
    :param xn1: drugie przybliżenie pierwiastka
    :param i_max: maksymalna liczba iteracji
    :param dok: dokładność
    :return: (przybliżona wartość pierwiastka, wartość funkcji od przybliżenia, tablica z danymi z każdej iteracji)
    """

    if i_max < 1:
        raise ValueError("i_max musi być większe od 0")

    df = pd.DataFrame(columns=["Krok", "x_n", "x_n+1", "x_n+2", "f(x_n+2)"])

    for i in range(i_max):

        # Wartości funkcji w dwóch pierwszych przybliżeniach
        yn0 = fun(xn0)
        yn1 = fun(xn1)

        # TODO
        # xn2 = ...

        yn2 = fun(xn2)

        df = df.append(pd.Series(data={"Krok": i + 1, "x_n": xn0, "x_n+1": xn1, "x_n+2": xn2, "f(x_n+2)": yn2}),
                                 ignore_index=True)

        # Sprawdzenie zbieżności
        if abs(yn2) < dok:
            print(f"Metoda siecznych zbieżna po {i + 1} krokach")
            break

        xn0, xn1 = xn1, xn2

    else:
        print("Funkcja nie mogła znaleźć przybliżenia z zadaną dokładnością w podanej liczbie iteracji")

    df = df.astype({"Krok": "int64"})
    df = df.set_index("Krok")

    return xn2, yn2, df

a, b, c = secant(lambda x: x*x + x - 5, -6, 1, 100, 0.01)
c
print(c.to_latex())
```

    Metoda siecznych zbieżna po 5 krokach
    \begin{tabular}{lrrrr}
    \toprule
    {} &       x\_n &     x\_n+1 &     x\_n+2 &  f(x\_n+2) \\
    Krok &           &           &           &           \\
    \midrule
    1    & -6.000000 &  1.000000 &  0.250000 & -4.687500 \\
    2    &  1.000000 &  0.250000 &  2.333333 &  2.777778 \\
    3    &  0.250000 &  2.333333 &  1.558140 & -1.014062 \\
    4    &  2.333333 &  1.558140 &  1.765452 & -0.117729 \\
    5    &  1.558140 &  1.765452 &  1.792681 &  0.006386 \\
    \bottomrule
    \end{tabular}

## Metoda Newtona

W metodzie Newtona jako aproksymację funkcji $f(x)$ przyjmujemy styczną do funkcji w $x_k$-tym miejscu. Następnym przybliżenie miejsca zerowego jest przecięcie stycznej z osią OX. Minusem takiego podejścia jest to, że  dodatkowo jest wymagana znajomość pochodnej zadanej funkcji. Metoda ta również nie gwarantuje zbieżności procesu. Jest ona jednak najszybszą z podstawowych metod i stąd często sięga się po nią jako pierwszą. Aby obliczyć
$k+1$-sze przybliżenie pierwiastka funkcji $f(x)$ należy zastosować wzór:

\begin{equation}
x_{k+1} = x_k - \frac{f(x_{k})}{f'(x_k)}
    \tag{5}
\end{equation}

Pseudo kod dla metody Newtona:

 Wejście:

 fun ← funkcja, dla której szukamy miejsca zerowego

 dfun ← pochodna funkcji fun

 xn0 ← pierwsze przybliżenia pierwiastka

 tol ← żądana dokładność

 i_max ← maksymalna ilość iteracji

 Wyjście:

 i → liczba iteracji

 xn1 → znalezione przybliżenie pierwiastka

 fn1 → wartość funkcji w punkcie xn1

 Algorytm:

  for  i ← 1  to  i_max  do

 xn1 ← xn0 - fun(xn0) / dfun(xn0)

 fn1 ← fun(xn1)

 if   |fn1| < dok   then

   return i, xn2, fn2

  end if

 xn0 ← xn1

  end for

## Ćwiczenie 5

Uzupełnij kod funkcji implementującej metodę Newtona.

```python
def newton(fun, dfun, xn0, i_max, dok):
    """
    :param fun: funkcja
    :param dfun: pochodna funkcji f
    :param xn0: przybliżenie pierwiastka
    :param i_max: maksymalna liczba iteracji
    :param dok: dokładność
    :return: (przybliżona wartość pierwiastka, wartość funkcji od przybliżenia, tablica z danymi z każdej iteracji)
    """

    if i_max < 1:
        raise ValueError("i_max musi być większe od 0")

    df = pd.DataFrame(columns=["Krok", "x_n", "x_n+1", "f(x_n+1)"])

    for i in range(i_max):

        # TODO
        # xn1 = ...
        yn1 = fun(xn1)

        df = df.append(pd.Series(data={"Krok": i + 1, "x_n": xn0, "x_n+1": xn1, "f(x_n+1)": yn1}),
                                 ignore_index=True)

        if abs(yn1) < dok:
            print(f"Metoda Newtona zbieżna po {i + 1} krokach")
            break

        xn0 = xn1
    else:
        print("Funkcja nie mogła znaleźć przybliżenia z zadaną dokładnością w podanej liczbie iteracji")

    df = df.astype({"Krok": "int64"})
    df = df.set_index("Krok")

    return xn1, yn1, df
```

## Metoda Newtona dla układów równań nieliniowych

Metoda Newtona dla układów równań nieliniowych jest uogólnieniem tej metody dla pojedyńczych równań. Jej algorytm  jest identyczny, wymaga tylko rozszerzenia bazowego równania tak aby pasował dla macierzy równań. W ten sposób dla  układu n równań, funkcja przyjmuje wektor n pierwszych przybliżeń pierwiastków, w miejsce pojedyńczego równania przyjmuje układ równań, a zamiast pochodnej funkcji - macierz pochodnych cząstkowych układu równań, czyli tzw. jakobian. Przytaczając wzór 5 z metody Newtona

\begin{equation}
x_{k+1} = x_k - \frac{f(x_{k})}{f'(x_k)}
\end{equation}

dla układu równań można przekształcić go do postaci

\begin{equation}
X_{k+1} - X_{k} =  - Y(X_{k})^{-1}  {J(X_k)}
    \tag{6}
\end{equation}

Zakładając teraz że lewa strona równania oznacza przyrost, należy rozwiązać równanie <Numer>  korzystając z arytmetyki macierzy

\begin{equation}
dX = -Y(X_{k})^{-1}  {J(X_k)}
    \tag{7}
\end{equation}

W języku Python, takie równanie można rozwiązać z pomocą biblioteki numpy, w następujący sposób:

```python
import numpy as np

J = ...
Y = ...

dX = -1 * inv(J).dot(Y)
# lub bardziej optymalny sposób
dX = -1 * np.linalg.solve(J, Y)
```

Należy przy tym pamiętać, że  dX ma znak ujemny

Podstawiając teraz równanie 7 do rownania 6 otrzymuje się wzór na kolejne przybliżenie wektora pierwiastków

\begin{equation}
X_{k+1} = X_{k} - dX
    \tag{8}
\end{equation}

Warunkiem stopu dla algorytmu dla układów równań, może być np. norma wektora dX, którą również można wyznaczyć za pomocą biblioteki numpy.

```python
import numpy as np

norm = np.linalg.norm(dX)
```

## Przykład 3. Wyznaczanie pochodnych cząstkowych równań

Biorąc za przykład poniższy układ równań

\begin{cases}
    f_{1} = x^{2} + y^{2} - 25  \\
    f_{2} = x + y - 7
\end{cases}

składa się on z dwóch równań oraz dwóch niewiadomych. Mówi to o kształcie macierzy Jacobi'ego która będzie miała rozmiar 2x2. Pierwszy rząd tej macierzy będzie się składał kolejno z pochodnej względem zmiennej x, a następnie z pochodnej względem zmiennej y pierwszego równania. Drugi rząd podobnie, tylko pod uwagę brane jest drugie równanie.

\begin{bmatrix}
{\frac{\partial f_{1}}{\partial x}} & {\frac{\partial f_{1}}{\partial y}}\\
{\frac{\partial f_{2}}{\partial x}} & {\frac{\partial f_{2}}{\partial y}}
\end{bmatrix}

Traktując teraz funkcję jako wielomian, pochodna cząstkowa względem $x$, będzie sumą pochodnych wyrazów w których ta zmienna występuje. Jeśli w wyrazie występują inne zmienne pozostają one w tym wyrazie. Czyli np. dla wyrazu
$x^{2}y$ pochodna cząstkowa względem $x$ będzie wynosić $2xy$. Tak samo postępuje się dla pozostałych zmiennych.

Jakobian dla układu równań wygląda następująco:

\begin{bmatrix}
2x  & 2y \\
1 & 1
\end{bmatrix}

## Ćwiczenie 6

Uzupełnij kod funkcji implementującej metodę Newtona dla układów równań.

```python
from numpy.linalg import inv, norm

def newton_nles(fun: np.array, jacobian: np.array, Xn0,  i_max, dok):
    """
    :param fun: układ X równań
    :param jacobian: jakobian równań
    :param Xn0: tablica pierwszych X przybliżeń
    :param i_max: maksymalna liczba iteracji
    :param dok: dokładność
    :return: (przybliżona wartość pierwiastka, wartość funkcji od przybliżenia, tablica z danymi z każdej iteracji)
    """

    if i_max < 1:
        raise ValueError("i_max musi być większe od 0")

    df = pd.DataFrame(columns=["Krok", "X_n", "f(X_n)"])

    for i in range(int(i_max)):
        J = jacobian(*Xn0)
        Y = fun(*Xn0)
        # TODO
        # dX = ...
        Xn1 = Xn0 - dX
        row = pd.Series(data={"Krok": i + 1, "X_n": Xn1, "f(X_n)": Y})
        df = df.append(row, ignore_index=True)

        if np.linalg.norm(dX) < dok:
            print(f'Metoda Newtona dla układu równań nieliniowych zbieżna po {i + 1} krokach')
            break

        Xn0 = Xn1

    df = df.astype({"Krok": "uint32"})
    df = df.set_index("Krok")

    return Xn1, df
```

## Ćwiczenie 7

Porównaj zbieżność poszczególnych metod szukając miejsca zerowego dla tych samych parametrów.

```python

```

## Ćwiczenie 8

Skorzystaj z funkcji omówionej w [przykładzie 1](#rrn), aby obliczyć pierwiastki równania

\begin{equation}
x^3+x^2-3x-3=0
\tag{9}
\end{equation}

```python

```

## Ćwiczenie 9

Zaproponuj takie dane wejściowe dla metody stycznych/siecznych, aby dla funkcji posiadającej miejsce zerowe w pobliżu pierwszych/-ego przybliżenia/- żeń metoda nie była zbieżna. Jakie cechy funkcji spowodowały rozbieżność?
    Zadanie to można wykonać modyfikując przybliżenie początkowe dla funkcji fun1 lub definiując inną/ne funkcję/-je.

```python

```

## Ćwiczenie 10

Równanie z [ćwiczenia 8](#cw8) można przekształcić do postaci, która umożliwia wykorzystanie metody iteracji prostej $x=\Phi(x)$ na kilka sposobów, m.in.:

$x = \frac{x^3+x^2-3}{3}$;    $x = \sqrt{-x^3+3x+3}$;

$x = \frac{-x^3+3}{x-3}$;      $x = (-x^2+3x+3)^{\frac{1}{3}}$

Zbadaj zbieżność metody iteracji prostej dla tych wzorów przyjmując różne przybliżenia początkowe pierwiastka, np: $x_0 = 1$ lub $x_0 = 0.1$.

```python

```


```
%pylab inline
figsize( 12.5, 4)
```

    Welcome to pylab, a matplotlib-based Python environment [backend: module://IPython.zmq.pylab.backend_inline].
    For more information, type 'help(pylab)'.

    c:\Python27\lib\site-packages\matplotlib\__init__.py:794: UserWarning: Bad val "348ABD, A60628, 7A68A6, 467821, CF4457, 188487, E24A33" on line #79
    	"patch.facecolor        : 348ABD, A60628, 7A68A6, 467821, CF4457, 188487, E24A33
    "
    	in file "c:\Python27\lib\site-packages\matplotlib\mpl-data\matplotlibrc"
    	348ABD, A60628, 7A68A6, 467821, CF4457, 188487, E24A33 does not look like a color arg
    Color tuples must be length 3
      "%s"\n\t%s' % (val, cnt, line, fname, msg))

#Chapter 6

This chapter focuses on the most discussed part of Bayesian methodologies: how to choose a prior distribution. We also present a very interesting connection between priors and linear regression.

## Getting our priorities straight

Up until now, we have been mostly ignored our priors. This is unfortunate as we can be very expressive with our priors, but we also must be careful about choosing them.

###Subjective vs Objective priors

Bayesian priors can be classified into two classes: *objective* priors, which aim to allow the data to speak the most, and *subjective* priors, which allow the practioner to express his or her views into the prior.

What is an example of a objective prior? We have seen a some already, including the *flat* prior (which is a uniform distribution over the entire range of the unknown). This implies we give each possible value an equal chance. Choosing this type of prior is invoking what is called "The Principle of Indifference."

Conversely, if we added more probability to certain areas of the prior, and less elsewhere, we are biasing our results towards the former area. This is an subjective, or *informative* prior. A subjective prior does not always reflect the practioner's opinion: more often the subjective prior was once a posterior to a previous problem, and know the practioner is updating this posterior with new data.

```
import scipy.stats as stats
figsize(12.5,2.5)
x = np.linspace(0,1)

y1, y2 = stats.beta.pdf(x, 1,1), stats.beta.pdf(x, 10,10)

plt.plot( x, y1,
    label='Objective \n( uninformative, \n"Principle of Indifference" )'  )
plt.plot( x,y2 ,
     label = "Subjective \n( informative)" )

leg = plt.legend()
leg.get_frame().set_alpha(0.4)
plt.title("Comparing objective vs. subjective priors" );
```

The choice, either *objective* or *subjective* mostly depend on the problem being solved, but there are a few cases where one is preferred over the other. In instances of scientific research, the choice of an objective prior is obvious. This eliminiates any biases in the results, and two researchers who might have differing prior opionions would feel an objective prior is fair. Consider an more extreme situation:

> A tobacco company publishes an report with a Bayesian methodology that retreated 60 years of medical research on tobacco use, would you believe them? No, probably because they choose a subjective prior that too strongly biased results in there favour.

Unfortunately, choosing a objective prior is not obvious, and the topic even today the problem is still not completely solved. Gelmen *et al.* [3] suggest that using a uniform distribution, with a large margin, is probably the best.

We must remember that choosing a prior, whether subjective or objective, is still part of the modelling process. To quote Gelman [3]

>...after the model has been ﬁt, one should look at the posterior distribution
and see if it makes sense. If the posterior distribution does not make sense, this implies
that additional prior knowledge is available that has not been included in the model,
and that contradicts the assumptions of the prior distribution that has been used. It is
then appropriate to go back and alter the prior distribution to be more consistent with
this external knowledge.

If the posterior does not make sense, then clearly one had an idea what the posterior *should* look like (not what one *hopes* it looks like), implying that the current prior does not contain all the prior information and should be updated. At this point, we can discard the objective prior and choose a more reflective one.

Although, one should be wary about using Uniform objective priors with large margins. Do you really think the unknown could be incredibly large? Often quantities are naturally biased towards 0. A Normal random variable with large variance (small precision) is often a better choice.

Either way, if using a subjective prior, **it is important to explain the rationale**, else you are no better than the tobacoo company's guilty parties.

We next introduce the Beta distribution.

### The Beta distribution

You may have seen the term `beta` in previous code in this book. Often, I was implementing a Beta distribution. The Beta distribution is very useful in Bayesian statistics. A random variable $X$ has a $\text{Beta}$ distribution, with paramters $(\alpha, \beta)$, if its density function is:

$$f_X(x | \; \alpha, \beta ) = \frac{ x^{(\alpha - 1)}(1-x)^{ (\beta - 1) } }{B(\alpha, \beta) }$$

where $B$ is the [Beta function](http://en.wikipedia.org/wiki/Beta_function) (hence the name). The random variable $X$ is only allowed in [0,1], making the Beta distribution a popular distribution for decimal values, probabilities and proportions. The values of $\alpha$ and $\beta$, both positive values, provide great flexibility in the shape of the distribution. Below we plot some distributions:

```
figsize( 12.5, 5 )
import scipy.stats as stats

params = [ (1,1), (2,5), (0.5, 0.5), ( 5, 5), (20, 4), (5, 1) ]
colors = ["#348ABD", "#A60628", "#7A68A6", "#467821",
        "#CF4457", "#188487", "#E24A33" ]
x = np.linspace( 0, 1, 100 )
beta = stats.beta
for c, (a, b) in zip(colors,params):
    y = beta.pdf( x, a, b )
    plt.plot( x, y, label = "(%.1f,%.1f)"%(a,b) )
    plt.fill_between( x, 0, y, alpha = 0.2, color = c )

plt.ylim(0)
plt.legend(loc = 'upper left');
```

One thing I'd like the reader to notice is the flat distribution above, specified by parameters $(1,1)$. This is the Uniform distribution. Hence the Beta distribution is a generalization of the Uniform distribution, something we will revisit many times.

There is an interesting connection beween the Beta distribution and the Binomial distribution. Suppose we are interested in some unknown proportion or probability $p$. We assign a $\text{Beta}(\alpha, \beta)$ prior to $p$. We observe some data genereted by a Binomial process, say $X \sim \text{Binomial}(N, p)$. Then our posterior *is a Beta disitribution*, i.e. $p | X \sim \text{Beta}( \alpha + X, \beta + N -X )$. Succinctly, one can relate the two by "a Beta prior with Binomial observations creates a Beta posterior". This is a very useful property, both computationally and heuristically.

In light of the above two paragraphs, if we start with a $\text{Beta}(1,1)$ prior on $p$ (which is a Uniform), observe data $X \sim \text{Binomial}(N, p)$, then our posterior is $\text{Beta}(1 + X, 1 + N - X)$.

#####Example: Bayesian Multi-Armed Bandits
*Adapted from an example by Ted Dunning of MapR Technologies*

> Suppose you are faced with $N$ slot machines (colourfully called multi-armed bandits). Each bandit has an unknown probability of distributing a prize (assume for now the prizes are the same for each bandit, only the probabilities differ). Some bandits are very generous, others not so much. Of course, you don't know what these probabilities are. By only choosing one bandit per round, our task is devise a strategy to maximize our winnings.

Of course, if we knew the bandit with the largest probability, then always picking this bandit would yield the maximum winnings. So our task can be phrased as "Find the best bandit, and as quickly as possible".

The task is complicated by the stochastic nature of the bandits. A suboptimal bandit can return many winnings, purely by chance, which would make us believe that it is a very profitable bandit. Similarly, the best bandit can return many duds. Should we keep trying losers then, or give up?

A more troublesome problem is, if we have a found a bandit that returns *pretty good* results, do we keep drawing from it to maintain our *pretty good score*, or do we try other bandits in hopes of finding an *even-better* bandit? This is the exploration vs. exploitation dilemma.

### Applications

The Multi-Armed Bandit problem at first seems very artificial, something only a mathematician would love, but that is only before we address some applications:

1. Internet display advertising: companies have a suite of potential ads they can display to visitors, but the company is not sure which ad strategy to follow to maximize sales. This is similar to A/B testing, but has the added advantage of naturally minimizing strategies that do not work (and generalizes to A/B/C/D... strategies)
2. Ecology: animals have a finite amount of energy to expend, and following certain behaviours has uncertain rewards. How does the animal maximize its fitness?
3. Finance: which stock option gives the highest return, under time-varying return profiles.
4. Clinical trials: a researcher would like to find the best treatment, out of many possible treatment, while minimizing losses.
5. Psychology: how does punishment and reward effect our behaviour? How do humans' learn?

Many of these questions above a fundamental to the application's field.

It turns out the *optimal solution* is incredibly difficult, and it took decades for an overall solution to develop. There are also many approximately-optimal solutions which are quite good. The one I wish to discuss is one of the few solutions that can scale incredibly well. The solution is known as *Bayesian Bandits*.

### A Proposed Solution

Any proposed strategy is called an *online algorithm* (not in the internet sense, but in the continuously-being-updated sense), and more specifically a reinforcement learning algorithm. The algorithm starts in an ignorant state, where it knows nothing, and begins to acquire data by testing the system. As it acquires data and results, it learns what the best and worst behaviours are (in this case, it learns which bandit is the best). With this in mind, perhaps we can add an additional application of the Multi-Armed Bandit problem:

- Psychology: how does punishment and reward effect our behaviour? How do humans' learn?

The Bayesian solution begins by assuming priors on the probability of winning for each bandit. In our vignette we assumed complete ignorance of the these probabilities. So a very natural prior is the flat prior over 0 to 1. The algorithm proceeds as follows:

For each round:

1. Sample a random variable $X_b$ from the prior of bandit $b$, for all $b$.
2. Select the bandit with largest sample, i.e. select $B = \text{argmax}\;\; X_b$.
3. Observe the result of pulling bandit $B$, and update your prior on bandit $B$.
4. Return to 1.

That's it. Computationally, the algorithm involves sampling from $N$ distributions. Since the initial priors are $\text{Beta}(\alpha=1,\beta=1)$ (a uniform distribution), and the observed result $X$ (a win or loss, encoded 1 and 0 respectfully) is Binomial, the posterior is a $\text{Beta}(\alpha=1+X,\beta=1+1−X)$.

To answer our question from before, this algorithm suggests that we should not discard losers, but we should pick them at a decreasing rate as we gather confidence that there exist *better* bandits. This follows because there is always a non-zero chance that a loser will achieve the status of $B$, but the probability of this event decreases as we play more rounds (see figure below).

Below we implement Bayesian Bandits using two classes, `Bandits` that defines the slot machines, and `BayesianStrategy` which implements the above learning strategy.

```
from pymc import rbeta

rand = np.random.rand

class Bandits(object):
    """
    This class represents N bandits machines.

    parameters:
        p_array: a (n,) Numpy array of probabilities >0, <1.

    methods:
        pull( i ): return the results, 0 or 1, of pulling
                   the ith bandit.
    """
    def __init__(self, p_array):
        self.p = p_array
        self.optimal = np.argmax(p_array)

    def pull( self, i ):
        #i is which arm to pull
        return rand() < self.p[i]

    def __len__(self):
        return len(self.p)

class BayesianStrategy( object ):
    """
    Implements a online, learning strategy to solve
    the Multi-Armed Bandit problem.

    parameters:
        bandits: a Bandit class with .pull method

    methods:
        sample_bandits(n): sample and train on n pulls.

    attributes:
        N: the cumulative number of samples
        choices: the historical choices as a (N,) array
        bb_score: the historical score as a (N,) array

    """

    def __init__(self, bandits):

        self.bandits = bandits
        n_bandits = len( self.bandits )
        self.wins = np.zeros( n_bandits )
        self.trials = np.zeros(n_bandits )
        self.N = 0
        self.choices = []
        self.bb_score = []

    def sample_bandits( self, n=1 ):

        bb_score = np.zeros( n )
        choices = np.zeros( n )

        for k in range(n):
            #sample from the bandits's priors, and select the largest sample
            choice = np.argmax( rbeta( 1 + self.wins, 1 + self.trials - self.wins) )

            #sample the chosen bandit
            result = self.bandits.pull( choice )

            #update priors and score
            self.wins[ choice ] += result
            self.trials[ choice ] += 1
            bb_score[ k ] = result
            self.N += 1
            choices[ k ] = choice

        self.bb_score = np.r_[ self.bb_score, bb_score ]
        self.choices = np.r_[ self.choices, choices ]
        return
```

Below we visualize the learning of the Bayesian Bandit solution.

```
import scipy.stats as stats
figsize( 11.0, 10)

beta = stats.beta
x = np.linspace(0,1,200)

def plot_priors(bayesian_strategy, prob):
    ## plotting function
    wins = bayesian_strategy.wins
    trials = bayesian_strategy.trials
    for i in range( prob.shape[0] ):
        y = beta( 1+wins[i], 1 + trials[i] - wins[i] )
        p = plt.plot( x, y.pdf(x) )
        c = p[0].get_markeredgecolor()
        plt.fill_between(x,y.pdf(x),0, color = c, alpha = 0.3,
                         label="underlying probability: %.2f"%prob[i])
        plt.vlines( prob[i], 0, y.pdf(prob[i]) ,
                   colors = c, linestyles = "--" )
        plt.title("Posteriors After %d pull"%bayesian_strategy.N +\
                    "s"*(bayesian_strategy.N>1) )
    return

hidden_prob = np.array([0.85, 0.60, 0.75] )
bandits = Bandits( hidden_prob )
bayesian_strat = BayesianStrategy( bandits )

for j,i in enumerate([1, 1, 3, 10, 10, 25, 50, 100, 200, 600 ]):
    subplot( 5, 2, j+1)
    bayesian_strat.sample_bandits(i)
    plot_priors( bayesian_strat, hidden_prob )
    #plt.legend()

plt.tight_layout()

```

Note that we don't real care how accurate we become about inference of the hidden probabilities -- for this problem we are more interested in choosing the best bandit (or more accurately, becoming *more confident* in choosing the best bandit). For this reason, the distribution of the red bandit is very wide (representing ignorance about what that hidden probability might be) but we are reasonably confident that it is not the best, so the algorithm chooses to ignore it.

### A Measure of *Good*

We need a metric to calculate how well we are doing. Recall the absolute *best* we can do is to always pick the bandit with the largest probability of winning. Denote this best bandit's probability of $w^*$. Our score should be relative to how well we would have done had we chosen the best bandit from the beginning. This motivates the *total regret* of a strategy, defined:

\begin{align}
R_T & = \sum_{i=1}^{T} \left( w^* - w_{B(i)} \right)\\\\
& = Tw^* - \sum_{i=1}^{T} \;  w_{B(i)}
\end{align}

where $w_{B(i)}$ is the probability of a prize of the chosen bandit in the $i$ round. A total regret of 0 means the strategy is matching the best possible score. This is likely not possible, as initially our algorithm will often make the wrong choice.  Ideally, a strategy's total regret should flatten as it learns the best bandit. (Mathematically we achieve $w_{B(i)}=w^*$ often)

Below we plot the total regret of this simulation, including the score if we randomly chose a bandit.

```
figsize( 12.5, 3 )

def regret( probabilities, choices ):
    w_opt = probabilities.max()
    return ( w_opt - probabilities[choices.astype(int)] ).cumsum()

#train 9000 more times. It's over 9000 now.
bayesian_strat.sample_bandits(9000)

bayesian_strategy_regret =  regret( hidden_prob,
                                    bayesian_strat.choices )
random_strategy_regret = regret( hidden_prob,
                                 np.random.randint( 0, 3, size=10000) )

#plotit
plt.plot(bayesian_strategy_regret, label = "Bayesian Bandit Strategy" )
plt.plot( random_strategy_regret, label = "Random guessing" )
plt.title("Total Regret of Bayesian Bandits Strategy vs. Random guessing" )
plt.xlabel("Number of pulls")
plt.ylabel("Regret after $n$ pulls");
plt.legend(loc = "upper left")

figure()
plt.plot(bayesian_strategy_regret, label = "Bayesian Bandit Strategy" )
plt.title("Total Regret of Bayesian Bandits Strategy" )
plt.xlabel("Number of pulls")
plt.ylabel("Regret after $n$ pulls");
plt.legend(loc = "upper left")
```

Like we wanted, the graph starts to flatten, representing we are achieving  optimal choices. To be more scientific so as to remove any possible luck in the above simulation, we should instead look at the *expected total regret*:

$$\bar{R_T} = E[ R_T ] $$

It can be shown that any *sub-optimal* strategy's expected total regret is bounded below logarithmically. Formally,

$$ E[R_T] = \Omega \left( \;\log(T)\; \right)$$

Thus, any strategy that matches logarithmic-growing regret is said to "solve" the Multi-Armed Bandit problem [3].

Using the Law of Large Numbers, we can approximate Bayesian Bandit's expected total regret by performing the same experiment many times (25 times, to be fair):

```
expected_total_regret = np.zeros(10000)
trials = 25

for i in range(trials):
    bayesian_strat = BayesianStrategy( bandits )
    bayesian_strat.sample_bandits(10000)
    bayesian_strategy_regret =  regret( hidden_prob,
                                    bayesian_strat.choices )
    expected_total_regret += bayesian_strategy_regret

plot(expected_total_regret/trials )
#plt.ylim( 0, expected_total_regret.max() )
plt.title("Expected Total Regret of Bayesian Bandits Strategy" )
plt.xlabel("Number of pulls")
plt.ylabel("Exepected Total Regret after $n$ pulls");

figure()
plot(expected_total_regret/trials )
#plt.ylim( 0, expected_total_regret.max() )
plt.title("Expected Total Regret of Bayesian Bandits Strategy, log-scale" )
plt.xlabel("Log Number of pulls")
plt.ylabel("Exepected Total Regret \n after $\log{n}$ pulls");
plt.xscale("log")

```

### Extending the algorithm

Because of the algorithm's simplicity, it is easy to extend it. Some possibilities:

1. If interested in the *minimum* probability (eg: where prizes are are bad thing), simply choose $B = \text{argmin} \; X_b$ and proceed.

2. Adding learning rates: Suppose the underlying environment may change over time. Technically the standard Bayesian Bandit algorithm would self-update itself (awesome) by noting that what it thought was the best is starting to fail more often, we can motivate the algorithm to learn changing environments quicker. We simply need to add a *rate* term upon updating:

        self.wins[ choice ] = rate*self.wins[ choice ] + result
        self.trials[ choice ] = rate*self.trials[ choice ] + 1

   If `rate < 1`, the algorithm will *forget* its previous wins quicker and there will be a downward  pressure towards ignorance. Conversely, setting `rate > 1` implies your algorithm will act more risky, and bet on earlier winners more often and be more resistant to changing environments.

2. Hierarchical algorithms: We can setup a Bayesian Bandit algorithm on top of smaller bandit algorithms. Suppose we have $N$ Bayesian Bandit models, each varying in some behavior (for example  different `rate` parameters, representing varying sensitivity to changing environments). On top of these $N$ models is another Bayesian Bandit learner that will select a sub-Bayesian Bandit. This chosen Bayesian Bandit will then make an internal choice as to which machine to pull. The super-Bayesian Bandit updates itself depending on whether the sub-Bayesian Bandit was correct or not.

```
figsize( 12.0, 12)
beta = stats.beta
hidden_prob = beta.rvs(1,1, size = 10 )
print hidden_prob
bandits = Bandits( hidden_prob )
bayesian_strat = BayesianStrategy( bandits )

for j,i in enumerate([1, 1, 3, 10, 10, 25, 50, 100, 200, 600 ]):
    subplot( 5, 2, j+1)
    bayesian_strat.sample_bandits(i)
    plot_priors( bayesian_strat, hidden_prob )
    #plt.legend()

```

```
def regret( probabilities, choices ):
    w_opt = probabilities.max()
    return ( w_opt - probabilities[choices.astype(int)] ).cumsum()

#train 9000 more times. It's over 9000 now.

bayesian_strat.sample_bandits(9000)

bayesian_strategy_regret =  regret( hidden_prob,
                                    bayesian_strat.choices )

figure()
plt.plot(bayesian_strategy_regret, label = "Bayesian Bandit Strategy" )
plt.title("Total Regret of Bayesian Bandits Strategy" )
plt.xlabel("Number of pulls")
plt.ylabel("Regret after $n$ pulls");
plt.legend(loc = "upper left")
plt.xscale("log")

```

##Effect of the prior as $N$ increases

In the first Chapter, I proposed that as the amount of observations, or data, that we posses, the less the prior matters. This is intuitive. After all, our prior is based on previous information, and eventually enough new information will shadow our previous information's value. The smothering of the prior by enough data is also helpful: if our prior is significantly wrong, then the self-correcting nature of the data will present to us a *less wrong*, and eventually *correct*, posterior.

We can see this mathematically. First, recall Bayes Theorem from Chapter 1 that relates the prior to the posterior. The following is a sample from [What is the relationship between sample size and the influence of prior on posterior?](http://stats.stackexchange.com/questions/30387/what-is-the-relationship-between-sample-size-and-the-influence-of-prior-on-poste)[1] on CrossValidated.

>The posterior distribution for a parameter $\theta$, given a data set ${\bf X}$ can be written as

\begin{equation}
p(\theta | {\bf X}) \propto \underbrace{p({\bf X} | \theta)}_{{\rm likelihood}} \cdot \underbrace{ p(\theta)}_{{\rm prior} }
\end{equation}

>or, as is more commonly displayed on the log scale,

$$ \log( p(\theta | {\bf X})  ) = c + L(\theta;{\bf X}) + \log(p(\theta)) $$

>The log-likelihood, $L(\theta;{\bf X}) = \log \left( p({\bf X}|\theta) \right)$, **scales with the sample size**, since it is a function of the data, while the prior density does not. Therefore, as the sample size increases, the absolute value of $L(\theta;{\bf X})$ is getting larger while $\log(p(\theta))$ stays fixed (for a fixed value of $\theta$), thus the sum $L(\theta;{\bf X}) + \log(p(\theta))$ becomes more heavily influenced by $L(\theta;{\bf X})$ as the sample size increases.

There is an interesting consequence not immediately apparent. For large sample sizes, the choice prior has less influence. Hence inference converges regardless of chosen prior, so long as areas of the prior assign non-zero probabilities are the same.

Below we visualize this. We examine the convergence of two posteriors of a Binomial's parameter $\theta$, one with a flat prior and the other with a biased prior towards 0. As the sample size increases, the posteriors, and hence the inference, converge.

```
figsize( 12.5, 15)
import scipy.stats as stats
import pymc as mc

p = 0.6
beta1_params = np.array( [1.,1.] )
beta2_params = np.array( [2,10] )
beta = stats.beta

x = np.linspace(0.00, 1, 125)
data = mc.rbernoulli(p, size=500)

figure()
for i,N in enumerate([0,4,8, 32,64, 128, 500]):
    s = data[:N].sum()
    subplot(8,1,i+1)
    params1 = beta1_params + np.array( [s, N-s] )
    params2 = beta2_params + np.array( [s, N-s] )
    y1,y2 = beta.pdf( x, *params1), beta.pdf( x, *params2)
    plt.plot( x,y1, label = r"flat prior", lw =3 )
    plt.plot( x, y2, label = "biased prior", lw= 3 )
    plt.fill_between( x, 0, y1, color ="#348ABD", alpha = 0.15)
    plt.fill_between( x, 0, y2, color ="#A60628", alpha = 0.15)
    plt.legend(title = "N=%d"%N)
    plt.vlines( p, 0.0, 7.5, linestyles = "--", linewidth=1)
    #plt.ylim( 0, 10)#

```

Keep in mind, not all posteriors will "forget" the prior this quickly. This example was just to show that *eventually* the prior is forgotten.

### Bayesian perspective of Penalized Linear Regressions

There is a very interesting relationship between the penalized least-squares regression and Bayesian priors. We will first describe the probablistic interpretation of Linear Regression. Denote our response variable $Y$, and features are contained in the data matrix $X$. The standard linear model is:

\begin{equation}
Y = X\beta + \epsilon
\end{equation}

where $\epsilon \sim \text{Normal}( {\bf 0}, \sigma{\bf I })$. Simpley, the observed $Y$ is a linear function of $X$ (with coeffcients $\beta$) plus some noise term. Our unknown to be learned is $\beta$. We use the following property of Normal random variables:

$$ \mu' + \text{Normal}( \mu, \sigma ) \sim \text{Normal}( \mu' + \mu , \sigma ) $$

to rewrite the above linear model as:

\begin{align}
& Y = X\beta + \text{Normal}( {\bf 0}, {\bf I }) \\\\
& Y = \text{Normal}( X\beta , {\bf I }) \\\\
\end{align}

In probabilitistic notation, denote $f_Y(y \; | \; \beta )$ the probabilitiy distribution of $Y$, and recall the density function for a Normal random variable:

$$ f_Y( Y \; |\; \beta, X) = L(\beta|\; X,Y)= \frac{1}{\sqrt{ 2\pi\sigma} } \exp \left( \frac{1}{2\sigma^2} (Y - X\beta)^T(Y - X\beta) \right) $$

This is the likelihood function for $\beta$ Taking the $\log$:

$$ \ell(\beta) = K - c(Y - X\beta)^T(Y - X\beta) $$

where $K$ and $c>0$ are constants. Maximum likelihood techniques wish to maximize this for $\beta$,

$$\hat{ \beta } = \text{argmax}_{\beta} \;\; - (Y - X\beta)^T(Y - X\beta) $$

Equivilantly we can minimize the negative of the above:

$$\hat{ \beta } = \text{argmin}_{\beta} \;\; (Y - X\beta)^T(Y - X\beta) $$

This is probably the familiar least-squares linear regression equation. We showed that the solution to this is the same as the maximum likelihood assuming Normal noise. Next we extend this to show how we can arrive at penalized linear regression by a suitable choice of priors on $\beta$.

#### Penalized least-squares

In the above, once we have the likelihood, we can include a prior distribution on $\beta$ to move to the equation for the posterior distribution:

$$P( \beta | Y, X ) = L(\beta|\;X,Y)p( \beta )$$

where $p(\beta)$ is a prior on the elements of $\beta$. What are some interesting priors?

1\. If we include *no explicit* prior term, we are actually including an uniformative prior, $P( \beta ) \propto 1$.

2\. If we have reason to believe the elements of $\beta$ are not too large, we can suppose that *apriori*:

$$ \beta \sim \text{Normal}({\bf 0 }, \lambda {\bf I } ) $$

The resulting posterior density function for $\beta$ is *proportional to*:

$$ \exp \left( \frac{1}{2\sigma^2} (Y - X\beta)^T(Y - X\beta) \right) \exp \left( \frac{1}{2\lambda^2} \beta^T\beta \right) $$

and taking the $\log$ of this, and combining some constants, we get:

$$ \ell(\beta) \propto K -  (Y - X\beta)^T(Y - X\beta) - \alpha \beta^T\beta  $$

we arrive at the function we wish to maximize (recall the point that maximizes the posterior distribution is the MAP, or *maximum a posterior*):

$$\hat{ \beta } = \text{argmax}_{\beta} \;\; -(Y - X\beta)^T(Y - X\beta) - \alpha \;\beta^T\beta $$

Equivilantly, we can minimize the negative of the above, and rewriting $\beta^T\beta = ||\beta||_2^2:

$$\hat{ \beta } = \text{argmin}_{\beta} \;\; (Y - X\beta)^T(Y - X\beta) + \alpha \;||\beta||_2^2$$

This above term is exactly Ridge Regression. Thus we can see that ridge regression corresponds to the MAP of a linear model with Normal errors and a Normal prior on $\beta$.

3\. Similarly, if we assume a *Laplace* prior on $\beta$, ie.

$$ f_\beta( \beta) \propto \exp \left(- \lambda ||\beta||_1 \right)$$

and following the same steps as above, we recover:

$$\hat{ \beta } = \text{argmin}_{\beta} \;\; (Y - X\beta)^T(Y - X\beta) + \alpha \;||\beta||_1$$

which is LASSO regression. Some important notes about this equilivance. The sparsity that is a result of using a LASSO regularization is not a result of the prior assigning high probability to sparsity. Quite the opposite actually. It is the combination of the $|| \cdot ||_1$ function and using the MAP that creates sparsity on $\beta$: purely a geometric argument. The prior does contribute to an overall shrinking of the coefficients towards 0 though. An interesing disucssion of this can be found in [2].

##### References

1. http://stats.stackexchange.com/questions/30387/what-is-the-relationship-between-sample-size-and-the-influence-of-prior-on-poste

2. Starck, J.-L., , et al. "Sparsity and the Bayesian Perspective." Astronomy & Astrophysics. (2013): n. page. Print.

3. Kuleshov, Volodymyr, and Doina Precup. "Algorithms for the multi-armed bandit problem." Journal of Machine Learning Research. (2000): 1-49. Print.

```
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
    }
    div.cell{
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    h1 {
        font-family: Helvetica, serif;
    }
    h4{
        margin-top:12px;
        margin-bottom: 3px;
       }
    div.text_cell_render{
        font-family: Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 145%;
        font-size: 120%;
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro", source-code-pro,Consolas, monospace;
    }
    .prompt{
        display: None;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 16pt;
        color: #4057A1;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }

</style>