#  CS109A  Introduction to Data Science

## Lab 3: plotting, K-NN Regression, Simple Linear Regression

**Harvard University**

**Fall 2019**

**Instructors:** Pavlos Protopapas, Kevin Rader, and Chris Tanner

**Material prepared by**: David Sondak, Will Claybaugh, Pavlos Protopapas, and Eleni Kaxiras.

##  Extended Edition

Same as the one done in class with the following additions/clarifications:

* I added another example to illustrate the difference between `.iloc` and `.loc` in `pandas` -- > [here](#iloc)
* I added some notes on why we are adding a constant in our linear regression model --> [here](#constant)
* How to run the solutions: Uncomment the following line and run the cell:

```python
# %load solutions/knn_regression.py
```
This will bring up the code in the cell but WILL NOT RUN it. You need to run the cell again in order to actually run the code

---

```python
#RUN THIS CELL
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2018-CS109A/master/content/styles/cs109.css").text
HTML(styles)
```

## Learning Goals

By the end of this lab, you should be able to:
* Review `numpy` including 2-D arrays and understand array reshaping
* Use `matplotlib` to make plots
* Feel comfortable with simple linear regression
* Feel comfortable with $k$ nearest neighbors

**This lab corresponds to lectures 4 and 5 and maps on to homework 2 and beyond.**

## Table of Contents

####  HIGHLIGHTS FROM PRE-LAB

* [1 - Review of numpy](#first-bullet)
* [2 - Intro to matplotlib plus more ](#second-bullet)

####  LAB 3 MATERIAL

* [3 - Simple Linear Regression](#third-bullet)
* [4 - Building a model with `statsmodels` and `sklearn`](#fourth-bullet)
* [5 - Example: Simple linear regression with automobile data](#fifth-bullet)
* [6 - $k$Nearest Neighbors](#sixth-bullet)

```python
import numpy as np
import scipy as sp
import matplotlib as mpl
import matplotlib.cm as cm
import matplotlib.pyplot as plt
import pandas as pd
import time
pd.set_option('display.width', 500)
pd.set_option('display.max_columns', 100)
pd.set_option('display.notebook_repr_html', True)
#import seaborn as sns
import warnings
warnings.filterwarnings('ignore')
# Displays the plots for us.
%matplotlib inline
```

```python
# Use this as a variable to load solutions: %load PATHTOSOLUTIONS/exercise1.py. It will be substituted in the code
# so do not worry if it disappears after you run the cell.
PATHTOSOLUTIONS = 'solutions'
```

## 1 - Review of  the  `numpy` Python library

In lab1 we learned about the `numpy` library [(documentation)](http://www.numpy.org/) and its fast array structure, called the `numpy array`.

```python
# import numpy
import numpy as np
```

```python
# make an array
my_array = np.array([1,4,9,16])
my_array
```

```python
print(f'Size of my array: {my_array.size}, or length of my array: {len(my_array)}')
print (f'Shape of my array: {my_array.shape}')
```

#### Notice the way the shape appears in numpy arrays

- For a 1D array, .shape returns a tuple with 1 element (n,)
- For a 2D array, .shape returns a tuple with 2 elements (n,m)
- For a 3D array, .shape returns a tuple with 3 elements (n,m,p)

```python
# How to reshape a 1D array to a 2D
my_array.reshape(-1,2)
```

Numpy arrays support the same operations as lists! Below we slice and iterate.

```python
print("array[2:4]:", my_array[2:4]) # A slice of the array

# Iterate over the array
for ele in my_array:
    print("element:", ele)
```

Remember `numpy` gains a lot of its efficiency from being **strongly typed** (all elements are of the same type, such as integer or floating point). If the elements of an array are of a different type, `numpy` will force them into the same type (the longest in terms of bytes)

```python
mixed = np.array([1, 2.3, 'eleni', True])
print(type(1), type(2.3), type('eleni'), type(True))
mixed # all elements will become strings
```

Next, we push ahead to two-dimensional arrays and begin to dive into some of the deeper aspects of `numpy`.

```python
# create a 2d-array by handing a list of lists
my_array2d = np.array([ [1, 2, 3, 4],
                        [5, 6, 7, 8],
                        [9, 10, 11, 12]
])

my_array2d
```

### Array Slicing (a reminder...)

Numpy arrays can be sliced, and can be iterated over with loops.  Below is a schematic illustrating slicing two-dimensional arrays.

Notice that the list slicing syntax still works!
`array[2:,3]` says "in the array, get rows 2 through the end, column 3]"
`array[3,:]` says "in the array, get row 3, all columns".

### Pandas Slicing (a reminder...)

`.iloc` is by position (position is unique), `.loc` is by label (label is not unique)

```python
# import cast dataframe
cast = pd.read_csv('../data/cast.csv', encoding='utf_8')
cast.head()
```

```python
# get me rows 10 to 13 (python slicing style : exclusive of end)
cast.iloc[10:13]
```

```python
# get me columns 0 to 2 but all rows - use head()
cast.iloc[:, 0:2].head()
```

```python
# get me rows 10 to 13 AND only columns 0 to 2
cast.iloc[10:13, 0:2]
```

```python
# COMPARE: get me rows 10 to 13 (pandas slicing style : inclusive of end)
cast.loc[10:13]
```

```python
# give me columns 'year' and 'type' by label but only for rows 5 to 10
cast.loc[5:10,['year','type']]
```

#### Another example of positioning with `.iloc` and `loc`

Look at the following data frame. It is a bad example because we have duplicate values for the index but that is legal in pandas. It's just a bad practice and we are doing it to illustrate the difference between positioning with `.iloc` and `loc`. To keep rows unique, though, internally, `pandas` has its own index which in this dataframe runs from `0` to `2`.

```python
index = ['A', 'Z', 'A']
famous = pd.DataFrame({'Elton': ['singer', 'Candle in the wind', 'male'],
                  'Maraie': ['actress' , 'Do not know', 'female'],
                  'num': np.random.randn(3)}, index=index)
famous
```

```python
# accessing elements by label can bring up duplicates!!
famous.loc['A']  # since we want all rows is the same as famous.loc['A',:]
```

```python
# accessing elements by position is unique - brings up only one row
famous.iloc[1]
```

## 2 - Plotting with matplotlib and beyond

`matplotlib` is a very powerful `python` library for making scientific plots.

We will not focus too much on the internal aspects of `matplotlib` in today's lab. There are many excellent tutorials out there for `matplotlib`.  For example,
* [`matplotlib` homepage](https://matplotlib.org/)
* [`matplotlib` tutorial](https://github.com/matplotlib/AnatomyOfMatplotlib)

Conveying your findings convincingly is an absolutely crucial part of any analysis. Therefore, you must be able to write well and make compelling visuals.  Creating informative visuals is an involved process and we won't cover that in this lab.  However, part of creating informative data visualizations means generating *readable* figures.  If people can't read your figures or have a difficult time interpreting them, they won't understand the results of your work.  Here are some non-negotiable commandments for any plot:
* Label $x$ and $y$ axes
* Axes labels should be informative
* Axes labels should be large enough to read
* Make tick labels large enough
* Include a legend if necessary
* Include a title if necessary
* Use appropriate line widths
* Use different line styles for different lines on the plot
* Use different markers for different lines

There are other important elements, but that list should get you started on your way.

We will work with `matplotlib` and `seaborn` for plotting in this class.  `matplotlib` is a very powerful `python` library for making scientific plots.  `seaborn` is a little more specialized in that it was developed for statistical data visualization.  We will cover some `seaborn` later in class. In the meantime you can look at the [seaborn documentation](https://seaborn.pydata.org)

First, let's generate some data.

#### Let's plot some functions

We will use the following three functions to make some plots:

* Logistic function:
  \begin{align*}
    f\left(z\right) = \dfrac{1}{1 + be^{-az}}
  \end{align*}
  where $a$ and $b$ are parameters.
* Hyperbolic tangent:
  \begin{align*}
    g\left(z\right) = b\tanh\left(az\right) + c
  \end{align*}
  where $a$, $b$, and $c$ are parameters.
* Rectified Linear Unit:
  \begin{align*}
    h\left(z\right) =
    \left\{
      \begin{array}{lr}
        z, \quad z > 0 \\
        \epsilon z, \quad z\leq 0
      \end{array}
    \right.
  \end{align*}
  where $\epsilon < 0$ is a small, positive parameter.

You are given the code for the first two functions.  Notice that $z$ is passed in as a `numpy` array and that the functions are returned as `numpy` arrays.  Parameters are passed in as floats.

You should write a function to compute the rectified linear unit.  The input should be a `numpy` array for $z$ and a positive float for $\epsilon$.

```python
import numpy as np

def logistic(z: np.ndarray, a: float, b: float) -> np.ndarray:
    """ Compute logistic function
      Inputs:
         a: exponential parameter
         b: exponential prefactor
         z: numpy array; domain
      Outputs:
         f: numpy array of floats, logistic function
    """

    den = 1.0 + b * np.exp(-a * z)
    return 1.0 / den

def stretch_tanh(z: np.ndarray, a: float, b: float, c: float) -> np.ndarray:
    """ Compute stretched hyperbolic tangent
      Inputs:
         a: horizontal stretch parameter (a>1 implies a horizontal squish)
         b: vertical stretch parameter
         c: vertical shift parameter
         z: numpy array; domain
      Outputs:
         g: numpy array of floats, stretched tanh
    """
    return b * np.tanh(a * z) + c

def relu(z: np.ndarray, eps: float = 0.01) -> np.ndarray:
    """ Compute rectificed linear unit
      Inputs:
         eps: small positive parameter
         z: numpy array; domain
      Outputs:
         h: numpy array; relu
    """
    return np.fmax(z, eps * z)
```

Now let's make some plots.  First, let's just warm up and plot the logistic function.

```python
x = np.linspace(-5.0, 5.0, 100) # Equally spaced grid of 100 pts between -5 and 5

f = logistic(x, 1.0, 1.0) # Generate data
```

```python
plt.plot(x, f)
plt.xlabel('x')
plt.ylabel('f')
plt.title('Logistic Function')
plt.grid(True)
```

#### Figures with subplots

Let's start thinking about the plots as objects. We have the `figure` object which is like a matrix of smaller plots named `axes`. You can use array notation when handling it.

```python
fig, ax = plt.subplots(1,1) # Get figure and axes objects

ax.plot(x, f) # Make a plot

# Create some labels
ax.set_xlabel('x')
ax.set_ylabel('f')
ax.set_title('Logistic Function')

# Grid
ax.grid(True)
```

Wow, it's *exactly* the same plot!  Notice, however, the use of `ax.set_xlabel()` instead of `plt.xlabel()`.  The difference is tiny, but you should be aware of it.  I will use this plotting syntax from now on.

What else do we need to do to make this figure better?  Here are some options:
* Make labels bigger!
* Make line fatter
* Make tick mark labels bigger
* Make the grid less pronounced
* Make figure bigger

Let's get to it.

```python
fig, ax = plt.subplots(1,1, figsize=(10,6)) # Make figure bigger

# Make line plot
ax.plot(x, f, lw=4)

# Update ticklabel size
ax.tick_params(labelsize=24)

# Make labels
ax.set_xlabel(r'$x$', fontsize=24) # Use TeX for mathematical rendering
ax.set_ylabel(r'$f(x)$', fontsize=24) # Use TeX for mathematical rendering
ax.set_title('Logistic Function', fontsize=24)

ax.grid(True, lw=1.5, ls='--', alpha=0.75)
```

Notice:
* `lw` stands for `linewidth`.  We could also write `ax.plot(x, f, linewidth=4)`
* `ls` stands for `linestyle`.
* `alpha` stands for transparency.

The only thing remaining to do is to change the $x$ limits.  Clearly these should go from $-5$ to $5$.

```python
#fig.savefig('logistic.png')

# Put this in a markdown cell and uncomment this to check what you saved.
#
```

#### Resources
If you want to see all the styles available, please take a look at the documentation.
* [Line styles](https://matplotlib.org/2.0.1/api/lines_api.html#matplotlib.lines.Line2D.set_linestyle)
* [Marker styles](https://matplotlib.org/2.0.1/api/markers_api.html#module-matplotlib.markers)
* [Everything you could ever want](https://matplotlib.org/2.0.1/api/lines_api.html#matplotlib.lines.Line2D.set_marker)

We haven't discussed it yet, but you can also put a legend on a figure.  You'll do that in the next exercise.  Here are some additional resources:
* [Legend](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.legend.html)
* [Grid](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.grid.html)

`ax.legend(loc='best', fontsize=24);`

Exercise

Do the following:
* Make a figure with the logistic function, hyperbolic tangent, and rectified linear unit.
* Use different line styles for each plot
* Put a legend on your figure

Here's an example of a figure:

```python
# your code here

# First get the data
f = logistic(x, 2.0, 1.0)
g = stretch_tanh(x, 2.0, 0.5, 0.5)
h = relu(x)

fig, ax = plt.subplots(1,1, figsize=(10,6)) # Create figure object

# Make actual plots
# (Notice the label argument!)
ax.plot(x, f, lw=4, ls='-', label=r'$L(x;1)$')
ax.plot(x, g, lw=4, ls='--', label=r'$\tanh(2x)$')
ax.plot(x, h, lw=4, ls='-.', label=r'$relu(x; 0.01)$')

# Make the tick labels readable
ax.tick_params(labelsize=24)

# Set axes limits to make the scale nice
ax.set_xlim(x.min(), x.max())
ax.set_ylim(h.min(), 1.1)

# Make readable labels
ax.set_xlabel(r'$x$', fontsize=24)
ax.set_ylabel(r'$h(x)$', fontsize=24)
ax.set_title('Activation Functions', fontsize=24)

# Set up grid
ax.grid(True, lw=1.75, ls='--', alpha=0.75)

# Put legend on figure
ax.legend(loc='best', fontsize=24);

fig.savefig('../images/nice_plots.png')
```

Exercise

These figures look nice in the plot and it makes sense for comparison. Now let's put the 3 different figures in separate plots.

* Make a separate plot for each figure and line them up on the same row.

```python
# your code here

```

```python
# %load solutions/three_subplots.py

```

Exercise

* Make a grid of 2 x 3 separate plots, 3 will be empty. Just plot the functions and do not worry about cosmetics. We just want you ro see the functionality.

```python
# your code here

```

```python
%load solutions/six_subplots.py

```

## 3 - Simple Linear Regression

Linear regression and its many extensions are a workhorse of the statistics and data science community, both in application and as a reference point for other models. Most of the major concepts in machine learning can be and often are discussed in terms of various linear regression models. Thus, this section will introduce you to building and fitting linear regression models and some of the process behind it, so that you can 1) fit models to data you encounter 2) experiment with different kinds of linear regression and observe their effects 3) see some of the technology that makes regression models work.

### Linear regression with a toy dataset
We first examine a toy problem, focusing our efforts on fitting a linear model to a small dataset with three observations.  Each observation consists of one predictor $x_i$ and one response $y_i$ for $i = 1, 2, 3$,

\begin{align*}
(x , y) = \{(x_1, y_1), (x_2, y_2), (x_3, y_3)\}.
\end{align*}

To be very concrete, let's set the values of the predictors and responses.

\begin{equation*}
(x , y) = \{(1, 2), (2, 2), (3, 4)\}
\end{equation*}

There is no line of the form $\beta_0 + \beta_1 x = y$ that passes through all three observations, since the data are not collinear. Thus our aim is to find the line that best fits these observations in the *least-squares sense*, as discussed in lecture.

Exercise (for home)

* Make two numpy arrays out of this data, x_train and y_train
* Check the dimentions of these arrays
* Try to reshape them into a different shape
* Make points into a very simple scatterplot
* Make a better scatterplot

```python
# your code here
```

```python
# solution
x_train = np.array([1,2,3])
y_train = np.array([2,3,6])
type(x_train)
```

```python
x_train.shape
```

```python
x_train = x_train.reshape(3,1)
x_train.shape
```

```python
# %load solutions/simple_scatterplot.py
# Make a simple scatterplot
plt.scatter(x_train,y_train)

# check dimensions
print(x_train.shape,y_train.shape)

```

```python
# %load solutions/nice_scatterplot.py
def nice_scatterplot(x, y, title):
    # font size
    f_size = 18

    # make the figure
    fig, ax = plt.subplots(1,1, figsize=(8,5)) # Create figure object

    # set axes limits to make the scale nice
    ax.set_xlim(np.min(x)-1, np.max(x) + 1)
    ax.set_ylim(np.min(y)-1, np.max(y) + 1)

    # adjust size of tickmarks in axes
    ax.tick_params(labelsize = f_size)

    # remove tick labels
    ax.tick_params(labelbottom=False,  bottom=False)

    # adjust size of axis label
    ax.set_xlabel(r'$x$', fontsize = f_size)
    ax.set_ylabel(r'$y$', fontsize = f_size)

    # set figure title label
    ax.set_title(title, fontsize = f_size)

    # you may set up grid with this
    ax.grid(True, lw=1.75, ls='--', alpha=0.15)

    # make actual plot (Notice the label argument!)
    #ax.scatter(x, y, label=r'$my points$')
    #ax.scatter(x, y, label='$my points$')
    ax.scatter(x, y, label=r'$my\,points$')
    ax.legend(loc='best', fontsize = f_size);

    return ax

nice_scatterplot(x_train, y_train, 'hello nice plot')

```

#### Formulae
Linear regression is special among the models we study because it can be solved explicitly. While most other models (and even some advanced versions of linear regression) must be solved itteratively, linear regression has a formula where you can simply plug in the data.

For the single predictor case it is:
    \begin{align}
      \beta_1 &= \frac{\sum_{i=1}^n{(x_i-\bar{x})(y_i-\bar{y})}}{\sum_{i=1}^n{(x_i-\bar{x})^2}}\\
      \beta_0 &= \bar{y} - \beta_1\bar{x}\
    \end{align}

Where $\bar{y}$ and $\bar{x}$ are the mean of the y values and the mean of the x values, respectively.

###  Building a model from scratch
In this part, we will solve the equations for simple linear regression and find the best fit solution to our toy problem.

```python

```

The snippets of code below implement the linear regression equations on the observed predictors and responses, which we'll call the training data set.  Let's walk through the code.

We have to reshape our arrrays to 2D. We will see later why.

Exercise

* make an array with shape (2,3)
* reshape it to a size that you want

```python
# your code here

```

```python
#solution
xx = np.array([[1,2,3],[4,6,8]])
xxx = xx.reshape(-1,2)
xxx.shape
```

```python
# Reshape to be a proper 2D array
x_train = x_train.reshape(x_train.shape[0], 1)
y_train = y_train.reshape(y_train.shape[0], 1)

print(x_train.shape)
```

```python
# first, compute means
y_bar = np.mean(y_train)
x_bar = np.mean(x_train)

# build the two terms
numerator = np.sum( (x_train - x_bar)*(y_train - y_bar) )
denominator = np.sum((x_train - x_bar)**2)

print(numerator.shape, denominator.shape) #check shapes
```

* Why the empty brackets? (The numerator and denominator are scalars, as expected.)

```python
#slope beta1
beta_1 = numerator/denominator

#intercept beta0
beta_0 = y_bar - beta_1*x_bar

print("The best-fit line is {0:3.2f} + {1:3.2f} * x".format(beta_0, beta_1))
print(f'The best fit is {beta_0}')
```

Exercise

Turn the code from the above cells into a function called `simple_linear_regression_fit`, that inputs the training data and returns `beta0` and `beta1`.

To do this, copy and paste the code from the above cells below and adjust the code as needed, so that the training data becomes the input and the betas become the output.

```python
def simple_linear_regression_fit(x_train: np.ndarray, y_train: np.ndarray) -> np.ndarray:

    return
```

Check your function by calling it with the training data from above and printing out the beta values.

```python
# Your code here
```

```python
# %load solutions/simple_linear_regression_fit.py
def simple_linear_regression_fit(x_train: np.ndarray, y_train: np.ndarray) -> np.ndarray:
    """
    Inputs:
    x_train: a (num observations by 1) array holding the values of the predictor variable
    y_train: a (num observations by 1) array holding the values of the response variable

    Returns:
    beta_vals:  a (num_features by 1) array holding the intercept and slope coeficients
    """

    # Check input array sizes
    if len(x_train.shape) < 2:
        print("Reshaping features array.")
        x_train = x_train.reshape(x_train.shape[0], 1)

    if len(y_train.shape) < 2:
        print("Reshaping observations array.")
        y_train = y_train.reshape(y_train.shape[0], 1)

    # first, compute means
    y_bar = np.mean(y_train)
    x_bar = np.mean(x_train)

    # build the two terms
    numerator = np.sum( (x_train - x_bar)*(y_train - y_bar) )
    denominator = np.sum((x_train - x_bar)**2)

    #slope beta1
    beta_1 = numerator/denominator

    #intercept beta0
    beta_0 = y_bar - beta_1*x_bar

    return np.array([beta_0,beta_1])

```

* Let's run this function and see the coefficients

```python
x_train = np.array([1 ,2, 3])
y_train = np.array([2, 2, 4])

betas = simple_linear_regression_fit(x_train, y_train)

beta_0 = betas[0]
beta_1 = betas[1]

print("The best-fit line is {0:8.6f} + {1:8.6f} * x".format(beta_0, beta_1))
```

Exercise

* Do the values of `beta0` and `beta1` seem reasonable?
* Plot the training data using a scatter plot.
* Plot the best fit line with `beta0` and `beta1` together with the training data.

```python
# Your code here
```

```python
# %load solutions/best_fit_scatterplot.py
fig_scat, ax_scat = plt.subplots(1,1, figsize=(10,6))

# Plot best-fit line
x_train = np.array([[1, 2, 3]]).T

best_fit = beta_0 + beta_1 * x_train

ax_scat.scatter(x_train, y_train, s=300, label='Training Data')
ax_scat.plot(x_train, best_fit, ls='--', label='Best Fit Line')

ax_scat.set_xlabel(r'$x_{train}$')
ax_scat.set_ylabel(r'$y$');

```

The values of `beta0` and `beta1` seem roughly reasonable.  They capture the positive correlation.  The line does appear to be trying to get as close as possible to all the points.

## 4 - Building a model with `statsmodels` and `sklearn`

Now that we can concretely fit the training data from scratch, let's learn two `python` packages to do it all for us:
* [statsmodels](http://www.statsmodels.org/stable/regression.html) and
* [scikit-learn (sklearn)](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html).

Our goal  is to show how to implement simple linear regression with these packages.  For an important sanity check, we compare the $\beta$ values from `statsmodels` and `sklearn` to the $\beta$ values that we found from above with our own implementation.

For the purposes of this lab, `statsmodels` and `sklearn` do the same thing.  More generally though, `statsmodels` tends to be easier for inference \[finding the values of the slope and intercept and dicussing uncertainty in those values\], whereas `sklearn` has machine-learning algorithms and is better for prediction \[guessing y values for a given x value\]. (Note that both packages make the same guesses, it's just a question of which activity they provide more support for.

**Note:** `statsmodels` and `sklearn` are different packages!  Unless we specify otherwise, you can use either one.

### Why do we need to add a constant in our simple linear regression model?

Let's say we a data set of two obsevations with one predictor and one response variable each. We would then have the following two equations if we run a simple linear regression model. $$y_1=\beta_0 + \beta_1*x_1$$ $$y_2=\beta_0 + \beta_1*x_2$$
 For simplicity and calculation efficiency we want to "absorb" the constant $b_0$ into an array with $b_1$ so we have only multiplication. To do this we introduce the constant ${x}^0=1$
$$y_1=\beta_0*{x_1}^0 + \beta_1*x_1$$ $$y_2=\beta_0 * {x_2}^0 + \beta_1*x_2$$
 That becomes:
$$y_1=\beta_0*1 + \beta_1*x_1$$ $$y_2=\beta_0 * 1 + \beta_1*x_2$$

In matrix notation:

$$
\left [
\begin{array}{c}
y_1 \\ y_2 \\
\end{array}
\right] =
\left [
\begin{array}{cc}
1& x_1 \\ 1 & x_2 \\
\end{array}
\right]
\cdot
\left [
\begin{array}{c}
\beta_0 \\ \beta_1 \\
\end{array}
\right]
$$

`sklearn` adds the constant for us where in `statsmodels` we need to explicitly add it using `sm.add_constant`

Below is the code for `statsmodels`.  `Statsmodels` does not by default include the column of ones in the $X$ matrix, so we include it manually with `sm.add_constant`.

```python
import statsmodels.api as sm
```

```python
# create the X matrix by appending a column of ones to x_train
X = sm.add_constant(x_train)

# this is the same matrix as in our scratch problem!
print(X)

# build the OLS model (ordinary least squares) from the training data
toyregr_sm = sm.OLS(y_train, X)

# do the fit and save regression info (parameters, etc) in results_sm
results_sm = toyregr_sm.fit()

# pull the beta parameters out from results_sm
beta0_sm = results_sm.params[0]
beta1_sm = results_sm.params[1]

print(f'The regression coef from statsmodels are: beta_0 = {beta0_sm:8.6f} and beta_1 = {beta1_sm:8.6f}')
```

Besides the beta parameters, `results_sm` contains a ton of other potentially useful information.

```python
import warnings
warnings.filterwarnings('ignore')
print(results_sm.summary())
```

Now let's turn our attention to the `sklearn` library.

```python
from sklearn import linear_model
```

```python
# build the least squares model
toyregr = linear_model.LinearRegression()

# save regression info (parameters, etc) in results_skl
results = toyregr.fit(x_train, y_train)

# pull the beta parameters out from results_skl
beta0_skl = toyregr.intercept_
beta1_skl = toyregr.coef_[0]

print("The regression coefficients from the sklearn package are: beta_0 = {0:8.6f} and beta_1 = {1:8.6f}".format(beta0_skl, beta1_skl))
```

We should feel pretty good about ourselves now, and we're ready to move on to a real problem!

### The `scikit-learn` library and the shape of things

Before diving into a "real" problem, let's discuss more of the details of `sklearn`.

`Scikit-learn` is the main `Python` machine learning library. It consists of many learners which can learn models from data, as well as a lot of utility functions such as `train_test_split()`.

Use the following to add the library into your code:

```python
import sklearn
```

In `scikit-learn`, an **estimator** is a Python object that implements the methods `fit(X, y)` and `predict(T)`

Let's see the structure of `scikit-learn` needed to make these fits. `fit()` always takes two arguments:
```python
estimator.fit(Xtrain, ytrain)
```
We will consider two estimators in this lab: `LinearRegression` and `KNeighborsRegressor`.

It is very important to understand that `Xtrain` must be in the form of a **2x2 array** with each row corresponding to one sample, and each column corresponding to the feature values for that sample.

`ytrain` on the other hand is a simple array of responses.  These are continuous for regression problems.

### Practice with `sklearn` and a real dataset
We begin by loading up the `mtcars` dataset. This data was extracted from the 1974 Motor Trend US magazine, and comprises of fuel consumption and 10 aspects of automobile design and performance for 32 automobiles (1973–74 models). We will load this data to a dataframe with 32 observations on 11 (numeric) variables. Here is an explanation of the features:

- `mpg` is Miles/(US) gallon
- `cyl` is Number of cylinders,
- `disp` is	Displacement (cu.in.),
- `hp` is	Gross horsepower,
- `drat` is	Rear axle ratio,
- `wt` is the Weight (1000 lbs),
- `qsec` is 1/4 mile time,
- `vs` is Engine (0 = V-shaped, 1 = straight),
- `am` is Transmission (0 = automatic, 1 = manual),
- `gear` is the Number of forward gears,
- `carb` is	Number of carburetors.

```python
import pandas as pd

#load mtcars
dfcars = pd.read_csv("../data/mtcars.csv")
dfcars.head()
```

```python
# Fix the column title
dfcars = dfcars.rename(columns={"Unnamed: 0":"car name"})
dfcars.head()
```

```python
dfcars.shape
```

#### Searching for values: how many cars have 4 gears?

```python
len(dfcars[dfcars.gear == 4].drop_duplicates(subset='car name', keep='first'))
```

Next, let's split the dataset into a training set and test set.

```python
# split into training set and testing set
from sklearn.model_selection import train_test_split

#set random_state to get the same split every time
traindf, testdf = train_test_split(dfcars, test_size=0.2, random_state=42)
```

```python
# testing set is around 20% of the total data; training set is around 80%
print("Shape of full dataset is: {0}".format(dfcars.shape))
print("Shape of training dataset is: {0}".format(traindf.shape))
print("Shape of test dataset is: {0}".format(testdf.shape))
```

Now we have training and test data.  We still need to select a predictor and a response from this dataset.  Keep in mind that we need to choose the predictor and response from both the training and test set.  You will do this in the exercises below.  However, we provide some starter code for you to get things going.

```python
traindf.head()
```

```python
# Extract the response variable that we're interested in
y_train = traindf.mpg
y_train
```

Exercise

Use slicing to get the same vector `y_train`

----

Now, notice the shape of `y_train`.

```python
y_train.shape, type(y_train)
```

### Array reshape
This is a 1D array as should be the case with the **Y** array.  Remember, `sklearn` requires a 2D array only for the predictor array.  You will have to pay close attention to this in the exercises later. `Sklearn` doesn't care too much about the shape of `y_train`.

The whole reason we went through that whole process was to show you how to reshape your data into the correct format.

**IMPORTANT:** Remember that your response variable `ytrain` can be a vector but your predictor variable `xtrain` ***must*** be an array!

## 5 - Example: Simple linear regression with automobile data
We will now use `sklearn` to predict automobile mileage per gallon (mpg) and evaluate these predictions. We already loaded the data and split them into a training set and a test set.

We need to choose the variables that we think will be good predictors for the dependent variable `mpg`.

Exercise in pairs

* Pick one variable to use as a predictor for simple linear regression.  Discuss your reasons with the person next to you.
* Justify your choice with some visualizations.
* Is there a second variable you'd like to use? For example, we're not doing multiple linear regression here, but if we were, is there another variable you'd like to include if we were using two predictors?

```python
x_wt = dfcars.wt
x_wt.shape
```

```python
# Your code here

```

```python
# %load solutions/cars_simple_EDA.py
```

Exercise

* Use `sklearn` to fit the training data using simple linear regression.
* Use the model to make mpg predictions on the test set.
* Plot the data and the prediction.
* Print out the mean squared error for the training set and the test set and compare.

```python
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

dfcars = pd.read_csv("../data/mtcars.csv")
dfcars = dfcars.rename(columns={"Unnamed: 0":"name"})

dfcars.head()
```

```python
traindf, testdf = train_test_split(dfcars, test_size=0.2, random_state=42)

y_train = np.array(traindf.mpg)
X_train = np.array(traindf.wt)
X_train = X_train.reshape(X_train.shape[0], 1)
```

```python
y_test = np.array(testdf.mpg)
X_test = np.array(testdf.wt)
X_test = X_test.reshape(X_test.shape[0], 1)
```

```python
# Let's take another look at our data
dfcars.head()
```

```python
# And out train and test sets
y_train.shape, X_train.shape
```

```python
y_test.shape, X_test.shape
```

```python
#create linear model
regression = LinearRegression()

#fit linear model
regression.fit(X_train, y_train)

predicted_y = regression.predict(X_test)

r2 = regression.score(X_test, y_test)
print(f'R^2 = {r2:.5}')
```

```python
print(regression.score(X_train, y_train))

print(mean_squared_error(predicted_y, y_test))
print(mean_squared_error(y_train, regression.predict(X_train)))

print('Coefficients: \n', regression.coef_[0], regression.intercept_)
```

```python
fig, ax = plt.subplots(1,1, figsize=(10,6))
ax.plot(y_test, predicted_y, 'o')
grid = np.linspace(np.min(dfcars.mpg), np.max(dfcars.mpg), 100)
ax.plot(grid, grid, color="black") # 45 degree line
ax.set_xlabel("actual y")
ax.set_ylabel("predicted y")

fig1, ax1 = plt.subplots(1,1, figsize=(10,6))
ax1.plot(dfcars.wt, dfcars.mpg, 'o')
xgrid = np.linspace(np.min(dfcars.wt), np.max(dfcars.wt), 100)
ax1.plot(xgrid, regression.predict(xgrid.reshape(100, 1)))
```

## 6 - $k$-nearest neighbors

Now that you're familiar with `sklearn`, you're ready to do a KNN regression.

Sklearn's regressor is called `sklearn.neighbors.KNeighborsRegressor`. Its main parameter is the `number of nearest neighbors`. There are other parameters such as the distance metric (default for 2 order is the Euclidean distance). For a list of all the parameters see the [Sklearn kNN Regressor Documentation](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsRegressor.html).

Let's use $5$ nearest neighbors.

```python
# Import the library
from sklearn.neighbors import KNeighborsRegressor
```

```python
# Set number of neighbors
k = 5
knnreg = KNeighborsRegressor(n_neighbors=k)
```

```python
# Fit the regressor - make sure your numpy arrays are the right shape
knnreg.fit(X_train, y_train)

# Evaluate the outcome on the train set using R^2
r2_train = knnreg.score(X_train, y_train)

# Print results
print(f'kNN model with {k} neighbors gives R^2 on the train set: {r2_train:.5}')
```

```python
knnreg.predict(X_test)
```

Exercise

Calculate and print the $R^{2}$ score on the test set

```python
# Your code here
```

Not so good? Lets vary the number of neighbors and see what we get.

```python
# Make our lives easy by storing the different regressors in a dictionary
regdict = {}

# Make our lives easier by entering the k values from a list
k_list = [1, 2, 4, 15]

# Do a bunch of KNN regressions
for k in k_list:
    knnreg = KNeighborsRegressor(n_neighbors=k)
    knnreg.fit(X_train, y_train)
    # Store the regressors in a dictionary
    regdict[k] = knnreg

# Print the dictionary to see what we have
regdict
```

Now let's plot all the k values in same plot.

```python
fig, ax = plt.subplots(1,1, figsize=(10,6))

ax.plot(dfcars.wt, dfcars.mpg, 'o', label="data")

xgrid = np.linspace(np.min(dfcars.wt), np.max(dfcars.wt), 100)

# let's unpack the dictionary to its elements (items) which is the k and Regressor
for k, regressor in regdict.items():
    predictions = regressor.predict(xgrid.reshape(-1,1))
    ax.plot(xgrid, predictions, label="{}-NN".format(k))

ax.legend();
```

Exercise

Explain what you see in the graph. **Hint** Notice how the $1$-NN goes through every point on the training set but utterly fails elsewhere.

Lets look at the scores on the training set.

```python
ks = range(1, 15) # Grid of k's
scores_train = [] # R2 scores
for k in ks:
    # Create KNN model
    knnreg = KNeighborsRegressor(n_neighbors=k)

    # Fit the model to training data
    knnreg.fit(X_train, y_train)

    # Calculate R^2 score
    score_train = knnreg.score(X_train, y_train)
    scores_train.append(score_train)

# Plot
fig, ax = plt.subplots(1,1, figsize=(12,8))
ax.plot(ks, scores_train,'o-')
ax.set_xlabel(r'$k$')
ax.set_ylabel(r'$R^{2}$')
```

Exercise

* Why do we get a perfect $R^2$ at k=1 for the training set?
* Make the same plot as above on the *test* set.
* What is the best $k$?

```python
# Your code here

```

```python
# %load solutions/knn_regression.py
```

```python
# solution to previous exercise
r2_test = knnreg.score(X_test, y_test)
print(f'kNN model with {k} neighbors gives R^2 on the test set: {r2_test:.5}')
```


# Technical information on how this blog is made
> Using Fastpages and Github Pages to host interactive Jupyter Notebooks in blog format

- toc: true
- badges: true
- comments: true
- categories: [jupyter]
- hide: true
- permalink: /technical
- image: images/chart-preview.png

# Introduction

This blog is written using [fastpages](https://github.com/fastai/fastpages){% cite fastpages_introduction%}, an easy-to-use (and to setup) way to convert Jupyter Notebooks to blogs, which are automatically hosted on a Github Pages website. This post serves as an introduction for potential users, as a way for me to test some things and as a reference for some more obscure parts of the Fastpages process. Let's begin!

# Why Fastpages?

A big part of my thesis will be working in python, specifically in Jupyter Notebooks. These are great ways to blend python scripts with markdown and some $\LaTeX$ features, meaning that in one cell I can mathmatically derive some expression and in the next I can script it in Python. This back and forth got me thinking about using these notebooks as the foundation for my thesis, which led me to fastpages. This means that if I write proper jupyter notebooks, I will be able to:

*   Work on the problem at hand
*   Share the problem at hand in a more informal way
*   Copy-paste a section straight into my final report

all from the same source file. Sounds great!

However, there are some limitations to *vanilla* fastpages

# Pedantic Limitations and how to solve them

## Equations

Using jupyter notebooks you can enter inline math, for example for variables like $x$ and $\eta_{NEP}$ by using the syntax `$some expression$`, just like you would do in $\LaTeX$ markdown files. However a centered function
 ```
$$
Complicated math
$$
```
 will not be numbered and is not referenceable. Thankfully Anthoine C.{% cite math_with_fastpages %} gave the solution, which is to override the default mathrenderer with a couple of lines of code and that seemed to do the trick.

 $$\begin{equation}
i\hbar\frac{\partial\mathbf{\Psi}(x,t)}{\partial t}=\left(-\frac{\hbar}{2m}\frac{\partial^2}{\partial x^2}+V(x,t)\right)\mathbf{\Psi}(x,t)
 \end{equation}$$
 and as we can see in \eqref{Schrodinger}, we have a beatifully typeset, referenced Schrodinger Equation. However, as we can see from the Identity matrix in $\mathbb{R}^4$ \eqref{I_R4}, the equation reference varies in size with the size of the equation, so maybe don't throw away your favorite $\LaTeX$ renderer yet.
$$\begin{equation}
I = \begin{pmatrix}1&0&0&0 \\ 0&1&0&0 \\ 0&0&1&0 \\ 0&0&0&1\end{pmatrix}
\end{equation}$$

## BibTex

Fastpages has a built-in way to handle in-text references, but in order to minimize the manual work in copy-pasting sections of notebook to blog to thesis, I have decided to swap it out for a BibTex compliant renderer, as described by the fastpages team {% cite bibtex_with_fastpages%}. As you can see from this post, this works well. The big advantage is that the entire website uses a single `references.bib` file, so I can cite the same reference across multiple posts and end up with a single reference file when I want to consolidate multiple posts in a $\LaTeX$ thesis or a single post.

One caveat in in-text refencing is the usual `\\cite{<source>}` doesn't work and `{{'{'}}% cite <source> %}` is needed instead. This can be changed in bulk using the find and replace feature of any editor however.

# Bibliography
{% bibliography --cited %}


Osnabrück University - Machine Learning (Summer Term 2016) - Prof. Dr.-Ing. G. Heidemann, Ulf Krumnack

# Exercise Sheet 03: Basics of Data Mining

## Introduction

This week's sheet should be solved and handed in before the end of **Sunday, April 29, 2018**. If you need help (and Google and other resources were not enough), feel free to contact your groups designated tutor or whomever of us you run into first. Please upload your results to your group's studip folder.

There are a lot of implementations with fewer theory questions on this sheet, but don't worry: To be able to implement most of the code, you have to understand the theory.

This week's assignments make use of two packages: `numpy` and `matplotlib`. We already expected you to install those as part of sheet 1. If you did not do so, go back to those instructions or just run the following command in the `terminal`/`cmd.exe` to do so. This will also upgrade your current installation.

    conda install jupyter numpy matplotlib

One note about `matplotlib`: If you run code which contains a plot like the cell below, it can sometimes take a while to execute the code and show the results. During that process the invocation count will be shown as a little Asterisk (\*) like this:

    In [*]:

Just be patient for a few seconds. The following cell tests if `numpy` and `matplotlib` are installed and work:

```python
%matplotlib notebook
import importlib

assert importlib.util.find_spec('numpy') is not None , 'numpy not found'
assert importlib.util.find_spec('matplotlib') is not None, 'matplotlib not found'

import numpy as np
import matplotlib.pyplot as plt

figure_intro = plt.figure('Example plot')
plt.plot(np.random.randn(1000, 1))
figure_intro.canvas.draw()
```

    <IPython.core.display.Javascript object>

## Assignment 0: Math recap (vector spaces) [2 Bonus Points]

This exercise is supposed to be very easy and is voluntary. There will be a similar exercise on every sheet.
It is intended to revise some basic mathematical notions that are assumed throughout this class and to allow you to check if you are comfortable with them.
Usually you should have no problem to answer these questions offhand, but if you feel unsure, this is a good time to look them up again. You are always welcome to discuss questions with the tutors or in the practice session.
Also, if you have a (math) topic you would like to recap, please let us know.

**a)** What is a *vector space*? What is the *basis* of a vector space and what is its *dimensionality*? Can you provide examples for finite- and infinite-dimensional vector spaces?

A vector space or linear space is a collection of vectors for which **vector addition** and **scalar multiplication**, that satisfy a set of axioms, hold. A basis is a infinite or finite set of vectors that are *linearly independent* and can be combined to a sum also called *linear combination* to express any vector of that space.

The dimension of a vector space is the cardinality of its basis. An example for a finite-dimensional vector space is the *Euclidean space* $\mathbb{R}^3$, which has the basis

$$\begin{Bmatrix}
\begin{pmatrix}
1\\ 0\\ 0
\end{pmatrix},
\begin{pmatrix}
0\\ 1\\ 0
\end{pmatrix},
\begin{pmatrix}
0\\ 0\\ 1
\end{pmatrix}
\end{Bmatrix}$$

and thereby a dimension of 3. An infinte-dimensional vector space can be found in the *Hilbert space*.

**b)** What is a *linear map*? What is the *image* and the *kernel* of such a map?

A linear mapping is a function that maps between two modules that adheres to certain rules regrading addition and scalar multiplication listed below.

\begin{align}
f(\vec{u}+\vec{v}) &= f(\vec{u}) + f(\vec{v})\\
f(c\vec{u}) &= cf(\vec{u})
\end{align}

The kernel and image of such alinear mapping can be defined as follows:

\begin{align}
ker(f) &= \{ x \in V: f(x) = 0 \}\\
im(f) &= \{ w \in W: f(x) = w, x \in V \}\\
\end{align}

Meaning that the kernel contains all vectors for which the operation f maps to 0 and the image holds all vectors in $W$ which can be mapped to from $V$ using f.

**c)** What is a *matrix*? What is the relation to linear maps?

A matrix is a $n\times m$ collection of values that can be used to perform linear mappings. A popular application of matrices that are linear mappings is the rotation or scaling of 2 or 3-dimensional objects in space (computer graphics).

## Assignment 1: Rosner test [5 Points]

The Rosner test is an iterative procedure to remove outliers of a data set via a z-test. In this exercise you will implement it and apply it to a sample data set.

### a) Outliers

First of all, think about why we use procedures like this and answer the following questions:

What are causes for outliers? And what are our options to deal with them?

A common cause for outliers is the imperfection of measuring technology, leading to incorrect data values. Other causes include the possibility that the data actually exhibits high variation or that the underlying model accounts for the extreme data values, while the data set *at hand* does not.

There are several options available to deal with outliers:
* Remove them
* Weight them according to their z-values (distance to $\mu$ in terms of $\sigma$)
* Remove them and replace the missing data according to a specified procedure

### b) Rosner test

In the following you find a stub for the implementation. The dataset is already generated. Now it is your turn to write the Rosner test and detect the outliers in the data.

`data` is a `np.array` of `[x, y]` coordinates. `outliers` is a list of `[x, y]` coordinates.

```python
%matplotlib notebook
import numpy as np
import matplotlib.pyplot as plt
from math import fabs

# generate dataset
data = list(zip(np.random.uniform(size=100), np.random.normal(size=100)))
data += list(zip(np.random.uniform(size=10), np.random.normal(0, 10, size=10)))
data = np.array(data)
outliers = []

# just to check if everything is pretty
fig_rosner_data = plt.figure('The Dataset')
plt.scatter(data[:,0], data[:,1], marker='x')
plt.axis([0, 1, -20, 20])
fig_rosner_data.canvas.draw()

# Now find the outliers, add them to 'outliers', and remove them from 'data'.
max_iter = 100
for i in range(max_iter):
    median = np.median(data, axis=0)
    dev = np.std(data, axis=0)
    # Find max z item
    z = np.array([[fabs(coord[0]-median[0])/dev[0], fabs(coord[1]-median[1])/dev[1]] for coord in data])
    maxz = np.unravel_index(np.argmax(z, axis=None), z.shape)
    if z[maxz] > 3.5:
        outliers.append(data[maxz[0]])
        data = np.delete(data, maxz[0], axis=0)

# plot results
outliers = np.array(outliers)
fig_rosner = plt.figure('Rosner Result')
plt.scatter(data[:,0], data[:,1], c='b', marker='x', label='cleared data')
plt.scatter(outliers[:,0], outliers[:,1], c='r', marker='x', label='outliers')
plt.axis([0, 1, -20, 20])
plt.legend(loc='lower right');
fig_rosner.canvas.draw()
```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

## Assignment 2: p-norm [5 Points]

A very well known norm is the euclidean distance. However, it is not the only norm: It is in fact just one of many p-norms where $p = 2$. In this assignment you will take a look at other p-norms and see how they behave.

Implement a function `pnorm` which expects a vector $x \in \mathcal{R}^n$ and a scalar $p \geq 1, p \in \mathcal{R}$ and returns the p-norm of $x$ which is defined as:

$$||x||_p = \left(\sum\limits_{i=1}^n |x_i|^p \right)^{\frac{1}{p}}$$

*Note:* Even though the norm is only defined for $p \geq 1$, values $0 < p < 1$ are still interesting. In that case we can not talk about a norm anymore, as the triangle inequality ($||a|| + ||b|| \geq ||a + b||$) does not hold. We will still take a look at some of these values, so your function should handle them as well.

```python
import numpy as np
from math import fsum, fabs

def pnorm(x, p):
    """
    Calculates the p-norm of x.

    Args:
        x (array): the vector for which the norm is to be computed.
        p (float): the p-value (a positive real number).

    Returns:
        The p-norm of x.
    """
    if p < 0:
        raise ValueError("P-Norm: p must be a positive number")

    if not isinstance(x, list):
        x = [x];

    return pow(fsum([pow(fabs(a),p) for a in x]), 1.0/p)
```

```python
# 1e-10 is 0.0000000001
assert pnorm(1, 2)      - 1          < 1e-10 , "pnorm is incorrect for x = 1, p = 2"
assert pnorm(2, 2)      - 2          < 1e-10 , "pnorm is incorrect for x = 2, p = 2"
assert pnorm([2, 1], 2) - np.sqrt(5) < 1e-10 , "pnorm is incorrect for x = [2, 1], p = 2"
assert pnorm(2, 0.5)    - 2          < 1e-10 , "pnorm is incorrect for x = 2, p = 0.5"
```

Implement another function `pdist` which expects two vectors $x_0 \in \mathcal{R}^n, x_1 \in \mathcal{R}^n$ and a scalar $p \geq 1, p \in \mathcal{R}$ and returns the distance between $x_0$ and $x_1$ on the p-norm defined by $p$. Again handle $0 < p < 1$ as well.

```python
import numpy as np
from math import fabs

def pdist(x0, x1, p):
    """
    Calculates the distance between x0 and x1
    using the p-norm.

    Arguments:
        x0 (array): the first vector.
        x1 (array): the second vector.
        p (float): the p-value (a positive real number).

    Returns:
        The p-distance between x0 and x1.
    """
    if p < 0:
        raise ValueError("P-Dist: p must be a positive number")

    if not isinstance(x0, list):
        x0 = [x0];

    if not isinstance(x1, list):
        x1 = [x1];

    return pnorm([a-b for a,b in zip(x0,x1)], p)
```

```python
# 1e-10 is 0.0000000001
assert pdist(1, 2, 2)           - 1          < 1e-10 , "pdist is incorrect for x0 = 1, x1 = 2, p = 2"
assert pdist(2, 5, 2)           - 3          < 1e-10 , "pdist is incorrect for x0 = 2, x1 = 5, p = 2"
assert pdist([2, 1], [1, 2], 2) - np.sqrt(2) < 1e-10 , "pdist is incorrect for x0 = [2, 1], x1 = [1, 2], p = 2"
assert pdist([2, 1], [0, 0], 2) - np.sqrt(5) < 1e-10 , "pdist is incorrect for x0 = [2, 1], x1 = [0, 0], p = 2"
assert pdist(2, 0, 0.5)         - 2          < 1e-10 , "pdist is incorrect for x0 = 2, x1 = 0, p = 0.5"
```

Now we will compare some different p-norms. Below is part of a code to plot data in nice scatter plots.

Your task is to calculate the data to plot. The variable `data` is currently simply filled with zeros. Instead, fill it as follows:

- Use the function `np.linspace()` to create a vector of `50` evenly distributed values between `-100` and `100` (inclusively).
- Fill `data`: Data is basically the cartesian product of the vector you created before with itself filled up with each value's norm. It should have 2500 rows. Each of the 2500 rows should contain `[x, y, d]`, where `x` is the x coordinate and `y` the y coordinate of a point, and `d` the p-norm of `(x, y)`. Use either `pnorm` or `pdist` to calculate `d`.
- Normalize the data in `data[:,2]` (i.e. all d-values) so that they are between 0 and 1.

Run your code and take a look at your results. Darker colors mean that a value is closer to the center (0, 0) according to the p-norm used.

*Hint:* To give you an idea of how `data` should look like, here is an example for three evenly distributed values between `-1` and `1` and a p-norm with `p = 2`.

Before normalization of the d-column:

```python
data = np.array([[-1.         -1.          1.41421356]
                 [-1.          0.          1.        ]
                 [-1.          1.          1.41421356]
                 [ 0.         -1.          1.        ]
                 [ 0.          0.          0.        ]
                 [ 0.          1.          1.        ]
                 [ 1.         -1.          1.41421356]
                 [ 1.          0.          1.        ]
                 [ 1.          1.          1.41421356]])
```

After normalization of the d-column:

```python
data = np.array([[-1.         -1.          1.        ]
                 [-1.          0.          0.70710678]
                 [-1.          1.          1.        ]
                 [ 0.         -1.          0.70710678]
                 [ 0.          0.          0.        ]
                 [ 0.          1.          0.70710678]
                 [ 1.         -1.          1.        ]
                 [ 1.          0.          0.70710678]
                 [ 1.          1.          1.        ]])
```

```python
%matplotlib notebook
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.colors import ColorConverter
from itertools import product

color = ColorConverter()
figure_norms = plt.figure('p-norm comparison')

# create the linspace vector
ls = np.linspace(-100, 100)

assert len(ls) == 50 , 'ls should be of length 50.'
assert (min(ls), max(ls)) == (-100, 100) , 'ls should range from -100 to 100, inclusively.'

for i, p in enumerate([1/8, 1/4, 1/2, 1, 1.5, 2, 4, 8, 128]):
    # Create a numpy array containing useful values instead of zeros
    # tmp_data = np.array([x for x in product(ls, ls)])
    data = np.array([[x[0], x[1], pnorm([x[0], x[1]], p)] for x in product(ls, ls)])

    normalize = np.amax(data[:,2])
    data[:,2] /= normalize

    assert all(data[:,2] <= 1), 'The third column should be normalized.'

    # Plot the data.
    colors = [color.to_rgb((1, 1-a, 1-a)) for a in data[:,2]]
    a = plt.subplot(3, 3, i + 1)
    plt.scatter(data[:,0], data[:,1], marker='.', color=colors)
    a.set_ylim([-100, 100])
    a.set_xlim([-100, 100])
    a.set_title('{:.3g}-norm'.format(p))
    a.set_aspect('equal')
    plt.tight_layout()
    figure_norms.canvas.draw()
```

    <IPython.core.display.Javascript object>

## Assignment 3: Expectation Maximization [10 Points]

In this assignment you will implement the Expectation Maximization algorithm (EM) for 1D data sets.

As some parts of this exercise would require some more knowledge of Python than what was already discussed in the practice sessions we built a small number of templates for you to use. However, if you prefer to do so you are also allowed to just go ahead and implement everything yourself! **Don't forget [task b)](#b%29-EM-and-missing-values)**!

### a) Implement Expectation Maximization

Use the next cell to implement your own solution or, if you want some more guidance, skip the next cell and continue the exercise at  [Step 1) Load the data](#Step-1%29-Load-the-data).

Here is an overview of what you have to do:

**1) Load the data:**

Load the provided data set. It is stored in `em_normdistdata.txt`. We call the set $X$ and each individual data $x \in X$.

**2) Initialize EM:**

Initialize three normal distributions whose parameters will be changed iteratively by the EM to converge close to the original distributions.

Each normal distribution $j$ has three parameters: $\mu_j$ (the mean), $\sigma_j$ (the standard deviation), $\alpha_j$ (the proportion of the normal distribution in the mixture, that means $\sum\limits_j\alpha_j=1$).

Initialize the three parameters using three random partitions $S_j$ of the data set. Calculate each $\mu_j$ and $\sigma_j$ and set $\alpha_j = \frac{|S_j|}{|X|}$.

**3) Implement the expectation step:**

Perform a soft classification of the data samples with the three normal distributions. That means: Calculate the likelihood that a data sample $x_i$ belongs to distribution $j$ given parameters $\mu_j$ and $\sigma_j$. Or in other words, what is the likelihood of $x_i$ to be drawn from $N_j(\mu_j, \sigma_j)$? When you got the likelihood, weight the result by $\alpha_j$.

As a last step normalize the results such that the likelihoods of a data sample $x_i$ sum up to $1$.

**4) Implement the maximization step:**

In the maximization step each $\mu_j$, $\sigma_j$ and $\alpha_j$ is updated. First calculate the new means:

$$\mu_j = \frac{1}{\sum\limits_{i=1}^{|X|} p_{ij}} \sum\limits_{i=1}^{|X|} p_{ij}x_i$$

That means $\mu_j$ is the weighted mean of all samples, where the weight is their likelihood of belonging to distribution $j$.

Then calculate the new $\sigma_j$. Each new $\sigma_j$ is the standard deviation of the normal distribution with the new $\mu_j$, so for the calculation you already use the new $\mu_j$:

$$\sigma_j = \sqrt{ \frac{1}{\sum\limits_{i=1}^{|X|} p_{ij}} \sum\limits_{i=1}^{|X|} p_{ij} \left(x_i - \mu_j\right)^2 }$$

To calculate the new $\alpha_j$ for each distribution, just take the mean of $p_j$ for each normal distribution $j$.

**5) Perform the complete EM and plot your results:**

Build a loop around the iterative procedure of expectation and maximization which stops when the changes in all $\mu_j$ and $\sigma_j$ are sufficiently small enough.

Plot your results after each step and mark which data points belong to which normal distribution. If you don't get it to work, just plot your final solution of the distributions.

```python
# Free space to implement your own solution -- either use this OR use the following step by step guide.
# You may use scipy.stats.norm.pdf for your own implementation.

```

#### Step 1) Load the data

Load the provided data set. It is stored in `em_normdistdata.txt`. We call the set $X$ and each individual data $x \in X$.

*Hint:* Figure out a way on how numpy can load text data.

```python
import numpy as np

def load_data(file_name):
    """
    Loads the data stored in file_name into a numpy array.
    """
    return np.loadtxt(file_name)

assert load_data('em_normdistdata.txt').shape == (200,) , "The data was not properly loaded."
```

*Optional:* The data consists of 200 data points drawn from three normal distributions. To get a feeling for the data set you can plot the data with the following cell. Change the number of bins to get a rough idea of how the three distributions might look like.

```python
%matplotlib notebook
import numpy as np
import matplotlib.pyplot as plt

data = load_data('em_normdistdata.txt')

fig_data_test = plt.figure('Data overview')
plt.hist(data, bins=5)
fig_data_test.canvas.draw()
```

    <IPython.core.display.Javascript object>

#### Step 2) Initialize EM

Below is a class definition `NormPDF` which represents the probability density function (pdf) of the normal distribution with an additional parameter $\alpha$. The class is explained in the next cells.

```python
import numpy as np
class NormPDF():
    """
    A representation of the probability density function of the normal distribution
    for the EM Algorithm.
    """

    def __init__(self, mu=0, sigma=1, alpha=1):
        """
        Initializes the normal distribution with mu, sigma and alpha.
        The defaults are 0, 1, and 1 respectively.
        """
        self.mu = mu
        self.sigma = sigma
        self.alpha = alpha

    def __call__(self, x):
        """
        Returns the evaluation of this normal distribution at x.
        Does not take alpha into account!
        """
        return np.exp(-(x - self.mu) ** 2 / (2 * self.sigma ** 2)) / (np.sqrt(np.pi * 2) * self.sigma)

    def __repr__(self):
        """
        A simple string representation of this instance.
        """
        return 'NormPDF({self.mu:.2f},{self.sigma:.2f},{self.alpha:.2f})'.format(self=self)
```

The class `NormPDF` offers several class methods: `__init__`, `__call__`, `__repr__`. They are all special Python functions which are overloaded so they can be used in a nice way. Note that all methods take as the first parameter `self`: this is just the python way of passing the instance itself to the method so that it becomes possible to access its data. You can always ignore it for now and just assume that the methods only need the parameters which follow.

`__init__`: This is the constructor. When a new instance of the class is created this method is used. It takes the parameters `mu`, `sigma`, and `alpha`. Note that if you leave out parameters, they will be set to some default values.
So you can create `NormPDF` instances like this:

```python
a = NormPDF()             # No parameters: mu = 0, sigma = 1, alpha = 1
b = NormPDF(1)            # mu = 1, sigma = 1, alpha = 1
c = NormPDF(1, alpha=0.4) # skips sigma but sets alpha, thus: mu = 1, sigma = 1, alpha = 0.4
d = NormPDF(0, 0.5)       # mu = 0, sigma = 0.5, alpha = 1
e = NormPDF(0, 0.5, 0.9)  # mu = 0, sigma = 0.5, alpha = 0.9
```

`__call__`: This is a very cool feature of Python. By implementing this method one can make an instance *callable*. That basically means one can use it as if it was a function. The `NormPDF` instances can be called with an x value (or a numpy array of x values) to get the evaluation of the normal distribution at x.

```python
normpdf = NormPDF()
print(normpdf(0))
print(normpdf(0.5))
print(normpdf(np.linspace(-2, 2, 10)))
```

    0.3989422804014327
    0.3520653267642995
    [0.05399097 0.11897819 0.21519246 0.31944801 0.38921247 0.38921247
     0.31944801 0.21519246 0.11897819 0.05399097]

`__repr__`: This method will be used in Python when one calls `repr(NormPDF())`. As long as `__str__` is not implemented (which you saw in last week's sheet) `str(NormPDF())` will also use this method. This comes in handy for printing:

```python
normpdf1 = NormPDF()
normpdf2 = NormPDF(1, 0.5, 0.9)
print(normpdf1)
print([normpdf1, normpdf2])
```

    NormPDF(0.00,1.00,1.00)
    [NormPDF(0.00,1.00,1.00), NormPDF(1.00,0.50,0.90)]

It is also possible to change the values of an instance of the NormPDF:

```python
normpdf1 = NormPDF()
print(normpdf1)
print(normpdf1(np.linspace(-2, 2, 10)))

normpdf1.mu = 1
normpdf1.sigma = 2
normpdf1.alpha = 0.9
print(normpdf1)
print(normpdf1(np.linspace(-2, 2, 10)))
```

    NormPDF(0.00,1.00,1.00)
    [0.05399097 0.11897819 0.21519246 0.31944801 0.38921247 0.38921247
     0.31944801 0.21519246 0.11897819 0.05399097]
    NormPDF(1.00,2.00,0.90)
    [0.0647588  0.08817395 0.11427077 0.14095594 0.16549503 0.18494385
     0.19671986 0.19916355 0.19192205 0.17603266]

Now that you know how the `NormPDF` class works, it is time for the implementation of the initialization function. Here is the task again:

Write a function `gaussians = initialize_EM(data, num_distributions)` to initialize the EM.

Each normal distribution $j$ has three parameters: $\mu_j$ (the mean), $\sigma_j$ (the standard deviation), $\alpha_j$ (the proportion of the normal distribution in the mixture, that means $\sum\limits_j\alpha_j=1$).
Initialize the three parameters using three random partitions $S_j$ of the data set. Calculate each $\mu_j$ and $\sigma_j$ and set $\alpha_j = \frac{|S_j|}{|X|}$.

```python
def initialize_EM(data, num_distributions):
    """
    Initializes the EM algorithm by calculating num_distributions NormPDFs
    from a random partitioning of data. I.e., the data set is randomly
    divided into num_distribution parts, and each part is used to initialize
    mean, standard deviation and alpha parameter of a NormPDF object.

    Args:
        data (array): A collection of data.
        num_distributions (int): The number of distributions to return.

    Returns:
        A list of num_distribution NormPDF objects, initialized from a
        random partioning of the data.
    """
    gaussians = []
    size = len(data)

    # Shuffle and split data to get 'randomized' partitioning
    np.random.shuffle(data)
    subsets = np.array_split(data, num_distributions)
    # For each subset, generate a pdf using the contained data
    for aset in subsets:
        gaussians.append(NormPDF(np.mean(aset), np.std(aset), len(aset)/size))

    return gaussians

normpdfs_ = initialize_EM(np.linspace(-1, 1, 100), 2)
assert len(normpdfs_) == 2, "The number of initialized distributions is not correct."
# 1e-10 is 0.0000000001
assert abs(1 - sum([normpdf.alpha for normpdf in normpdfs_])) < 1e-10 , "Sum of all alphas is not 1.0!"
```

#### Step 3) Implement the expectation step

Perform a soft classification of the data samples with the normal distributions. That means: Calculate the likelihood that a data sample $x_i$ belongs to distribution $j$ given parameters $\mu_j$ and $\sigma_j$. Or in other words, what is the likelihood of $x_i$ to be drawn from $N_j(\mu_j, \sigma_j)$? When you got the likelihood, weight the result by $\alpha_j$.

As a last step normalize the results such that the likelihoods of a data sample $x_i$ sum up to $1$.

*Hint:* Store the data in a different array before you normalize it to not run into problems with partly normalized data.

```python
from math import fsum

def expectation_step(gaussians, data):
    """
    Performs the expectation step of the EM.

    Args:
        gaussians (list): A list of NormPDF objects.
        data (array): The data vector.

    Returns:
        An array of shape (len(data), len(gaussians))
        which contains normalized likelihoods for each sample
        to denote to which of the normal distributions it
        most likely belongs to.
    """
    tmp = np.array([[g(entry)*g.alpha for g in gaussians] for entry in data])
    expectation = np.array(tmp)

    for i in range(len(data)):
        whole = fsum(tmp[i,:])
        expectation[i,:] = [x/whole for x in tmp[i,:]]

    return expectation

assert expectation_step([NormPDF(), NormPDF()], np.linspace(-2, 2, 100)).shape == (100, 2) , "Shape is not correct!"
```

#### Step 4) Implement the maximization step

In the maximization step each $\mu_j$, $\sigma_j$ and $\alpha_j$ is updated. First calculate the new means:

$$\mu_j = \frac{1}{\sum\limits_{i=1}^{|X|} p_{ij}} \sum\limits_{i=1}^{|X|} p_{ij}x_i$$

That means $\mu_j$ is the weighted mean of all samples, where the weight is their likelihood of belonging to distribution $j$.

Then calculate the new $\sigma_j$. Each new $\sigma_j$ is the standard deviation of the normal distribution with the new $\mu_j$, so for the calculation you already use the new $\mu_j$:

$$\sigma_j = \sqrt{ \frac{1}{\sum\limits_{i=1}^{|X|} p_{ij}} \sum\limits_{i=1}^{|X|} p_{ij} \left(x_i - \mu_j\right)^2 }$$

To calculate the new $\alpha_j$ for each distribution, just take the mean of $p_j$ for each normal distribution $j$.

**Caution:** For the next step it is necessary to know how much all $\mu$ and $\sigma$ changed. For that the function `maximization_step` should return a numpy array of those (absolute) changes. For example if $\mu_0$ changed from 0.1 to 0.15, $\sigma_0$ from 1 to 0.9, $\mu_1$ from 0.5 to 0.6, and $\sigma_1$, $\mu_2$, and $\sigma_2$ stayed the same, we expect the function to return `np.array([0.05, 0.1, 0.1, 0, 0, 0])` (however, the order is not important).

```python
from math import sqrt, fabs, fsum

def weighted_mean(data, pij):
    """
    Calculates the new weighted mean

    Args:
        data (array): the data vector
        pij (array): expected values for the data

    Returns:
        The new weighted mean for the distribution
    """
    return fsum([x*y for x,y in zip(data,pij)])/fsum(pij)

def std_dev(mu, data, pij):
    """
    Calculates the new standard deviation

    Args:
        mu (float): mean
        data (array): the data vector
        pij (array): expected values for the data

    Returns:
        The new weighted mean for the distribution
    """
    return sqrt(fsum([y*pow(x-mu, 2) for x,y in zip(data,pij)])/fsum(pij))

def maximization_step(gaussians, data, expectation):
    """
    Performs the maximization step of the EM.
    Modifies the gaussians by updating their mus and sigmas.

    Args:
        gaussians (list): A list of NormPDF objects.
        data (array): The data vector.
        expectation (array): The expectation values for data element
            (as computed by expectation_step()).

    Returns:
        A numpy array of absolute changes in any mu or sigma,
        that means the returned array has twice as many elements as
        the supplied list of gaussians.
    """
    changes = []
    for i in range(len(gaussians)):
        new_mu = weighted_mean(data, expectation[:,i])
        new_sigma = std_dev(new_mu, data, expectation[:,i])
        gaussians[i].alpha = sum(expectation[:,i])/len(data)

        changes.append(fabs(gaussians[i].mu - new_mu))
        changes.append(fabs(gaussians[i].sigma - new_sigma))
        gaussians[i].mu = new_mu
        gaussians[i].sigma = new_sigma

    return np.array(changes)
```

**5) Perform the complete EM and plot your results:**

Initialize three normal distributions whose parameters will be changed iteratively by the EM to converge close to the original distributions.

Build a loop around the iterative procedure of expectation and maximization which stops when the changes in all $\mu_j$ and $\sigma_j$ are sufficiently small enough.

Plot your results after each step and mark which data points belong to which normal distribution. If you don't get it to work, just plot your final solution.

*Hint:* Remember to load the data and initialize the EM before the loop.

*Hint:* A function `plot_intermediate_result` to plot your result after each step is already defined in the next cell. Take a look at what arguments it takes and try to use it in your loop.

*Hint:* To plot your final result the first three images and corresponding code examples on the tutorial of [`plt.plot(...)`](http://matplotlib.org/users/pyplot_tutorial.html) should help you.

*Optional:* Run the code multiple times. If your results are changing, use `np.random.seed(2)` in the beginning of the cell to get consistent results (any other integer will work as well, but 2 has some good results for the example solutions).

```python
%matplotlib notebook
import time
import itertools

import numpy as np
import matplotlib.pyplot as plt
# Sets the random seed to a fix value to make results consistent
# np.random.seed(2)

colors = itertools.cycle(['r', 'g', 'b', 'c', 'm', 'y', 'k'])
figure, axis = plt.subplots(1)
axis.set_xlim(-5, 5)
axis.set_ylim(-0.2, 4)
axis.set_title('Intermediate Results')
final_figure = plt.figure('Final Result')

def plot_intermediate_result(gaussians, data, mapping):
    """
    Gets a list of gaussians and data input. The mapping
    parameter is a list of indices of gaussians. Each value
    corresponds to the data value at the same position and
    maps this data value to the proper gaussian.
    """
    x = np.linspace(-5, 5, 100)
    if len(axis.lines):
        for j, N in enumerate(gaussians):
            axis.lines[j * 2].set_xdata(x)
            axis.lines[j * 2].set_ydata(N(x))
            axis.lines[j * 2 + 1].set_xdata(data[mapping == j])
            axis.lines[j * 2 + 1].set_ydata([0] * len(data[mapping == j]))
    else:
        for j, N in enumerate(gaussians):
            axis.plot(x, N(x), data[mapping == j], [0] * len(data[mapping == j]), 'x', color=next(colors), markersize=5)
    figure.canvas.draw()
    time.sleep(1.0)

def plot_final(gaussians, data, mapping):
    x = np.linspace(-5, 5, 100)
    for j, g in enumerate(gaussians):
        plt.plot(x, g(x), data[mapping == j], [0] * len(data[mapping == j]), 'x', color=next(colors), markersize=5)
    final_figure.canvas.draw()

# Perform the initialization.
data = load_data('em_normdistdata.txt')
gaussians = initialize_EM(data, 3)

# Loop until the changes are small enough.
eps = 0.05
changes = [float('inf')] * 2
expectation = []
while max(changes) > eps:
    # E-step
    expectation = expectation_step(gaussians, data)
    # M-step
    changes = maximization_step(gaussians, data, expectation)
    # Optional: Calculate the parameters to update the plot and call the function to do it.
    plot_intermediate_result(gaussians, data, np.argmax(expectation, 1))

# Plot your final result and print the final parameters.
plot_final(gaussians, data, np.argmax(expectation, 1))
```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

### b) EM and missing values

Describe in your own words: How does the EM-algorithm deal with the missing value problem?

In order to deal with missing values, the EM-algorithm estimates the distribution underlying the data and then generates numbers that are sampled from that distribution to fill up the missing values.

To find the distribution, or rather the correct parameters of that distribution, the algorithm makes an initial guess and then, in each iteration, the probability for each data point to belong to a distribution is calculated. That result is used to refactor each distribution's parameters (in our case $\mu$ and $\sigma$) by recalculating them with weighted samples, the weight being that samples' probability of belonging to that distribution.


# Introduction

Classical mechanics is a topic which has been taught intensively over
several centuries. It is, with its many variants and ways of
presenting the educational material, normally the first **real** physics
course many of us meet and it lays the foundation for further physics
studies. Many of the equations and ways of reasoning about the
underlying laws of motion and pertinent forces, shape our approaches and understanding
of the scientific method and discourse, as well as the way we develop our insights
and deeper understanding about physical systems.

There is a wealth of
well-tested (from both a physics point of view and a pedagogical
standpoint) exercises and problems which can be solved
analytically. However, many of these problems represent idealized and
less realistic situations.  The large majority of these problems are
solved by paper and pencil and are traditionally aimed
at what we normally refer to as continuous models from which we may find an analytical solution.  As a consequence,
when teaching mechanics, it implies that we can seldomly venture beyond an idealized case
in order to develop our understandings and insights about the
underlying forces and laws of motion.

On the other hand, numerical algorithms call for approximate discrete
models and much of the development of methods for continuous models
are nowadays being replaced by methods for discrete models in science and
industry, simply because **much larger classes of problems can be addressed** with discrete models, often  by simpler and more
generic methodologies.

As we will see below, when properly scaling the equations at hand,
discrete models open up for more advanced abstractions and the possibility to
study  real life systems, with the added bonus that we can explore and
deepen our basic understanding of various physical systems

Analytical solutions are as important as before. In addition, such
solutions provide us with invaluable benchmarks and tests for our
discrete models. Such benchmarks, as we will see below, allow us
to discuss possible sources of errors and their behaviors.  And
finally, since most of our models are based on various algorithms from
numerical mathematics, we have a unique oppotunity to gain a deeper
understanding of the mathematical approaches we are using.

With computing and data science as important elements in essentially
all aspects of a modern society, we could  then try to define Computing as
**solving scientific problems using all possible tools, including
symbolic computing, computers and numerical algorithms, and analytical
paper and pencil solutions**.
Computing provides us with the tools to develope our own understanding of the scientific method by enhancing algorithmic thinking.

The way we will teach this course reflects
this definition of computing. The course contains both classical paper
and pencil exercises as well as  computational projects and exercises. The
hope is that this will allow you to explore the physics of systems
governed by the degrees of freedom of classical mechanics at a deeper
level, and that these insights about the scientific method will help
you to develop a better understanding of how the underlying forces and
equations of motion and how they impact a given system. Furthermore, by introducing various numerical methods
via computational projects and exercises, we aim at developing your competences and skills about these topics.

These competences will enable you to

* understand how algorithms are used to solve mathematical problems,

* derive, verify, and implement algorithms,

* understand what can go wrong with algorithms,

* use these algorithms to construct reproducible scientific outcomes and to engage in science in ethical ways, and

* think algorithmically for the purposes of gaining deeper insights about scientific problems.

All these elements are central for maturing and gaining a better understanding of the modern scientific process *per se*.

The power of the scientific method lies in identifying a given problem
as a special case of an abstract class of problems, identifying
general solution methods for this class of problems, and applying a
general method to the specific problem (applying means, in the case of
computing, calculations by pen and paper, symbolic computing, or
numerical computing by ready-made and/or self-written software). This
generic view on problems and methods is particularly important for
understanding how to apply available, generic software to solve a
particular problem.

*However, verification of algorithms and understanding their limitations requires much of the classical knowledge about continuous models.*

## A well-known examples to illustrate many of the above concepts

Before we venture into a reminder on Python and mechanics relevant applications, let us briefly outline some of the
abovementioned topics using an example many of you may have seen before in for example CMSE201.
A simple algorithm for integration is the Trapezoidal rule.
Integration of a function $f(x)$ by the Trapezoidal Rule is given by following algorithm for an interval $x \in [a,b]$

$$
\int_a^b(f(x) dx = \frac{1}{2}\left [f(a)+2f(a+h)+\dots+2f(b-h)+f(b)\right] +O(h^2),
$$

where $h$ is the so-called stepsize defined by the number of integration points $N$ as $h=(b-a)/(n)$.
Python offers an  extremely versatile programming  environment, allowing for
the inclusion of analytical studies in a numerical program. Here we show an
example code with the **trapezoidal rule**. We use also **SymPy** to evaluate the exact value of the integral and compute the absolute error
with respect to the numerically evaluated one of the integral
$\int_0^1 dx x^2 = 1/3$.
The following code for  the trapezoidal rule allows you  to plot the relative error by comparing with the exact result. By increasing to $10^8$ points one arrives at a region where numerical errors start to accumulate.

```python
%matplotlib inline

from math import log10
import numpy as np
from sympy import Symbol, integrate
import matplotlib.pyplot as plt
# function for the trapezoidal rule
def Trapez(a,b,f,n):
   h = (b-a)/float(n)
   s = 0
   x = a
   for i in range(1,n,1):
       x = x+h
       s = s+ f(x)
   s = 0.5*(f(a)+f(b)) +s
   return h*s
#  function to compute pi
def function(x):
    return x*x
# define integration limits
a = 0.0;  b = 1.0;
# find result from sympy
# define x as a symbol to be used by sympy
x = Symbol('x')
exact = integrate(function(x), (x, a, b))
# set up the arrays for plotting the relative error
n = np.zeros(9); y = np.zeros(9);
# find the relative error as function of integration points
for i in range(1, 8, 1):
    npts = 10**i
    result = Trapez(a,b,function,npts)
    RelativeError = abs((exact-result)/exact)
    n[i] = log10(npts); y[i] = log10(RelativeError);
plt.plot(n,y, 'ro')
plt.xlabel('n')
plt.ylabel('Relative error')
plt.show()
```

This example shows the potential of combining numerical algorithms with symbolic calculations, allowing us to

* Validate and verify  their  algorithms.

* Including concepts like unit testing, one has the possibility to test and test several or all parts of the code.

* Validation and verification are then included *naturally* and one can develop a better attitude to what is meant with an ethically sound scientific approach.

* The above example allows the student to also test the mathematical error of the algorithm for the trapezoidal rule by changing the number of integration points. The students get **trained from day one to think error analysis**.

* With a Jupyter notebook you can keep exploring similar examples and turn them in as your own notebooks.

In this process we can easily bake in
1. How to structure a code in terms of functions

2. How to make a module

3. How to read input data flexibly from the command line

4. How to create graphical/web user interfaces

5. How to write unit tests (test functions or doctests)

6. How to refactor code in terms of classes (instead of functions only)

7. How to conduct and automate large-scale numerical experiments

8. How to write scientific reports in various formats (LaTeX, HTML)

The conventions and techniques outlined here will save you a lot of time when you incrementally extend software over time from simpler to more complicated problems. In particular, you will benefit from many good habits:
1. New code is added in a modular fashion to a library (modules)

2. Programs are run through convenient user interfaces

3. It takes one quick command to let all your code undergo heavy testing

4. Tedious manual work with running programs is automated,

5. Your scientific investigations are reproducible, scientific reports with top quality typesetting are produced both for paper and electronic devices.


```python
# Erasmus+ ICCT project (2018-1-SI01-KA203-047081)

# Toggle cell visibility

from IPython.display import HTML
tag = HTML('''
Promijeni vidljivost <a href="javascript:code_toggle()">ovdje</a>.''')
display(tag)

# Hide the code completely

# from IPython.display import HTML
# tag = HTML('''<style>
# div.input {
#     display:none;
# }
# </style>''')
# display(tag)

```

Promijeni vidljivost ovdje.

## Unutarnja stabilnost

Koncept stabilnosti bilježi ponašanje evolucije stanja sustava kada je isti "izbačen" iz ravnotežnog stanja: stabilnost opisuje divergira li evolucija stanja koja nastaje nakon poremećaja iz točke ravnoteže ili ne.

### Definicija
S obzirom na vremenski nepromjenjiv dinamički sustav opisan vektorom stanja $x(t)\in \mathbb{R}^n$, točkom ravnoteže $x_e$, početnim stanjem $x_0$ i početnim vremenom $t_0$, ako vrijedi

$$
\forall \, \epsilon \in \mathbb{R}, \, \epsilon > 0 \quad \exists \delta \in \mathbb{R}, \, \delta > 0 : \quad ||x_0-x_e|| < \delta \, \Rightarrow  \, ||x(t)-x_e|| < \epsilon \quad \forall t \ge t_0
$$
to bi se moglo protumačiti kao: ako postoji dovoljno mala početna perturbacija $\delta$ od točke ravnoteže takva da evolucija stanja $x(t)$ od točke poremećaja ne odlazi predaleko (više od $\epsilon$) od same ravnoteže, tada je točka ravnoteže stabilna.

Ako se također dogodi $\lim_{t\to\infty}||x(t)-x_e|| = 0$, što se može protumačiti kao: evolucija stanja se vraća natrag u točku ravnoteže, tada se kaže da je ravnoteža asimptotski stabilna.

U slučaju linearnih vremenski nepromjenjivih sustava:

\begin{cases}
\dot{x} = Ax +Bu \\
y = Cx + Du,
\end{cases}

moguće je dokazati da stabilnost jedne točke ravnoteže podrazumijeva stabilnost svih točaka ravnoteže, pa možemo govoriti o stabilnosti sustava čak i ako je, općenito, svojstvo stabilnosti povezano s točkom ravnoteže. Posebnost ovog linearnog sustava posljedica je činjenice da je evolucija ove vrste sustava strogo povezana sa svojstvenim vrijednostima matrice dinamike $A$, koje su invarijantne s obzirom na rotaciju, translaciju, početne uvjete i vrijeme.

Podsjetite se što je objašnjeno u primjeru o modalnoj analizi:

> Rješenje diferencijalne jednadžbe (u zatvorenoj formi), od početnog vremena $t_0$, s početnim uvjetima $x(t_0)$, je
$$
x(t) = e^{A(t-t_0)}x(t_0).
$$ Matrica $e^{A(t-t_0)}x(t_0)$ se sastoji od linearnih kombinacija funkcije vremena $t$, svake tipa: $$e^{\lambda t},$$ gdje su $\lambda$-e svojstvene vrijednosti matrice $A$; ove su funckije modovi sustava.

stoga:
- linearni dinamički sustav stabilan je ako i samo ako svi njegovi modovi nisu divergentni,
- linearni dinamički sustav je asimptotski stabilan ako i samo ako su svi njegovi modovi konvergentni,
- linearni dinamički sustav je nestabilan ako ima barem jedan divergentni mod.

i, s obzirom na svojstvene vrijednosti matrice dinamike, to se događa ako:
- sve svojstvene vrijednosti matrice $A$ pripadaju zatvorenoj lijevoj polovici kompleksne ravnine (tj. realna vrijednost im je negativna ili nula), a, u slučaju da imaju realnu vrijednost nula, njihova je algebarska višestrukost ista kao geometrijska višestrukost (množnost), ili, ekvivalentno tome, imaju skalarne blokove u Jordanovom obliku;
- sve svojstvene vrijednosti pripadaju otvorenoj lijevoj polovici imaginarne ravnine, odnosno imaju strogo negativne realne dijelove;
- barem jedna svojstvena vrijednost ima pozitivan realni dio ili postoje svojstvene vrijednosti s realnom vrijednošću 0 i neskalarni Jordanovi blokovi.

Ovaj interaktivni primjer predstavlja matricu dinamike $A$ koju je moguće uređivati, a prikazuje slobodni odziv sustava uz odgovarajuće svojstvene vrijednosti.

### Kako koristiti ovaj interaktivni primjer?
- Pokušajte promijeniti svojstvene vrijednosti i početni uvjet $x_0$ i pogledajte kako se mijenja odziv.

```python
%matplotlib inline
#%matplotlib notebook
import control as control
import numpy
import sympy as sym
from IPython.display import display, Markdown
import ipywidgets as widgets
import matplotlib.pyplot as plt

#print a matrix latex-like
def bmatrix(a):
     """Returns a LaTeX bmatrix - by Damir Arbula (ICCT project)

     :a: numpy array
     :returns: LaTeX bmatrix as a string
     """
     if len(a.shape) > 2:
         raise ValueError('bmatrix can at most display two dimensions')
     lines = str(a).replace('[', '').replace(']', '').splitlines()
     rv = [r'\begin{bmatrix}']
     rv += ['  ' + ' & '.join(l.split()) + r'\\' for l in lines]
     rv +=  [r'\end{bmatrix}']
     return '\n'.join(rv)

# Display formatted matrix:
def vmatrix(a):
    if len(a.shape) > 2:
         raise ValueError('bmatrix can at most display two dimensions')
    lines = str(a).replace('[', '').replace(']', '').splitlines()
    rv = [r'\begin{vmatrix}']
    rv += ['  ' + ' & '.join(l.split()) + r'\\' for l in lines]
    rv +=  [r'\end{vmatrix}']
    return '\n'.join(rv)

#matrixWidget is a matrix looking widget built with a VBox of HBox(es) that returns a numPy array as value !
class matrixWidget(widgets.VBox):
    def updateM(self,change):
        for irow in range(0,self.n):
            for icol in range(0,self.m):
                self.M_[irow,icol] = self.children[irow].children[icol].value
                #print(self.M_[irow,icol])
        self.value = self.M_

    def dummychangecallback(self,change):
        pass

    def __init__(self,n,m):
        self.n = n
        self.m = m
        self.M_ = numpy.matrix(numpy.zeros((self.n,self.m)))
        self.value = self.M_
        widgets.VBox.__init__(self,
                             children = [
                                 widgets.HBox(children =
                                              [widgets.FloatText(value=0.0, layout=widgets.Layout(width='90px')) for i in range(m)]
                                             )
                                 for j in range(n)
                             ])

        #fill in widgets and tell interact to call updateM each time a children changes value
        for irow in range(0,self.n):
            for icol in range(0,self.m):
                self.children[irow].children[icol].value = self.M_[irow,icol]
                self.children[irow].children[icol].observe(self.updateM, names='value')
        #value = Unicode('example@example.com', help="The email value.").tag(sync=True)
        self.observe(self.updateM, names='value', type= 'All')

    def setM(self, newM):
        #disable callbacks, change values, and reenable
        self.unobserve(self.updateM, names='value', type= 'All')
        for irow in range(0,self.n):
            for icol in range(0,self.m):
                self.children[irow].children[icol].unobserve(self.updateM, names='value')
        self.M_ = newM
        self.value = self.M_
        for irow in range(0,self.n):
            for icol in range(0,self.m):
                self.children[irow].children[icol].value = self.M_[irow,icol]
        for irow in range(0,self.n):
            for icol in range(0,self.m):
                self.children[irow].children[icol].observe(self.updateM, names='value')
        self.observe(self.updateM, names='value', type= 'All')

                #self.children[irow].children[icol].observe(self.updateM, names='value')

#overlaod class for state space systems that DO NOT remove "useless" states (what "professor" of automatic control would do this?)
class sss(control.StateSpace):
    def __init__(self,*args):
        #call base class init constructor
        control.StateSpace.__init__(self,*args)
    #disable function below in base class
    def _remove_useless_states(self):
        pass
```

```python
# Preparatory cell

A = numpy.matrix([[0,1],[-2/5,-1/5]])
X0 = numpy.matrix('5; 3')

Aw = matrixWidget(2,2)
Aw.setM(A)
X0w = matrixWidget(2,1)
X0w.setM(X0)
```

```python
# Misc

#create dummy widget
DW = widgets.FloatText(layout=widgets.Layout(width='0px', height='0px'))

#create button widget
START = widgets.Button(
    description='Test',
    disabled=False,
    button_style='', # 'success', 'info', 'warning', 'danger' or ''
    tooltip='Test',
    icon='check'
)

def on_start_button_clicked(b):
    #This is a workaround to have intreactive_output call the callback:
    #   force the value of the dummy widget to change
    if DW.value> 0 :
        DW.value = -1
    else:
        DW.value = 1
    pass
START.on_click(on_start_button_clicked)
```

```python
# Main cell

def main_callback(A, X0, DW):
    sols = numpy.linalg.eig(A)
    sys = sss(A,[[1],[0]],[0,1],0)
    pole = control.pole(sys)
    if numpy.real(pole[0]) != 0:
        p1r = abs(numpy.real(pole[0]))
    else:
        p1r = 1
    if numpy.real(pole[1]) != 0:
        p2r = abs(numpy.real(pole[1]))
    else:
        p2r = 1
    if numpy.imag(pole[0]) != 0:
        p1i = abs(numpy.imag(pole[0]))
    else:
        p1i = 1
    if numpy.imag(pole[1]) != 0:
        p2i = abs(numpy.imag(pole[1]))
    else:
        p2i = 1

    print('Svojstvene vrijednosti matrice A su:',round(sols[0][0],4),'i',round(sols[0][1],4))

    #T = numpy.linspace(0, 60, 1000)
    T, yout, xout = control.initial_response(sys,X0=X0,return_x=True)

    fig = plt.figure("Svojstvene vrijednosti od A", figsize=(16,16))
    ax = fig.add_subplot(311,title='Polovi (Re vs Img)')
    #plt.axis(True)
    # Move left y-axis and bottim x-axis to centre, passing through (0,0)
    # Eliminate upper and right axes
    ax.spines['left'].set_position(('data',0.0))
    ax.spines['bottom'].set_position(('data',0.0))
    ax.spines['right'].set_color('none')
    ax.spines['top'].set_color('none')
    ax.set_xlim(-max([p1r+p1r/3,p2r+p2r/3]),
                max([p1r+p1r/3,p2r+p2r/3]))
    ax.set_ylim(-max([p1i+p1i/3,p2i+p2i/3]),
                max([p1i+p1i/3,p2i+p2i/3]))

    plt.plot([numpy.real(pole[0]),numpy.real(pole[1])],[numpy.imag(pole[0]),numpy.imag(pole[1])],'o')
    plt.grid()

    ax1 = fig.add_subplot(312,title='Slobodni odziv')
    plt.plot(T,xout[0])
    plt.grid()
    ax1.set_xlabel('vrijeme [s]')
    ax1.set_ylabel('$x_1$')
    ax1.axvline(x=0,color='black',linewidth='0.8')
    ax1.axhline(y=0,color='black',linewidth='0.8')
    ax2 = fig.add_subplot(313)
    plt.plot(T,xout[1])
    plt.grid()
    ax2.set_xlabel('vrijeme [s]')
    ax2.set_ylabel('$x_2$')
    ax2.axvline(x=0,color='black',linewidth='0.8')
    ax2.axhline(y=0,color='black',linewidth='0.8')

    #plt.show()

alltogether = widgets.HBox([widgets.VBox([widgets.Label('$A$:',border=3),
                                          Aw]),
                                          widgets.Label('   ',border=3),
                            widgets.VBox([widgets.Label('$X_0$:',border=3),
                                          X0w]),
                                          START])
out = widgets.interactive_output(main_callback, {'A':Aw, 'X0':X0w, 'DW':DW})
out.layout.height = '1000px'
display(out, alltogether)
```

    Output(layout=Layout(height='1000px'))

    HBox(children=(VBox(children=(Label(value='$A$:'), matrixWidget(children=(HBox(children=(FloatText(value=0.0, …