Natural Language Processing From Scratch

Text Representation

Bruno Gonçalves

        www.data4sci.com

        @bgoncalves, @data4sci

In this lesson we will see in some details how we can best represent text in our application. Let's start by importing the modules we will be using:

```python
import string
from collections import Counter
from pprint import pprint
import gzip
import matplotlib.pyplot as plt
import numpy as np

%matplotlib inline
%load_ext watermark
```

List out the versions of all loaded libraries

```python
%watermark -n -v -m -g -iv
```

    matplotlib 3.1.0
    numpy      1.16.2
    Mon Nov 11 2019

    CPython 3.7.3
    IPython 6.2.1

    compiler   : Clang 4.0.1 (tags/RELEASE_401/final)
    system     : Darwin
    release    : 18.7.0
    machine    : x86_64
    processor  : i386
    CPU cores  : 8
    interpreter: 64bit
    Git hash   : b9e7a934ea44d9018471e17a5808b247be788f1f

Set the default style

```python
plt.style.use('./d4sci.mplstyle')
```

We choose a well known nursery rhyme, that has the added distinction of having been the first audio ever recorded, to be the short snippet of text that we will use in our examples:

```python
text = """Mary had a little lamb, little lamb,
    little lamb. Mary had a little lamb
    whose fleece was white as snow.
    And everywhere that Mary went
    Mary went, Mary went. Everywhere
    that Mary went,
    The lamb was sure to go"""
```

## Tokenization

The first step in any analysis is to tokenize the text. What this means is that we will extract all the individual words in the text. For the sake of simplicity, we will assume that our text is well formed and that our words are delimited either by white space or punctuation characters.

```python
print(string.punctuation)
```

    !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~

```python
def extract_words(text):
    temp = text.split() # Split the text on whitespace
    text_words = []

    for word in temp:
        # Remove any punctuation characters present in the beginning of the word
        while word[0] in string.punctuation:
            word = word[1:]

        # Remove any punctuation characters present in the end of the word
        while word[-1] in string.punctuation:
            word = word[:-1]

        # Append this word into our list of words.
        text_words.append(word.lower())

    return text_words
```

After this step we now have our text represented as an array of individual, lowercase, words:

```python
text_words = extract_words(text)
print(text_words)
```

    ['mary', 'had', 'a', 'little', 'lamb', 'little', 'lamb', 'little', 'lamb', 'mary', 'had', 'a', 'little', 'lamb', 'whose', 'fleece', 'was', 'white', 'as', 'snow', 'and', 'everywhere', 'that', 'mary', 'went', 'mary', 'went', 'mary', 'went', 'everywhere', 'that', 'mary', 'went', 'the', 'lamb', 'was', 'sure', 'to', 'go']

As we saw during the video, this is a wasteful way to represent text. We can be much more efficient by representing each word by a number

```python
word_dict = {}
word_list = []
vocabulary_size = 0
text_tokens = []

for word in text_words:
    # If we are seeing this word for the first time, create an id for it and added it to our word dictionary
    if word not in word_dict:
        word_dict[word] = vocabulary_size
        word_list.append(word)
        vocabulary_size += 1

    # add the token corresponding to the current word to the tokenized text.
    text_tokens.append(word_dict[word])
```

When we were tokenizing our text, we also generated a dictionary **word_dict** that maps words to integers and a **word_list** that maps each integer to the corresponding word.

```python
print("Word list:", word_list, "\n\n Word dictionary:")
pprint(word_dict)
```

    Word list: ['mary', 'had', 'a', 'little', 'lamb', 'whose', 'fleece', 'was', 'white', 'as', 'snow', 'and', 'everywhere', 'that', 'went', 'the', 'sure', 'to', 'go']

     Word dictionary:
    {'a': 2,
     'and': 11,
     'as': 9,
     'everywhere': 12,
     'fleece': 6,
     'go': 18,
     'had': 1,
     'lamb': 4,
     'little': 3,
     'mary': 0,
     'snow': 10,
     'sure': 16,
     'that': 13,
     'the': 15,
     'to': 17,
     'was': 7,
     'went': 14,
     'white': 8,
     'whose': 5}

These two datastructures already proved their usefulness when we converted our text to a list of tokens.

```python
print(text_tokens)
```

    [0, 1, 2, 3, 4, 3, 4, 3, 4, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 0, 14, 0, 14, 0, 14, 12, 13, 0, 14, 15, 4, 7, 16, 17, 18]

Unfortunately, while this representation is convenient for memory reasons it has some severe limitations. Perhaps the most important of which is the fact that computers naturally assume that numbers can be operated on mathematically (by addition, subtraction, etc) in a way that doesn't match our understanding of words.

## One-hot encoding

One typical way of overcoming this difficulty is to represent each word by a one-hot encoded vector where every element is zero except the one corresponding to a specific word.

```python
def one_hot(word, word_dict):
    """
        Generate a one-hot encoded vector corresponding to *word*
    """

    vector = np.zeros(len(word_dict))
    vector[word_dict[word]] = 1

    return vector
```

So, for example, the word "fleece" would be represented by:

```python
fleece_hot = one_hot("fleece", word_dict)
print(fleece_hot)
```

    [0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]

This vector has every element set to zero, except element 6, since:

```python
print(word_dict["fleece"])
fleece_hot[6] == 1
```

    6

    True

```python
print(fleece_hot.sum())
```

    1.0

## Bag of words

We can now use the one-hot encoded vector for each word to produce a vector representation of our original text, by simply adding up all the one-hot encoded vectors:

```python
text_vector1 = np.zeros(vocabulary_size)

for word in text_words:
    hot_word = one_hot(word, word_dict)
    text_vector1 += hot_word

print(text_vector1)
```

    [6. 2. 2. 4. 5. 1. 1. 2. 1. 1. 1. 1. 2. 2. 4. 1. 1. 1. 1.]

In practice, we can also easily skip the encoding step at the word level by using the *word_dict* defined above:

```python
text_vector = np.zeros(vocabulary_size)

for word in text_words:
    text_vector[word_dict[word]] += 1

print(text_vector)
```

    [6. 2. 2. 4. 5. 1. 1. 2. 1. 1. 1. 1. 2. 2. 4. 1. 1. 1. 1.]

Naturally, this approach is completely equivalent to the previous one and has the added advantage of being more efficient in terms of both speed and memory requirements.

This is known as the __bag of words__ representation of the text. It should be noted that these vectors simply contains the number of times each word appears in our document, so we can easily tell that the word *mary* appears exactly 6 times in our little nursery rhyme.

```python
text_vector[word_dict["mary"]]
```

    6.0

A more pythonic (and efficient) way of producing the same result is to use the standard __Counter__ module:

```python
text_words
```

    ['mary',
     'had',
     'a',
     'little',
     'lamb',
     'little',
     'lamb',
     'little',
     'lamb',
     'mary',
     'had',
     'a',
     'little',
     'lamb',
     'whose',
     'fleece',
     'was',
     'white',
     'as',
     'snow',
     'and',
     'everywhere',
     'that',
     'mary',
     'went',
     'mary',
     'went',
     'mary',
     'went',
     'everywhere',
     'that',
     'mary',
     'went',
     'the',
     'lamb',
     'was',
     'sure',
     'to',
     'go']

```python
word_counts = Counter(text_words)
pprint(word_counts)
```

    Counter({'mary': 6,
             'lamb': 5,
             'little': 4,
             'went': 4,
             'had': 2,
             'a': 2,
             'was': 2,
             'everywhere': 2,
             'that': 2,
             'whose': 1,
             'fleece': 1,
             'white': 1,
             'as': 1,
             'snow': 1,
             'and': 1,
             'the': 1,
             'sure': 1,
             'to': 1,
             'go': 1})

From which we can easily generate the __text_vector__ and __word_dict__ data structures:

```python
items = list(word_counts.items())

# Extract word dictionary and vector representation
word_dict2 = dict([[items[i][0], i] for i in range(len(items))])
text_vector2 = [items[i][1] for i in range(len(items))]
```

```python
word_counts['mary']
```

    6

And let's take a look at them:

```python
text_vector
```

    array([6., 2., 2., 4., 5., 1., 1., 2., 1., 1., 1., 1., 2., 2., 4., 1., 1.,
           1., 1.])

```python
print("Text vector:", text_vector2, "\n\nWord dictionary:")
pprint(word_dict2)
```

    Text vector: [6, 2, 2, 4, 5, 1, 1, 2, 1, 1, 1, 1, 2, 2, 4, 1, 1, 1, 1]

    Word dictionary:
    {'a': 2,
     'and': 11,
     'as': 9,
     'everywhere': 12,
     'fleece': 6,
     'go': 18,
     'had': 1,
     'lamb': 4,
     'little': 3,
     'mary': 0,
     'snow': 10,
     'sure': 16,
     'that': 13,
     'the': 15,
     'to': 17,
     'was': 7,
     'went': 14,
     'white': 8,
     'whose': 5}

The results using this approach are slightly different than the previous ones, because the words are mapped to different integer ids but the corresponding values are the same:

```python
for word in word_dict.keys():
    if text_vector[word_dict[word]] != text_vector2[word_dict2[word]]:
        print("Error!")
```

As expected, there are no differences!

## Term Frequency

The bag of words vector representation introduced above relies simply on the frequency of occurence of each word. Following a long tradition of giving fancy names to simple ideas, this is known as __Term Frequency__.

Intuitively, we expect the the frequency with which a given word is mentioned should correspond to the relevance of that word for the piece of text we are considering. For example, **Mary** is a pretty important word in our little nursery rhyme and indeed it is the one that occurs the most often:

```python
sorted(items, key=lambda x:x[1], reverse=True)
```

    [('mary', 6),
     ('lamb', 5),
     ('little', 4),
     ('went', 4),
     ('had', 2),
     ('a', 2),
     ('was', 2),
     ('everywhere', 2),
     ('that', 2),
     ('whose', 1),
     ('fleece', 1),
     ('white', 1),
     ('as', 1),
     ('snow', 1),
     ('and', 1),
     ('the', 1),
     ('sure', 1),
     ('to', 1),
     ('go', 1)]

However, it's hard to draw conclusions from such a small piece of text. Let us consider a significantly larger piece of text, the first 100 MB of the english Wikipedia from: http://mattmahoney.net/dc/textdata. For the sake of convenience, text8.gz has been included in this repository in the **data/** directory. We start by loading it's contents into memory as an array of words:

```python
data = []

for line in gzip.open("data/text8.gz", 'rt'):
    data.extend(line.strip().split())
```

Now let's take a look at the first 50 words in this large corpus:

```python
data[:50]
```

    ['anarchism',
     'originated',
     'as',
     'a',
     'term',
     'of',
     'abuse',
     'first',
     'used',
     'against',
     'early',
     'working',
     'class',
     'radicals',
     'including',
     'the',
     'diggers',
     'of',
     'the',
     'english',
     'revolution',
     'and',
     'the',
     'sans',
     'culottes',
     'of',
     'the',
     'french',
     'revolution',
     'whilst',
     'the',
     'term',
     'is',
     'still',
     'used',
     'in',
     'a',
     'pejorative',
     'way',
     'to',
     'describe',
     'any',
     'act',
     'that',
     'used',
     'violent',
     'means',
     'to',
     'destroy',
     'the']

And the top 10 most common words

```python
counts = Counter(data)

sorted_counts = sorted(list(counts.items()), key=lambda x:x[1], reverse=True)

for word, count in sorted_counts[:10]:
    print(word, count)
```

    the 1061396
    of 593677
    and 416629
    one 411764
    in 372201
    a 325873
    to 316376
    zero 264975
    nine 250430
    two 192644

Surprisingly, we find that the most common words are not particularly meaningful. Indeed, this is a common occurence in Natural Language Processing. The most frequent words are typically auxiliaries required due to gramatical rules.

On the other hand, there is also a large number of words that occur very infrequently as can be easily seen by glancing at the word freqency distribution.

```python
dist = Counter(counts.values())
dist = list(dist.items())
dist.sort(key=lambda x:x[0])
dist = np.array(dist)

norm = np.dot(dist.T[0], dist.T[1])

plt.loglog(dist.T[0], dist.T[1]/norm)
plt.xlabel("count")
plt.ylabel("P(count)")
plt.title("Word frequency distribution")
plt.gcf().set_size_inches(11, 8)
```

## Stopwords

One common technique to simplify NLP tasks is to remove what are known as Stopwords, words that are very frequent but not meaningful. If we simply remove the most common 100 words, we significantly reduce the amount of data we have to consider while losing little information.

```python
stopwords = set([word for word, count in sorted_counts[:100]])

clean_data = []

for word in data:
    if word not in stopwords:
        clean_data.append(word)

print("Original size:", len(data))
print("Clean size:", len(clean_data))
print("Reduction:", 1-len(clean_data)/len(data))
```

    Original size: 17005207
    Clean size: 9006229
    Reduction: 0.470384041782026

```python
clean_data[:50]
```

    ['anarchism',
     'originated',
     'term',
     'abuse',
     'against',
     'early',
     'working',
     'class',
     'radicals',
     'including',
     'diggers',
     'english',
     'revolution',
     'sans',
     'culottes',
     'french',
     'revolution',
     'whilst',
     'term',
     'still',
     'pejorative',
     'way',
     'describe',
     'any',
     'act',
     'violent',
     'means',
     'destroy',
     'organization',
     'society',
     'taken',
     'positive',
     'label',
     'self',
     'defined',
     'anarchists',
     'word',
     'anarchism',
     'derived',
     'greek',
     'without',
     'archons',
     'ruler',
     'chief',
     'king',
     'anarchism',
     'political',
     'philosophy',
     'belief',
     'rulers']

Wow, our dataset size was reduced almost in half!

In practice, we don't simply remove the most common words in our corpus but rather a manually curate list of stopwords. Lists for dozens of languages and applications can easily be found online.

## Term Frequency/Inverse Document Frequency

One way of determining of the relative importance of a word is to see how often it appears across multiple documents. Words that are relevant to a specific topic are more likely to appear in documents about that topic and much less in documents about other topics. On the other hand, less meaningful words (like **the**) will be common across documents about any subject.

To measure the document frequency of a word we will need to have multiple documents. For the sake of simplicity, we will treat each sentence of our nursery rhyme as an individual document:

```python
print(text)
```

    Mary had a little lamb, little lamb,
        little lamb. Mary had a little lamb
        whose fleece was white as snow.
        And everywhere that Mary went
        Mary went, Mary went. Everywhere
        that Mary went,
        The lamb was sure to go

```python
corpus_text = text.split('.')
corpus_words = []

for document in corpus_text:
    doc_words = extract_words(document)
    corpus_words.append(doc_words)
```

Now our corpus is represented as a list of word lists, where each list is just the word representation of the corresponding sentence:

```python
pprint(corpus_words)
```

    [['mary', 'had', 'a', 'little', 'lamb', 'little', 'lamb', 'little', 'lamb'],
     ['mary',
      'had',
      'a',
      'little',
      'lamb',
      'whose',
      'fleece',
      'was',
      'white',
      'as',
      'snow'],
     ['and', 'everywhere', 'that', 'mary', 'went', 'mary', 'went', 'mary', 'went'],
     ['everywhere',
      'that',
      'mary',
      'went',
      'the',
      'lamb',
      'was',
      'sure',
      'to',
      'go']]

Let us now calculate the number of documents in which each word appears:

```python
document_count = {}

for document in corpus_words:
    word_set = set(document)

    for word in word_set:
        document_count[word] = document_count.get(word, 0) + 1

pprint(document_count)
```

    {'a': 2,
     'and': 1,
     'as': 1,
     'everywhere': 2,
     'fleece': 1,
     'go': 1,
     'had': 2,
     'lamb': 3,
     'little': 2,
     'mary': 4,
     'snow': 1,
     'sure': 1,
     'that': 2,
     'the': 1,
     'to': 1,
     'was': 2,
     'went': 2,
     'white': 1,
     'whose': 1}

As we can see, the word __Mary__ appears in all 4 of our documents, making it useless when it comes to distinguish between the different sentences. On the other hand, words like __white__ which appear in only one document are very discriminative. Using this approach we can define a new quantity, the ___Inverse Document Frequency__ that tells us how frequent a word is across the documents in a specific corpus:

```python
def inv_doc_freq(corpus_words):
    number_docs = len(corpus_words)

    document_count = {}

    for document in corpus_words:
        word_set = set(document)

        for word in word_set:
            document_count[word] = document_count.get(word, 0) + 1

    IDF = {}

    for word in document_count:
        IDF[word] = np.log(number_docs/document_count[word])

    return IDF
```

Where we followed the convention of using the logarithm of the inverse document frequency. This has the numerical advantage of avoiding to have to handle small fractional numbers.

We can easily see that the IDF gives a smaller weight to the most common words and a higher weight to the less frequent:

```python
corpus_words
```

    [['mary', 'had', 'a', 'little', 'lamb', 'little', 'lamb', 'little', 'lamb'],
     ['mary',
      'had',
      'a',
      'little',
      'lamb',
      'whose',
      'fleece',
      'was',
      'white',
      'as',
      'snow'],
     ['and', 'everywhere', 'that', 'mary', 'went', 'mary', 'went', 'mary', 'went'],
     ['everywhere',
      'that',
      'mary',
      'went',
      'the',
      'lamb',
      'was',
      'sure',
      'to',
      'go']]

```python
IDF = inv_doc_freq(corpus_words)

pprint(IDF)
```

    {'a': 0.6931471805599453,
     'and': 1.3862943611198906,
     'as': 1.3862943611198906,
     'everywhere': 0.6931471805599453,
     'fleece': 1.3862943611198906,
     'go': 1.3862943611198906,
     'had': 0.6931471805599453,
     'lamb': 0.28768207245178085,
     'little': 0.6931471805599453,
     'mary': 0.0,
     'snow': 1.3862943611198906,
     'sure': 1.3862943611198906,
     'that': 0.6931471805599453,
     'the': 1.3862943611198906,
     'to': 1.3862943611198906,
     'was': 0.6931471805599453,
     'went': 0.6931471805599453,
     'white': 1.3862943611198906,
     'whose': 1.3862943611198906}

As expected **Mary** has the smallest weight of all words 0, meaning that it is effectively removed from the dataset. You can consider this as a way of implicitly identify and remove stopwords. In case you do want to keep even the words that appear in every document, you can just add a 1. to the argument of the logarithm above:

\begin{equation}
\log\left[1+\frac{N_d}{N_d\left(w\right)}\right]
\end{equation}

When we multiply the term frequency of each word by it's inverse document frequency, we have a good way of quantifying how relevant a word is to understand the meaning of a specific document.

```python
def tf_idf(corpus_words):
    IDF = inv_doc_freq(corpus_words)

    TFIDF = []

    for document in corpus_words:
        TFIDF.append(Counter(document))

    for document in TFIDF:
        for word in document:
            document[word] = document[word]*IDF[word]

    return TFIDF
```

```python
tf_idf(corpus_words)
```

    [Counter({'a': 0.6931471805599453,
              'had': 0.6931471805599453,
              'lamb': 0.8630462173553426,
              'little': 2.0794415416798357,
              'mary': 0.0}),
     Counter({'a': 0.6931471805599453,
              'as': 1.3862943611198906,
              'fleece': 1.3862943611198906,
              'had': 0.6931471805599453,
              'lamb': 0.28768207245178085,
              'little': 0.6931471805599453,
              'mary': 0.0,
              'snow': 1.3862943611198906,
              'was': 0.6931471805599453,
              'white': 1.3862943611198906,
              'whose': 1.3862943611198906}),
     Counter({'and': 1.3862943611198906,
              'everywhere': 0.6931471805599453,
              'mary': 0.0,
              'that': 0.6931471805599453,
              'went': 2.0794415416798357}),
     Counter({'everywhere': 0.6931471805599453,
              'go': 1.3862943611198906,
              'lamb': 0.28768207245178085,
              'mary': 0.0,
              'sure': 1.3862943611198906,
              'that': 0.6931471805599453,
              'the': 1.3862943611198906,
              'to': 1.3862943611198906,
              'was': 0.6931471805599453,
              'went': 0.6931471805599453})]

Now we finally have a vector representation of each of our documents that takes the informational contributions of each word into account. Each of these vectors provides us with a unique representation of each document, in the context (corpus) in which it occurs, making it posssible to define the similarity of two documents, etc.

## Porter Stemmer

There is still, however, one issue with our approach to representing text. Since we treat each word as a unique token and completely independently from all others, for large documents we will end up with many variations of the same word such as verb conjugations, the corresponding adverbs and nouns, etc.

One way around this difficulty is to use stemming algorithm to reduce words to their root (or stem) version. The most famous Stemming algorithm is known as the **Porter Stemmer** and was introduced by Martin Porter in 1980 [Program 14, 130 (1980)](https://dl.acm.org/citation.cfm?id=275705)

The algorithm starts by defining consonants (C) and vowels (V):

```python
V = set('aeiouy')
C = set('bcdfghjklmnpqrstvwxz')
```

The stem of a word is what is left of that word after a speficic ending has been removed. A function to do this is easy to implement:

```python
def get_stem(suffix, word):
    """
        Extract the stem of a word
    """

    if word.lower().endswith(suffix.lower()): # Case insensitive comparison
        return word[:-len(suffix)]

    return None
```

It also defines words (or stems) to be sequences of vowels and consonants of the form:

\begin{equation}
[C](VC)^m[V]
\end{equation}

where $m$ is called the **measure** of the word and [] represent optional sections.

```python
def measure(orig_word):
    """
        Calculate the "measure" m of a word or stem, according to the Porter Stemmer algorthim
    """

    word = orig_word.lower()

    optV = False
    optC = False
    VC = False
    m = 0

    pos = 0

    # We can think of this implementation as a simple finite state machine
    # looks for sequences of vowels or consonants depending of the state
    # in which it's in, while keeping track of how many VC sequences it
    # has encountered.
    # The presence of the optional V and C portions is recorded in the
    # optV and optC booleans.

    # We're at the initial state.
    # gobble up all the optional consonants at the beginning of the word
    while pos < len(word) and word[pos] in C:
        pos += 1
        optC = True

    while pos < len(word):
        # Now we know that the next state must be a vowel
        while pos < len(word) and word[pos] in V:
            pos += 1
            optV = True

        # Followed by a consonant
        while pos < len(word) and word[pos] in C:
            pos += 1
            optV = False

        # If a consonant was found, then we matched VC
        # so we should increment m by one. Otherwise,
        # optV remained true and we simply had a dangling
        # V sequence.
        if not optV:
            m += 1

    return m
```

Let's consider a simple example. The word __crepusculars__ should have measure 4:

[cr] (ep) (usc) (ul) (ars)

and indeed it does.

```python
word = "crepusculars"
print(measure(word))
```

    4

The Porter algorithm sequentially applies a series of transformation rules over a series of 5 steps (step 1 is divided in 3 substeps and step 5 in 2). The rules are only applied if a certain condition is true.

In addition to possibily specifying a requirement on the measure of a word, conditions can make use of different boolean functions as well:

```python
def ends_with(char, stem):
    """
        Checks the ending of the word
    """
    return stem[-1] == char

def double_consonant(stem):
    """
        Checks the ending of a word for a double consonant
    """
    if len(stem) < 2:
        return False

    if stem[-1] in C and stem[-2] == stem[-1]:
        return True

    return False

def contains_vowel(stem):
    """
        Checks if a word contains a vowel or not
    """
    return len(set(stem) & V) > 0
```

Finally, we define a function to apply a specific rule to a word or stem:

```python
def apply_rule(condition, suffix, replacement, word):
    """
        Apply Porter Stemmer rule.
        if "condition" is True replace "suffix" by "replacement" in "word"
    """

    stem = get_stem(suffix, word)

    if stem is not None and condition is True:
        # Remove the suffix
        word = stem

        # Add the replacement suffix, if any
        if replacement is not None:
            word += replacement

    return word
```

Now we can see how rules can be applied. For example, this rule, from step 1b is successfully applied to __pastered__:

```python
word = "plastered"
suffix = "ed"
stem = get_stem(suffix, word)
apply_rule(contains_vowel(stem), suffix, None, word)
```

    'plaster'

```python
stem
```

    'plaster'

```python
contains_vowel(stem)
```

    True

While try applying the same rule to **bled** will fail to pass the condition resulting in no change.

```python
word = "bled"
suffix = "ed"
stem = get_stem(suffix, word)
apply_rule(contains_vowel(stem), suffix, None, word)
```

    'bled'

```python
stem
```

    'bl'

```python
contains_vowel(stem)
```

    False

For a more complex example, we have, in Step 4:

```python
word = "adoption"
suffix = "ion"
stem = get_stem(suffix, word)
apply_rule(measure(stem) > 1 and (ends_with("s", stem) or ends_with("t", stem)), suffix, None, word)
```

    'adopt'

```python
ends_with("t", stem)
```

    True

```python
measure(stem)
```

    2

In total, the Porter Stemmer algorithm (for the English language) applies several dozen rules (see https://tartarus.org/martin/PorterStemmer/def.txt for a complete list). Implementing all of them is both tedious and error prone, so we abstain from providing a full implementation of the algorithm here. High quality implementations can be found in all major NLP libraries such as [NLTK](http://www.nltk.org/howto/stem.html).

The dificulties of defining matching rules to arbitrary text cannot be fully resolved without the use of Regular Expressions (typically implemented as Finite State Machines like our __measure__ implementation above), a more advanced topic that is beyond the scope of this course.


# Homework 1
(c) 2017 Justin Bois. This work is licensed under a [Creative Commons Attribution License CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/). All code contained herein is licensed under an [MIT license](https://opensource.org/licenses/MIT).

```python
import numpy as np
import pandas as pd

import bokeh.io
import bokeh.plotting

from bokeh.models import Legend
from bokeh.plotting import figure, show, output_file

bokeh.io.output_notebook()
```

    Loading BokehJS ...

## Problem 1.3 (Microtubule catastrophes)
For this exercise, we will analyze the paper by Gardner, Zanic, et al., Depolymerizing kinesins Kip3 and MCAK shape cellular microtubule architecture by differential control of catastrophe, *Cell*, 147, 1092-1103, 2011

In this paper, the authors investigated the dynamics of microtubule catastrophe, the switching of a microtubule from a growing to a shrinking state. In particular, they were interested in the time between the start of growth of a microtubule and the catastrophe event. They monitored microtubules by using tubulin (the monomer that comprises a microtubule) that was labeled with a fluorescent marker. As a control to make sure that fluorescent labels and exposure to laser light did not affect the microtubule dynamics, they performed a similar experiment using differential interference contrast (DIC) microscopy. They measured the time until catastrophe with labeled and unlabeled tubulin.

We will use their data to generate the plot which is similar to the Fig. 2a of their paper.

First, let's load the data into a Pandas `DataFrame`

```python
# Read the data from the data file into a DataFrame
df = pd.read_csv('data/gardner_et_al_2011_time_to_catastrophe_dic.csv', comment='#')

# Let's take a look
df
```

      time to catastrophe with labeled tubulin (s)
      time to catastrophe with unlabeled tubulin (s)

      0
      470
      355.0

      1
      1415
      425.0

      2
      130
      540.0

      3
      280
      265.0

      4
      550
      1815.0

      5
      65
      160.0

      6
      330
      370.0

      7
      325
      460.0

      8
      340
      190.0

      9
      95
      130.0

      10
      55
      460.0

      11
      360
      140.0

      12
      220
      295.0

      13
      225
      445.0

      14
      320
      210.0

      15
      60
      360.0

      16
      210
      575.0

      17
      155
      320.0

      18
      875
      180.0

      19
      1000
      180.0

      20
      475
      615.0

      21
      295
      145.0

      22
      245
      870.0

      23
      415
      420.0

      24
      305
      460.0

      25
      320
      265.0

      26
      370
      150.0

      27
      505
      770.0

      28
      1180
      75.0

      29
      440
      40.0

      ...
      ...
      ...

      181
      350
      NaN

      182
      155
      NaN

      183
      1095
      NaN

      184
      465
      NaN

      185
      215
      NaN

      186
      365
      NaN

      187
      605
      NaN

      188
      520
      NaN

      189
      280
      NaN

      190
      610
      NaN

      191
      765
      NaN

      192
      290
      NaN

      193
      240
      NaN

      194
      180
      NaN

      195
      225
      NaN

      196
      1030
      NaN

      197
      155
      NaN

      198
      215
      NaN

      199
      705
      NaN

      200
      285
      NaN

      201
      135
      NaN

      202
      465
      NaN

      203
      80
      NaN

      204
      370
      NaN

      205
      480
      NaN

      206
      195
      NaN

      207
      705
      NaN

      208
      300
      NaN

      209
      605
      NaN

      210
      600
      NaN

211 rows × 2 columns

The data above are not tidy. Let's remember the three rules for the tidy data:
* Each variable forms a column.
* Each observation forms a separate row.
* Each type of observational unit forms a separate table.

To tidy these data, we should drop `NaN`s. We could not use these values for analysis anyways, as we need to match control measurements (unlabeled) with the measurements of interest (labeled)

```python
# Drop NaN values
df = df.dropna()

# Let's look at it
df.head()
```

      time to catastrophe with labeled tubulin (s)
      time to catastrophe with unlabeled tubulin (s)

      0
      470
      355.0

      1
      1415
      425.0

      2
      130
      540.0

      3
      280
      265.0

      4
      550
      1815.0

Apparently, the `NaN` datapoints have been removed.

In the Fig. 2a of their paper, Gardner, Zanic, et al. have the empirical cumulative distribution function (ECDF). We will try to reconstruct this plot. First, we need to write a function `ecdf_vals(data)` which takes a one-dimensional Numpy array (or Pandas `Series`; same construction will work) of data anad returns the `x` and `y` values for plotting the ECDF. The definition of ECDF is
\begin{align}
ECDF(x) = fraction \ of \ data \ points \leq x
\end{align}

```python
def ecdf_vals(data):
    """Function returns the x and y values for the plotting of ECDF.
        Input: data (Numpy array or Pandas Series)
        Output: a pair of Numpy arrays (xaxis data and yaxis data)."""
    x = np.sort(data)
    y = np.arange(1, len(data)+1)/len(data)

    return x, y
```

```python
# First let's rename the columns to have shorter names
rename_dict = {'time to catastrophe with labeled tubulin (s)' : 'tc_lab',
               'time to catastrophe with unlabeled tubulin (s)' : 'tc_unlab'}

df = df.rename(columns=rename_dict)

# Let's look at it
df.head()
```

      tc_lab
      tc_unlab

      0
      470
      355.0

      1
      1415
      425.0

      2
      130
      540.0

      3
      280
      265.0

      4
      550
      1815.0

```python
# Select labeled tubulin data
d_lab = df['tc_lab']

# Select unlabeled tubulin data
d_unlab = df['tc_unlab']

# Apply the ECDF functions to both labeled and unlabeled data
xlab, ylab = ecdf_vals(d_lab)
xunlab, yunlab = ecdf_vals(d_unlab)

# Now use bokeh to the ECDFs on the single graph
# Set up the plot
f = bokeh.plotting.figure(plot_height=300,
                          plot_width=500,
                          x_axis_label='time to catastrophe (s)',
                          y_axis_label='ECDF')

# Add a scatter plot
f1 = f.circle(xlab, ylab, color='red')
f2 = f.circle(xunlab, yunlab, color='green')

# Make a legend object
legend = Legend(items=[
            ('labeled tubulin', [f1]),
            ('unlabeled tubulin', [f2])
            ])
# Add legend
f.add_layout(legend)
f.legend.location = 'bottom_right'

# Add axis labels
f.xaxis.axis_label = 'time to catastrophe (s)'
f.yaxis.axis_label = 'ECDF'

bokeh.io.show(f)
```

Although the ECDFs are often plotted as the scatter graphs, it is not a typical convention. Given the definition of the ECDF, it is defined for *all* values of *x* along the real *x-axis*. So, formally, the ECDF should be plotted as a line.

Now, we will write a function `plot_ecdf_formal(data)` that takes a one-dimensional Numpy array of dat and returns a Bokeh figure with the ECDF plotted as a line.

```python
def plot_ecdf_formal(data, data_legend='data', plot_height=300, plot_width=500,
                     x_axis_label='data', y_axis_label='ECDF'):
    """Returns the figure with the line plot of the ECDF data."""
    # Compute the ECDF
    x, y = ecdf_vals(data)

    # Make the Bokeh figure
    f = bokeh.plotting.figure(plot_height=plot_height,
                              plot_width=plot_width,
                              x_axis_label=x_axis_label,
                              y_axis_label=y_axis_label)

    # Make a line plot
    f1 = f.line(x,y, color='red')

    # Make a legend object
    legend = Legend(items=[
            (data_legend, [f1])
            ])
    # Add legend
    f.add_layout(legend)
    f.legend.location = 'bottom_right'

    # Add axis labels
    f.xaxis.axis_label = x_axis_label
    f.yaxis.axis_label = y_axis_label

    bokeh.io.show(f)

    return None
```

Let's try to reproduce our ECDF plot for the data for labeled tubulin.

```python
plot_ecdf_formal(d_lab, data_legend='labeled tubulin',
                 x_axis_label='time to catastrophe (s)')
```

The plot looks good!

## Bayes's Theorem and statistical inference
Let's do a quick recap of the *Bayes's theorem* and apply it to the data before we will talk about the marginalization. We start from the product rule, where we consider three events, $A$, $B$, and $C$ and their probabilities of occuring. Product rule states that
\begin{align}
P(A, B \mid C) = P(A \mid B,C) \, P(B \mid C) \quad \textbf{(product rule)}
\end{align}
This says that the probability of $A$ *and* $B$ occuring, given that $C$ happened is equal to probability of $A$ occuring given that $B$ and $C$ happened, multiplied by the probability of $B$ occuring, given that $C$ happened.

Now we can add some meaning to the events $A$, $B$, and $C$. How about this:
* $ A = H_{i} $ is the hypothesis (or parameter value) that we are testing.
* $ B = D $ is the measured data.
* $ C = I $ is all the other information we know.

Now we can rewrite the product rule as
\begin{align}
P(H_{i}, D \mid I) = P(H_{i} \mid D,I) \, P(D \mid I)
\end{align}
The *and* operation is commutative. It means that $P(H_{i}, D \mid I) = P(D, H_{i} \mid I)$. It means that we can express these equations as
\begin{align}
P(H_{i} \mid D,I) \, P(D \mid I) = P(D \mid H_{i},I) \, P(H_{i} \mid I)
\end{align}
After rearranging, we get
\begin{align}
P(H_{i} \mid D,I) = \frac{P(D \mid H_{i},I) \, P(H_{i} \mid I)}{P(D \mid I)}
\quad \textbf{(Bayes's theorem)}
\end{align}
This is exactly what we wanted - all the quantities on the right side have meaning and they describe the probability that our hypothesis is true, given the data and the other information we have. Let's assign the names to the terms in the equation and rewrite the equation as follows:
\begin{align}
posterior = \frac{likelihood \times prior}{evidence}
\end{align}

*Note: The Bayes's Theorem is a statement about the probability, which is valid for both Bayesian and frequentist interpretation of the probability.*

Let's go through each term:

* **The prior probability.** This represents the plausibility of the hypothesis $H_{i}$ given everything we know *before* we did the experiment to get the data.

* **The likelihood.** The likelihood describes how likely it is to obtain the data, *given that the hypothesis $H_{i}$ is true.* It also contains information about our expectations from data, given our measurement method. For instance, instrument noise, its model, and the other external circumstances.

* **The evidence.** It can be computed from the likelihood and prior and is also called the *marginal likelihood*.

* **The posterior probability.** And this is what we want - how plausible is the hypothesis, given that we have measured some new data? It is calculated directly from the likelihood and prior.

## Marginalization
We said that the evidence can be computed from the likelihood and the prior. To see this, let's apply the sum rule to the posterior probability:

\begin{align}
P(H_{j} \mid D,I) + P(\bar{H_{j}} \mid D, I) = 1 \\[1em]
P(H_{j} \mid D,I) + \sum_{i \neq j}{P(H_{i} \mid D,I)} = 1 \\
\sum_{i}{P(H_{i} \mid D,I)} = 1
\end{align}
for some hypothesis $H_{j}$. We can now apply the Bayes's Theorem as

\begin{align}
\sum_{i}{P(H_{i} \mid D,I)} = \sum_{i}{\frac{P(D \mid H_{i}, I) \, P(H_{i} \mid I)}{P(D \mid I)}} = 1 \\[1em]
\sum_{i}{P(H_{i} \mid D,I)} = \frac{1}{{P(D \mid I)}} \sum_{i}{P(D \mid H_{i}, I) \, P(H_{i} \mid I)} = 1 \\[1em]
\end{align}
Therefore, we can compute the *evidence* by summing over the products of likelihoods and priors:

\begin{align}
P(D \mid I) = \sum_{i}{P(D \mid H_{i}, I) \, P(H_{i} \mid I)} \\[1em]
\end{align}
This process of elimination of a variable (in this case the hypotheses) from a probability by summing is called *marginalization*.

How to grasp the marginalization? I think that we can imagine that we are trying to look through the plausible "Hypothesis space"  on all the possible hypotheses, taking into account the measured data and the prior information to assess the "influence" and the implications of each hypothesis for the observed experimental data, given the prior information.

## Probability distribution of a marginalized parameter
Let's say we have a statistical model with two continuous parameters, that is, $\theta = (\theta_{1}, \theta{2})$. A statement of Bayes's theorem in this case is
\begin{align}
P(\theta \mid D,I) = \frac{P(D \mid \theta,I) \, P(\theta \mid I)}{P(D \mid I)}
\end{align}
This can be explicitly written as
\begin{align}
P(\theta_{1}, \theta_{2} \mid D,I) = \frac{P(D \mid \theta_{1}, \theta_{2},I) \, P(\theta_{1}, \theta_{2} \mid I)}{P(D \mid I)}
\end{align}
Let's try to marginalize the statistical model by $\theta_{2}$ to obtain $P(\theta_{1} \mid D,I)$:
Considering that we have continuous hypothesis space, we can write
\begin{align}
P(\theta_{1} \mid D,I) = \int{ \frac{P(D \mid \theta_{1}, \theta_{2},I) \, P(\theta_{1}, \theta_{2} \mid I)}{P(D \mid I)} d\theta_{2}}
\end{align}

```python

```


# **Save this file as studentid1_studentid2_lab#.ipynb**
(Your student-id is the number shown on your student card.)

E.g. if you work with 3 people, the notebook should be named:
12301230_3434343_1238938934_lab1.ipynb.

**This will be parsed by a regexp, so please double check your filename.**

Before you turn this problem in, please make sure everything runs correctly. First, **restart the kernel** (in the menubar, select Kernel$\rightarrow$Restart) and then **run all cells** (in the menubar, select Cell$\rightarrow$Run All).

**Make sure you fill in any place that says `YOUR CODE HERE` or "YOUR ANSWER HERE", as well as your names and email adresses below.**

```python
NAME = "Pascal Esser"
NAME2 = "Jana Leible"
NAME3 = "Tom de Bruijn"
EMAIL = "pascal.esser@student.uva.nl"
EMAIL2 = "jana.leible@web.de"
EMAIL3 = "tommdebruijn@gmail.com"
```

---

# Lab 2: Classification

### Machine Learning 1, September 2017

Notes on implementation:

* You should write your code and answers in this IPython Notebook: http://ipython.org/notebook.html. If you have problems, please contact your teaching assistant.
* Please write your answers right below the questions.
* Among the first lines of your notebook should be "%pylab inline". This imports all required modules, and your plots will appear inline.
* Use the provided test cells to check if your answers are correct
* **Make sure your output and plots are correct before handing in your assignment with Kernel -> Restart & Run All**

$\newcommand{\bx}{\mathbf{x}}$
$\newcommand{\bw}{\mathbf{w}}$
$\newcommand{\bt}{\mathbf{t}}$
$\newcommand{\by}{\mathbf{y}}$
$\newcommand{\bm}{\mathbf{m}}$
$\newcommand{\bb}{\mathbf{b}}$
$\newcommand{\bS}{\mathbf{S}}$
$\newcommand{\ba}{\mathbf{a}}$
$\newcommand{\bz}{\mathbf{z}}$
$\newcommand{\bv}{\mathbf{v}}$
$\newcommand{\bq}{\mathbf{q}}$
$\newcommand{\bp}{\mathbf{p}}$
$\newcommand{\bh}{\mathbf{h}}$
$\newcommand{\bI}{\mathbf{I}}$
$\newcommand{\bX}{\mathbf{X}}$
$\newcommand{\bT}{\mathbf{T}}$
$\newcommand{\bPhi}{\mathbf{\Phi}}$
$\newcommand{\bW}{\mathbf{W}}$
$\newcommand{\bV}{\mathbf{V}}$

```python
%pylab inline
plt.rcParams["figure.figsize"] = [9,5]

import numpy as np

import matplotlib.pyplot as plt
```

    Populating the interactive namespace from numpy and matplotlib

# Part 1. Multiclass logistic regression

Scenario: you have a friend with one big problem: she's completely blind. You decided to help her: she has a special smartphone for blind people, and you are going to develop a mobile phone app that can do _machine vision_ using the mobile camera: converting a picture (from the camera) to the meaning of the image. You decide to start with an app that can read handwritten digits, i.e. convert an image of handwritten digits to text (e.g. it would enable her to read precious handwritten phone numbers).

A key building block for such an app would be a function `predict_digit(x)` that returns the digit class of an image patch $\bx$. Since hand-coding this function is highly non-trivial, you decide to solve this problem using machine learning, such that the internal parameters of this function are automatically learned using machine learning techniques.

The dataset you're going to use for this is the MNIST handwritten digits dataset (`http://yann.lecun.com/exdb/mnist/`). You can download the data with scikit learn, and load it as follows:

```python
from sklearn.datasets import fetch_mldata
# Fetch the data
mnist = fetch_mldata('MNIST original')
data, target = mnist.data, mnist.target.astype('int')
# Shuffle
indices = np.arange(len(data))
np.random.seed(123)
np.random.shuffle(indices)
data, target = data[indices].astype('float32'), target[indices]

# Normalize the data between 0.0 and 1.0:
data /= 255.

# Split
x_train, x_valid, x_test = data[:50000], data[50000:60000], data[60000: 70000]
t_train, t_valid, t_test = target[:50000], target[50000:60000], target[60000: 70000]
```

MNIST consists of small 28 by 28 pixel images of written digits (0-9). We split the dataset into a training, validation and testing arrays. The variables `x_train`, `x_valid` and `x_test` are $N \times M$ matrices, where $N$ is the number of datapoints in the respective set, and $M = 28^2 = 784$ is the dimensionality of the data. The second set of variables `t_train`, `t_valid` and `t_test` contain the corresponding $N$-dimensional vector of integers, containing the true class labels.

Here's a visualisation of the first 8 digits of the trainingset:

```python
def plot_digits(data, num_cols, targets=None, shape=(28,28)):
    num_digits = data.shape[0]
    num_rows = int(num_digits/num_cols)
    for i in range(num_digits):
        plt.subplot(num_rows, num_cols, i+1)
        plt.imshow(data[i].reshape(shape), interpolation='none', cmap='Greys')
        if targets is not None:
            plt.title(int(targets[i]))
        plt.colorbar()
        plt.axis('off')
    plt.tight_layout()
    plt.show()

plot_digits(x_train[0:40000:5000], num_cols=4, targets=t_train[0:40000:5000])
```

In _multiclass_ logistic regression, the conditional probability of class label $j$ given the image $\bx$ for some datapoint is given by:

$ \log p(t = j \;|\; \bx, \bb, \bW) = \log q_j - \log Z$

where $\log q_j = \bw_j^T \bx + b_j$ (the log of the unnormalized probability of the class $j$), and $Z = \sum_k q_k$ is the normalizing factor. $\bw_j$ is the $j$-th column of $\bW$ (a matrix of size $784 \times 10$) corresponding to the class label, $b_j$ is the $j$-th element of $\bb$.

Given an input image, the multiclass logistic regression model first computes the intermediate vector $\log \bq$ (of size $10 \times 1$), using $\log q_j = \bw_j^T \bx + b_j$, containing the unnormalized log-probabilities per class.

The unnormalized probabilities are then normalized by $Z$ such that $\sum_j p_j = \sum_j \exp(\log p_j) = 1$. This is done by $\log p_j = \log q_j - \log Z$ where $Z = \sum_i \exp(\log q_i)$. This is known as the _softmax_ transformation, and is also used as a last layer of many classifcation neural network models, to ensure that the output of the network is a normalized distribution, regardless of the values of second-to-last layer ($\log \bq$)

**Warning**: when computing $\log Z$, you are likely to encounter numerical problems. Save yourself countless hours of debugging and learn the [log-sum-exp trick](https://hips.seas.harvard.edu/blog/2013/01/09/computing-log-sum-exp/ "Title").

The network's output $\log \bp$ of size $10 \times 1$ then contains the conditional log-probabilities $\log p(t = j \;|\; \bx, \bb, \bW)$ for each digit class $j$. In summary, the computations are done in this order:

$\bx \rightarrow \log \bq \rightarrow Z \rightarrow \log \bp$

Given some dataset with $N$ independent, identically distributed datapoints, the log-likelihood is given by:

$ \mathcal{L}(\bb, \bW) = \sum_{n=1}^N \mathcal{L}^{(n)}$

where we use $\mathcal{L}^{(n)}$ to denote the partial log-likelihood evaluated over a single datapoint. It is important to see that the log-probability of the class label $t^{(n)}$ given the image, is given by the $t^{(n)}$-th element of the network's output $\log \bp$, denoted by $\log p_{t^{(n)}}$:

$\mathcal{L}^{(n)} = \log p(t = t^{(n)} \;|\; \bx = \bx^{(n)}, \bb, \bW) = \log p_{t^{(n)}} = \log q_{t^{(n)}} - \log Z^{(n)}$

where $\bx^{(n)}$ and $t^{(n)}$ are the input (image) and class label (integer) of the $n$-th datapoint, and $Z^{(n)}$ is the normalizing constant for the distribution over $t^{(n)}$.

## 1.1 Gradient-based stochastic optimization
### 1.1.1 Derive gradient equations (20 points)

Derive the equations for computing the (first) partial derivatives of the log-likelihood w.r.t. all the parameters, evaluated at a _single_ datapoint $n$.

You should start deriving the equations for $\frac{\partial \mathcal{L}^{(n)}}{\partial \log q_j}$ for each $j$. For clarity, we'll use the shorthand $\delta^q_j = \frac{\partial \mathcal{L}^{(n)}}{\partial \log q_j}$.

For $j = t^{(n)}$:
$
\delta^q_j
= \frac{\partial \mathcal{L}^{(n)}}{\partial \log p_j}
\frac{\partial \log p_j}{\partial \log q_j}
+ \frac{\partial \mathcal{L}^{(n)}}{\partial \log Z}
\frac{\partial \log Z}{\partial Z}
\frac{\partial Z}{\partial \log q_j}
= 1 \cdot 1 - \frac{\partial \log Z}{\partial Z}
\frac{\partial Z}{\partial \log q_j}
= 1 - \frac{\partial \log Z}{\partial Z}
\frac{\partial Z}{\partial \log q_j}
$

For $j \neq t^{(n)}$:
$
\delta^q_j
= \frac{\partial \mathcal{L}^{(n)}}{\partial \log Z}
\frac{\partial \log Z}{\partial Z}
\frac{\partial Z}{\partial \log q_j}
= - \frac{\partial \log Z}{\partial Z}
\frac{\partial Z}{\partial \log q_j}
$

Complete the above derivations for $\delta^q_j$ by furtherly developing $\frac{\partial \log Z}{\partial Z}$ and $\frac{\partial Z}{\partial \log q_j}$. Both are quite simple. For these it doesn't matter whether $j = t^{(n)}$ or not.

$
\frac{\partial\log Z}{\partial Z} = \frac{1}{Z}
$
$
\frac{\partial Z}{\partial \log q_i }=\frac{\partial\sum_k q_k}{\partial \log q_i}= \frac{p_i}{\partial \log q_i}=\frac{\partial\exp(\log q_i)}{\partial \log q_j} = \exp(\log q_i)
$

For $j = t^{(n)}$:
\begin{align}
\delta^q_j
&=1-\frac{1}{Z}\exp(\log q_i)
\end{align}
For $j \neq t^{(n)}$:
\begin{align}
\delta^q_j
&= -\frac{1}{Z}\exp(\log q_i)
\end{align}

Given your equations for computing the gradients $\delta^q_j$ it should be quite straightforward to derive the equations for the gradients of the parameters of the model, $\frac{\partial \mathcal{L}^{(n)}}{\partial W_{ij}}$ and $\frac{\partial \mathcal{L}^{(n)}}{\partial b_j}$. The gradients for the biases $\bb$ are given by:

$
\frac{\partial \mathcal{L}^{(n)}}{\partial b_j}
= \frac{\partial \mathcal{L}^{(n)}}{\partial \log q_j}
\frac{\partial \log q_j}{\partial b_j}
= \delta^q_j
\cdot 1
= \delta^q_j
$

The equation above gives the derivative of $\mathcal{L}^{(n)}$ w.r.t. a single element of $\bb$, so the vector $\nabla_\bb \mathcal{L}^{(n)}$ with all derivatives of $\mathcal{L}^{(n)}$ w.r.t. the bias parameters $\bb$ is:

$
\nabla_\bb \mathcal{L}^{(n)} = \mathbf{\delta}^q
$

where $\mathbf{\delta}^q$ denotes the vector of size $10 \times 1$ with elements $\mathbf{\delta}_j^q$.

The (not fully developed) equation for computing the derivative of $\mathcal{L}^{(n)}$ w.r.t. a single element $W_{ij}$ of $\bW$ is:

$
\frac{\partial \mathcal{L}^{(n)}}{\partial W_{ij}} =
\frac{\partial \mathcal{L}^{(n)}}{\partial \log q_j}
\frac{\partial \log q_j}{\partial W_{ij}}
= \mathbf{\delta}_j^q
\frac{\partial \log q_j}{\partial W_{ij}}
$

What is $\frac{\partial \log q_j}{\partial W_{ij}}$? Complete the equation above.

If you want, you can give the resulting equation in vector format ($\nabla_{\bw_j} \mathcal{L}^{(n)} = ...$), like we did for $\nabla_\bb \mathcal{L}^{(n)}$.

\begin{equation}
\log q_i = \bm w_i^T \bm x = b_i +\sum \bm  W_{ij} \bm x_i
\end{equation}
\begin{equation}
\frac{\partial\log q_i}{\partial \bm W_{ij}} =\bm x_i
\end{equation}

\begin{equation}
\frac{\partial\mathcal{L}^{(n)}}{\partial \bm W_{ij}} =\delta^q_j\bm x_i
\end{equation}

### 1.1.2 Implement gradient computations (10 points)

Implement the gradient calculations you derived in the previous question. Write a function `logreg_gradient(x, t, w, b)` that returns the gradients $\nabla_{\bw_j} \mathcal{L}^{(n)}$ (for each $j$) and $\nabla_{\bb} \mathcal{L}^{(n)}$, i.e. the first partial derivatives of the log-likelihood w.r.t. the parameters $\bW$ and $\bb$, evaluated at a single datapoint (`x`, `t`).
The computation will contain roughly the following intermediate variables:

$
\log \bq \rightarrow Z \rightarrow \log \bp\,,\, \mathbf{\delta}^q
$

followed by computation of the gradient vectors $\nabla_{\bw_j} \mathcal{L}^{(n)}$ (contained in a $784 \times 10$ matrix) and $\nabla_{\bb} \mathcal{L}^{(n)}$ (a $10 \times 1$ vector).

For maximum points, ensure the function is numerically stable.

```python
# 1.1.2 Compute gradient of log p(t|x;w,b) wrt w and b
def logreg_gradient(x, t, w, b):

    log_q = w.T.dot(x.T).squeeze() + b

    # calculate 'a' from the log sum trick
    a = np.max(log_q)
    # log sum trick
    log_Z = a + np.log(np.sum(np.exp(log_q - a)))
    Z = np.exp(log_Z)

    logp = log_q - log_Z

    # compute derivertives
    dL_db = -np.exp(log_q) / Z
    dL_db[t] += 1
    dL_dw = dL_db[:, np.newaxis].dot(x).squeeze().T

    return logp[t].squeeze(), dL_dw, dL_db.squeeze()
```

```python
np.random.seed(123)
# scalar, 10 X 768  matrix, 10 X 1 vector
w = np.random.normal(size=(28*28,10), scale=0.001)
# w = np.zeros((784,10))
b = np.zeros((10,))

# test gradients, train on 1 sample
logpt, grad_w, grad_b = logreg_gradient(x_train[0:1,:], t_train[0:1], w, b)

print("Test gradient on one point")
print("Likelihood:\t", logpt)
print("\nGrad_W_ij\t",grad_w.shape,"matrix")
print("Grad_W_ij[0,152:158]=\t", grad_w[152:158,0])
print("\nGrad_B_i shape\t",grad_b.shape,"vector")
print("Grad_B_i=\t", grad_b.T)
print("i in {0,...,9}; j in M")

assert logpt.shape == (), logpt.shape
assert grad_w.shape == (784, 10), grad_w.shape
assert grad_b.shape == (10,), grad_b.shape

```

    Test gradient on one point
    Likelihood:	 -2.2959726720744777

    Grad_W_ij	 (784, 10) matrix
    Grad_W_ij[0,152:158]=	 [-0.04518971 -0.06758809 -0.07819784 -0.09077237 -0.07584012 -0.06365855]

    Grad_B_i shape	 (10,) vector
    Grad_B_i=	 [-0.10020327 -0.09977827 -0.1003198   0.89933657 -0.10037941 -0.10072863
     -0.09982729 -0.09928672 -0.09949324 -0.09931994]
    i in {0,...,9}; j in M

```python
# It's always good to check your gradient implementations with finite difference checking:
# Scipy provides the check_grad function, which requires flat input variables.
# So we write two helper functions that provide can compute the gradient and output with 'flat' weights:
from scipy.optimize import check_grad

np.random.seed(123)
# scalar, 10 X 768  matrix, 10 X 1 vector
w = np.random.normal(size=(28*28,10), scale=0.001)
# w = np.zeros((784,10))
b = np.zeros((10,))

def func(w):
    logpt, grad_w, grad_b = logreg_gradient(x_train[0:1,:], t_train[0:1], w.reshape(784,10), b)
    return logpt
def grad(w):
    logpt, grad_w, grad_b = logreg_gradient(x_train[0:1,:], t_train[0:1], w.reshape(784,10), b)
    return grad_w.flatten()
finite_diff_error = check_grad(func, grad, w.flatten())
print('Finite difference error grad_w:', finite_diff_error)
assert finite_diff_error < 1e-3, 'Your gradient computation for w seems off'

def func(b):
    logpt, grad_w, grad_b = logreg_gradient(x_train[0:1,:], t_train[0:1], w, b)
    return logpt
def grad(b):
    logpt, grad_w, grad_b = logreg_gradient(x_train[0:1,:], t_train[0:1], w, b)
    return grad_b.flatten()
finite_diff_error = check_grad(func, grad, b)
print('Finite difference error grad_b:', finite_diff_error)
assert finite_diff_error < 1e-3, 'Your gradient computation for b seems off'

```

    Finite difference error grad_w: 6.3612946893e-07
    Finite difference error grad_b: 5.23511748609e-08

```python

```

### 1.1.3 Stochastic gradient descent (10 points)

Write a function `sgd_iter(x_train, t_train, w, b)` that performs one iteration of stochastic gradient descent (SGD), and returns the new weights. It should go through the trainingset once in randomized order, call `logreg_gradient(x, t, w, b)` for each datapoint to get the gradients, and update the parameters **using a small learning rate of `1E-6`**. Note that in this case we're maximizing the likelihood function, so we should actually performing gradient ___ascent___... For more information about SGD, see Bishop 5.2.4 or an online source (i.e. https://en.wikipedia.org/wiki/Stochastic_gradient_descent)

```python
def sgd_iter(x_train, t_train, W, b):
    # create a shuffled list of indecies of the same
    # length as the training set
    index_list = np.arange(len(x_train) - 1)
    np.random.shuffle(index_list)
    # define learning rate:
    learning_rate = 10 ** -6

    # initialize the sum
    logp_train_sum = 0

    for i in index_list:
        logp_train, grad_w, grad_b = logreg_gradient(x_train[i:i + 1, :],
                                                     t_train[i:i + 1], W, b)
        # add logp to sum
        logp_train_sum += logp_train
        # update w and b with SGD
        W += learning_rate * grad_w
        b += learning_rate * grad_b

    return logp_train_sum, W, b

```

```python
# Sanity check:
np.random.seed(1243)
w = np.zeros((28*28, 10))
b = np.zeros(10)

logp_train, W, b = sgd_iter(x_train[:5], t_train[:5], w, b)

```

## 1.2. Train

### 1.2.1 Train (10 points)
Perform 10 SGD iterations through the trainingset. Plot (in one graph) the conditional log-probability of the trainingset and validation set after each iteration.

```python
def test_sgd(x_train, t_train, w, b):
    # lists for log probabilities for each iteration
    log_prob_train = []
    log_prob_valid = []
    # run over 10 iterations
    for _ in range(10):
        logp_valid_sum = 0
        # call SGD
        logp_train, w, b = sgd_iter(x_train, t_train, w, b)
        log_prob_train.append(logp_train)
        # apply new w and b on the validation set
        for i in range(10000):
            logpt, _, _ = logreg_gradient(x_valid[i:i + 1, :],
                                          t_valid[i:i + 1], w, b)
            logp_valid_sum += logpt
        log_prob_valid.append(logp_valid_sum)

    return w, b, log_prob_train, log_prob_valid

np.random.seed(1243)
w = np.zeros((28 * 28, 10))
b = np.zeros(10)
w, b, log_prob_train_out, log_prob_valid_out = test_sgd(x_train, t_train, w, b)

```

```python
# put plotting in extra cell, so you dont have to run the
# SGD for fine tuning the plots

x_axis = range(len(log_prob_train_out))

# Normalize the training and validation set by deviding
# it by the number of elements.
log_prob_train_out_scale = np.array(log_prob_train_out) / 50000
log_prob_valid_out_scale = np.array(log_prob_valid_out) / 10000

plt.plot(x_axis, log_prob_train_out_scale, label='Training Set')
plt.plot(x_axis, log_prob_valid_out_scale, label='Validation Set')

plt.scatter(x_axis, log_prob_train_out_scale)
plt.scatter(x_axis, log_prob_valid_out_scale)

plt.legend(loc='upper left')

plt.show()

```

### 1.2.2 Visualize weights (10 points)
Visualize the resulting parameters $\bW$ after a few iterations through the training set, by treating each column of $\bW$ as an image. If you want, you can use or edit the `plot_digits(...)` above.

```python
# plot weights
plot_digits(w.T, num_cols=5)

```

**Describe in less than 100 words why these weights minimize the loss**

weights into: a medium-gray one (0 values), a black one (high positive values) and a white one (high negative values).
* Gray: 0-weights are assigned to the parts of the input images that do not add any information, namely those that are the same in all input images
* Black: For weight-vector $w_j$ high positive weights are assigned to those pixels that, if black, increase the probability of the image being labeled $j$
White: With the same reasoning high negative weights are assigned to those pixels that, if black, decrease the probability of the image being labeled $j$

### 1.2.3. Visualize the 8 hardest and 8 easiest digits (10 points)
Visualize the 8 digits in the validation set with the highest probability of the true class label under the model.
Also plot the 8 digits that were assigned the lowest probability.
Ask yourself if these results make sense.

```python
# redefine because of the change w and b values

# calculate log_q for the validation set
log_q = x_valid.dot(w) + b
index_list = []

# create list of validation targets in numbers and index
for (index,), number in np.ndenumerate(t_valid):
    index_list.append(log_q[index, number])

# Sort after log_q
valid_sorted = np.argsort(np.array(index_list))

# easiest are the first 8 in the list
easy = valid_sorted[-8:]
print('Easiest 8 digits')
plot_digits(x_valid[easy], num_cols=4)

# hardest are the last 8 in the list
print('Hardest 8 digits')
hard = valid_sorted[:8]
plot_digits(x_valid[hard], num_cols=4)

```

Intuitively, easy digits are the ones that are not easily confused with other digits. As the zero is very different from the other numbers - and everyone tends to write zeros the same way -, it makes sense that it is a digit that is easy to classify.
On the other hand, hard digits are the ones that are ambiguous or come in a number of different shapes (7 with and without a horitontal bar, 1 with and without a serif) and thus easily confused with others. For the human eye this might not be the case for all hard digits (for example the 5 and the 9 are easy to identify) but for example the last digit could easily be a 4 or a 6.

# Part 2. Multilayer perceptron

You discover that the predictions by the logistic regression classifier are not good enough for your application: the model is too simple. You want to increase the accuracy of your predictions by using a better model. For this purpose, you're going to use a multilayer perceptron (MLP), a simple kind of neural network. The perceptron wil have a single hidden layer $\bh$ with $L$ elements. The parameters of the model are $\bV$ (connections between input $\bx$ and hidden layer $\bh$), $\ba$ (the biases/intercepts of $\bh$), $\bW$ (connections between $\bh$ and $\log q$) and $\bb$ (the biases/intercepts of $\log q$.

The conditional probability of the class label $j$ is given by:

$\log p(t = j \;|\; \bx, \bb, \bW) = \log q_j - \log Z$

where $q_j$ are again the unnormalized probabilities per class, and $Z = \sum_j q_j$ is again the probability normalizing factor. Each $q_j$ is computed using:

$\log q_j = \bw_j^T \bh + b_j$

where $\bh$ is a $L \times 1$ vector with the hidden layer activations (of a hidden layer with size $L$), and $\bw_j$ is the $j$-th column of $\bW$ (a $L \times 10$ matrix). Each element of the hidden layer is computed from the input vector $\bx$ using:

$h_j = \sigma(\bv_j^T \bx + a_j)$

where $\bv_j$ is the $j$-th column of $\bV$ (a $784 \times L$ matrix), $a_j$ is the $j$-th element of $\ba$, and $\sigma(.)$ is the so-called sigmoid activation function, defined by:

$\sigma(x) = \frac{1}{1 + \exp(-x)}$

Note that this model is almost equal to the multiclass logistic regression model, but with an extra 'hidden layer' $\bh$. The activations of this hidden layer can be viewed as features computed from the input, where the feature transformation ($\bV$ and $\ba$) is learned.

## 2.1 Derive gradient equations (20 points)

State (shortly) why $\nabla_{\bb} \mathcal{L}^{(n)}$ is equal to the earlier (multiclass logistic regression) case, and why $\nabla_{\bw_j} \mathcal{L}^{(n)}$ is almost equal to the earlier case.

Like in multiclass logistic regression, you should use intermediate variables $\mathbf{\delta}_j^q$. In addition, you should use intermediate variables $\mathbf{\delta}_j^h = \frac{\partial \mathcal{L}^{(n)}}{\partial h_j}$.

Given an input image, roughly the following intermediate variables should be computed:

$
\log \bq \rightarrow Z \rightarrow \log \bp \rightarrow \mathbf{\delta}^q \rightarrow \mathbf{\delta}^h
$

where $\mathbf{\delta}_j^h = \frac{\partial \mathcal{L}^{(n)}}{\partial \bh_j}$.

Give the equations for computing $\mathbf{\delta}^h$, and for computing the derivatives of $\mathcal{L}^{(n)}$ w.r.t. $\bW$, $\bb$, $\bV$ and $\ba$.

You can use the convenient fact that $\frac{\partial}{\partial x} \sigma(x) = \sigma(x) (1 - \sigma(x))$.

\begin{align*}
	\delta_j^q &= \frac{\partial\mathcal{L}^{(n)}}{\partial\ln(q_j)} \\
	&= \begin{cases}
		j = t^{(n)}: 1 - \frac{1}{Z} \exp{} (\ln{} (q_j)) \\
		j \neq t^{(n)}: - \frac{1}{Z} \exp{} (\ln{} (q_j))
	\end{cases}
\end{align*}
\begin{align*}
	\delta_j^h &=
		\frac{\partial\mathcal{L}^{(n)}}{\partial{} h_i}
		= \frac{\partial\mathcal{L}^{(n)}}{\partial\ln{}(q_i)}
		\frac{\partial\ln{}(q_i)}{\partial{} h_i}
		= \delta_j^qW_{ij}
\end{align*}
\begin{align*}
	\frac{\partial\mathcal{L}^{(n)}}{\partial{} b_j}
		&= \frac{\partial\mathcal{L}^{(n)}}{\partial\ln{}(q_i)}
		\frac{\partial\ln{}(q_i)}{\partial{} b_j} = \delta_j^q
\end{align*}
\begin{align*}
	\frac{\partial\mathcal{L}^{(n)}}{\partial{} W_{ij}}
		&= \frac{\partial\mathcal{L}^{(n)}}{\partial\ln{}(q_i)}
		\frac{\partial\ln{}(q_i)}{\partial{} W_{ij}} = \delta_j^qh_i
\end{align*}
\begin{align*}
	\frac{\partial\mathcal{L}^{(n)}}{\partial{} a_i}
		&= \frac{\partial\mathcal{L}^{(n)}}{\partial{}(h_i)}
		\frac{\partial{}(h_i)}{\partial{} v_i^Tx+a_i} \frac{\partial{} v_i^tx+a_i}{\partial{} a_i}\\
		&=\delta_j^h \sigma (v_i^Tx + a_i)(1 - \sigma (v_i^Tx + a_i)) = \delta_j^hh_i(1-h_i)
\end{align*}
\begin{align*}
	\frac{\partial\mathcal{L}^{(n)}}{\partial v_i}
	&= \frac{\partial\mathcal{L}^{(n)}}{\partial{}(h_i)}
		\frac{\partial{}(h_i)}{\partial{} v_i^Tx+a} \frac{\partial{} v_i^tx+a}{\partial{} v_i^T}\\
		&= \delta_j^hh_i(1-h_i)x_h
\end{align*}

## 2.2 MAP optimization (10 points)

You derived equations for finding the _maximum likelihood_ solution of the parameters. Explain, in a few sentences, how you could extend this approach so that it optimizes towards a _maximum a posteriori_ (MAP) solution of the parameters, with a Gaussian prior on the parameters.

The maximum likelihood solution we found above results from deriving the log-likelihood with respect to $\ln(q_j)$. In order to obtain the MAP solution we'd have to multiply the likelihood with the prior prior to deriving. Since we're deriving the log-likelihood multiplying with the prior becomes equivalent to adding the log-prior. Deriving that sum is the same as adding the log-prior derived w.r.t. $\ln(q_j)$ to the ML-solution for $\delta^q_j$.
This could be implemented as:

\begin{align*}
	\mathcal{L}^{(n)} &= \ln(q_j) - \ln(Z) - \frac{\alpha}{2}(V_{hi}-W_{ij})^2\\
	&= \ln(q_j) - \ln(Z) - \frac{\alpha_1}{2}\sum^{784}_{h=1} \sum_{i=0}^{\mathcal{L}} V_{hi}^2 - \frac{\alpha_2}{2}\sum_{i=0}^{\mathcal{L}} \sum_{j=1}^{10}W_{ij}^2
\end{align*}

Leading to:

\begin{align*}
	\frac{\partial\mathcal{L}^{(n)}}{\partial{} W_{ij}}
		&= \delta_j^qh_i - \alpha_2 W_{ij}
\end{align*}
\begin{align*}
	\frac{\partial\mathcal{L}^{(n)}}{\partial v_hi}
	&= \delta_j^hh_i(1-h_i)x_h -\alpha_1 v_{hi}
\end{align*}

with all other derivertives staying the same.

## 2.3. Implement and train a MLP (15 points)

Implement a MLP model with a single hidden layer of **20 neurons**.
Train the model for **10 epochs**.
Plot (in one graph) the conditional log-probability of the trainingset and validation set after each two iterations, as well as the weights.

- 10 points: Working MLP that learns with plots
- +5 points: Fast, numerically stable, vectorized implementation

```python
import numpy as np

np.random.seed(123)

# initialize weights, so that we dont get a
# bias by testing for differnt learning rates
def init_weights():
    # initialize variables
    w = (np.random.random_sample((20, 10)) - 0.5) * 0.001
    b = (np.random.random_sample(10) - 0.5) * 0.001
    v = (np.random.random_sample((28 * 28, 20)) - 0.5) * 0.001
    a = (np.random.random_sample(20) - 0.5) * 0.001
    return w, b, v, a

# Calculate all gradients
def MLP_gradient(x, t, w, b, v, a):
    h = 1 / (1 + np.exp(-x.dot(v).squeeze() - a))

    log_q = h.dot(w) + b
    # calculate 'alpha' from the log sum trick
    alpha = np.max(log_q)
    # log sum trick
    log_Z = alpha + np.log(np.sum(np.exp(log_q - alpha)))
    Z = np.exp(log_Z)

    logp = log_q - log_Z

    # calculate derivertives
    dL_db = -np.exp(log_q) / Z
    dL_db[t] += 1
    dL_dw = h[np.newaxis].T.dot(dL_db[np.newaxis])
    dL_dh = w[:, t].dot(dL_db[t])
    dL_da = dL_dh * h * (1 - h)
    dL_dv = x.T.dot(dL_da[np.newaxis])

    return logp[t].squeeze(), dL_dw, dL_db, dL_dv, dL_da

# update  w, b, v, a with SGD
def sgd_iter_MLP(x_train, t_train, w, b, v, a, learning_rate):
    # create a shuffled list of indecies of the same
    # length as the training set
    index_list = np.arange(len(x_train) - 1)
    np.random.shuffle(index_list)
    # define learning rate:
    learning_rate_ = learning_rate

    logp_train_sum = 0

    for i in index_list:
        logp_train, grad_w, grad_b, grad_v, grad_a = MLP_gradient(
            x_train[i:i + 1, :],
            t_train[i:i + 1], w, b, v, a)
        logp_train_sum += logp_train
        w += learning_rate_ * grad_w
        b += learning_rate_ * grad_b
        v += learning_rate_ * grad_v
        a += learning_rate_ * grad_a

    return logp_train_sum, w, b, v, a

# optimize over 10 iterations
def test_sgd_MLP(x_train, t_train, w, b, v, a, learning_rate):
    log_prob_train = []
    log_prob_valid = []
    for _ in range(10):
        logp_valid_sum = 0
        logp_train, w, b, v, a = sgd_iter_MLP(x_train, t_train, w, b, v, a,
                                              learning_rate)
        log_prob_train.append(logp_train)
        for i in range(10000):
            logp_temp, _, _, _, _ = MLP_gradient(x_valid[i:i + 1, :],
                                                 t_valid[i:i + 1], w, b, v, a)
            logp_valid_sum += logp_temp
        log_prob_valid.append(logp_valid_sum)

    return w, b, v, a, log_prob_train, log_prob_valid

# function for testing the model
def test_model(w, b, v, a, x_valid):
    h = 1 / (1 + np.exp(-x_valid.dot(v) - a))
    log_q = h.dot(w) + b
    predict = np.argmax(log_q, axis=1)
    correct = t_valid == predict
    accuracy = len(correct[correct == True]) / float(len(t_valid))

    return accuracy

```

```python

# Learn and test the model
log_prob_train_out_, log_prob_valid_out_ = [], []
accuracy_ = 0
v_ = 0

# run over differtn learning rates
for learning_rate in [10 ** -1, 10 ** -2, 10 ** -3, 10 ** -4]:
    # initialize weights
    w, b, v, a = init_weights()
    # calculate MLP with SGD
    w, b, v, a, log_prob_train_out, log_prob_valid_out = test_sgd_MLP(x_train,
                                                                      t_train,
                                                                      w, b, v,
                                                                      a,
                                                                      learning_rate)
    # calculate accuracy
    accuracy = test_model(w, b, v, a, x_valid)
    print('Accuracy: {}     | for learning rate: {}'.format(accuracy,
                                                          learning_rate))
    # update optimal values
    if accuracy > accuracy_:
        accuracy_ = accuracy
        log_prob_train_out_, log_prob_valid_out_ = log_prob_train_out, log_prob_valid_out
        v_ = v

```

    Accuracy: 0.8995     | for learning rate: 0.1
    Accuracy: 0.9269     | for learning rate: 0.01
    Accuracy: 0.8951     | for learning rate: 0.001
    Accuracy: 0.2235     | for learning rate: 0.0001

```python
# put plotting in extra cell, so you dont have to run the
# SGD for fine tuning the plots
#print(w.shape, b.shape, v.shape, a.shape)

x_axis = range(len(log_prob_train_out))

log_prob_train_out_scale = np.array(log_prob_train_out_) / 50000
log_prob_valid_out_scale = np.array(log_prob_valid_out_) / 10000

plt.plot(log_prob_train_out_scale)
plt.plot(log_prob_valid_out_scale)
#print(log_prob_train_out)
plt.scatter(x_axis, log_prob_train_out_scale)
plt.scatter(x_axis, log_prob_valid_out_scale)

plt.show()

#plot_digits(, num_cols=5)
plot_digits(v_.T, num_cols=5, targets=None, shape=(28, 28))

```

### 2.3.1. Explain the weights (5 points)
In less than 80 words, explain how and why the weights of the hidden layer of the MLP differ from the logistic regression model, and relate this to the stronger performance of the MLP.

In this model, each number is modelled out of a combination or different weightvectors and no longer by one single weightvector. The weights are now combinations of areas, with higher probabilities and lower probabilities. This means, that the weights now no longer model a whole number, but specific edges (dark areas) or explicitly areas where it is unlikely, that something is written (white / grey). This is more powerful because it allows for a exacter and more flexible digit definition.

### 2.3.1. Less than 250 misclassifications on the test set (10 bonus points)

You receive an additional 10 bonus points if you manage to train a model with very high accuracy: at most 2.5% misclasified digits on the test set. Note that the test set contains 10000 digits, so you model should misclassify at most 250 digits. This should be achievable with a MLP model with one hidden layer. See results of various models at : `http://yann.lecun.com/exdb/mnist/index.html`. To reach such a low accuracy, you probably need to have a very high $L$ (many hidden units), probably $L > 200$, and apply a strong Gaussian prior on the weights. In this case you are allowed to use the validation set for training.
You are allowed to add additional layers, and use convolutional networks, although that is probably not required to reach 2.5% misclassifications.

```python
predict_test = np.zeros(len(t_test))
# Fill predict_test with the predicted targets from your model, don't cheat :-).
# YOUR CODE HERE
raise NotImplementedError()
```

```python
assert predict_test.shape == t_test.shape
n_errors = np.sum(predict_test != t_test)
print('Test errors: %d' % n_errors)
```