```
import torch, os, ipywidgets, json
import torch.nn as nn
import numpy as np
import pandas as pd
import seaborn as sns
import torch.nn.functional as F
import torchvision as tv
import matplotlib.pyplot as plt
from google.colab import drive
from google.colab import widgets as clwidgets
from tqdm import tqdm_notebook as tqdm
from torchvision.datasets import MNIST
from torch.utils.data import DataLoader
PI = torch.tensor(np.pi, dtype=torch.float32)
sns.set_style('whitegrid')
%matplotlib inline
```

```
save_path = '/content/gdrive/My Drive/Tutorials/InfoGAN/MNIST/'
drive.mount('/content/gdrive', force_remount=True)
os.makedirs(os.path.join(save_path, 'ckpt'), exist_ok=True)
```

# A quick tour on Information Maximising Generative Adversarial Networks (InfoGAN)
*Yen Yu* $\diamond$ *yen.yu@araya.org* $\diamond$ *yen.yu.10@ucl.ac.uk*

*17/Jun/2019*

## Background
* This Notebook traces my steps from my only knowing about GAN on a conceptual level to actually trying to learn the theory behind by implementing it.
* I intend this Notebook to be a tutorial accessible to those who wants to learn more about geneartive modelling (there is a plan to cover other GANs and VAE+Normalising Flows).

## Goal
* The goal of this Notebook is to to provide theoretical details of GAN just enough to motivate an implementation.
* The main focus will be on the implementation of InfoGAN
* To successfully train an InfoGAN on the MNIST dataset.

## References
[1] The origin GAN paper by Goodfellow

[2] The InfoGAN paper by Xi Chen

[3] Ferenc Huszar's [blog post](https://www.inference.vc/infogan-variational-bound-on-mutual-information-twice/)—very insightful

[4] [Useful GAN hacks (multiple authors)](https://github.com/soumith/ganhacks#authors)

## Classic GAN
### Theory
#### Layperson example

You are visiting a friend who is working in the packaging department of a factory. His/her job is to load the products onto a conveyer belt to initiate automatic packaging. You noticed, from the control room, that the conveyer belt has a layout resembling the shape of the letter Y. Products are loaded onto the branches on the top end and get merged into a single line at the junction in the middle. The packaging mechnism is located at the bottom end. An electronic gate at the junction controls which branch gets to pass. Whenever the indicator light on the gate shines red the left branch feeds into the bottom branch. And when a green light is shone, the opposite is the case.

You know the factory has a long-term partnership with a particular supplier who provides the standard products for this line. When the supply is short, the factory seeks additional supply from another supplier but thier quality can be a hit and miss.

Today, both branches are operating but you have no idea whether there has been a supply shortage. If there is a shortage, one of the branches must be carrying products of "somewhat questionable quality".

From the control room terminal, you see the gate camera view, along with its indicator light. Out of curiosity, you decide to challenge yourself to answer whether there *is* a shortage by finding out whether different lights signal a discernable impression on the product quality.

This time, you are positive because the non-standard products have a grey-ish tint.

Hopefully, on your next visit, knowing that there is still a shortage going on, you will find yourself having a hard time arriving at a definite conclusion. This can almost mean the additional supplier has boosted their quality to be on a par with the standard.

#### Remarks

* A Generative Adversarial Network (GAN) typically consists of a generator and a discriminator, both as trainable neural networks. The generator tries to generate samples to fool the discriminator, making it believe the generated samples are as good as the real ones. And the discriminator tries to unfool itself.
* The standard supplier provides the *real* data: $x_{real}$.
* The second supplier, which is the generator, provides the *fake* data: $x_{fake}$
* You, the discriminator, serve to differentiate the real from the fake. But you also help the fake fake better.
* The role of the indicator light $d$ will become clear in the following part.
* The layperson example lends itself to an information-theoretical interpretation as described below.

#### An information-theoretic perspective

Let us consider two random variables $x_{real}$ and $x_{fake}$, one representing data coming from a *real* distribution and another trying to mimic it. And $d \sim \mathrm{Bern}(0.5)$ being a mixing factor allows the following interaction:

$$
\begin{align}
x &= \left\{
    \begin{array}{l}
    x_{real},\; \hbox{ if } d=1 \\
    x_{fake},\; \hbox{ if } d = 0
    \end{array}
    \right.
\end{align}
$$

The task for a classic GAN is to make $x_{fake}$ as real as possible such that knowing $d$ provides minimal, if not none, insights into the actual identity of $x$. In the language of information theory, this is equivalent to saying

$$\min I(x : d)$$

The learning rule of the classic GAN can be derived from here by considering:

$$
\begin{align}
I(x : d) &= H(d) - H(d | x)  \\
    &= H(d) + \mathbb E_x \mathbb E_{d|x} \log p(d|x)  \\
    &= H(d) + \mathbb E_{d,x} \frac {\log p(d|x)} {\log q(d|x)} \log q(d|x)  \\
    &\ge H(d) + \mathbb E_{d,x} \log q(d|x)
\end{align}
$$

Here, an auxiliary density $q_{d|x}$ is introduced to create a lower bound on the mutual information. This bound is tight when $p(d|x) = q(d|x)$. Then the following holds:

$$
\begin{align}
I(x:d) &= H(d) + \max_{q_{d|x}} \mathbb E_{d,x} \log q(d|x)  \\
    &\ge H(d) + \max_\varphi \mathbb E_{d,x} \log q(d|x; \varphi)
\end{align}
$$

where the last line turns $q_{d|x}$ into a parametric family by assumption.

Then, one can expand the expectation to rewrite the expression since we knew $q$ is a coin-flip:

$$
\begin{align}
I(x:d) &\ge H(d) + \max_\varphi \left(
  \mathbb E_{x_{real}} \log q(1 | x_{real};\varphi) +
  \mathbb E_{x_{fake}} \log q(0 | x_{fake}; \varphi)
  \right)
\end{align}
$$

### Implementation
Following the derivation immediately above, it becomes clear that we can treat the approximate density $q(d=\{0, 1\} | x; \varphi)$ as a deep neural network that works as a discriminator ($\mathcal D(x; \varphi)$) . Specifically, we want this D network to take as input, say, an image tensor and output a scalar between 0 and 1. This D network should be able to tell $x$ apart by learning to assign 1 to $x_{real}$ and 0 to $x_{fake}$:
$$
\begin{align}
&= H(d) + \max_\varphi \left(
  \mathbb E_{x_{real}} \log \mathcal D(x_{real};\varphi) +
  \mathbb E_{x_{fake}} \log [1 - \mathcal D(x_{fake}; \varphi)]
  \right)
\end{align}
$$
On the other hand, $x_{fake}$ is something we have to create from nothing (well, from noise, to be exact). And we can assign this task to another neural network that we'd like to call the generator $\mathcal G(z; \vartheta)$, where $z$ is the noise vector. By replaceing $x_{fake}$ with $\mathcal D(z; \vartheta)$ we sidestep the need to actually evaluate the expectation $\mathbb E_{x_{fake}}$. Instead, we are now dealing with a much simpler $\mathbb E_z$, whcih can be Monte Carlo approximated:

$$
\begin{align}
&=H(d) + \max_\varphi \left(
  \mathbb E_{x_{real}} \log \mathcal D(x_{real};\varphi) +
  \mathbb E_{z} \log [1 - \mathcal D(\mathcal G(z; \vartheta); \varphi)]
  \right)
\end{align}
$$

where $\vartheta$ parametrises the generator network $\mathcal G$ and $z \sim N(0,1)$ is some Gaussian noise conventionally used by GAN.

The G network has to learn to fool the D network as the D network tries to tell the real and fake apart. This brings us back to our initial objective of minimising the mutual information. We can now write down the classic GAN objective:

$$
\begin{align}
\min_\vartheta \max_\varphi \left(
  \mathbb E_{x_{real}} \log \mathcal D(x_{real};\varphi) +
  \mathbb E_{z} \log [1 - \mathcal D(\mathcal G(z; \vartheta); \varphi)]
  \right)
\end{align}
$$

#### Task
To implement a classic GAN, we need to program the following:
* A discriminator network $\mathcal D$ that maps an input to a real number between 0 and 1 (as representing probability)
* A generator $\mathcal G$ that takes some noise vector of arbitrary dimension
* Write down the objective function

```
%%script false
# This is only a demonstration cell

# Taks 1: Program a discriminator network
class Discriminator(nn.Module):
    """A Discriminator network represents a function
    that maps an input to a scalar between (0, 1)."""
    def __init__(self):
        super(Discriminator, self).__init__()

        # The following contains the main component of the
        # desired function. It can take any valid architecture.
        # But for the sake of demonstration, I will only include
        # minimal numbers of network layers, assuming the inputs
        # are mini-batches of MNIST images which are monochromatic,
        # 28x28 images.
        self.layers = nn.Sequential(
            # NB. setting bias=False only because BN is used. The beta parameter
            # in a BN works just like a bias.
            nn.Conv2d(1, 64, (3, 3), stride=(2, 2), padding=1, bias=False),
            nn.BatchNorm2d(64),
            nn.LeakyReLU(0.1),
            nn.Conv2d(64, 128, (3, 3), stride=(2, 2), padding=1, bias=False),
            nn.BatchNorm2d(128),
            nn.LeakyReLU(0.1),
            nn.Conv2d(128, 1024, (7, 7), stride=(1, 1), padding=0, bias=False),
            nn.BatchNorm2d(1024)
            nn.LeakyReLU(0.1),
            nn.Conv2d(1024, 1, (1, 1), stride=(1, 1), padding=0, bias=True),
            nn.Sigmoid()
        )

    def forward(self, x):
        return self.layers(x)

# Task 2: Program a generator network
class Generator(nn.Module):
    """A Generator is a function that maps a latent code (usually a noise vector) of
    arbitrary dimension (here 74 is used) to a tensor that mimics real dataset."""
    def __init__(self):
        super(Generator, self).__init__()

        self.layers = nn.Sequential(
            nn.Conv2d(74, 1024, (1, 1), (1, 1), bias=False, padding=0),
            nn.LeakyReLU(0.1),
            nn.BatchNorm2d(1024),
            nn.ConvTranspose2d(1024, 128, (7, 7), (1, 1), bias=False, padding=0),
            nn.BatchNorm2d(128),
            nn.LeakyReLU(0.1),
            nn.ConvTranspose2d(128, 64, (4, 4), (2, 2), bias=False, padding=1),
            nn.BatchNorm2d(64),
            nn.ConvTranspose2d(64, 1, (4, 4), (2, 2), bias=True, padding=1),
            nn.Sigmoid())

        self.apply(initialise_weights)

    def forward(self, x):
        return self.layers(x)

# Task 3: Write down the GAN loss
class ClassicGANCriterion:
    def __call__(self, prob, label):
        prob = prob.view(-1)
        label = torch.zeros_like(prob).fill_(label)
        return F.binary_cross_entropy(prob, label)
```

## InfoGAN

The classic GAN generator places no restrictions on how the noise vector should be used to generate data. It can inadvertently come to ignore a big part of that noise vector by essentially map a range of value to the same output. As a result, the generator will fail to capture the true data distribution and quite often end up with one with very low entropy. Naturally, a generator behaviing like this will do a very poor job in fooling the discriminator. This failure mode is marked by high generator loss and 0 discriminator loss, otherwise known as "mode collapse" of GAN.

The authors of InfoGAN attempted to prevent mode collapse by introducing one regulariser as restriction. This restriction is such that any generator output, give a noise vector, has to retain maximum information about that vector. This is the same as saying:

$$
\max I(z, x_{fake})
$$

InfoGAN does this by allowing a partition in the noise vector. In classic GAN, a noise vector can be as simple as $z \sim N(\boldsymbol{0}, Id)$, where $\boldsymbol{0}$ is a zero vector and $Id$ is an identity matrix of arbitrary dimension. InfoGAN chooses to have $z = \{ c, z'\}$, where $c$ is a noise vector from a known distribution whose sufficient statistics must be recovered after the generation process (in the form of posterior estimates, but more on that later!). Whereas, $z'$ is the same old noise vector ($z$) used by classic GAN, meaning there will be no restrictions placed on $z'$.

We can then rewrite the mutual information above to only include $c$, i.e., $I(c, x_{fake})$. For InforGAN, the following is to be minimised

$$
I(d, x) - \lambda I(c, x_{fake})
$$
where $\lambda$ serves to adjust scale (entropy and mutual information are not scale-invariant).

### Implementation
To see how this can be implemented using neural network, we trace our steps in the previous section and attempt a similar work on the newly introduced regulariser. We have

$$
\begin{align}
I(c, x_{fake}) &= H(c) - H(c | x_{fake})  \\
&= H(c) + \mathbb E_{z\sim p(z)} \mathbb E_{c \sim p(c)} \log p(c | x_{fake}) \\
&= H(c) + \mathbb E_{z, c} \log p(c | \mathcal G(c, z))
\end{align}
$$

If we create a variational lower bound just like we did earlier we would have a difficult time evaluating the log-probability in the second term. However, by Law of Total Expectation, we can now see this expression in a different light:

$$
\begin{align}
\mathbb E_{z, c} \log p(c |\mathcal G(c, z)) &=
  \mathbb E_{z, c} \mathbb E_{c' \sim p(c'|x_{fake})} \log p(c' | \mathcal G (c, z)) \\
&= \mathbb E_{z, c} \mathbb E_{c' \sim p(c'|x_{fake})} \log \frac {p(c'|x_{fake})}{q(c'|x_{fake})} q(c'|x_{fake})  \\
&\ge \mathbb E_{z, c} \mathbb E_{c' \sim p(c'|x)} \log q(c'|x_{fake})
\end{align}
$$

This means we will use $c$ and $z$ to get our $x_{fake}$ through the generator $\mathcal G$. Then program a neural network as $q(c'|x_{fake})$—assuming a known family of probability distributions—that takes as input $x_{fake}$ and outputs the sufficient statistics of the approximate density $q_{c'|x_{fake}}$ (which is called a Recognition Model in the paper). There! We can work out the log-probability with much less effort.

#### Generator and utility objects
First, let us take care of the usual generator and we will get to the discriminator and recognition model after that. We have done this bit in the classic GAN section and there is nothing new in particular, only we have introduced a different normalisation module (which I might want to compare aginst BatchNorm later).

```
def initialise_weights(module):
    if isinstance(module, nn.Conv2d) or isinstance(module, nn.ConvTranspose2d):
        nn.init.xavier_normal_(module.weight.data)
        if isinstance(module.bias, torch.Tensor):
            module.bias.data.fill_(0.)

class GroupNorm2d(nn.Module):
    """Group Normalisation Layer."""
    def __init__(self, channels, groups, eps=1e-5):
        super(GroupNorm2d, self).__init__()

        self.gamma = nn.Parameter(torch.ones(1, channels, 1, 1))
        self.beta = nn.Parameter(torch.zeros(1, channels, 1, 1))
        self.num_groups = groups
        self.eps = eps

    def forward(self, x):
        N, C, H, W = x.size()
        G = self.num_groups

        x = x.view(N, G, -1)
        mean = x.mean(dim=2, keepdim=True)
        var = (x - mean).pow(2).sum(2, keepdim=True) / x.size(2)

        x = (x - mean) / (var + self.eps).sqrt()
        x = x.view(N, C, H, W)

        return x * self.gamma + self.beta

class Generator(nn.Module):
    """The Generator Network."""
    def __init__(self):
        super(Generator, self).__init__()
        # Network architecture follows the original InfoGAN paper (Chen et al.)
        self.layer = nn.Sequential(
            nn.Conv2d(74, 1024, (1, 1), (1, 1), bias=False, padding=0),
            nn.LeakyReLU(0.1),
            GroupNorm2d(1024, 32),
            nn.ConvTranspose2d(1024, 128, (7, 7), (1, 1), bias=False, padding=0),
            GroupNorm2d(128, 32),
            nn.LeakyReLU(0.1),
            nn.ConvTranspose2d(128, 64, (4, 4), (2, 2), bias=False, padding=1),
            GroupNorm2d(64, 32),
            nn.ConvTranspose2d(64, 1, (4, 4), (2, 2), bias=True, padding=1),
            nn.Tanh())

        self.apply(initialise_weights)

    def sample_latent(self, batch_size, device):
        # all dimension information is mentioned in the original InfoGAN paper
        # incompressible noise (z)
        z = torch.randn(batch_size, 62, device=device)

        # categorical latent code (cd)
        # c ~ Categ(K=10, p=0.1)
        index = torch.randint(0, 10, (batch_size,), device=device)
        cd = torch.zeros(batch_size, 10, device=device)
        cd[torch.arange(batch_size), index] = 1

        # continuous latent codes (cc)
        # c ~ Unif(-1., 1.)
        cc = torch.rand((batch_size, 2), device=device) * 2. - 1.

        noise = torch.cat([z, cd, cc], dim=1).view(-1, 74, 1, 1)
        return noise, index

    def forward(self, x):
        return self.layer(x)
```

#### Discriminator and Recognition model
Here, we have split the discriminator network into two parts (the "SharedNetwork" and "DiscriminatorEnd") and have one part shared with the Recognition model. This can be interpreted as using the Recognition model (which works as inferring the sufficient statistics of the seed noise vector) as an auxiliary task for the discriminator network. Auxiliary task tends to help obtain good representation. The same technique is also seen in reinforcement learning. We also attached another network to the discriminator to classify images from the true dataset which seems to help stablise training.

```
class SharedNetwork(nn.Module):
    """Network shared between the Discriminator and Recognition model."""
    def __init__(self):
        super(SharedNetwork, self).__init__()

        # Average pooling is used here in place of the usual max pooling or strides
        # Empirically, this avoids sparse gradients and tends to prevent mode collapse.
        self.net_base = nn.Sequential(
            nn.Conv2d(1, 64, (4, 4), (1, 1), bias=False, padding=0),
            nn.LeakyReLU(0.1),
            nn.AvgPool2d((2, 2)),
            GroupNorm2d(64, 32),
            nn.Conv2d(64, 128, (4, 4), (1, 1), bias=False, padding=0),
            nn.LeakyReLU(0.1),
            nn.AvgPool2d((2, 2)),
            GroupNorm2d(128, 32),
            nn.Conv2d(128, 1024, (4, 4), (1, 1), bias=False, padding=0),
            nn.LeakyReLU(0.1),
            GroupNorm2d(1024, 32))

        self.apply(initialise_weights)

    def forward(self, x):
        x = self.net_base(x)
        return x

class DiscriminatorEnd(nn.Module):
    """The Discriminator."""
    def __init__(self):

        super(DiscriminatorEnd, self).__init__()

        self.net_d = nn.Sequential(
            nn.Conv2d(1024, 1, (1, 1), (1, 1), bias=True, padding=0),
            nn.Sigmoid())

        self.apply(initialise_weights)

    def forward(self, x):
        return self.net_d(x).squeeze(3).squeeze(2)

class RecognitionEnd(nn.Module):
    """The Recognition model."""
    def __init__(self):
        super(RecognitionEnd, self).__init__()

        self.net_r = nn.Sequential(
            nn.Conv2d(1024, 128, (1, 1), (1, 1), bias=True, padding=0),
            nn.LeakyReLU(0.1),
            GroupNorm2d(128, 32))

        self.cat_r = nn.Conv2d(128, 10, (1, 1), (1, 1), bias=True, padding=0)
        self.gau_r = nn.Conv2d(128,  4, (1, 1), (1, 1), bias=True, padding=0)

        self.apply(initialise_weights)

    def forward(self, x):
        r = self.net_r(x)
        cat = self.cat_r(r).squeeze(3).squeeze(2)
        gau = self.gau_r(r).squeeze(3).squeeze(2)
        mean, logv = torch.chunk(gau, 2, dim=1)
        return cat, mean, logv

class ClassifierEnd(nn.Module):
    """Auxiliary Classification for the D Network."""
    # Introducing an auxiliary task whenever possible tends to help stabalise training.
    def __init__(self):
        super(ClassifierEnd, self).__init__()
        self.net_c = nn.Sequential(
            nn.Conv2d(1024, 10, (1, 1), (1, 1), bias=True, padding=0))

    def forward(self, x):
        logits = self.net_c(x).view(-1, 10)
        return F.softmax(logits, dim=1)
```

## Training InfoGAN to generate MNIST digits
### Loss functions

```
class NaNError(Exception):
    pass

class CriterionDiscriminator:
    """Binary Cross Entropy Loss."""
    def __call__(self, prob, label):
        prob = prob.view(-1)
        label = torch.zeros_like(prob).fill_(label)
        return F.binary_cross_entropy(prob, label)

class CriterionRecognitionCategorial:
    """Categorial Cross Entropy Loss"""
    def __call__(self, logits, index):
        return F.cross_entropy(logits, index)

class CriterionRecognitionNLL:
    """Negative log-likelihood for factorised Normal distribution."""
    def __call__(self, sample, mean, logv):
        sample = sample.squeeze()
        scale = (0.5 * logv).exp()
        normal = torch.distributions.Normal(mean, scale)
        # variance = logv.exp()
        # nll = (0.5 * torch.log(2 * PI).add(logv) +
        #        (sample - mean).pow(2).div(2 * variance + 1e-5))

        return - normal.log_prob(sample).sum(dim=1).mean()
```

### Setting up training
- Download the MNIST dataset and prepare data sampler
- Create new instances of network
- Define optimisers (for G and D networks, respectively)

```
%%capture
# Download the MNIST dataset
transforms = tv.transforms.Compose([
    tv.transforms.ToTensor(),
    tv.transforms.Normalize((0.5,), (0.5,))
])
mnist_data = {
    'train': MNIST('.', train=True,  transform=transforms, download=True),
    'test':  MNIST('.', train=False, transform=transforms, download=True)}
```

```
# Training settings and save names
checkpoint = 0
settings = {
    'batch_size': 128,
    'num_epochs': 300,
    'lr_g': 2e-4,
    'lr_d': 2e-4,
    'beta1': 0.5,
    'beta2': 0.999,
    'save_every': 50,
    'trained_d': 'trained_d.pth',
    'trained_g': 'trained_g.pth',
    'trained_r': 'trained_r.pth',
    'trained_c': 'trained_c.pth',
    'trained_shared': 'trained_shared.pth',
    'training_log': 'training_log.csv'}

# Pytorch DataLoader
data_loader = {
    'train': DataLoader(mnist_data['train'], batch_size=settings['batch_size'], shuffle=True, drop_last=True),
    'test':  DataLoader(mnist_data['test'],  batch_size=settings['batch_size'], shuffle=True, drop_last=True)}

# --- Create networks:
# One can potentially introduce an auxiliary classifier network as part of the D network.
# An auxiliary task, empirically, tends to help the stability of GAN training.
nets = {'G': Generator().cuda(), 'D': DiscriminatorEnd().cuda(), 'R': RecognitionEnd().cuda(),
        'Shared': SharedNetwork().cuda(), 'C': ClassifierEnd().cuda()}

# --- Optimisers:
# ADAM was chosen for G network and its hyperparameters (betas) follow that of DC-GAN
# SGD may be chosen for D network (empirically, this prevents discriminator mode collapse, i.e., D loss becomes zero)
optimiser = {
    'G': torch.optim.Adam([{'params': nets['G'].parameters()},
                           {'params': nets['R'].parameters()}],
                          lr=settings['lr_g'],
                          betas=[settings['beta1'], settings['beta2']]),
    'D': torch.optim.SGD([{'params': nets['D'].parameters()},
                          {'params': nets['C'].parameters()},
                          {'params': nets['Shared'].parameters()}],
                          lr=settings['lr_d'])}

# --- Loss functions:
# For D network, use binary cross entropy
# For Recognition (R) network, use categorical cross entropy for discrete latent codes;
# use negative Gaussian log-likelihood for continuous latent codes.
crit_d = CriterionDiscriminator()
crit_r_ce = CriterionRecognitionCategorial()
crit_r_nll = CriterionRecognitionNLL()

# Save the settings for later reference.
with open(os.path.join(save_path, 'notebook_settings.json'), 'w') as file:
    json.dump(settings, file, sort_keys=True)
```

```
%%script false  # Comment out this line to enable this cell

# Recover from previous checkpoint
checkpoint = 250
ckpt_path = os.path.join(save_path, 'ckpt')
nets['D'].load_state_dict(torch.load(os.path.join(ckpt_path, 'ckpt_d_{:03d}.pth'.format(checkpoint-1))))
nets['G'].load_state_dict(torch.load(os.path.join(ckpt_path, 'ckpt_g_{:03d}.pth'.format(checkpoint-1))))
nets['R'].load_state_dict(torch.load(os.path.join(ckpt_path, 'ckpt_r_{:03d}.pth'.format(checkpoint-1))))
nets['C'].load_state_dict(torch.load(os.path.join(ckpt_path, 'ckpt_c_{:03d}.pth'.format(checkpoint-1))))
nets['Shared'].load_state_dict(torch.load(os.path.join(ckpt_path, 'ckpt_shared_{:03d}.pth'.format(checkpoint-1))))
training_log = pd.read_csv(os.path.join(ckpt_path, 'ckpt_training_log_{:03d}.csv').format(checkpoint-1))
print('Recovered checkpoint: {}'.format(checkpoint))
```

### Main training script

```
%%script false

training_log = pd.DataFrame(columns=['epoch', 'phase', 'd_loss', 'g_loss', 'r_loss_cat', 'r_loss_gaus', 'c_loss'])
pgrid = clwidgets.Grid(1, 6)

# Main training script
for epoch in tqdm(range(checkpoint, settings['num_epochs'])):
    for phase in ['train']:
        for img_real, label in data_loader[phase]:
            i = len(training_log)
            img_real = img_real.to('cuda')
            label = label.to('cuda')
            noise, cat_index = nets['G'].sample_latent(settings['batch_size'], 'cuda')

            # Improve the Discriminator
            with torch.set_grad_enabled(phase == 'train'):
                optimiser['D'].zero_grad()
                shared_real = nets['Shared'](img_real)
                pr_real = nets['D'](shared_real)
                pr_auxc = nets['C'](shared_real)
                img_fake = nets['G'](noise)
                pr_fake = nets['D'](nets['Shared'](img_fake.detach()))

                loss_d = crit_d(pr_real, 1) + crit_d(pr_fake, 0)
                loss_c = F.cross_entropy(pr_auxc, label)

                if phase == 'train':
                    (loss_d + loss_c).backward()
                    optimiser['D'].step()

            # Improve the Generator and Recognition model
            if phase == 'train':
                optimiser['G'].zero_grad()
                shared_fake = nets['Shared'](img_fake)
                pr_fake = nets['D'](shared_fake)
                r_logit, r_mean, r_logv = nets['R'](shared_fake)

                loss_g = crit_d(pr_fake, 1)
                loss_cat = crit_r_ce(r_logit, cat_index)
                loss_gaus = crit_r_nll(noise[:, -2:], r_mean, r_logv)

                (loss_g + loss_cat + 0.1 * loss_gaus).backward()
                optimiser['G'].step()

                training_log.loc[i] = [epoch, phase, loss_d.item(), loss_g.item(), loss_cat.item(), loss_gaus.item(), loss_c.item()]

                # some sanity check for training
                if training_log.loc[i].isnull().any():
                    raise NaNError
            else:
                # if not in training phase, treat g-losses as missing
                training_log.loc[i] = [epoch, phase, loss_d.item(), np.nan, np.nan, np.nan, loss_c.item()]

    # keep a training checkpoint
    if (epoch + 1) % settings['save_every'] == 0:
        torch.save(nets['D'].state_dict(), os.path.join(save_path, 'ckpt', 'ckpt_d_{:03d}.pth'.format(epoch)))
        torch.save(nets['G'].state_dict(), os.path.join(save_path, 'ckpt', 'ckpt_g_{:03d}.pth'.format(epoch)))
        torch.save(nets['R'].state_dict(), os.path.join(save_path, 'ckpt', 'ckpt_r_{:03d}.pth'.format(epoch)))
        torch.save(nets['C'].state_dict(), os.path.join(save_path, 'ckpt', 'ckpt_c_{:03d}.pth'.format(epoch)))
        torch.save(nets['Shared'].state_dict(), os.path.join(save_path, 'ckpt', 'ckpt_shared_{:03d}.pth'.format(epoch)))
        training_log.to_csv(os.path.join(save_path, 'ckpt', 'ckpt_training_log_{:03d}.csv'.format(epoch)))

  # plot progress at the end of each epoch
    with pgrid.output_to(0, 0):
        pgrid.clear_cell()
        plt.figure(figsize=(3, 3))
        plt.imshow(img_real[0, 0].detach().cpu().numpy())
        plt.gca().set_title('Real Example ({})'.format(epoch))
        plt.gca().grid(False)
        plt.gca().set_axis_off()
    with pgrid.output_to(0, 1):
        pgrid.clear_cell()
        plt.figure(figsize=(3, 3))
        plt.imshow(img_fake[0, 0].detach().cpu().numpy())
        plt.gca().set_title('Fake Example ({})'.format(epoch))
        plt.gca().grid(False)
        plt.gca().set_axis_off()
    with pgrid.output_to(0, 2):
        pgrid.clear_cell()
        plt.figure(figsize=(3, 3))
        sns.lineplot(x='epoch', y='d_loss', data=training_log.tail(20000))
    with pgrid.output_to(0, 3):
        pgrid.clear_cell()
        plt.figure(figsize=(3, 3))
        sns.lineplot(x='epoch', y='g_loss', data=training_log.tail(20000))
    with pgrid.output_to(0, 4):
        pgrid.clear_cell()
        plt.figure(figsize=(3, 3))
        sns.lineplot(x='epoch', y='r_loss_cat', data=training_log.tail(20000))
    with pgrid.output_to(0, 5):
        pgrid.clear_cell()
        plt.figure(figsize=(3, 3))
        sns.lineplot(x='epoch', y='r_loss_gaus', data=training_log.tail(20000))

# save model and training log for later use
torch.save(nets['D'].state_dict(), os.path.join(save_path, settings['trained_d']))
torch.save(nets['G'].state_dict(), os.path.join(save_path, settings['trained_g']))
torch.save(nets['R'].state_dict(), os.path.join(save_path, settings['trained_r']))
torch.save(nets['C'].state_dict(), os.path.join(save_path, settings['trained_c']))
torch.save(nets['Shared'].state_dict(), os.path.join(save_path, settings['trained_shared']))
training_log.to_csv(os.path.join(save_path, settings['training_log']))
```

```
# %%script false
# load trained model
with open(os.path.join(save_path, 'notebook_settings.json'), 'r') as file:
    settings = json.load(file)

nets['D'].load_state_dict(torch.load(os.path.join(save_path, settings['trained_d'])))
nets['G'].load_state_dict(torch.load(os.path.join(save_path, settings['trained_g'])))
nets['R'].load_state_dict(torch.load(os.path.join(save_path, settings['trained_r'])))
nets['C'].load_state_dict(torch.load(os.path.join(save_path, settings['trained_c'])))
nets['Shared'].load_state_dict(torch.load(os.path.join(save_path, settings['trained_shared'])))
training_log = pd.read_csv(os.path.join(save_path, settings['training_log']))

# Download the MNIST dataset
transforms = tv.transforms.Compose([
    tv.transforms.ToTensor(),
    tv.transforms.Normalize((0.5,), (0.5,))
])
mnist_data = {
    'train': MNIST('.', train=True,  transform=transforms, download=True),
    'test':  MNIST('.', train=False, transform=transforms, download=True)}
```

```
noise, _ = nets['G'].sample_latent(1, 'cuda')
img_fake = nets['G'](noise)
img_real = mnist_data['test'][0][0].view(1, 1, 28, 28)
pgrid = clwidgets.Grid(1, 6)
with pgrid.output_to(0, 0):
    pgrid.clear_cell()
    plt.figure(figsize=(3, 3))
    plt.imshow(img_real[0, 0].detach().cpu().numpy())
    plt.gca().set_title('Real Example')
    plt.gca().grid(False)
    plt.gca().set_axis_off()
with pgrid.output_to(0, 1):
    pgrid.clear_cell()
    plt.figure(figsize=(3, 3))
    plt.imshow(img_fake[0, 0].detach().cpu().numpy())
    plt.gca().set_title('Generated Example')
    plt.gca().grid(False)
    plt.gca().set_axis_off()
with pgrid.output_to(0, 2):
    pgrid.clear_cell()
    plt.figure(figsize=(3, 3))
    sns.lineplot(x='epoch', y='d_loss', data=training_log)
with pgrid.output_to(0, 3):
    pgrid.clear_cell()
    plt.figure(figsize=(3, 3))
    sns.lineplot(x='epoch', y='g_loss', data=training_log)
with pgrid.output_to(0, 4):
    pgrid.clear_cell()
    plt.figure(figsize=(3, 3))
    sns.lineplot(x='epoch', y='r_loss_cat', data=training_log)
with pgrid.output_to(0, 5):
    pgrid.clear_cell()
    plt.figure(figsize=(3, 3))
    sns.lineplot(x='epoch', y='r_loss_gaus', data=training_log)
```

###Plot a number of random samples
Below shows 100 images, arranged into a 10x10 grid, generated by the GAN we just trained.

```
z = torch.randn(100, 62, 1, 1, device='cuda')
k = torch.zeros(100, 10, 1, 1, device='cuda')
k[torch.arange(100), torch.randint(0, 10, (100,))] = 1.
c = torch.stack(torch.meshgrid(torch.linspace(-1, 1, 10), torch.linspace(-1, 1, 10)), dim=2).view(100, 2, 1, 1).cuda()

with torch.no_grad():
    latc = torch.cat([z, k, c], dim=1)
    img_fake = nets['G'](latc)
    img_fake = img_fake.view(10, 10, 28, 28).permute(0, 2, 1, 3).contiguous().view(280, 280)
    img_fake = img_fake.cpu().numpy()
    plt.figure(figsize=(8, 8))
    plt.imshow(img_fake)
    plt.gca().grid(False)
    plt.gca().set_axis_off()
```

### Examining the categorical latent code
The 10x10 grid below shows how the continuous latent code affects the generated images. The continuous latent code has 2 dimensions, one varies from -1 to 1 along the horizontal axis and the other along the vertical axis. The noise vector $z$ is kept constant for each generation.

The slider on top of the plot controls the 1 position in the categorical latent code.

```
@ipywidgets.interact(index=(0, 9, 1))
def generate(index):
    k.fill_(0.)
    k[:, index] = 1

    with torch.no_grad():
        latc = torch.cat([z, k, c], dim=1)
        img_fake = nets['G'](latc)
        img_fake = img_fake.view(10, 10, 28, 28).permute(0, 2, 1, 3).contiguous().view(280, 280)
        img_fake = img_fake.cpu().numpy()
        plt.figure(figsize=(8, 8))
        plt.imshow(img_fake)
        plt.gca().grid(False)
        plt.gca().set_axis_off()
```

    interactive(children=(IntSlider(value=4, description='index', max=9), Output()), _dom_classes=('widget-interac…

```

```


+ This notebook is part of *Final exam review* in the OCW MIT course 18.06 by Prof Gilbert Strang [1]
+ Created by me, Dr Juan H Klopper
    + Head of Acute Care Surgery
    + Groote Schuur Hospital
    + University Cape Town
    + Email me with your thoughts, comments, suggestions and corrections

Linear Algebra OCW MIT18.06 IPython notebook [2] study notes by Dr Juan H Klopper is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

+ [1] OCW MIT 18.06
+ [2] Fernando Pérez, Brian E. Granger, IPython: A System for Interactive Scientific Computing, Computing in Science and Engineering, vol. 9, no. 3, pp. 21-29, May/June 2007, doi:10.1109/MCSE.2007.53. URL: http://ipython.org

```python
from IPython.core.display import HTML, Image
css_file = 'style.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Philosopher:400,700,400italic,700italic' rel='stylesheet' type='text/css'>

```python
from sympy import init_printing, Matrix, symbols, sqrt, Rational
from sympy.solvers import solve
from numpy import matrix, transpose, sqrt, eye
from numpy.linalg import pinv, inv, det, svd, norm, eig
from scipy.linalg import pinv2
from warnings import filterwarnings
```

```python
init_printing(use_latex = 'mathjax')
filterwarnings('ignore')
```

# Final examination review

## Previous examination questions

### Question 1

+ If A is a *m* × *n* matrix of rank *r* and the following holds
    + No solution
    $$ Ax=\begin{bmatrix}1\\0\\0\end{bmatrix} $$
    + One solution
    $$ Ax=\begin{bmatrix}0\\1\\0\end{bmatrix} $$

+ How many rows in this matrix?
    + *m* = 3

+ What is the rank?
    + If there are no solutions then *r* < *m*
    + If there is only a single solution then the nullspace has only the zero vector as so *r* = *n*

+ How many columns?
    + For one solution (as above) *r* = *n* and with *m* = 3 and *r* < *m* we have *r* = *n* < 3

+ Write down a matrix that fits the description above
$$ A=\begin{bmatrix}0&0\\1&0\\0&1\end{bmatrix} $$

+ True or False for the above
    + Determinant of ATA is same as determinant of AAT
        + False
    + ATA is invertible
        + If *r* = *n* (independent columns of A) then TRUE
    + AATA is positive definite
        + False (it is going to be 3 × 3, but still with only rank 2)

```python
A = Matrix([[0, 0], [1, 0], [0, 1]])
A
```

$$\left[\begin{matrix}0 & 0\\1 & 0\\0 & 1\end{matrix}\right]$$

```python
(A.transpose() * A).inv()
```

$$\left[\begin{matrix}1 & 0\\0 & 1\end{matrix}\right]$$

```python
(A.transpose() * A).det() == (A * A.transpose())
```

    False

```python
A * A.transpose()
```

$$\left[\begin{matrix}0 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right]$$

+ Prove that AT*y* = *c* has at least one solution for every *c* and in fact infinitely many solution for every *c*
    + It has at least one solution because the new number of rows (*n*) is equal to *r*
    + The dimension of the nullspace of AT is *m* - *r*, which in our example here would be > 0, thus infinitely many solutions

### Question 2

+ Suppose we have a matrix A with columns containing vectors *v*1, *v*2, and *v*3

+ Solve A**x** = *v*1 - *v*2 + *v*3
    + This is simple multiplication by columns
    $$ x=\begin{bmatrix}1\\-1\\1\end{bmatrix} $$

+ Suppose *v*1 - *v*2 + *v*3 = 0
    + Is the solution unique or are there more
        + Uniqueness means nothing in the nullspace except the zero vector, so in this cane the solutions are not unique
+ Suppose the columns are orthonormal (would be called *q*1, *q*2, *q*3)
    + What combination of *v*1 and *v*2 are closet to *v*3?
        + Zero for each *v*1 and *v*2

### Question 3

+ Consider the Markov matrix
$$ \begin{bmatrix}0.2&0.4&0.3\\0.4&0.2&0.3\\0.4&0.4&0.4\end{bmatrix} $$

+ Calculate the eigenvalues
    + The matrix is singular (note how ½ of columns 1 plys ½ of column 2 equals columns 3) so one eigenvalue will be zero
    + Another must be 1
    + The trace adds to 0.8 and so must the sums of the eigenvalues, thus the last eigenvalue is -0.2

+ If for the following the *u*(0) vector is as indicated, what would teh solution be after *k* steps?
$$ {u}_{k}={A}^{k};\quad u\left({0}\right)=\begin{bmatrix}0\\10\\0\end{bmatrix} $$
    + The following will hold
    $$ {u}_{k}={A}^{k};\quad u\left({0}\right)={c}_{1}{\lambda}_{1}^{k}{x}_{1}+{c}_{2}{\lambda}_{2}^{k}{x}_{2}+{c}_{3}{\lambda}_{3}^{k}{x}_{3} \\ {u}_{k}={A}^{k};\quad u\left({0}\right)=0+{c}_{2}\left({1}\right){x}_{2}+{c}_{3}\left({-0.2}\right)^{k}{x}_{3} $$
    + So at ∞ the only term that survives is *c*2*x*2
    + Indeed, the key eigenvalue in any Markov matrix is 1

+ Consider the eigenvector and calculate *u* at ∞
    + We already know that we have to use the λ = 1 eigenvalue
    + The distribution at ∞ will be as follows (see python code below)
    $$ u\left({\infty}\right)=\begin{bmatrix}3\\3\\4\end{bmatrix} $$

```python
A = Matrix([[0.2, 0.4, 0.3], [0.4, 0.2, 0.3], [0.4, 0.4, 0.4]])
A
```

$$\left[\begin{matrix}0.2 & 0.4 & 0.3\\0.4 & 0.2 & 0.3\\0.4 & 0.4 & 0.4\end{matrix}\right]$$

```python
A.eigenvects() # Looking for eigenvector of eigenvalue 1
# Have to distribute the totals into 10 (were 10 total intiallly)
```

$$\begin{bmatrix}\begin{pmatrix}-0.2, & 1, & \begin{bmatrix}\left[\begin{matrix}-1.0\\1.0\\0\end{matrix}\right]\end{bmatrix}\end{pmatrix}, & \begin{pmatrix}0, & 1, & \begin{bmatrix}\left[\begin{matrix}-0.5\\-0.5\\1.0\end{matrix}\right]\end{bmatrix}\end{pmatrix}, & \begin{pmatrix}1.0, & 1, & \begin{bmatrix}\left[\begin{matrix}0.75\\0.75\\1.0\end{matrix}\right]\end{bmatrix}\end{pmatrix}\end{bmatrix}$$

### Question 4

+ Calculate the projection onto the following line
$$ a=\begin{bmatrix}4\\-3\end{bmatrix} $$
    + The projection matrix is
    $$ P=\frac{{a}{a}^{T}}{{a}^{T}{a}} $$

```python
a = matrix([[4], [-3]]) # Using scipy
(a * transpose(a)) / (transpose(a) * a)
```

    matrix([[ 0.64, -0.48],
            [-0.48,  0.36]])

+ Consider the matrix with eigenvalues 0 and 3 and the following eigenvectors
$$ 0,\begin{bmatrix}1\\2\end{bmatrix}\quad 3,\begin{bmatrix}2\\1\end{bmatrix} $$
    + We use the following decomposition
    $$ A={S}{\Lambda}{S}^{-1} $$

```python
S = matrix([[1, 2], [2, 1]])
L = matrix([[0, 0], [0, 3]])
S_inv = inv(S)
```

```python
A = S * L * S_inv
A
```

    matrix([[ 4., -2.],
            [ 2., -1.]])

+ Give a 2 × 2 matrix A such that A ≠ BTB for any B
    + BT is always symmetric, so A can be any non-symmetric matrix

+ A matrix that has orthogonal eigenvectors, but is not symmetric
    + Any skew-symmetric matrix (transpose = negative of matrix)
    $$ \begin{bmatrix}0&1\\-1&0\end{bmatrix} $$
    + Any orthogonal matrix
    $$ \begin{bmatrix}\cos&-\sin\\\sin&\cos\end{bmatrix} $$

### Question 5

+ Consider the following system A**x**=**b**, with the least squares solution shown and calculate the projection of **b** onto the columnspace of A
$$ \begin{bmatrix} 1 & 0 \\ 1 & 1 \\ 1 & 2 \end{bmatrix}=\begin{bmatrix} c \\ d \end{bmatrix}=\begin{bmatrix} 3 \\ 4 \\ 1 \end{bmatrix}, \quad \begin{bmatrix} \hat { c }  \\ \hat { d }  \end{bmatrix}=\begin{bmatrix} \frac { 11 }{ 3 }  \\ -1 \end{bmatrix} $$
    + The least square solution is given, so simply multiply each entry by its column
    $$ \frac{ 11 }{ 3 } \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}-1\begin{bmatrix} 0 \\ 1 \\ 2 \end{bmatrix} $$

+ Calculate a different vector **b** such that all the least square solutions are zero
    + This requires **b** to be orthogonal to those columns, such as the following
    $$ \begin{bmatrix}1\\-2\\1\end{bmatrix} $$

### Question 6 (from recitation)

+ Consider then 3 × 3 matrix A, with λ1=1 and λ2=2 and the first two pivots *d*1=*d*2=1
$$ A=\begin{bmatrix}1&0&1\\0&1&1\\1&1&0\end{bmatrix} $$

+ Find λ3 and *d*3
    + The sum of the eigenvalues must equal the trace, thus λ3=-1
    + Constant multiples of a row subtracted from another won't change the determinant leaving *d*1×*d*2×*d*3=|A| (just watching out for singular matrices which will have a zero on the main diagonal; here though we have three non-zero eigenvalues, so the matrix is not-singular), leaving *d*3=-2 (product of eigenvalues is also the determinant of A)

+ Calculate the smallest *a*33 entry that will make positive semi-definite
    + For positive definite the eigenvalues must all be ≥ zero
    + The determinant must also be ≥ 0

```python
a33 = symbols('a33')
A = Matrix([[1, 0, 1], [0, 1, 1], [1, 1, a33]])
A
```

$$\left[\begin{matrix}1 & 0 & 1\\0 & 1 & 1\\1 & 1 & a_{33}\end{matrix}\right]$$

```python
A.det() # Thus a33 must be grteater tha or equal to 2
```

$$a_{33} - 2$$

+ Calculate the smallest values of *c* such that the following is still positive semi-definite
$$ A-cI $$
    + We can calculate the determinant using sympy (see below) or we can make use of the fact that adding a constant multiple of the identity matrix will only add that constant to each eigenvalue, leaving the eigenvectors intact
       $$ 1+c,\quad 2+c,\quad -1+c $$
   + Each must be ≥ 0, so the smallest value of *c* is 1

```python
c = symbols('c')
A = Matrix([[1, 0, 1], [0, 1, 1], [1, 1, 0]])
(A - c * eye(3))
```

$$\left[\begin{matrix}- 1.0 c + 1 & 0 & 1\\0 & - 1.0 c + 1 & 1\\1 & 1 & - 1.0 c\end{matrix}\right]$$

```python
(A - c * eye(3)).det() # From here we can calulcate the smallest value of c such
# that the determinant is still greater than or equal to zero
```

$$- 1.0 c^{3} + 2.0 c^{2} + 1.0 c - 2$$

```python
f = -c ** 3 + 2 * c ** 2 + c - 2
f
```

$$- c^{3} + 2 c^{2} + c - 2$$

```python
solve(f, c) # solve the equation f for the variable c
```

$$\begin{bmatrix}-1, & 1, & 2\end{bmatrix}$$

+ Consider now one of the starting vectors *u*0 below and with *u*k+1 = ½A*u*k calculate the limiting behavior of *u*k as *k* approaches ∞
$$ {u}_{0}=\begin{bmatrix}3\\0\\0\end{bmatrix},\quad\begin{bmatrix}0\\3\\0\end{bmatrix},\quad\begin{bmatrix}0\\0\\3\end{bmatrix} $$
    + Notice that ½ is a Markov matrix
    + We cannot be sure that there will be a steady state as there are zero entries in ½A
    + Multiplying a matrix by a constant scalar will not change the eigenvectors, but will change the eigenvalues by the same scalar multiple and we will have λ1=½ and λ2=1 and λ3=-½
    + We do have an eigenvalue of 1, so we will reach a steady-state
    + The eigenvector of λ2=1 is the following (see below)
    $$ \begin{bmatrix}1\\1\\1\end{bmatrix} $$
    + This already sums to 3, so will be *u*∞

```python

```


# What do word vectors represent?

On Monday we saw the result of running a word embedding algorithm on two collections. We "embed" words in a vector space, such that words that are subtitutable ("ship" and "boat") or that occur together ("Stop" and "thief!").

In today's work we will look at the intuition for how the properties of embedding vectors relate to properties we can directly observe in texts. They are really just representations of the words that occur near a given word.

What can we tell about a word from the words that occur near it?

```python
import numpy, sys, math
from IPython.display import display, clear_output, Markdown, Latex

from collections import Counter
```

```python
## Helper functions to nicely display numeric word scores

def show(sorted_words, n=20):
    markdown_table = "|Score | Word|\n|---:|:---|\n"
    for score, word in sorted_words[:n]:
        markdown_table += "|{:.3f}|{}|\n".format(score, word)
    display(Markdown(markdown_table))

def show_counter(counter, n=20):
    markdown_table = "|Count | Word|\n|---:|:---|\n"
    for word, score in counter.most_common(n):
        markdown_table += "|{}|{}|\n".format(score, word)
    display(Markdown(markdown_table))
```

First we'll read the texts. I've already split the tokens with Spacy and written the output to a file with one sentence per line, so punctuation will be included as distinct tokens.

While we read this, we'll also count the frequency of each word type in `all_counter`.

```python
text_filename = "../data/Sagas/sagas_en_split.txt"

sentences = []
all_counter = Counter()

with open(text_filename, encoding="utf-8") as reader:
    for line in reader:
        ## The file has already been tokenized, so we can split on whitespace
        tokens = line.strip().split()
        all_counter.update(tokens)

        sentences.append(tokens)
```

Let's start by looking at the context that words appear in. The next block defines a *key word in context* (KWIC) view.

```python
window_size = 5
```

```python
def keyword_in_context(query):
    table_markdown = "|left context|word|right context|\n|--:|--|:--|\n"
    for sentence in sentences:

        if not query in sentence:
            continue

        for i, word in enumerate(sentence):
            if word == query:
                start = max(i-window_size, 0)
                left_context = sentence[start:i]
                right_context = sentence[(i+1):(i+window_size+1)]
                table_markdown += "|{}|{}|{}|\n".format(" ".join(left_context), word, " ".join(right_context))

    display(Markdown(table_markdown))
```

### Part 1

I've given you an example, for *Shetland*, a chain of islands north of Scotland near the Orkney islands.

Add 10 additional cells, each with one call to the `keyword_in_context` function. Choose five pairs of words that you think might be similar (e.g. *Shetland* and *Orkneys*). Select a variety of parts of speech, such as nouns, verbs, adjectives, prepositions, and proper names.

Discuss what you notice about the similarities and differences between the contexts of these words.

**Answer here**

```python
keyword_in_context("Shetland")
```

```python
# add more `keyword_in_context` cells here
```

Now let's look at the distribution of words immediately preceding (*left* or *previous* context) and immediately following (*right* or *next* context) a word. This block creates two dictionaries, which map a string to the `Counter` of the words that follow that word and precede it, respectively.

```python
previous_context_counters = {} # count words that precede the key
next_context_counters = {} # count words that follow the key

for sentence in sentences:
    for i in range(len(sentence) - 1): # stop at the next-to-last token
        word = sentence[i]
        next_word = sentence[i+1]

        if not word in next_context_counters:
            next_context_counters[word] = Counter()
        if not next_word in previous_context_counters:
            previous_context_counters[next_word] = Counter()

        next_context_counters[word][next_word] += 1
        previous_context_counters[next_word][word] += 1
```

### Part 2

In the next code cell I'm demonstrating how to get the most frequent following words for a query word.

Use this function like a "predictive text" feature. Generate two Viking sentences of 10-20 words.
* In the first sentence, start with "Then" and pick the most frequent following word. Record your sentence, and comment on why always picking the most common word might not be a good idea.
* In the second sentence, start with "Then" but choose the next word based on both the frequency distribution and your artistic sensibilities.

**First sentence here**

**Comment on first sentence**

**Second sentence here**

Add cells to show previous *and* next context words for at least 10 more words. Use a selection of nouns, verbs, adjectives, prepositions, and names. These may be the same words you looked at before, but you may also want to add additional examples.

Discuss whether the words to the right or left of a word indicate its part of speech. Cite examples to support your argument. Are the two contexts equally informative for a given part of speech, and is that consistent across different parts of speech?

**Answer here**

```python
show_counter(next_context_counters["she"])
```

```python
show_counter(previous_context_counters["she"])
```

```python
## add cells here
```

Next we'll look at sums over the full five-word context window. This code creates one `Counter` for each word type, which adds up all the words that appear within the window around the word.

```python
word_context_counters = {}

for sentence in sentences:

    for i, word in enumerate(sentence):
        start = max(i-window_size, 0)
        left_context = sentence[start:i]
        right_context = sentence[(i+1):(i+window_size+1)]

        if not word in word_context_counters:
            word_context_counters[word] = Counter()

        word_context_counters[word].update(left_context)
        word_context_counters[word].update(right_context)
```

### Part 3

This next cell is an example showing output for the full context counts of a word, essentially adding up all the words you saw in the KWIC view earlier.

Show output for at least 10 words, from a mix of parts of speech.

Discuss how this view of a word's context differs from the single-previous-word and single-next-word context views we saw in Part 2.

**Answer here**

```python
show_counter(word_context_counters["Shetland"], 15)
```

```python
## add cells here
```

Finally, let's look at a way of comparing the word frequencies we actually observed to the word frequencies in the collection as a whole. We'll use a method called *pointwise mutual information*.

PMI is closely related to KL divergence. In this case, the two distributions we want to compare are the probability of context word $c$ *near* word $w$ and the probabilty of $c$ anywhere. The word *the* is common throughout the collection, so we expect to see it. This metric measures the ratio between the frequency that we actually saw it in the context and our expectation for any random context.

Notation:
* $N(c|w)$ is `word_context_counters[w][c]`
* $N(w)$ is `sum(word_context_counters[w].values()`
* $N(c)$ is `all_counter[c]`
* $N$ is `all_sum`

$$
\begin{align}
PMI(c, w) & = P(c, w) \log \frac{P(c,w)}{P(c)P(w)} \\
& = P(c, w) \log \frac{P(c|w)P(w)}{P(c)P(w)} \\
& = P(c, w) \log \frac{P(c|w)}{P(c)} \\
& \propto N(c|w) \log \frac{\frac{N(c|w)}{N(w)} }{ \frac{N(c)}{N}  } \\
& = N(c|w) \log \frac{N(c|w)N}{N(w)N(c)}
\end{align}$$

```python
def log_ratio(word):
    counter = word_context_counters[word]

    all_sum = sum(all_counter.values()) ## N
    word_sum = sum(counter.values())    ## N(w)

    comparisons = []
    for c in counter.keys():
        score = counter[c] * math.log((counter[c] * all_sum) / (word_sum * all_counter[c]))
        comparisons.append((score, c))

    return sorted(comparisons, reverse=True)

```

### Part 4

Compare results using this `log_ratio` function to the output of the `nearest` function used in Monday's notebook.

Provide some examples, and describe how they are similar or different from the output of the word embedding. If there are "missing" words in the output here that are close in the embedding space, show the `log_ratio` output for those words. Do the two words have similar context words? Describe whether this is true and mention examples.

**Answer here**

```python
## 'spae' is a Scots word for prophecy. Gunnhilda was the wife of Eric Bloodaxe.
## She was ordered to be drowned in a bog by King Harald Bluetooth, the namesake
## of the wireless standard. Think about that next time you put on some headphones.

show(log_ratio("queen"))
```

```python
## add cells with examples here
```

**Extra bonus for those interested** The embedding algorithm adds an additional step: subsampling the most frequent words. Here's code that generates this subsampling probability.

```python
sampling_probs = {}
all_sum = sum(all_counter.values())
for word in all_counter.keys():
    p_word = all_counter[word] / all_sum
    score = 1.0 / (10000 * p_word)
    sampling_probs[word] = math.sqrt(score) + score
```


Natural Language Processing For Everyone

Text Representation

Bruno Gonçalves

        www.data4sci.com

        @bgoncalves, @data4sci

In this lesson we will see in some details how we can best represent text in our application. Let's start by importing the modules we will be using:

```python
import string
from collections import Counter
from pprint import pprint
import gzip
import matplotlib.pyplot as plt
import numpy as np

import watermark

%matplotlib inline
%load_ext watermark
```

List out the versions of all loaded libraries

```python
%watermark -n -v -m -g -iv
```

    autopep8 1.5
    numpy    1.18.1
    json     2.0.9
    Mon May 04 2020

    CPython 3.7.3
    IPython 6.2.1

    compiler   : Clang 4.0.1 (tags/RELEASE_401/final)
    system     : Darwin
    release    : 19.4.0
    machine    : x86_64
    processor  : i386
    CPU cores  : 8
    interpreter: 64bit
    Git hash   : 8c3b24b7cfb0371a17c86c11dece1a06155ce164

Set the default style

```python
plt.style.use('./d4sci.mplstyle')
```

We choose a well known nursery rhyme, that has the added distinction of having been the first audio ever recorded, to be the short snippet of text that we will use in our examples:

```python
text = """Mary had a little lamb, little lamb,
    little lamb. Mary had a little lamb
    whose fleece was white as snow.
    And everywhere that Mary went
    Mary went, Mary went. Everywhere
    that Mary went,
    The lamb was sure to go"""
```

## Tokenization

The first step in any analysis is to tokenize the text. What this means is that we will extract all the individual words in the text. For the sake of simplicity, we will assume that our text is well formed and that our words are delimited either by white space or punctuation characters.

```python
print(string.punctuation)
```

    !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~

```python
def extract_words(text):
    temp = text.split() # Split the text on whitespace
    text_words = []

    for word in temp:
        # Remove any punctuation characters present in the beginning of the word
        while word[0] in string.punctuation:
            word = word[1:]

        # Remove any punctuation characters present in the end of the word
        while word[-1] in string.punctuation:
            word = word[:-1]

        # Append this word into our list of words.
        text_words.append(word.lower())

    return text_words
```

After this step we now have our text represented as an array of individual, lowercase, words:

```python
text_words = extract_words(text)
print(text_words)
```

    ['mary', 'had', 'a', 'little', 'lamb', 'little', 'lamb', 'little', 'lamb', 'mary', 'had', 'a', 'little', 'lamb', 'whose', 'fleece', 'was', 'white', 'as', 'snow', 'and', 'everywhere', 'that', 'mary', 'went', 'mary', 'went', 'mary', 'went', 'everywhere', 'that', 'mary', 'went', 'the', 'lamb', 'was', 'sure', 'to', 'go']

As we saw during the video, this is a wasteful way to represent text. We can be much more efficient by representing each word by a number

```python
word_dict = {}
word_list = []
vocabulary_size = 0
text_tokens = []

for word in text_words:
    # If we are seeing this word for the first time, create an id for it and added it to our word dictionary
    if word not in word_dict:
        word_dict[word] = vocabulary_size
        word_list.append(word)
        vocabulary_size += 1

    # add the token corresponding to the current word to the tokenized text.
    text_tokens.append(word_dict[word])
```

When we were tokenizing our text, we also generated a dictionary **word_dict** that maps words to integers and a **word_list** that maps each integer to the corresponding word.

```python
print("Word list:", word_list, "\n\n Word dictionary:")
pprint(word_dict)
```

    Word list: ['mary', 'had', 'a', 'little', 'lamb', 'whose', 'fleece', 'was', 'white', 'as', 'snow', 'and', 'everywhere', 'that', 'went', 'the', 'sure', 'to', 'go']

     Word dictionary:
    {'a': 2,
     'and': 11,
     'as': 9,
     'everywhere': 12,
     'fleece': 6,
     'go': 18,
     'had': 1,
     'lamb': 4,
     'little': 3,
     'mary': 0,
     'snow': 10,
     'sure': 16,
     'that': 13,
     'the': 15,
     'to': 17,
     'was': 7,
     'went': 14,
     'white': 8,
     'whose': 5}

These two datastructures already proved their usefulness when we converted our text to a list of tokens.

```python
print(text_tokens)
```

    [0, 1, 2, 3, 4, 3, 4, 3, 4, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 0, 14, 0, 14, 0, 14, 12, 13, 0, 14, 15, 4, 7, 16, 17, 18]

Unfortunately, while this representation is convenient for memory reasons it has some severe limitations. Perhaps the most important of which is the fact that computers naturally assume that numbers can be operated on mathematically (by addition, subtraction, etc) in a way that doesn't match our understanding of words.

## One-hot encoding

One typical way of overcoming this difficulty is to represent each word by a one-hot encoded vector where every element is zero except the one corresponding to a specific word.

```python
def one_hot(word, word_dict):
    """
        Generate a one-hot encoded vector corresponding to *word*
    """

    vector = np.zeros(len(word_dict))
    vector[word_dict[word]] = 1

    return vector
```

So, for example, the word "fleece" would be represented by:

```python
fleece_hot = one_hot("fleece", word_dict)
print(fleece_hot)
```

    [0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]

This vector has every element set to zero, except element 6, since:

```python
print(word_dict["fleece"])
fleece_hot[6] == 1
```

    6

    True

```python
print(fleece_hot.sum())
```

    1.0

## Bag of words

We can now use the one-hot encoded vector for each word to produce a vector representation of our original text, by simply adding up all the one-hot encoded vectors:

```python
text_vector1 = np.zeros(vocabulary_size)

for word in text_words:
    hot_word = one_hot(word, word_dict)
    text_vector1 += hot_word

print(text_vector1)
```

    [6. 2. 2. 4. 5. 1. 1. 2. 1. 1. 1. 1. 2. 2. 4. 1. 1. 1. 1.]

In practice, we can also easily skip the encoding step at the word level by using the *word_dict* defined above:

```python
text_vector = np.zeros(vocabulary_size)

for word in text_words:
    text_vector[word_dict[word]] += 1

print(text_vector)
```

    [6. 2. 2. 4. 5. 1. 1. 2. 1. 1. 1. 1. 2. 2. 4. 1. 1. 1. 1.]

Naturally, this approach is completely equivalent to the previous one and has the added advantage of being more efficient in terms of both speed and memory requirements.

This is known as the __bag of words__ representation of the text. It should be noted that these vectors simply contains the number of times each word appears in our document, so we can easily tell that the word *mary* appears exactly 6 times in our little nursery rhyme.

```python
text_vector[word_dict["mary"]]
```

    6.0

A more pythonic (and efficient) way of producing the same result is to use the standard __Counter__ module:

```python
text_words
```

    ['mary',
     'had',
     'a',
     'little',
     'lamb',
     'little',
     'lamb',
     'little',
     'lamb',
     'mary',
     'had',
     'a',
     'little',
     'lamb',
     'whose',
     'fleece',
     'was',
     'white',
     'as',
     'snow',
     'and',
     'everywhere',
     'that',
     'mary',
     'went',
     'mary',
     'went',
     'mary',
     'went',
     'everywhere',
     'that',
     'mary',
     'went',
     'the',
     'lamb',
     'was',
     'sure',
     'to',
     'go']

```python
word_counts = Counter(text_words)
pprint(word_counts)
```

    Counter({'mary': 6,
             'lamb': 5,
             'little': 4,
             'went': 4,
             'had': 2,
             'a': 2,
             'was': 2,
             'everywhere': 2,
             'that': 2,
             'whose': 1,
             'fleece': 1,
             'white': 1,
             'as': 1,
             'snow': 1,
             'and': 1,
             'the': 1,
             'sure': 1,
             'to': 1,
             'go': 1})

From which we can easily generate the __text_vector__ and __word_dict__ data structures:

```python
items = list(word_counts.items())

# Extract word dictionary and vector representation
word_dict2 = dict([[items[i][0], i] for i in range(len(items))])
text_vector2 = [items[i][1] for i in range(len(items))]
```

```python
word_counts['mary']
```

    6

And let's take a look at them:

```python
text_vector
```

    array([6., 2., 2., 4., 5., 1., 1., 2., 1., 1., 1., 1., 2., 2., 4., 1., 1.,
           1., 1.])

```python
print("Text vector:", text_vector2, "\n\nWord dictionary:")
pprint(word_dict2)
```

    Text vector: [6, 2, 2, 4, 5, 1, 1, 2, 1, 1, 1, 1, 2, 2, 4, 1, 1, 1, 1]

    Word dictionary:
    {'a': 2,
     'and': 11,
     'as': 9,
     'everywhere': 12,
     'fleece': 6,
     'go': 18,
     'had': 1,
     'lamb': 4,
     'little': 3,
     'mary': 0,
     'snow': 10,
     'sure': 16,
     'that': 13,
     'the': 15,
     'to': 17,
     'was': 7,
     'went': 14,
     'white': 8,
     'whose': 5}

The results using this approach are slightly different than the previous ones, because the words are mapped to different integer ids but the corresponding values are the same:

```python
for word in word_dict.keys():
    if text_vector[word_dict[word]] != text_vector2[word_dict2[word]]:
        print("Error!")
```

As expected, there are no differences!

## Term Frequency

The bag of words vector representation introduced above relies simply on the frequency of occurence of each word. Following a long tradition of giving fancy names to simple ideas, this is known as __Term Frequency__.

Intuitively, we expect the the frequency with which a given word is mentioned should correspond to the relevance of that word for the piece of text we are considering. For example, **Mary** is a pretty important word in our little nursery rhyme and indeed it is the one that occurs the most often:

```python
sorted(items, key=lambda x:x[1], reverse=True)
```

    [('mary', 6),
     ('lamb', 5),
     ('little', 4),
     ('went', 4),
     ('had', 2),
     ('a', 2),
     ('was', 2),
     ('everywhere', 2),
     ('that', 2),
     ('whose', 1),
     ('fleece', 1),
     ('white', 1),
     ('as', 1),
     ('snow', 1),
     ('and', 1),
     ('the', 1),
     ('sure', 1),
     ('to', 1),
     ('go', 1)]

However, it's hard to draw conclusions from such a small piece of text. Let us consider a significantly larger piece of text, the first 100 MB of the english Wikipedia from: http://mattmahoney.net/dc/textdata. For the sake of convenience, text8.gz has been included in this repository in the **data/** directory. We start by loading it's contents into memory as an array of words:

```python
data = []

for line in gzip.open("data/text8.gz", 'rt'):
    data.extend(line.strip().split())
```

Now let's take a look at the first 50 words in this large corpus:

```python
data[:50]
```

    ['anarchism',
     'originated',
     'as',
     'a',
     'term',
     'of',
     'abuse',
     'first',
     'used',
     'against',
     'early',
     'working',
     'class',
     'radicals',
     'including',
     'the',
     'diggers',
     'of',
     'the',
     'english',
     'revolution',
     'and',
     'the',
     'sans',
     'culottes',
     'of',
     'the',
     'french',
     'revolution',
     'whilst',
     'the',
     'term',
     'is',
     'still',
     'used',
     'in',
     'a',
     'pejorative',
     'way',
     'to',
     'describe',
     'any',
     'act',
     'that',
     'used',
     'violent',
     'means',
     'to',
     'destroy',
     'the']

And the top 10 most common words

```python
counts = Counter(data)

sorted_counts = sorted(list(counts.items()), key=lambda x: x[1], reverse=True)

for word, count in sorted_counts[:10]:
    print(word, count)
```

    the 1061396
    of 593677
    and 416629
    one 411764
    in 372201
    a 325873
    to 316376
    zero 264975
    nine 250430
    two 192644

Surprisingly, we find that the most common words are not particularly meaningful. Indeed, this is a common occurence in Natural Language Processing. The most frequent words are typically auxiliaries required due to gramatical rules.

On the other hand, there is also a large number of words that occur very infrequently as can be easily seen by glancing at the word freqency distribution.

```python
dist = Counter(counts.values())
dist = list(dist.items())
dist.sort(key=lambda x:x[0])
dist = np.array(dist)

norm = np.dot(dist.T[0], dist.T[1])

plt.loglog(dist.T[0], dist.T[1]/norm)
plt.xlabel("count")
plt.ylabel("P(count)")
plt.title("Word frequency distribution")
plt.gcf().set_size_inches(11, 8)
```

## Stopwords

One common technique to simplify NLP tasks is to remove what are known as Stopwords, words that are very frequent but not meaningful. If we simply remove the most common 100 words, we significantly reduce the amount of data we have to consider while losing little information.

```python
stopwords = set([word for word, count in sorted_counts[:100]])

clean_data = []

for word in data:
    if word not in stopwords:
        clean_data.append(word)

print("Original size:", len(data))
print("Clean size:", len(clean_data))
print("Reduction:", 1-len(clean_data)/len(data))
```

    Original size: 17005207
    Clean size: 9006229
    Reduction: 0.470384041782026

```python
clean_data[:50]
```

    ['anarchism',
     'originated',
     'term',
     'abuse',
     'against',
     'early',
     'working',
     'class',
     'radicals',
     'including',
     'diggers',
     'english',
     'revolution',
     'sans',
     'culottes',
     'french',
     'revolution',
     'whilst',
     'term',
     'still',
     'pejorative',
     'way',
     'describe',
     'any',
     'act',
     'violent',
     'means',
     'destroy',
     'organization',
     'society',
     'taken',
     'positive',
     'label',
     'self',
     'defined',
     'anarchists',
     'word',
     'anarchism',
     'derived',
     'greek',
     'without',
     'archons',
     'ruler',
     'chief',
     'king',
     'anarchism',
     'political',
     'philosophy',
     'belief',
     'rulers']

Wow, our dataset size was reduced almost in half!

In practice, we don't simply remove the most common words in our corpus but rather a manually curate list of stopwords. Lists for dozens of languages and applications can easily be found online.

## Term Frequency/Inverse Document Frequency

One way of determining of the relative importance of a word is to see how often it appears across multiple documents. Words that are relevant to a specific topic are more likely to appear in documents about that topic and much less in documents about other topics. On the other hand, less meaningful words (like **the**) will be common across documents about any subject.

To measure the document frequency of a word we will need to have multiple documents. For the sake of simplicity, we will treat each sentence of our nursery rhyme as an individual document:

```python
print(text)
```

    Mary had a little lamb, little lamb,
        little lamb. Mary had a little lamb
        whose fleece was white as snow.
        And everywhere that Mary went
        Mary went, Mary went. Everywhere
        that Mary went,
        The lamb was sure to go

```python
corpus_text = text.split('.')
corpus_words = []

for document in corpus_text:
    doc_words = extract_words(document)
    corpus_words.append(doc_words)
```

Now our corpus is represented as a list of word lists, where each list is just the word representation of the corresponding sentence:

```python
print(len(corpus_words))
```

    4

```python
pprint(corpus_words)
```

    [['mary', 'had', 'a', 'little', 'lamb', 'little', 'lamb', 'little', 'lamb'],
     ['mary',
      'had',
      'a',
      'little',
      'lamb',
      'whose',
      'fleece',
      'was',
      'white',
      'as',
      'snow'],
     ['and', 'everywhere', 'that', 'mary', 'went', 'mary', 'went', 'mary', 'went'],
     ['everywhere',
      'that',
      'mary',
      'went',
      'the',
      'lamb',
      'was',
      'sure',
      'to',
      'go']]

Let us now calculate the number of documents in which each word appears:

```python
document_count = {}

for document in corpus_words:
    word_set = set(document)

    for word in word_set:
        document_count[word] = document_count.get(word, 0) + 1

pprint(document_count)
```

    {'a': 2,
     'and': 1,
     'as': 1,
     'everywhere': 2,
     'fleece': 1,
     'go': 1,
     'had': 2,
     'lamb': 3,
     'little': 2,
     'mary': 4,
     'snow': 1,
     'sure': 1,
     'that': 2,
     'the': 1,
     'to': 1,
     'was': 2,
     'went': 2,
     'white': 1,
     'whose': 1}

As we can see, the word __Mary__ appears in all 4 of our documents, making it useless when it comes to distinguish between the different sentences. On the other hand, words like __white__ which appear in only one document are very discriminative. Using this approach we can define a new quantity, the ___Inverse Document Frequency__ that tells us how frequent a word is across the documents in a specific corpus:

```python
def inv_doc_freq(corpus_words):
    number_docs = len(corpus_words)

    document_count = {}

    for document in corpus_words:
        word_set = set(document)

        for word in word_set:
            document_count[word] = document_count.get(word, 0) + 1

    IDF = {}

    for word in document_count:
        IDF[word] = np.log(number_docs/document_count[word])

    return IDF
```

Where we followed the convention of using the logarithm of the inverse document frequency. This has the numerical advantage of avoiding to have to handle small fractional numbers.

We can easily see that the IDF gives a smaller weight to the most common words and a higher weight to the less frequent:

```python
corpus_words
```

    [['mary', 'had', 'a', 'little', 'lamb', 'little', 'lamb', 'little', 'lamb'],
     ['mary',
      'had',
      'a',
      'little',
      'lamb',
      'whose',
      'fleece',
      'was',
      'white',
      'as',
      'snow'],
     ['and', 'everywhere', 'that', 'mary', 'went', 'mary', 'went', 'mary', 'went'],
     ['everywhere',
      'that',
      'mary',
      'went',
      'the',
      'lamb',
      'was',
      'sure',
      'to',
      'go']]

```python
IDF = inv_doc_freq(corpus_words)

pprint(IDF)
```

    {'a': 0.6931471805599453,
     'and': 1.3862943611198906,
     'as': 1.3862943611198906,
     'everywhere': 0.6931471805599453,
     'fleece': 1.3862943611198906,
     'go': 1.3862943611198906,
     'had': 0.6931471805599453,
     'lamb': 0.28768207245178085,
     'little': 0.6931471805599453,
     'mary': 0.0,
     'snow': 1.3862943611198906,
     'sure': 1.3862943611198906,
     'that': 0.6931471805599453,
     'the': 1.3862943611198906,
     'to': 1.3862943611198906,
     'was': 0.6931471805599453,
     'went': 0.6931471805599453,
     'white': 1.3862943611198906,
     'whose': 1.3862943611198906}

As expected **Mary** has the smallest weight of all words 0, meaning that it is effectively removed from the dataset. You can consider this as a way of implicitly identify and remove stopwords. In case you do want to keep even the words that appear in every document, you can just add a 1. to the argument of the logarithm above:

\begin{equation}
\log\left[1+\frac{N_d}{N_d\left(w\right)}\right]
\end{equation}

When we multiply the term frequency of each word by it's inverse document frequency, we have a good way of quantifying how relevant a word is to understand the meaning of a specific document.

```python
def tf_idf(corpus_words):
    IDF = inv_doc_freq(corpus_words)

    TFIDF = []

    for document in corpus_words:
        TFIDF.append(Counter(document))

    for document in TFIDF:
        for word in document:
            document[word] = document[word]*IDF[word]

    return TFIDF
```

```python
tf_idf(corpus_words)
```

    [Counter({'a': 0.6931471805599453,
              'had': 0.6931471805599453,
              'lamb': 0.8630462173553426,
              'little': 2.0794415416798357,
              'mary': 0.0}),
     Counter({'a': 0.6931471805599453,
              'as': 1.3862943611198906,
              'fleece': 1.3862943611198906,
              'had': 0.6931471805599453,
              'lamb': 0.28768207245178085,
              'little': 0.6931471805599453,
              'mary': 0.0,
              'snow': 1.3862943611198906,
              'was': 0.6931471805599453,
              'white': 1.3862943611198906,
              'whose': 1.3862943611198906}),
     Counter({'and': 1.3862943611198906,
              'everywhere': 0.6931471805599453,
              'mary': 0.0,
              'that': 0.6931471805599453,
              'went': 2.0794415416798357}),
     Counter({'everywhere': 0.6931471805599453,
              'go': 1.3862943611198906,
              'lamb': 0.28768207245178085,
              'mary': 0.0,
              'sure': 1.3862943611198906,
              'that': 0.6931471805599453,
              'the': 1.3862943611198906,
              'to': 1.3862943611198906,
              'was': 0.6931471805599453,
              'went': 0.6931471805599453})]

Now we finally have a vector representation of each of our documents that takes the informational contributions of each word into account. Each of these vectors provides us with a unique representation of each document, in the context (corpus) in which it occurs, making it posssible to define the similarity of two documents, etc.

## Porter Stemmer

There is still, however, one issue with our approach to representing text. Since we treat each word as a unique token and completely independently from all others, for large documents we will end up with many variations of the same word such as verb conjugations, the corresponding adverbs and nouns, etc.

One way around this difficulty is to use stemming algorithm to reduce words to their root (or stem) version. The most famous Stemming algorithm is known as the **Porter Stemmer** and was introduced by Martin Porter in 1980 [Program 14, 130 (1980)](https://dl.acm.org/citation.cfm?id=275705)

The algorithm starts by defining consonants (C) and vowels (V):

```python
V = set('aeiouy')
C = set('bcdfghjklmnpqrstvwxz')
```

The stem of a word is what is left of that word after a speficic ending has been removed. A function to do this is easy to implement:

```python
def get_stem(suffix, word):
    """
        Extract the stem of a word
    """

    if word.lower().endswith(suffix.lower()): # Case insensitive comparison
        return word[:-len(suffix)]

    return None
```

It also defines words (or stems) to be sequences of vowels and consonants of the form:

\begin{equation}
[C](VC)^m[V]
\end{equation}

where $m$ is called the **measure** of the word and [] represent optional sections.

```python
def measure(orig_word):
    """
        Calculate the "measure" m of a word or stem, according to the Porter Stemmer algorthim
    """

    word = orig_word.lower()

    optV = False
    optC = False
    VC = False
    m = 0

    pos = 0

    # We can think of this implementation as a simple finite state machine
    # looks for sequences of vowels or consonants depending of the state
    # in which it's in, while keeping track of how many VC sequences it
    # has encountered.
    # The presence of the optional V and C portions is recorded in the
    # optV and optC booleans.

    # We're at the initial state.
    # gobble up all the optional consonants at the beginning of the word
    while pos < len(word) and word[pos] in C:
        pos += 1
        optC = True

    while pos < len(word):
        # Now we know that the next state must be a vowel
        while pos < len(word) and word[pos] in V:
            pos += 1
            optV = True

        # Followed by a consonant
        while pos < len(word) and word[pos] in C:
            pos += 1
            optV = False

        # If a consonant was found, then we matched VC
        # so we should increment m by one. Otherwise,
        # optV remained true and we simply had a dangling
        # V sequence.
        if not optV:
            m += 1

    return m
```

Let's consider a simple example. The word __crepusculars__ should have measure 4:

[cr] (ep) (usc) (ul) (ars)

and indeed it does.

```python
word = "crepusculars"
print(measure(word))
```

    4

(agr) = (VC)

```python
word = "agr"
print(measure(word))
```

    1

The Porter algorithm sequentially applies a series of transformation rules over a series of 5 steps (step 1 is divided in 3 substeps and step 5 in 2). The rules are only applied if a certain condition is true.

In addition to possibily specifying a requirement on the measure of a word, conditions can make use of different boolean functions as well:

```python
def ends_with(char, stem):
    """
        Checks the ending of the word
    """
    return stem[-1] == char

def double_consonant(stem):
    """
        Checks the ending of a word for a double consonant
    """
    if len(stem) < 2:
        return False

    if stem[-1] in C and stem[-2] == stem[-1]:
        return True

    return False

def contains_vowel(stem):
    """
        Checks if a word contains a vowel or not
    """
    return len(set(stem) & V) > 0
```

Finally, we define a function to apply a specific rule to a word or stem:

```python
def apply_rule(condition, suffix, replacement, word):
    """
        Apply Porter Stemmer rule.
        if "condition" is True replace "suffix" by "replacement" in "word"
    """

    stem = get_stem(suffix, word)

    if stem is not None and condition is True:
        # Remove the suffix
        word = stem

        # Add the replacement suffix, if any
        if replacement is not None:
            word += replacement

    return word
```

Now we can see how rules can be applied. For example, this rule, from step 1b is successfully applied to __pastered__:

```python
word = "plastered"
suffix = "ed"
stem = get_stem(suffix, word)
apply_rule(contains_vowel(stem), suffix, None, word)
```

    'plaster'

```python
stem
```

    'plaster'

```python
contains_vowel(stem)
```

    True

While try applying the same rule to **bled** will fail to pass the condition resulting in no change.

```python
measure('agr')
```

    1

```python
word = "bled"
suffix = "ed"
stem = get_stem(suffix, word)
apply_rule(contains_vowel(stem), suffix, None, word)
```

    'bled'

```python
stem
```

    'bl'

```python
contains_vowel(stem)
```

    False

For a more complex example, we have, in Step 4:

```python
word = "adoption"
suffix = "ion"
stem = get_stem(suffix, word)
apply_rule(measure(stem) > 1 and (ends_with("s", stem) or ends_with("t", stem)), suffix, None, word)
```

    'adopt'

```python
ends_with("t", stem)
```

    True

```python
ends_with("s", stem)
```

    False

```python
measure(stem)
```

    2

In total, the Porter Stemmer algorithm (for the English language) applies several dozen rules (see https://tartarus.org/martin/PorterStemmer/def.txt for a complete list). Implementing all of them is both tedious and error prone, so we abstain from providing a full implementation of the algorithm here. High quality implementations can be found in all major NLP libraries such as [NLTK](http://www.nltk.org/howto/stem.html).

The dificulties of defining matching rules to arbitrary text cannot be fully resolved without the use of Regular Expressions (typically implemented as Finite State Machines like our __measure__ implementation above), a more advanced topic that is beyond the scope of this course.