|
Name
               |
Date
 |
| ---------------------------------------------------| ------------------------------------- |
|
Diaaeldin SHALABY
   | 23.05.2021                            |

Hands-on AI II

Unit 4 — Recurrent Neural Networks (Assignment)

Authors: B. Schäfl, S. Lehner, J. Brandstetter

Date: 30-04-2021

This file is part of the "Hands-on AI II" lecture material. The following copyright statement applies to all code within this file.

Copyright statement:

This  material,  no  matter  whether  in  printed  or  electronic  form,  may  be  used  for personal  and non-commercial educational use only.  Any reproduction of this manuscript, no matter whether as a whole or in parts, no matter whether in printed or in electronic form, requires explicit prior acceptance of the authors.

Table of contents

The Latch Sequence Data Set

Visualizing data set statistics

Splitting and preparing

Tackling Sequence Data with CNNs

Approximating performances of random models

Applying 1D convolutions to sequences

Analyzing gradients of a CNN model

Tackling Sequence Data with LSTMs

Applying LSTMs to sequences

Analyzing gradients of an initialized LSTM model

Analyzing gradients of a trained LSTM model

The effect of the forget gate in an LSTM

How to use this notebook

This notebook is designed to run from start to finish. There are different tasks (displayed in orange boxes) which require your contribution (in form of code, plain text, ...). Most/All of the supplied functions are imported from the file u4_utils.py which can be seen and treated as a black box. However, for further understanding, you can look at the implementations of the helper functions. In order to run this notebook, the packages which are imported at the beginning of u4_utils.py need to be installed.

```python
# Import pre-defined utilities specific to this notebook.
import u4_utils as u4

# Import additional utilities needed in this notebook.
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
import torch

# Setup Jupyter notebook (warning: this may affect all Jupyter notebooks running on the same Jupyter server).
u4.setup_jupyter()
```

Setting up notebook ... finished.

Module versions

As mentioned in the introductiory slides, specific minimum versions of Python itself as well as of used modules is recommended.

```python
u4.check_module_versions()
```

    Installed Python version: 3.8 (✓)
    Installed numpy version: 1.19.1 (✓)
    Installed pandas version: 1.1.3 (✓)
    Installed PyTorch version: 1.7.1 (✓)
    Installed scikit-learn version: 0.23.2 (✓)
    Installed scipy version: 1.5.0 (✓)
    Installed matplotlib version: 3.3.1 (✓)
    Installed seaborn version: 0.11.0 (✓)
    Installed PIL version: 8.0.0 (✓)
    Installed rdkit version: 2020.09.1 (✓)

The Latch Sequence Data Set

In the accompanying excercise class, the latch task was presented. You'll be working with the same data set in this assignment. The original latch task was introduced by Hochreiter and Mozer:

    Sepp Hochreiter, Michael Mozer, 2001. A discrete probabilistic memory model for discovering dependencies in time. Artificial Neural Networks -- ICANN 2001, 13, pp.661-668.

The essence of this task is that a sequence of inputs is presented, beginning with one of two symbols, A or B, and after a variable number of time steps, the model has to output a corresponding symbol. Thus, the task requires memorizing the original input over time. It has to be noted, that in the original task desription, both class-defining symbols must only appear at the first position of an instance.

The modified version of this task used in this assignment is identical to the one discussed during the accompanying exercise, with the difference of a higher amount of possible targets. Defining arguments are:

        Argument
        Value (used in this assignment)
        Description

        num_samples
        4096
        Amount of samples of the full dataset.

        num_instances
        48
        Amount of instances per sample (sample length).

        num_characters
        26
        Amount of different characters (size of the one-hot encoded vector).

        num_targets
        25
        Amount of different characters used as possible targets.

        seed
        42
        Random seed used to generate the samples of the data set.

    Execute the notebook until here and try to solve the following tasks:

Generate a latch sequence data set with the properties as described in the table above.

Visualize the last sequence of the data set in tabular form, with all $1$
            in bold magenta and all $0$ in lighter.

Visualize the first $32$ samples in a heatmap, once without and once with a corresponding prefix-mask.

Interpreting the previous visualizations, which character of the chosen alphabet determines the prefix?

```python
data_latch = u4.LatchSequenceSet(
    num_samples=4096,
    num_instances=48,
    num_characters=26,
    num_targets=25,
    seed=42)

# Visualize the last generated sequence of the latch data set.
data_sample = pd.DataFrame(data_latch[-1][0].transpose(0, 1).numpy()).astype(int)

# data_sample.style.applymap(lambda _: f'color: {r"magenta" if _ == 1.0 else r"lighter"};font-weight: bold')
```

```python
# Set random seed for reproducibility.
np.random.seed(seed=42)
torch.manual_seed(seed=42)

# Specify batch as well as test size.
batch_size = 32

# Create data loader of training set.
sampler_train = torch.utils.data.SubsetRandomSampler(list(range(32)))

```

```python
# Set default plotting style as well as random seed for reproducibility.
sns.set()
np.random.seed(seed=42)
torch.manual_seed(seed=42)

# Prepare mini-batch of latch sequence data set for plotting.
data_heatmap = iter(data_loader_train).next()[0]
data_heatmap = pd.DataFrame(map(lambda _: torch.argmax(_, dim=1).numpy(), data_heatmap))

# Plot heatmap of a mini-batch of the latch sequence data set w.r.t. the comprised characters.
u4.plot_heatmap(data=data_heatmap, prefix_mask=True, prefix_index=data_latch.num_targets, figsize=(14, 7))
u4.plot_heatmap(data=data_heatmap, prefix_mask=False, prefix_index=data_latch.num_targets, figsize=(14, 7))
```

    Execute the notebook until here and try to solve the following tasks:

Linearly split the data set into a training and a test set in a ratio of $3:1$ (use a <tt>SubsetRandomSampler</tt> and a batch size of $48$).

Compute and print the amount of samples of each of the respective sets and verify the $3 : 1$ split.

Visualize the character counts of the first training mini-batch appropriately. What is the count of the prefix character?

```python
# Set random seed for reproducibility.
np.random.seed(seed=42)
torch.manual_seed(seed=42)

# Specify batch as well as test size.
batch_size = 48
test_size = min(max(1, data_latch.num_samples // 4), len(data_latch) - 1)

# Create data loader of training set.
sampler_train = torch.utils.data.SubsetRandomSampler(list(range(
    test_size, data_latch.num_samples)))
data_loader_train = torch.utils.data.DataLoader(
    dataset=data_latch, batch_size=batch_size, sampler=sampler_train)

# Create data loader of test set.
sampler_test = torch.utils.data.SubsetRandomSampler(list(range(test_size)))
data_loader_test = torch.utils.data.DataLoader(
    dataset=data_latch, batch_size=batch_size, sampler=sampler_test)
```

```python
print(f'The training set consists of "{len(sampler_train)}" samples, '
      f'whereas the test set comprises "{len(sampler_test)}" samples.')
```

    The training set consists of "3072" samples, whereas the test set comprises "1024" samples.

```python
# Set default plotting style as well as random seed for reproducibility.
sns.set()
np.random.seed(seed=42)
torch.manual_seed(seed=42)

# Prepare mini-batch of latch sequence data set for plotting.
data_histogram = iter(data_loader_train).next()[0]
data_histogram = pd.DataFrame(map(lambda _: torch.argmax(_, dim=1).numpy(), data_histogram))
data_histogram = data_histogram.to_numpy().flatten()

# Plot heatmap of a mini-batch of the latch sequence data set w.r.t. the comprised characters.
fig, ax = plt.subplots(figsize=(14, 7))
count_plot = sns.countplot(x=data_histogram, ax=ax)
_ = count_plot.set(xlabel=r'Character', ylabel=r'Count')
```

Tackling Sequence Data with CNNs

During the accompanying exercise class, a dense feed-forward network was presented as some kind of baseline. Afterwards, recurrent architectures were applied. In this exercise, you'll be tasked with implementing a convolutional architecture for handling sequence data.

    The following code snippet is taken from the accompanying exercise notebook. You do not need to modify it for this assignment.

```python
class TheMightyDice(torch.nn.Module):
    """
    Dice roll "network" tailored to deliver random outcomes.
    """

    def __init__(self, output_size: int):
        super(TheMightyDice, self).__init__()
        self.__output_size = output_size

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return torch.rand(size=(x.shape[0], self.__output_size))
```

    Execute the notebook until here and try to solve the following tasks:

Test a <tt>TheMightyDice</tt> instance on the latch sequence test set. Do you expect this result? Comment on your answer.

Assume uniformly distributed targets. If a model would always predict the same class, what would the accuracy be?

```python
# Set random seed for reproducibility.
np.random.seed(seed=42)
torch.manual_seed(seed=42)

# Evaluate dice model on test data set.

performance = u4.test_network(
    model=TheMightyDice(25), data_loader=data_loader_test)
print(f'\nFinal loss: {performance[0]:.4f} / Final accuracy: {performance[1]:.4f}')
```

    Final loss: 0.0701 / Final accuracy: 0.0420

The accuracy is very low as expected for a random function.

If a model would always predict the same class, I think the accuracy would be 50%.

    Execute the notebook until here and try to solve the following tasks:

Implement a function for computing the output size of a convolution operation. Hint: have a look at the PyTorch documentation. You may also use your implementation from the first assignment.

Implement a class <tt>CNN</tt> with the following architecture:

        Position
        Element
        Comment

            0
            input
            -

            1
            1D convolution
            $256$ output channels and a configurable kernel size (specified as an argument to <tt>\_\_init\_\_</tt>)

            2
            ReLU
            -

            3
            1D convolution
            $256$ output channels and the same kernel size as the 1D convolution at position $1$

            4
            ReLU
            -

            5
            fully connected
            <tt>num_targets</tt> output features (as specified during the data set creation)

            6
            output
            -

Train a <tt>CNN</tt> network for $15$ epochs, print the training accuracy as well as the loss per epoch and report the final test set loss and accuracy. Use a kernel size of $1$.

Repeat the same procedure with a second <tt>CNN</tt> but a kernel size of $3$. Do you observe any differences? Comment and interpret your results.

```python
class CNN(torch.nn.Module):
    """
    CNN tailored to process Fashion-MNIST data.
    """

    def __init__(self,kernel_size):
        super(CNN, self).__init__()
        self.kernel_size = kernel_size
        self.output_features = data_latch.num_targets
        self.network = torch.nn.Sequential(
            torch.nn.Conv1d(48, 256, self.kernel_size),
            torch.nn.ReLU(True),
            torch.nn.Conv1d(256, 256, self.kernel_size),
            torch.nn.ReLU(True),
            torch.nn.Flatten(),
            torch.nn.Linear(6656, self.output_features)
        )

    def forward(self, x: torch.Tensor) -> torch.Tensor:

        return self.network(x)
```

```python
# Set random seed for reproducibility.
np.random.seed(seed=42)
torch.manual_seed(seed=42)

# Create DenseFNN instance and the corresponding optimizer to use.
target_device = torch.device(r'cpu')
input_size = data_latch.num_instances * data_latch.num_characters

cnn_model = CNN(kernel_size=1).to(target_device)
optimizer = torch.optim.Adam(cnn_model.parameters(), lr=1e-3)

# Print the architecture of the DenseFNN instance.
print(cnn_model, end='\n\n')

# Train and evaluate DenseFNN instance on the latch sequence training set.
num_epochs = 15
for epoch in range(num_epochs):

    # Train DenseFNN instance for one epoch.
    u4.train_network(
        model=cnn_model, data_loader=data_loader_train, device=target_device, optimizer=optimizer)

    # Evaluate current DenseFNN instance.
    performance = u4.test_network(
        model=cnn_model, data_loader=data_loader_train, device=target_device)

    # Print result of current epoch to standard out.
    print(f'Epoch: {str(epoch + 1).zfill(len(str(num_epochs)))} ' +
          f'/ Loss: {performance[0]:.4f} / Accuracy: {performance[1]:.4f}')

# Evaluate final model on test data set.
performance = u4.test_network(
    model=cnn_model, data_loader=data_loader_test, device=target_device)
print(f'\nFinal loss: {performance[0]:.4f} / Final accuracy: {performance[1]:.4f}')
```

    CNN(
      (network): Sequential(
        (0): Conv1d(48, 256, kernel_size=(1,), stride=(1,))
        (1): ReLU(inplace=True)
        (2): Conv1d(256, 256, kernel_size=(1,), stride=(1,))
        (3): ReLU(inplace=True)
        (4): Flatten(start_dim=1, end_dim=-1)
        (5): Linear(in_features=6656, out_features=25, bias=True)
      )
    )

    Epoch: 01 / Loss: 0.0663 / Accuracy: 0.0833
    Epoch: 02 / Loss: 0.0583 / Accuracy: 0.1777
    Epoch: 03 / Loss: 0.0463 / Accuracy: 0.4072
    Epoch: 04 / Loss: 0.0323 / Accuracy: 0.6257
    Epoch: 05 / Loss: 0.0198 / Accuracy: 0.8076
    Epoch: 06 / Loss: 0.0112 / Accuracy: 0.9271
    Epoch: 07 / Loss: 0.0065 / Accuracy: 0.9714
    Epoch: 08 / Loss: 0.0028 / Accuracy: 0.9967
    Epoch: 09 / Loss: 0.0015 / Accuracy: 0.9997
    Epoch: 10 / Loss: 0.0008 / Accuracy: 1.0000
    Epoch: 11 / Loss: 0.0006 / Accuracy: 1.0000
    Epoch: 12 / Loss: 0.0004 / Accuracy: 1.0000
    Epoch: 13 / Loss: 0.0003 / Accuracy: 1.0000
    Epoch: 14 / Loss: 0.0003 / Accuracy: 1.0000
    Epoch: 15 / Loss: 0.0002 / Accuracy: 1.0000

    Final loss: 0.1828 / Final accuracy: 0.0547

```python
class CNN(torch.nn.Module):
    """
    CNN tailored to process Fashion-MNIST data.
    """

    def __init__(self,kernel_size):
        super(CNN, self).__init__()
        self.kernel_size = kernel_size
        self.output_features = data_latch.num_targets
        self.network = torch.nn.Sequential(
            torch.nn.Conv1d(48, 256, self.kernel_size),
            torch.nn.ReLU(True),
            torch.nn.Conv1d(256, 256, self.kernel_size),
            torch.nn.ReLU(True),
            torch.nn.Flatten(),
            torch.nn.Linear(5632, self.output_features)
        )

    def forward(self, x: torch.Tensor) -> torch.Tensor:

        return self.network(x)
```

```python
# Set random seed for reproducibility.
np.random.seed(seed=42)
torch.manual_seed(seed=42)

# Create DenseFNN instance and the corresponding optimizer to use.
target_device = torch.device(r'cpu')
input_size = data_latch.num_instances * data_latch.num_characters

cnn_model = CNN(kernel_size=3).to(target_device)
optimizer = torch.optim.Adam(cnn_model.parameters(), lr=1e-3)

# Print the architecture of the DenseFNN instance.
print(cnn_model, end='\n\n')

# Train and evaluate DenseFNN instance on the latch sequence training set.
num_epochs = 15
for epoch in range(num_epochs):

    # Train DenseFNN instance for one epoch.
    u4.train_network(
        model=cnn_model, data_loader=data_loader_train, device=target_device, optimizer=optimizer)

    # Evaluate current DenseFNN instance.
    performance = u4.test_network(
        model=cnn_model, data_loader=data_loader_train, device=target_device)

    # Print result of current epoch to standard out.
    print(f'Epoch: {str(epoch + 1).zfill(len(str(num_epochs)))} ' +
          f'/ Loss: {performance[0]:.4f} / Accuracy: {performance[1]:.4f}')

# Evaluate final model on test data set.
performance = u4.test_network(
    model=cnn_model, data_loader=data_loader_test, device=target_device)
print(f'\nFinal loss: {performance[0]:.4f} / Final accuracy: {performance[1]:.4f}')
```

    CNN(
      (network): Sequential(
        (0): Conv1d(48, 256, kernel_size=(3,), stride=(1,))
        (1): ReLU(inplace=True)
        (2): Conv1d(256, 256, kernel_size=(3,), stride=(1,))
        (3): ReLU(inplace=True)
        (4): Flatten(start_dim=1, end_dim=-1)
        (5): Linear(in_features=5632, out_features=25, bias=True)
      )
    )

    Epoch: 01 / Loss: 0.0669 / Accuracy: 0.0895
    Epoch: 02 / Loss: 0.0644 / Accuracy: 0.1904
    Epoch: 03 / Loss: 0.0552 / Accuracy: 0.2604
    Epoch: 04 / Loss: 0.0445 / Accuracy: 0.4502
    Epoch: 05 / Loss: 0.0324 / Accuracy: 0.6178
    Epoch: 06 / Loss: 0.0241 / Accuracy: 0.7116
    Epoch: 07 / Loss: 0.0174 / Accuracy: 0.8271
    Epoch: 08 / Loss: 0.0111 / Accuracy: 0.9141
    Epoch: 09 / Loss: 0.0064 / Accuracy: 0.9769
    Epoch: 10 / Loss: 0.0047 / Accuracy: 0.9889
    Epoch: 11 / Loss: 0.0027 / Accuracy: 0.9997
    Epoch: 12 / Loss: 0.0017 / Accuracy: 1.0000
    Epoch: 13 / Loss: 0.0011 / Accuracy: 1.0000
    Epoch: 14 / Loss: 0.0008 / Accuracy: 1.0000
    Epoch: 15 / Loss: 0.0007 / Accuracy: 1.0000

    Final loss: 0.1524 / Final accuracy: 0.0703

final accuracy is surprisingly very low for both.

    The following code snippet is taken from the accompanying exercise notebook. You do not need to modify it for this assignment.

```python
def collect_gradients(model: torch.nn.Module, loader: torch.utils.data.DataLoader) -> pd.DataFrame:
    """
    Auxiliary function for collecting gradient magnitudes of a corresponding model w.r.t. the network input.

    :param model: model instance to be used for collecting gradients
    :param device: device to use for gradient collection
    :param loader: data loader supplying the samples used for collecting gradients
    :return: data frame comprising the gradient magnitudes of the loss function w.r.t. each input element
    """
    model_state = model.training
    model.train()
    model.zero_grad()

    # Iterating over the data set and computing the corresponding gradients.
    device, gradients = next(model.parameters())[0].device, []
    criterion = torch.nn.CrossEntropyLoss()
    for batch_index, (data, target) in enumerate(loader):
        data, target = data.float().to(device), target.long().to(device)

        # Prepare network input for gradient recording.
        data.requires_grad_(True)
        data.register_hook(lambda _: gradients.append(_.cpu().abs()))

        # One forward
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        model.zero_grad()

    # Reset model state and return collected gradients.
    model.train(mode=model_state)
    return pd.DataFrame(torch.cat(gradients, dim=0).mean(dim=2).numpy())
```

    Execute the notebook until here and try to solve the following tasks:

Create a fresh instance of <tt>CNN</tt> and collect its gradients w.r.t. the network input using the latch sequence training set.

Visualize the collected gradients accordingly. What do you observe? Comment on your results.

```python
# Set random seed for reproducibility.
np.random.seed(seed=42)
torch.manual_seed(seed=42)

# Create DenseFNN instance and the corresponding optimizer to use.
target_device = torch.device(r'cpu')
input_size = data_latch.num_instances * data_latch.num_characters

cnn_model_2 = CNN(kernel_size=1).to(target_device)

```

```python
gradient_data_CNN = collect_gradients(model=cnn_model_2, loader=data_loader_train)
```

```python
# Set default plotting style.
sns.set()

# Prepare collected gradients for plotting.
gradients_prepared = pd.melt(gradient_data_CNN, value_vars=gradient_data_CNN.columns)
gradients_prepared.columns = (r'Timestep', r'Gradient Magnitude')

# Define plotting figure and corresponding attributes.
fig, ax = plt.subplots(figsize=(14, 7))
ax.set_title(r'CNN Gradient Magnitudes', fontsize=14)
ax.set(yscale=r'log')

# Plot pre-processed gradients.
_ = sns.boxplot(x=r'Timestep', y=r'Gradient Magnitude', data=gradients_prepared, ax=ax)
```

Tackling Sequence Data with LSTMs

During the accompanying exercise class, the Long Short-Term Memory (LSTM) was presented as a quite prominent and often used architecture in the recurrent case. It was designed and published by Hochreiter and Schmidhuber:

            Hochreiter, S. and Schmidhuber, J., 1997. Long short-term memory. Neural computation, 9(8), pp.1735-1780.

It has to be noted, that the most crucial part of the LSTM, the constant error carousel (CEC), was already discussed during Hochreiter's diploma thesis (in German):

                Hochreiter, S., 1991. Untersuchungen zu dynamischen neuronalen Netzen. Diploma, Technische Universität München, 91(1).

In contrast to most other recurrent architectures like the Elman RNN, the LSTM is a bit more complex, but equally more powerful:

        \begin{equation}
            \begin{split}
                i_{t} &= \sigma{\left(W_{ii}x_{t} + b_{ii} + W_{hi}h_{t-1} + b_{hi}\right)} \\
                \color{red}{f_{t}} &\color{red}{= \sigma{\left(W_{if}x_{t} + b_{if} + W_{hf}h_{t-1} + b_{hf}\right)}} \\
                g_{t} &= \tanh{\left(W_{ig}x_{t} + b_{ig} + W_{hg}h_{t-1} + b_{hg}\right)} \\
                o_{t} &= \sigma{\left(W_{io}x_{t} + b_{io} + W_{ho}h_{t-1} + b_{ho}\right)} \\
                c_{t} &= \color{red}{f_{t}\odot{}}c_{t-1} + i_{t}\odot{}g_{t} \\
                h_{t} &= o_{t}\odot{}\tanh{\left(c_{t}\right)}
            \end{split}
        \end{equation}

We are using the implementation provided by PyTorch, more information may be found in the official documentation. It has to be noted, the the original formulation did not contain an additional forget gate $f_{t}$ (see equations above), as this completely destroys the constant error carousel – it was introduced by Gers et al.:

    Gers, F.A., Schmidhuber, J. and Cummins, F., 1999. Learning to forget: Continual prediction with LSTM.

Nonetheless, for some tasks, the forget gate seems to provide a useful addition. Hence, in this exercise you'll be tasked with activating the forget gate and interpreting the results.

    Execute the notebook until here and try to solve the following tasks:

Implement a class <tt>LSTM</tt> with the following architecture:

        Position
        Element
        Comment

            0
            input
            -

            1
            LSTM
            $256$ memory cells and a configurable initial forget gate bias (specified as an argument to <tt>\_\_init\_\_</tt>)

            2
            fully connected
            <tt>num_targets</tt> output features (as specified during the data set creation)

            3
            output
            -

Train an <tt>LSTM</tt> network for $15$ epochs, print the training accuracy as well as the loss per epoch and report the final test set loss and accuracy. Use an initial forget gate bias of $0.0$. Do you expect the resulting performance?

```python
class LSTM(torch.nn.Module):
    """
    LSTM tailored to process latch sequence data.
    """

    def __init__(self, input_size: int, hidden_size: int = 256, output_size: int = 2, forget_gate_bias: int = 0.0):
        super(LSTM, self).__init__()
        self.lstm1 = torch.nn.LSTM(input_size, hidden_size, batch_first=True)
        self.fc1 = torch.nn.Linear(self.lstm1.hidden_size, data_latch.num_targets)

        # Deactivate forget gate to be in line with the original definition.
        def _reset_forget_gate_hook(_gradients: torch.Tensor) -> torch.Tensor:
            _gradients[_gradients.shape[0] // 4:_gradients.shape[0] // 2].fill_(forget_gate_bias)
            return _gradients

        for name, parameter in self.lstm1.named_parameters():
            if r'bias' in name:
                parameter.data[(parameter.shape[0] // 4):(parameter.shape[0] // 2)].fill_(1e6)
                parameter.register_hook(_reset_forget_gate_hook)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = self.lstm1(x)[0][:, -1, :]
        return self.fc1(x)
```

```python
# Set random seed for reproducibility.
np.random.seed(seed=42)
torch.manual_seed(seed=42)

# Create LSTM instance and the corresponding optimizer to use.
target_device = torch.device(r'cuda' if torch.cuda.is_available() else r'cpu')
input_size = data_latch.num_characters
output_size = data_latch.num_targets
lstm_model = LSTM(input_size=input_size, output_size=output_size).to(target_device)
optimizer = torch.optim.Adam(lstm_model.parameters(), lr=1e-2)

# Print the architecture of the LSTM instance.
print(lstm_model, end='\n\n')

# Train and evaluate LSTM instance on the latch sequence training set.
num_epochs = 15
for epoch in range(num_epochs):

    # Train LSTM instance for one epoch.
    u4.train_network(
        model=lstm_model, data_loader=data_loader_train, device=target_device, optimizer=optimizer)

    # Evaluate current LSTM instance.
    performance = u4.test_network(
        model=lstm_model, data_loader=data_loader_train, device=target_device)

    # Print result of current epoch to standard out.
    print(f'Epoch: {str(epoch + 1).zfill(len(str(num_epochs)))} ' +
          f'/ Loss: {performance[0]:.4f} / Accuracy: {performance[1]:.4f}')

# Evaluate final model on test data set.
performance = u4.test_network(
    model=lstm_model, data_loader=data_loader_test, device=target_device)
print(f'\nFinal loss: {performance[0]:.4f} / Final accuracy: {performance[1]:.4f}')
```

    LSTM(
      (lstm1): LSTM(26, 256, batch_first=True)
      (fc1): Linear(in_features=256, out_features=25, bias=True)
    )

    Epoch: 01 / Loss: 0.0677 / Accuracy: 0.0381
    Epoch: 02 / Loss: 0.0676 / Accuracy: 0.0352
    Epoch: 03 / Loss: 0.0674 / Accuracy: 0.0508
    Epoch: 04 / Loss: 0.0667 / Accuracy: 0.0492
    Epoch: 05 / Loss: 0.0667 / Accuracy: 0.0547
    Epoch: 06 / Loss: 0.0663 / Accuracy: 0.0674
    Epoch: 07 / Loss: 0.0662 / Accuracy: 0.0752
    Epoch: 08 / Loss: 0.0655 / Accuracy: 0.0804
    Epoch: 09 / Loss: 0.0651 / Accuracy: 0.0872
    Epoch: 10 / Loss: 0.0644 / Accuracy: 0.0931
    Epoch: 11 / Loss: 0.0637 / Accuracy: 0.1090
    Epoch: 12 / Loss: 0.0403 / Accuracy: 0.4307
    Epoch: 13 / Loss: 0.0186 / Accuracy: 0.6706
    Epoch: 14 / Loss: 0.0012 / Accuracy: 0.9974
    Epoch: 15 / Loss: 0.0005 / Accuracy: 0.9997

    Final loss: 0.0008 / Final accuracy: 0.9961

    Execute the notebook until here and try to solve the following tasks:

Create a fresh instance of <tt>LSTM</tt> and collect its gradients w.r.t. the network input using the latch sequence training set.

Visualize the collected gradients accordingly. What do you observe? Comment on your results.

```python
def collect_gradients(model: torch.nn.Module, loader: torch.utils.data.DataLoader) -> pd.DataFrame:
    """
    Auxiliary function for collecting gradient magnitudes of a corresponding model w.r.t. the network input.

    :param model: model instance to be used for collecting gradients
    :param device: device to use for gradient collection
    :param loader: data loader supplying the samples used for collecting gradients
    :return: data frame comprising the gradient magnitudes of the loss function w.r.t. each input element
    """
    model_state = model.training
    model.train()
    model.zero_grad()

    # Iterating over the data set and computing the corresponding gradients.
    device, gradients = next(model.parameters())[0].device, []
    criterion = torch.nn.CrossEntropyLoss()
    for batch_index, (data, target) in enumerate(loader):
        data, target = data.float().to(device), target.long().to(device)

        # Prepare network input for gradient recording.
        data.requires_grad_(True)
        data.register_hook(lambda _: gradients.append(_.cpu().abs()))

        # One forward
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        model.zero_grad()

    # Reset model state and return collected gradients.
    model.train(mode=model_state)
    return pd.DataFrame(torch.cat(gradients, dim=0).mean(dim=2).numpy())
```

```python
# Set random seed for reproducibility.
np.random.seed(seed=42)
torch.manual_seed(seed=42)

# Create ElmanRNN instance for gradient recording.
target_device = torch.device(r'cuda' if torch.cuda.is_available() else r'cpu')
input_size = data_latch.num_characters
output_size = data_latch.num_targets
new_lstm_model = LSTM(input_size=input_size, output_size=output_size).to(target_device)
```

```python
gradient_data = collect_gradients(model=new_lstm_model, loader=data_loader_train)
# Set default plotting style.
sns.set()

# Prepare collected gradients for plotting.
gradients_prepared = pd.melt(gradient_data, value_vars=gradient_data.columns)
gradients_prepared.columns = (r'Timestep', r'Gradient Magnitude')

# Define plotting figure and corresponding attributes.
fig, ax = plt.subplots(figsize=(14, 7))
ax.set_title(r'LSTM Gradient Magnitudes', fontsize=14)
ax.set(yscale=r'log')

# Plot pre-processed gradients.
_ = sns.boxplot(x=r'Timestep', y=r'Gradient Magnitude', data=gradients_prepared, ax=ax)
```

Strangely, exponentially decreasing gradient in LSTMs

    Execute the notebook until here and try to solve the following tasks:

Use the already trained <tt>LSTM</tt> instance from the beginning of this section for collecting its gradients w.r.t. the network input using the latch sequence training set.

Visualize the collected gradients together with the gradients of the freshly created <tt>LSTM</tt> instance accordingly. What do you observe? Comment on your results.

```python
gradient_data = collect_gradients(model=lstm_model, loader=data_loader_train)
```

```python
# Set default plotting style.
sns.set()

# Prepare collected gradients for plotting.
gradients_prepared = pd.melt(gradient_data, value_vars=gradient_data.columns)
gradients_prepared.columns = (r'Timestep', r'Gradient Magnitude')

# Define plotting figure and corresponding attributes.
fig, ax = plt.subplots(figsize=(14, 7))
ax.set_title(r'LSTM Gradient Magnitudes', fontsize=14)
ax.set(yscale=r'log')

# Plot pre-processed gradients.
_ = sns.boxplot(x=r'Timestep', y=r'Gradient Magnitude', data=gradients_prepared, ax=ax)
```

The diffrences in the gradients of the trained model are negligible which proves even more strongly how powerful LSTMs are.

    Execute the notebook until here and try to solve the following tasks:

Train an <tt>LSTM</tt> network for $15$ epochs, print the training accuracy as well as the loss per epoch and report the final test set loss and accuracy. Use an initial forget gate bias of $1.0$. Do you expect the resulting performance?

Use the newly trained <tt>LSTM</tt> instance for collecting its gradients w.r.t. the network input using the latch sequence training set.

Visualize the collected gradients together with the gradients of the freshly created <tt>LSTM</tt> instance accordingly. What do you observe? Comment on your results.

```python
# Set random seed for reproducibility.
np.random.seed(seed=42)
torch.manual_seed(seed=42)

# Create LSTM instance and the corresponding optimizer to use.
target_device = torch.device(r'cuda' if torch.cuda.is_available() else r'cpu')
input_size = data_latch.num_characters
output_size = data_latch.num_targets

lstm_model_2 = LSTM(input_size=input_size, output_size=output_size, forget_gate_bias=1.0).to(target_device)
optimizer = torch.optim.Adam(lstm_model_2.parameters(), lr=1e-2)

# Print the architecture of the LSTM instance.
print(lstm_model_2, end='\n\n')

# Train and evaluate LSTM instance on the latch sequence training set.
num_epochs = 15
for epoch in range(num_epochs):

    # Train LSTM instance for one epoch.
    u4.train_network(
        model=lstm_mode_2, data_loader=data_loader_train, device=target_device, optimizer=optimizer)

    # Evaluate current LSTM instance.
    performance = u4.test_network(
        model=lstm_mode_2, data_loader=data_loader_train, device=target_device)

    # Print result of current epoch to standard out.
    print(f'Epoch: {str(epoch + 1).zfill(len(str(num_epochs)))} ' +
          f'/ Loss: {performance[0]:.4f} / Accuracy: {performance[1]:.4f}')
```

    LSTM(
      (lstm1): LSTM(26, 256, batch_first=True)
      (fc1): Linear(in_features=256, out_features=25, bias=True)
    )

    Epoch: 01 / Loss: 0.0680 / Accuracy: 0.0358
    Epoch: 02 / Loss: 0.0680 / Accuracy: 0.0358
    Epoch: 03 / Loss: 0.0680 / Accuracy: 0.0358
    Epoch: 04 / Loss: 0.0680 / Accuracy: 0.0358
    Epoch: 05 / Loss: 0.0680 / Accuracy: 0.0358
    Epoch: 06 / Loss: 0.0680 / Accuracy: 0.0358
    Epoch: 07 / Loss: 0.0680 / Accuracy: 0.0358
    Epoch: 08 / Loss: 0.0680 / Accuracy: 0.0358
    Epoch: 09 / Loss: 0.0680 / Accuracy: 0.0358
    Epoch: 10 / Loss: 0.0680 / Accuracy: 0.0358
    Epoch: 11 / Loss: 0.0680 / Accuracy: 0.0358
    Epoch: 12 / Loss: 0.0680 / Accuracy: 0.0358
    Epoch: 13 / Loss: 0.0680 / Accuracy: 0.0358
    Epoch: 14 / Loss: 0.0680 / Accuracy: 0.0358
    Epoch: 15 / Loss: 0.0680 / Accuracy: 0.0358

    Final loss: 0.0001 / Final accuracy: 0.9990

```python
# Evaluate final model on test data set.
performance = u4.test_network(
    model=lstm_model_2, data_loader=data_loader_test, device=target_device)
print(f'\nFinal loss: {performance[0]:.4f} / Final accuracy: {performance[1]:.4f}')
```

    Final loss: 0.0702 / Final accuracy: 0.0400

```python
gradient_data = collect_gradients(model=lstm_mode_2, loader=data_loader_train)
```

```python
# Set default plotting style.
sns.set()

# Prepare collected gradients for plotting.
gradients_prepared = pd.melt(gradient_data, value_vars=gradient_data.columns)
gradients_prepared.columns = (r'Timestep', r'Gradient Magnitude')

# Define plotting figure and corresponding attributes.
fig, ax = plt.subplots(figsize=(14, 7))
ax.set_title(r'LSTM Gradient Magnitudes', fontsize=14)
ax.set(yscale=r'log')

# Plot pre-processed gradients.
_ = sns.boxplot(x=r'Timestep', y=r'Gradient Magnitude', data=gradients_prepared, ax=ax)
```

```python

```


#  Macro 318: Lecture 4 - 6 / Tutorial 2

##  Data, Stats and Math with Julia

####  Lecturer:
 Dawie van Lill (dvanlill@sun.ac.za)

# Introduction

In this tutorial we will start our discussion of how to work with data in Julia.

We will then cover some basic statistics and in the last section move on to some fundamental ideas in mathematics (mostly related to calculus).

Please note that working with data in Julia is going to be different than working with data in Stata.

I am just showing basic principles here so that you are aware of them.

You do not need to memorise everything in this notebook. It is simply here as a good reference to have if you want to do some useful data work for macroeconomics.

If you are more comfortable with Stata for working with data then you can continue on that path. I am simply offering an alternative.

In the job market there are a few languages that are used for data analysis.

The most popular ones are Stata, R, Python and Julia.

At this stage Julia is not the most popular for data work, but it shares similarities with Python.

So if you know Julia well, it will be easy to pick up Python.

Julia is more popular for work related to numerical / scientific computation, which we will cover in some of the future tutorials.

If you are interested in Python as an alternative to Julia you can always contact me and I can refer you to some resources. However, for most students it is more important to get the programming principles right without worrying too much about the language that they are using.

```julia
import Pkg
```

```julia
Pkg.add("CategoricalArrays")
Pkg.add("CSV")
Pkg.add("DataFrames")
Pkg.add("DataFramesMeta")
Pkg.add("Downloads")
Pkg.add("ForwardDiff")
Pkg.add("GLM")
Pkg.add("LinearAlgebra")
Pkg.add("Plots")
Pkg.add("Random")
Pkg.add("RDatasets")
Pkg.add("Roots")
Pkg.add("ShiftedArrays")
Pkg.add("SparseArrays")
Pkg.add("Statistics")
Pkg.add("Symbolics")
Pkg.add("Zygote")
```

        Updating registry at `C:\Users\andie\.julia\registries\General.toml`
       Resolving package versions...
    ┌ Warning: The active manifest file at `C:\Users\andie\Documents\Macro-318-andie\Manifest.toml` has an old format that is being maintained.
    │ To update to the new format run `Pkg.upgrade_manifest()` which will upgrade the format without re-resolving.
    └ @ Pkg.Types C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\Pkg\src\manifest.jl:287
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Project.toml`
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Manifest.toml`
    ┌ Warning: The active manifest file is an older format with no julia version entry. Dependencies may have been resolved with a different julia version.
    └ @ nothing C:\Users\andie\Documents\Macro-318-andie\Manifest.toml:0
       Resolving package versions...
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Project.toml`
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Manifest.toml`
       Resolving package versions...
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Project.toml`
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Manifest.toml`
       Resolving package versions...
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Project.toml`
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Manifest.toml`
       Resolving package versions...
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Project.toml`
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Manifest.toml`
       Resolving package versions...
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Project.toml`
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Manifest.toml`
       Resolving package versions...
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Project.toml`
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Manifest.toml`
       Resolving package versions...
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Project.toml`
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Manifest.toml`
       Resolving package versions...
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Project.toml`
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Manifest.toml`
       Resolving package versions...
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Project.toml`
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Manifest.toml`
       Resolving package versions...
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Project.toml`
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Manifest.toml`
       Resolving package versions...
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Project.toml`
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Manifest.toml`
       Resolving package versions...
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Project.toml`
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Manifest.toml`
       Resolving package versions...
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Project.toml`
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Manifest.toml`
       Resolving package versions...
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Project.toml`
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Manifest.toml`
       Resolving package versions...
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Project.toml`
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Manifest.toml`
       Resolving package versions...
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Project.toml`
      No Changes to `C:\Users\andie\Documents\Macro-318-andie\Manifest.toml`

```julia
using CategoricalArrays
using CSV
using DataFrames
using DataFramesMeta
using Downloads
using ForwardDiff
using GLM
using LinearAlgebra
using Plots
using Random
using RDatasets
using Roots
using ShiftedArrays
using SparseArrays
using Statistics
using Symbolics
using Zygote
```

# Working with data

The primary package for working with data in Julia is `DataFrames.jl`.

 For a comprehensive tutorial series on this package I would recommend Bogumił Kamiński's [Introduction to DataFrames](https://github.com/bkamins/Julia-DataFrames-Tutorial).

# DataFrames basics

In this section we discuss basic principles from the DataFrames package.

For the first topic we look at how to construct and access DataFrames.

The fundamental object that we care about is the `DataFrame`. This is similar to a `dataframe` that you would find in R or in Pandas (Python).

DataFrames are essentially matrices, with the rows being observations and the columns indicating the variables.

Below is some code that sets the maximum number of rows and columns to be printed in the notebook.

```julia
ENV["LINES"] = 10; # set the max number of lines that will be printed
ENV["COLUMNS"] = 100; # set the max number of columns that will be printed
```

## Constructors

The easiest thing to construct is an empty DataFrame.

```julia
DataFrame() # empty DataFrame
```

0 rows × 0 columns

You could also construct a DataFrame with different keyword arguments. Notice the different types of the different columns.

```julia
DataFrame(A = 2:5, B = randn(4), C = "Hello")
```

4 rows × 3 columns

	A	B	C

	Int64	Float64	String

1	2	-1.05687	Hello

2	3	-1.22619	Hello

3	4	-1.04417	Hello

4	5	-0.723848	Hello

One of the most common ways to use constructors is through arrays.

```julia
commodities = ["crude", "gas", "gold", "silver"] # commodities
last_price = [4.2, 11.3, 12.1, missing] # prices of the commodities (notice that the last value is missing)

df = DataFrame(commod = commodities, price = last_price) # give names to columns
```

4 rows × 2 columns

	commod	price

	String	Float64?

1	crude	4.2

2	gas	11.3

3	gold	12.1

4	silver	missing

Notice above that the `commod` column contains only string values.

In the case of `price` we have decimal numbers with one missing number.

The `DataFrames` package infers that these are floating point numbers.

The question mark indicates the uncertainty about how to handle the missing value.

One can also easily add a new row to an existing `DataFrame` using the `push!` function.

This is equivalent to adding new observations (rows) to the variables.

```julia
new_row = (commod = "nickel", price = 5.1)
push!(df, new_row)
```

5 rows × 2 columns

	commod	price

	String	Float64?

1	crude	4.2

2	gas	11.3

3	gold	12.1

4	silver	missing

5	nickel	5.1

One could also use array comprehensions to generate values for the DataFrame,

```julia
DataFrame([rand(3) for i in 1:3], [:x1, :x2, :x3]) # see how we named the columns with the symbol notation [:x1, :x2, :x3]
```

3 rows × 3 columns

	x1	x2	x3

	Float64	Float64	Float64

1	0.665034	0.0563188	0.791727

2	0.045603	0.990904	0.960224

3	0.880942	0.824061	0.245943

You can also create a DataFrame from a matrix,

```julia
x = DataFrame(rand(3, 3), :auto) # automatically assign column names with :auto
```

3 rows × 3 columns

	x1	x2	x3

	Float64	Float64	Float64

1	0.683878	0.840128	0.352071

2	0.0315821	0.477672	0.723665

3	0.525703	0.286144	0.73931

Incidentally, you can convert the DataFrame into a matrix or array if you so wished,

```julia
Matrix(x)
```

    3×3 Matrix{Float64}:
     0.683878   0.840128  0.352071
     0.0315821  0.477672  0.723665
     0.525703   0.286144  0.73931

In the next section we talk about accessing the elements of a DataFrame as well as looking at some basic information about the DataFrame that we have on hand.

## Accessing data

Once we have our data set up in a DataFrame, we are often going to want to know some basic things about the contents. Let us construct a relatively large DataFrame. Most of the time we will be working with large datasets in economics, with thousands of rows and columns. You might be used to working with data in Excel, so things might feel foreign right now. However, I promise that once you start working with data in a programming language such as R, Julia or Python, your productivity will greatly increase. You only need to get over that initial apprehension of learning something new.

```julia
y = DataFrame(rand(1:10, 1000, 10), :auto);
```

We can get some basic summary statistics on the data in the DataFrame using the `describe` function.

```julia
describe(y)
```

10 rows × 7 columns

	variable	mean	min	median	max	nmissing	eltype

	Symbol	Float64	Int64	Float64	Int64	Int64	DataType

1	x1	5.597	1	6.0	10	0	Int64

2	x2	5.519	1	6.0	10	0	Int64

3	x3	5.599	1	6.0	10	0	Int64

4	x4	5.598	1	6.0	10	0	Int64

5	x5	5.451	1	5.0	10	0	Int64

6	x6	5.47	1	6.0	10	0	Int64

7	x7	5.404	1	5.0	10	0	Int64

8	x8	5.551	1	6.0	10	0	Int64

9	x9	5.357	1	5.0	10	0	Int64

10	x10	5.435	1	5.0	10	0	Int64

If we want to take a peak at the first few rows of the data we can use the `first` function.

```julia
first(y, 5) # first 5 rows
```

5 rows × 10 columns

	x1	x2	x3	x4	x5	x6	x7	x8	x9	x10

	Int64	Int64	Int64	Int64	Int64	Int64	Int64	Int64	Int64	Int64

1	3	2	7	10	3	5	5	9	1	8

2	4	1	10	1	2	8	9	1	2	2

3	10	1	4	7	5	10	8	3	10	9

4	2	8	9	2	4	8	8	8	5	7

5	10	1	7	7	5	7	1	10	1	5

There are multiple ways to access particular columns of the DataFrame that we have created. The most obvious way is to to use `y.col` where `col` stands for the column name. This provides us the column in vector format.

```julia
y.x2; # get a single column
```

Another interesting way to access the column is the following,

```julia
y[!, :x2]; # or y[!, 2] or y[:, :x2]
```

You can access several columns (the first two in this case) with the following command,

```julia
y[:, [:x1, :x2]];
```

Getting rows is also quite easy (and similar to the way in which we access rows in arrays),

```julia
y[1, :]
```

DataFrameRow (10 columns)

	x1	x2	x3	x4	x5	x6	x7	x8	x9	x10

	Int64	Int64	Int64	Int64	Int64	Int64	Int64	Int64	Int64	Int64

1	3	2	7	10	3	5	5	9	1	8

With the code above you can also easily change the values in the DataFrame. We could, for example, multiply each of the values in the second column by $2$ if we wanted.

```julia
first(y.x2, 2) # observe first two values in the column
```

    2-element Vector{Int64}:
     2
     1

```julia
z = y[!, :x2];
z *= 2; # multiply column by two

first(z, 2) # observe newly mutated column
```

    2-element Vector{Int64}:
     4
     2

```julia
first(y.x2, 2) # important to note that this is unchanged
```

    2-element Vector{Int64}:
     2
     1

# Importing data

Now let us import some data and play around with it a bit. This is generally referred to as data wrangling. If you want to become a data scientist, then a significant portion of your work is going to involve gathering and cleaning data. The analysis part only makes up a small percentage.

I have created a dataset that is hosted on github at the following location -- https://github.com/DawievLill/Macro-318/blob/main/data/tut2_data.csv

Now let us download this data with Julia. This is one possible way to do it.

```julia
Downloads.download(
    "https://raw.githubusercontent.com/DawievLill/Macro-318/main/data/tut2_data.csv",
    "tut2_data.csv"
)
```

    "tut2_data.csv"

Now get the data into Julia!

```julia
sa_data = DataFrame(CSV.File("tut2_data.csv", dateformat = "yyyy/mm/dd")) # specify the date format
```

71 rows × 5 columns

	date	gdp	repo	cpi	inflation

	Date	Int64	Float64	Float64	Float64

1	2004-03-31	1571580	8.0	51.1689	-2.05565

2	2004-06-30	1640953	8.0	51.4474	-2.02507

3	2004-09-30	1674699	7.66667	51.5677	-1.00851

4	2004-12-31	1731000	7.5	51.8483	1.62524

5	2005-03-31	1768828	7.5	52.1796	1.97509

6	2005-06-30	1803783	7.0	52.456	1.9603

7	2005-09-30	1873743	7.0	52.8125	2.41398

8	2005-12-31	1918423	7.0	52.9391	2.10385

9	2006-03-31	1960150	7.0	53.243	2.03793

10	2006-06-30	2048534	7.16667	53.7536	2.47365

11	2006-09-30	2120013	7.83333	54.7726	3.7114

12	2006-12-31	2231030	8.66667	55.3275	4.51158

13	2007-03-31	2270667	9.0	55.9521	5.08817

14	2007-06-30	2350848	9.16667	56.9458	5.93869

15	2007-09-30	2415988	9.83333	58.1548	6.17488

16	2007-12-31	2462412	10.6667	59.2646	7.11589

17	2008-03-31	2569803	11.0	60.9667	8.96234

18	2008-06-30	2633146	11.6667	62.6333	9.98761

19	2008-09-30	2723120	12.0	64.6	11.0829

20	2008-12-31	2673514	11.8333	65.0333	9.73391

21	2009-03-31	2742517	10.5	66.0667	8.36523

22	2009-06-30	2732647	8.16667	67.4667	7.71687

23	2009-09-30	2805594	7.16667	68.7	6.34675

24	2009-12-31	2853911	7.0	69.0333	6.15069

25	2010-03-31	2936248	6.83333	69.8	5.65086

26	2010-06-30	3010051	6.5	70.5333	4.54545

27	2010-09-30	3053147	6.33333	71.1	3.49345

28	2010-12-31	3081135	5.66667	71.4	3.4283

29	2011-03-31	3183146	5.5	72.4667	3.82044

30	2011-06-30	3251316	5.5	73.8	4.63138

⋮	⋮	⋮	⋮	⋮	⋮

Alternatively, we could have done the following, since we know the data is located in the `data` folder,

```julia
sa_data_1 = DataFrame(CSV.File("../data/tut2_data.csv")); # if you don't understand this piece of code, you can simply move on.
```

So we have succesfully imported data into Julia.

What can we do with this data?

We see that the data contains information on GDP, the repo rate, CPI and inflation.

The first thing that we might want to do is visualise the data.

This is always a good first step.

After that we might want to look at some basic descriptive statistics, to get an idea of the properties of the data.

However, before we do that, let us take a look at how to rename columns.

## Renaming

Two functions can be used to rename columns.

The `names` function returns column names as a vector of strings, while the `propertynames` function returns a vector of symbols.

```julia
names(sa_data)
```

    5-element Vector{String}:
     "date"
     "gdp"
     "repo"
     "cpi"
     "inflation"

```julia
propertynames(sa_data)
```

    5-element Vector{Symbol}:
     :date
     :gdp
     :repo
     :cpi
     :inflation

We use the `rename!` function to change column names.

This function can be used to rename all columns at once.

```julia
rename!(sa_data, [:date, :GDP, :interest, :CPI, :infl])
```

71 rows × 5 columns

	date	GDP	interest	CPI	infl

	Date	Int64	Float64	Float64	Float64

1	2004-03-31	1571580	8.0	51.1689	-2.05565

2	2004-06-30	1640953	8.0	51.4474	-2.02507

3	2004-09-30	1674699	7.66667	51.5677	-1.00851

4	2004-12-31	1731000	7.5	51.8483	1.62524

5	2005-03-31	1768828	7.5	52.1796	1.97509

6	2005-06-30	1803783	7.0	52.456	1.9603

7	2005-09-30	1873743	7.0	52.8125	2.41398

8	2005-12-31	1918423	7.0	52.9391	2.10385

9	2006-03-31	1960150	7.0	53.243	2.03793

10	2006-06-30	2048534	7.16667	53.7536	2.47365

⋮	⋮	⋮	⋮	⋮	⋮

Another option is to rename only some of the columns specified by their names,

```julia
rename!(sa_data, :GDP => :gdp, :interest => :repo, :CPI => :cpi, :infl => :inflation)
```

71 rows × 5 columns

	date	gdp	repo	cpi	inflation

	Date	Int64	Float64	Float64	Float64

1	2004-03-31	1571580	8.0	51.1689	-2.05565

2	2004-06-30	1640953	8.0	51.4474	-2.02507

3	2004-09-30	1674699	7.66667	51.5677	-1.00851

4	2004-12-31	1731000	7.5	51.8483	1.62524

5	2005-03-31	1768828	7.5	52.1796	1.97509

6	2005-06-30	1803783	7.0	52.456	1.9603

7	2005-09-30	1873743	7.0	52.8125	2.41398

8	2005-12-31	1918423	7.0	52.9391	2.10385

9	2006-03-31	1960150	7.0	53.243	2.03793

10	2006-06-30	2048534	7.16667	53.7536	2.47365

⋮	⋮	⋮	⋮	⋮	⋮

## Plotting the data

For us to plot the data let us look at one of the variables in the dataset.

Let us consider GDP, which is the second column in the dataset. We can access GDP by calling the variable name, but we can also use the fact that it is located in the second column of the table.

```julia
gdp_1 = sa_data.gdp;
gdp_2 = sa_data[!, 2];
gdp_3 = sa_data[!, :gdp];
```

We can check whether these variables give the same result as follows,

```julia
gdp_1 == gdp_2 == gdp_3 # check that these give exactly the same result
```

    true

Now let us draw a basic plot of GDP and see if it aligns with your expectation of what GDP would look like in level terms. You can also compare this with US GDP in Chapter 2 of the Williamson textbook. I have used **nominal GDP** in this dataset.

We will compare nominal with real GDP soon, with a simple calculation involving the inflation rate (although technically we should be using the GDP deflator to move from nominal to real terms).

```julia
date_sa = sa_data[!, :date]
plot(date_sa, gdp_1, legend = false, lw = 2, color = :blue, alpha = 0.8)
```

One of the most popular transformations of GDP data is to take a natural logarithm. The reason for this is that differences between adjacent values in the GDP series represent growth rates once the series is "logged". We will get back to this point at a later stage. If we take a natural log of the series then the plot of GDP looks as follows,

```julia
log_gdp = log.(gdp_1); # remember the dot syntax, since we are broadcasting the log over all the values of GDP
```

```julia
plot(date_sa, log_gdp, legend = false, lw = 2, color = :blue, alpha = 0.8)
```

You will see that the scale on the y-axis has changed after this transformation. The slope of this graph indicates the growth rate. A point that we will touch on soon.

We can also plot some of the other variables in our dataset, such as the inflation rate.

```julia
inflation_rate = sa_data.inflation
plot(date_sa, inflation_rate, legend = false, lw = 2, color = :blue, alpha = 0.8)
plot!([0], legend = false, lw = 1.5, seriestype = :hline, color = :black, ls = :dash, alpha = 0.5)
```

### To do:

1. Calculate real GDP

### Hodrick-Prescott filter (technical)

The following section is for the more technically inclined students. However, you can quickly read through this section to get a feeling for the main result.

The Hodrick-Prescott filter is a tool that is used to remove the cyclical component of a time series. In other words it extracts the trend component of a time series.

We will provide a brief formal description of what the filter is doing, and provide some code for how to operate it.

It is not expected that you fully understand this process.

This is a topic that you will encounter again in the Honours program.

For now it is good enough to simply understand what the HP filter is doing. Please do not worry about the code. Those that are interested in the code can look at it, but it is a bit beyond what is expected of most of the students in this class, unless you have a background in programming and statistics.

Formally, a time series $y_t$, such as GDP, is made up of a trend component $\tau_t$, a cyclical component $c_t$ and an error component $\varepsilon_t$ such that

$$
y_t = \tau_t + c_t + \varepsilon_{t}
$$

Given some value of $\lambda$, there is a trend component that will solve the following minimisation problem,
$$
\min _{\tau }\left(\sum _{t=1}^{T}{(y_{t}-\tau _{t})^{2}}+\lambda \sum _{t=2}^{T-1}{[(\tau _{t+1}-\tau _{t})-(\tau _{t}-\tau _{t-1})]^{2}}\right)
$$

The first term $(y_{t}-\tau _{t})^{2}$ represents the squared deviation of the trend from the series, which acts as a penalty on the cyclical component. The second term $[(\tau _{t+1}-\tau _{t})-(\tau _{t}-\tau _{t-1})]^{2}$, which is multiplied by $\lambda$, represents the second difference of the squared trend component. This part penalises variations in the growth rate of the trend component. The higher the value of $\lambda$, the higher this penalty.

The value for $\lambda$ is normally set to $1600$ for quarterly data.

```julia
function hp_filter(y; w = 1600)

    # y is original series to be smoothed
    # w is smoothing parameter
    # s is the output / filtered series

    # ensure the correct shape (column vector)
    if size(y, 1) < size(y, 2)
        y = y'
    end

    t = size(y, 1)

    a = 6 * w + 1
    b = -4 * w
    c = w

    d = [c b a]

    d = ones(t, 1) * d

    m = spdiagm(d[:,3]) + spdiagm(1 => d[1:t-1, 2]) + spdiagm(-1 => d[1:t-1,2])
    m += spdiagm(2 => d[1:t-2, 1]) + spdiagm(-2 => d[1:t-2, 1])

    m[1, 1] = 1 + w
    m[1, 2] = -2 * w
    m[2, 1] = -2 * w
    m[2, 2] = 5 * w +1
    m[t-1, t-1] = 5 * w + 1
    m[t-1, t] = -2 * w
    m[t, t-1] = -2 * w
    m[t, t] = 1 + w

    return s = m \ y
end
```

    hp_filter (generic function with 1 method)

Below we see a plot with the trend in red and the original series in black. The trend is retrieved using the HP filter, with a value of $\lambda = 1600$

```julia
hp_plot = plot(hp_filter(gdp_1), legend = false, lw = 2, alpha = 0.8, color = :red)
plot!(hp_plot, gdp_1, lw = 2, alpha = 0.7, color = :black)
```

Below is the detrended business cycle, which we retrieve by subtracting the trend from the original series.

```julia
bus_cycle = gdp_1 - hp_filter(gdp_1);
plot(date_sa, bus_cycle, lw = 2, alpha = 0.7, color = :black, legend = false)
```

Finally, we can see what happens if we use different values of $\lambda$ for the trend.

```julia
lambdas = [200, 1600, 10000]

p1 = plot()

for i in lambdas
    plot!(p1, hp_filter(gdp_1, w = i), label = "lambda = $i", legend = :topleft, lw = 2, alpha = 0.8)
end

plot(p1)
```

```julia
p2 = plot()

for i in lambdas
    plot!(p2, gdp_1 - hp_filter(gdp_1, w = i), label = "lambda = $i", legend = :bottomleft, lw = 2, alpha = 0.8)
end

plot(p2)
```

# Descriptive statistics

Some of the most widely used statistics are the mean (average) and standard deviation. They can be easily computed with the computer as follows,

```julia
inflation_1 = sa_data[!, :inflation] # select the inflation column
mean(inflation_1) # mean value of inflation
```

    4.830130044713535

```julia
std(inflation_1) # standard deviation of inflation
```

    2.3122543025163282

Let us plot a histogram for the data and also indicate where the mean is in this plot.

```julia
histogram(inflation_1, legend = false, alpha = 0.5, bins = 20) # a histogram gives a general idea of what the distribution of values for inflation looks like.
plot!([mean(inflation_1)],  seriestype = :vline, lw = 3, colour = :black, ls = :dash) # plots the mean value
```

In general, if we want descriptive statistics we can use the `describe()` function to give us some more information about the dataset.

```julia
describe(sa_data)
```

5 rows × 7 columns

	variable	mean	min	median	max	nmissing	eltype

	Symbol	Union…	Any	Any	Any	Int64	DataType

1	date		2004-03-31	2012-12-31	2021-09-30	0	Date

2	gdp	3.72775e6	1571580	3.71359e6	5819983	0	Int64

3	interest	6.73709	3.5	6.66667	12.0	0	Float64

4	cpi	82.2662	51.1689	80.0	122.233	0	Float64

5	infl	4.83013	-2.05565	4.86163	11.0829	0	Float64

# Growth rates

Mathematics is so much easier when we get to use a computer. In this section I will introduce some of the basic mathematical theory that you need as a macroeconomist and then we will show you how that relates to programming.

Computing growth rates is quite important in macroeconomics. This is something that you will frequently encounter and it is important to know how to do this. The growth rate between two subsequent dates can be calculated as follows,

$$
\left(\frac{Y_{t} - Y_{t-1}}{Y_{t-1}}\right) \times 100 = \left(\frac{Y_{t}}{Y_{t-1}} - 1\right) \times 100
$$

If you are working with quarterly data and want to calculate the growth rate from one quarter to the same quarter next year then you should use,

$$
\left(\frac{Y_{t}}{Y_{t-4}} - 1\right) \times 100
$$

In addition, we can calculate the monthly or quarterly growth rate at an annual rate,

$$
\left(\left[\frac{Y_{t}}{Y_{t-1}}\right]^{n} - 1\right) \times 100
$$
where $n = 4$ represents quarterly growth and $n = 12$ gives monthly growth.

Finally, we can calculate the average growth rate over $n$ years with the following formula,

$$
\left(\left[\frac{Y_{t}}{Y_{t-n}}\right]^{1/n} - 1\right) \times 100
$$

### Examples of growth rate calculations

Let us try calculating some growth rates with the data at hand. We will focus on GDP growth rates for this example,

```julia
gdp_first = sa_data[1, :gdp]; # first value of the GDP series
gdp_second = sa_data[2, :gdp]; # second value of the GDP series

gdp_growth_1 = ((gdp_second - gdp_first)/gdp_first) * 100 # using the formula for growth between two periods.
```

    4.414220084246427

There is an alternative way to calculate an approximation to the growth rate from above. We can simply take the natural logarithm of the two values and subtract them from each other. In other words we have that,

$$
\left(\frac{Y_{t}}{Y_{t-1}} - 1\right) \times 100 \approx \log(Y_{t}) - \log(Y_{t-1}) \times 100
$$

```julia
(log(gdp_second) - log(gdp_first)) * 100
```

    4.319568788852912

The values are not exactly the same. This is simply an approximation which is often used in practice.

As another example, let us calculate the quarterly growth rate of GDP and then plot the resulting values. This calculation is a bit more tricky since we are now going to apply it to the entire dataset. Let me explain the logic of what we are going to do here.

First, we are going to create a new column that contains a lagged version of the original column for GDP. Why do we want to do this? The reason is that we want to be able to divide $Y_t$ by $Y_{t-4}$ from our formula above. However, we only have values for $Y_t$ in the $t$-th column and not $Y_{t-4}$. So for every row we need to create a corresponding lagged version of the original column. Before we do anything, let us take a look at the first few values of the dataset to see what we need to change. We only care about the first two columns, so we will only select those.

```julia
first(sa_data[!, 1:2], 8)
```

8 rows × 2 columns

	date	gdp

	Date	Int64

1	2004-03-31	1571580

2	2004-06-30	1640953

3	2004-09-30	1674699

4	2004-12-31	1731000

5	2005-03-31	1768828

6	2005-06-30	1803783

7	2005-09-30	1873743

8	2005-12-31	1918423

```julia
sa_data.lagged_gdp = lag(sa_data[!, 2], 4);
```

We can see that the lagged GDP column has successfully been created and added to the DataFrame.

```julia
first(sa_data[!, :], 8)
```

8 rows × 6 columns

	date	gdp	repo	cpi	inflation	lagged_gdp

	Date	Int64	Float64	Float64	Float64	Int64?

1	2004-03-31	1571580	8.0	51.1689	-2.05565	missing

2	2004-06-30	1640953	8.0	51.4474	-2.02507	missing

3	2004-09-30	1674699	7.66667	51.5677	-1.00851	missing

4	2004-12-31	1731000	7.5	51.8483	1.62524	missing

5	2005-03-31	1768828	7.5	52.1796	1.97509	1571580

6	2005-06-30	1803783	7.0	52.456	1.9603	1640953

7	2005-09-30	1873743	7.0	52.8125	2.41398	1674699

8	2005-12-31	1918423	7.0	52.9391	2.10385	1731000

Now we can calculate the growth rate,

```julia
gdp_growth_quarterly = (log.(sa_data.gdp) .- log.(sa_data.lagged_gdp)) .* 100; # log method
gdp_growth_quarterly_1 = ((sa_data.gdp ./ sa_data.lagged_gdp) .- 1) .* 100; # formula method
```

```julia
plot(date_sa, gdp_growth_quarterly, legend = false, lw = 2, alpha = 0.7) # log method
plot!(date_sa, gdp_growth_quarterly_1, legend = false, lw = 2, ls = :dash, color = :black) # formula method
```

For the other formulas you can perform similar calculations. As an exercise you can attempt to use the other formulas to calculate annualised growth rates.

# Basic math for macroeconomics

In this section we will be discussing very basic mathematical concepts that relate to mathematical modeling. For this section we will be making heavy use of the notes on mathematics for economists by [Fan Wang](https://fanwangecon.github.io/Math4Econ/). Please go look at his website for more cool notes and give his repository a star. His code is mostly in Matlab, but he also has some Python and R code. You can easily translate Matlab to Julia, since the syntax of the languages are quite similar.

## Functions

A function is a **rule** that assigns to every element of $x \in X$ a **single element** of the set $Y$. This is written as,

$$
f:X \rightarrow Y
$$

The arrow indicates the mapping from the one set to another. When we write $y = f(x)$ we are mapping from the argument $x$ in the domain $X$ to a value in the co-domain $Y$.

It is important to note that for a function we are assigning a single element from the set $X$ to the set $Y$.

Let us illustrate this with some examples of functions and non-functions in Julia.

```julia
x = 0:π/100:2π
y = sin.(x)

plot(x, y, title = "This is a function", legend = false, lw = 2)
```

The function above is a portion of the $\sin$ function over the interval from $0$ to $2\pi$. However, the following graph that depicts a circle is not a function. It is a relation, but NOT a function. Can you see why this is the case? What is the defining feature of a function?

```julia
x = 1; y = 1; r = 1
θ = 0:π/50:2π

x_unit = r .* cos.(θ) .+ x
y_unit = r .* sin.(θ) .+ y

plot(x_unit, y_unit, title = "This is NOT a function", legend = false, lw = 2)
```

A linear function, which is also known as polynomial of degree 1 has slope $m$ and intercept $b$. Linear functions have constant slope. We will encounter the idea of slope again later when we talk about derivatives.

```julia
m = 0.5 # slope
b = 1 # intercept

ar_x = LinRange(-5, 10, 100)
ar_y = ar_x .* m .+ b

plot(ar_x, ar_y, legend = false, title = "Linear function with slope $m and intercept $b", lw = 2)

vline!([0], ls = :dash, color = :black, alpha = 0.5, xticks = ([-2]))
hline!([0], ls = :dash, color = :red, alpha = 0.5, yticks = ([1]))
```

In high school you probably determined the slope of this function using a the method of rise over run. In other words, the change in $y$ over the change in $x$. More "formally", you calculated $m = \frac{\Delta{y}}{\Delta{x}}$. In this example, $\Delta{y} = 1 - 0$ and $\Delta{x} = 0 - (-2)$, so we have that $m = 1 / 2 = 0.5$. We will talk about this method of using a difference quotient again when we want to calculate the slope of a tangent line to a function in the section on derivatives.

## Monomials and polynomials

Functions that take the form $a \cdot x^{k}$ are considered **monomials**. In this case $a$ is any real number and $k$ is a positive integer. The value of $k$ represent the degree of the monomial. Monomials can be added together to form **polynomials**. A general formulation for a polynomial of degree four would be,

$$
a + b\cdot{x} + c \cdot x^2 + d \cdot x^3 + e \cdot x^4
$$

where the coefficients $a, b, c, d, e$ could be positive or negative. In order to determine the degree of the polynomial, consider the monomials that constitute the polynomial. The monomial with the highest degree determines the degree of the whole polynomial. An explicit representation of a polynomial that you should all know quite well is something along the following lines,

$$
y = 2 + 5x^2
$$

Can you identify what the degree of this polynomial is? Let us plot some polynomials to get an idea of what they look like.

```julia
x = -1:0.01:1
y = 2 .+ 5 .* x .^ 2

plot(x, y, legend = false, title = "Polynomial of degree two", lw = 2)
```

Let us plot a polynomial of degree three to see how this might differ.

```julia
x = -3:0.01:3
y = 2 .- 1 .* x .^ 2 .- 10 .* x .^ 3

plot(x, y, legend = false, title = "Polynomial of degree three", lw = 2)
```

## Local and global maximum

A function $f$ has a global maximum at $x^{*} \in X$ if for all $x \in X$, $f(x) \leq f(x^{*})$. A function $f$ has a local maximum at $x^{*} \in X$ if there exists and open interval $(a, b)$ such that $x^{*} \in (a, b)$, and for all $x \in (a, b), f(x) \leq f(x^{x})$. Not all functions have a maximum.

We are going to be dealing with functions such as utility functions, production functions and budget constraints in many of our economics models.

As an example, in the case of a utility function the household is choosing the bundle of goods that provides the highest level of utility.

Most production and utility functions will NOT have a local or global maximum.

In the case of a log-utility function, higher consumption will give higher levels of utility, so no maximum exists here either.

If we have a Cobb-Douglas production function, then output will increase with labour and capital. This means that the function is even increasing in its inputs and has no maximum.

**NB**: It is only when we combine our preferences with a budget that we will be able to think about the optimal bundle choice given a fixed budget. The maximisation problem in this case will reveal some maximum. We will cover this example at length in the next tutorial.

### Quadratic utility

There is one nice utility function that has a maximum, the quadratic utility function.

$$
U(x) = x - \alpha \cdot x^2
$$

We can quickly draw a plot of this utility function and by inspection determine where the maximum is going to be. The most efficient way to determine the maximum would be using a derivative and setting it equal to zero. However, we haven't introduced derivatives yet, so let us use this inefficient way to determine the maximum for now.

```julia
npoints = 100
a, b = (-10, 10)
x = range(a, b, length = npoints)
α = 0.2

U(x) = x .- α .* x .^ 2

plot(x, U.(x), legend = false, lw = 2, title = "Can you find the max?!")
```

As you can see, we can approximately guess from the graph where the highest value for this graph is going to be. It is somewhere in the interval between 10 and 15. However, this is not precise enough. We need some method to give us the exact answer, this is where derivatives will enter. Another approach is just to consider each of the utility values and then pick the maximum from the list. Luckily Julia has a function called `findmax()` that can do this for us.

Don't worry too much about the code for now, this won't make too much sense at first. I will try and explain the basic idea, but if you don't get it don't worry. Try and come back to this code at a later stage once you are more familiar with programming in Julia. Then try and evaluate every line and see if you can make sense of what is happening here.

```julia
fmax, ix = findmax(U.(x))
```

    (1.249872461993674, 63)

We can see from above that if we evaluate our utility function with the `findmax()` function a tuple is returned. The first value in the tuple is the function output at the maximum. In other words it is the value of utility where this function is maximised. It is the value on the y-axis. The second value in the tuple is the position in the list of $x$ values that we evaluated. If we go back in our code you will see that we created a range of $x$ values in the interval $-10$ to $10$. We created a grid of 100 points within the interval (generated 100 potential values for $x$) and inserted those $x$ values into the utility function at each of the given points.

According to the `findmax()` function, the 63rd value in that grid of $x$ values was the one that maximised the function. This is also referred to as the $\argmax$ in mathematics. We can then find the $x$ value by looking at `x[ix]`, which is basically the same as `x[63]` in this case. Remember how to access elements in an array from the previous tutorial.

```julia
x[ix] == x[63]
```

    true

The maximum here is $2.52525252 \ldots$. If we chose a finer grid for $x$ we would be able to get a better approximation for the true value that maximises this function.

```julia
x[ix]
```

    2.525252525252525

In the graph below we simply draw a scatter plot with the $x$ value being represented by the 63rd point in the $x$ vector.

```julia
scatter!([x[ix]], [fmax], color = :red, ms = 5)
vline!([x[ix]], lw = 1.5, color = :black, ls = :dash, alpha = 0.5, xticks = ([x[ix]]))
```

## Derivatives

Linear functions have a constant slope, as we have seen in one of the earlier examples. However, what about non-linear functions? What is the rate of change for a non-linear function as we move along its domain?

Let $(x_0, f(x_0))$ be a point on the graph of $y = f(x)$. The derivative of $f$ at the point $x_0$ is the slope of the tangent line to the graph of $f$ at $(x_0, f(x_0))$. We can denote this derivative of a function $f$ at $x_0$ as

$$
f'(x_0) = \frac{df(x_0)}{dx} = f_{x}(x_0)
$$

The notation above is interchangeable. The derivative is defined formally as,

$$
f'(x_0) = \lim_{h \rightarrow 0} \frac{f(x_{0} + h) - f(x_0)}{h}
$$

In taking derivatives we normally revert to the derivative rules. The derivative rules are the following,

1. Constant rule
2. Power rule
3. Chain rule
4. Sum (difference) rule
5. Product rule
6. Quotient rule (we won't cover this, since you can use the product rule if needed)

Then we can also speak of the exponential and log rule for derivatives. These are ones that will be frequently used. Let us have a brief discussion and examples of each of the rules. We will also accompany the solutions that are written out by hand with a solution generated by the computer to check that we are correct in our calculation.

It is important to note that there are several ways in which you can take derivatives on the computer. The primary methods are,

1. Symbolic differentiation
2. Automatic differentiation
3. Numerical differentiation

We will not be going into detail on how these methods work. We will just use them in practice. We will show the symbolic and automatic differentiation approaches in this tutorial. For a good introduction to the approximation of derivatives using a computer you can look at these notes [here](https://mth229.github.io/derivatives.html).

#### Constant rule

If $f(x) = k$ where $k$ is some constant then $f'(k) = 0$. As an application of the constant rule, let us determine the derivative of $f(x) = 8$.

$$
f'(8) = 0
$$

This is the easiest rule and can be computed in one line. Let us see what the computer does with this example. First we use the `Symbolics` package.

```julia
@variables x
D = Differential(x)

y = 8
D(y) # we can see that this now gives us the correct formulation
```

\begin{equation}
\mathrm{\frac{d}{d x}}\left( 8 \right)
\end{equation}

```julia
expand_derivatives(D(y)) # solution to the problem -- we see that this is the same as our answer.
```

    0

Now we try the `Zygote` package. This uses automatic differentiation. Once again, we will not explain what this method entails. If you want to read more on the topic you are more than welcome. For those that are interested in machine learning and deep learning it will be compulsory to learn more about the concept of automatic differentiation at some point.

```julia
f(x) = 8

f'(x)
```

We see that there is no answer here. This seems strange. Let us try another approach with this rule. We can calculate the gradient at a certain point in another way.

```julia
Zygote.gradient(y -> 8, 8)
```

    (nothing,)

We see that the output is `nothing` here. This shows why we had no output in the previous code. When `nothing` is returned then there will be nothing to display. Another option for automatic differentiation is the `ForwardDiff` package. Let us quickly look at an example with this package.

```julia
ForwardDiff.derivative.(f, 8)
```

    0

In this case we get an answer of zero. Which is more in line with what one would expect.

#### Power rule

For any positive integer $k$ the derivative of $f(x) = x^{k}$ at $x_0$ is,

$$
f'(x_0) = k \cdot x_{0}^{k - 1}
$$

An example of this would be the following. Find the answer to

$$
f'(x) = x ^ 3
$$

What do you think the answer should be? Let us check with the computer to see what we get. First we try symbolic differentiation.

```julia
y = x ^ 3
D(y)
```

\begin{equation}
\mathrm{\frac{d}{d x}}\left( x^{3} \right)
\end{equation}

```julia
expand_derivatives(D(y))
```

\begin{equation}
3 x^{2}
\end{equation}

Does this answer match with what you calculated by hand? Next we try automatic differentiation.

```julia
f(x) = x ^ 3

f'(x)
```

\begin{equation}
3 x^{2}
\end{equation}

It seems that `Zygote` gives the same answer. Let us evaluate this function at a particular point with `ForwardDiff`.

```julia
f'(5) == ForwardDiff.derivative.(f, 5) # evaluated at the point x = 5
```

    true

```julia
f'(5)
```

    75

#### Chain rule

The chain rule is a bit more complicated then some of the other rules, but it is used frequently in economics. If we have a function $f(x) = p(q(x)) = (p \circ q)(x)$ which is a composite of two differentiable functions $p(x)$ and $q(x)$ then the derivative at $x_0$ according to the chain rule is,

$$
f^{\prime } (x_0 )=p^{\prime } (q(x_0 ))\cdot q^{\prime } (x_0 )
$$

A good example of where this is applicable is the function

$$
f(x) = \sqrt{(5x - 8)}
$$

In this case we have a composition of two functions, $p(x) = \sqrt{x}$ and $q(x) = 5x - 8$. Here we need to take the derivatives separately and combine with our rule. We start with $p'(x)$. We haven't really talked about taking the derivative of a square root. So how do we proceed? Well, our square root can actually be written as $p(x)^{1/2}$. Now we can use our power rule from before to get $p'(x) = \frac{x^{-1/2}}{2}$. Next we need to take the derivative of $q(x)$, which gives is $q'(x) = 5$ using the power rule (since $5 = 5 \cdot x^{(1 - 1)}$). Now we combine all the components as per the chain rule,

$$
\begin{align*}
f'\left( x \right) & = p'\left( {q\left( x \right)} \right)\,\,q'\left( x \right)\\
&  = p'\left( {5x - 8} \right)\,\,q'\left( x \right)\\
&  = \frac{1}{2}{\left( {5x - 8} \right)^{ - \frac{1}{2}}}\,\left( 5 \right)\\
&  = \frac{1}{{2\sqrt {5x - 8} }}\,\,\left( 5 \right)\\
&  = \frac{5}{{2\sqrt {5x - 8} }}\end{align*}
$$

Let us check whether the computer gives the same answers,

```julia
y = sqrt(5x - 8)
D(y)
```

\begin{equation}
\frac{dsqrt(-8 + 5x)}{dx}
\end{equation}

```julia
expand_derivatives(D(y))
```

\begin{equation}
\frac{\frac{5}{2}}{\sqrt{-8 + 5 x}}
\end{equation}

```julia
f(x) = sqrt(5x - 8)

f'(x)
```

\begin{equation}
\frac{5}{2 \sqrt{-8 + 5 x}}
\end{equation}

#### Sum (difference) rule

Given functions $p$ and $q$ that are differentiable at $x$ and with $f(x) = p(x) + q(x)$ we have that the derivative according to the sum rule is given by,

$$
f^{\prime } (x)=p^{\prime } (x)+q^{\prime } (x)
$$

Let us consider an example with the application of this rule. Find the derivative of $f(x) = 2x^5 + 7$. In this case, $p(x) = 2x^5$ and $q(x) = 7$.

$$
\begin{align*}
f'(x)&=\dfrac{d}{dx}\left(2x^5+7\right)\\
&=\dfrac{d}{dx}(2x^5)+\dfrac{d}{dx}(7) & & \text{Apply the sum rule.}\\
&=2\dfrac{d}{dx}(x^5)+\dfrac{d}{dx}(7) & & \text{Apply the constant multiple rule.}\\
&=2(5x^4)+0 & & \text{Apply the power rule and the constant rule.}\\
&=10x^4 & & Simplify.
\end{align*}
$$

As usual, we check the answer on the computer.

```julia
y = 2x^5 + 7
D(y)
```

\begin{equation}
\mathrm{\frac{d}{d x}}\left( 7 + 2 x^{5} \right)
\end{equation}

```julia
expand_derivatives(D(y))
```

\begin{equation}
10 x^{4}
\end{equation}

```julia
f(x) = 2x^5 + 7

f'(x)
```

\begin{equation}
10 x^{4}
\end{equation}

These are the most important rules that we will use the most. The following rules are also used, but less frequently.

#### Product rule

Given functions $p$ and $q$ that are differentiable at $x$ and with $f(x)=p(x)\cdot q(x)$ we have that the derivative is,

$$
f^{\prime } (x)=p^{\prime } (x)\cdot q(x)+p(x)\cdot q^{\prime } (x)
$$

Consider the following example, find $f'(x) = (x^2+2)(3x^3−5x)$ by applying the product rule.

If we set $p(x)=x2+2$ and $q(x)=3x3−5x$, then $p'(x)=2x$ and $q'(x)=9x2−5$ and therefore,

$$
f'(x)=p'(x)q(x)+q'(x)p(x)=(2x)(3x^3−5x)+(9x^2−5)(x^2+2)
$$

If we simplify this we have $f'(x) = 15x4+3x2−10$

Check against the answer from computer,

```julia
y = (x^2 + 2)*(3x^3 − 5x)
D(y)
```

\begin{equation}
\mathrm{\frac{d}{d x}}\left( \left( 2 + x^{2} \right) \left( 3 x^{3} - 5 x \right) \right)
\end{equation}

```julia
expand_derivatives(D(y))
```

\begin{equation}
\left( 2 + x^{2} \right) \left( -5 + 9 x^{2} \right) + 2 x \left( 3 x^{3} - 5 x \right)
\end{equation}

Check against the automatic differentiation answer,

```julia
f(x) = (x^2 + 2)*(3x^3 − 5x)

f'(x)
```

\begin{equation}
-10 - 5 x^{2} + 2 x \left( 3 x^{3} - 5 x \right) + 3 x^{2} \left( 6 + 3 x^{2} \right)
\end{equation}

#### Exponential

The exponential function is used a lot in economics. We will often have functions such as $f(x) = \exp(a \cdot x)$. The derivative in this case is given by,

$$
f'(x) = a \cdot \exp(a \cdot x)
$$

In economics you will find the exponential function in the compounding of interest. The accumulated value of interest that is compounded continuously is given by

$$
A(t) = Pe^{rt}
$$

If you wanted to take a derivative of this function with respect to the variable $t$ then you would get the following answer,

$$
A'(t) = r \cdot Pe^{rt}
$$

```julia
P = 10; r = 0.05 # choose arbitrary values for P and r

f(x) = P * exp(r*x)

f'(x)
```

\begin{equation}
0.5 e^{0.05 x}
\end{equation}

#### Logarithm

Finally, we have the derivative of a log function, which is used almost everywhere in economics. If we have the function $f(x) = \log(x)$ then the derivative is given by,

$$
f'(x) = 1 / x
$$

```julia
f(x) = log(x)

f'(x)
```

\begin{equation}
\frac{1}{x}
\end{equation}

## Higher order derivatives

With these rules you should be able to tackle most problems that involve derivatives. These derivatives that we calculated were first order derivatives. We can actually take the derivative again with respect to the variable of interest and then we would have **second order derivatives**. The second order derivative is then a derivative of the first order derivative.

While the first order derivative provides a rate of change (often a slope of a tangent line), the second order derivative gives us information on the rate of change of the rate of change.

### Example: Cobb-Douglas

We can illustrate the usage of derivatives in economics with a basic example. Consider the Cobb-Douglas production function (which we will see again in the notebook on the Solow model). This is a simple explicit form for a production function that we often encounter in economics. The form of the function is as follows,

$$
F(K,L)=K^{\alpha } \cdot L^{\beta}
$$

where $F$ is the production function and it is a function of the capital ($K$) and labour ($L$) inputs. In order to make this a univariate example, we fix the value of $K$, so that it isn't variable. The derivative with respect to labour gives us the marginal product of labour. The derivative is then given as,

$$
\frac{{\textrm{d}} Y(K,L)}{dL } = MPL(K,L)=(\beta)\cdot K^{\alpha } \cdot L^{\beta - 1}
$$

The derivative that we have obtained above is simply another function and we can take additional derivatives. The second order derivative is given by,

$$
\frac{{\textrm{d}}^2 Y(K,L)}{dL^2 }=(\beta )\cdot (\beta -1)\cdot K^{\alpha } \cdot L^{\beta -2}
$$

We can calculate the derivatives with `Symbolics` as well,

```julia
@variables K L α β

D = Differential(L)

Y = K ^ (α) * L ^ (β)
D(Y)
```

\begin{equation}
\mathrm{\frac{d}{d L}}\left( L^{\beta} K^{\alpha} \right)
\end{equation}

```julia
expand_derivatives(D(Y))
```

\begin{equation}
L^{-1 + \beta} K^{\alpha} \beta
\end{equation}

```julia
D2 = Differential(L) * Differential(L)
```

    Differential(L) ∘ Differential(L)

```julia
D2(Y)
```

\begin{equation}
\frac{d(d / (d * L))(L ^ \beta * K ^ \alpha)}{dL}
\end{equation}

```julia
expand_derivatives(D2(Y)) # second order derivative
```

\begin{equation}
L^{-2 + \beta} K^{\alpha} \beta \left( -1 + \beta \right)
\end{equation}

We seem to get the same answer as above if we do a quick check.

### Curvature and second derivative

For our Cobb-Douglas example, we will graph the second derivative when $\beta = 0.5$. The production function is concave. For any function that is twice continuously differentiable, the function is concave if and only if its second derivative is non-positive.

```julia
α = 0.5; β = 0.5

K = 1

@variables L

D = Differential(L)

Y = K ^ (α) * L ^ (β)
D(Y)

diff_1 = expand_derivatives(D(Y));
diff_2 = expand_derivatives(D2(Y));
```

```julia
plot(Y, 0.2, 3, label = "f(x)", lw = 2, alpha = 0.7)
plot!(diff_1, 0.2, 3, label = "First derivative", lw = 2, ls = :dashdot, alpha = 0.7)
plot!(diff_2, 0.2, 3, label = "Second derivative", legend = :bottomright, lw = 2, ls = :dash, alpha = 0.7)
```

We see that the first derivative is positive, which indicates that the slope of the function is always positive. In terms of the curvature though, the second derivative is always negative, which indicates that the slope of the function is positive but decreasing with an increase in labour. This means that the marginal return to labour is positive but decreasing with an increase in labour.

## Conditions for local maxima

Previously we looked at the definitions for global and local maxima. In this section we describe the necessary conditions for optimality with respect to derivatives. In the univariate case (with only one variable), which we have mostly looked at thus far, the first and second order necessary condition for a local maxima is the following,

1. **FONC:** $f'(x^*) =0$
2. **SONC** $f''(x^*) \leq 0$ (and $f''(x^*) \geq 0$ for local minima)
2. (**SOSC** $f''(x^*) < 0$ (and $f''(x^*) > 0$ for local minima))

#### Examples

An easier way to determine the turning points for functions is to use the first and second order conditions. Say that we want to determine the minima and maxima of the following function,

$$
f(x) = 200 - 30x + 8x^2 - 1/2x^3
$$

```julia
f(x) = 200 - 30 * x + 8 * x^2 - 0.5 * x^3
```

    f (generic function with 1 method)

```julia
plot(f, 0, 12, lw = 2, legend = false)
```

It appears from the graph that there are going to be two turning points. We want to calculate the local maximum in the interval $[0, 12]$. First we take the first order condition to find the place where the slope is zero. Then we look at the second order condition to determine if it is a maximum or minimum. For this we are going to be using a root finding package in Julia, called `Roots`.

```julia
roots = find_zeros(f', (0, 12)) # we are essentially setting f'(x) = 0 with this package.
```

    2-element Vector{Float64}:
     2.427400704306218
     8.239265962360449

```julia
plot(f, 0, 12, lw = 2, legend = false)
vline!([roots[1], roots[2]], lw = 2, alpha = 0.6, ls = :dash, color = :black, xticks = ([roots[1], roots[2]]))
```

Next we need to check if the second order derivatives evaluated at the turning points are positive or negative in order to determine if they are maxima or minima.

```julia
f''(roots[1])
```

    8.717797887081346

This value is positive, so we don't believe that this is a maximum. We are looking for the maximum, and for that to be true we must have that the second order derivative is negative.

```julia
f''(roots[2])
```

    -8.717797887081346

In this case we see that the value is negative, which indicates to us that this is a **maximum**. We can also look at original quadratic utility function to see whether that was a maximum. The code to find this maximum is the same as above.

```julia
α = 0.2

U(x) = x .- α .* x .^ 2

utility_roots = find_zeros(U', (-10, 10))
```

    1-element Vector{Float64}:
     2.5

In this case we get the correct answer, which differs marginally from the value we retrieved from the brute force grid method we used before.

#### Exercise

Total revenue of the firm is given by $TR = 20Q - 2Q^2$ and the total cost function is $TC = Q^3 - 8Q^{2} + 20Q + 2$. Find the level of output that maximises total profit. First do this by hand and then try to solve the problem on the computer and see if you get the same answer. Verify that this level of output satisfies the condition that marginal revenue is equal to marginal cost.

## Functions of multiple variables

Here we talk about how to deal with functions $f: \mathbb{R}^{n} \rightarrow \mathbb{R}$. This means functions that take values in $\mathbb{R}^n$ and return a value along the real line. An example of this would be,

$$
f(x, y) = x^{2} + y^{2}
$$

This is a function that takes in two inputs and returns one output. We will mostly be working with functions that have two inputs. In other words, those functions for which $n = 2$, so that we are operating initially in $\mathbb{R}^2$. Before we continue, I am going to be using two helper functions, that will allow us to easily create some 3D plots. Do not worry too much about the code in the `minmax()` and `mmplotter()` functions, they are a bit more difficult to understand fully on the first reading. You are, however, equipped with all the tools to understand what is happening. This will simply require a careful reading of the first tutorial to figure out.

### Plotting in 3D

```julia
function minmax()

	v = collect(range(-2, stop = 2, length = 30))  # values
	mini = [x^2 + y^2 for x in v, y in v]
	maxi = -mini   # max is just negative min
	saddle = [x^2 + y^3 for x in v, y in v]

	return Dict(:x => v,:min => mini, :max => maxi, :saddle => saddle)
end;
```

```julia
function mmplotter(s::Symbol)

    d = minmax()

    surface(d[:x], d[:x], d[s], fillalpha = 0.7, legend = false, fillcolor =:viridis)
end;
```

```julia
function mmcontour(s::Symbol)

    d = minmax()

    contour(d[:x], d[:x], d[s], fillalpha = 0.5, legend = false, fill = true, fillcolor = :viridis, color = :black)
end;
```

We can plot a set of points $(x, y, f(x, y))$ using a surface plot. The following provides the plot for the function $f(x, y) = x^{2} + y^{2}$ that we mentioned earlier.

```julia
mmplotter(:max)
```

In addition to the surface plot, we can have a contour plot, which shows a top down view of the surface plot above.

```julia
mmcontour(:max)
```

### Partial derivatives

For a function $f: \mathbb{R}^{n} \rightarrow \mathbb{R}$ the concept of a derivative is extended to partial derivatives for $n$ variables. The partial of $x$ is defined by holding $y$ constant while a derivative in $x$ is taken. We saw an example of this with our calculation of marginal product of labour in a previous example.

The partial derivative of a function $f$ with respect to the variable $x$, can be written as $\frac{\partial{f}}{\partial{x}}$, which is defined as,

$$
\dfrac{∂f}{∂x}=f_x(x,y)=\lim_{h→0}\dfrac{f(x+h,y)−f(x,y)}{h}
$$

In this definition we see a new symbol, $\partial$, which indicates a partial derivative. In this case we can also write the derivative with respect to $y$ so there are two partial derivatives for this function, since it is a function of two variables.

The gradient of $f$, which is referred to as $\nabla f$, is the vector valued function of partial derivatives $[\partial f / \partial x, \partial f / \partial y]$.

#### Calculating partial derivatives by hand

If we wanted to calculate the partial derivative for some function we would have to do the following. Consider the following function,

$$
f(x,y)=x^2−3xy+2y^2−4x+5y−12
$$

The partial derivative of $f$ with respect to $x$ can be calculated using our definition above. First calculate $f(x + h, y)$,

$$
\begin{align*} f(x+h,y) &=(x+h)^2−3(x+h)y+2y^2−4(x+h)+5y−12 \\ &=x^2+2xh+h^2−3xy−3hy+2y^2−4x−4h+5y−12. \end{align*}
$$

Now we need to substitute this into our initial equation above and then simplify,

$$
\begin{align*} \dfrac{∂f}{∂x} &=\lim_{h→0}\dfrac{f(x+h,y)−f(x,y)}{h} \\
&=\lim_{h→0}\dfrac{(x^2+2xh+h^2−3xy−3hy+2y^2−4x−4h+5y−12)−(x^2−3xy+2y^2−4x+5y−12)}{h} \\ &=\lim_{h→0}\dfrac{x^2+2xh+h^2−3xy−3hy+2y^2−4x−4h+5y−12−x^2+3xy−2y^2+4x−5y+12}{h} \\
&=\lim_{h→0}\dfrac{2xh+h^2−3hy−4h}{h}\\
&=\lim_{h→0}\dfrac{h(2x+h−3y−4)}{h} \\
&=\lim_{h→0}(2x+h−3y−4) \\
&=2x−3y−4. \end{align*}
$$

The same thing can be done for the partial derivative of $f$ with respect to $y$. As an exercise, you might want to try and do this for $y$. The answer should be $-3x + 4y + 5$.

Now, we won't always use this method to find the partial derivative. There are easier ways than using the definition above. For the partial derivative, we can think about it as taking a derivative of the one variable while holding the other constant. Exactly like we did for our example with the Cobb-Douglas production function.

So the easier way to take a partial derivative of $f$ with respect to $x$ would be to treat the variable $y$ like it was a constant and apply the usual single variable calculus rules.

Let us consider the individual components of the function.

In the case of $x^{2}$, we would have that the derivative is $2x$.

For $-3xy$ we treat the $y$ like a constant, so the derivative is $-3y$.

The next term, $2y^2$ falls away completely since it is constant. The same is true for $5y$ and $12$.

Finally, we are left with $-4x$, with a derivative given by $-4$.

If we then add these components together, we get our answer $2x - 3y - 4$

#### Differentiation via the computer

We can compute the partial derivatives using the `Symbolics` package as follows,

```julia
@variables x y

l = x^2 − 3x*y + 2y^2 − 4x + 5y − 12

H = Differential(x);
I = Differential(y);
```

```julia
expand_derivatives(H(l)) # partial derivative of f(x, y) wrt to x
```

\begin{equation}
-4 + 2 x - 3 y
\end{equation}

Alternatively we can use automatic differentiation to calculate the partial derivatives.

```julia
f(x) = x[1]^2 − 3x[1]*x[2] + 2x[2]^2 − 4x[2] + 5x[2] − 12

∇f(x) = ForwardDiff.gradient(f, x)

∂f_∂x(x, y) = ∇f([x, y])[1];
∂f_∂y(x, y) = ∇f([x, y])[2];
```

With this we can now find the gradient at any particular point, say for example we want the partial derivative of $f(x, y)$ with respect to $x$ at the point $(1, 2)$ we can do the following,

```julia
∂f_∂x(1, 2)
```

    -4

This has been a brief introduction to unconstrained optimisation. We will continue this discussion in our next tutorial.