\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[backend=biber]{biblatex}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{dsfont}
\addbibresource{bib.bib}
\setlength{\parindent}{0em}
\bibliography{bib}
\setlength{\parskip}{6pt}
\usepackage[margin=1.0in]{geometry}
\usepackage{graphicx}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{wrapfig}
\usepackage{url}

\title{Intro to deep learning with PyTorch}
\author{Miguel A. Saavedra-Ruiz}
\date{May 2020}
\linespread{1.0}

\nocite{*}

\begin{document}

\maketitle

\section*{Convolutional Neural Networks}

Convolutional Neural Networks (CNNs) are a variation of neural networks with the ability to efficiently operate over images. CNNs can look at images as a whole and learn to identify spatial patterns such as prominent colors and shapes, or whether a texture is fuzzy or smooth and so on. The shapes and colors that define any image and any object in an image are often called \textbf{features} Fig. \ref{fig:f1}.

An example of a feature would be what do we look at to distinguish a cat and a dog. These characteristics might be the shape of the eyes, the size, and how they move are just a couple of examples of visual features.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.45\textwidth,height=0.45\textheight,keepaspectratio]{images/features.png}
    \captionsetup{justification=centering}
    \caption{Features in an image}
    \label{fig:f1}
\end{figure}

To understand neural networks, it is important to also understand how an image is represented in a computer. In the case of gray scale images, these are interpreted by a computer as an array. A grid of values for each grid cell is called a pixel, and each pixel has a numerical value. In the MNIST database, each image is 28 pixels height and wide. Therefore, these images are understood by a computer as a 28 by 28 array. In a typical gray scale image, white pixels are encoded as the value 255, and black pixels are encoded as zero. Gray pixels fall somewhere in between, with light-gray being closer to 255. This description can be easily undertood with Fig. \ref{fig:f2}

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.4\textwidth,height=0.4\textheight,keepaspectratio]{images/image_representation.png}
    \captionsetup{justification=centering}
    \caption{Image representation as a matrix}
    \label{fig:f2}
\end{figure}

Nevertheless, the images in the MNIST dataset have actually gone through a quick pre-processing step. They've been re-scaled so that each image has pixel values in a range from zero to one Fig. \ref{fig:f3}. To go from a range of 0-255 to zero and one, it is necessary to divide every pixel value by 25 Eq. \eqref{eq:1}5. This step is called \textbf{normalization}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.4\textwidth,height=0.4\textheight,keepaspectratio]{images/image_norm.png}
    \captionsetup{justification=centering}
    \caption{Image representation as a normalized matrix}
    \label{fig:f3}
\end{figure}

\begin{equation}
NormX_{ij} = \frac{X_{ij}}{max(X)} \label{eq:1}
\end{equation}

Normalization is a useful tool to help the algorithm to train better. Thiss tep is quite useful due to neural networks rely on gradient calculations. These networks are trying to learn how important or how weighty a certain pixel should be in determining the class of an image. Normalizing the pixel values helps these gradient calculations stay consistent, and not get so large that they slow down or prevent a network from training.

Recall that to classify objects, the most common technique used for this was to use a MLP or fully connected neural network. However, these networks require 1-dimensional data. Hence, it is necessary to convert any array image into a vector. This process is called \textbf{flattening} and can be seen in Fig. \ref{fig:f4}. In this example, a four-by-four image which is a matrix with 16 pixel values. Instead of representing this as a four-by-four matrix, it is possible to construct a vector with 16 entries, where the first first entries of the vector correspond to the first row of the old array. The second four entries correspond to the second row and so on.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.4\textwidth,height=0.4\textheight,keepaspectratio]{images/flattening.png}
    \captionsetup{justification=centering}
    \caption{Flattening a simple matrix into a vector}
    \label{fig:f4}
\end{figure}

Once an image has been flattened, it is easy to pass the data to a MLP and obtain the classification result of the image.

It is important to note that Pytorch does Data normalization by subtracting the mean (the average of all pixel values) from each pixel, and then dividing the result by the standard deviation of all the pixel values.

\[\frac{P_{ij} - \mu}{\sigma}\]

Where for \(\mu = 0.5\) and \(\sigma=0.5\) the minimum (pixel coordinate equals 0) and maximum (pixel coordinate equals 1) values of the normalization are.

\[\frac{1 - 0.5}{0.5} = 1\]
\[\frac{0 - 0.5}{0.5} = -1\]

Resulting in a normalization of \([-1,1]\) for all the pixel values in the image.

A feasible network architecture to train the neural network is the one presented in Fig. \ref{fig:f50}. A MLP with 784 input units (number of inputs after flattening a MNIST image), two hidden layers with 512 nodes each and 10 output layers with a softmax activation function (one output per character) to produce a probability distribution in the final layer of the network.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.55\textwidth,height=0.55\textheight,keepaspectratio]{images/mlp.png}
    \captionsetup{justification=centering}
    \caption{MLP to train a MNIST image}
    \label{fig:f5}
\end{figure}

Recall that neural networks learn from mistakes, hence, the next list of stems gives a gives of how to optimize the weights of a neural network.

\begin{enumerate}
  \item Loss: Measure any mistake between a predicted and true class
  \item Backpropagation: Quantify how bad a particular weight is in making a mistake
  \item Optimization: Gives a way to calculate a better weight value
\end{enumerate}

As this problem is categorized as a multi-classes problem, the \textbf{loss function} needed is called \textbf{categorical cross-entropy loss}. This loss function is defined as the negative log of the output layer (with softmax activation function).

Therefore, if the probability of a node is \(p(x_i) = 0.162\) the negative log is given in Eq. \eqref{eq:2}.

\begin{equation}
loss = -\log(0.162) = 1.82 \label{eq:2}
\end{equation}

If the probability is something like \(p(x_i) = 0.441\), then the loss would be \(loss = 0.819\)

It is possible to conclude that the categorical cross-entropy loss is \textbf{lower} when the prediction and label agree and \textbf{higher} when the prediction and label disagree (a probability close to one yields to a low loss value).

The optimizer is used to minimize the error of the loss function and change the weights towards the minimum error expression. In the Fig. \ref{fig:f6}, in the top right box it is possible to see a list of different optimizer and their performance minimizing the error. Each optimizer has its own benefits and the selection of one or other depends on the problem.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.4\textwidth,height=0.4\textheight,keepaspectratio]{images/optimizer.png}
    \captionsetup{justification=centering}
    \caption{Optimizer used to train a neural network}
    \label{fig:f6}
\end{figure}

A complete solution to the problem described before can be seen in \textit{mnist\_mlp\_exercise.ipynb}.

To decide whether the model is doing well or bad during training, a new set needs to be introduced. \textbf{The validation set} is defined as a sub-sample of the training set, usually the 20\% of the training set. During training, the training set is used to change the weights and the validation set checks the performance of the model to avoid over-fitting. Finally, once a model has been chosen, a test set is used (this set has never been seen by the network before) to check the accuracy of the trained model. The pipeline described before can be seen in Fig. \ref{fig:f7}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.55\textwidth,height=0.55\textheight,keepaspectratio]{images/sets.png}
    \captionsetup{justification=centering}
    \caption{Sets the evaluate and train a model}
    \label{fig:f7}
\end{figure}

Overall, the steps to train a neural network are:

\begin{itemize}
  \item Visualize data
  \item Pre-process normalize transform
  \item Define a model
  \item Train the model define loss \& optimization function
  \item Save the best model
  \item Test the model
\end{itemize}

Despite that flattening images and using multi-layer perceptrons might sound like a good solution to deal with images, these techniques have a set of drawbacks which affects its implementation in real world images. For example, the MLP only uses fully-connected layers, which for the case of the MNIST dataset is almost half a million parameters. Furthermore, this sort of architecture only support data in the form on vectors whereas the image is a matrix with spatial information which can be relevant around its close neighborhood.

To sort this problems a new architecture is employed and this is \textbf{called convolutional neural networks (CNNs)}. This networks instead of use fully connected layers use sparsely connected layers where the connection between layers are informed by the 2-D structure of the image matrix. Moreover, this network architecture accepts matrices as inputs.

To understand the concept of sparsely connected and its benefits, consider breaking the image into four regions Fig. \ref{fig:f8}. The image is painted in four sections: red, green, yellow, and blue. Then, each hidden node could be connected to only the pixels in one of these four regions.Here, each headed node sees only a quarter of the original image. With this regional breakdown and the assignment of small local groups of pixels to different hidden nodes, every hidden node finds patterns in only one of the four regions in the image. Then, each hidden node still reports to the output layer where the output layer combines the findings for the discovered patterns learned separately in each region

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.55\textwidth,height=0.55\textheight,keepaspectratio]{images/sparsely.png}
    \captionsetup{justification=centering}
    \caption{Sets the evaluate and train a model}
    \label{fig:f8}
\end{figure}

This locally connected layer uses far fewer parameters than a densely connected layer. It's less prone to overfitting and truly understands how to tease out the patterns contained in image data.

Rearranging the vector as a matrix it is possible to see the relationships between the nodes in each layer. It is possible to expand the number of patterns that the network is able to detect while still making use of the 2-D structure to selectively and conservatively add weights to the model by introducing more hidden nodes, where each is still confined to analyzing a single small region within the image. The red nodes in the hidden layer are still only connected to the red nodes in the input layer, with the same color coding for all other colors Fig. \ref{fig:f9}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.55\textwidth,height=0.55\textheight,keepaspectratio]{images/matrix_input.png}
    \captionsetup{justification=centering}
    \caption{Arranging the matrix as an input for a CNNs}
    \label{fig:f9}
\end{figure}

Recall that by expanding the number of nodes in the hidden layer it is possible to discover more complex patterns in the data. In this case, two collections of hidden nodes is being used where each collection contains nodes responsible for examining a different region of the image as shown by Fig. \ref{fig:f10}. It will prove useful to have each of the hidden nodes within a collection share a common group of weights. The idea being that different regions within the image can share the same kind of information. In other words, every pattern that's relevant towards understanding the image could appear anywhere within the image.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.55\textwidth,height=0.55\textheight,keepaspectratio]{images/shared_weights.png}
    \captionsetup{justification=centering}
    \caption{Sharing weights in a CNNs}
    \label{fig:f10}
\end{figure}

A convolutional neural network is a special kind of neural network which can remember spatial information. The common neural networks only look at individual inputs, whereas convolutional neural networks, can look at an image as a whole, or in patches and analyze groups of pixels at a time.

The key to preserving the \textbf{spatial information} in CNNs is called the convolutional layer. A \textbf{convolutional layer} applies a series of different image filters also known as \textbf{convolutional kernels} to an input image. The resulting filtered images have different appearances. The filters may have extracted features like the edges of objects in that image, or the colors that distinguish the different classes of images.

Spatial patterns in an image can be for example color or shape. The changes in shape can be described as the change of intensity in a group of pixels. These changes can be used to detect objects in an image. It is possible to identify these changes with convolutional kernels with a specific form and values.

To measure how the pixels change in an image, the frecuency can be used to identify the rate of change. A high frequency image Fig. \ref{fig:f11} is one where the intensity changes a lot  and the level of brightness changes quickly from one pixel to the next (pink square). A low frequency image may be one that is relatively uniform in brightness or changes very slowly (blue square).

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.55\textwidth,height=0.55\textheight,keepaspectratio]{images/frec.png}
    \captionsetup{justification=centering}
    \caption{High and low frecuency image patterns}
    \label{fig:f11}
\end{figure}

High-frequency components in an image also correspond to the edges of objects and can be used to classify them.

In an image, \textbf{filters} are used to filter out unwanted information or amplify features of interest as corners. An example of a filter is a high-pass filter which is used to sharpen an image and enhance high-frecuency parts of an image. In the Fig. \ref{fig:f12} the high-pass filter has enhanced the areas of the image where rapid changes have been presented (emphasizing edges). It is important to mention that this simple filters have to be used on one-channel images.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.55\textwidth,height=0.55\textheight,keepaspectratio]{images/high_pass.png}
    \captionsetup{justification=centering}
    \caption{Image after a High-pass filter}
    \label{fig:f12}
\end{figure}

A filter or convolutional filter is just a matrix of numbers that modifies the image. A high-ass filter is the one shown in Fig. \ref{fig:f13}. For edge detection, all the values inside the filter must add to zero.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.2\textwidth,height=0.2\textheight,keepaspectratio]{images/filter.png}
    \captionsetup{justification=centering}
    \caption{High-pass filter}
    \label{fig:f13}
\end{figure}

To apply a filter to an image an operation called convolution needs to be applied to the image. In this operation, the kernel sweeps the whole image, multiplying value by value the pixel and kernel. Finally, all the values are added and the result is places in a new image which will counting the result of the convolution. The image Fig. \ref{fig:f14} shows this convolution process in a simple set of pixels in an image. Is is important to mention that the convolution is denoted by the symbol \(\*\).

\[K * I(x,y) = \text{resulting image}\]

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.55\textwidth,height=0.55\textheight,keepaspectratio]{images/convolution.png}
    \captionsetup{justification=centering}
    \caption{The convolution process}
    \label{fig:f14}
\end{figure}

When applying this process to the corners of an image leads to unknown pixel values, techniques as extend, padding or crop are applied in the input image to generate an image of the same size of the input one.

Getting more into detail, CNNs are a kind of deep learning model that can learn to do things like image classification and object recognition. They keep track of spatial information and learn to extract features like the edges of objects in something called a convolutional layer. The Fig. \ref{fig:f15} shows a simple CNN structure, made of multiple layers.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.65\textwidth,height=0.65\textheight,keepaspectratio]{images/layers_cnn.png}
    \captionsetup{justification=centering}
    \caption{Layers in a CNN}
    \label{fig:f15}
\end{figure}

The convolutional layer is produced by applying a series of many different image filters, also known as convolutional kernels, to an input image Fig. \ref{fig:f16}. In this case, four kernels mean four differently filtered output images.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.55\textwidth,height=0.55\textheight,keepaspectratio]{images/conv_layer.png}
    \captionsetup{justification=centering}
    \caption{Convolutional layer}
    \label{fig:f16}
\end{figure}

When these outputs are stacked it is possible to form a complete convolutional layer with a depth of four Fig. \ref{fig:f17}

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.45\textwidth,height=0.45\textheight,keepaspectratio]{images/depth.png}
    \captionsetup{justification=centering}
    \caption{Depth in a convolutional layer}
    \label{fig:f17}
\end{figure}

The main difference between a filter and a convolutional kernel is that neural networks will actually learn the best filter weights as they train on a set of image data.

To see an example of a convolutional network in a grayscale image in pytorch, see the notebook \textit{conv\_visualization.ipynb}.

If the image has three channels (RGB) the filters need to be stacked in such way that there is a filter for each channel. This means that if the image has a depth of 3, the set of filters needed for this task will also have a depth of three. The result of this stacked filters will be the sum of the convolution of all the channels. Therefore for an input image with \(depth = 3\) and a set of filters \((20x4x4)\) where 20 is the amount of stacked filters and 4x4 is the size of the filter, the result will be a set of 20 images. A convolutional stacked filter can be seen in Fig. \ref{fig:f18}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.45\textwidth,height=0.45\textheight,keepaspectratio]{images/conv_2d.png}
    \captionsetup{justification=centering}
    \caption{Convolution of RGB image}
    \label{fig:f18}
\end{figure}

In the Fig. \ref{fig:f19} is a brief explanation of how the filters operates through a 3-channels image to generate a single-channel output as the sum of the convolution of the three channels.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.75\textwidth,height=0.75\textheight,keepaspectratio]{images/conv_operation.png}
    \captionsetup{justification=centering}
    \caption{Result of a convolution with depth 3 is a image of one single channel}
    \label{fig:f19}
\end{figure}

It is important to mention that the multiple 2D outputs of a convolutional layer can be stacked together to generate inputs to more convolutional layers and discover more patterns among the patterns and improve the detection of features in the image. As the fully connected layers, convolutional layers have weights and bias (kernels) which are initially randomly generated and can be trained to achieve a specific task at hand. Overall, training a CNNs determine what kind of patterns it needs to detect based on the loss function, therefore, the filters will be learned based on the given data in the training process.

Besides the filter size and amount of filters, it is possible to tune additional parameters. One parameter is the \textbf{stride} and this is defined as the amount by which the filter slides over the image. For example, a stride of one means that the filter will slide only one pixel horizontally and vertically at each move. A stride of one will result in a feature map of almost the same size of the input image, whereas a stride of 2 will shrink the image in both width and height reducing the feature map size.

Another important parameter to use in CNNs is the \textbf{padding}. Zero padding can be added to an image with the objective to maintain the original image size in the output feature map. This process will allows a better extraction of features and avoid the loss of information in corners. To pad an image it is just necessary to add zeros in the surroundings of the image Fig. \ref{fig:f20}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.5\textwidth,height=0.5\textheight,keepaspectratio]{images/padding.png}
    \captionsetup{justification=centering}
    \caption{Zero padding}
    \label{fig:f20}
\end{figure}

In Fig. \ref{fig:f15} is is possible to see that a CNN is composed of Convolutional layers, fully connected and pooling layers. \textbf{Pooling layers} are used as a method to reduce dimensionality and the amount of parameters after a convolution. There are basically two types of pooling layers. The first one is the max pooling layer and it works by just taking the maximum of the pixels contained in the window. Fig. \ref{fig:f21} shows an example of a max pooling layer which extracts the maximum value in each window. In this case, the stride of the max pooling is two. A max pooling layer reduce the size of the feature map to the half, reducing the amount of parameters in the neural network. An example of max pooling can be seen in the script \textit{maxpooling\_visualization.ipynb}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.5\textwidth,height=0.5\textheight,keepaspectratio]{images/max_pooling.png}
    \captionsetup{justification=centering}
    \caption{Max pooling in a feature map}
    \label{fig:f21}
\end{figure}

Overall, the general architecture of a convolutional neural network is a set con convolutional layers (with RELU activation function) and max pooling connected and finally flattened to feed a fully connected neural network Fig. \ref{fig:f22}

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.4\textwidth,height=0.4\textheight,keepaspectratio]{images/architecture.png}
    \captionsetup{justification=centering}
    \caption{CNN common architecture}
    \label{fig:f22}
\end{figure}

It is quite important to recall that the max pooling layer is a good alternative to reduce parameter in a CNN. Nevertheless, it throws away some image information.

An alternative to classification methods that do not discard spatial information and instead learn to spatial relationships between parts is called \textbf{Capsule Networks}. This network architecture provide a way to detect parts of objects in an image and represent spatial relationships between those parts. This means that capsule networks are able to recognize the same object, like a face, in a variety of different poses and with the typical number of features (eyes, nose , mouth) even if they have not seen that pose in training data.

Capsule networks are made of parent and child nodes that build up a complete picture of an object Fig. \ref{fig:f23}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.3\textwidth,height=0.3\textheight,keepaspectratio]{images/faces.png}
    \captionsetup{justification=centering}
    \caption{Capsule Networks}
    \label{fig:f23}
\end{figure}

A very important pre-processing step in CNNs is to resize the images to the same image size. Typically, the images are resized as a square with the spatial dimension equal to a power of two \(2^x\). Fig. \ref{fig:f24} presents the general architecture of a CNN and this is as follows. It began with convolutional layers which detect regional patterns in an image using a series of image filters. Typically a ReLu activation function is applied to the output of these filters to standardize their output values. Then max pooling layers, which appear after convolutional layers to reduce the dimensionality of the input arrays. These new layers, along with fully-connected layers, are often the only layers used in this sort of networks.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.6\textwidth,height=0.6\textheight,keepaspectratio]{images/full_cnn.png}
    \captionsetup{justification=centering}
    \caption{A complete CNN architecture}
    \label{fig:f24}
\end{figure}

It is possible to compute the spatial size of the output volume as a function of the input volume size (W), the receptive field size of the Conv Layer neurons (F), the stride with which they are applied (S), and the amount of zero padding used (P) on the border. The equation of the output is given by Eq. \eqref{eq:3}. For example for a 7x7 input and a 3x3 filter with stride 1 and pad 0 the output will be 5x5. With stride 2 the output would be 3x3.

\begin{equation}
\frac{W - F + 2P} {S} + 1
\label{eq:3}
\end{equation}

The number of parameters in a CNN is defined by Eq. \eqref{eq:4}, where \textbf{K} is the number of filters in the convolutional layer, \textbf{F} is the filter size, \textbf{\(D_{in}\)} corresponds to the depth of the previous layer and \textbf{K} is the bias term.

\begin{equation}
Param = K * F * F * D_{in}  + K
\label{eq:4}
\end{equation}

The idea behind the amount of layers in a CNNs is to extract the most revelant information of an image and finally, flatten it to construct a feature vector which can feed a fully connected layer. The general pipeline of a CNN with a fully connected layer is given in Fig. \ref{fig:f25}

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.75\textwidth,height=0.75\textheight,keepaspectratio]{images/fcn.png}
    \captionsetup{justification=centering}
    \caption{A complete CNN architecture with fully connected layers}
    \label{fig:f25}
\end{figure}

An implementation of a CNN can be seen in \textit{cifar10\_cnn.ipynb} where a fully convolutional neural network was used to classify the Cifar-10 dataset.

It is important that a CNN learns an invariant representation of the image which does not get affected by changes in scale, position, or angle in the interest object. Scale invariance is when the algorithm does not get affected by changes of size of the object in the image. Rotation invariance is immunity to change in the angle of the object in the image. Finally, translation invariance is when the algorithm is not affected by changes in the position of the object in the image frame.

To generate a CNN statically invariant and make it robust against those changes, it is necessary to apply a technique called \textbf{image augmentation}. This technique will allow the creation of artificial data with random rotation, translations and scale changes that will allow the algorithm to generalize better over different variations in an image. The Fig. \ref{fig:f26} shows an example of image augmentation in an image with an avocado, realizing changes in the position and orientation of the object. This technique is quite important because it prevents over fitting and improves generalization of the CNN.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.6\textwidth,height=0.6\textheight,keepaspectratio]{images/augmentation.png}
    \captionsetup{justification=centering}
    \caption{Data augmentation}
    \label{fig:f26}
\end{figure}

The code \textit{cifar10\_cnn\_augmentation.ipynb} shows how to do image augmentation to improve generalization over a set of images.

\printbibliography

\end{document}


         %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
         % Information sheet for the Matlab lab - Maths 6111 %
         %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\documentclass[10pt]{article}
\input ma_no_html_header

\usepackage{color}
\usepackage{hyperref}
\hypersetup{breaklinks=true,colorlinks=true}
%\input ma_header
\setlength{\parindent}{0pt}
\pagestyle{myheadings}
% \markright{
% \protect {\protect \epsfxsize=0.2 true cm \protect \epsffile {dolph.line.eps}}
% \it Maths 3018/6111 - Numerical methods \hfill}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{document}

\thispagestyle{empty}
\begin{center}
\textbf{\Large Maths 3018/6111 - Numerical Methods \\*[8mm]
Worksheet 1 - Solutions}\\*[.8cm]
\end{center}

\section*{Theory}

\begin{enumerate}
\item Write down the $1$, $2$ and $\infty$ vector norms of
  \begin{equation*}
    {\bf v}_1 =
    \begin{pmatrix}
      1 \\ 3 \\ -1
    \end{pmatrix}, \quad
    {\bf v}_2 =
    \begin{pmatrix}
      1 \\ -2
    \end{pmatrix}, \quad
    {\bf v}_3 =
    \begin{pmatrix}
      1 \\ 6 \\ -3 \\ 1
    \end{pmatrix}.
  \end{equation*}
  %
  \begin{center}
    \rule{0.9\textwidth}{.1pt}
  \end{center}
  %
  We know that the 1-norm is the sum of the absolute values, so
  \begin{align*}
    |{\bf v}_1|_1 & = 1 + 3 + 1 = 5, \\
    |{\bf v}_2|_1 & = 1 + 2  = 3, \\
    |{\bf v}_3|_1 & = 1 + 6 + 3 + 1 = 11.
  \end{align*}
  The 2-norm is the square root of the sum of the squares, so
  \begin{align*}
    |{\bf v}_1|_2 & = \sqrt{1^2 + 3^2 + 1^2} = \sqrt{11} \approx 3.3166, \\
    |{\bf v}_2|_2 & = \sqrt{1^2 + 2^2}  = \sqrt{5} \approx 2.2361, \\
    |{\bf v}_3|_2 & = \sqrt{1^2 + 6^2 + 3^2 + 1^2} = \sqrt{47} \approx
    6.8557.
  \end{align*}
  The infinity norm is the maximum absolute value, so
  \begin{align*}
    |{\bf v}_1|_{\infty} & = 3 \\
    |{\bf v}_2|_{\infty} & = 2 \\
    |{\bf v}_3|_{\infty} & = 6.
  \end{align*}
  %
  \begin{center}
    \rule{0.9\textwidth}{.1pt}
  \end{center}
  %
\item Find the $1$ and $\infty$ matrix norms of
  \begin{equation*}
    A_1 =
    \begin{pmatrix}
      1 & 2 \\ 3 & 4
    \end{pmatrix}, \quad
    A_2 =
    \begin{pmatrix}
      -3 & 2 \\ 3 & 6
    \end{pmatrix}
  \end{equation*}
  %
  \begin{center}
    \rule{0.9\textwidth}{.1pt}
  \end{center}
  %
  The 1-norm of a matrix is the maximum of the 1-norms of the column
  vectors. For $A_1$ the 1-norms are 4 and 6 respectively. For $A_2$
  they are 6 and 8 respectively. So we have
  \begin{align*}
    \|A_1\|_1 & = 6, \\
    \|A_2\|_2 & = 8.
  \end{align*}

  The infinity norm of a matrix is the maximum of the 1-norms of the row
  vectors. For $A_1$ the 1-norms are 3 and 7 respectively. For $A_2$
  they are 5 and 9 respectively. So we have
  \begin{align*}
    \|A_1\|_{\infty} & = 7, \\
    \|A_2\|_{\infty} & = 9.
  \end{align*}
  %
  \begin{center}
    \rule{0.9\textwidth}{.1pt}
  \end{center}
  %
\item Find the condition numbers of the above matrices. What does this
  suggest about the numerical behaviour of an algorithm that used such
  a matrix?
  %
  \begin{center}
    \rule{0.9\textwidth}{.1pt}
  \end{center}
  %
  The inverse matrices are
  \begin{align*}
    A_1^{-1} & =
      -\frac{1}{2}
    \begin{pmatrix}
      4 & -2 \\ -3 & 1
    \end{pmatrix}, \\
    A_2^{-1} & =
      -\frac{1}{24}
    \begin{pmatrix}
      6 & -2 \\ -3 & -3
    \end{pmatrix}.
  \end{align*}
  It follows that the matrix norms of the inverse matrices are
  \begin{align*}
    \|A_1^{-1}\|_1 & = \frac{7}{2}, & \|A_2^{-1}\|_1 & = \frac{3}{8}, \\
    \|A_1^{-1}\|_{\infty} & = 3, & \|A_2^{-1}\|_{\infty} & = \frac{1}{3}.
  \end{align*}
  Therefore the condition numbers with respect to the 1-norm are
  \begin{align*}
    K(A_1) & = \|A_1\|_1 \|A_1^{-1}\|_1 \\
           & = 21, \\
    K(A_2) & = \|A_2\|_1 \|A_2^{-1}\|_1 \\
           & = 3,
  \end{align*}
  and the condition numbers with respect to the infinity norm are
  \begin{align*}
    K(A_1) & = \|A_1\|_{\infty} \|A_1^{-1}\|_{\infty} \\
           & = 21, \\
    K(A_2) & = \|A_2\|_{\infty} \|A_2^{-1}\|_{\infty} \\
           & = 3.
  \end{align*}
  In this case they are identical.

  This suggests that if the numerical algorithm used the matrix $A_1$
  then the errors intrinsic in the algorithm would increase by a
  factor of order 10, whilst using $A_2$ would increase them by a
  factor order unity. That is, we expect $A_2$ to be better behaved
  than $A_1$ (this is a rather wooly way of putting it).
  %
  \begin{center}
    \rule{0.9\textwidth}{.1pt}
  \end{center}
  %
\item Explain the difference between direct and indirect methods for
  solving linear systems. Give an example of when the latter may be
  more useful.
  %
  \begin{center}
    \rule{0.9\textwidth}{.1pt}
  \end{center}
  %
  {\it Standard exam question; see, e.g., 07/08}.

  \textit{Direct methods} consist of a finite list of transformations
  of the original matrix of the coefficients that reduce the linear
  systems to one that is easily solved.  \textit{Indirect or iterative
    methods}, consist of algorithms that specify a series of steps,
  possibly infinite, that lead closer and closer to the solution;
  there may not be a guarantee that they ever exactly reach it.  This
  may not seem a very desirable feature until we remember that we
  cannot in any case perfectly represent an exact solution: most
  iterative methods provide us with a highly accurate solution in
  relatively few iterations.

  Large, sparse matrices are ideally solved using iterative methods.
  %
  \begin{center}
    \rule{0.9\textwidth}{.1pt}
  \end{center}
  %
\end{enumerate}

\end{document}


\chapter{Stan Program Style Guide}

\noindent
This chapter describes the preferred style for laying out Stan
models. These are not rules of the language, but simply
recommendations for laying out programs in a text editor.  Although
these recommendations may seem arbitrary, they are similar to those of
many teams for many programming languages.  Like rules for typesetting
text, the goal is to achieve readability without wasting white space
either vertically or horizontally.

\section{Choose a Consistent Style}

The most important point of style is consistency.  Consistent coding
style makes it easier to read not only a single program, but multiple
programs.  So when departing from this style guide, the number one
recommendation is to do so consistently.

\section{Line Length}

Line lengths should not exceed 80 characters.%
%
\footnote{Even 80 characters may be too many for rendering in print;
  for instance, in this manual, the number of code characters that fit
  on a line is about 65.}
%
This is a typical recommendation for many programming language style
guides because it makes it easier to lay out text edit windows side by
side and to view the code on the web without wrapping, easier to view
diffs from version control, etc.  About the only thing that is
sacrificed is laying out expressions on a single line.

\section{File Extensions}

The recommended file extension for Stan model files is \code{.stan}.
For Stan data dump files, the recommended extension is \code{.R}, or
more informatively, \code{.data.R}.

\section{Variable Naming}

The recommended variable naming is to follow C/\Cpp naming
conventions, in which variables are lowercase, with the underscore
character (\Verb|_|) used as a separator.  Thus it is preferred to use
\Verb|sigma_y|, rather than the run together \Verb|sigmay|, camel-case
\Verb|sigmaY|, or capitalized camel-case \Verb|SigmaY|.  Even matrix
variables should be lowercased.

The exception to the lowercasing recommendation, which also follows
the C/\Cpp conventions, is for size constants, for which the
recommended form is a single uppercase letter.  The reason for this is
that it allows the loop variables to match.  So loops over the indices of
an $M \times N$ matrix $a$ would look as follows.
%
\begin{stancode}
for (m in 1:M)
  for (n in 1:N)
     a[m,n] = ...
\end{stancode}

\section{Local Variable Scope}

Declaring local variables in the block in which they are used aids in
understanding programs because it cuts down on the amount of text
scanning or memory required to reunite the declaration and definition.

The following Stan program corresponds to a direct translation of a
BUGS model, which uses a different element of \code{mu} in each
iteration.
%
\begin{stancode}
model {
  real mu[N];
  for (n in 1:N) {
    mu[n] = alpha * x[n] + beta;
    y[n] ~ normal(mu[n],sigma);
  }
}
\end{stancode}
%
Because variables can be reused in Stan and because they should be
declared locally for clarity, this model should be recoded as follows.
%
\begin{stancode}
model {
  for (n in 1:N) {
    real mu;
    mu = alpha * x[n] + beta;
    y[n] ~ normal(mu,sigma);
  }
}
\end{stancode}
%
The local variable can be eliminated altogether, as follows.
%
\begin{stancode}
model {
  for (n in 1:N)
    y[n] ~ normal(alpha * x[n] + beta, sigma);
}
\end{stancode}
%
There is unlikely to be any measurable efficiency difference
between the last two implementations, but both should be a bit
more efficient than the BUGS translation.

\subsubsection{Scope of Compound Structures with Componentwise Assignment}

In the case of local variables for compound structures, such as
arrays, vectors, or matrices, if they are built up component by
component rather than in large chunks, it can be more efficient to
declare a local variable for the structure outside of the block
in which it is used.  This allows it to be allocated once and then
reused.
%
\begin{stancode}
model {
  vector[K] mu;
  for (n in 1:N) {
    for (k in 1:K)
      mu[k] = ...;
    y[n] ~ multi_normal(mu,Sigma);
}
\end{stancode}
%
In this case, the vector \code{mu} will be allocated
outside of both loops, and used a total of \code{N} times.

\section{Parentheses and Brackets}

\subsection{Optional Parentheses for Single-Statement Blocks}

Single-statement blocks can be rendered in one of two ways.  The fully
explicit bracketed way is as follows.
%
\begin{stancode}
for (n in 1:N) {
  y[n] ~ normal(mu,1);
}
\end{stancode}
%
The following statement without brackets has the same effect.
%
\begin{stancode}
for (n in 1:N)
  y[n] ~ normal(mu,1);
\end{stancode}
%
Single-statement blocks can also be written on a single line, as
in the following example.
%
\begin{stancode}
for (n in 1:N) y[n] ~ normal(mu,1);
\end{stancode}
%
These can be much harder to read than the first example. Only use this
style if the statement is very simple, as in this example.  Unless
there are many similar cases, it's almost always clearer to put
each sampling statement on its own line.

Conditional and looping statements may also be written without brackets.

The use of for loops without brackets can be dangerous.  For instance,
consider this program.
%
\begin{stancode}
for (n in 1:N)
  z[n] ~ normal(nu,1);
  y[n] ~ normal(mu,1);
\end{stancode}
%
Because Stan ignores whitespace and the parser completes a statement
as eagerly as possible (just as in C++), the previous program is
equivalent to the following program.
%
\begin{stancode}
for (n in 1:N) {
  z[n] ~ normal(nu,1);
}
y[n] ~ normal(mu,1);
\end{stancode}
%

\subsection{Parentheses in Nested Operator Expressions}

The preferred style for operators minimizes parentheses.  This reduces
clutter in code that can actually make it harder to read expressions.
For example, the expression \code{a~+~b~*~c} is preferred to the
equivalent \code{a~+~(b~*~c)} or \code{(a~+~(b~*~c))}.  The operator
precedences and associativities follow those of pretty much every
programming language including Fortran, C++, R, and Python;  full
details are provided in the reference manual.

Similarly, comparison operators can usually be written with minimal
bracketing, with the form \code{y[n] > 0 || x[n] != 0} preferred to
the bracketed form \code{(y[n] > 0) || (x[n] != 0)}.

\subsection{No Open Brackets on Own Line}

Vertical space is valuable as it controls how much of a program you
can see.  The preferred Stan style is as shown in the previous
section, not as follows.
%
\begin{stancode}
for (n in 1:N)
{
  y[n] ~ normal(mu,1);
}
\end{stancode}
%
This also goes for parameters blocks, transformed data blocks,
which should look as follows.
%
\begin{stancode}
transformed parameters {
  real sigma;
  ...
}
\end{stancode}
%

\section{Conditionals}

Stan supports the full \Cpp-style conditional syntax,
allowing real or integer values to act as conditions, as follows.
%
\begin{stancode}
real x;
...
if (x) {
   // executes if x not equal to 0
   ...
}
\end{stancode}
%

\subsection{Explicit Comparisons of Non-Boolean Conditions}

The preferred form is to write the condition out explicitly for
integer or real values that are not produced as the result of a
comparison or boolean operation, as follows.
%
\begin{stancode}
if (x != 0) ...
\end{stancode}

\section{Functions}

Functions are laid out the same way as in languages such as Java and
\Cpp.  For example,
%
\begin{stancode}
real foo(real x, real y) {
  return sqrt(x * log(y));
}
\end{stancode}
%
The return type is flush left, the parentheses for the arguments are
adjacent to the arguments and function name, and there is a space
after the comma for arguments after the first.  The open curly brace
for the body is on the same line as the function name, following the
layout of loops and conditionals.  The body itself is indented; here
we use two spaces.  The close curly brace appears on its own line.
%
If function names or argument lists are long, they can be
written as
%
\begin{stancode}
matrix
function_to_do_some_hairy_algebra(matrix thingamabob,
                                  vector doohickey2) {
  ...body...
}
\end{stancode}
%
The function starts a new line, under the type.  The arguments are
aligned under each other.

Function documentation should follow the Javadoc and Doxygen styles.
Here's an example repeated from \refsection{documenting-functions}.
%
\begin{stancode}
/**
 * Return a data matrix of specified size with rows
 * corresponding to items and the first column filled
 * with the value 1 to represent the intercept and the
 * remaining columns randomly filled with unit-normal draws.
 *
 * @param N Number of rows correspond to data items
 * @param K Number of predictors, counting the intercept, per
 *          item.
 * @return Simulated predictor matrix.
 */
matrix predictors_rng(int N, int K) {
  ...
\end{stancode}
%
The open comment is \code{/**}, asterisks are aligned below the first
asterisk of the open comment, and the end comment \code{*/} is also
aligned on the asterisk.  The tags \code{@param} and \code{@return}
are used to label function arguments (i.e., parameters) and return
values.

\section{White Space}

Stan allows spaces between elements of a program.  The white space
characters allowed in Stan programs include the space (ASCII
\code{0x20}), line feed (ASCII \code{0x0A}), carriage return
(\code{0x0D}), and tab (\code{0x09}).  Stan treats all whitespace
characters interchangeably, with any sequence of whitespace characters
being syntactically equivalent to a single space character.
Nevertheless, effective use of whitespace is the key to good program
layout.

\subsection{Line Breaks Between Statements and Declarations}

It is dispreferred to have multiple statements or declarations on the
same line, as in the following example.
%
\begin{stancode}
transformed parameters {
  real mu_centered;  real sigma;
  mu = (mu_raw - mean_mu_raw);    sigma = pow(tau,-2);
}
\end{stancode}
%
These should be broken into four separate lines.

\subsection{No Tabs}

Stan programs should not contain tab characters.  They are legal and
may be used anywhere other whitespace occurs.  Using tabs to layout a
program is highly unportable because the number of spaces
represented by a single tab character varies depending on which
program is doing the rendering and how it is configured.

\subsection{Two-Character Indents}

Stan has standardized on two space characters of indentation, which is
the standard convention for C/C++ code.  Another sensible choice is
four spaces, which is the convention for Java and Python.  Just be
consistent.

\subsection{Space Between \code{if} and Condition}

Use a space after \code{if}s.  For instance, use \code{if (x < y) ...}, not
\code{if(x < y) ...}.

\subsection{No Space For Function Calls}

There is no space between a function name and the function it applies
to.  For instance, use \code{normal(0,1)}, not \code{normal (0,1)}.

\subsection{Spaces Around Operators}

There should be spaces around binary operators.  For instance, use
\code{y[1]~=~x}, not \code{y[1]=x}, use \code{(x~+~y)~*~z} not
\code{(x+y)*z}.

\subsection{Breaking Expressions across Lines}

Sometimes expressions are too long to fit on a single line.  In that
case, the recommended form is to break \emph{before} an operator,%
%
\footnote{This is the usual convention in both typesetting and other
  programming languages. Neither R nor BUGS allows breaks before an
  operator because they allow newlines to signal the end of an
  expression or statement.}
%
aligning the operator to indicate scoping.  For example, use the
following form (though not the content; inverting matrices is almost
always a bad idea).
%
\begin{stancode}
target += (y - mu)' * inv(Sigma) * (y - mu);
\end{stancode}
%
Here, the multiplication operator (\code{*}) is aligned to clearly
signal the multiplicands in the product.

For function arguments, break after a comma and line the next
argument up underneath as follows.
%
\begin{stancode}
y[n] ~ normal(alpha + beta * x + gamma * y,
              pow(tau,-0.5));
\end{stancode}
%

\subsection{Optional Spaces after Commas}

Optionally use spaces after commas in function arguments for clarity.
For example, \code{normal(alpha * x[n] + beta,sigma)} can also be
written as \code{normal(alpha~*~x[n]~+~beta,~sigma)}.

\subsection{Unix Newlines}

Wherever possible, Stan programs should use a single line feed
character to separate lines.  All of the Stan developers (so far, at
least) work on Unix-like operating systems and using a standard
newline makes the programs easier for us to read and share.

\subsubsection{Platform Specificity of Newlines}

Newlines are signaled in Unix-like operating systems such as Linux and
Mac OS X with a single line-feed (LF) character (ASCII code point
\code{0x0A}).  Newlines are signaled in Windows using two characters,
a carriage return (CR) character (ASCII code point \code{0x0D})
followed by a line-feed (LF) character.


% $Header: /cvsroot/latex-beamer/latex-beamer/solutions/conference-talks/conference-ornate-20min.en.tex,v 1.6 2004/10/07 20:53:08 tantau Exp $

\documentclass{beamer}
%\documentclass[handout]{beamer}
%\usepackage{pgfpages}
%\pgfpagesuselayout{2 on 1}[a4paper,border shrink=5mm]

% This file is a solution template for:

% - Talk at a conference/colloquium.
% - Talk length is about 20min.
% - Style is ornate.

% Copyright 2004 by Till Tantau <tantau@users.sourceforge.net>.
%
% In principle, this file can be redistributed and/or modified under
% the terms of the GNU Public License, version 2.
%
% However, this file is supposed to be a template to be modified
% for your own needs. For this reason, if you use this file as a
% template and not specifically distribute it as part of a another
% package/program, I grant the extra permission to freely copy and
% modify this file as you see fit and even to delete this copyright
% notice.

\mode<presentation>
{
%  \usetheme{Warsaw}
%  \usetheme{Boadilla}
%  \usetheme{Goettingen}
%  \usetheme{Hannover}
%  \usetheme{Madrid}
%  \usetheme{Marburg}
%  \usetheme{Montpellier}
%  \usetheme{Pittsburgh}
  \usetheme{Hawke}
  % or ...

  \setbeamercovered{transparent}
  % or whatever (possibly just delete it)
}

\usepackage[english]{babel}
% or whatever

\usepackage[latin1]{inputenc}
% or whatever

\usepackage{times}
\usepackage[T1]{fontenc}
% Or whatever. Note that the encoding and the font should match. If T1
% does not look nice, try deleting the line with the fontenc.

\usepackage{multimedia}

%%%%%%
% My Commands
%%%%%%

\newcommand{\ml}{{\sc matlab}}
\newcommand{\bb}{{\boldsymbol{b}}}
\newcommand{\bx}{{\boldsymbol{x}}}
\newcommand{\by}{{\boldsymbol{y}}}
\newcommand{\bfm}[1]{{\boldsymbol{#1}}}

%%%%

\title[Lecture 14] % (optional, use only with long paper titles)
{Lecture 14 - Adaptive and Gaussian Quadrature}

% \subtitle
% {Include Only If Paper Has a Subtitle}

\author[I. Hawke] % (optional, use only with lots of authors)
{I.~Hawke}
% - Give the names in the same order as the appear in the paper.
% - Use the \inst{?} command only if the authors have different
%   affiliation.

\institute[University of Southampton] % (optional, but mostly needed)
{
%  \inst{1}%
  School of Mathematics, \\
  University of Southampton, UK
}
% - Use the \inst command only if there are several affiliations.
% - Keep it simple, no one is interested in your street address.

\date[Semester 1] % (optional, should be abbreviation of conference name)
{MATH3018/6141, Semester 1}
% - Either use conference name or its abbreviation.
% - Not really informative to the audience, more for people (including
%   yourself) who are reading the slides online

\subject{Numerical methods}
% This is only inserted into the PDF information catalog. Can be left
% out.

% If you have a file called "university-logo-filename.xxx", where xxx
% is a graphic format that can be processed by latex or pdflatex,
% resp., then you can add a logo as follows:

\pgfdeclareimage[height=0.5cm]{university-logo}{mathematics_7469}
\logo{\pgfuseimage{university-logo}}

% Delete this, if you do not want the table of contents to pop up at
% the beginning of each subsection:
%  \AtBeginSubsection[]
%  {
%    \begin{frame}<beamer>
%      \frametitle{Outline}
%      \tableofcontents[currentsection,currentsubsection]
%    \end{frame}
%  }
\AtBeginSection[]
{
  \begin{frame}<beamer>
    \frametitle{Outline}
    \tableofcontents[currentsection]
  \end{frame}
}

% If you wish to uncover everything in a step-wise fashion, uncomment
% the following command:

%\beamerdefaultoverlayspecification{<+->}

\begin{document}

\begin{frame}
  \titlepage
\end{frame}

\section{Adaptive Quadrature}

\subsection{Adaptive Quadrature}

\begin{frame}
  \frametitle{Minimizing cost}

  The aim of numerical quadrature is to compute
  \begin{equation*}
    \int_a^b f(x) \, \text{d}x
  \end{equation*}
  where $f$ is a real function of a single variable $x$.

  \vspace{1ex}

  Looked at polynomial quadrature with equally spaced nodes. Can be
  inefficient and inaccurate.  \pause

  \vspace{1ex}

  Efficiency requires minimizing the number of function evaluations,
  which means minimizing the number of nodes.  Either
  \begin{itemize}
  \item Use standard quadrature methods, placing nodes depending on
    the local behaviour of the function $f$, or \pause
  \item Use methods that give the best accuracy for a ``generic''
    function $f$.
  \end{itemize}

\end{frame}

\begin{frame}
  \frametitle{Adaptive Quadrature}

  The aim of adaptive quadrature is to compute the integral
  \begin{equation*}
    \int_a^b f(x) \, \text{d}x
  \end{equation*}
  for arbitrary $f$ to a given tolerance $\epsilon$.

  \vspace{1ex}

  Error depends on derivatives of $f$ and the spacing $h$. Most
  efficient to put more nodes (decrease $h$) where the function varies
  rapidly. \pause

  \vspace{1ex}

  Adaptive quadrature therefore
  \begin{enumerate}
  \item Computes an approximation with few subintervals;
  \item Estimates the error on each subinterval;
  \item Subdivides subintervals where error is ``too large''.
  \end{enumerate}

\end{frame}

\begin{frame}
  \frametitle{Example subdivision}

  \begin{center}
    \includegraphics<1|handout:0>[height=0.7\textheight]{figures/Adaptive0}
    \includegraphics<2|handout:0>[height=0.7\textheight]{figures/Adaptive1}
    \includegraphics<3>[height=0.7\textheight]{figures/Adaptive2}
  \end{center}
  An example of the subintervals created by adaptive quadrature.

\end{frame}

\begin{frame}
  \frametitle{Estimating the error}

  Recursively dividing an interval into two equal subintervals is
  straightforward. But how do we choose which intervals need
  subdividing? For this we need some error measure. \pause

  \vspace{1ex}

  Use Richardson extrapolation. For any (sub)interval:
  \begin{enumerate}
  \item Compute $I_{2h}$; \pause
  \item Compute $I_h$; \pause
  \item If $I \simeq I_h + C h^s$ then the \emph{computable estimate of
      the error} is
    \begin{equation*}
      E_{2 h} \equiv \frac{|I_{2h} - I_h|}{2^s - 1}.
    \end{equation*}
  \end{enumerate}

\end{frame}

\begin{frame}
  \frametitle{Total vs.\ local error}

  Computable error is the \emph{local} error for one subinterval.
  Aim: To make the \emph{total} error over the \emph{whole} interval
  less than $\epsilon$. \pause

  \vspace{1ex}

  Total error bounded by the sum of the errors in each subinterval,
  \begin{equation*}
    |{\cal E}| \leq \sum_{\text{intervals } j} |{\cal E}_j|.
  \end{equation*} \pause

  Expect ${\cal E}_j$ to be proportional to the width of the
  subinterval,
  \begin{equation*}
     |{\cal E}_j| \leq D (x_j - x_{j-1}).
  \end{equation*} \pause

  If we bound the error on each subinterval by
  \begin{equation*}
     |{\cal E}_j| \leq \frac{\epsilon}{b - a} (x_j - x_{j-1})
  \end{equation*}
  then the resulting total error ${\cal E}$ will be less than our
  required tolerance $\epsilon$.

\end{frame}

\begin{frame}
  \frametitle{Example 1}

  \begin{center}
    \includegraphics<1|handout:0>[height=0.7\textheight]{figures/AdaptiveQuad1}
    \includegraphics<2|handout:0>[height=0.7\textheight]{figures/AdaptiveQuad2}
    \includegraphics<3>[height=0.7\textheight]{figures/AdaptiveQuad3}
  \end{center}
  The nodes for adaptive quadrature of a Heaviside function - the
  accuracy is $\sim 10^{-9}$ with $\sim 50$ nodes.

\end{frame}

\begin{frame}
  \frametitle{Example 2}

  \begin{center}
    \includegraphics<1|handout:0>[height=0.7\textheight]{figures/AdaptiveQuadSin1}
    \includegraphics<2>[height=0.7\textheight]{figures/AdaptiveQuadSin2}
    \includegraphics<3|handout:0>[height=0.7\textheight]{figures/AdaptiveQuadSin3}
  \end{center}
  The nodes for adaptive quadrature of a more complex function.

\end{frame}

\section{Gaussian Quadrature}

\subsection{Gaussian Quadrature}

\begin{frame}
  \frametitle{Gaussian Quadrature}

  Algorithms used so far use nodes $\{x_j\}$ to find a formula of type
  \begin{equation*}
    \int_a^b f(x) \, \text{d}x \simeq \sum_{j=1}^n w_j f_j.
  \end{equation*}
  \begin{overlayarea}{\textwidth}{0.5\textheight}
    \only<1-|handout:1->
    {
      Location of the nodes fixed in advance. Choice of
      polynomial then fixes \emph{weights} $w_j$.
    }
    \only<2-|handout:2->
    {
      In Gaussian Quadrature instead both weights \emph{and} nodes are
      to be determined.  Use this freedom to evaluate integral ``as
      accurately as possible''.
      \vspace{1ex}

    }
    \only<3|handout:2>
    {
      We have $2n$ degrees of freedom in the above formula, so
      can perfectly represent a polynomial of order
      $2n-1$. So  fix nodes and weights by ensuring
      \begin{equation*}
        \int_a^b x^s \, \text{d}x = \sum_{j=1}^n w_j x_j^s \quad
        \forall s = 0, 1, \dots, 2 n - 1.
      \end{equation*}
    }
    \only<4|handout:3>
    {
      If $f$ behaves in known ways (i.e., decays exponentially), can
      improve accuracy by introducing a \emph{weight function} $W(x)$:
      \begin{equation*}
        \int_a^b W(x) x^s \, \text{d}x = \sum_{j=1}^n w_j x_j^s
        \quad \forall s = 0, 1, \dots, 2 n - 1.
      \end{equation*}
    }
  \end{overlayarea}

\end{frame}

\begin{frame}
  \frametitle{Example}

  Gauss-Legendre quadrature on $[-1,1]$, $n=2$ nodes:
  \begin{equation*}
    \int_{-1}^1 x^s \, \text{d}x = \sum_{j=1}^n w_j x_j^s \quad
     s = 0, 1, 2, 3.
  \end{equation*} \pause

  This gives the four equations
  \begin{align*}
  &&s & = 0 : & w_1 + w_2 & = 2 &&\\
  &&s & = 1 : & w_1 x_1 + w_2 x_2 & = 0 &&\\
  &&s & = 2 : & w_1 x_1^2 + w_2 x_2^2 & = 2 / 3 &&\\
  &&s & = 3 : & w_1 x_1^3 + w_2 x_2^3 & = 0.&&
  \end{align*}
  The solution is $w_1 = 1 = w_2$, $x_1 = - 1 / \sqrt{3} = -
  x_2$. \pause

  \vspace{1ex}

  Therefore the Gauss-Legendre formula is
  \begin{equation*}
    \int_{-1}^{1} f(x) \, \text{d}x \simeq f(-1 / \sqrt{3}) +  f(1 /
    \sqrt{3}).
  \end{equation*}

\end{frame}

\begin{frame}
  \frametitle{Example: 2}

  If we allow ourself $n=3$ nodes we have to solve
  \begin{equation*}
    \int_{-1}^1 x^s \, \text{d}x = \sum_{j=1}^n w_j x_j^s \quad
     s = 0, 1, 2, 3, 4, 5.
  \end{equation*} \pause

  This has solution for the weights $w_1 = \tfrac{5}{9} = w_3$, $w_2 =
  \tfrac{8}{9}$ when evaluated at the nodes $x_1 = -
  \sqrt{\tfrac{3}{5}} = - x_3$, $x_2 = 0$. \pause

  \vspace{1ex}

  Therefore the Gauss-Legendre formula is
  \begin{equation*}
    \int_{-1}^{1} f(x) \, \text{d}x \simeq \frac{1}{9} \left( 5
      f(-\sqrt{\tfrac{3}{5}}) + 8 f(0) + 5 f(\sqrt{\tfrac{3}{5}}) \right).
  \end{equation*} \pause

  \vspace{1ex}

  In general, nodes are given by zeros of orthogonal polynomials (type
  depends on weight function).

\end{frame}

\begin{frame}
  \frametitle{Example: 2}

  If we apply the Gauss-Legendre formula with 2 nodes to
  \begin{equation*}
    \int_{-1}^{1} \cos^2 \left(\tfrac{\pi x}{2}\right) \, \text{d}x
  \end{equation*}
  then we find
  \begin{equation*}
    \int_{-1}^{1} \cos^2 \left(\tfrac{\pi x}{2}\right) \, \text{d}x
    \simeq 0.75938.
  \end{equation*} \pause

  The Gauss-Legendre formula with 3 nodes gives
  \begin{equation*}
    \int_{-1}^{1} \cos^2 \left(\tfrac{\pi x}{2}\right) \, \text{d}x
    \simeq 1.133565.
  \end{equation*} \pause

  Compare with the trapezoidal rule ($0$!) and Simpson's rule ($4/3$)
  -- the exact answer is $1$ and only two (three) function evaluations
  are needed.

\end{frame}

\begin{frame}
  \frametitle{Example: 3}

  \begin{overlayarea}{\textwidth}{0.6\textheight}
    \only<1|handout:1>
    {
      \begin{center}
        \includegraphics[height=0.6\textheight]{figures/Gauss1}
      \end{center}
    }
    \only<2|handout:2>
    {
      \begin{center}
        \includegraphics[height=0.6\textheight]{figures/Gauss2}
      \end{center}
    }
  \end{overlayarea}
  Simple example: very few nodes sufficient for high accuracy. \pause
  More complex example: more nodes required, but Gauss quadrature
  still more accurate than standard Newton-Cotes methods.
\end{frame}

\section{Summary}

\subsection{Summary}

\begin{frame}
  \frametitle{Summary}

  \begin{itemize}
  \item If the function is expensive to compute then to improve speed
    we need to minimize the number of nodes used.
  \item Adaptive quadrature uses standard quadrature methods
    (Simpson's rule, trapezoidal rule) by splitting the interval into
    subintervals depending on the function itself.
  \item Adaptive quadrature allows you to bound the error
    (approximately).
  \item Gaussian quadrature gives the ``best'' result for a generic
    function with a given number of nodes.
  \item The location of the nodes is computed, not given.
  \end{itemize}

\end{frame}

\end{document}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End:


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%                                                                 %
%  GEANT manual in LaTeX form                                     %
%                                                                 %
%  Michel Goossens (for translation into LaTeX)                   %
%  Version 1.00                                                   %
%  Last Mod. Jan 24 1991  1300   MG + IB                          %
%                                                                 %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Origin{M.S. Dixit}
\Revision{G.Lynch}
\Documentation{F.Carminati}
\Submitted {17.06.85}  \Revised {16.12.93}
\Version{Geant 3.16}\Routid{PHYS325}
\Makehead {Moli\`ere  scattering}
\section{Subroutines}
\subsection{Initialisation}

\Shubr{GMULOF}{}
\Rind{GMULOF} calculates the table of maximum
steps corresponding to the
energies in the array {\tt ELOW} for electrons and muons.
It is called during initialisation by \Rind{GPHYSI}.
For hadrons and ions the computation is performed during
tracking in the routines \Rind{GTHADR} and \Rind{GTHION}.

\Shubr{GMOLI}{(AC,ZC,WMAT,NLM,DENS,OMC*,CHC*)}
\begin{DLtt}{MMMMMMMM}
\item[AC] {\tt REAL} array of dimenstion {\tt NLM} containing
the mass numbers;
\item[ZC] {\tt REAL} array  of dimenstion {\tt NLM}
containing the atomic numbers;
\item[WMAT] {\tt REAL} array containing the proportions by
weight of the component of a mixture/compound. {\tt 1} in case
of a single component;
\item[NLM] {\tt INTEGER} number of the elements in the mixture/compound;
\item[DENS] {\tt REAL} density in g cm$^{-3}$;
\item[OMC] {\tt REAL} $b_{c}$ constant of Moli\`{e}re theory;
\item[CHC] {\tt REAL} $\chi_{cc}$ constant of Moli\`{e}re theory.
\end{DLtt}

\Rind{GMOLI} calculates the two material-dependent constants
({\tt OMC} and {\tt CHC}). These are
needed during initialisation by \Rind{GMULOF} and during tracking
to sample the scattering angle, as explained below.
It is called at initialisation time by the routine
\Rind{GPROBI} which initialises some material constants and computes
the probabilities for various processes.

\subsection{Sampling of the multiple scattering angle}

\Shubr{GMULTS}{}
\Rind{GMULTS} is the steering routine for the sampling
of the multiple scattering angle when the variable {\tt IMULTS}
in common \FCind{/GCPHYS/} is {\tt 1} or {\tt 2}. This variable
can be controlled by the {\tt MULT} data record.
The routine decides if the material-dependent constant {\tt OMC}
computed by \Rind{GMOLI} during initialisation has to
be corrected due to its dependence on $\beta$ and $Z_{inc}$, and
selects Moli\`ere theory (\Rind{GMOLIE})
or single Coulomb scattering (\Rind{GMCOUL}).
It updates the direction of the particle in the array
{\tt VECT} (COMMON \FCind{GCTRAK}).
It is called by the tracking routines \Rind{GTELEC}, \Rind{GTHADR},
\Rind{GTHION} and \Rind{GTMUON}.

\Shubr{GMOLIO}{(AC,ZC,WMAT,NLM,DENS,BETA2,CHARG2,OMC*)}
\begin{DLtt}{MMMMMMMM}
\item[BETA2] {\tt REAL} $\beta^2$ of the particle;
\item[CHARG2] {\tt REAL} charge squared of the particle in electron
charge unit ($Z_{inc}^2$ in the following).
\end{DLtt}

All the other arguments are routine \Rind{GMOLI}.
\Rind{GMOLIO} re-evaluates the material-dependent constant {\tt OMC}.
It is called by \Rind{GMULTS}, if needed.

\Shubr{GMOLIE}{(OMEGA,BETA2,DIN*)}
\begin{DLtt}{MMMMMMMM}
\item[OMEGA] {\tt REAL} $\Omega_0$ of the Moli\`{e}re theory;
\item[BETA2] {\tt REAL} $\beta^2$ of the particle;
\item[DIN]   {\tt REAL} array of dimension 3 containing the
new direction cosines with respect to the original direction of the
particle.
\end{DLtt}
\Rind{GMOLIE} samples the multiple scattering angle according to
Moli\`{e}re theory, as explained in the next section.
It is called by the multiple scattering steering routine \Rind{GMULTS},

\Shubr{GMOL4}{(Y*,X,VAL,ARG,EPS,IER*)}
\begin{DLtt}{MMMMMMMM}
\item[Y] {\tt REAL} value of the interpolated function;
\item[X] {\tt REAL} value at which the function should be interpolated;
\item[VAL] {\tt REAL} array of four values of the function to
interpolate;
\item[ARG] {\tt REAL} array of four values of the argument of the
function to which the values contained in {\tt VAL} correspond;
\item[EPS]  {\tt REAL} required precision for the interpolation;
\item[IER] {\tt INTEGER} error flag;
\end{DLtt}
\Rind{GMOL4} inverts the integral of Moli\`ere distribution
function using four-points continued-fraction interpolation.
It is called by \Rind{GMOLIE}.

\section{Method}
\subsection{Introduction}
When traversing ordinary matter, elementary particles undergo deflection
from their originary trajectory due to the interaction with the atoms.
This effect is rather large for charged particles, which are deflected
by the electric field of nuclei and electrons via a large number of
small elastic {\it collisions}. To simulate precisely the transport of
particles in matter, it is important to provide a precise description
of this effect. Considerable effort has gone over the years into
the formulation of a theory of Coulomb multiple scattering. In
{\tt GEANT} we follow the theory originally formulated by Moli\`{e}re
\cite{bib-MOLI} \cite{bib-MOL1} and then elaborated by Bethe
\cite{bib-BET1}. A review of this theory can be found in \cite{bib-SCOT}.

It should be noted that this is not the only theory which describes
multiple Coulomb scattering, and a thorough analysis of the problem has
been performed by Goudsmit and Saunderson \cite{bib-GOUD} \cite{bib-GOU1}.

A complete account of the Moli\`{e}re theory is outside the scope of this
manual. For more information the interested reader could refer to the
works cited above and to the EGS4 manual
\cite{bib-EGS4}, from which the notation used in the present chapter
has been derived.

The principal limitations of Moli\`{e}re theory are the following:
\begin{enumerate}
\item the angular deflection is {\it small}. Effectively this
constraint provides the upper limit for the step-size;
\item the theory is a {\it multiple scattering} theory; that is, many
atomic collisions participate in causing the incident particle to be
deflected. In some implementations, this constraints provides a lower
limit on the step size. In {\tt GEANT}, when the number of scatters is
below the limit of applicability of Moli\`{e}re single Coulomb scatterings
are simulated, so that there is no effective minimum step size below which
multiple scattering is {\it switched off};
\item the theory applies only in semi-infinite homogeneous media. This
constraint calls for a very careful handling of particles near to a
medium's boundary;
\item there is no energy loss built into the theory. This again introduces
the necessity to take small steps;
\end{enumerate}

In Moli\`{e}re theory corrected for finite angle
scattering ($\sin\theta \neq \theta$) as described by
Bethe~\latexhtml{\cite{bib-BET1,bib-SCOT}}%
                {\cite{bib-BET1} \cite{bib-SCOT}},
the angular distribution is given by:
\begin{eqnarray}
      f(\theta) \: \theta d\theta & = & \sqrt{\frac{\sin\theta}{\theta}} \:
f_r(\eta) \: \eta d\eta,
\end{eqnarray}

where for $f_r(\eta)$ we use the first three terms of Bethe's expansion:
\begin{eqnarray}
\label{eq:phys325-1}
f_r(\eta) & = & f_{r}^{(0)}(\eta)+
f_{r}^{(1)}(\eta)B^{-1}+f_{r}^{(2)}(\eta)B^{-2} \\
\eta & = & \frac{\theta}{\chi_c \sqrt{ B }} \nonumber
\end{eqnarray}
$\eta$ is called the {\it reduced angle}.

\subsection{Calculation of the reduced angle}
In formula (\ref{eq:phys325-1}) above $\chi_c$ the {\it critical
scattering angle} defined as:
\begin{equation}
\chi^2_c = \frac{4 \pi e^4  Z_s \rho N_{Av} Z_{inc}^2 t}{W c^2 p^2\beta^2}
= \chi^2_{cc} Z_{inc}^2 \frac{t}{E^2\beta^4}
\end{equation}
with
\[
\begin{array}{ll}
Z_{inc} & \mbox{incident particle charge}               \\
Z_{s}   & \mbox{defined below}                           \\
N_{Av}  & \mbox{Avogadro's number}                      \\
\rho  & \mbox{density}                                  \\
W     & \sum_{i=1}^{N} n_i A_i \mbox{\hspace{0.5cm}molecular weight}  \\
E     & \mbox{particle energy in $MeV$}                    \\
r_0   & e^2/(m c^2) \mbox{\hspace{0.5cm} classical electron radius} \\
t     & \mbox{total path length in the scatterer (and not its thickness).}
\end{array}
\]

The variable $\chi_c$ is factorised into two parts.
The first part, $\chi_{cc}$, is a constant of the medium for
a given incident particle.
The second part depends on the incident particle energy and on the
path length in the medium.

\begin{equation}
\chi^2_{cc} = \frac{4 \pi e^{4} Z_s \rho N_{Av}}{W} = \frac{4 \pi c^4 r_0^2
m_e^2 Z_s \rho N_{Av}}{W} \approx (0.39612 \: 10^{-3})^2
Z_s \frac{\rho}{W}
\left [ \: GeV^2 \: cm^{-1} \right ]
\end{equation}

The variable {\tt CHC} or {\tt CHCMOL} stored in {\tt Q(JPROB+25)} corresponds
to $\chi_{cc}$.
$B$ is defined by the equation
\begin{equation}
\label{eq:phys325-2}
B-\ln B =\ln\Omega_0,
\end{equation}

where $\Omega_0$ can be interpreted as the number of collisions in the step:

\begin{equation}
\Omega_0 =\frac{ \chi^2 _c}{e^{2\gamma-1}\chi_\alpha^2}
= b_c Z_{inc}^2\frac{t}{\beta^2}
\end{equation}

Here $\gamma$ is Euler's constant ($\gamma =0.57721\dots$)
and $\chi_\alpha$ the atomic electron
screening angle. For a single element this is given by:

\[
\chi_\alpha^2 = \left ( \frac{\lambda_0}{2 \pi r_{TF}} \right ) ^2
\left [ 1.13 + 3.76 \left ( \frac{\alpha Z Z_{inc}}{\beta} \right ) ^2 \right ]
\]

where

\begin{eqnarray*}
\frac{\lambda_0}{2 \pi} & = & \hbar/p
\mbox{\hspace{0.5cm}Compton wavelength of the electron} \\
r_{TF} & = & 0.885 a_0 Z^{-\frac{1}{3}} \mbox{\hspace{0.5cm}Thomas-Fermi
radius of the atom} \\
a_0 & = & \frac{\hbar^2}{m e^2} \mbox{\hspace{0.5cm}Bohr's radius} \\
\alpha & = & 1/137.035 \dots \mbox{\hspace{0.5cm}the fine-structure constant}
\end{eqnarray*}

so that we have

\[
\chi_\alpha^2 = \frac{m^2 e^4 1.13}{p^2 \hbar^2 (0.885)^2} Z^{\frac{2}{3}}
\left [ 1 + 3.34 \left ( \frac{\alpha Z Z_{inc}}{\beta} \right ) ^2 \right ].
\]

For a compound/mixture the following rule holds:

\begin{eqnarray*}
\ln(\chi_\alpha^2) & = & \frac{\displaystyle \sum_{i=1}^N n_i Z_i (Z_i+1) \ln
( \chi_{\alpha i}^2)}{\displaystyle \sum_{i=1}^N n_i Z_i (Z_i+1)} =
\frac{\displaystyle \sum_{i=1}^N \frac{p_i}{A_i} Z_i (Z_i+1) \ln
( \chi_{\alpha i}^2) }{\displaystyle \sum_{i=1}^N \frac{p_i}{A_i} Z_i (Z_i+1)}\\
& = & \frac{\displaystyle \sum_{i=1}^N \frac{p_i}{A_i} Z_i (Z_i+1) \ln \left \{
\frac{m^2 e^4 1.13}{p^2 \hbar^2 (0.885)^2} Z_i^{\frac{2}{3}}
\left [ 1 + 3.34 \left ( \frac{\alpha Z_i Z_{inc}}{\beta} \right ) ^2 \right ]
\right \} }{\displaystyle \sum_{i=1}^N \frac{p_i}{A_i} Z_i (Z_i+1)} \\
& = & \ln \left ( \frac{m^2 e^4 1.13}{p^2 \hbar^2 (0.885)^2} \right ) -
\frac{\displaystyle \sum_{i=1}^N \frac{p_i}{A_i} Z_i (Z_i+1) \ln
Z_i^{-\frac{2}{3}}}{\displaystyle \sum_{i=1}^N \frac{p_i}{A_i} Z_i (Z_i+1)} \\
 & & + \frac{\displaystyle \sum_{i=1}^N \frac{p_i}{A_i} Z_i (Z_i+1)
\left [ 1 + 3.34 \left ( \frac{\alpha Z_i Z_{inc}}{\beta} \right ) ^2 \right ]}
{\displaystyle \sum_{i=1}^N \frac{p_i}{A_i} Z_i (Z_i+1)} \\
\end{eqnarray*}

To understand the transformations in the above formulae, the following
should be noted. Let $n_i$ be the number of atoms
of type $i$ in a compound/mixture, that is the number of moles
of element $i$ in a mole of material, and $p_i$ the proportion by weight
of that element. We have the following relation:

\[
p_i = \frac{n_i A_i}{\sum_{j=1}^{N} n_j A_j} \mbox{\hspace{2cm} that
is\hspace{2cm}} n_i = \left ( \sum_{j=1}^{N} n_j A_j \right )
\frac{p_i}{A_i} = W  \frac{p_i}{A_i}
\]

so that we can simplify expressions in the following way:

\[
\frac{\sum_{i=1}^{N} n_i \dots}{W} = \sum_{i=1}^{N} (p_i/A_i) \dots
\]

If now we set

\[
\begin{array}{L@{}c@{}L@{}c@{}L@{}c@{}L}
Z_s & = & \sum^{N}_{i = 1} n_i Z_i(Z_i+1)  & =  &
W \sum^{N}_{i = 1} \frac{p_i}{A_i} Z_i(Z_i+1) & = & W Z_s'  \\
Z_E & = & \sum^{N}_{i = 1} n_i Z_i(Z_i + 1 )\ln Z_i^{-2/3}  & = &
W\sum \frac{p_i}{A_i} Z_i(Z_i + 1 )\ln Z_i^{-2/3} & = & W Z_E' \\
Z_x & = & \sum^{N}_{i = 1} n_i Z_i(Z_i + 1 ) \ln \left [ 1+3.34
\left( \frac{\alpha Z_i Z_{inc}}{\beta}\right)^2 \right ] \\
 & = & W \sum^{N}_{i = 1} \frac{p_i}{A_i} Z_i(Z_i + 1 ) \ln \left [ 1+3.34
\left( \frac{\alpha Z_i Z_{inc}}{\beta}\right)^2 \right ] & &  & = & W Z_x'
\end{array}
\]

we can write:

\begin{eqnarray*}
\ln(\chi_{\alpha}^2) & = &
\ln \left ( \frac{m^2 e^4 1.13}{p^2 \hbar^2 (0.885)^2} \right )
+ \frac{Z_x-Z_E}{Z_s} \\
\chi_{\alpha}^2 & = & \frac{m^2 e^4 1.13}{p^2 \hbar^2 (0.885)^2}
e^{(Z_x  - Z_E )/Z_s} = \frac{m^2 e^4 1.13}{p^2 \hbar^2 (0.885)^2}
e^{(Z_x'  - Z_E' )/Z_s'}
\end{eqnarray*}

and finally:

\begin{eqnarray*}
\Omega_0 & = & \frac{ \chi^2 _c}{e^{2\gamma-1}\chi_\alpha^2} \\
 & = & \frac{1}{1.167} \:
\frac{4 \pi e^4  Z_s \rho N_{Av} Z_{inc}^2 t}{W c^2 p^2\beta^2} \:
\frac{p^2 \hbar^2 (0.885)^2}{1.13 m^2 e^4}e^{-(Z_x'  - Z_E' )/Z_s'} \\
 & = & b_c Z_{inc}^2\frac{t}{\beta^2} \\
b_c & = & \frac{4 \pi N_{Av} \hbar^2}{m^2 c^2}
\frac{(0.885)^2}{1.13 \times
1.167} \rho Z_s' e^{(Z_E'  - Z_x' )/Z_s'} \approx 6702.33 \rho Z_s'
e^{(Z_E'  - Z_x' )/Z_s'}
\end{eqnarray*}

The quantity $b_c$ is calculated during initialisation by
the routine \Rind{GMOLI} setting $\beta=1$ and $Z_{inc}=1$.
This variable is called
{\tt OMC} or {\tt OMCMOL} and it is stored in {\tt Q(JPROB+21)}.
It has, via the atomic electron screening angle $\chi_{\alpha}^2$,
a small dependence
on $\beta$  and $Z_{inc}$ in the term $Z_x$. The quantity $b_c \rho Z_s'$
is re-evaluated during tracking by the routine \Rind{GMOLIO}, called by
\Rind{GMULTS}, only when necessary.

The dependence of $Z_x$ on $\beta$ and $Z_{inc}$ is via a term of the form:
$1+3.34\left(\alpha Z Z_{inc}\beta^{-1} \right )^2$, so it is not
necessary to recalculate it as long as
$1 + 3.34\left(\alpha Z Z_{inc}\beta^{-1} \right )^2 \approx
1 + 3.34\left(\alpha Z \right )^2$ that is:

\[
\begin{array}{RcLcLcL}
3.34 (Z \alpha)^2  \left [ \left( \frac{Z_{inc}}{\beta} \right )^2
-1 \right] & \ll & 1 & \Rightarrow &
Z^2 \frac{Z_{inc}^2 - \beta^2}{\beta^2} & \ll & 3.34 \alpha^{-2}
\approx 5500 \\ [.2cm]
 Z^2 ( Z_{inc}^2 - \beta^2 ) & \leq & 50 \beta^2
\end{array}
\]

For mixtures/compounds this condition should be checked for every component
and this would be unacceptable from the point of view of computer time. So
we simply make the condition more restrictive multiplying the first term
by the number of elements in the mixture/compound.

\subsection{Sampling of the distribution function}

The component distribution functions (\ref{eq:phys325-1}) are given by

\[
\begin{array}{lcccr}
f_{r}^{(0)}(\eta)   & = & 2e^{-\eta^2}  & = &  2D_0(1,1,-\eta^2)         \\
f_{r}^{(1)}(\eta)   &   &               & = &  2D_1(2,1,-\eta^2)         \\
f_{r}^{(2)}(\eta)   &   &               & = &   D_2(3,1,-\eta^2),
\end{array}
\]

where $ D_n(a,b,z) = d^n[\Gamma(a)M(a,b,z)]/d a^n $, with
$M(a,b,z)$ the Kummers hypergeometric function.

Integrals of the functions $f_{r}^{(1)}$ and $f_{r}^{(2)}$ needed for the
Monte Carlo can be written down directly in terms of the $D$ functions
\begin{eqnarray*}
\int_{\eta}^{\infty}\eta f_{r}^{(1)} \: \eta  d\eta & = &
     D_1 ( 2,1, - \eta^2)-D_1( 1,1, - \eta^2 ) - D_0 ( 1,1,-\eta^2)\\
\int_{\eta}^{\infty}\eta f_{r}^{(2)} \: \eta d\eta    & = &
\frac{1}{2} D_2 (3,1,-\eta^2) - D_2 (2,1,-\eta^2) - D_1(2,1,-\eta^2)
\end{eqnarray*}

We note that the first term is just the Gaussian part and it dominates
for large $B$, that is, for large number of scatters, as it has to
be expected. Recalling the definition of $\eta$ in (\ref{eq:phys325-1})
we can say that the half-width of the Gaussian part of the Moli\`{e}re
distribution is $\sigma^{2} = \chi_{c}^2 B/2$.
Routine \Rind{GMOLIE} performs the sampling of the Moli\`{e}re
distribution via the following steps:

\begin{enumerate}
\item the value of $B$ is calculated recursively. If we set $f(B) =
B$ with $f(B) = \ln \Omega_0  + \ln B$ the relation used is
$B_n = B_{n-1} + \Delta B = B_{n-1} + (f(B_{n-1}) - B_{n-1})/
(1-1/B_{n-1})$. Convergence is assumed when $|\Delta B| \leq 10^{-4}$.

\item a random number $r_1$ is sampled and
a table of four values $F_{r}(\eta_i)$ is built with
$i = j-2,j-1,j,j+1$ and $F_{r}(\eta_j) \geq r_1$. $F$ is the distribution
function derived from (\ref{eq:phys325-1}):
\begin{eqnarray*}
F_{r}(\eta) & = & \int_{0}^{\eta}{f_{r}(t) \: dt}
= \int_{0}^{\eta}{f_{r}^{(0)}(t)+f_{r}^{(1)}(t)B^{-1}+f_{r}^{(2)}(t)B^{-2} \: dt} \\
& = & \int_{0}^{\eta}{f_{r}^{(0)}(t) \: dt} +
B^{-1} \int_{0}^{\eta}{f_{r}^{(1)}(t) \: dt} +
B^{-2} \int_{0}^{\eta}{f_{r}^{(2)}(t) \: dt} \\
& = & F_{r}^{(0)}(\eta) +
B^{-1} F_{r}^{(1)}(\eta) + B^{-2} F_{r}^{(2)}(\eta)
\end{eqnarray*}

40 values of the functions $F_{r}^{(n)}$ are tabulated.

\item using a four-points continued-fraction interpolation method
(\Rind{GMOL4}) a table of $\eta_i^2$ is interpolated using
the values of $r_1$ and $F$ tabulated in the previous step.
This is similar
to the inversion of the distribution function, but instead of
obtaining directly the random variable, we interpolate a table
of its squares.
This provides a better fit to the inverse function;

\item the value of $\theta = \chi_{c} \sqrt{B \eta^2}$ is calculated;

\item a random number $r_2$ is extracted, and the rejection function
$g(\theta) = \sqrt{\sin\theta/\theta}$ is computed. If $r_2 > g(\theta)$
resampling begins from step 2, otherwise the value is accepted.
\end{enumerate}

The Moli\`ere distribution gives the space scattering angle. A similar
expression may be written for the lateral displacement of the
scattered particles. However, the problem of joint angle lateral displacement
in the Moli\`ere approximation has not been solved, and,  for small
step size, lateral displacement is of second order and may be neglected.
This introduces a problem when large step sizes are taken. The step
size can be artificially limited via the use of the variable {\tt
STEMAX}, which is an argument to the routine \Rind{GSTMED}. For more
information on this point the user is invited to consult chapter
{\tt [CONS200]}.

\section{Path and step length}
\subsection{Restrictions on the step length}
Restrictions on the length of the step arise from:
\begin{enumerate}
\item the number of scatters
$\Omega_0 \geq 20 $ to stay within the multiple scattering regime. When
$\Omega_0 < 20$, an appropriate number of {\it single} scatterings is
used. See routine \Rind{GMCOUL} {\tt [PHYS328]}.
\item
$\chi_c^2B \leq 1 $ i.e. the width of the Gaussian part of the distribution
should be less than one radian. This
condition induces a maximum step length for the multiple scattering called
$t_{Bethe}$. In order to find this value we write the limiting condition
as $\chi_c^2(t_{Bethe})B(t_{Bethe}) = 1 $, that is
$B(t_{Bethe}) = 1/\chi_c^2(t_{Bethe})$. Now using equation
(\ref{eq:phys325-2}) we take the exponential of both members
$B(t_{Bethe}) \Omega_0(t_{Bethe}) = \exp(1/\chi_c^2(t_{Bethe}))$.
Dividing the two last equalities we obtain:
\begin{eqnarray*}
\Omega_0(t_{Bethe}) = \frac{b_c Z_{inc}^2}{\beta^2} t_{Bethe} & = &
\chi_c^2 (t_{Bethe})\exp(1/\chi_c^2(t_{Bethe})) \\
&  = & \frac{\chi_{cc}^2 Z_{inc}^2 t_{Bethe}
\exp{[(E^2\beta^4)/(\chi_{cc}^2 Z_{inc}^2 t_{Bethe})]}}
{E^2\beta^4} \\
\exp{[(E^2\beta^4)/(\chi_{cc}^2 Z_{inc}^2 t_{Bethe})]} & = &
\frac{b_c E^2 \beta^2}{\chi_{cc}^2} \\
t_{Bethe} & = & \frac{E^2 \beta^4}{\chi_{cc}^2 \: Z_{inc}^2 \: \ln
\left [b_c E^2 \beta^2/\chi_{cc}^2 \right]}
\end{eqnarray*}

For electrons and muons this constraint on the step-length is tabulated
at initialisation time in the routine \Rind{GMULOF} {\tt [PHYS201]}.
For hadrons this formula can be approximated as:
\[
t_{Gauss}
\approx \left(\frac{1}{14.1 \: 10^{-3}} \: \frac{E^2 \beta}{Z_{inc}}
\right )^2 X_0,
\]
where $E$ is in GeV and $X_0$ is the radiation length in centimeters and
the formula has been taken from the Gaussian approximation to multiple
scattering (see {\tt [PHYS320]}). This condition is more restrictive,
because it is equivalent to require that the width of the Gaussian
part of the distribution be less than $0.5$, but it has been found that
the two conditions are numerically equivalent;
\item limitation due to the path length correction algorithm used (see
below).
\end{enumerate}
\subsection{Path Length Correction}

A path length correction may be applied in an approximate manner.
We have from the Fermi-Eyges theory \cite{bib-EGS4}
\begin{equation}
\label{eq:phys325-3}
t = S + \frac{1}{2}\int_{0}^{t}{\bar{\theta}^{2}\bar{(t)} \: dt}
\end{equation}
where
\[
\begin{array}{LL}
\bar{\theta}^2\bar{(t)}  & \mbox{the mean square angle of scattering;} \\
S                     & \mbox{step size along a straight line;} \\
t                      & \mbox{actual path length.}
\end{array}
\]

We have further:
\begin{equation}
\label{eq:phys325-4}
\bar{\theta}^2\bar{(t)} = \left( 0.0212 \frac{Z_{inc}}{p\beta} \right)^2
\frac{t}{X_0}
\end{equation}
$X_0$ is the radiation length. From (\ref{eq:phys325-3}) and
(\ref{eq:phys325-4}) we get
\begin{equation}
\label{eq:phys325-5}
S = t - K t^2
\makebox[4cm]{with}
K = 1.12 \times 10^{-4} \: \frac{Z_{inc}^2}{p^2\beta^2X_0}
\end{equation}
Equation (\ref{eq:phys325-5})
may be used to make the path length correction. Solving equation
(\ref{eq:phys325-5}) with respect to $t$ implies that, in order to
obtain real solutions:
\begin{equation}
\label{eq:phys325-6}
S \leq \frac{1}{4K} \makebox[3cm]{i.e.} S \leq 2232 \:
\frac{X_0 p^2 \beta^2}{Z_{inc}^2}
\end{equation}
This condition provides an additional constraint to the maximum step
length for multiple scattering (variable {\tt TMXCOR} in routine \Rind{GMULOF}).
The corrected step can be approximated as:
\[
t \approx S(1 + KS) = S ( 1 + \mbox{\tt CORR} )
\]
where {\tt CORR} $ \leq 0.25$ due to condition (\ref{eq:phys325-6}).


\section{Bayesian inference of species tree}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Species \& gene trees}

\begin{frame}\frametitle{Species tree}
	\begin{itemize}
		\item Species tree --- the phylogeny representing the relationships among a group of species
		\begin{figure}[h!]
 			\includegraphics[height=4cm]{figures/primateSpeciesTree}
			\figureCaption{\cite{Rogers:2014ka}}{}
		\end{figure}
		\item Gene tree --- the phylogeny for sequences at a particular gene locus from those species
	\end{itemize}
\end{frame}

\begin{frame}\frametitle{Gene tree discordance}
	\begin{itemize}
		\item Incomplete lineage sorting
	\end{itemize}
	\begin{figure}[h!]
 		\includegraphics[height=4cm]{figures/incompleteLineageSorting}
		\figureCaption{\cite{Patterson:2006cm}}{}
	\end{figure}
\end{frame}

\begin{frame}\frametitle{Gene tree discordance}
	\begin{itemize}
		\item Horizontal gene transfer
		\item Gene duplication and loss
		\begin{figure}[h!]
 			\includegraphics[height=3.5cm]{figures/HGTandDuplicationLoss}
 			\figureCaption{\cite{Degnan:2009hr}}{}
		\end{figure}
	\end{itemize}
\end{frame}

\begin{frame}\frametitle{Gene tree discordance}
	\begin{itemize}
		\item Hybridization
		\begin{figure}[h!]
 			\includegraphics[height=6cm]{figures/hybridization}
 			\figureCaption{\cite{Li:2016ko}}{}
		\end{figure}
	\end{itemize}
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{*BEAST}

\begin{frame}\frametitle{Species tree inference and *BEAST}
	\begin{itemize}
		\item A Bayesian method to infer species tree from multilocus sequence data \cite{Heled:2010ia}
		\item *BEAST, a functionality of BEAST2
	\end{itemize}
	\begin{columns}
	\column{4.6cm}
	\begin{itemize}
		\item Gene trees are embedded in the species tree under the multispecies coalescent model \cite{Rannala:2003vt}
		\begin{itemize}
			\item incomplete lineage sorting
		\end{itemize}
		\item Gene trees are independent among loci
	\end{itemize}
	\column{4.7cm}
    \begin{figure}[h!]
    	\includegraphics[width=1.0\textwidth]{figures/geneTreesInSpeciesTree}
  	\end{figure}
	\end{columns}
\end{frame}

%\begin{frame}\frametitle{Species tree inference and *BEAST}
%	\begin{itemize}
%		\item TODO: draw a graphical model representation
%	\end{itemize}
%\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Species tree prior}

\begin{frame}\frametitle{Species tree prior}
	\begin{itemize}
		\item The prior for species tree $S$ has two parts:
		\[ P(S) = P(S_T)P(N) \]
		\begin{itemize}
			\item $S_T$ --- species time tree
			\item $N$ --- population size functions
		\end{itemize}
		\item $P(S_T)$ --- typically a Yule (pure-birth) or birth-death prior
		\begin{itemize}
			\item we can assign a hyperprior for the speciation (birth) rate (and extinction (death) rate, if birth-death)
		\end{itemize}
		\item $P(N)$ --- constant or continuous-linear
	\end{itemize}
\end{frame}

\begin{frame}\frametitle{Species tree prior}
	\begin{itemize}
		\item Constant population sizes
		\begin{figure}[h!]
 			\includegraphics[height=5cm]{figures/popFunctionConstant}
 			\figureCaption{\cite{Drummond:2015tz}}{}
		\end{figure}
	\end{itemize}
\end{frame}

\begin{frame}\frametitle{Species tree prior}
	\begin{itemize}
		\item Continuous-linear population sizes
		\begin{figure}[h!]
 			\includegraphics[height=5.3cm]{figures/popFunctionLinear}
 			\figureCaption{\cite{Drummond:2015tz}}{}
		\end{figure}
	\end{itemize}
\end{frame}

\begin{frame}\frametitle{Species tree prior}
	\begin{itemize}
		\item In *BEAST, the prior type for $N$ is fixed to gamma
		\item The gamma shape parameter $k$ is fixed to 2, but we can assign a hyperprior for $\psi$, the scale parameter of the gamma
		\item (This $\psi$ parameter is called "population mean" in Beauti, but the prior mean is actually $2\psi$ when the population sizes are constant)
	\end{itemize}
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Multispecies coalescent}

\begin{frame}\frametitle{Multispecies coalescent model}
	\begin{itemize}
		\item The prior for gene tree $g$, given species tree $S$
	\end{itemize}
	\begin{figure}[h!]
 		\includegraphics[height=5.5cm]{figures/mscOverview}
 		\figureCaption{\cite{Drummond:2015tz}}{}
	\end{figure}
\end{frame}

\begin{frame}\frametitle{Multispecies coalescent model}
	\begin{itemize}
		\item The prob. distribution of gene time tree $g$ given species tree $S$, is:
		\[ P(g|S) = \prod_{j=1}^{2s-1} P(L_j(g)|N_j(t)) \]
	\end{itemize}
	\begin{columns}
	\column{5.4cm}
		\begin{itemize}
			\item $s$ --- number of extant species ($2s-1$ branches totally)
			\item $N_j(t)$ --- population size function (linear)
			\item $L_j(g)$ --- coalescent intervals for genealogy $g$ that are contained in the $j$'th branch of species tree $S$
		\end{itemize}
	\column{4cm}
    \begin{figure}[h!]
    	\includegraphics[width=1.0\textwidth]{figures/mscPopulation4}
  	\end{figure}
	\end{columns}
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Molecular clock model}

\begin{frame}\frametitle{Molecular clock model}
	\begin{itemize}
		\item $P(c)$ --- prior for the molecular clock model of genealogy $g$
		\begin{itemize}
			\item strict clock --- typically fix to 1.0 for the first locus, and infer the relative clock rates for the rest loci
			\item relaxed clock
		\end{itemize}
	\end{itemize}
	\begin{itemize}
		\item $P(\theta)$ --- prior for the substitution model parameters
		\item e.g. HKY85,
		\begin{itemize}
			\item Prior for transition/transversion rate ratio ($\kappa$), e.g. gamma(2,1)
  			\item Prior for base frequencies ($\pi_T, \pi_C, \pi_A, \pi_G$), e.g. Dirichlet(1,1,1,1)
		\end{itemize}
	\end{itemize}
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Felsenstein likelihood}

\begin{frame}\frametitle{Felsenstein likelihood}
	\begin{itemize}
		\item The probability (likelihood) of data $d_i$ (alignment at locus $i$), given the gene time tree $g_i$, molecular clock $c_i$, and substitution model $\theta_i$, is:
		\[ P(d_i|g_i,c_i,\theta_i) \]
	\end{itemize}
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Posterior distribution}

\begin{frame}\frametitle{Priors and likelihood}
	\begin{itemize}
		\item $P(S)$ --- prior for species tree
		\vskip 0.4cm
		\item $P(g_i|S)$ --- prior for gene tree $i$ (multispecies coalescent)
		\vskip 0.4cm
		\item $P(c_i)$ --- prior for clock rate of locus $i$
		\vskip 0.4cm
		\item $P(\theta_i)$ --- prior for substitution parameters of locus $i$
		\vskip 0.4cm
		\item $P(d_i|g_i,c_i,\theta_i)$ --- likelihood of data at locus $i$
	\end{itemize}
\end{frame}

\begin{frame}\frametitle{Posterior}
	\begin{itemize}
		\item The posterior distribution of the species tree $S$ and other paremeters given data $D$ is:
		\[ P(S, \mathbf{g, c}, \Theta|D) \propto P(S) \prod_{i=1}^n P(g_i|S) P(c_i) P(\theta_i) P(d_i|g_i,c_i,\theta_i) \]
		\item The data $D = \{d_1, d_2, \dots, d_n\}$ is composed of $n$ alignments, one per locus.
	\end{itemize}
\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{starBEAST2}

\begin{frame}\frametitle{Integrating out population sizes}
	\begin{itemize}
		\item Assume constant population sizes
		\item Assign i.i.d inverse-gamma($\alpha$, $\beta$) prior for $N_j$
		\begin{itemize}
			\item mean = $\beta / (\alpha -1)$
		\end{itemize}
		\item The population sizes $N$ can be integrated out from $P(g|S)$ \cite{Jones:2015jf}
		\vskip 0.4cm
		\item Specify $\alpha$ and $\beta$ in the invgamma prior (instead of $\psi$ in the gamma prior)
	\end{itemize}
\end{frame}

\begin{frame}\frametitle{starBEAST2}
	\begin{itemize}
		\item A more efficient implementation and an upgrade of *BEAST
		\begin{itemize}
			\item Population sizes integrated out \cite{Jones:2015jf}
			\item Relaxed molecular clock per species tree branch (instead of per gene tree branch)
			\item More efficient MCMC proposals for the species tree and gene trees (coordinated operators) \cite{Jones:2015jf, Rannala:2015wk}
		\end{itemize}
		\item Available at \url{github.com/genomescale/starbeast2}, will be released soon (as a BEAST2 add-on)
	\end{itemize}
\end{frame}