% $Id: ant8.tex,v 1.10 2002/10/10 02:38:49 ellard Exp $

\section{Overview}

This document describes the architecture of the 8-bit {\sc Ant-8} processor.

{\sc Ant-8} is a very small and simple processor.

It contains 16 user-visible registers:  14 general-purpose registers
and two special registers.

The instruction set consists of 16 instructions.

\section{{\sc Ant-8} Memory Organization}

The memory of the {\sc Ant-8} processor consists of 256 8-bit
bytes.  This memory is shared by the instructions and the
data.

The {\sc Ant-8} architecture is a load/store architecture; the only
instructions that can access memory are the {\em load} and
{\em store} instructions.  All other operations access only
registers.

\section{The {\sc Ant-8} Register Set}

The {\sc Ant-8} processor has 16 registers that can be accessed directly by
the programmer.  In {\sc Ant-8} assembler, they are named {\tt r0} through
{\tt r15}.  In {\sc Ant-8} machine language, they are the 4-bit numbers 0
through 15, and are usually written as a single hexadecimal digit
({\tt 0x0} through {\tt 0xf}).

Registers {\tt r2} through {\tt r15} are general purpose registers.
These registers can be used as both the source and destination
registers in any of the instructions that use source and destination
registers; they are read/write registers.

Registers {\tt r0} and {\tt r1} are not general purpose registers and
can be used only as source registers; they are read-only registers.
{\tt r0} always contains the constant zero, and {\tt r1} is used to
hold values related to the result of the previous instruction.  Some
instructions do not modify {\tt r1} at all, but several use {\tt r1}
to store useful results.

If an instruction attempts to write a value to either {\tt r0}
or {\tt r1}, the instruction executes in the normal manner,
but no changes are made to the register.

The program counter (or {\sc PC}) is a special 8-bit register that
contains the offset (or index) into memory of the next instruction to
execute.  Each instruction is 2 bytes long, and each instruction must
begin on an even address.  Note that the offset is interpreted as an
unsigned number and therefore ranges from $0 \cdots \MaxUIntWord$.
The {\sc PC} is not directly accessible to the program.

\section{Execution of Programs}

Programs are executed in the following manner:

\subsection{Initialization}

\begin{enumerate}

	\item Each location in memory is filled with zero.

	\item All of the registers are set to zero.

	\item The program counter (PC) is set to zero.

	\item The program is loaded into memory from a file.

		See section \ref{executable-file-sec} for information
		about the program file format.

	\item The fetch and execute loop (described in Section
		\ref{fetch-and-exec}) is executed until the program
		halts via the {\tt hlt} instruction, or because the
		execution encounters an error.

		Whenever the {\sc Ant-8} processor halts due to any error,
		it dumps core to a file named {\tt ant8.core}.

\end{enumerate}

\subsection{The Fetch and Execute Loop}
\label{fetch-and-exec}

\begin{enumerate}

\item Fetch the instruction at the offset in memory indicated
	by the {\sc PC}.

\item Set {\sc PC} $\leftarrow$ {\sc PC} $+ ~ 2$.

\item Execute the instruction.

	\begin{enumerate}

	\item Get the value of the source registers (if any).

	\item Perform the specified operation.

	\item Place the result, if any, into the destination register.

	\item Update register {\tt r1}, if necessary.

	\item Update the {\sc PC}, if necessary (only for branching or
			jumping instructions:  {\tt beq}, {\tt bgt}, and
			{\tt jmp} instructions).

	\end{enumerate}

\end{enumerate}

\section{The Instruction Set}

\subsection{Instruction Formats}

Most of the instructions have the following general instruction format:

\newcommand{\OPCODE}{{\em Operator \newline (4 bits)}}

\begin{description}

\item \ThreeRegisterOp{Op}{Description}
	{\parbox[t]{0.7in} \OPCODE }
	{\parbox[t]{0.7in}{Register 1 \\ (4 bits) }}
	{\parbox[t]{0.7in}{Register 2 \\ (4 bits) }}
	{\parbox[t]{0.7in}{Register 3 \\ (4 bits) }}
\end{description}

The lc (load constant), inc (increment), and I/O
({\tt in}/{\tt out}) instructions have the following format:

\begin{description}
\item \OneRegisterOp{Op}{Description}
	{\parbox[t]{0.7in} \OPCODE }
	{\parbox[t]{0.7in}{Register 1 \\ (4 bits) }}
	{\parbox[t]{0.7in}{Constant \\ (8 bits) }}
\end{description}

The final two exceptions to this are the {\tt ld1} (load) and {\tt
st1} (store) instructions, which have the following format:

\begin{description}
\item \TwoRegisterOp{Op}{Description}
	{\parbox[t]{0.7in} \OPCODE }
	{\parbox[t]{0.7in}{Register 1 \\ (4 bits) }}
	{\parbox[t]{0.7in}{Register 2 \\ (4 bits) }}
	{\parbox[t]{0.7in}{Constant \\ (4 bits) }}
\end{description}

The 4 bits of the {\em Operator} are a 1-digit hexadecimal number that
represents the name of the operator or instruction.  The 4 bits of the
Register(s) are the number of the register (i.e., {\tt 0x3} will
represent register 3).

\subsection{Notation}

The notation we will use to describe the operands of the
instructions is given in Figure \ref{operand-types-table}.

\begin{figure}[h]
\caption{ \label{operand-types-table} {\sc Ant-8} Machine Language Operand Types}
\vspace{3mm}
\begin{center}
\begin{tabular}{|lp{4.5in}|}
\hline

{\em des}       & Must always be a register index.  \\
{\em reg}       & Must always be a register index. \\
{\em src1}      & Must always be a register index. \\
{\em src2}      & Must always be a register index. \\
{\em const8}	& Must be an 8-bit constant
		($\MinIntWord \cdots \MaxIntWord$). \\
{\em uconst8}	& Must be an 8-bit constant
		($\MinUIntWord \cdots \MaxUIntWord$). \\
{\em uconst4}	& Must be a 4-bit constant integer
			($\MinUIntHWord \cdots \MaxUIntHWord$). \\
\hline
\end{tabular}

\vspace{3mm}
\begin{tabular}{|lrl|}
\hline
\MaxIntWord	&	127	& Maximum signed 8-bit integer. \\
\MinIntWord	&	-128	& Minimum signed 8-bit integer. \\
\hline
\MaxUIntWord	&	255	& Maximum unsigned 8-bit integer. \\
\MinUIntWord	&	0	& Minimum unsigned 8-bit integer. \\
\hline
\MaxUIntHWord	&	15	& Maximum unsigned 4-bit integer. \\
\MinUIntHWord	&	0	& Minimum unsigned 4-bit integer. \\
\hline
\end{tabular}
\end{center}
\vspace{3mm}
\end{figure}

Note that the same register can serve as both a source and destination
in one command.  For instance, you can double the contents of a
register by adding that register to itself and putting the result back
in that register, all in one command.

\subsection{Instruction Descriptions}

\begin{description}

\item	\renewcommand{\OPCODE}{{\tt 0 0 0 0}} \input{Arch/hlt}
\item	\renewcommand{\OPCODE}{{\tt 0 0 0 1}} \input{Arch/lc}
\item	\renewcommand{\OPCODE}{{\tt 0 0 1 0}} \input{Arch/inc}
\item	\renewcommand{\OPCODE}{{\tt 0 0 1 1}} \input{Arch/jmp}
\item	\renewcommand{\OPCODE}{{\tt 0 1 0 0}} \input{Arch/beq}
\item	\renewcommand{\OPCODE}{{\tt 0 1 0 1}} \input{Arch/bgt}
\item	\renewcommand{\OPCODE}{{\tt 0 1 1 0}} \input{Arch/ld1}
\item	\renewcommand{\OPCODE}{{\tt 0 1 1 1}} \input{Arch/st1}
\item	\renewcommand{\OPCODE}{{\tt 1 0 0 0}} \input{Arch/add}
\item	\renewcommand{\OPCODE}{{\tt 1 0 0 1}} \input{Arch/sub}
\item	\renewcommand{\OPCODE}{{\tt 1 0 1 0}} \input{Arch/mul}
\item	\renewcommand{\OPCODE}{{\tt 1 0 1 1}} \input{Arch/shf}
\item	\renewcommand{\OPCODE}{{\tt 1 1 0 0}} \input{Arch/and}
\item	\renewcommand{\OPCODE}{{\tt 1 1 0 1}} \input{Arch/nor}
\item	\renewcommand{\OPCODE}{{\tt 1 1 1 0}} \input{Arch/in}
\item	\renewcommand{\OPCODE}{{\tt 1 1 1 1}} \input{Arch/out}

\end{description}

\section{{\sc Ant-8} Executable Files}
\label{executable-file-sec}

{\sc Ant-8} program files are stored as text, as a sequence of
hexadecimal numbers, one per line.  Anything that appears after the
number on each line is ignored by the program loader, although it may
contain information that is used by the debugger.  Empty blank lines
or lines that begin with a \verb$#$ are ignored during the execution
of the program, although they may contain information used by the {\sc
Ant-8} debugger.  Each line in a program file must be less than 512
characters in length, and each line must end with a newline.

The program begins with the instructions, which are written as pairs
of 8-bit hexadecimal numbers.  If any of the hexadecimal numbers that
are supposed to represent instructions are too large to fit into 8
bits, then the program is invalid.

The loader reads bytes from the file until either the end of file is
reached, or 256 bytes have been read.

If there are fewer than 256 bytes specified in the file, the bytes not
specified are implicitly {\tt 0x00}.


\documentclass[journal]{IEEEtran}
\usepackage{amssymb}
\usepackage{lineno}
\usepackage{fixltx2e}
\usepackage{hyperref}
\usepackage{enumitem}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{listings}
\usepackage{tabularx}
\usepackage{float}
\usepackage{pbox}
\usepackage{xcolor}
\usepackage[hang, small,labelfont=bf,up,textfont=it,up]{caption} % Custom captions under/above floats in tables or figures
\usepackage{booktabs} % Horizontal rules in tables
\usepackage{microtype} % Slightly tweak font spacing for aesthetics
\usepackage[english]{babel} % Language hyphenation and typographical rules
\usepackage[normalem]{ulem}
\useunder{\uline}{\ul}{}

\lstset{ %
	language=C,                % choose the language of the code
	basicstyle=\footnotesize,       % the size of the fonts that are used for the code
	numbers=none,                   % where to put the line-numbers
	numberstyle=\footnotesize,      % the size of the fonts that are used for the line-numbers
	stepnumber=1,                   % the step between two line-numbers. If it is 1 each line will be numbered
	backgroundcolor=\color{white},  % choose the background color. You must add \usepackage{color}
	showspaces=false,               % show spaces adding particular underscores
	showstringspaces=false,         % underline spaces within strings
	showtabs=false,                 % sh-ow tabs within strings adding particular underscores
	frame=single,           % adds a frame around the code
	tabsize=2,          % sets default tabsize to 2 spaces
	captionpos=b,           % sets the caption-position to bottom
	breaklines=true,        % sets automatic line breaking
	breakatwhitespace=false,    % sets if automatic breaks should only happen at whitespace
	escapeinside={\%*}{*)}          % if you want to add a comment within your code
}

\DeclareMathOperator*{\argmax}{arg\,max}

\begin{document}

\title{Comparing five common classification algorithms on the ORL and MNIST data set}

%% use optional labels to link authors explicitly to addresses:
%% \author[label1,label2]{<author name>}
%% \address[label1]{<address>}
%% \address[label2]{<address>}

\author{Christian~M.~Lillelund,~\IEEEmembership{201408354@post.au.dk,~School of Engineering, Aarhus University}
\thanks{Report revised December 10, 2018.}}
\maketitle

%\address{School of Engineering, Aarhus University}

%\address{X}

\begin{abstract}
Today's computer systems offer a significant potential for training and testing machine learning techniques and classification algorithms, however the choice of algorithm depend on the problem at hand, one's amount of train and test data and the computational resources at one's disposal. In this report we review one dimensionality reduction technique and five common classification schemes and evaluate their performance on the ORL (human faces) and MNIST (hand-written numbers) data set. We find that simple algorithms like the Nearest Neighbor classifier work great performance and accuracy wise when training on the small data set, ORL, but struggle to keep up with more complex algorithms like a Perceptron using backpropagation on the large set, MNIST, as the dimensionality and amount of samples is significantly increased.
\end{abstract}

\begin{IEEEkeywords}
	IEEE, IEEEtran, journal, \LaTeX, paper, template.
\end{IEEEkeywords}

\section{Introduction}
For humans, it does not take much effort to tell the difference between a picture of a dog or a cat. A natural number or a letter. A happy person or a sad person. For computers, these sort of problems are notorious hard to solve and often require many computational resources\footnote{\cite{TerenceMills2018}}. Machine learning and computer vision deals with these issues as they encompass a range of algorithms and classification techniques to produce a model or scheme that can tell images apart and recognize similarities. In this report we study the recognition and classification of a data set containing human faces and one featuring hand-written numbers by implementing and testing five commonly used techniques (Nearest class centroid classifier, nearest sub-class centroid, nearest neighbor and two perceptron variants) using MATLAB. We split the data in a training and a test set, then train our respective classifier and evaluate its ability to classify correctly on the testing set. A version using all the dimensions of the training data samples and a principal component version will be applied. We use some MATLAB visualization techniques to better communicate the data representation and tables to compare them. We start by going over the basic theory behind the classification schemes, then look at the data, briefly go over implementation details and then turn to results. At the end we review and argue which scheme would make the most sense to use with these two classification problems.

Initially, we split the ORL image data set, containing 400 40x30 pixel faces of 40 different persons, and the MNIST set containing 70.000 28x28 pixel hand-written numbers from zero to nine. The data is vectorized, so we use built-in MATLAB functions to construct the original pictures as such:

\begin{figure}[h]
	\centering
	\includegraphics[width=0.7\linewidth]{orlfaces}
	\caption{Ten random images of faces from the ORL set.}
	\label{fig:orlfaces}
\end{figure}

\begin{figure}[h]
	\centering
	\includegraphics[width=0.7\linewidth]{mnistnumbers}
	\caption{Ten random images of numbers from the MNIST set.}
	\label{fig:mnistnumbers}
\end{figure}

\section{Theory}
This section will explain fundamental theory behind the dimensional reduction technique (PCA) and the five schemes used in this report.

\subsection{Principal Component Analysis}

PCA is a procedure that transform a set of observations or samples in dimension $D$ to a lower dimension $D-n$ while still preserving a smaller number of variables explaining the main features $X_{1}, X_{2}, ..., X_{p}$ in the original set\footnote{\cite{Casella}}. This is particularly useful when dealing with high dimension data, as this can be computationally hard and challenging to visualize. With PCA, we compute principal components $d$ of $n$ original samples with $p$ features, where $d$ is the desired output dimensionality and each dimension is a linear combination of the $p$ features. In practice, we find eigenvectors of the covariance matrix of the original data set, sort them by highest eigenvalue score and use these as weights $W$ in computing a linear transformation: $$y_{i} = W^T*x_{i}, i = 1, ..., N$$. The scattering of the transformed data is the scatter matrix given by \footnote{\cite{Iosifidis2017}}: $$S_{T} = \sum_{i=1}^{N} [W^T(x_{i}-\mu)][W^T(x_{i}-\mu)]^T$$. The weights $W$ can be found by applying eigenanalysis and taking the eigenvectors with the  highest score, more formally optimizing \footnote{\cite{Iosifidis2017}}: \[ W^* = \argmax_c Tr (W^TS_{T}) \] subject to $W^TW=I$. We end with a data set containing fewer ($d < D$) dimensions.

\subsection{Nearest class centroid classifier}

The NCC classifier assigns labels $l_{i}$ to $N$ observations determined by which class $c_{k}$'s mean (centroid) the observation $x_{i}$ is closest to. We make the assumption that each class follow a normal distribution, as they are given equal importance in the classification algorithm. The mean class vector is given by \footnote{\cite{Iosifidis2017}}: $$ \mu_{k} = \frac{1}{N_{k}} \sum_{i,li=k}^{} x_{i}, k = 1, ... K.$$

To classify any observation $x_{i}$ we find the smallest distance to any mean vector and assign the label of that vector to $x_{i}$, more formally: $$ d(x_{i},\mu_{k}) = ||x_{i}-\mu_{k}||^2_{2} $$

\subsection{Nearest subclass centroid classifier}

Similar to nearest class centroid, but each class $c_{k}$ now has subclasses $m$ that follow a normal distribution and has a mean vector $\mu_{km}$: $$ \mu_{km} = \frac{1}{N_{km}} \sum_{i,li=k, qi=m}^{} x_{i} $$ where $N_{km}$ denotes the number of observations in a given subclass and $x_{i}$ is a observation with a subclass label $qi$. Like NCC, the distance to the subclass mean is used to classify each observation \footnote{\cite{Iosifidis2017}}: $$ d(x_{i},\mu_{km}) = ||x_{i}-\mu_{km}||^2_{2} $$

The number of subclasses is a hyper parameter for NSC and must be decided prior. Nearest subclass classifier is a compromise between nearest mean and nearest neighbor and combines the flexibility of nearest neighbor with the robustness of nearest mean \footnote{\cite{Veenman2005}}. When the number of subclasses $m$ is equal to $N$ samples, we have the nearest neighbor classifier.

\subsection{Nearest neighbor classifier}

The nearest neighbor (NN) classifier is a simple algorithm where each sample is assigned to the class of its closest neighbor, or the most common class among its $k$ nearest neighbors in the k-NN variant. Pure NN is when $k=1$, but often $k>1$ where a majority vote takes place. The algorithm uses euclidean distance between a test sample $x_{i}$ with class $c_{k}$ and a training one $y_{i}$: $$ d(x_{i},y_{i}) = ||x_{i}-y_{i}||^2_{2} $$

\subsection{Perceptron learning with backpropagation}

A perceptron is a supervised algorithm of binary classifiers, that classify whether or not an input sample $x_{i}$ belong to a class $c_{k}$. It uses the linear discriminant function $f(x) = w^T*x + w_{0}$, where $w$ are weights, $x\varepsilon$ $\mathbb{R^D}$ a feature vector and $w_{0}$ the bias\footnote{\cite{Iosifidis2017}}. $w$ represent the orientation of the discriminant hyperplane, and $w_{0}$ a offset from the origin. Given weights $w$, sample $x_{i}$ and classes $c_{1}$ and $c_{2}$, the function splits the feature space into these two classes. If $g(w,x_{i}) > 0$, $x_{i}$ belongs to $c_{1}$ and if $g(w,x_{i}) < 0$, $x_{i}$ belongs to $c_{1}$. If $g(w,x_{i}) = 0$, it can be classified to either class. The resulting scalar is the distance of $x_{i}$ to the hyperplane. The function is used in our perceptron to follow, where we define a binary label $l_{i}$ for each sample to express the criterion function: $$ f(w^*,x_{i}) = l_{i} g(w,x_{i}) = l_{i} w^{*T} x_{i} \geq 0, i = 1, ..., N $$

All samples are correctly classified, if $f(w_{*}, x_{i}) \geq 0$. To produce such result, we need to optimize weights $w_{*}$ by the perceptron criterion function for $w_{*}$, where ${\jmath}_{p}(w_{*}) = 0$ would be a solution for\footnote{\cite{Iosifidis2017}} $w_{*}$: $$ {\jmath}_{p}(w) = \sum_{x_{i} \varepsilon \chi}^{} -f(w,x_{i}) = \sum_{x_{i} \varepsilon \chi}^{} -l_{i} w^T x_{i} $$

To optimize ${\jmath}_{p}(w)$, we use the gradient at $w$ and follow it to update $w$, where $\eta$ is the rate of change (learning rate) and expresses how fast it converges, and $\varepsilon$ express a vector set of mislabeled samples:

$$ w(t+1) = w(t) - \eta(t) \nabla{\jmath}_{p} = w(t) + \eta(t) \sum_{x_{i} \varepsilon \chi}^{} l_{i} x_{i} $$

At each learning iteration, the weights $w(t)$ are "punished" by the sum of misclassified samples scaled, leading to convergence as the result of criterion function ${\jmath}_{p}$ will be higher. This is known as backpropagation. The algorithm to come will elaborate on this.

\subsection{Perceptron learning with MSE}

As an alternative to backpropagation, one can apply least-mean square regression to obtain the perceptron weights $W$. Here we use a criterion function ${\jmath}_{LSE}$ that involves all of the samples, not just the misclassified ones. Previously we made the inner products of $f(X) = W^T*X$ positive, now we look to find a solution to $W^T X = t_{i}, i = 1, ..., N$, where $t_{i}$ are arbitrarily set positive constants. This is written as $X^T*w = b$ in matrix form. If X is nonsingular, we could write $W = X^-1 b$ and have a formal solution, but this is often not the case, however by obtaining the pseudo-inverse of $X$ one can find a solution for the weights\footnote{\cite{Duda2012}}: $$ X^\dagger = (X^tX)^{-1}X^T $$

Examining the criterion function, we look to minimize: $$ {\jmath}_{LSE} = ||W^T X - B||^2_{F} $$

By setting the derivative of this to zero with respect to $W$, we get: $$ \nabla{\jmath}_{LSE} = 0 => 2XX^W = 2XB^T $$

Hence: $$ W = (XX^T)^{-1} XB^T = W^\dagger B^T $$

If W is square and nonsingular, the pseudo-inverse is simply the regular inverse. Also it holds at $X^\dagger X = I$, but $X X^\dagger \neq I$, though a MSE always exists, hence $W = X^\dagger b$ is an MSE solution to $X^\dagger w = b$.

\section{Implementation}

The implementation uses MATLAB to design and run the different schemes on the training and test data. For understanding purposes, this section will briefly lay out the algorithms described in theory as pseudo-code.

\subsection{PCA}

The algorithm centers the data, then simply calculates eigenvectors from the variance matrix, sort them after highest eigenvalue and uses the $n$ vector as a principal component in representing the data. Listing \ref{lst:pca} shows the PCA as pseudo-code.

\begin{lstlisting}[caption=Implementation of PCA., label={lst:pca}, belowskip=-0.8 \baselineskip]
	pca_reduce(trainData, nDimensions) {
		trainData = trainData - mean(trainData) for rows in trainData
		eigVectors, eigValues = eig(cov(trainData))
		eigVectors = sort(eigVectors, 'descending')
		pc = eigVectors(from 1 to nDimensions)*trainData
	}
\end{lstlisting}

\subsection{NCC}

NCC simply calculates a mean vector for $c_{k}$ classes by summing the samples that belong to the class and dividing that with the number of $N$ samples in a given class, as expressed in listing \ref{lst:ncc}.

\begin{lstlisting}[caption=Implementation of NCC., label={lst:ncc}]
	train_ncc(trainData, trainLbls, nClasses) {
		mu = zero vector by size of trainData*nClasses
		n = zero vector by size of 1*nClasses
		index = 1;

		for (i = 1; i <= nClasses;i++) {
			n(i) = sum(num of classes i in trainLbls)
			mu(i) = sum(each traning sample from index to n(i))/n(i)
			index = index+n(i)
		}
	}
\end{lstlisting}

\subsection{NSC}

NSC makes an initial k-means clustering of the training data for $k=m$ subclasses as hyper parameter and use the resulting centroids to construct a new centroid matrix for each subclass $m$ spanning $c_{k}$*$m$. Listing \ref{lst:nsc} shows this.

\begin{lstlisting}[caption=Implementation of NSC., label={lst:nsc}]
	train_nsc(trainData, trainLbls, nClasses, nSubClasses) {
		centroids = zero vector by size of trainData*(nClasses*nSubClasses)
		n = zero vector by size of 1*nClasses
		index = 1;
		centroidIdx = vector to keep idx of centroids

		for (i = 1; i <= nClasses;i++) {
			n(i) = sum(num of classes i in trainLbls)
			[idx,clst] = kmeans(trainData for index+n(i), nSubClasses)
			centroids(for idx in centroidIdx) = clst
			index = index+n(i)-1
		} }
\end{lstlisting}

\subsection{NNC}

The algorithm takes as input the training and test data, as well as training labels. It then iterates through every test sample $x_i$ and compares to every single training sample $X_{i}$. It classifies $x_i$ to the class of $X_{i}$ it is closest too in Euclidean distance. Listing \ref{lst:nnc} shows this.

\begin{minipage}[H]{0.95\linewidth}
	\begin{lstlisting}[caption=Implementation of NNC., label={lst:nnc}]
	train_nnc(trainData, trainLbls, testData) {
		distances = zero vector by size testData*trainData
		resultLabels = zero vector by size testData*1
		nTestImages = num of test images
		nTrainImages = num of train images

		for (i = 1; i <= nTestImages; i++) {
			for (j = 1; j <= nTrainImages; j++) {
				distances(i,j) = norm(testData(i)-trainData(j), 2)^2.
			}
			trainIndex = min(distances(i))
			resultLabels = trainLbls(trainIndex)
		}
	\end{lstlisting}
\end{minipage}

\subsection{Perceptron with backpropagation}

The algorithm trains a perceptron to correctly classify each sample $x_{i}$ to class $c_{k}$. We start with an empty vector $\chi$, then create a label $l_{i}$ for sample $x_{i}$ setting it to 1 if it corresponds to our current class, otherwise -1. We then use a one-vs-rest criterion function $f(w,x)$ for the $x_{i}$'s, save the wrongly classified samples and labels, then summing the wrong samples times their label and use this value to update weights $w_{k}$ iteratively. The learning eta $\eta$ is a hyper parameter. Listing \ref{lst:pbp} shows the pseudo-code for this.

\begin{lstlisting}[caption=Implementation of a perceptron with backprogapation., label={lst:pbp}]
	train_perp_bp(trainData, trainLbls, eta, nClasses) {
	w = random W matrix with size trainData*nCLasses
	maxIters = max itertions
	for (k = 1; k <= nClasses; k++) {
		wrongLabels = empty matrix to save labels
		done = false, i = 0
		while (not done and nIters < maxIters) {
			X = empty matrix for wrong x_i's
			for (i = 1; i <= nTrainImages, i++) {
				x_i = trainData(i)
				if(trainlbls(i) = k) label = 1 else label = -1
				if (label*w(k)*x_i) < 0
					save x_i to X, save label to WrongLabels
			}
			sumOfWrongs = sum(WrongLabels * X)
			w(k) = w(k) + eta * sumOfWrongs
			ifEmpty(X)
				done = true
			}
		}
	}
\end{lstlisting}

\subsection{Perceptron with MSE}

The algorithm for MSE is simplified, but possibly more computational demanding. We start by finding the pseudo-inverse of our training data set, then create a label vector for each class $c_{k}$ with $l_{i} = 1$ if the label is in the class or $l_{i} = -1$ if it is not. Finally we find the MSE solution by multiplying the two for weight $W_{k}$. Listing \ref{lst:pmse} shows this.

\begin{lstlisting}[caption=Implementation of NSC., label={lst:pmse}]
	train_perp_mse(trainData, trainLbls, nClasses) {
		w = random W matrix with size trainData*nCLasses
		X = pseudo-inverse of trainData

		for (k = 1; k <= nClasses; k++) {
			Labels = empty matrix to save labels
			for (i = 1; i <= nTrainImages; i++)
				if (trainlbls(i) = k)
					save 1 to Labels
				else save -1 to Labels
			}
			w(:,k) = X*Labels;
		}
	}
\end{lstlisting}

\section{Results}

The five classification schemes are trained and tested on the ORL and MNIST data set with scatter plots to visualize the resulting labels compared to the testing ones and a confusion matrix that depicts the predicated and true classes. Each algorithm is tested with both the PCA and the full version of the data. All results were done with MATLAB R2018a on an Intel i5 4690K @ 3.50Ghz CPU. We start by showing the output of the PCA analysis, and then proceed to the classifications.

\subsection{PCA}

PCA was performed on both ORL and MNIST data set for number of dimensions $D=2$ with MATLAB. Table \ref{table:pca} details the computation time for PCA and figure \ref{fig:orlpca} and \ref{fig:mnistpca} shows a scatter plot with two principal components, i.e. the eigenvectors with highest eigenvalues.

\begin{table}[H]
	\centering
	\begin{tabular}{|l|l|l|l} \hline
		Data set & \pbox{18cm}{Method} & \pbox{5cm}{Execution time in $s$} \\ \hline
		ORL & PCA & 0.33 \\ \hline
		MNIST & PCA & 0.83 \\ \hline
	\end{tabular}
	\caption{PCA on ORL and MNIST data set.}
	\label{table:pca}
\end{table}

\begin{figure}[H]
	\centering
	\frame{\includegraphics[width=0.7\linewidth]{../ML/results/ORL_PCA}}
	\caption{Plot of two PC's on ORL. Data labels are colors.}
	\label{fig:orlpca}
\end{figure}

\begin{figure}[H]
	\centering
	\frame{\includegraphics[width=0.7\linewidth]{../ML/results/MNIST_PCA}}
	\caption{Plot of two PC's on MNIST. Data labels are colors.}
	\label{fig:mnistpca}
\end{figure}

\subsection{NCC}

Nearest Centroid was conducted on ORL and MNIST. To test the algorithm, we find the class $c_{k}$'s mean to which a sample $x_{i}$ is closest to by calculating the Euclidean distance (L2-norm) between a test sample and the mean vector. Table \ref{table:ncc} details performance and accuracy, while figure \ref{fig:orlnc} and \ref{fig:mnistnc} shows the resulting labels on scatter plots. The tests were performed on full-dimensional sets and PCA reduced sets. Accuracy is the number of resulting labels that equal the testing labels divided by the number of test samples.

\begin{table}[H]
	\centering
	\begin{tabular}{|l|l|l|} \hline
		Data set & \pbox{18cm}{Accuracy in $\%$} & \pbox{18cm}{Execution time in $s$} \\ \hline
		ORL & 94,1\% & 0.00 \\ \hline
		ORL w. PCA & 15,8\% & 0.00 \\ \hline
		MNIST & 75,2\% & 0.34 \\ \hline
		MNIST w. PCA & 14,9\% & 0.00 \\ \hline
	\end{tabular}
	\caption{Results for NCC performed on ORL and MNIST.}
	\label{table:ncc}
\end{table}

\begin{figure}[H]
	\centering
	\frame{\includegraphics[width=0.6\linewidth]{../ML/results/ORL_NC}}
	\caption{Plot results of NCC on ORL.}
	\label{fig:orlnc}
\end{figure}

\begin{figure}[H]
	\centering
	\frame{\includegraphics[width=0.6\linewidth]{../ML/results/MNIST_NC}}
	\caption{Plot results of NCC on MNIST.}
	\label{fig:mnistnc}
\end{figure}

\subsection{NSC}

Nearest Subclass Centroid was performed on ORL and MNIST with hyper-parameter $N_{sc}=2, N_{sc}=3, N_{sc}=5$ for number of subclasses. Our algorithm returns centroids for number of classes $N_{c}=N_{ck}*N_{sc}$. We classify a testing sample to a class of which its centroid it is closest to. Table \ref{table:nsc} details the result. $N_{sc}$ is number of subclasses. Figure \ref{fig:orlnsc} and \ref{fig:mnistnsc} show the NSC on the ORL and MNIST data set. The blue color represents test labels, the red color represents result labels. Each test sample is plotted to their class.

\begin{table}[H]
	\centering
	\begin{tabular}{|l|l|l|l|} \hline
		Data set & \pbox{18cm}{$N_{sc}$} & \pbox{18cm}{Accuracy in $\%$} & \pbox{18cm}{Execution time in $s$} \\ \hline
		ORL & 2 & 93.3 & 0.15 \\ \hline
		ORL & 3 & 95 & 0.15 \\ \hline
		ORL & 5 & 95.8 & 0.15 \\ \hline
		ORL w. PCA & 2 & 9.1 & 0.13 \\ \hline
		MNIST & 2 & 86.1 & 4.07 \\ \hline
		MNIST & 3 & 88 & 5.99 \\ \hline
		MNIST & 5 & 90 & 9.21 \\ \hline
		MNIST w. PCA & 2 & 14.3 & 0.14 \\ \hline
	\end{tabular}
	\caption{Results for NSC performed on ORL and MNIST.}
	\label{table:nsc}
\end{table}

\begin{figure}[H]
	\centering
	\frame{\includegraphics[width=0.6\linewidth]{../ML/results/ORL_NSC}}
	\caption{Plot results of NSC on ORL with $N_{sc}=2$.}
	\label{fig:orlnsc}
\end{figure}

\begin{figure}[H]
	\centering
	\frame{\includegraphics[width=0.6\linewidth]{../ML/results/MNIST_NSC}}
	\caption{Plot results of NSC on MNIST with $N_{sc}=2$.}
	\label{fig:mnistnsc}
\end{figure}

\subsection{NNC}

Nearest Neighbor classifier was performed on ORL and MNIST. To classify a test sample, the algorithm calculates the Euclidean distance to every training sample and assigns the test to whichever training sample it is closest to. Table \ref{table:nn} shows the results with accuracy and execution times. Table \ref{table:nnconfusion} is a confusion matrix to visualize the classification result. It shows samples known to be in class $c_{k}$ classified correctly to $c_{k}$, i.e. 973 samples were rightly classified to class 1.

\begin{table}[H]
	\centering
	\begin{tabular}{|l|l|l|} \hline
		Data set & \pbox{18cm}{Accuracy in $\%$} & \pbox{18cm}{Execution time in $s$} \\ \hline
		ORL & 95.8\% & 0.04 \\ \hline
		ORL w. PCA & 11.6\% & 0.01 \\ \hline
		MNIST & 96.9\% & 676.07 \\ \hline
		MNIST w. PCA & 14.9\% & 264.65 \\ \hline
	\end{tabular}
	\caption{Results of NNC performed on ORL and MNIST.}
	\label{table:nn}
\end{table}

\begin{table}[H]
	\centering
	\begin{tabular}{|l|l|l|l|l|l|l|l|l|l|l|} \hline
		{\ul }      & \textbf{1} & \textbf{2} & \textbf{3} & \textbf{4} & \textbf{5} & \textbf{6} & \textbf{7} & \textbf{8} & \textbf{9} & \textbf{10} \\ \hline
		\textbf{1}  & 973        & 1          & 1          & 0          & 0          & 1          & 3          & 1          & 0          & 0           \\
		\textbf{2}  & 0          & 1129       & 3          & 0          & 1          & 1          & 1          & 0          & 0          & 0           \\
		\textbf{3}  & 7          & 6          & 992        & 5          & 1          & 0          & 2          & 16         & 3          & 0           \\
		\textbf{4}  & 0          & 1          & 2          & 970        & 1          & 19         & 0          & 7          & 7          & 3           \\
		\textbf{5}  & 0          & 7          & 0          & 0          & 944        & 0          & 3          & 5          & 1          & 22          \\
		\textbf{6}  & 1          & 1          & 0          & 12         & 2          & 860        & 5          & 1          & 6          & 4           \\
		\textbf{7}  & 4          & 2          & 0          & 0          & 3          & 5          & 944        & 0          & 0          & 0           \\
		\textbf{8}  & 0          & 14         & 6          & 2          & 4          & 0          & 0          & 992        & 0          & 10          \\
		\textbf{9}  & 6          & 1          & 3          & 14         & 5          & 13         & 5          & 4          & 920        & 5           \\
		\textbf{10} & 2          & 5          & 1          & 6          & 10         & 5          & 1          & 11         & 1          & 967 \\ \hline
	\end{tabular}
	\caption{Confusion matrix for a NNC model tested on MNIST.}
	\label{table:nnconfusion}
\end{table}

\subsection{Perceptron with BP}

A perceptron was trained on both data sets with the learning rate $\eta$ as hyper-parameter for $\eta=0.01$, $\eta=0.1$, $\eta=1$ and $\eta=10$. For each test sample, we calculate the decision function $y = w(k)*x_{i}$ for each class $c_{k}$ and classify it to the class with maximizes the function. A maximum number of iterations per class is set to 100. Table \ref{table:perceptronbp} details the results, figure \ref{fig:orlperceptronbp} and \ref{fig:mnistperceptronbp} show plots of the resulting labels and \ref{fig:mnistperceptronlinebp1dpca} and \ref{fig:mnistperceptronlinebp2dpca} show the classification perceptron line in 1D and 2D with PCA.

\begin{table}[H]
	\centering
	\begin{tabular}{|l|l|l|l|} \hline
		Data set & \pbox{18cm}{$\eta$} & \pbox{18cm}{Accuracy in $\%$} & \pbox{18cm}{Execution time in $s$} \\ \hline
		ORL & 0.01 & 86.6 & 1.39 \\ \hline
		ORL & 0.1 & 84.1 & 1.30 \\ \hline
		ORL & 1 & 82.5 & 1.36 \\ \hline
		ORL & 10 & 84.1 & 1.34 \\ \hline
		ORL w. PCA & 0.1 & 4.1 & 1.13 \\ \hline
		MNIST & 0.01 & 87.3 & 131.39 \\ \hline
		MNIST & 0.1 & 86.6 & 115.50 \\ \hline
		MNIST & 1 & 86.2 & 115.60 \\ \hline
		MNIST & 10 & 79.3 & 113.25 \\ \hline
		MNIST w. PCA & 0.1 & 7.4 & 1.16 \\ \hline
	\end{tabular}
	\caption{Results for a perceptron with BP performed on ORL and MNIST.}
	\label{table:perceptronbp}
\end{table}

\begin{figure}[H]
	\centering
	\frame{\includegraphics[width=0.7\linewidth]{../ML/results/ORL_Perceptron_BP}}
	\caption{Plot results of training a perceptron with BP on ORL with $\eta=0.1$.}
	\label{fig:orlperceptronbp}
\end{figure}

\begin{figure}[H]
	\centering
	\frame{\includegraphics[width=0.7\linewidth]{../ML/results/MNIST_Perceptron_BP}}
	\caption{Plot results of training a perceptron with BP on MNIST with $\eta=1$.}
	\label{fig:mnistperceptronbp}
\end{figure}

\begin{figure}[H]
	\centering
	\frame{\includegraphics[width=0.7\linewidth]{../ML/results/MNIST_PerceptronLine_BP_1D_PCA}}
	\caption{Plot lines of perceptron with BP for classes on MNIST in 1D.}
	\label{fig:mnistperceptronlinebp1dpca}
\end{figure}

\begin{figure}[H]
	\centering
	\frame{\includegraphics[width=0.7\linewidth]{../ML/results/MNIST_PerceptronLine_BP_2D_PCA}}
	\caption{Plot lines of perceptron with BP for classes on MNIST in 2D.}
	\label{fig:mnistperceptronlinebp2dpca}
\end{figure}

\subsection{Perceptron with MSE}

A perceptron using least-squares was trained on both data sets. Like with backpropagation, we find the class that maximize the decision function $y = w(k)*x_{i}$ for each sample and classify it to that class. Table \ref{table:perceptronmse} details the results, figure \ref{fig:orlperceptronmse} and \ref{fig:mnistperceptronmse} shows plots with the resulting labels.

\begin{table}[H]
	\centering
	\begin{tabular}{|l|l|l|} \hline
		Data set & \pbox{18cm}{Accuracy in $\%$} & \pbox{18cm}{Execution time in $s$} \\ \hline
		ORL & 90\% & 0.05 \\ \hline
		ORL w. PCA & 10\% & 0.01 \\ \hline
		MNIST & 86\% & 6.79 \\ \hline
		MNIST w. PCA & 18.4\% & 0.53 \\ \hline
	\end{tabular}
	\caption{Results for a perceptron with least-squares (MSE) performed on ORL and MNIST.}
	\label{table:perceptronmse}
\end{table}

\begin{figure}[H]
	\centering
	\frame{\includegraphics[width=0.7\linewidth]{../ML/results/ORL_Perceptron_MSE}}
	\caption{Plot results of training a perceptron with least-squares (MSE) on ORL.}
	\label{fig:orlperceptronmse}
\end{figure}

\begin{figure}[H]
	\centering
	\frame{\frame{\includegraphics[width=0.7\linewidth]{../ML/results/MNIST_Perceptron_MSE}}}
	\caption{Plot results of training a perceptron with least-squares (MSE) on MNIST.}
	\label{fig:mnistperceptronmse}
\end{figure}

\section{Discussion}

As expected, the algorithms performed differently given the complexity and dimensionality of the data set. If a classification scheme is fast and accurate on a small data set, like ORL ($D=1200$), it might not replicate on a larger set like MNIST ($D=60000$). PCA showed a computational easy way to reduce dimensionality, but accuracy with PCA data lacked in all our tests, averaging around 15\%, even though the execution times were similar to that of the full dimensional sets. Nearest Centroid Classifier (NSC) gained an impressive accuracy on ORL (94.1\%), but struggled on MNIST (75.2\%). Nearest Subclass Centroid (NSC) proved high classification precision, with a 2\% difference between 2 and 5 subclasses on ORL, and 3.9\% on MNIST, however execution times doubled on MNIST with five subclasses compared to just two. Nearest Neighbor (NNC) gave us the best results in terms of accuracy (95.8\% and 96.9\%), but suffered from very high execution times. Our perceptron with BP showed that a lower learning rate $\eta < 0.01$ gave better accuracy with similar execution times as tests with higher learning rate $\eta > 1$. The perceptron performed faster than nearest neighbor (around 2 and 11 minutes on MNIST, respectively) with similar accuracy, showing that a perceptron should be preferred over NNC on larger data sets. The one using least-squares (MSE) gave a sound accuracy on both sets, lower execution times and is a much more simple algorithm to implement and comprehend, though solving the generalized inverse often requires plenty computational resources.

\section{Conclusion}

In this report we have explained the workings of five common classification algorithms including principal component analysis (PCA), the mathematics behind them and benchmarked them on two famous data sets, the ORL set containing 40 different types of facial images and the MNIST set featuring images of numbers from zero to nine. We found that simple algorithms like nearest neighbor classifier (NNC) performs excellent on simple data sets with low dimensionality (like ORL), but struggles on more complex sets like MNIST. In such case, we found that a perceptron trained either by backpropagation or MSE is more suited because it is computational more cost effective, given a boundary is set on number of convergence attempts.

\bibliographystyle{plainnat}
\bibliography{literature}

\end{document}


%==========================================================================
\chapter{ROSE-HPCToolKit Interface}
\label{chap:rosehpct}
ROSE-HPCToolKit is designed to read in performance data generated
by HPCToolkit (and recently GNU gprof) and annotate ROSE AST with performance
metrics.
It is included in the ROSE distribution and
enabled by default if  an existing installation of the Gnome XML library, libxml2
(\url{http://xmlsoft.org}) can be detected by ROSE's configure script.
Or it can be enabled explicitly by specifying the \texttt{--enable-rosehpct} option
when running configure.
% and be sure that you have an existing installation of the Gnome XML library, libxml2 (\url{http://xmlsoft.org}).

The HPCToolkit (\url{http://www.hipersoft.rice.edu/hpctoolkit}) is a
set of tools for analyzing the dynamic performance behavior of
applications. It includes a tool that instruments a program's binary,
in order to observe CPU hardware counters during execution; additional
post-processing tools attribute the observed data to statements in the
original source code. HPCToolkit stores this data and the source
attributions in XML files. In this chapter, we give an overview of
simple interfaces in ROSE that can read this data and attach it to the
AST.

GNU gprof is a basic but easy to use profiling tool.
It produces an execution profile of applications.
gprof's output is a flat profile for each function by default, which is not
very interesting to us.
We use a line-by-line output with full file path information generated by using
option {\tt -l -L} with gprof.

\input{\TutorialExampleBuildDirectory/roseHPCT/is-static}

%==========================================================================
\section{An HPCToolkit Example Run}
\label{chap:rosehpct:run}

%--------------------------------------------------------------------------
\begin{figure}[!h]
{\indent
{\mySmallFontSize
% Do this when processing latex to generate non-html (not using latex2html)
\begin{latexonly}
   \lstinputlisting[language=C,numbers=left,stepnumber=5]{\TutorialExampleBuildDirectory/roseHPCT/profiled.c.aa}
\end{latexonly}

% Do this when processing latex to build html (using latex2html)
\begin{htmlonly}
   \verbatiminput{\TutorialExampleBuildDirectory/roseHPCT/profiled.c.aa}
\end{htmlonly}

% end of scope in font size
}
% End of scope in indentation
}
\caption{profiled.c (part 1 of 2): Sample input program, profiled using the HPCToolkit.}
\label{Tutorial:roseHPCT:profiled:aa}
\end{figure}
%--------------------------------------------------------------------------

%--------------------------------------------------------------------------
\begin{figure}[!h]
{\indent
{\mySmallFontSize
% Do this when processing latex to generate non-html (not using latex2html)
\begin{latexonly}
   \lstinputlisting[language=C,numbers=left,stepnumber=5,firstnumber=66]{\TutorialExampleBuildDirectory/roseHPCT/profiled.c.ab}
\end{latexonly}

% Do this when processing latex to build html (using latex2html)
\begin{htmlonly}
   \verbatiminput{\TutorialExampleBuildDirectory/roseHPCT/profiled.c.ab}
\end{htmlonly}

% end of scope in font size
}
% End of scope in indentation
}
\caption{profiled.c (part 2 of 2): Sample input program, profiled using the HPCToolkit.}
\label{Tutorial:roseHPCT:profiled:ab}
\end{figure}
%--------------------------------------------------------------------------

Consider the sample source program shown in
Figures~\ref{Tutorial:roseHPCT:profiled:aa}--\ref{Tutorial:roseHPCT:profiled:ab}.
This program takes an integer $n$ on the command line, and has a
number of loops whose flop and memory-operation complexity are either
$\Theta(n)$ or $\Theta(n^2)$. For this example, we would expect the
loop nest at line 56, which has $O(n^2)$ cost, to be the most
expensive loop in the program for large $n$.

%--------------------------------------------------------------------------
\begin{figure}[!h]
{\indent
{\mySmallFontSize
% Do this when processing latex to generate non-html (not using latex2html)
\begin{latexonly}
   \lstinputlisting[language=XML,numbers=left,stepnumber=5]{\TutorialExampleBuildDirectory/roseHPCT/PAPI_TOT_CYC.xml.aa}
\end{latexonly}

% Do this when processing latex to build html (using latex2html)
\begin{htmlonly}
   \verbatiminput{\TutorialExampleBuildDirectory/roseHPCT/PAPI_TOT_CYC.xml.aa}
\end{htmlonly}

% end of scope in font size
}
% End of scope in indentation
}
\caption{XML schema for HPCToolkit data files: This schema, prepended
to each of the HPCToolkit-generated XML files, describes the format of
the profiling data. This particular schema was generated by HPCToolkit
1.0.4.}
\label{Tutorial:roseHPCT:xml:schema}
\end{figure}
%--------------------------------------------------------------------------

%--------------------------------------------------------------------------
\begin{figure}[!h]
{\indent
{\mySmallFontSize
% Do this when processing latex to generate non-html (not using latex2html)
\begin{latexonly}
   \lstinputlisting[language=XML,numbers=left,stepnumber=5,firstnumber=62]{\TutorialExampleBuildDirectory/roseHPCT/PAPI_TOT_CYC.xml.ab}
\end{latexonly}

% Do this when processing latex to build html (using latex2html)
\begin{htmlonly}
   \verbatiminput{\TutorialExampleBuildDirectory/roseHPCT/PAPI_TOT_CYC.xml.ab}
\end{htmlonly}

% end of scope in font size
}
% End of scope in indentation
}
\caption{PAPI\_TOT\_CYC.xml: Sample cycle counts observed during
profiling, generated from running the HPCToolkit on profiled.c
(Figures~\ref{Tutorial:roseHPCT:profiled:aa}--\ref{Tutorial:roseHPCT:profiled:ab}.)
These lines would appear after the schema shown in
Figure~\ref{Tutorial:roseHPCT:xml:schema}.}
\label{Tutorial:roseHPCT:xml:cycles}
\end{figure}
%--------------------------------------------------------------------------

%--------------------------------------------------------------------------
\begin{figure}[!h]
{\indent
{\mySmallFontSize
% Do this when processing latex to generate non-html (not using latex2html)
\begin{latexonly}
   \lstinputlisting[language=XML,numbers=left,stepnumber=5,firstnumber=62]{\TutorialExampleBuildDirectory/roseHPCT/PAPI_FP_OPS.xml.ab}
\end{latexonly}

% Do this when processing latex to build html (using latex2html)
\begin{htmlonly}
   \verbatiminput{\TutorialExampleBuildDirectory/roseHPCT/PAPI_FP_OPS.xml.ab}
\end{htmlonly}

% end of scope in font size
}
% End of scope in indentation
}
\caption{PAPI\_FP\_OPS.xml: Sample flop counts observed during
profiling, generated from running the HPCToolkit on profiled.c
(Figures~\ref{Tutorial:roseHPCT:profiled:aa}--\ref{Tutorial:roseHPCT:profiled:ab}.)
These lines would appear after the schema shown in
Figure~\ref{Tutorial:roseHPCT:xml:schema}.}
\label{Tutorial:roseHPCT:xml:flops}
\end{figure}
%--------------------------------------------------------------------------

Suppose we use the HPCToolkit to profile this program, collecting
cycle counts and floating-point instructions.\footnote{In this
example, HPCToolkit uses the PAPI to read CPU counters
(\url{http://icl.cs.utk.edu/papi}).} HPCToolkit will generate one XML
file for each metric.

A schema specifying the format of these XML files appears in
Figure~\ref{Tutorial:roseHPCT:xml:schema}. In essence, this schema
specifies that the XML file will contain a structured, abstract
representation of the program in terms of abstract program entities
such as ``modules,'' ``procedures,'' ``loops,'' and ``statements.''
Each of these entities may have line number information and a
``metric'' value. (Refer to the HPCToolkit documentation for more
information.) This schema is always the first part of an
HPCToolkit-generated XML profile data file.

We ran HPCToolkit on the program in
Figures~\ref{Tutorial:roseHPCT:profiled:aa}--\ref{Tutorial:roseHPCT:profiled:ab},
and collected cycle and flop counter data. The actual XML files
storing this data appear in Figures~\ref{Tutorial:roseHPCT:xml:cycles}
and~\ref{Tutorial:roseHPCT:xml:flops}. By convention, these metrics
are named according to their PAPI symbolic name, as shown on line 67
in both Figures~\ref{Tutorial:roseHPCT:xml:cycles}
and~\ref{Tutorial:roseHPCT:xml:flops}. According to the cycle data on
line 90 of Figure~\ref{Tutorial:roseHPCT:xml:cycles}, the most
expensive statement in profiled.c is line 62 of
Figure~\ref{Tutorial:roseHPCT:profiled:aa}, as expected.

%==========================================================================
\clearpage
\section{Attaching HPCToolkit Data to the ROSE AST}
\label{chap:rosehpct:attach}

To attach the data of Figures~\ref{Tutorial:roseHPCT:xml:cycles}
and~\ref{Tutorial:roseHPCT:xml:flops} to the AST, we augment a basic
ROSE translator with two additional calls, as shown in
Figure~\ref{Tutorial:roseHPCT:attach}, lines 47--48 and 54. We
describe these calls below.

%--------------------------------------------------------------------------
\begin{figure}[!h]
{\indent
{\mySmallFontSize

% Do this when processing latex to generate non-html (not using latex2html)
\begin{latexonly}
   \lstinputlisting[numbers=left,stepnumber=5]{\TutorialExampleDirectory/roseHPCT/attachMetrics.cc}
\end{latexonly}

% Do this when processing latex to build html (using latex2html)
\begin{htmlonly}
   \verbatiminput{\TutorialExampleDirectory/roseHPCT/attachMetrics.cc}
\end{htmlonly}

% end of scope in font size
}
% End of scope in indentation
}
\caption{attachMetrics.cc: Sample translator to attach HPCToolkit metrics to the AST.}
\label{Tutorial:roseHPCT:attach}
\end{figure}
%--------------------------------------------------------------------------

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Calling ROSE-HPCT}

We must first include rosehpct/rosehpct.hh, as shown on line 6 of
Figure~\ref{Tutorial:roseHPCT:attach}. All ROSE-HPCT routines and
intermediate data structures reside in the RoseHPCT namespace.

Next, lines 47--48 of Figure~\ref{Tutorial:roseHPCT:attach} store the
contents of the raw XML file into an intermediate data structure of
type RoseHPCT::ProgramTreeList\_t. The RoseHPCT::loadProfilingFiles()
routine processes command-line arguments, extracting
ROSE-HPCT-specific options that specify the files. We discuss these
options in Section~\ref{sec:rosehpct:cmdlineopts}.

Line 54 of Figure~\ref{Tutorial:roseHPCT:attach}, attaches the
intermediate profile data structure to the ROSE AST. The
RoseHPCT::attachMetrics() routine creates new persistent attributes
that store the counter data.\footnote{The last parameter to
RoseHPCT::attachMetrics() is a boolean that, when true, enables
verbose (debugging) messages to standard error.}  The attributes are
named using the metric name taken from the XML file (see lines 67 of
Figures~\ref{Tutorial:roseHPCT:xml:cycles}--\ref{Tutorial:roseHPCT:xml:flops});
in this example, the attributes are named PAPI\_TOT\_CYC and
PAPI\_FP\_OPS. Following the conventions of persistent attribute
mechanism as described in Chapter~\ref{chap:traversals}, the
attributes themselves are of type RoseHPCT::MetricAttr, which derives
from the AstAttribute type.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Retrieving the attribute values}

We retrieve the attribute values as described in
Chapter~\ref{chap:traversals}.  In particular, given a located node
with cycle and flop attribute values, the printFlopRate() routine
defined in lines 11--42 of Figure~\ref{Tutorial:roseHPCT:attach}
prints the source position, AST node type, and estimated flops per
cycle. We call printFlopRate() for each expression statement
(SgExpressionStmt), for-initializer (SgForInitStatement), and
for-statement (SgForStatement) in lines 59--66 of
Figure~\ref{Tutorial:roseHPCT:attach}. The output is shown in
Figure~\ref{Tutorial:roseHPCT:profiled:out}.

Inspecting the output carefully, you may notice seeming discrepancies
between the values shown and the values that appear in the XML files,
or other values which seem unintuitive. We explain how these values
are derived in Section~\ref{sec:rosehpct:metprop}.

This example dumps the AST as a PDF file, as shown on line 68 of
Figure~\ref{Tutorial:roseHPCT:attach}. You can inspect this file to
confirm where attributes have been attached. We show an example of
such a page in Figure~\ref{Tutorial:roseHPCT:pdf}. This page is the
SgExprStatement node representing the sum-accumulate on line 26 of
Figure~\ref{Tutorial:roseHPCT:profiled:aa}.

\begin{figure}[!ht]
{\mySmallFontSize
  \verbatiminput{\TutorialExampleBuildDirectory/roseHPCT/profiled.out}
}
\caption{Sample output, when running attachMetrics.cc
(Figure~\ref{Tutorial:roseHPCT:attach}) with the XML inputs in
Figures~\ref{Tutorial:roseHPCT:xml:cycles}--\ref{Tutorial:roseHPCT:xml:flops}. Here,
we only show the output sent to standard output (\emph{i.e.}, cout and
not cerr).}
\label{Tutorial:roseHPCT:profiled:out}
\end{figure}

\begin{figure}[!hb]
\includegraphics[width=6in,trim=0in 7.8in 2in 0in, clip]{\TutorialExampleBuildDirectory/roseHPCT/profiled-p93.pdf}
\caption{Sample PDF showing attributes.}
\label{Tutorial:roseHPCT:pdf}
\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Metric propagation}
\label{sec:rosehpct:metprop}

The example program in Figure~\ref{Tutorial:roseHPCT:attach} dumps
metric values at each expression statement, for-initializer, and
for-statement, but the input XML files in
Figure~\ref{Tutorial:roseHPCT:xml:cycles}--\ref{Tutorial:roseHPCT:xml:flops}
only attribute the profile data to ``statements'' that are not loop
constructs. (The \texttt{<S \ldots>} XML tags refer to statements,
intended to be ``simple'' non-scoping executable statements; a
separate \texttt{<L \ldots>} tag would refer to a loop.) Since the XML
file specifies statements only by source line number,
RoseHPCT::attachMetrics() attributes measurements to AST nodes in a
heuristic way.

For example, lines 78--80 of Figure~\ref{Tutorial:roseHPCT:xml:cycles}
indicate that all executions of the ``simple statements'' of line 25
of the original source (Figure~\ref{Tutorial:roseHPCT:profiled:aa})
accounted for 65534 observed cycles, and that line 26 accounted for an
additional 65534 cycles. In the AST, there are multiple ``statement''
and expression nodes that occur on line 25; indeed,
Figure~\ref{Tutorial:roseHPCT:profiled:out} lists 4 such
statements.

The ROSE-HPCT modules uses a heuristic which only assigns \texttt{<S
\ldots>} metric values to non-scoping nodes derived from
SgStatement. When multiple SgStatement nodes occur at a particular
source line,
%ROSE-HPCT simply divides the observed metric equally among all the SgStatement nodes on that line.
ROSE-HPCT simply attaches the metric to each of them.
But only one of them will be used for propagating metrics to parent scopes.

How is the measurement of 65534 cycles attributed to all of the AST
nodes corresponding to line 25 of
Figure~\ref{Tutorial:roseHPCT:profiled:aa}? Indeed, line 25 actually
``contains'' four different SgStatement nodes: an SgForStatement
representing the whole loop on lines 25--26, an SgForInitStatement
(initializer), and two SgExprStatements (one which is a child of the
SgForInitStatement, and another for the for-loop's test
expression). The loop's increment is stored in the SgForStatement node
as an SgExpression, not an SgStatement. The SgForStatement node is a
scoping statement, and so it ``receives'' none of the 65534
cycles. Since the increment is not a statement and one of the
SgExprStatements is a child of the initializer, this leaves only two
direct descendants of the SgForStatement---the initializer and the
test expression statement---among which to divide the 65534
cycles. Thus, each receives 32767 cycles. The initializer's
SgExprStatement child gets the same 32767 as its parent, since the two
nodes are equivalent (see first two cases of
Figure~\ref{Tutorial:roseHPCT:profiled:out}).

For the entire loop on lines 25--26 of
Figure~\ref{Tutorial:roseHPCT:profiled:aa}, the original XML files
attribute 65534 cycles to line 25, and another 65534 cycles to line 26
(see Figure~\ref{Tutorial:roseHPCT:xml:cycles}). Moreover, the XML
files do not attribute any costs to this loop \emph{via} an explicit
\texttt{<L \ldots>} tag. Thus, the best we can infer is that the
entire for-statement's costs is the sum of its immediate child costs;
in this case, 131068 cycles. The RoseHPCT::attachMetrics() routine
will heuristically accumulate and propagate metrics in this way to
assign higher-level scopes approximate costs.

The RoseHPCT::attachMetrics() routine automatically propagates metric
values through parent scopes. A given metric attribute,
RoseHPCT::MetricAttr* x, is ``derived'' through propagation if
x-$>$isDerived() returns true. In fact, if you call x-$>$toString() to
obtain a string representation of the metric's value, two asterisks
will be appended to the string as a visual indicator that the metric
is derived. We called RoseHPCT::MetricAttr::toString() on lines 27 and
29 of Figure~\ref{Tutorial:roseHPCT:attach}, and all of the
SgForStatement nodes appearing in the output in
Figure~\ref{Tutorial:roseHPCT:profiled:out} are marked as derived.

Alternatively, you cann call RoseHPCT::attachMetricsRaw(), rather than
calling RoseHPCT::attachMetrics().  The ``raw'' routine takes the same
arguments but only attaches the raw data, \emph{i.e.}, without
attempting to propagate metric values through parent scopes.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Working with GNU gprof}
ROSE-HPCT can also accept the line-by-line profiling output generated by
GNU gprof.
Currently, we only use the self seconds associated with each line
and attach them to ROSE AST as AST attributes named {\tt WALLCLK}.

A typical session to generate compatible gprof profiling file for ROSE-HPCT
is given below:
\begin{verbatim}
[liao@codes]$ gcc -g seq-pi.c -pg
[liao@codes]$ ./a.out
[liao@codes]$ gprof -l -L a.out gmon.out &>profile.result
\end{verbatim}

{\tt -l} tells gprof to output line-by-line profiling information and {\tt
-L} causes gprof to output full file path information.

An excerpt of an output file looks like the following:
{\scriptsize
\begin{verbatim}
Flat profile:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total
 time   seconds   seconds    calls  Ts/call  Ts/call  name
 38.20      8.84     8.84                             jacobi (/home/liao6/temp/jacobi.c:193 @ 804899c)
 36.43     17.27     8.43                             jacobi (/home/liao6/temp/jacobi.c:196 @ 8048a3f)
 11.00     19.82     2.54                             jacobi (/home/liao6/temp/jacobi.c:188 @ 804893e)
  5.66     21.12     1.31                             jacobi (/home/liao6/temp/jacobi.c:187 @ 8048968)
  3.93     22.04     0.91                             jacobi (/home/liao6/temp/jacobi.c:197 @ 8048a71)
  3.24     22.79     0.75                             jacobi (/home/liao6/temp/jacobi.c:191 @ 8048a7f)
  0.95     23.00     0.22                             jacobi (/home/liao6/temp/jacobi.c:186 @ 8048976)
  0.50     23.12     0.12                             jacobi (/home/liao6/temp/jacobi.c:187 @ 8048935)
  0.09     23.14     0.02                             jacobi (/home/liao6/temp/jacobi.c:190 @ 8048a94)
  0.00     23.14     0.00        1     0.00     0.00  driver (/home/liao6/temp/jacobi.c:91 @ 8048660)
  0.00     23.14     0.00        1     0.00     0.00  error_check (/home/liao6/temp/jacobi.c:220 @ 8048b7c)
  0.00     23.14     0.00        1     0.00     0.00  initialize (/home/liao6/temp/jacobi.c:116 @ 8048722)
  0.00     23.14     0.00        1     0.00     0.00  jacobi (/home/liao6/temp/jacobi.c:160 @ 8048892)
\end{verbatim}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Command-line options}
\label{sec:rosehpct:cmdlineopts}

The call to RoseHPCT::loadProfilingFiles() on line 49 of
Figure~\ref{Tutorial:roseHPCT:attach} processes and extracts
ROSE-HPCT-specific command-line options. To generate the output in
this chapter, we invoked Figure~\ref{Tutorial:roseHPCT:attach} with
the following command-line:

\verbatiminput{\TutorialExampleBuildDirectory/roseHPCT/command-line}

The main option is \texttt{-rose:hpct:prof <file>}, which specifies
the HPCToolkit-generated XML file containing metric data. Here, we use
this option twice to specify the names of the cycle and flop data
files
(Figures~\ref{Tutorial:roseHPCT:xml:cycles}--\ref{Tutorial:roseHPCT:xml:flops}).
To accept gprof output file, please use another option \texttt{-rose:gprof:linebyline <file>}.
This option cannot be used with \texttt{-rose:hpct:prof <file>} currently.

We need the other option, \texttt{-rose:hpct:eqpath <A>=<B>}, to
specify how paths in the HPCToolkit XML files can be mapped to file
paths in the ROSE AST.
This option allows users to generate performance files on one machine and
analyze the results on another machine.
In this example, the XML
files specify the source file as, ``./profiled.c'' (line 73 of
Figures~\ref{Tutorial:roseHPCT:xml:cycles}
and~\ref{Tutorial:roseHPCT:xml:flops}); the ``eqpath'' command-line
option above remaps the relative path ``.'' to an absolute path as it
would appear in the ROSE AST.
Another example is to use the same performance file even after the original source tree is moved to another location.
ROSE-HPCT can still correctly match performance data if the root source paths are given as \texttt{-rose:hpct:eqpath <oldRootPath>=<newRootPath>}.

Yet another option  \texttt{-rose:hpct:enable\_debug} is provided to display runtime debugging information
such as metrics reading, attaching, and propagating.
It also adds performance metrics into the ROSE output source file as source comments as shown below.
Users can examine the source comments to make sure performance metrics are attached and propagated properly.
As we can see, ROSE-HPCT attaches each performance metric to each matching statement.
If there are multiple statements showing in the same line, the same metric will be attached to each of them.
The metric propagation step will only propagate one of them to upper-level language constructs to ensure the correctness.

{\scriptsize
\begin{verbatim}
/* ROSE-HPCT propagated metrics WALLCLK:18.95[SgForStatement at 0xb7beb218] */
  for (
/* ROSE-HPCT raw data: Statement  WALLCLK:0.02@File jacobi.c  190-0 -> SgForInitStatement 0x94e8d08 at 190 */
i = 1;
/* ROSE-HPCT raw data: Statement  WALLCLK:0.02@File jacobi.c  190-0 -> SgExprStatement 0x94516d8 at 190 */
i < (n - 1); i++)
/* ROSE-HPCT propagated metrics WALLCLK:18.93[SgForStatement at 0xb7beb29c] */
    for (
/* ROSE-HPCT raw data: Statement  WALLCLK:0.75@File jacobi.c  191-0 -> SgForInitStatement 0x94e8d38 at 191 */
j = 1;
/* ROSE-HPCT raw data: Statement  WALLCLK:0.75@File jacobi.c  191-0 -> SgExprStatement 0x9451728 at 191 */
j < (m - 1); j++)
/* ROSE-HPCT propagated metrics WALLCLK:18.18[SgBasicBlock at 0x93f60b4] */
{
/* ROSE-HPCT raw data: Statement  WALLCLK:8.84@File jacobi.c  193-0 -> SgExprStatement 0x9451750 at 193 */
      resid = (((((ax * (((( *uold)[i - 1])[j]) + ((( *uold)[i + 1])[j]))) + (ay * (((( *uold)[i])[j - 1])
               + ((( *uold)[i])[j + 1])))) + (b * ((( *uold)[i])[j]))) - ((( *f)[i])[j])) / b);
/* ROSE-HPCT raw data: Statement  WALLCLK:8.43@File jacobi.c  196-0 -> SgExprStatement 0x9451778 at 196 */
      (( *u)[i])[j] = (((( *uold)[i])[j]) - (omega * resid));
/* ROSE-HPCT raw data: Statement  WALLCLK:0.91@File jacobi.c  197-0 -> SgExprStatement 0x94517a0 at 197 */
      error = (error + (resid * resid));
    }
\end{verbatim}
}
% eof


\documentclass{article}
\pagestyle{empty}
\usepackage{amsmath,amssymb,amsfonts}
\usepackage{graphicx}
\usepackage{multicol}
\setlength{\oddsidemargin}{0in} \setlength{\evensidemargin}{0in}
\setlength{\topmargin}{0in} \setlength{\textheight}{8.5in}
\setlength{\textwidth}{6.5in}

\makeatletter
\renewcommand*\env@matrix[1][*\c@MaxMatrixCols c]{%
	\hskip -\arraycolsep
	\let\@ifnextchar\new@ifnextchar
	\array{#1}}
\makeatother

\begin{document}
\begin{flushleft}
	\bfseries{MATH 260, Linear Algebra, Spring `14}\\
	\bfseries{Activity 11:  Diagonalization}\\
	\bfseries{Honor Code:} \hspace{3.5in}\bfseries{Names:}\\
\end{flushleft}
\begin{flushleft}
\vspace{.75in}
Directions:  Everyone should work on the assignment and should fill out their paper.  You are expected to make corrections based on what is presented on the board.  \\
If you need more explanations after class, you can read Section 5.4 of your textbook.\\
\vspace{0.1in}
\Large
In-Class Learning Goals:\\
\normalsize
\begin{enumerate}
\item Be able diagonalize a matrix
\item Be able to identify matrices that cannot be diagonalized
\item (Stretch/Homework) Be able to use the diagonalization of a matrix to find a matrix power or check similarity
%\item Be able to determine if a set of vectors is a basis for a vector space
%\item (stretch) Be able to find the dimension of a set or space (by the end of homework for sure!)

\end{enumerate}

\vspace{0.1in}

\section*{Warm-up:  Eigenvectors and Spaces of Repeated Roots}
%This is the matrix from Sect. 5.4, pg 339, #43

$\textbf{G}
=\begin{bmatrix}
1 & 0 & 0\\
-4 & 3 & 0 \\
-4 & 2 & 1
\end{bmatrix}
$
\hspace{0.5in}
The eigenvalues of \textbf{G} are $\lambda_{1,2}=1$ and $\lambda_3 = 3$.

\vspace{0.25in}
1. If I had asked you to find the eigenvalues of \textbf{G}, you would NOT need to calculate $det(\textbf{G}-\lambda \textbf{I})=0$. Explain why.

\vspace{1in}

2. Find the three eigenvectors associated with the given eigenvalues.

\vspace{3in}

3. What is the dimension of each corresponding eigenspace? What is the sum of the eigenspace dimensions for \textbf{G}?

\vspace{1in}

\section*{Diagonalized Equations}
Define two matrices: $\textbf{P} = \begin{bmatrix}
| & | & & |\\
\vec{v}_1 & \vec{v}_2 & \ldots &\vec{v}_n \\
| & | & & |
\end{bmatrix} $
\hspace{0.5in}
$\textbf{D} = \begin{bmatrix}
\lambda_1 & 0 & \ldots & 0 \\
0 & \lambda_2 & \ldots & 0 \\
\vdots & \vdots & \ddots & \vdots \\
0 & 0 & \ldots & \lambda_n
\end{bmatrix} $

\vspace{0.1in}

We are going to check some matrix equations using \textbf{P}, $\textbf{P}^{-1}$ and \textbf{D}.
For each of the following be sure to show some intermediate steps, since we already know what the final result should be!

\vspace{0.1in}

For \# 4-8 use the matrix: $\textbf{A} = \begin{bmatrix} 1 & 1 \\ 4 & 1 \end{bmatrix}$ \hspace{0.2in} with eigenvalues $\lambda_1 = 3$ and $\lambda_2 = -1$ with corresponding eigenvectors $\vec{v}_1 = \begin{bmatrix} 1 \\ 2 \end{bmatrix}$ and $\vec{v}_2 = \begin{bmatrix} 1 \\ -2 \end{bmatrix}$ respectively.

\vspace{0.2in}

4. Find $\textbf{AP}$ and $\textbf{PD}$. How do they compare?

\vspace{2.5in}

5. Find $\textbf{PDP}^{-1}$. What does it equal that we have already defined? \\
\textit{Hint: You already calculated $\textbf{PD}$ you just need to find $\textbf{P}^{-1}$, and do one multiplication}

\vspace{2.5in}

6. Find $\textbf{P}^{-1} \textbf{AP}$. What does it equal that we have already defined?\\
\textit{Hint: You only need to do ONE new multiplication here!}

\vspace{2in}

7. Write all three of the equations you have calculated above here, using the variables \textbf{A}, \textbf{P}, \textbf{D}, and $\textbf{P}^{-1}$. \\
\textit{This is a good place to check with your table, and professor to be sure everything came out right!}

\vspace{1in}

8. a) Define the matrix $\textbf{F}=\begin{bmatrix} \lambda_2 & 0 \\ 0 & \lambda_1 \end{bmatrix} = \begin{bmatrix} -1 & 0 \\ 0 & 3 \end{bmatrix}$, and use the same \textbf{P} as above. Find $\textbf{PFP}^{-1}$.

\vspace{2.5in}

8. b) Does it matter to the equations you wrote in \# 6 what order the eigenvalues and eigenvectors are in \textbf{P} and \textbf{D}? What relationship is required between \textbf{P} and \textbf{D}?

\vspace{1in}

\section*{Diagonalizing with Repeated Eigenvalues}
9. In the warm-up you found the eigenvectors for a matrix with a repeated eigenvalue, but which had 3 linearly independent eigenvectors. Pick your favorite equation from \# 6. Is the equation valid for \textbf{G}?

\vspace{2.5in}

%The matrix below is from Sect 5.4, pg 339, #29
10. The matrix $\begin{bmatrix} 3 & 1 \\ -1 & 5 \end{bmatrix}$ has a repeated eigenvalue of 4, with one eigenvector of $\vec{v}_1=\begin{bmatrix} 1 \\ 1 \end{bmatrix}$. \\
If we let $\textbf{P}=\begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}$ and $\textbf{D} = \begin{bmatrix} 4 & 0 \\ 0 & 4 \end{bmatrix}$. Does the equation $\textbf{A} = \textbf{PDP}^{-1}$ still hold? \\

\textit{ Hint: Think about \textbf{P}. What is $det(\textbf{P})$? What does that tell you about $\textbf{P}^{-1}$? }

\vspace{2in}

11. If the results from \# 9 and \# 10 are generalizable (they are), what must be true about a matrix for us to diagonalize it?

\vspace{1in}

%\section*{Applications of Diagonalization: Matrix Powers}
%We are again going to use the matrix: $\textbf{A} = \begin{bmatrix} 1 & 1 \\ 4 & 1 \end{bmatrix}$ \hspace{0.2in} with eigenvalues $\lambda_1 = 3$ and $\lambda_2 = -1$ with corresponding eigenvectors $\vec{v}_1 = \begin{bmatrix} 1 \\ 2 \end{bmatrix}$ and $\vec{v}_2 = \begin{bmatrix} 1 \\ -2 \end{bmatrix}$ respectively.
%
%\vspace{0.1in}
%
%12. Compute $\textbf{PD}^2 \textbf{P}^{-1}$ and $\textbf{A}^2 $. What do you notice?
%
%\vspace{2.5in}
%
%13. Compute $\textbf{PD}^3 \textbf{P}^{-1}$. Is it equal to $\textbf{A}^3 = \begin{bmatrix} 13 & 7 \\ 28 & 13 \end{bmatrix}$?
%
%\vspace{2in}
%
%14. Give a general equation to compute $\textbf{A}^k$.
%
%\vspace{1in}

%\subsection*{Similar Matrices}
%For this problem we'll use the matrices:\\
%$\textbf{X}=\begin{bmatrix} 4 & -2 \\ 1 & 1 \end{bmatrix}$
%\hspace{0.25in}
%$\textbf{Y}=\begin{bmatrix} -3 & 10 \\ -3 & 8 \end{bmatrix}$\\
%Their associated \textbf{P} matrices are: \\
%$\textbf{P}_X = \begin{bmatrix} 2 & 1 \\ 1 & 1 \end{bmatrix}$
%\hspace{0.25in}
%$\textbf{P}_Y = \begin{bmatrix} 5 & 2 \\ 3 & 1 \end{bmatrix}$
%
%\vspace{0.2in}
%
%15. Find $\textbf{P}_X \textbf{X} \textbf{P}^{-1}_X$ and $\textbf{P}_Y \textbf{Y} \textbf{P}^{-1}_Y$.
%
%\vspace{3in}
%
%16. Find the trace and determinant of \textbf{X} and \textbf{Y}.

\end{flushleft}
\end{document}

Spring 2014 - Schmitt/Activity Latex/Activity_11_diagonalization.tex


\documentclass[10pt,twocolumn,letterpaper]{article}

\usepackage{cvpr}
\usepackage{times}
\usepackage{epsfig}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{algorithm} % algorithm
\usepackage{algpseudocode} % algorithm
\usepackage{subfig} % sub-figure

% Include other packages here, before hyperref.

% If you comment hyperref and then uncomment it, you should delete
% egpaper.aux before re-running latex.  (Or just hit 'q' on the first latex
% run, let it finish, and you should be clear).
\usepackage[breaklinks=true,bookmarks=false]{hyperref}

\cvprfinalcopy % *** Uncomment this line for the final submission

\def\cvprPaperID{****} % *** Enter the CVPR Paper ID here
\def\httilde{\mbox{\tt\raisebox{-.5ex}{\symbol{126}}}}

% new commands
\newcommand{\given}{\,|\,}
\newcommand{\I}{\mathbb{I}}

% Pages are numbered in submission mode, and unnumbered in camera-ready
%\ifcvprfinal\pagestyle{empty}\fi
\setcounter{page}{1}
\begin{document}

%%%%%%%%% TITLE
\title{RANSAC With Likelihood Sampling}

\author{Xiangyu Kong\\
   University of Toronto\\
   1002109620 kongxi16
   \and
   Zun Cao\\
   University of Toronto\\
   1003871250 caozun
}

\maketitle
%\thispagestyle{empty}

%%%%%%%%% ABSTRACT
\begin{abstract}
   RANSAC (RAndom SAmple Consensus) has been a popular algorithm in computer vision for fitting a model to data points containing outliers \cite{RansacPaper}.
   Traditional RANSAC algorithm uses random sampling to choose data points, and use these data points to generate models \cite{RansacSlides}.
   Instead of sampling randomly, we explore the possibility of sampling based on the likelihoods of the data point to be inliers.
   A previously developed form of RANSAC, BAYSAC \cite{BaysacPaper}, is used in the experiment.
   Experiments were conducted on the noisy circle finding problem \cite{NoisyCircleFindingProblem} in order to demonstrate the differences and similarities between BAYSAC and RANSAC.
\end{abstract}

%%%%%%%%% BODY TEXT
\section{Introduction}

%-------------------------------------------------------------------------
\subsection{RANSAC}

RANSAC (RAndom SAmple Consensus) is mainly used in computer vision for fitting a model to data points containing outliers \cite{RansacPaper}.
In general, RANSAC algorithm follows the algorithm given in Algorithm~\ref{algo:ransac}:

\begin{algorithm}
   \caption{RANSAC($num\_iter$, $threshold$)}\label{algo:ransac}
   \begin{algorithmic}[1]
      \State $t = 0$, $\I = \{\}$
      \While{$t < num\_iter$ and $\I.size() < desired\_size$}
      \State Randomly Sample dataset $H_{i}$ of size ($n + 1$).
      \State Fit a polynomial $model$ of degree $n$.
      \If{$\forall j \in H_{i}$, $dist(model, j) < threshold$}
      \State $\I = \I \cup H_{i}$.
      \EndIf
      \State $t ++$
      \EndWhile
      \State\Return the fitted $model$ using all points in $\I$.
   \end{algorithmic}
\end{algorithm}

%-------------------------------------------------------------------------
\subsection{Problem Definition}

The problem of noisy circle finding problem is defined as follows. Given two input arrays of pixel locations where $X = [x_{1}, x_{2}, \dots, x_{n}]$, $Y = [y_{1}, y_{2}, \dots, y_{n}]$, $n \geq 3$, find a best-fit circle with center $(x, y)$, and its radius $r$ \cite{NoisyCircleFindingProblem}. An example of the problem is shown in Figure~\ref{fig:circle_finding}

\begin{figure}[!htp]
   \centering
   \includegraphics[width=0.6\linewidth]{./images/noisy_circle.png}
   \caption{Example of Circle Finding Problem}
   \label{fig:circle_finding}
\end{figure}

%------------------------------------------------------------------------
\section{Related Work}
\label{section:related_work}

%-------------------------------------------------------------------------
\subsection{Noisy Circle Finding Problem}

The noisy circle finding problem has been a classic exercise for computer vision students and an ongoing research topic for scholars. There are many approaches to solving this problem, including RANSAC, Patch Matching, Neural Network, etc.

For our experiment, we adopted and extended on an existing RANSAC circle detection project \cite{RansacGitHub}. The original project uses RANSAC with a twist. Instead of counting the number of inliers, the author chooses the model that has the minimum distance to all data points. We modified this part back to counting the number of inliers, and extended the class to implement the BAYSAC algorithm described in Section~\ref{section:baysac}.

%-------------------------------------------------------------------------
\subsection{BAYSAC}
\label{section:baysac}

The BAYSAC algorithm \cite{BaysacPaper} first assign all points' likelihoods to be inliers to $0.5$.
For each iteration, it updates the points' likelihoods using Equation~(\ref{eq:1}), where $P_{t}(i \in \I)$ indicates the probability of point $i$ being in the inliers set $\I$ at iteration $t$;
$P(H_{t} \subseteq \I)$ indicates the probability of the sample $H$ at iteration $t$ being a subset of the inlier set $\I$. I.e. for all points in the sample $j \in H_{t}$, $j \in \I$.

\begin{align} \label{eq:1}
   P_{t}(i \in \I)
    & =
   \begin{cases}
      \dfrac{P_{t - 1}(i \in \I) P(H_{t} \not\subseteq \I \given i \in \I)}{P(H_{t} \not\subseteq \I)}
       & i \in H_{t}
      \\
      P_{t - 1}(i \in \I)
       & i \not \in H_{t}
   \end{cases}
\end{align}

To calculate $P(H_{t} \not\subseteq \I \given i \in \I)$ and $P(H_{t} \not\subseteq \I)$, Equation~(\ref{eq:2}) and Equation~(\ref{eq:3}) are used.

\begin{align} \label{eq:2}
   P(H_{t} \not\subseteq \I)
    & = 1 - P(H_{t} \subseteq \I) \nonumber
   \\
    & = 1 - \prod^{}_{j \in H_{t}} P_{t - 1}(j \in \I)
\end{align}

\begin{align} \label{eq:3}
   P(H_{t} \not\subseteq \I \given i \in \I)
    & = 1 - P(H_{t} \subseteq \I \given i \in \I) \nonumber
   \\
    & = 1 - \prod^{}_{j \in H_{t}; j \neq i} P_{t - 1}(j \in \I) \nonumber
   \\
    & = 1 - \dfrac{P(H_{t} \subseteq \I)}{P_{t - 1}(i \in \I)}
\end{align}

Finally, combining Equations~(\ref{eq:2}, \ref{eq:3}) with Equation~(\ref{eq:1}), we derive Equation~(\ref{eq:4})

\begin{align} \label{eq:4}
   P_{t}(i \in \I)
    & =
   \begin{cases}
      \dfrac{P_{t - 1}(i \in \I) - P(H_{t} \subseteq \I)}{1 - P(H_{t} \subseteq \I)}
       & i \in H_{t}
      \\
      P_{t - 1}(i \in \I)
       & i \not \in H_{t}
   \end{cases}
\end{align}

%------------------------------------------------------------------------
\section{Experiment}

%-------------------------------------------------------------------------
\subsection{Algorithm}

Combining the two related work described in the Section~\ref{section:related_work}, the final BAYSAC algorithm can be written as Algorithm~\ref{algo:baysac}, where $dist(model, j)$ is the L2 distance defined by Equation~\ref{eq:5}.

\begin{align} \label{eq:5}
   dist(x_{c}, y_{c}, r_{c}, x_{j}, y_{j}) = | \sqrt{(x_{j} - x_{c})^{2} + (y_{j} - y_{c})^{2}} - r_{c} |
\end{align}

\begin{algorithm}
   \caption{BAYSAC($num\_iter$, $threshold$)}\label{algo:baysac}
   \begin{algorithmic}[1]
      \State $t = 0$, $\I = \{\}$
      \While{$t < num\_iter$ and $\I.size() < desired\_size$}
      \State Sample $H_{t}$ of ($n + 1$) points ordered by highest $P_{t - 1}(i \in \I)$
      \State Fit a polynomial $model$ of degree $n$.
      \If{$\forall j \in H_{t}$, $dist(model, j) < threshold$}
      \State $\I = \I \cup H_{t}$.
      \EndIf
      \State update all points' $P_{t}(i \in \I)$ base on Equation~\ref{eq:4}
      \State $t ++$
      \EndWhile
      \State\Return the fitted $model$ using all points in $\I$.
   \end{algorithmic}
\end{algorithm}

%-------------------------------------------------------------------------
\subsection{Experimental Design}

The parameters for the algorithms are the total number of datasets ($total\_num$), the circle to noise data point ratio ($ratio$), the circle points noise variance ($circle\_noise$) and the noisy points variance ($noisy\_noise$).

For each parameter setting, 10 iterations were run. For each iteration, two algorithms are run on the same set of generated data points. The analytic data are collected for each iteration and then averaged and plotted.

The analytic data collected after the experiment are percentage of inliers that were actual circle data (accuracy), runtime, the distance from the fitted model to the inliers, and the differences between the fitted circle's centers and radiuses using two algorithms.

%-------------------------------------------------------------------------
\subsection{Results}
\label{sec:results}

% ===== baseline =====
\textbf{Baseline Group}

The default parameters we set were $total\_num = 1000$, $ratio = 0.8$, $circle\_noise = 0.1$, $noisy\_noise = 1.0$. Due to the page constraint, only three pairs of results of the two algorithms are shown in Figure~\ref{fig:baseline_results}. See project repository \cite{ProjectRepo} for complete result. Observe that with the same dataset, RANSAC and BAYSAC produce very similar results with slight deviation.

% ===== total num =====
\textbf{Total Number of Data Points}

This part of the experiment tests the effect of the total number of data points. The number of data varies from 100 to 10100. For each epoch, the number of data is increased by 1000. The accuracy, runtime, distance to inliers and difference between two algorithms' fitted circles' centers and radiuses for each epoch is then plotted and shown in Figure~\ref{fig:total_num}.

% ===== ratio =====
\textbf{Circle to Noise Ratio}

This part of the experiment tests the circle to noise data point ratio. The ratio varies from 0.0 (only noise) to 1.0 (only circle). For each epoch, the ratio is increased by 0.1. The accuracy, runtime, distance to inliers and difference between two algorithms' fitted circles' centers and radiuses for each epoch is then plotted and shown in Figure~\ref{fig:ratio}.

% ===== num iter =====
\textbf{Number of Iterations}

This part of the experiment tests the number of iterations. The number of iterations varies from 2 to 40. For each epoch, the number of iterations is increased by 2. The accuracy, runtime, distance to inliers and difference between two algorithms' fitted circles' centers and radiuses for each epoch is then plotted and shown in Figure~\ref{fig:num_iter}.

% ===== circle noise =====
\textbf{Circle Noise}

This part of the experiment tests the circle noise variance. The variance of circle noise varies from 0.0 to 1.0. For each epoch, the variance is increased by 0.2. The accuracy, runtime, distance to inliers and difference between two algorithms' fitted circles' centers and radiuses for each epoch is then plotted and shown in Figure~\ref{fig:circle_noise}.

% ===== noisy noise =====
\textbf{Noisy Noise}

This part of the experiment tests the Noisy points' noise variance. The variance varies from 0.0 to 2.0. For each epoch, the variance is increased by 0.5. The accuracy, runtime, distance to inliers and difference between two algorithms' fitted circles' centers and radiuses for each epoch is then plotted and shown in Figure~\ref{fig:noisy_noise}.

% ===== baseline =====
\begin{figure}[!h]
   \centering
   \subfloat[Ransac-0]{\includegraphics[width=0.23\textwidth]{images/baseline/0_ransac.png}}
   \hfill
   \subfloat[BAYSAC-0]{\includegraphics[width=0.23\textwidth]{images/baseline/0_baysac.png}}
   \hfill
   \subfloat[Ransac-1]{\includegraphics[width=0.23\textwidth]{images/baseline/1_ransac.png}}
   \hfill
   \subfloat[BAYSAC-1]{\includegraphics[width=0.23\textwidth]{images/baseline/1_baysac.png}}

   \caption{Baseline Results}
   \label{fig:baseline_results}
\end{figure}

% ===== total num =====
\begin{figure}[!h]
   \centering
   \subfloat[Accuracy]{\includegraphics[width=0.23\textwidth]{images/total_num/accuracy.png}}
   \hfill
   \subfloat[Runtime]{\includegraphics[width=0.23\textwidth]{images/total_num/runtime.png}}
   \\
   \subfloat[Inlier Distance]{\includegraphics[width=0.23\textwidth]{images/total_num/inlier_dist.png}}
   \hfill
   \subfloat[Center and Radius Difference]{\includegraphics[width=0.23\textwidth]{images/total_num/center_radius_diff.png}}

   \caption{Total Number of Data}
   \label{fig:total_num}
\end{figure}

% ===== ratio =====
\begin{figure}[!h]
   \centering
   \subfloat[Accuracy]{\includegraphics[width=0.23\textwidth]{images/ratio/accuracy.png}}
   \hfill
   \subfloat[Runtime]{\includegraphics[width=0.23\textwidth]{images/ratio/runtime.png}}
   \\
   \subfloat[Inlier Distance]{\includegraphics[width=0.23\textwidth]{images/ratio/inlier_dist.png}}
   \hfill
   \subfloat[Center and Radius Difference]{\includegraphics[width=0.23\textwidth]{images/ratio/center_radius_diff.png}}

   \caption{Circle to Noise Ratio}
   \label{fig:ratio}
\end{figure}

% ===== num iter =====
\begin{figure}[!h]
   \centering
   \subfloat[Accuracy]{\includegraphics[width=0.23\textwidth]{images/num_iter/accuracy.png}}
   \hfill
   \subfloat[Runtime]{\includegraphics[width=0.23\textwidth]{images/num_iter/runtime.png}}
   \\
   \subfloat[Inlier Distance]{\includegraphics[width=0.23\textwidth]{images/num_iter/inlier_dist.png}}
   \hfill
   \subfloat[Center and Radius Difference]{\includegraphics[width=0.23\textwidth]{images/num_iter/center_radius_diff.png}}

   \caption{Number of Iterations}
   \label{fig:num_iter}
\end{figure}

% ===== circle noise =====
\begin{figure}[!h]
   \centering
   \subfloat[Accuracy]{\includegraphics[width=0.23\textwidth]{images/circle_noise/accuracy.png}}
   \hfill
   \subfloat[Runtime]{\includegraphics[width=0.23\textwidth]{images/circle_noise/runtime.png}}
   \\
   \subfloat[Inlier Distance]{\includegraphics[width=0.23\textwidth]{images/circle_noise/inlier_dist.png}}
   \hfill
   \subfloat[Center and Radius Difference]{\includegraphics[width=0.23\textwidth]{images/circle_noise/center_radius_diff.png}}

   \caption{Circle Noise}
   \label{fig:circle_noise}
\end{figure}

% ===== noisy noise =====
\begin{figure}[!h]
   \centering
   \subfloat[Accuracy]{\includegraphics[width=0.23\textwidth]{images/noisy_noise/accuracy.png}\label{fig:noisy_noise_accuracy}}
   \hfill
   \subfloat[Runtime]{\includegraphics[width=0.23\textwidth]{images/noisy_noise/runtime.png}}
   \\
   \subfloat[Inlier Distance]{\includegraphics[width=0.23\textwidth]{images/noisy_noise/inlier_dist.png}}
   \hfill
   \subfloat[Center and Radius Difference]{\includegraphics[width=0.23\textwidth]{images/noisy_noise/center_radius_diff.png}}

   \caption{Noisy Noise}
   \label{fig:noisy_noise}
\end{figure}

%------------------------------------------------------------------------
\section{Conclusions}

Base on the results shown in Section~\ref{sec:results}, we can see that under the problem of noisy circle detection, RANSAC and BAYSAC performs almost equally well.
The accuracies and runtimes for two the two algorithms are very similar. Note that for Figure~\ref{fig:noisy_noise_accuracy}, the y-axis scale is enlarged because the accuracies for RANSAC and BAYSAC are very similar.

By looking at the Center and Radius Differences, the results of RANSAC and BAYSAC converges as the number of iterations increases, circle to noise ratio increases (there are more circle points than random noise given the same number of total data points) and circle noise decreases.

In the original BAYSAC paper \cite{BaysacPaper}, the author claimed that BAYSAC could potentially help reduce the failure rate by up to $78\%$ compared to vanilla RANSAC. However, we were not able to show this through our experiment. One potential reason is that the noisy circle detection problem is at a very low dimension, and circle is a very low degree polynomial ($n = 2$). In the future, one possible improvement to this experiment is to compare RANSAC and BAYSAC results for solving a higher dimension problem.

%------------------------------------------------------------------------
\pagebreak
{\small
   \bibliographystyle{ieee_fullname}
   \bibliography{reportbib}
}

\end{document}


%%%%%%%%%%%%%%
% To compile the latex source you the following packages
\documentclass[11pt,a4paper]{article}
\usepackage{color}
\usepackage{listings}
\usepackage{graphicx}
\usepackage{url}
\usepackage[ddmmyyyy]{datetime}
\renewcommand{\dateseparator}{-}

\definecolor{bbb}{rgb}{.9,.9,.9}
\lstset{stepnumber=2, basicstyle=\small \ttfamily, language=matlab,
  frame=single, framerule=0.pt}

\usepackage[colorlinks=true, pdfstartview=FitV, linkcolor=blue,
citecolor=blue, urlcolor=blue]{hyperref}

\newcommand\kms{$\mathrm{km} \, \mathrm{s}^{-1}$}
\newcommand\kmsmath{\mathrm{km} \, \mathrm{s}^{-1}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%% Page dimensions %%%%%
%%%%%%  DO NOT CHANGE  %%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%\textheight=247mm
\textwidth=163mm
\topmargin=-7mm
\oddsidemargin=-10mm
\evensidemargin=-10mm
%\parindent 10pt

\setlength{\textwidth}{16.3cm}
\setlength{\oddsidemargin}{0cm}
\setlength{\evensidemargin}{0cm}
\addtolength{\textheight}{20mm}
\addtolength{\voffset}{-5mm}
\usepackage{sectsty}
\usepackage[small,bf,hang]{caption}

\allsectionsfont{\sffamily}
\chapterfont{\sffamily}
\chaptertitlefont{\sffamily}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%% Start of document %%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{document}
\pagestyle{plain}
\pagenumbering{arabic}
\title{\textsf{Analysing data from SALSA using \textsc{SalsaJ}}}
\author{\textsf{Eskil Varenius}}
\yyyymmdddate
\date{\textsf{Revised: \today \, \currenttime}}

\maketitle

\section*{Abstract}
This document describes how to analyse FITS files generated by the SALSA Onsala
radio telescopes by using the software \textsc{SalsaJ}.  We describe step by
step how to open the FITS files, improve the analysis by removing receiver
artifacts (spectral baseline subtraction), and fit Gaussian intensity
distributions to derive velocities and intensities for the observed components
(peaks).

\tableofcontents

\section{Introduction}
\label{sec:introduction}

The SALSA Onsala telescope is a 2.3\,m in diameter antenna located at
Onsala space observatory outside Gothenburg. The telescopes are
designed to detect the faint radiation from cold hydrogen gas in our
galaxy, the Milky way. The hydrogen gas emits radiation at a
wavelength of 21 cm (or a frequency of 1420.4 MHz). This signal can be
detected by the microwave receiver connected to the telescope.
Having completed the observations, the observer may want to process
the data further in order to more accurately deduce the kinematics,
as well as the amount of hydrogen gas in the Milky Way. From the webpage,
you can download your data in FITS-format. There
are currently two main ways to open FITS-files from SALSA:

\begin{itemize}
\item \textbf{SalsaJ} was developed within the EU-HOU project.
	SalsaJ can be used to analyse spectral data from the SALSA telscopes, but
	can also be used as a simple image editor and processor.  The main
	advantage of SalsaJ is its easy-to-use point-and-click interface. The main
	disadvantage is that reducing many spectra can be tedious.
\item \textbf{SalsaSpectrum} is a data reduction environment written
  in the popular mathematical software \textsc{\textsc{Matlab}} -
  aimed for reduction of data from SALSA Onsala, developed by Daniel
  Dahlin. The main advantage is that many spectra can be processed
  quickly. The main disadvantage is that it requires the user to have
  a working installation of Matlab, which is non-free software. Please note
  that the observatory cannot provide Matlab licenses, but many university
  students have free access to Matlab.
\end{itemize}
Both SalsaJ and SalsaSpectrum can be downloaded from the webpage at
{\url{http://vale.oso.chalmers.se/salsa}}.

\section{Analysing SALSA data with SalsaJ}
\label{sec:salsaj}
In this section we describe step by step how to analyse the FITS files
generated by the SALSA Onsala telescopes in the software SalsaJ. We will
open the FITS files, switch to velocity, and inspect the data using the
pointer. Further analysis, such as removing receiver artifacts
(spectral baseline subtraction), and fit Gaussian intensity distributions,
is described in Sect. \ref{sect:adv}.
\textbf{IMPORTANT:} SalsaJ can only read correctly spectra produced by SALSA after
2015-06-23.  If you have older spectra, you cannot use SalsaJ for processing
them.

\subsection{Starting SalsaJ}
Once you have downloaded the file SalsaJ2.jar from the website,
open it by double clicking. If this does not work for you,
try right-clicking on the file SalsaJ2.jar and chose \emph{Open with}
and find \emph{Java}. Once the software open you should see a window looking
like Fig. \ref{fig:salsajstart}.
\begin{figure}[h!]
  \centering
  \scalebox{0.55}{\includegraphics{../figures/start}}
  \caption{The startup window of the software SalsaJ.}
  \label{fig:salsajstart}
\end{figure}

\subsection{Opening a spectrum}
To open a spectrum, i.e. a FITS file, click the \emph{File} menu, and then
\emph{Radio spectrum}. Navigate to the file you want to analyse and click OK.
You should now see the spectrum presented as a figure similar to Fig.
\ref{fig:opened}.
\begin{figure}[h!]
  \centering
  \scalebox{0.55}{\includegraphics{../figures/spectrumopened}}
  \caption{A spectrum has been opened and is by default presented with channel
  index on the horisontal axis. We want to change to velocity.}
  \label{fig:opened}
\end{figure}
Note: If you in Mac OS X do not see the \emph{Radio spectrum} option in the
\emph{File} menu, make sure you focus the SalsaJ window by clicking on it.

\subsection{Set the scale}
By default, SalsaJ presents spectra as Intensity vs channel number. For our
purposes it is more useful to switch the horisontal axis to velocity. This is
done by clicking the button labelled \emph{Set scale}, choose \emph{Velocity},
and click OK. The horisontal axis should now be set to km/s, as in Fig.
\ref{fig:velocity}. Note that you can inspect the measured values by moving the
pointer over the graph: the intensity and velocity are reported in the lower part
of the window. If you are only interested in rough estimates of velocities,
you can note down the values you find by visual inspection and consider yourself finished
with this spectrum. However, if you want to get the best possible results you may
want to consider the baseline subtraction and gaussian fitting described in Sect.
\ref{sect:adv} below.
\begin{figure}[h!]
  \centering
  \scalebox{0.55}{\includegraphics{../figures/spectrumvel}}
  \caption{The spectrum is now shown with velocity on the horisontal axis. You can now
  inspect the spectrum directly by hovering with the pointer over the peaks. The current value
  marked by the cross is displayed in the bottom part of the window.}
  \label{fig:velocity}
\end{figure}

\section{Advanced processing}
\label{sect:adv}
You can get pretty good estimates of the observed velocities by moving the
pointer over the graph: the intensity and velocity are reported in the lower
part of the window.  However, if you want to get the best possible results you
may want to consider the baseline subtraction and gaussian fitting as described
below.  Note that this can be tedious work if you have many spectra, in this
case consider using the Matlab program SalsaSpectrum if you have access to
Matlab.

\subsection{Spectral baseline subtraction}
The spectra presented in Fig. \ref{fig:velocity} is pretty good already from
the start, but looking carefully we can see that the zero-level is not flat.
Instead, it is curved due to residual effects from the receiver which were
not subtracted during the observation.
For accurate results, we need to
remove this effect. This removal is called \emph{baseline subtraction} and
is started by clicking the button \emph{Baseline} in the SalsaJ window.
From quickly inspecting the graph
with the pointer, we note that we have no line emission in the velocity regions
[-170 to -130] and [50 to 240] km/s.
We enter the boundaries of the two regions and chose poynomial order 2,
see Fig. \ref{fig:baseline}.

\begin{figure}[h!]
  \centering
  \scalebox{0.55}{\includegraphics{../figures/baseline}}
  \caption{The input window for baseline subtraction, where we have defined
  two regions of the spectra with no line emission and chosen a degree of
  polynomial to fit to the data.}
  \label{fig:baseline}
\end{figure}

After you press OK, a baseline will be fitted and plotted, see Fig.
\ref{fig:baselinefit}.
\begin{figure}[h!]
  \centering
  \scalebox{0.55}{\includegraphics{../figures/baselinefitted}}
  \caption{A baseline has been fitted based on the input in Fig.
  \ref{fig:baseline}. The fitted baseline is shown in red, and the
  zero-level in blue.}
  \label{fig:baselinefit}
\end{figure}
If you are happy with the fit, click \emph{Subtract baseline}. If the fit looks
bad, click \emph{reset} and redo the subtraction with different intervals or
polynomial orders until you get a good fit to the line-free parts of the spectrum.
After subtracting a baseline, the line-free parts should be flat, as in
Fig. \ref{fig:baselinesubtracted}.
\begin{figure}[h!]
  \centering
  \scalebox{0.55}{\includegraphics{../figures/baselinesubtracted}}
  \caption{The baseline fitted in Fig. \ref{fig:baselinesubtracted} has been
	  subtracted and the line-free regions are flat with noise around zero. We are now
	  ready to measure velocities and intensities.}
  \label{fig:baselinesubtracted}
\end{figure}

\subsection{Measuring velocities: fitting Gaussians}
Because of noise it can be hard to measure exactly the center velocity
of peaks by simple manual inspection. A good way is to fit Gaussian
intensity distributions to the peak, which can be done by clicking the
button \emph{Gaussian fit} in SalsaJ. For the fitting we first need to
note down starting guesses for the peaks we want to fit. Here visual
inspection suggest four peaks roughly at positions -90, -52, -35, and 0 km/s.
Hence we want to fit four Gaussians to the spectrum, but SalsaJ can only
handle one at the time (the SalsaSpectrum Matlab class can fit multiple Gaussians
at once). Let us start with the brightest peak around 0 km/s.

After clicking the button\emph{Gaussian fit} we need to give an interval of data to use
for fitting this peak, specified by Xmin and Xmax. Usually a range of 10km/s above and below the
peak center works fine, in this case -10 to +10. Then we need to give initial guesses for the fitting
to start. This does not have to be very accurate, but the closer to the real value the better.
We guess an amplitude of 30 (which is far below the real one), a center at 0 as noted for the
peak positions in the previous paragraph, and a width of 20. Then we click OK. The fitted Gaussian
is shown in red, see Fig. \ref{fig:fittedgaussian}.

NOTE: If your fit goes wrong, for example
if you specify an interval of the data where there is no peak, then you can remove the previous fitted
gaussian by checking the box \emph{erase the last}.

We proceed in a similar way to fit the other three peaks, the result can be seen in Fig.
\ref{fig:allfitted}. After fitting you will also get a list of fitted values, see Fig.
\ref{fig:fitres}. From this list you can read the galactic coordinates of the spectrum,
and the velocities and intensities of the all fitted peaks. The velocities can be used
to make a rotation curve or map of the Milky Way as described in the SALSA Milky Way project
instructions.

\begin{figure}[h!]
  \centering
  \scalebox{0.55}{\includegraphics{../figures/fitpars}}
  \caption{A set of parameters to fit a Gaussian intensity distribution to
	  one of the peaks in the spectrum. The resulting fit can be seen in Fig.
  \ref{fig:fittedgaussian}.} \label{fig:fitpars}
\end{figure}

\begin{figure}[h!]
  \centering
  \scalebox{0.55}{\includegraphics{../figures/fittedgauss}}
  \caption{A gaussian has been fitted to the data using the parameters
  specified in Fig. \ref{fig:fitpars}.} \label{fig:fittedgaussian}
\end{figure}

\begin{figure}[h!]
  \centering
  \scalebox{0.55}{\includegraphics{../figures/allfitted}}
  \caption{All four peaks have been fitted. The fitted velocities and intensities can be
	  found in the \emph{Gaussian fit results}, see Fig. \ref{fig:fitres}.} \label{fig:allfitted}
\end{figure}

\begin{figure}[h!]
  \centering
  \scalebox{0.55}{\includegraphics{../figures/fitres}}
  \caption{All four peaks have been fitted. In this window you can read the fitted velocities and
	  intensities. These values can be used to make a rotation curve or map of the Milky Way, as
  described in the SALSA project instructions document.} \label{fig:fitres}
\end{figure}

\end{document}