%!TEX root = ../OGtextbook.tex
% TODO: Place this chapter as an appendix chapter after Git chapter

\LaTeX is a document preparation system that produces high-quality typesetting and is ideal for technical, scientific, and computational subjects. The mathematical equation engine in \LaTeX is the standard in technical and mathematical typesetting. Further, the broader philosophy of \LaTeX documents is to provide document-type and reference-type formatting functions separate from the content of the document. This allows the \LaTeX user to focus on content, and let separate commands determine the formatting. This can also allow for particular content to be quickly reformatted to another style.

An encylopedic reference for most \LaTeX functionality is \citet{MittelbachGoossens:2004}. But it valuable to search the resources available online because they often describe the typesetting innovations that come from writing your own source \LaTeX code.

Programs like \href{https://www.mackichan.com/index.html?products/swp.html~mainFrame}{Scientific WorkPlace} provide a GUI interface for \LaTeX. But this program is expensive, and it is not as flexible as writing your own \LaTeX document code. This book recommends writing your own \LaTeX document source code. Very soon, you will begin to think in terms of the the compiled \LaTeX document even though you are typing source code.

\section{Installing \LaTeX}\label{SecLaTeXInstall}

  A great summary page for installing \LaTeX on Mac OS X, Windows, or Linux is available from \href{https://www.latex-project.org/get/}{this link} (https://www.latex-project.org/get/). Following these instructions will install on your local machine a number of libraries, packages, and tools to help you use \LaTeX for your typesetting.

\section{Running \LaTeX}\label{SecLaTeXRun}

  The traditional use and workflow of creating a document in \LaTeX is to write a plain text source document (.tex subscript) in any text editor and then have the \LaTeX engine compile that document into a format you want to use (typically a PDF). You can use any text editor for creating \LaTeX source documents (.tex files), but there are advantages to using text editors that ``work well'' with \LaTeX.

  Most distributions of \LaTeX that you would download from the link in Section \ref{SecLaTeXInstall} will come with a text editor, such as TeXShop for Mac, WinEdt for Windows, and TeXMaker for Linux. The \href{http://www.vim.org/}{Vim} text editor is a sign of a true programmer. Vim is most effective for the coder that prefers to keep his hands on the keyboard and avoid the mouse. Vim has a \LaTeX plug in that allows the user to compile .tex files to PDF directly. The \href{https://www.sublimetext.com/3}{Sublime Text 3} is a more versatile text editor that has a nice LaTeXTools package that allows you to compile PDFs from .tex files directly from the editor. On Mac OS X, Sublime Text also requires the Skim PDF viewer. Sublime Text and Vim also have various color and syntax highlighting options for .tex and other \LaTeX source files. Figure \ref{FigSubTxtSkimPDF} shows a screen shot of a .tex file in Sublime Text 3 and its compiled PDF version.

  \begin{figure}[htbp]\centering\captionsetup{width=6.0in}
    \caption{\textbf{Screenshot Sublime Text 3 .tex document and compiled PDF in Skim}}\label{FigSubTxtSkimPDF}
    \fbox{\resizebox{6.0in}{3.3in}{\includegraphics{./images/SubTxtSkimPDF.png}}}
  \end{figure}

\section{Learning \LaTeX}\label{SecLaTeXLearning}

  Many tutorials are available online for learning \LaTeX. However, most of those tutorials focus on functionality specific to a particular field. The best way to start using \LaTeX is to get a template of a source .tex document and then use online resources to learn more specific functionality. Figure \ref{FigLaTeXtemplate} is a snapshot of a template.tex file.

  \begin{figure}[htbp]\centering\captionsetup{width=4.0in}
    \caption{\textbf{Screenshot of \LaTeX template .tex file}}\label{FigLaTeXtemplate}
    \fbox{\resizebox{4.0in}{5.0in}{\includegraphics{./images/LaTeXtemplate.png}}}
  \end{figure}

  \href{http://stackoverflow.com/}{Stackoverflow.com} is a great resource for answers to particular \LaTeX questions. Anyone can publicly browse the questions and answers stored at Stackoverflow. If you register for an account, you will be able to ask questions through Stackoverflow and even give answers. In most cases, you will find that someone else has already asked a question similar to yours and that a series of answers have been given.

\section{\LaTeX\space Cheat Sheet}\label{SecLaTeXCheat}

  This section provides a list of some of the key \LaTeX commands and structures that you will likely use.

  \subsection{Math symbols and Greek Letters}

    A good list of the \LaTeX commands for math symbols and Greek letters is available at \href{http://web.ift.uib.no/Teori/KURS/WRK/TeX/symALL.html}{this link}.\footnote{See \href{http://web.ift.uib.no/Teori/KURS/WRK/TeX/symALL.html}{http://web.ift.uib.no/Teori/KURS/WRK/TeX/symALL.html}.} Math symbols and Greek letters are included in the text by using dollar signs to bracket the commands that are to be rendered in math mode. Typing the text, ``The Greek letter \verb|$\theta$| represents \verb|$x + y$|'' will be rendered as ``The Greek letter $\theta$ represents $x + y$''.

    \textbf{Equations.} The \texttt{amsmath} package (American Mathematical Society) allows for a number of higher math display options that you will use often. The standard construct for equations is the \verb|\begin{equation}| environment. The following text will render the following equation.

    \begin{lstlisting}[frame=single]
      \usepackage{amsmath}
      \begin{equation}
        u\left(c_{s,t}\right) = \frac{(c_{s,t})^{1-\gamma} - 1}{1 - \gamma}
      \end{equation}
    \end{lstlisting}
    \begin{equation}\tag{1}
      u\left(c_{s,t}\right) = \frac{(c_{s,t})^{1-\gamma} - 1}{1 - \gamma}
    \end{equation}

    \noindent Table \ref{TabLaTeXAMStypes} shows various display environment options for the \texttt{amsmath} package.

    \begin{table}[htbp] \centering \captionsetup{width=6.0in}
    \caption{\label{TabLaTeXAMStypes}\textbf{Display environment options in the \texttt{amsmath} package}}
      \begin{threeparttable}
      \begin{tabular}{>{\small}l >{\small}l |>{\small}l}
        \hline\hline
        \multicolumn{1}{c}{\textbf{\small{Numbered}}} & \multicolumn{1}{c}{\textbf{\small{Not numbered}}} & \multicolumn{1}{c}{\textbf{\small{Description}}} \\
        \hline
        \texttt{equation} & \texttt{equation*} & One line, one equation \\
        \texttt{multline} & \texttt{multline*} & One unaligned multiple-line equation, one equation number \\
        \texttt{gather} & \texttt{gather*} & Several equations without alignment \\
        \texttt{align} & \texttt{align*} & Several equations with multiple alignments \\
        \texttt{flalign} & \texttt{flalign*} & Several equations: horizontally spread form of \texttt{align} \\
        \texttt{split} &  & A simple alignment within a multiple-line equation \\
        \texttt{gathered} &  & A ``mini-page'' with unaligned equations \\
        \texttt{aligned} &  & A ``mini-page'' with multiple alignments \\
        \hline\hline
      \end{tabular}
      \begin{tablenotes}
        \scriptsize{\item[*]See Table 8.1 in \citet{MittelbachGoossens:2004}.}
      \end{tablenotes}
      \end{threeparttable}
    \end{table}

  \subsection{Brackets and parentheses in equations}

    TODO: Add content.

  \subsection{Matrices in equations}

    TODO: Add content.

  \subsection{Tables}

    TODO: Add content.

  \subsection{Figures}

    A good package and set up for producing figures is the \texttt{graphicx} package. These figures are called ``float'' objects because \LaTeX intelligently chooses their placement in the document based on natural breaks in the text. The code that produced Figure \ref{FigSubTxtSkimPDF} is the following.

    \begin{lstlisting}[frame=single]
      \usepackage{graphicx}
      \usepackage[format=hang,font=normalsize,labelfont=bf]{caption}

      \begin{figure}[htbp]\centering\captionsetup{width=4.0in}
        \caption{\textbf{Screenshot of \LaTeX template .tex file}}\label{FigLaTeXtemplate}
        \fbox{\resizebox{4.0in}{5.0in}{\includegraphics{./images/LaTeXtemplate.png}}}
      \end{figure}
    \end{lstlisting}

  \subsection{BibTeX and references}

    TODO: Add content.

\section{Debugging}\label{SecLaTeXDebug}

  At some point, you will have created a \LaTeX source .tex file and hit compile and you will receive an error. The PDF will not compile. \LaTeX produces an error log file in this case. The first time you read an error log, it may look incomprehensible. But as you experience more of these error logs, you find where the key information is located to help you solve your problem most efficiently.

  The error log often displays at the bottom of your text editor. In Sublime Text, the error log is displayed in the window at the bottom of the screen, pictured in Figure \ref{FigSubTxtSkimPDF} at the bottom of the left-hand-side. Key information is the line number of the source code where the error occured. And the error log will also usually give you some kind of helpful description as to what the problem was.

  In many cases, the error will be an incomplete closure, such as  \$\:\$, (), [], \{\}, \verb|\begin{}\end{}|. Other common errors include misspelled function names and missing source packages in the preamble.

  One last comment with debugging \LaTeX code is that \href{http://stackoverflow.com/}{Stackoverflow.com} is your friend. As was mentioned in Section \ref{SecLaTeXRun}, Stackoverflow is a great resource for answers to particular \LaTeX questions. Anyone can publicly browse the questions and answers stored at Stackoverflow. This book recommends that you register for an account. This will allow you to ask questions through Stackoverflow and even give answers. In most cases, you will find that someone else has already asked a question similar to yours and that a series of answers have been given.


\chapter{Design and Implementation}\label{C:work}

A significant hurdle for the project was gaining enough understanding of the existing work on RBMs to be able to implement the ORBMs algorithm and architecture. This was crucial as an incorrect implementation invalidates Contributions \ref{item-c2} and \ref{item-c3} of this project.

\section{Implementation Design}

\subsection{Language Choice}

The implementation of the ORBM and inference algorithm is in \emph{Python}, with \emph{Matlab} and \emph{Java} being the other languages originally considered.

I have spent my university career working in Java, and nowdays it is efficient to perform machine learning tasks. Python's module and class system promotes composition, and multiple inherietance in particular allows for composition of different classes. Matlab is a popular numerical computing environment and has a lot of online resources as well as machine learing papers~\cite{hinton2006reducing} that include snippets or full Matlab programs that were used in the paper. Also Matlab is built around strong, efficient support for matrix operations which are very prolific in the RBM and ORBM implementations and evaluations.

Python has a very succinct syntax and shallow learning curve, as well as being the language of choice of my supervisor. This is favourable as I can have langauge level support for translating the algorithm into python correctly. Through libraries that supply wrappers for C-bindings, efficient code can be written in Python. Overall, being an interpreted language Python is slower than Matlab and Java. Three main factors outweighed this and resulted in the choice of Python:

\begin{description}
\item[Up front learning] Given the amount of up front learning of concepts required to implement the solution, Python was favoured over Java or Matlab because is has a very shallow learning curve. Despite having experience in Java as well as Python, I had never used machine learning or linear algebra libraries in either languages. Given the amount of up front learning required to understand the concepts in this project let alone implement it, I opted for the shallower learning curve of Python. Also my supervisor has experience using Python and the libraries involved with this project. A shallower learning curve meant that more time could be allocated to the evaluation which is the foremost contribution of this project.
\item[Library Support] Matlab and Python were two contenders in this factor, as a lot of matlab machine learning code is avaliable with supporting papers. Python has libraries such as NumPy, SciPy and Matplotlib~\cite{2015HistSciPy} which mirror Matlab functionality. However, NumPy and SciPy are all open source, while there is argument for treating an API as a black box --- i.e. not writing code that is dependant on the underlying implementation, having the option to view source helped me better grasp the concepts. In particular being able to compare my implementation of the RBM to  the Sklearn implementation~\cite{scikit-learn} allowed for some performance improvements to my implementation.
\item[Ease of Evaluation] Python provides a strong plotting library that is inspired by Matlab and R --- MatplotLib. Matplotlib is interoperable with the NumPy library, meaning that visualising results was really easy.

Also the evaluation requires repeatedly running a test to gain more confidence in the stochastic result. This is made possible by the ECS-Grid. Python only required a simple script that manages input and output. However the Java support on the ECS grid requires significantly more setup. This seemed like an unnecessary risk to introduce to the project at the later stage when the evaluation would being carried out.
\end{description}
%
% All of the languages considered have the efficiency for machine learning tasks, despite Python being on the slower end of the spectrum of languages, versus that of the compiled JVM languages like Matlab or Java, it is still fast enough. The complexity of the traditional RBM is relative to what is trying to be modelled, less of the language implementing it.
% Java and Python have robust testing suites given they are used for non-academic goals. Matlab and R do have suites however the author has not had experience with them. Choosing from one of these two languages would increase the amount of upfront learning required and therefore increase this risk.
% The author had the most experience with Python and Java, however Python was favoured due to it's Matlab-like library NumPy. Expressive matrix syntax combined with the familiarity and brevity of Python made it a compelling option.

\subsection{Program Architecture}

The implementation of the RBM, ORBM and algorithm is implemented with unit testing and composability in mind. It is important that the design supports comparing the Full and Approximated Correction (Equations~\cite{eq:Full-Corretion} and ~\cite{eq:Approx-Correction}. By using Python, multiple inheritance could be leveraged to achieve this composability. For instance adding support for continuous pixel values to the \texttt{Full Correction Sampler}, one simply needed to extend \texttt{Continuous Approximate Correction Sampler} and the \texttt{Full Correction Sampler}, with no code actually in the new class. The architecture is pictured in the class diagram \ref{F:Prog-Arch}. There were three main roles these classes filled:
\begin{enumerate}
  \item The \texttt{Trainer} was used to train the weights of a supplied \texttt{RBM}. It would do so using a supplied \texttt{sampler}, decoupling how samples were generated from the training process.
  \item The \texttt{RBM} was the model of an RBM, storing the weight matrix, as well as parameter information. Also this supported conversion of the SciPy Sklearn libraries' RBM implementation into the implementation used in this project. Decoupling the concept of an \texttt{RBM} from the \texttt{Sampler} and the \texttt{Trainer} meant that RBMs could be `plugged' into the ORBM architecture with ease.
  \item The \texttt{Sampler} defined how to perform Gibbs sampling, the subclasses defining whether it is standard RBM sampling or ORBM sampling. This made it trivial to compose \texttt{samplers}, which was required for the ORBM samplers.
\end{enumerate}

\begin{figure}[h]
\begin{center}
  \includegraphics[width = 1\textwidth]{Assets/ENGR489-Architecture.png}
\caption{A figure showing the architecture for the implemented test suite in UML notation.}
\label{F:Prog-Arch}
\end{center}
\end{figure}

\subsection{Testing Approach}

The inference algorithm is the most crutial part of the implementation to test. In larger dimensional tasks, unit testing to ensure the algorithm produces the correct values becomes difficult. In smaller cases I was able to ensure the probabilities of $P(h|v)$ and $P(v|h)$ calculated by the implementation matched calculations I made by hand. Being an unsupervised black boxes, determining if an RBM (and by extension the ORBM) hidden representation is `correct` for larger tasks such as the MNIST handwritten digit dataset is non-trivial. Concerned by this risk, I designed my evaluations to build confidence in the algorithm and the implementation.

\section{Evaluation Design}
\subsection{Examining the mixing time of inference in the ORBM}

As described in Section \ref{S:ORBM-Inference}, sampling from $h^A$ and $h^B$ suffers from the effect of explaining away (Section \ref{S:Explaining-Away}) which requires a Gibbs chain to be run. While intractable for a large SBN, as only two causes are being modeled, ideally the Gibbs chain for this process won't take many iterations to mix. As the correction needs to be computed for each Gibbs iteration, a long Gibbs chain could be detrimental to performance.
Finding this \emph{optimal mixing time} for the ORBM is only an issue as the number of dimensions and training examples increase. In smaller dimensions mixing can be overcome by using a large ($>1000$) number of Gibbs iterations, as it computational feasible to do so. However in the larger problems this becomes intractable, so an optimal mixing time is desired.

This project examines the mixing time by way of examining reconstructions at various points of the Gibbs chain. This is also interesting in that we can observe the dynamics of the network and the effect of applying the \emph{correction} has on reconstructions. Hinton~\cite{Hinton-Animations} has had success observing reconstructions in an animated setting.

\subsection{Evaluating RBMs and ORBMs}

  A challenge faced by this project and work with RBMs is that they are non-trivial to evaluate. Being an unsupervised black box, one cannot merely inspect the hidden units sampled after a given input and be sure that a good model has been learnt. As I plug RBMs into the ORBM architecture it is important for the overall results that the RBMs are well trained.

\subsection{Hinton diagrams}
  We can examine the weights into a given hidden unit in the shape of the visible vector. For instance in the context of images, \texttt{Hinton Diagrams} allow visualisation of what a given hidden unit is `doing' by visualising the matrix of weights into that unit. This was first used by Hinton in the context of Boltzmann Machines in~\cite{Hinton:1986:LRB:104279.104291}. They also give insight into hidden unit utilisation. Weights are often initialised to small random values resulting in a Hinton diagram with little structure to it. This is illustrated in figures \ref{F:Hinton-Good} and \ref{F:Hinton-Bad}. The former showing a hidden units weights where some structure has been learnt, and the latter showing the opposite.

  \begin{figure}[htb]
  \centering
  \begin{subfigure}[t]{0.3\textwidth}
      \includegraphics[width=\textwidth]{Assets/HINTON1.png}
      \caption{Utilised hidden unit Hinton Diagram}
      \label{F:Hinton-Good}
  \end{subfigure}
  ~ %add desired spacing between images, e. g. ~, \quad, \qquad, \hfill etc.
    %(or a blank line to force the subfigure onto a new line)
  \begin{subfigure}[t]{0.3\textwidth}
      \includegraphics[width=\textwidth]{Assets/HINTON2.png}
      \caption{Unutilised hidden unit Hinton Diagram}
      \label{F:Hinton-Bad}
  \end{subfigure}
  \caption{Hinton Diagrams illustrating a trained hidden unit, versus that of an untrained/unutilised hidden unit. This is with no measures to enforce sparsity.}\label{fig:mnist-worse-best-results}
\end{figure}

\subsection{Reconstructions: a measure of performance}

Reconstructions, as described in Section \ref{SS:RBM-Reconstructions}, are a good measure of performance. Reconstructions will be the main way I evaluate the RBM in comparison to the ORBM. To be able to use this reconstruction based evaluation, knowledge of the ground truth is required. This is so:
\begin{itemize}
  \item RBMs could be trained and then plugged into the ORBM network.
  \item Reconstructions could be compared directly (by score) to the ground truth.
\end{itemize}

\subsection{Evaluating RBMs: Problem dependent}

  Evaluating RBMs is problem dependent, we can examine the different approaches available by splitting problems into two classes:
  \begin{description}
  \item[Problems with small dimensionality] When the dimensionality of the input is very small ($<10$) and all possible inputs are known, we can perform analytical evaluations of the RBM. By clamping the visible units of the RBM to each input in the training set, reconstructions can be created. This process can be repeated for each input, recording the frequency that each reconstruction occurs on a bar graph. We can then ensure that the RBM is reconstructing the input perfectly.

  In a similar way to the reconstructions, samples can be drawn from RBM in a `free phase` without the input clamped to a given visible. The bar graph should exhibit dream visible patterns from the whole training set approximately an equal proportion to that of the training set.

  \item[Problems with large dimensionality] In non-trivial cases, with larger datasets, reconstructions can be inspected and compared to the training dataset. However, empirically detecting if a model is trained is difficult, especially given the unsupervised, black box nature of RBMs.
  Alternatively, a `score` can be assigned to each item in the training set based on how well the RBM can reconstruct it.
  This project will use two scores the \emph{Cross Entropy}~\cite{golik2013cross} and the \emph{Negative Cosine Angle} to compare the reconstructions to the ground truth images.
  Dreams can also be examined in larger cases, but empirically detecting if they are `correct' is infeasible. Nevertheless, Dreams are useful to examine, as if they look like items in the training set, this gives confidence the RBM is capturing the model.

\end{description}

All evaluations followed the same high level process, working from trivial cases to more challenging tasks. By changing only a few aspects from problem to problem it allows me to identify what qualities of the problem make the algorithm less or more effective. These asepcts were typically the dimensionality and then ultimately the difficultly of the problem. The justification for this being that trivial cases make unit testing feasible, therefore ensuring conclusions can be drawn with regard to the algorithm and model, not an incorrect implementation.

The evaluations in this project aimed to evaluate the ORBM and it's inference algorithm by examining reconstructions given a multi-cause image. Because the multi-cause inputs were images composed of known images, I could then compare the reconstructions to the ground truth. An example of a multi-cause input with two quadrilaterals is shown in Figure \ref{F:Composite-Example}. The optimal reconstructions are equivalent to the two images that combined form the input.

\begin{wrapfigure}{r}{0.6\textwidth}
  \begin{center}
    \includegraphics[width=0.48\textwidth]{Assets/Composite-Example.png}
  \end{center}
  \caption{A figure illustrating two five by five pixel images combining to form a composite/multicause input. The images are binary.}
  \label{F:Composite-Example}
\end{wrapfigure}

In the smaller dimensional cases the reconstructions are inspected by plotting the reconstructions and the frequency with which they occurred after a large amount of repetitions. In the larger dimensional tasks, the two `scores' are used to evaluate reconstructions against the corresponding item of the training set.

\subsection{Choice of Evaluation Datasets}

As my evaluations follow the same outline of training RBMs, plugging them into the ORBM, and evaluating the reconstructions, the key aspect of the evaluation that needs to be designed is the datasets for these tasks. It is important the images of the datasets are \emph{multi-subject} and that the individual images used to compose the training images are known. This is to allow the reconstructions of the ORBM (and RBM) to be compared to these underlying images, exploring how well the ORBM and RBM can perform source separation.

\begin{description}
  \item[2 Bit XOR] The minimal case of image source separation. The RBMs are trained on a single bit being on in a two bit pattern. The training set then resembles a 2 bit XOR truth table. Because the dimensions of this task are so small, reconstructions and dreams can be evaluated empirically. Also the algorithm can be unit tested ensuring the outputs of the different steps are correct, comparing them to values calculated by hand. Also as the dimensionality is so small, the Approximated Correction and the Full Correction can be compared to ensure that the approximated correction works well in practice.
  \item[$X$ neighboring bits on in $Y$ bit pattern] This is a natural next step from 2 bit XOR, and is effectively the same task but in larger dimensions. It is trivial to train an RBM to represent this problem, and the algorithm is quick to run on such a small dimensionality creating a quick feedback loop for development.
  \item[2 by 2 squares in a 5 by 5 pixel image] This dataset extends the previous increasing to 25 dimensions. The dataset is trivial to construct and corresponds to a square of 2 by 2 pixels being on in a 5 by 5 pixel image. Interesting cases can be explored and then inspected visually as images, which are easier to interpret than bit strings (especially as the bit strings get larger than 10).
  \item[Rectangles in a 5 by 5 pixel image] This dataset builds on the previous by introducing different shaped \emph{subjects} to the images. This means that two RBMs need to be trained, one for each subject. This dataset, much like the previous allows for compelling cases to be separated. It sets the scene for the larger dimension cases.
  \item[MNIST Handwritten Digit Dataset] This is a prolific machine learning dataset~\cite{mnistlecun}. By using the dataset in this project it aids in reproducibility and as there has been previous work it is known that RBMs can be trained successfully on the data. The dataset is comprised of 28 by 28 pixel handwritten digit images, where a pixel value is between 0 and 1. There are digits 0 through 9. To make a multi-cause input, each digits it is composed with every other digit, making the novel, yet non-trival task of separating two digits from a single image.
\end{description}


\section{Introduction}
\label{JM:sec:introduction}

In the world of today's sciences, the use of computers has transformed the field and requirements for researchers tremendously.
The ability to collect and process data at high rates using (semi-) automatic approaches pushes the classical research fields like
physics, biology, medicine, engineering, and social sciences towards hybrid models and big data evaluation. A recent example is the detection of gravitational waves,
where multiple software components have been used to process the data and help the researchers to classify the events \cite{JMMacleodEtAl2021} .\\

The requirements on software development used for research purposes have decreased below expert knowledge. Research software and coding are much more of a general tool, similar to text editors, for many scientific disciplines.
As a result scientists from fields where programming and numerical computation play at most a minor role, e.g. psychology, sociology but also biology, start to develop code on their own.
This is reflected in the rise of high-level languages like \textit{Python}, \textit{Ruby}, \textit{R}, or \textit{Lua} \cite{JMVanRossumDrake2009,JMThomasEtAl2005, JMRCT2016, JMIerusalimschy2006} which are semantically easier to program as opposed to lower-level languages such as
\textit{C}, \textit{C++}, \textit{Fortran}, \textit{Go}, and \textit{Rust} \cite{JMKernighanRitchie2006, JMStroustrup2013, JMBackusHeising1964, JMMeyerson2014, JMMatsakisKlockII2014}.
The broader, more diverse user base of high level languages results in a more versatile ecosystem, giving rise to various packages and modules designed
for specific tasks with a clean interface. Despise these advantages, the price to be paid is commonly linked to the performance in terms of execution time and memory consumption of the resulting code.
A common pattern used is hence to write an application programming interface (API) in high-level languages and move the computationally expensive parts to a low-level implementation after finishing prototyping.
This is referred to as the \textit{two language problem}.
This approach, while being general, has some major drawbacks related to datatype availability, effort of maintanance, and dependency management - to name just a few.\\

To resolve this - and other - problems of scientific computation the \textit{Julia} programming language has been created \cite{JMBezansonEtAl2015}. It aims to provide performant code, leveraging the best features of
prominent languages without complicated semantic structures, contradicting the common misconception that computational performance comes at the price of ease of use \cite{JMEdelman2019}.
Even though being a relatively new language, initially released in 2012 for the broader public, \textit{Julia} gained much momentum. \\

In the following, \textit{Julia} will be introduced as a language for scientifc computation. Section \ref{JM:sec:basics} starts by introducing how to define and analyse functions before introducing the concept of multiple dispatch to the reader.
Afterward, section \ref{JM:sec:JITOpt} explains benchmarking, optimizing code, and just-in-time compilation. Section \ref{JM:sec:GOMP} a simple greedy algorithm for sparse regression is implemented, showcasing the ease of use to transpile mathematical notation to fast code.
Section \ref{JM:sec:ECO} highlights some of the most useful and prominent packages of the language.
A conclusion is drawn in section \ref{JM:sec:CONC}.

\newpage


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{The physics of magnetic monopoles}
\label{sec:theory}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
To date there has been no solid, reproducible experimental evidence that
magnetic monopoles exist. For now, magnetic charge remains an entirely
theoretical concept that would, amongst other things, neaten up Maxwell's
equations of electrodynamics.
%
So why would we -- \emph{why should we} -- spend time and effort looking
for them? There are, in fact, some very good reasons to believe they exist
and that we just haven't found them yet.
%
One theorist has even said that they are ``\emph{one of the safest
bets one can make about physics not yet seen}''~\cite{Polchinski2004}.
%
In this section we look at some of these reasons -- so you
can decide for yourself!

%=============================================================================
\subsection{Dirac's magnetic monopole}
\label{sec:diracmonopole}
%=============================================================================
Paul Adrien Maurice Dirac (Figure~\ref{fig:pamdirac}) was a theoretical
theorist who played an important part in the development of quantum mechanics.
He was one of the first theoreticians to combine quantum mechanics with
special relativity with his eponymous equation~\cite{Dirac1928},
which allowed so-called ``negative energy'' solutions~\cite{Dirac1930}.
%
At first Dirac interpreted these solutions,
which would require the particle corresponding to the negatively-charged
electron to have a positive charge, as protons.
%
He noted in a subsequent paper~\cite{Dirac1931} that, actually,
the negative energy particles would need to have the same mass as the
electron -- the positively-charged anti-electron.
%
He had predicted antimatter.
%
This prediction was confirmed with Anderson's discovery of the
positron~\cite{Anderson1933} and the rest, as they say, is history.

%
\begin{figure}[htbp]
  \centering
  \includegraphics[width=0.25\textwidth]{assets/images/dirac-portrait/dirac-portrait.jpg}
  \caption[Paul A. M. Dirac]
  {\label{fig:pamdirac}%
Paul Adrien Maurice Dirac (1902-1984) predicted the existence of antimatter~\cite{Dirac1930}. %
Could he have been right about magnetic monopoles too?~\cite{Dirac1931}. %
Image credit: \href{https://commons.wikimedia.org/wiki/File:Dirac_4.jpg}{nobelprize.org} (public domain); %
please refer to their website regarding re-use of this image.}
\end{figure}
%

Funnily enough, in the same paper Dirac had made another prediction.
(In fact, this prediction was the main point of the paper!)
In trying to explain why there was a smallest, discrete value of the
electric charge (i.e. why charge is quantised),
he postulated the existence of isolated magnetic charge that would show
a ``\emph{symmetry between electricity and magnetism quite foreign to current views}''.
He had predicted the magnetic monopole.

%
\begin{figure}[htbp]
  \centering
  \includegraphics[width=0.5\textwidth]{assets/images/dirac-monopole/dirac-monopole.png}
  \caption[A schematic representation of Dirac's magnetic monopole]
  {\label{fig:diracmonopole}%
A schematic representation of Dirac's magnetic monopole.
The monopole is imagined to be the end point of a semi-infinitely long,
infinitesimally thin solenoid known as a ``Dirac string'',
here shown from the side (red lines).
%
The magnetic field lines (black arrows) are shown emanating from the point
at the end of the string (imagine the red lines are infinitely close together).
If such Dirac monopoles exist, it is the requirement that the string is
undetectable that means electric charge must be quantised.
%
Image credit: \href{http://researchinschools.org}{Institute for Research in Schools}.}
\end{figure}
%

Dirac's monopole was bit of a funny object.
Like a particle with electric charge,
the monopole would need to exude magnetic field field lines spherically
from its centre. Other magnetic charges placed in this field would experience
a force exactly analogous to the Coulomb force that pushes like charges apart
and pulls opposite charges together.
To create such a magnetic field configuration,
Dirac imagined a semi-infinitely long, infinitesimally thin solenoid
(i.e. a very, very tightly-wound coil with a flowing electric current).
%
The end of this ``Dirac string'' -- as shown schematically in
Figure~\ref{fig:diracmonopole} 2 -- represents the monopole.

While it is easy to imagine such a string,
one might not necessarily want to get tangled up in it.
Dirac showed that if such a string/monopole configuration did exist in
Nature, the only way that the mathematics could work out such that the
strings were undetectable by experiments was if electric charge was quantised.
There is an excellent explanation of the quantum theory behind this assertion
using the double slit experiment (as well as many other aspects of magnetic
monopoles) in~\cite{Rajantie2012}.
%
As the quantisation of charge is observed experimentally,
Dirac reasoned that magnetic monopoles (and their associated strings)
must exist.
%
Unfortunately, unlike the positron, monopoles were not found a few years
later.

%=============================================================================
\subsection{Grand Unified Theories}
\label{sec:guts}
%=============================================================================
Theorists, in the meantime, had been developing \ac{QFT}
as a way of describing matter and forces not in terms of particles or waves
but mathematical constructs called fields.
%
(If you want to get your head around the wave-particle duality,
study physics at university and take a \ac{QFT} course.
Your hands will thank you from all of the ``hand-waving'' you won’t have to do.)
%
The 1960s saw the electromagnetic and weak forces -- two of the fundamental
forces of Nature -- united into a single electroweak force,
winning Glashow, Salam and Weinberg the 1979 Nobel Prize for
Physics.
The 1970s then saw a great deal of interest in literally going one
better -- uniting the electromagnetic, weak, and strong forces into one force.
Such a theory is known as a \ac{GUT}. %Grand Unified Theories (GUTs).
Figure~\ref{fig:forceunification} shows roughly where this ``Grand Unification''
might occur in terms of energy (around $10^{13}$ \ac{TeV}) and time
(about $10^{-36}$ seconds after the Big Bang).

%
\begin{figure}[htbp]
  \centering
  \includegraphics[width=0.6\textwidth]{assets/images/force-unification/force-unification.jpg}
  \caption[The unification of the four fundamental forces of Nature]
  {\label{fig:forceunification}%
The unification of the four fundamental forces of Nature,
as shown in this ``\emph{Postcard from the Terascale}''.
Image credit: Symmetry Magazine/Sandbox Illustrations; please contact them
regarding licensing/re-use of this image.}
\end{figure}
%

What does this have to do with magnetic monopoles?
Well, in the course of trying to get the mathematics to work,
two theorists independently found that an inevitable consequence of bringing
the three forces together was the appearance of terms in the equations that
represented particles (well, fields) with
magnetic charge~\cite{tHooft1974,Polyakov1974}.
You couldn't unite the strong force with the electroweak force
without magnetic monopoles\footnote{%
As it happens, the monopoles appear when you squeeze the \ac{GUT} to
quantise the electromagnetic field into particles.
So, as noted in many reviews of the magnetic monopole literature,
Dirac’s argument is reversed: the quantisation of the electromagnetic
field necessitates the existence of magnetic monopoles.}.
%
Of course, no Grand Unified Theory has been experimentally verified,
but if we believe in the ultimate unification of the
fundamental forces -- which, let’s be honest, would be rather nice -- it
turns out that we \emph{need} magnetic monopoles.

%=============================================================================
\subsection{Monopoles in the Standard Model}
\label{sec:smmonopole}
%=============================================================================
There is, however, one slight problem with \ac{GUT}-scale magnetic monopoles.
While one cannot predict the mass of the Dirac monopole,
\ac{GUT} monopole masses (which can be calculated) tend to be billions
and billions times the energies reachable by the \ac{LHC}, or indeed any
physical process occurring more than a fraction of a second after the Big Bang.
While we can look for ``relic'' monopoles in cosmic rays that reach the
Earth --and~\cite{Patrizii2015} provides an excellent summary of the searches
conducted so far -- we would need something between the Dirac monopole
and the \ac{GUT} monopole in order to have some hope of creating
magnetic monopoles in accelerators with all of the benefits that the
controlled conditions of the laboratory bring.

%
\begin{figure}[htbp]
  \centering
  \includegraphics[width=1.0\textwidth]{assets/images/monopole-artist/monopole-artist.png}
  \caption[An artist's impression of magnetic monopole production at the LHC]
  {\label{fig:monopoleartist}%
An artist's impression of magnetic monopole production in a proton-proton
collision at the \acf{LHC}. Image credit: The MoEDAL Collaboration/H. Valja 2015;
please contact them regarding licensing/re-use of this image.}
\end{figure}
%

Fortunately, such a monopole has been proposed.
An ``\term{electroweak monopole}'' (which, incidentally, has twice the
magnetic charge of Dirac's monopole) can be made to appear in the
Standard Model of particle physics by adding some extra terms to the
field equations~\cite{Cho1997,Yang1998}.
In fact, the theorists behind the electroweak monopole insist that the
Standard Model is incomplete without the electroweak monopole -- and
given that we know the Standard Model is already broken
by the fact neutrinos have mass, we know that we can't just leave it at
the discovery of the Higgs boson.
%
Semantics aside, what's important is that some
recent papers~\cite{Kimm2016,Ellis2016} have suggested that the
electroweak monopole would have a mass of below 7 \acs{TeV},
putting it within tantalising reach of the Large Hadron Collider and
the MoEDAL experiment.
%
Figure~\ref{fig:monopoleartist} shows an artist's impression of
monopole-antimonopole production in \acs{LHC} proton-proton collisions.
%
The question is -- can Nature produce something as beautiful?


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{graphicx}
  \DeclareGraphicsExtensions{.png, .jpeg}
\usepackage{caption}
\usepackage[top=1in, bottom=1in, left=1in, right=1in]{geometry}

\title{STAT 775: Machine Learning \\ Midterm}
\author{Terence Henriod}
\date{\today}

\begin{document}

\clearpage            % All of
\maketitle            % this,
\thispagestyle{empty} % removes the page number from the title page

\begin{abstract}
This take-home Midterm is due Wednesday at 1:00 p.m. at the beginning of class.
\end{abstract}

\newpage
\section{Problem 01}
\subsection{Problem Statement}
ESL Exercise 4.2:

\subsection{Solution}
\begin{enumerate}
  \item Recall that linear discriminant functions take the form
  $$\delta_{k}(x) = x^{T}\Sigma^{-1}\mu_{k} - \frac{1}{2}\mu_{k}^{T}\Sigma^{-1}\mu_{k} + log\pi_{k}$$
  and for classification we choose the class with the greatest discriminant for a given $x$. In the two class with the parameters as defined in the problem, in order to classify to class 2, we have $\delta_{2}(x) > \delta_{1}(x)$, so
  \begin{align*}
    x^{T}\Sigma^{-1}\mu_{2} - \frac{1}{2}\mu_{2}^{T}\Sigma^{-1}\mu_{2} + log\pi_{2} &> x^{T}\Sigma^{-1}\mu_{1} - \frac{1}{2}\mu_{1}^{T}\Sigma^{-1}\mu_{1} + log\pi_{1}\\
    x^{T}\Sigma^{-1}\mu_{2} - \frac{1}{2}\mu_{2}^{T}\Sigma^{-1}\mu_{2} &> x^{T}\Sigma^{-1}\mu_{1} - \frac{1}{2}\mu_{1}^{T}\Sigma^{-1}\mu_{1} + log\pi_{1} - log\pi_{2}\\
    x^{T}\Sigma^{-1}\mu_{2} &> x^{T}\Sigma^{-1}\mu_{1} + \frac{1}{2}\mu_{2}^{T}\Sigma^{-1}\mu_{2} - \frac{1}{2}\mu_{1}^{T}\Sigma^{-1}\mu_{1} + log\pi_{1} - log\pi_{2}\\
    x^{T}\Sigma^{-1}\mu_{2} - x^{T}\Sigma^{-1}\mu_{1} &> \frac{1}{2}\mu_{2}^{T}\Sigma^{-1}\mu_{2} - \frac{1}{2}\mu_{1}^{T}\Sigma^{-1}\mu_{1} + log\pi_{1} - log\pi_{2}\\
    x^{T}\Sigma^{-1}(\mu_{2} - \mu_{1}) &> \frac{1}{2}\mu_{2}^{T}\Sigma^{-1}\mu_{2} - \frac{1}{2}\mu_{1}^{T}\Sigma^{-1}\mu_{1} + log\pi_{1} - log\pi_{2}\\
  \end{align*}
  Now, since we have class sizes $N_{1}$ and $N_{2}$ with $N = N_{1} + N_{2}$, this means that $\pi_{k} = N_{k} / N$. This gives the final equation
  $$ x^{T}\Sigma^{-1}(\mu_{2} - \mu_{1}) > \frac{1}{2}\mu_{2}^{T}\Sigma^{-1}\mu_{2} - \frac{1}{2}\mu_{1}^{T}\Sigma^{-1}\mu_{1} + log(\frac{N_{1}}{N}) - log(\frac{N_{2}}{N}) $$

  \item I don't think I'm able to solve this one.

  \item In the previous problem statement, the assertion $ \hat{\Sigma_{B}} = (\hat{mu_{2}} - \hat{\mu_{1}})(\hat{mu_{2}} - \hat{\mu_{1}})^{T}$ is made, we can say that
  $$ \hat{\Sigma_{B}}\beta = (\hat{mu_{2}} - \hat{\mu_{1}})(\hat{mu_{2}} - \hat{\mu_{1}})^{T} $$
  which is equivalent to:
  $$ (\hat{mu_{2}} - \hat{\mu_{1}}) * k $$
  for some scalar $k$. Since $k$ is a scalar, $ (\hat{mu_{2}} - \hat{\mu_{1}}) * k $ will be in the same direction as $ (\hat{mu_{2}} - \hat{\mu_{1}}) $, just in a different direction.

  \item Our results thus far have used a distinct coding of the classes. So the previous result holds in that regard. As for the class codings being arbitrary, the encodings used in the problem statement seem arbitrary enough, so I would assume the result holds in this regard as well.

  \item I don't think I'm able to solve this one.
\end{enumerate}

\newpage
\section{Problem 02}
\subsection{Problem Statement}
ESL Exercise 4.6:

\subsection{Solution}
\begin{enumerate}
  \item The problem statement allows the assumption that the data are separable, and in fact separable by some $\beta_{SEP}$. By Eq. 4.46 from the ESL book, in the notation of the problem, then $ y_{i}\beta_{SEP}^{T}z_{i} \ge D $ for some minimum distance from the decision boundary $D$ must hold. Since for any $\beta_{1}$ and $\beta_{0}$ (which compose $\beta_{SEP}$) that satisfy the inequality, any positively scaled such $\beta$s will satisfy the inequality also. Thus we can choose some scaling factor $\rho$ s.t. $ \rho * D = 1$, there must exist a
  $\beta_{SEP} = \beta_{SEP}\prime * \rho$ ($\beta_{SEP}\prime$ is some other, acceptable solution) s.t.:
  $$ y_{i}\beta_{SEP}^{T}z_{i} \ge 1 $$

  \item First, to show that the inequality holds:
  \begin{align*}
    ||\beta_{new} - \beta_{sep}||^{2}
      &= ||\beta_{new}||^{2} + ||\beta_{sep}||^{2} - 2\beta_{sep}^{T}\beta_new \\
    &= ||\beta_{old} + y_{i}z_{i}||^{2} + ||\beta_{sep}||^{2} - 2\beta_{sep}^{T}(\beta_{old} + y_{i}z_{i}) \\
    &= ||\beta_{old}||^{2} + ||y_{i}z_{i}||^{2} + 2\beta_{old}^{T}y_{i}z_{i} + ||\beta_{sep}||^{2} - 2\beta_{sep}^{T}(\beta_{old} + y_{i}z_{i}) \\
    &= ||\beta_{old}||^{2} + 1 + 2\beta_{old}^{T}y_{i}z_{i} + ||\beta_{sep}||^{2} - 2\beta_{sep}^{T}(\beta_{old} + y_{i}z_{i}) \\
    &= ||\beta_{old}||^{2} + 2\beta_{old}^{T}y_{i}z_{i} + ||\beta_{sep}||^{2} - 2\beta_{sep}^{T}\beta_{old} - 2y_{i}\beta_{sep}^{T}z_{i} + 1\\
    &\le ||\beta_{old}||^{2} + 2\beta_{old}^{T}y_{i}z_{i} + ||\beta_{sep}||^{2} - 2\beta_{sep}^{T}\beta_{old} - 2 + 1\\
    &\le ||\beta_{old}||^{2} + ||\beta_{sep}||^{2} + 2\beta_{old}^{T}y_{i}z_{i} - 2\beta_{sep}^{T}\beta_{old} - 1 \\
    &\le ||\beta_{old} - \beta_{sep}||^{2} - 1 \\
  \end{align*}
  using $y_{i}\beta_{sep}^{T}z_{i} \ge 1$ from the previous part of the exercise. Also, in the previous exercise, because the result $y_{i}\beta_{sep}^{T}z_{i} \ge 1$ indicated a maximum separation distance, we could say that $2\beta_{old}^{T}y_{i}z_{i} \le y_{i}\beta_{sep}^{T}z_{i} $ and so that term is dwarfed by the
  $-2\beta_{sep}^{T}\beta_{old}$ term, dropping it from the equation. This result means that the inequality
  \begin{align*}
  ||\beta_{old} - \beta_{sep}||^{2} + 2\beta_{old}^{T}y_{i}z_{i} - 1 &\le ||\beta_{old} - \beta_{sep}||^{2} - 1 \\
  ||\beta_{new} - \beta_{sep}||^{2} &\le ||\beta_{old} - \beta_{sep}||^{2} - 1
  \end{align*}
  must hold. \\

  Second, if we are to reach a solution, then $||\beta_{old} - \beta_{sep}||^{2} = 0$ at some point. By the previous result, each iteration improves our $\beta$ solution by at least $1$ each time. Suppose we start at some $\beta$ $\beta_{start}$, where $||\beta_{start} - \beta_{sep}||^{2} = i$. Since the algorithm improves the solution and reduces $i$ by $1$ with each iteration, we will reach a $\beta_{new} = \beta_{sep}$ in $i$ or fewer iterations.
\end{enumerate}

\newpage
\section{Problem 03}
\subsection{Problem Statement}
Build a logistic resgression classifier for the South African Heart Disease data and compare its performance with a neural network with one hidden layer.

\subsection{Results}
The results were somewhat disappointing. Neither classifier offered particularly good performance, and the neural net only got the performance it did by classifying everyithing as the most likely class. (I hope this was just the nature of the data and not because I forgot to do perform some technique or operation correctly). The results use a neural net with a hidden layer with 3 units, but increasing the number of hidden units produces the same result. Even dimensionality reduction techniques did not seem to help much. Shuffling the data proved only slightly helpful. As we discussed, the neural net should never under-perform logistic regression since LR is a subset of the neural network approach.

The confusion matrices were as follows.\\
For LR:\\
\newline
\begin{tabular}{| c || c | c | c |}
  \hline
  Actual / Predicted & No CHD & CHD & \% Correct \\
  \hline
  \hline
  No CHD             &     90 &  17 &         84 \\
  \hline
  CHD                &     16 &  31 &         66 \\
  \hline
  \hline
  Overall            &        &     &         79 \\
  \hline
\end{tabular}
%
%
\hfill\newline\hfill\\
For Neural Net:\\
\begin{tabular}{| c || c | c | c |}
  \hline
  Actual / Predicted & No CHD & CHD & \% Correct \\
  \hline
  \hline
  No CHD             &    107 &     &        100 \\
  \hline
  CHD                &     47 &     &          0 \\
  \hline
  \hline
  Overall            &        &     &         69 \\
  \hline
\end{tabular}

\subsection{Code}
\begin{verbatim}
#
# Initial Setup
#
setwd("C:/Users/Terence/Documents/GitHub/STAT775/Midterm/")
# setwd("~/STAT775/HW03/")

DATA.FILE.NAME <- "../DataSets/south.african.heart.disease/data.disease"

#
# Data Handling
#
shuffle.data <- function(x) {
  # x: a numeric matrix

  for (i in 1:nrow(x)) {
    one <- round(runif(1, 1, nrow(x)))
    other <- round(runif(1, 1, nrow(x)))

    temp <- x[one, ]
    x[one, ] <- x[other, ]
    x[other, ] <- temp
  }

  return(x)
}

code.data <- function(dat.a) {
  family.history <- rep(0.0, nrow(dat.a))

  data <- data.frame(dat.a[, 1:4], family.history, dat.a[6:10])

  for (i in 1:nrow(data)) {
    if (dat.a[i, 'famhist'] == 'Present') {
      data[i, 'family.history'] <- 1.0
    }
  }

  return(shuffle.data(data.matrix(data)))
}

get.data.tuples <- function(data) {
  last.column <- ncol(data)

  return(list(
    observations = (data.matrix(data[, -last.column])),
    targets = data.matrix(data[, last.column])
  ))
}

#
# Logistic Regression
#
logistic.predict <- function(x, b) {
  #
  # Args:
  #   x: a p x 1 column vector of observation values
  #   b: a p X 1 column vector of model weights (betas)

  e <- exp(matrix(b, nrow = 1) %*% rbind(1.0, matrix(x, ncol = 1)))
  return(e / (1.0 + e))
}

newton.raphson.iteration <- function(X, y, old.betas, step.size = 0.5) {
  #
  # Args:
  #   X: N x p matrix of observations
  #   y: column vector of N target labels
  #   old.betas: column vector of (p + 1) model parameters
  #   step.size: a rate controlling parameter

  p <- matrix(0, nrow = nrow(y), ncol = 1)
  for (i in 1:nrow(y)) {
    p[i, 1] <- logistic.predict(x = data.matrix(X[i, ]), b = old.betas)
  }

  W <- diag(1, nrow(y), nrow(y))
  for (i in 1:nrow(y)) {
    W[i, i] <- p[i, 1] * (1.0 - p[i, 1])
  }

  X.b <- cbind(1.0, X)  # X with 1s prepended for bias term

  z <- (X.b %*% old.betas) +
    (solve(W + diag(0.001, nrow(W), ncol(W))) %*% (y - p))

  step <- solve((t(X.b) %*% W %*% X.b) + diag(0.001, ncol(X.b), ncol(X.b))) %*%
    t(X.b) %*% (y - p)
  step <- step * step.size

  return(old.betas + step)
}

train.logistic.model <- function(X, y, n.iterations = 400, step.size = 0.2) {
  #
  # Args:
  #   X: N x p matrix of observations, observations are rows
  #   y: N x 1 column vector of class labels {0, 1}
  #   n.iterations: the number of approximation iterations to perform
  #   step.size: used to slow the rate of convergence to prevent overshooting

  # start with model parameters at 0 - arbitrary starting point
  betas <- matrix(0, nrow = ncol(X) + 1, 1)

  # iterate until parameter updates become arbitrarily small
  old.betas <- betas + 1.0
  i <- 0
  while (i < n.iterations) {
    betas <- newton.raphson.iteration(
      X = X,
      y = y,
      old.betas = betas,
      step.size = step.size
    )

    # TODO: use method for finding convergence, not just arbitrary iterations

    i <- i + 1
  }

  return(betas)
}

#
# Neural Net
#
sigmoid <- function(x) {
  #
  # Args:
  #   x: a numeric or vector; the function is applied element-wise

  return(1.0 / (1.0 + exp(-x)))
}

sigmoid.derivative <- function(x) {
  #
  # Args:
  #   x: a numeric or vector

  return(sigmoid(x) * (1.0 - sigmoid(x)))
}

construct.neural.net <- function(
  topology = c(2, 2, 1),
  activation = sigmoid,
  activation.derivative = sigmoid.derivative,
  debug = F) {
  #
  # Args:
  #   topology: a list or vector of the dimensions of each layer
  #   activation: a function to be used for the activation of each unit
  #   activation.derivative: a function that is the derivative of activation

  layer.weights <- list()
  derivative.matrices <- list()
  outputs <- list()

  previous.layer.dim <- 1
  next.layer.dim <- 1
  for (i in 1:(length(topology) - 1)) {
    previous.layer.dim <- topology[[i]] + 1  # +1 for bias
    next.layer.dim <- topology[[i + 1]]
    num.elements <- (previous.layer.dim) * next.layer.dim

    layer.weights[[i]] <- matrix(
      if(debug) {rep(1, num.elements)}
      else {runif(n = num.elements, min = -0.001, max = 0.001)},
      nrow = previous.layer.dim,
      ncol = next.layer.dim
    )

    outputs[[i]] <- matrix(0, nrow = next.layer.dim, ncol = 1)
    derivative.matrices[[i]] <- diag(0, next.layer.dim)
  }

  return(list(
    input.dim = topology[[1]],
    output.dim = next.layer.dim,  # should be dim of last layer
    n.layers = length(layer.weights),
    activation = activation,
    activation.deriv = activation.derivative,
    input = matrix(0, nrow = 1, ncol = topology[[1]]),
    output = matrix(0, nrow = tail(topology, 1)[[1]], 1),
    weights = layer.weights,
    outputs = outputs,
    derivatives = derivative.matrices
  ))
}

apply.inputs <- function(net, x, for.training = T) {
  #
  # Args:
  #   x: a 1 x n vector of inputs; n should be the same as for net
  #   net: a structure with all of the appropriate data for a neural network,
  #        as created by construct.neural.net()
  #   for.training[T]: currently unused

  net$input <- matrix(x, nrow = 1)
  previous.output <- cbind(net$input, 1)
  for (i in 1:net$n.layers) {
    weighted.sums <- previous.output %*% net$weights[[i]]

    net$outputs[[i]] <- net$activation(weighted.sums)

    net$derivatives[[i]] <- diag(
      as.list(net$activation.deriv(weighted.sums)),
      length(net$outputs[[i]])
    )

    previous.output <- cbind(net$outputs[[i]], 1)
  }

  net$output <- t(tail(net$outputs, 1)[[1]])

  return(net)
}

backprop.weight.update <- function(net, target, learning.rate = 2) {
  #
  # Args:
  #   net: a neural net object that has had inputs applied and derivatives stored
  #   target: a column vector; the target output that should have been observed
  #   learning.rate: the learning rate of the network
  #                  TODO: refactor learning.rate to be less hacky, allow for
  #                        advanced techniques

  last.index <- net$n.layers

  deltas <- list()
  error <- matrix(net$output, ncol = 1) - matrix(target, ncol = 1)
  deltas[[last.index + 1]] <- error
  W <- diag(1, nrow = nrow(error))
  D <- net$derivatives[[last.index]]
  for (i in last.index:1) {
    deltas[[i]] <- D %*% W %*% deltas[[i + 1]]

    if (i > 1) {
      D <- net$derivatives[[i - 1]]
      W <- net$weights[[i]]
      W <- W[1:(nrow(W) - 1), ]
    }
  }

  weight.updates <- list()
  o.hat <- cbind(net$input, 1)
  for (i in 1:last.index) {
    weight.updates[[i]] <- -learning.rate * t(deltas[[i]] %*% o.hat)

    if (i < last.index) {
      o.hat <- cbind(net$outputs[[i]], 1)
    }
  }

  for (i in 1:length(weight.updates)) {
    net$weights[[i]] <- net$weights[[i]] + weight.updates[[i]]
  }

  return(net)
}

######
# MAIN
######
data.fram.e <- read.table(
  "http://www-stat.stanford.edu/~tibs/ElemStatLearn/datasets/SAheart.data",
  sep = ",",
  head = T,
  row.names = 1
)
data <- code.data(data.fram.e)
boundary <- 2 * nrow(data) / 3
train <- get.data.tuples(data[1:boundary, ])
test <- get.data.tuples(data[-(1:boundary), ])

# Logistic
logistic.model <- train.logistic.model(X = train$observations, y = train$targets)

logistic.confusion <- matrix(0, nrow = 3, ncol = 3)
for (i in 1:nrow(test$targets)) {
  prediction <- round(logistic.predict(x = test$observations[i, ], b = logistic.model))
  label <- test$targets[[i]]

  logistic.confusion[label + 1, prediction + 1] <-
    logistic.confusion[label + 1, prediction + 1] + 1
}

# Neural Net
NUM.EPOCHS <- 700
chd.net <- construct.neural.net(
  topology = c(9, 3, 1),
  activation = sigmoid,
  activation.deriv = sigmoid.derivative
)

for (t in 1:NUM.EPOCHS) {
  for (i in 1:nrow(train$targets)) {
    chd.net <- apply.inputs(
      net = chd.net,
      x = matrix(train$observations[i, ], nrow = 1)
    )
    chd.net <- backprop.weight.update(
      net = chd.net,
      target = matrix(train$targets[i, ], ncol = 1),
      learning.rate = 0.2
    )
  }
}

net.confusion <- matrix(0, 3, 3)
for (i in 1:nrow(test$observations)) {
  prediction <- round(
    apply.inputs(
      net = chd.net,
      x = matrix(test$observations[i, ], nrow = 1),
      for.training = F
    )$output
  )

  label <- test$targets[[i]]

  net.confusion[label + 1, prediction + 1] <-
    net.confusion[label + 1, prediction + 1] + 1
}

# Results
compute.confusion.percentages <- function(confusion) {
  rownames(confusion) <- c('Actual No CHD', 'Actual CHD', '%.Correct')
  colnames(confusion) <- c('Actual No CHD', 'Actual CHD', '%.Correct')

  row.sums <- rowSums(confusion)
  col.sums <- colSums(confusion)
  correct <- 0
  for (i in 1:2) {
    confusion[i, 3] <- confusion[i, i] / row.sums[[i]]
    confusion[3, i] <- confusion[i, i] / col.sums[[i]]
    correct <- correct + confusion[i, i]
  }
  confusion[[3,3]] <- correct / sum(row.sums)

  return(confusion)
}

logistic.confusion <-compute.confusion.percentages(logistic.confusion)
net.confusion <-compute.confusion.percentages(net.confusion)
print(logistic.confusion, digits = 2)
print(net.confusion, digits = 2)
\end{verbatim}

\newpage
\section{Problem 03}
\subsection{Problem Statement}
Build a two classifiers for lpsa in the prostate data base. Compare the performance of linear regresssion against a neural network with a hidden layer (scale the output).

\subsection{Results}
Once again the neural net was outperformed. This could be because I did not ``scale the outputs" properly (I was unsure about how to go about this), or because the nerual net is too simple to be effective.

I was not sure how to present the results, so here are some screenshots of various R output indicating a comparison of the Residual Sum of Squares (RSS) for each method and example comparisons:

\begin{figure}[h!]
  \centering
  \includegraphics[width=.8\linewidth]{Problem_04_results}
  \caption{Error results for the logistic regression and neural net models.}
  \label{fig:P04_results}
\end{figure}

\subsection{Code}
\begin{verbatim}
#
# Initial Setup
#
setwd("C:/Users/Terence/Documents/GitHub/STAT775/HW03/")
DATA.FILE.NAME <- "../DataSets/prostate_cancer/prostate_cancer_data"

#
# Data Handling
#
# these include the intercept that gets added to the matrices
FEATURE.INDICES <- 1:9
TARGET.INDEX <- 10
TRAIN.SET.FLAG <- 11

scale.outputs <- function(x) {
  # Args:
  #  x: a numerical vector, column or row

  range <- max(x) - min(x)
  scaled.x <- (x - min(x)) / range
  return(scaled.x)
}

unscale.outputs <- function(x, max.unscaled, min.unscaled) {
  range <- max.unscaled - min.unscaled
  return((x * range) + min.unscaled)
}

data <- read.table(DATA.FILE.NAME)
# according to the dataset info, this needs to be done
data[, 1:8] <- scale(data[, 1:8], T, T)
intercept <- rep(1, nrow(data))
data <- data.frame(Intercept = intercept, data)

# store data with rows as observations
train.data <- data.matrix(subset(data, data$train == T))[, FEATURE.INDICES]
train.targets <- data.matrix(subset(data, data$train == T))[, TARGET.INDEX]

test.data <- data.matrix(subset(data, data$train == F))[, FEATURE.INDICES]
test.targets <- data.matrix(subset(data, data$train == F))[, TARGET.INDEX]

max.test.target <- max(test.targets)
min.test.target <- min(test.targets)
unscaled.test.targets <- test.targets

train.targets <- scale.outputs(train.targets)
test.targets <- scale.outputs(test.targets)

#
# Regression
#
ols.regression <- function(x, y) {
  #
  # Args:
  #   x: a matrix of observations; each observation is stored as a row
  #   y: a column vector of the targets, or actual value/label, for each
  #      observation

  return(list(
    beta.hat = solve(t(x) %*% x) %*% t(x) %*% matrix(y, ncol = 1)
  ))
}

ols.predict <- function(model, x) {
  #
  # Args:
  #   model: an n x 1 column vector of regression coefficients
  #   x: a ? x n matrix of observation data

  return(t(model$beta.hat) %*% t(x))
}

#
# Neural Net
#

sigmoid <- function(x) {
  #
  # Args:
  #   x: a numeric or vector; the function is applied element-wise

  return(1.0 / (1.0 + exp(-x)))
}

sigmoid.derivative <- function(x) {
  #
  # Args:
  #   x: a numeric or vector

  return(sigmoid(x) * (1.0 - sigmoid(x)))
}

construct.neural.net <- function(
  topology = c(2, 2, 1),
  activation = sigmoid,
  activation.derivative = sigmoid.derivative,
  debug = F) {
  #
  # Args:
  #   topology: a list or vector of the dimensions of each layer
  #   activation: a function to be used for the activation of each unit
  #   activation.derivative: a function that is the derivative of activation

  layer.weights <- list()
  derivative.matrices <- list()
  outputs <- list()

  previous.layer.dim <- 1
  next.layer.dim <- 1
  for (i in 1:(length(topology) - 1)) {
    previous.layer.dim <- topology[[i]] + 1  # +1 for bias
    next.layer.dim <- topology[[i + 1]]
    num.elements <- (previous.layer.dim) * next.layer.dim

    layer.weights[[i]] <- matrix(
      if(debug) {rep(1, num.elements)}
      else {runif(n = num.elements, min = -0.001, max = 0.001)},
      nrow = previous.layer.dim,
      ncol = next.layer.dim
    )

    outputs[[i]] <- matrix(0, nrow = next.layer.dim, ncol = 1)
    derivative.matrices[[i]] <- diag(0, next.layer.dim)
  }

  return(list(
    input.dim = topology[[1]],
    output.dim = next.layer.dim,  # should be dim of last layer
    n.layers = length(layer.weights),
    activation = activation,
    activation.deriv = activation.derivative,
    input = matrix(0, nrow = 1, ncol = topology[[1]]),
    output = matrix(0, nrow = tail(topology, 1)[[1]], 1),
    weights = layer.weights,
    outputs = outputs,
    derivatives = derivative.matrices
  ))
}

apply.inputs <- function(net, x, for.training = T) {
  #
  # Args:
  #   x: a 1 x n vector of inputs; n should be the same as for net
  #   net: a structure with all of the appropriate data for a neural network,
  #        as created by construct.neural.net()
  #   for.training[T]: currently unused

  net$input <- matrix(x, nrow = 1)
  previous.output <- cbind(net$input, 1)
  for (i in 1:net$n.layers) {
    weighted.sums <- previous.output %*% net$weights[[i]]

    net$outputs[[i]] <- net$activation(weighted.sums)

    net$derivatives[[i]] <- diag(
      as.list(net$activation.deriv(weighted.sums)),
      length(net$outputs[[i]])
    )

    previous.output <- cbind(net$outputs[[i]], 1)
  }

  net$output <- t(tail(net$outputs, 1)[[1]])

  return(net)
}

backprop.weight.update <- function(net, target, learning.rate = 2) {
  #
  # Args:
  #   net: a neural net object that has had inputs applied and derivatives stored
  #   target: a column vector; the target output that should have been observed
  #   learning.rate: the learning rate of the network
  #                  TODO: refactor learning.rate to be less hacky, allow for
  #                        advanced techniques

  last.index <- net$n.layers

  deltas <- list()
  error <- matrix(net$output, ncol = 1) - matrix(target, ncol = 1)
  deltas[[last.index + 1]] <- error
  W <- diag(1, nrow = nrow(error))
  D <- net$derivatives[[last.index]]
  for (i in last.index:1) {
    deltas[[i]] <- D %*% W %*% deltas[[i + 1]]

    if (i > 1) {
      D <- net$derivatives[[i - 1]]
      W <- net$weights[[i]]
      W <- W[1:(nrow(W) - 1), ]
    }
  }

  weight.updates <- list()
  o.hat <- cbind(net$input, 1)
  for (i in 1:last.index) {
    weight.updates[[i]] <- -learning.rate * t(deltas[[i]] %*% o.hat)

    if (i < last.index) {
      o.hat <- cbind(net$outputs[[i]], 1)
    }
  }

  for (i in 1:length(weight.updates)) {
    net$weights[[i]] <- net$weights[[i]] + weight.updates[[i]]
  }

  return(net)
}

######
# MAIN
######
ols.model <- ols.regression(x = train.data, y = train.targets)
predictions <- ols.predict(model = ols.model, x = test.data)
OLS.Predictions <- as.list(predictions)
# OLS.Predictions <- as.list(unscale.outputs(predictions, max.test.target, min.test.target))

ols.RSS <- sum((test.targets - predictions) ^ 2)
print(ols.RSS)

# Neural Net
NUM.EPOCHS <- 1000
prostate.net <- construct.neural.net(
  topology = c(ncol(train.data), 10, 1),
  activation = sigmoid,
  activation.deriv = sigmoid.derivative
)

for (t in 1:NUM.EPOCHS) {
  for (i in 1:length(train.targets)) {
    prostate.net <- apply.inputs(
      net = prostate.net,
      x = matrix(train.data[i, ], nrow = 1)
    )
    prostate.net <- backprop.weight.update(
      net = prostate.net,
      target = matrix(train.targets[[i]], ncol = 1),
      learning.rate = 0.2
    )
  }
}

net.RSS <- 0
net.predictions <- list()
for (i in 1:length(test.targets)) {
  prostate.net <- apply.inputs(
    net = prostate.net,
    x = matrix(test.data[i, ], nrow = 1)
  )

  net.predictions[[i]] <- prostate.net$output

  error.squared <- (prostate.net$output - test.targets[[i]]) ^ 2

  net.RSS <- net.RSS + error.squared
}
NET.Predictions <- as.list(net.predictions)
# NET.Predictions <- as.list(unscale.outputs(unlist(net.predictions), max.test.target, min.test.target))

unscaled.test.targets <- test.targets

ols.error <- list()
net.error <- list()
for (i in 1:length(unscaled.test.targets)) {
  ols.error[[i]] <- unscaled.test.targets[[i]] - OLS.Predictions[[i]]
  net.error[[i]] <- unscaled.test.targets[[i]] - NET.Predictions[[i]]
}

results <- data.frame(
  cbind(
    unscaled.test.targets,
    OLS.Predictions,
    NET.Predictions,
    ols.error,
    net.error
  )
)

print(data.frame('OLS RSS' = ols.RSS, 'Neural Net RSS' = net.RSS))
print(results)
\end{verbatim}

\end{document}


\documentclass[journal,hidelinks]{IEEEtran}
\usepackage[utf8]{inputenc}
\usepackage[
  pdftitle={Course Project -- Initial Phase},
  pdfauthor={Andrei Purcarus},
  pdfsubject={ECSE-534 -- Analog Microelectronics}
]{hyperref}
\usepackage{graphicx}
\usepackage[all]{hypcap}
\usepackage{cleveref}
\usepackage{indentfirst}
\usepackage[per-mode=symbol]{siunitx}
\usepackage{amsmath}

\title{ECSE-534 \\ Analog Microelectronics \\ Course Project -- Initial Phase}
\author{Andrei~Purcarus,~260631911,~\IEEEmembership{McGill~University} \\ \href{mailto:andrei.purcaruscraciun@mail.mcgill.ca}{andrei.purcaruscraciun@mail.mcgill.ca}}

\begin{document}
\sloppy

\maketitle

\begin{abstract}

A first order, single bit, $\Delta \Sigma$ A/D converter was designed, simulated, and analyzed using MATLAB and Cadence. This converter was required to achieve a peak SNDR of at least $\SI{60}{\deci\bel}$ with a bandwidth of $\SI{38.22}{\kilo\hertz}$ and a full scale input range of $\pm \SI{0.2}{\volt}$. To meet these goals, a sampling frequency of $\SI{10}{\mega\hertz}$ was used, which resulted in a peak SNDR of $\SI{76.69}{\deci\bel}$ in a system level simulation and of $\SI{55.7223}{\deci\bel}$ in a circuit level simulation. Given that the circuit level simulation SNDR does not meet the specifications, several avenues of inquiry to pursue in the final phase of the project were also suggested.

\end{abstract}

\section{Introduction}

\IEEEPARstart{T}{his} report follows the design, simulation, and analysis of a first order, single bit, low pass $\Delta \Sigma$ A/D converter using a 0.13 micron process from IBM. We used MATLAB and Simulink to perform system level simulations and Cadence and Analog Environment to perform circuit level simulations in said process. The converter was required to meet the following specifications:
\begin{itemize}
  \item Bandwidth: $(1 + \alpha) \cdot \SI{20}{\kilo\hertz}$, where $\alpha =$ (last 3 digits of student ID) / 1000.
  \item SNDR (peak): \SI{60}{\deci\bel} ($>$ 10 bits).
  \item Input Voltage Range: $\SI{\pm 0.2}{\volt}$ relative to AGND.
  \item Power Dissipation: Minimum.
\end{itemize}

In addition, we were assigned a supply voltage of $\SI{1.2}{\volt}$ and an analog ground voltage of AGND $= \SI{0.6}{\volt}$. To meet these specifications, we first performed a system level simulation using MATLAB and Simulink to determine the high level parameters required. Then, we mapped the system to a circuit and performed a circuit level simulation using Cadence and Analog Environment. Finally, we analyzed the performance of the converter and verified if it meets the specifications.

\section{System Level Simulation}

We first simulated the ADC at the system level. We began by calculating the requirements of our system. From our student ID, we can determine that $\alpha = 0.911$, which means our bandwidth is $f_{3dB} = \SI{38.22}{\kilo\hertz}$. To compute our minimal SNR, we assume that the power of the noise in our system will be about equal to the power of the signal harmonics. Thus, we require SNR $= 2$ SNDR, or SNR $= \SI{63}{\deci\bel}$ at the minimum. We can therefore compute our desired oversampling rate as
\[ OSR = 10^{(SNR_{peak} - 2.61)/30} = 103.04 \]
Using this result, we can determine that our minimum sampling frequency must be
\[ f_s = 2 \  OSR \  f_{3dB} = \SI{7.88}{\mega\hertz} \]
To allow for some margin, we chose $f_s = \SI{10}{\mega\hertz}$.

Next, we set up the simulation in Simulink, as shown in \Cref{fig:sim_diagram}. We used a $\SI{0.2}{\volt}$ amplitude sine wave with a DC offset of $\SI{0.6}{\volt}$ as our input to the system, and we sampled this input at a rate of $\SI{10}{\mega\hertz}$. We also used $\SI{1}{\kilo\hertz}$ bins for all SNDR calculations.

\begin{figure}[!htb]
  \centering
  \includegraphics[width=\columnwidth]{simulation/adc_sim.pdf}
  \caption{A system level block diagram of the $\Delta \Sigma$ A/D converter.}
  \label{fig:sim_diagram}
\end{figure}

To find our peak SNDR, we first set the relay outputs to $\SI{0.4}{\volt}/\SI{0.8}{\volt}$ and swept the input frequency. The results are shown in \Cref{fig:sim_freq}. From this plot, we can see that the frequency with the best SNDR is around $\SI{20}{\kilo\hertz}$. This can be explained by the fact that the first harmonic for this input frequency occurs at $\SI{40}{\kilo\hertz}$, which is outside our bandwidth and hence does not add distortion.

\begin{figure}[!htb]
  \centering
  \includegraphics[width=0.8\columnwidth]{simulation/adc_sim_freq.eps}
  \caption{SNDR vs. input frequency for the $\Delta \Sigma$ A/D converter with the relay outputs set at $\SI{0.4}{\volt}/\SI{0.8}{\volt}$.}
  \label{fig:sim_freq}
\end{figure}

Next, we kept the input frequency at $\SI{20}{\kilo\hertz}$ and swept the amplitude of the relay output around a DC offset of $\SI{0.6}{\volt}$. The results are shown in \Cref{fig:sim_vo}. From this plot, we can see that the peak SNDR of $\SI{76.69}{\deci\bel}$ occurs when the relay output amplitude is $\SI{0.4}{\volt}$, corresponding to relay outputs of $\SI{0.2}{\volt}/\SI{1.0}{\volt}$.

\begin{figure}[!htb]
  \centering
  \includegraphics[width=0.8\columnwidth]{simulation/adc_sim_vo.eps}
  \caption{SNDR vs. relay output amplitude for the $\Delta \Sigma$ A/D converter with the input frequency set at $\SI{20}{\kilo\hertz}$.}
  \label{fig:sim_vo}
\end{figure}

Thus, we chose to use an input frequency of $\SI{20}{\kilo\hertz}$ and a relay output amplitude of $\SI{0.4}{\volt}$ for our design. The power spectral density of the output with these settings is shown in \Cref{fig:sim_psd}. This plot clearly shows that the output has a peak at the input frequency of $\SI{20}{\kilo\hertz}$, with the first harmonic occurring outside of our bandwidth at $\SI{40}{\kilo\hertz}$.

\begin{figure}[!htb]
  \centering
  \includegraphics[width=0.8\columnwidth]{simulation/adc_sim_psd.eps}
  \caption{Power spectral density for the output of the $\Delta \Sigma$ A/D converter with the input frequency set at $\SI{20}{\kilo\hertz}$ and the relay outputs set at $\SI{0.2}{\volt}/\SI{1.0}{\volt}$.}
  \label{fig:sim_psd}
\end{figure}

\section{Circuit Level Simulation}

We next mapped the system to a circuit using Cadence. We reused the operational amplifier designed in Assignments \#2 and \#3 to implement an integrator. We also used a modified version of the operational amplifier with the compensation network removed to implement a comparator. These are shown in \Cref{fig:adc} as $U_1$ and $U_2$ respectively. The remaining components required digital logic. As a starting point, we used $\SI{10}{\micro\meter}/\SI{0.12}{\micro\meter}$ NMOS transistors and $\SI{25}{\micro\meter}/\SI{0.12}{\micro\meter}$ PMOS transistors for all our digital logic.

We first implemented a NOT gate and a NOR gate to provide building blocks for our other circuit elements. Schematics of these implementations are shown in \Cref{fig:not_gate,fig:nor_gate}. Then, we used the NOT gate to implement a basic transmission gate, as shown in \Cref{fig:transmission_gate}.

\begin{figure}[!htb]
  \centering
  \includegraphics[height=0.5\columnwidth]{diagrams/not_gate.pdf}
  \caption{A schematic of a NOT gate (inverter) implementation.}
  \label{fig:not_gate}
\end{figure}

\begin{figure}[!htb]
  \centering
  \includegraphics[height=0.8\columnwidth]{diagrams/nor_gate.pdf}
  \caption{A schematic of a two-input NOR gate implementation.}
  \label{fig:nor_gate}
\end{figure}

\begin{figure}[!htb]
  \centering
  \includegraphics[height=0.5\columnwidth]{diagrams/transmission_gate.pdf}
  \caption{A schematic of a transmission gate implementation.}
  \label{fig:transmission_gate}
\end{figure}

\begin{figure*}[!htb]
  \centering
  \includegraphics[width=\textwidth]{diagrams/adc.pdf}
  \caption{A schematic of the $\Delta \Sigma$ A/D converter.}
  \label{fig:adc}
\end{figure*}

Then, we implemented a D flip-flop in order to add a sampling element to our circuit. A schematic of this implementation is shown in \Cref{fig:ff}. This flip-flop is triggered by the falling edge of the clock signal.

\begin{figure}[!htb]
  \centering
  \includegraphics[width=0.8\columnwidth]{diagrams/ff.pdf}
  \caption{A schematic of a D flip-flop implementation.}
  \label{fig:ff}
\end{figure}

Finally, we implemented the circuit shown in \Cref{fig:clock} to generate a 2-phase clock signal that we could use to operate our switched capacitor circuit. We achieved this by simply negating the clock to provide the second phase, and then using an SR latch and a series of buffers to ensure that the two phases never overlap.

\begin{figure}[!htb]
  \centering
  \includegraphics[width=\columnwidth]{diagrams/clock.pdf}
  \caption{A schematic of a 2-phase clock generator implementation.}
  \label{fig:clock}
\end{figure}

With these subcomponents in hand, we then implemented the complete ADC, as shown in \Cref{fig:adc}. We chose values of $C_1 = C_2 = \SI{0.5}{\pico\farad}$ as a starting point, making sure to match them in order to implement a discrete integrator. In addition, we used the reference voltages of $V_{ref+} = \SI{1.0}{\volt}$ and $V_{ref-} = \SI{0.2}{\volt}$ that were previously found to give the best SNDR.

We next performed a transient analysis of the circuit. As we did in our system level simulation, we used a $\SI{10}{\mega\hertz}$ clock and a $\SI{0.2}{\volt}$ amplitude, $\SI{20}{\kilo\hertz}$ sine wave input signal. We then ran the simulation for $\SI{2}{\milli\second}$, which allowed us to get $\SI{500}{\hertz}$ bins in our FFT. One issue we encountered was that the FFT expects uniform sampling while Analog Environment produces non-uniformly sampled outputs. To fix this, we set the \verb|strobeperiod| option of the transient simulation to $\SI{10}{\nano\second}$, which produces a uniformly sampled output and gives us 10 points per period of the sampling clock.

A portion of the results are shown in \Cref{fig:debug}. In this plot, we can see the integrator output for a given input signal. We see that in this case, with the signal being greater than analog ground, the negative error when the DAC outputs $\SI{1.0}{\volt}$ produces small steps downward, which eventually leads to crossing the analog ground line. At that point, the DAC output switches to $\SI{0.2}{\volt}$, and the larger positive error causes the integrator output to rise rapidly above the analog ground line. This illustrates how the ADC produces an output whose mean value is correlated with the value of the input signal.

\begin{figure}[!htb]
  \centering
  \includegraphics[width=0.8\columnwidth]{circuit/adc_cir_debug.eps}
  \caption{Transient response of the $\Delta \Sigma$ A/D converter with the input frequency set at $\SI{20}{\kilo\hertz}$ and the relay outputs set at $\SI{0.2}{\volt}/\SI{1.0}{\volt}$.}
  \label{fig:debug}
\end{figure}

We next processed the simulation data over the entire $\SI{2}{\milli\second}$ and obtained a SNDR of $\SI{55.7223}{\deci\bel}$ over our bandwidth of $\SI{38.22}{\kilo\hertz}$. We used $\SI{1}{\kilo\hertz}$ bins to match the system level simulation. A plot of the power spectral density we obtained is shown in \Cref{fig:cir_psd}.

\begin{figure}[!htb]
  \centering
  \includegraphics[width=0.8\columnwidth]{circuit/adc_cir_psd.eps}
  \caption{Power spectral density for the output of the $\Delta \Sigma$ A/D converter with the input frequency set at $\SI{20}{\kilo\hertz}$ and the relay outputs set at $\SI{0.2}{\volt}/\SI{1.0}{\volt}$.}
  \label{fig:cir_psd}
\end{figure}

Finally, we measured the power supply current and multiplied it by $V_{DD} = \SI{1.2}{\volt}$ in order to calculate the circuit's power dissipation. The results are shown in \Cref{fig:cir_power}. We can see that large spikes occur when the circuit's digital components are switching. Integrating the curve and dividing by the interval yields an average power dissipation of $\SI{195}{\micro\watt}$.

\begin{figure}[!htb]
  \centering
  \includegraphics[width=0.8\columnwidth]{circuit/adc_cir_power.eps}
  \caption{Power dissipation of the $\Delta \Sigma$ A/D converter with the input frequency set at $\SI{20}{\kilo\hertz}$ and the relay outputs set at $\SI{0.2}{\volt}/\SI{1.0}{\volt}$.}
  \label{fig:cir_power}
\end{figure}

\section{Analysis}

In terms of SNDR, the circuit level simulation produced a much lower result than the system level simulation. In fact, the value of $\SI{55.7223}{\deci\bel}$ obtained is below the spec of $\SI{60}{\deci\bel}$, and far below the system level simulation value of $\SI{76.69}{\deci\bel}$. This can be explained by several factors. As shown in \Cref{fig:sim_psd,fig:cir_psd}, the noise floor for the system level simulation is around $\SI{-140}{\deci\bel}$, while it is around $\SI{-90}{\deci\bel}$ for the circuit level simulation. This could be a result of additional quantization error, or a result of additional noise being introduced by the other components in the circuit. Since the system level simulation meets the specification, it seems more likely that the latter is the cause. However, more investigation is required for the final phase.

Several possible improvements that we will examine include the use of a 4-phase clock system and the use of bootstrapped transmission gates to reduce charge injection and improve linearity. In addition, we will look at second order effects to try to determine the principal causes of the reduced SNDR.

In terms of power dissipation, we were not given an explicit requirement. However, we can evaluate our implementation by comparing our average power dissipation of $\SI{195}{\micro\watt}$ to the Assignment 2 maximum of $\SI{200}{\micro\watt}$. This comparison shows that our power dissipation is quite low, given that our combined circuit consumes less power on average than the maximum we were given for a single operational amplifier. However, there are some improvements that can be made. In particular, the bias circuits of the operational amplifier and the comparator should be merged in order to save on power and silicon area. This will also improve the layout by reducing the need to lay out multiple current references.

\section{Conclusions}

In conclusion, we designed, simulated, and analyzed a first order, single bit, $\Delta \Sigma$ A/D converter using MATLAB and Cadence. To meet our specifications of a $\SI{38.22}{\kilo\hertz}$ bandwidth and a peak SNDR of at least $\SI{60}{\deci\bel}$, we used a sampling frequency of $\SI{10}{\mega\hertz}$. We then found that our peak SNDR occurs when the input frequency is $\SI{20}{\kilo\hertz}$ and our DAC output voltage levels are $\SI{0.2}{\volt}/\SI{1.0}{\volt}$. With these parameters, our system level simulation yielded a SNDR of $\SI{76.69}{\deci\bel}$, while our circuit level simulation yielded a SNDR of $\SI{55.7223}{\deci\bel}$. Given that our circuit level simulation SNDR does not meet the specifications, we also suggested several avenues of inquiry to pursue for the final phase of the project.

\bibliographystyle{IEEEtran}
\bibliography{IEEEabrv,references}

\end{document}