\definecolor{dkgreen}{rgb}{0,0.6,0}
\definecolor{gray}{rgb}{0.5,0.5,0.5}
\definecolor{mauve}{rgb}{0.58,0,0.82}

\lstset{frame=none,
  language=Python,
  aboveskip=5mm,
  belowskip=5mm,
  showstringspaces=false,
  columns=flexible,
  basicstyle={\linespread{0.8}\small\ttfamily},
  numbers=none,
  numberstyle=\tiny\color{gray},
  keywordstyle=\color{dkgreen},
  commentstyle=\color{blue},
  stringstyle=\color{mauve},
  breaklines=false,
  breakatwhitespace=true,
  tabsize=3
}

\chapter{Evaluation}

\section{Test and Development Environment}

Millipyde was tested using a desktop workstation with the following specifications:
\begin{itemize}
  \item Intel Xeon CPU E5-1620 @ 3.50GHz
  \item 32 GB memory
  \item Ubuntu 20.04.3 LTS Operating System
\end{itemize}

The workstation was equipped with two of the following GPU:
\begin{itemize}
  \item AMD Vega 10 XTX (gfx900) GPU
  \item Wavefront size: 64
  \item 64 CUs with 4 SIMDs per CU
  \item 4 Shader Engines
  \item 16KB L1 cache, 4096KB L2 cache
\end{itemize}

\quad Using the system BIOS, the workstation was configured to use large-BAR configuration to expose GPU local memory via PCIe memory BARs. This increased the BAR size to 64 bits, and was required to allow Millipyde to use DMA peer-to-peer memory transfers for any experiments that used both GPUs \cite{rocmrdma}.

\section{Single Function Benchmarks}

To test performance with individual functions, we used select Millipyde functions that had functionally-equivalent implementations in the non-GPU accelerated library scikit-image. These functions were run on identical images, and their performance was measured using Python's \verb|time.perf_counter| function. This measurement only included the time for the function itself to execute. It excludes the time for the image to be opened and saved. To ensure accuracy and consistency across both implementations, very pixel value in both the scikit-image output and the Millipyde output were compared to 4 decimal places. These benchmark were run on 12 copies of the same image in different sizes. They start at 500 pixels wide and go up to 6000 pixels wide, stepping by 500 pixels every time.

\subsection{Gamma Correction}

One of the most basic categories of parallel computations are element-wise calculations where each data value in an array undergoes a standalone operation to compute the respective output data value in the resulting array. One such example is Millipyde's \verb|adjust_gamma| function. Each pixel is transformed using the equation $V_{out} = A * V_{in} ^ \gamma$ where  $A$ is a constant multiplier called `gain', and $\gamma$ is the gamma adjustment value. This results in a brightness correction where gamma values less than one increase the overall brightness, and values greater than one decrease the brightness. The effects of this can be seen in Figure \ref{gammaExample}. For our experiments, we ran both the Millipyde function and the scikit-image equivalent with a gamma of 2 and gain of 1.

\begin{figure}[H]
\includegraphics[width=\textwidth]{figures/GammaExample.png}
\centering
\caption{The effects of gamma correction using a gamma value of 2.}
\label{gammaExample}
\end{figure}

\begin{table}[h!]
\centering
\begin{tabular}{ |c|c|c|c| }
\hline
image size & scikit-image time (s) & Millipyde time (s) & \% difference \\
\hline
500&0.0030651&0.0001561&1964 \\
1000&0.0105069&0.0001869&5622 \\
1500&0.0262136&0.0002929&8950 \\
2000&0.0403756&0.0004912&8220 \\
2500&0.0624683&0.0006293&9927 \\
3000&0.0890932&0.0009215&9668 \\
3500&0.1531737&0.0011845&12932 \\
4000&0.1878477&0.0015242&12324 \\
4500&0.2309318&0.0018687&12358 \\
5000&0.2741731&0.0022279&12306 \\
5500&0.3252361&0.0027007&12043 \\
6000&0.3819223&0.0031833&11998 \\
\hline
\end{tabular}
\caption{Timing results for gamma correction using scikit-image and Millipyde.}
\label{gammaTable}
\end{table}

\begin{figure}[H]
\includegraphics[width=\textwidth]{figures/GammaComparison.png}
\centering
\caption{Graphical representation of the time spent performing gamma correction on an image in scikit-image compared to its GPU-accelerated equivalent in Millipyde.}
\label{gammaComparison}
\end{figure}

\quad The results demonstrated a predictable decrease in runtime for the GPU-accelerated function. It ranged from around 20x faster for the 500-pixel image to more than 100x faster for larger images. These results when graphed followed a clean curve that closely mirrored the scikit-image equivalent as shown in Table \ref{gammaTable} and Figure \ref{gammaComparison}. Users can reliably expect the Millipyde gamma correction functionality to be faster than CPU-only equivalent functions.

\subsection{Greyscale}

 A slightly more complicated variation on a element-wise kernels happens when the dimensions of the input array are different than that of the output array. This can be seen in greyscale conversion functions which translates from a 3-dimensional RGB array of 8-bit integer values to a 2-dimensional array of floating point values. This computation is traditionally done using a weighted sum of the RGB components in an image to produce a single greyscale value between 1 and 0. Both Millipyde and scikit-image use the following weights in this calculation: $Y = 0.2125 R + 0.7154 G + 0.0721 B$ \cite{skirgb2grey}.
This makes it a good function to use in comparing scikit-image's functionality with the GPU-parallelized implementation in Millipyde.

\begin{figure}[hbtp]
\includegraphics[width=\textwidth]{figures/GreyscaleExample.png}
\centering
\caption{The effects of greyscaling an RGB image.}
\label{greyscaleExample}
\end{figure}

\begin{table}[H]
\centering
\begin{tabular}{ |c|c|c|c| }
\hline
image size & scikit-image time (s) & Millipyde time (s) & \% difference \\
\hline
500 & 0.0063839 & 0.0014073 & 454 \\
1000 & 0.0260336&0.0014537&1790 \\
1500&0.0535500&0.0052456&1021 \\
2000&0.0942788&0.0055702&1693 \\
2500&0.1466724&0.0054552&2689 \\
3000&0.2102404&0.0065436&3213 \\
3500&0.2964064&0.0091423&3242 \\
4000&0.3863234&0.0085288&4530 \\
4500&0.4974762&0.0089737&5544 \\
5000&0.6113694&0.0092912&6580 \\
5500&0.7201406&0.0092047&7824 \\
6000&0.8563366&0.0115259&7516 \\
\hline
\end{tabular}
\caption{Timing results for greyscaling images using scikit-image and Millipyde.}
\label{greyscaleTable}
\end{table}

\begin{figure}[H]
\includegraphics[width=\textwidth]{figures/greyscaleComparison.png}
\centering
\caption{Graphical representation of the time spent greyscaling an image in scikit-image compared to its GPU-accelerated equivalent in Millipyde.}
\label{greyscaleComparison}
\end{figure}

\quad As shown in Table \ref{greyscaleTable} and Figure \ref{greyscaleComparison}, these results differed surprisingly from those of the gamma correction function. On the surface, the kernels for both functions look startlingly similar -- both only perform a single element-wise computation on each input pixel. This difference in performance is therefore likely to do with the difference in shape between the input and output arrays. The resulting Millipyde graph almost resembles stair steps with significant jumps in performance between some image sizes, and stagnated performance between others. We were unable to find a clear cause based on studying the profile results of the function's runtime using the ROCm profiler 'rocprof'. Future experimentation is needed to understand this behavior, and to see if better/more predictable performance is possible.

\subsection{Gaussian Blur Convolution}

Another large category of parallel computation is the convolution, often referred to as the stencil computation, with applications ranging from signal processing, image processing, video processing, and more \cite{greenBook}. It involves a more mathematically-complex array computation where each output element is a weighted sum of N number of neighboring elements. The weights come from an input mask array called the kernel. For this example, we will use a Gaussian blur convolution which is commonly used in image processing and computer vision for applications such as edge detection \cite{gaussEdge}. For this computation, the kernel can be modeled as a 2D convolution using the kernel $G(x, y) = \frac{1}{2 \pi \sigma^2} e^{-\frac{x^2 + y^2}{2 \sigma^2}}$ where $\sigma$ represents standard deviation. This results in a symmetric kernel with stronger weights in the middle. The Gaussian blur is also separable, meaning the same result can be achieved by a separate pass of a 1-dimensional Gaussian kernel on each axis. The calculation for the 1-dimensional kernel is $G(x) = \frac{1}{\sqrt{2 \pi \sigma^2}} e^{-\frac{x^2}{2 \sigma^2}}$.

\begin{figure}[H]
\includegraphics[width=100mm]{figures/GaussianExample2.png}
\centering
\caption{The effects of a Gaussian Blur.}
\label{augmentExample}
\end{figure}

\quad Both Millipyde and scikit-image's gaussian functions use two passes of a 1-dimensional Gaussian kernel which makes it a great example for comparison \cite{scipyGaussianSrc}. Millipyde also exploits the shared memory system on the GPU for the Gaussian function. A block of the input image is first loaded into shared memory so that each thread block has access to these pixel values. This technique was previously studied in CUDA, and it was shown to produce excellent results for GPU runtime \cite{cudaConvolution}. For this test, we used a sigma value of 2 for both Millipyde and scikit-image's benchmarks. To match the way Millipyde handle's pixel edge values and kernel widths, the following parameters were used for the scikit-image comparison:
\verb|cval=0|, \verb|truncate=8|, \verb|mode="constant"|. This ensures equal kernel sizes, and caps pixels outside of the image to a value of 0.

\begin{table}[H]
\centering
\begin{tabular}{ |c|c|c|c| }
\hline
image size & scikit-image time (s) & Millipyde time (s) & \% difference \\
\hline
500&0.0105292&0.0002608&4037 \\
1000&0.0408449&0.0004123&9907 \\
1500&0.0902537&0.0006834&13207 \\
2000&0.1697805&0.0010704&15861 \\
2500&0.2523343&0.0015845&15925 \\
3000&0.3749286&0.0022314&16802 \\
3500&0.5563606&0.0029901&18607 \\
4000&0.7121980&0.0037107&19193 \\
4500&0.8735706&0.0047103&18546 \\
5000&1.0703184&0.0058559&18278 \\
5500&1.2964685&0.0069618&18623 \\
6000&1.5977196&0.0082425&19384 \\
\hline
\end{tabular}
\caption{Timing results for Gaussian blur using scikit-image and Millipyde.}
\label{gaussianTable}
\end{table}

\begin{figure}[H]
\includegraphics[width=\textwidth]{figures/gaussianComparison.png}
\centering
\caption{Graphical representation of the time spent performing a Gaussian blur on an image in scikit-image compared to its GPU-accelerated equivalent in Millipyde.}
\label{gaussianComparison}
\end{figure}

\quad This algorithm proved to be incredibly efficient in our Millipyde environment. As shown in Table \ref{gaussianTable} and Figure \ref{gaussianComparison}, our performance ranged from around 40x faster for smaller images up to just under 200x faster in larger images. Much of this performance increase is thanks to the shared memory blocks whose access times were much faster than always reading from global memory. This shows a lot of promise for future convolutional functions in Millipyde, and more experimentation should be done on utilizing shared memory in other image algorithms.

\section{Pipeline Benchmarks}

Since our Pipeline objects exploit multiple forms of parallelism, it is worth studying the performance of various Pipeline configurations. Four our tests, we bench-marked the Pipelines using multiple copies of the same image for our list of inputs. Each image was 3500 pixels wide by 4666 pixels tall. The input list sizes ranged from 1 to 100. From there, four different configurations were run on each group of inputs. The first configuration, labeled `Control' in each respective chart, executed every Operation sequentially in a Python loop rather than using a Pipeline object at all. The test labeled `Single GPU' ran a single Pipeline that was bound to one given device so that it would not use all GPUs available on the system. The 3rd test, labeled `Dual GPUs' was not bound to a single device, so it was allowed to use both available devices for scheduling. The final test, labeled `Connected GPUs' split the work in half so that each of two Pipeline objects had half of the Operations to be completed. These two Pipelines were each bound to separate devices in the system and connected so that the outputs that completed on the first Pipeline would be transferred over to the second Pipeline to complete the remainder of the Operations.

\subsection{Pipelines with 5 Operations}

Our first experiment represented a standard small set of operations that may occur in a variety of data manipulation programs. Each iteration of the experiment used a total of five image transformation Operations. These included a Gaussian blur, gamma adjustment, horizontal flip, 45-degree rotation, and a greyscale conversion. For the connected Pipeline, these Operations were split in half so that the Gaussian blur, gamma adjustment, and flip were performed inside of the first Pipeline, and the rotation and greyscale conversion were performed on the second Pipeline.

\begin{table}[H]
\centering
\begin{tabular}{ |c|c|c|c|c| }
\hline
Number of inputs & Control & Single GPU & Dual GPUs & Connected GPUs \\
\hline
1&	0.0244971&	0.0269347&	0.2622321&	0.0357248 \\
5&	0.1214234&	0.1228287&	0.1046105&	0.1197764\\
10&	0.2419561&	0.2456722&	0.1764512&	0.2654271\\
20&	0.4817377&	0.4817598&	0.3661450&	0.4735247\\
50&	1.2170164&	1.3320313&	0.8369642&	1.1232160\\
75&	1.9473421&	2.1429327&	1.2548219&	1.8599911\\
100&	2.9957998&	2.9042716&	1.6321600&	2.5250520\\
\hline
\end{tabular}
\caption{Timing results for running a short 5-Operation pipeline}
\label{pipelineTable1}
\end{table}

\begin{figure}[H]
\includegraphics[width=\textwidth]{figures/pipelineChart1.png}
\centering
\caption{Graphical results for running a short 5-Operation pipeline.}
\label{pipelineChart1}
\end{figure}

\quad The results of this experiment are shown in Table \ref{pipelineTable1} and Figure \ref{pipelineChart1}. Predictably, the Pipeline that was able to use two GPUs performed the best. Its runtime was often half of the single-GPU Pipeline and the control experiment. The two connected Pipelines performed notably worse than the standard Pipeline despite the fact that both Pipelines should theoretically be splitting work across two GPUs. This is easily explained by the overhead from the transfer itself. When data is transferred from one Pipeline to the next, it has to be offloaded from the first device's work pool into the second device's work pool, and many small costs are incurred from operating managing both Pipeline objects simultaneously.

\quad The single GPU pipeline showed disappointing performance that was often comparable to the control. Since the operations used were image-based transformations, little to no work was done on the CPU besides for immediate GPU kernel launches. This means that the work pools associated with each device were not fully utilized, and often only added scheduling overhead to the experiment. More work can be done in the future to make Millipyde's Pipeline scheduling more adaptable in these situations.

\subsection{CPU-bound Pipelines with 5 Operations}

To expose the effects of having the work pools associated with each device, we ran the same Pipeline tests as the previous experiment with the addition of simulated CPU work. At the beginning of each of the five Operations, 0.002 seconds of sleep was performed on the CPU. This was meant to represent time that could be incurred with future functions that require CPU-based kernel preparation prior to the kernel launch itself. Ideally, these experiments will be re-run and analyzed in the future when a function like this is added to Millipyde.

\begin{table}[H]
\centering
\begin{tabular}{ |c|c|c|c|c| }
\hline
Number of inputs & Control & Single GPU & Dual GPUs & Connected GPUs \\
\hline
1&	0.0373815&	0.0361518&	0.0370423&	0.0470410 \\
5&	0.1813750&	0.1406366&	0.1128026&	0.1417612 \\
10&	0.3653580&	0.2656119&	0.1972203&	0.2660218 \\
20&	0.7243550&	0.5269443&	0.3712618&	0.5205974 \\
50&	1.9165390&	1.6008312&	0.9192495&	1.2318530 \\
75&	3.0087808&	2.4173886&	1.3291795&	1.8750082 \\
100&	4.4714676&	3.3045429&	1.7332253&	2.5254060 \\
\hline
\end{tabular}
\caption{Timing results for running a 5-Operation pipeline with simulated CPU-bound pre-processing work.}
\label{pipelineTable2}
\end{table}

\begin{figure}[H]
\includegraphics[width=\textwidth]{figures/pipelineChart3.png}
\centering
\caption{Timing results for running a 5-Operation pipeline with simulated CPU-bound pre-processing work.}
\label{pipelineChart2}
\end{figure}

As shown in Table \ref{pipelineTable2} and Figure \ref{pipelineChart2}, these results were more dramatic than the prior five-operation Pipeline experiment. The single-GPU pipeline was able to more predictably outperform the control experiment. We still saw strong performance gains from both of the two-GPU configurations. Despite this, the connected Pipelines still under-performed due to the cost of transferring the work from one Pipeline to the next.

\subsection{Pipelines with 20 Operations}

Lastly, an experiment was performed to see the effects of having longer Pipelines with many Operations. Each Pipeline in this test performed the same 20 Operations: seven Gaussian blurs, four gamma adjustments, two horizontal flips, six 45-degree rotations, and one greyscale conversion. This test is meant to represent more intensive GPU-operations that should further outweigh the costs incurred by the Pipeline's own overhead. For the two connected Pipelines, this work was split down the middle so that each Pipeline performed 10 Operations.

\begin{table}[H]
\centering
\begin{tabular}{ |c|c|c|c|c| }
\hline
Number of inputs & Control & Single GPU & Dual GPUs & Connected GPUs \\
\hline
1&	0.1210601&	0.1236825&	0.1229053&	0.1325121\\
5&	0.6136912&	0.6081436&	0.4897915&	0.5424913\\
10&	1.2329561&	1.2234442&	0.7826877&	0.8564503\\
20&	3.2071803&	2.9295116&	1.6075147&	1.6007919\\
50&	14.1225833&	17.9376053&	4.3081120&	4.4900887\\
75&	30.5186737&	30.8800205&	8.4228492&	8.4681248\\
100&	48.8984402&	46.5367387&	14.6403225&	14.5740577\\
\hline
\end{tabular}
\caption{Timing results for running a long 20-Operation Pipeline.}
\label{pipelineTable3}
\end{table}

\begin{figure}[H]
\includegraphics[width=\textwidth]{figures/pipelineChart2.png}
\centering
\caption{Graphical results for running a long 20-Operation Pipeline.}
\label{pipelineChart3}
\end{figure}

\quad As seen in Table \ref{pipelineTable3} and Figure \ref{pipelineChart3}, this experiment produced the most dramatic results. Now that the costs of Pipeline transfer were outweighed by the costs of the GPU computations, both dual-GPU configurations were able to perform roughly the same -- almost always within 0.1 seconds of each other. Surprisingly, both the control and the single-GPU pipeline performed poorly with more than double the runtime of their dual-GPU equivalents.


\title{\bf Detectors}

\section{Basics \& Nomenclature}

Modern astronomical detectors come in several varieties depending on
wavelength and use case. The traditional detector introduced in the
1800s, the photographic plate, is too inefficient ($\sim$ 5\%) and too
inconvenient to convert into digital form to be of use today, though
it continued to be used until about 2000. Here, we will not review the
physics behind modern detectors except insofar as necessary to explain
how to interpret data taken with them.

In the optical, out to about a micron, the most common detector in use
is the {\it charge-coupled device} (CCD). Out to about 28 $\mu$m, the
most common detectors are infrared detector arrays. Although CCD and
infrared arrays are different technologies, both are
semiconductor-based detectors that detect photons of energy higher
than their band gap between their bound valence electron energies and
their conduction band. Silicon is the best-developed CCD technology
and has a band gap of 1.1 $\mu$m.  Germanium CCDs could in principle
extend this sensitivity to $1.8$ $\mu$m, but their technology is not
cost-effective today. HgCdTe (``MER-CAH-TEL'') infrared detectors have
a band gap of that can be designed anywhere between 0.4--12
$\mu$m. Si:As detectors can extend to 28 $\mu$m (and are for example
used in the two redder WISE bands).

Each pixel of a CCD detects individual photons that hit it, each of
which usually contributes one electron to the overall signal. For most
hardware, the charge is {\it read out} at amplifiers along the side of
the CCD, by transferring the charge from pixel to pixel to the edge of
the CCD. At the last pixel the charge is converted to a digital signal
by the amplifiers. CCDs may be readout by various numbers of
amplifiers. Because of the finite temperature of the devices, some
number of electrons are released even when there are no photons
entering the device; this phenomenon is known as the {\it dark
current}. To reduce the dark current, CCD detectors need to be cooled;
variations in temperature can cause variations in the dark current.

Infrared detector arrays operate similarly, but the infrared-sensitive
materials cannot today be used to support the circuits necessary for
CCD operation. Instead, the infrared sensitive material is bonded to a
silicon detector on which the read out occurs. These detectors are not
CCDs but multiplexers, which are read directly out on each pixel. The
charge can be read out nondestructively, allowing many reads on the
same pixel. However, the read out is intrinsically noisier.

From this basic understanding, there results an interpretation of the
digital numbers reported by a semiconductor device:
\begin{equation}
\mathrm{DN} =
\frac{n_e}{\mathrm{Gain~}} =
\frac{n_p + \mathrm{Dark}}
{\mathrm{Gain~}}
\end{equation}
where $n_e$ is the number of electrons recorded by the device, $n_p$
is the number of photons actually detected, i.e. which are converted
to electrons (which will include all background photon sources). Gain
represents the {\it gain}, the number of electrons per DN reported by
the electronics. Dark represents the dark current in units of
electrons.

The noise in the DN is due primarily to two sources: Poisson noise
around the mean $n_e$ and {\it read noise} associated with the
electronics. The read noise does not usually depend on the
signal. Therefore:
\begin{equation}
\sigma_{\rm DN}^2 = \frac{n_e}{\mathrm{Gain}^2} + \mathrm{Read~Noise}^2
\end{equation}
The Poisson noise includes the ``object'' signal, the background
signal, and the dark current contributions to the expected number of
electrons $n_e$.

Over their usuable dynamic range in number of electrons, CCD devices
are remarkably but not perfectly linear.  CCD devices are limited in
their dynamic range by the number of electrons each pixel can
store. If the number of electron approaches or exceed this limit,
those electrons typically bleed to neighboring pixels; since the
electronics is not isotropic, they typically bleed along the same
direction the device is read out. They also may be limited by the
dynamic range of their analog-to-digital converter, which is often
16-bits.

CCDs are also sensitive to cosmic rays and these will release
electrons that contribute to the signal. A characteristic feature of a
cosmic ray is that it will be a sharper feature than the atmosphere
and optics allow. The exact nature of the cosmic ray distribution
depends on altitude (with more reaching higher altitude and space
detectors) and orientation of the detector relative to vertical (since
underneath the atmosphere the cosmic rays will be directed
preferentially downwards).

In the ultraviolet, other detectors are still in use. For example,
GALEX uses a position sensitive proportional counter. These devices
are essentially a crossed grid of wires under voltage inside a chamber
filled with an inert gas that can be ionized by UV photons. When a UV
photon causes a charge track, the voltage change is detected and
recorded. Unlike CCDs, these detectors can detect individual photons.

In the X-rays, CCD detectors are now typically used. CCDs lose
sensitivity in the UV due to absorption on their surfaces, but at
energies $> 120$ eV ($<100$ \AA) they become sensitive again. Unlike
in the UV and optical, X-ray photons can release many electrons. This
fact, and that the photons much rarer, allows X-ray CCDs to be energy
sensitive. At high count rates, they become complicated to analyze due
to latency effects; for some period of time after a photon is detected
subsequent photons cannot be.

In the low frequency radio, antennae of various sorts are generally
used as detectors, and the signal is read out as a changing
voltage. Radio detectors therefore preserve the phase information in
the electromagnetic signal, unlike higher frequency instruments. This
allows interoferometry on very wide baselines to achieve very high
angular resolution (low $\lambda/D$). In the higher frequency radio
(sub-mm), superconductor-insulator-superconductor (SIS) devices are
used, which are photon-sensing and so do not preserve the phase.  In
both regimes, feedhorns in the focal plane of the telescope are often
used to match the focal plane to the antennae. In some configurations,
antennae are used in arrays without any focusing element, and images
are reconstructed through cross-correlating in the resulting
signals. Noise in radio instruments is often dominated by thermal
noise due to the finite temperature of the receiver electronics.

No detectors capture all of the photons hitting each pixel. The
fraction of photons detected is known as the {\it quantum efficiency}
of the device. It is a function of wavelength (most notably near the
band gap). CCD and infrared devices have quantum efficiencies that are
typically quite high, e.g. up to 95\%.

\section{Commentary}

CCDs only became widely used in astronomy until the mid-1990s. Prior
to that time most imaging and spectroscopy was performed with
photographic plates. Photographic plates have only $\sim 5$\% quantum
efficiency typically, and obviously are harder to translate into
digital results. However, they remained competitive for wide field
imaging surprisingly long, because the high cost of CCDs kept them
from being used to completely cover the focal planes of wide field
telescopes.

\section{Key References}

\begin{itemize}
  \item
    {\it Design and Construction of Large Telescopes},
      \citet{bely03a}
  \item
    {\it Astrophysical Techniques},
      \citet{kitchin09a}
\end{itemize}

\section{Order-of-magnitude Exercises}

\begin{enumerate}
\item Modern CCDs have read noise can be as low as 2 electrons,
    whereas infrared detectors often have read noise of $\sim 20$
    electrons. How many independent reads would have to be performed
    of an infrared detector to beat down the read noise to compete
    with CCD read noise?
\item The cosmic X-ray background at 1 keV is about $3\times 10^{-11}$
    W m$^{-2}$ sr$^{-1}$ (\citealt{fabian92a}). The Chandra X-ray
    telescope often takes exposures of $\sim 10^4$ s, and has a
    resolution of order an arcsec. What is the expected number of
    background photons within the a FWHM of a source that are
    contributed by this background for such an exposure?
\item What is the highest angular resolution that can be achieved by a
    ground-based interferometric radio experiment at $\sim 1.6$ GHz?

\begin{answer}[Author: Trey Jensen]
In interferometry, one can use aperture synthesis to create a larger
baseline aperture than traditional telescopes by placing an array of
multiple telescopes. The distance between any pair of two telescopes,
the baseline, sets the diffraction limit for images produced by
correlating the phase signal between the telescopes.  The diffraction
limit is $\theta\sim \frac{\lambda}{D}$ and the largest possible
distance we can separate two ground-based telescopes is of order the
diameter of the Earth $D\sim 10^7$ meters. For 1.6 GHz light, we have
an angular resolution of
\begin{align*}
    \theta\sim \frac{3\times 10^8}{(1.6\times 10^9)(10^7)}\sim 2\times 10^{-8} \textrm{ rad}\sim 4\times 10^{-3} \textrm{ arcseconds}.
\end{align*}
\end{answer}
\end{enumerate}

\section{Analytic Exercises}

\begin{enumerate}
\item The detected electrons contributing to the noise include the
object, background, and dark current signal. Write the fractional
error in the object counts. Separate the terms involving the other
sources of noise. For low object signal, how does the fractional error
depend on signal? For high object signal, how does the fractional
error depend on signal?

\begin{answer}[Author: Dou Liu]
The noise $N$ includes the Poisson noise in the object, sky, dark
counts ($n_\mathrm{obj}$, $n_\mathrm{sky}$, and $n_\mathrm{dark}$),
added in quadrature with each other and the read noise
$N_\mathrm{read}$:
\begin{equation}
\label{ }
N
= \sqrt{N_\mathrm{obj}^2+N_\mathrm{sky}^2+N_\mathrm{dark}^2+N_\mathrm{read}^2}=\sqrt{n_\mathrm{obj}+n_\mathrm{sky}+n_\mathrm{dark}+N_\mathrm{read}^2}
\end{equation}
The fractional error in the object counts is:
\begin{equation}
\label{ }
\frac{N}{n_\mathrm{obj}}=\frac{\sqrt{n_\mathrm{obj}+n_\mathrm{sky}+n_\mathrm{dark}+N_\mathrm{read}^2}}{n_\mathrm{obj}}
= \sqrt{\frac{1}{n_\mathrm{obj}}\left(1+
\frac{n_\mathrm{sky}+n_\mathrm{dark}+N_\mathrm{read}^2}{n_\mathrm{obj}}\right)}
\end{equation}
Defining:
\begin{equation}
\sigma_\mathrm{other}^2 =
n_\mathrm{sky}+n_\mathrm{dark}+N_\mathrm{read}^2,
\end{equation}
we find that if $n_\mathrm{obj}\gg \sigma_\mathrm{other}^2$ (the
object-dominated regime), then
\begin{equation}
\frac{N}{n_\mathrm{obj}} \approx \frac{1}{\sqrt{n_\mathrm{obj}}}.
\end{equation}
If $n_\mathrm{obj}\ll \sigma_\mathrm{other}^2$ (the
background-dominated regime), then
\begin{equation}
\frac{N}{n_\mathrm{obj}} \approx \frac{\sigma_\mathrm{other}}{n_\mathrm{obj}}.
\end{equation}
Therefore, for the faintest fluxes (below the background) the
fractional errors are a stronger function of the flux than they are at
brighter fluxes (above the background).
\end{answer}
\item The noise in a data set is determined by the Poisson noise in
the expectation value of the signal. However, observationally we often
only have access to one realization of the signal, and often the
quoted errors are based on the Poisson noise in that signal.  If we
take multiple observations and combine them together weighting by the
inverse variance estimated in this way, it leads to a bias. Ignoring
any background contribution to the noise, estimate this bias as a
function of $n_e$.
\end{enumerate}

\section{Numerics and Data Exercises}

\begin{enumerate}
\item CCD images from SDSS are available
as \href{https://www.sdss.org/dr14/imaging/images/#corr}{corrected
frames}. Those images are calibrated in physical units, and have the
sky background subtracted. Read the documentation there, including the
data model. Download a corrected frame in the $r$-band. Use the
formulae in the documentation to translate the image into uncalibrated
DN units with the sky background reinstated (the files have all the
information necessary to do that). Is it all integers?  Use the result
to estimate the noise per pixel in a region without stars or
galaxies. Can you independently estimate that noise from the
distribution of flux values themselves, and do the estimates agree?
\end{enumerate}

\bibliographystyle{apj}
\bibliography{exex}


\documentclass{memoir}
\usepackage{pgfplots}
\usepackage{multicol}
\usepackage{tikz}
\usepackage{amsmath,amssymb}
\usepackage{listings}
\usepackage{color}
\usepackage{caption}
\usepackage[babel=hyphen,backend=biber,citestyle=authoryear]{biblatex}

\addbibresource{PPSources.bib}

\title{ML for HS}
\author{Aidan Sciortino}
\date{March 2017}

%---Custom Colors---%
\definecolor{light-gray}{gray}{0.85}

%---Code Style Template---%
\lstdefinestyle{customj}{
    backgroundcolor=\color{light-gray},
    belowcaptionskip=1\baselineskip,
    breaklines=true,
    frame=L,
    xleftmargin=\parindent,
    language=Java,
    showstringspaces=false,
    basicstyle=\ttfamily,
    keywordstyle=\bfseries\color{green!40!black},
    commentstyle=\itshape\color{purple!40!black},
    identifierstyle=\color{blue},
    stringstyle=\color{orange},
    numbers = left,
}
\lstset{style=customj}

%--Custom Graph Style--%
\pgfplotsset{plot/.style={height=5cm,width=0.5\textwidth,axis lines= middle, axis line style = <->, >=latex,tick align = center,legend style={legend pos = south east, font = \tiny}}}

%---Custom Math Commands---%
\newcommand{\ddx}{ \frac{d}{dx} }
\newcommand{\e}{\text{e}}

%---Neural Network Diagram Shapes---%
\tikzstyle{neuron} = [circle, minimum width= 0.5cm, draw = black]
\tikzstyle{arrow} = [->,>=stealth]
\usetikzlibrary{shapes.geometric, arrows}

\begin{document}
\pagenumbering{roman}

\tableofcontents
\listoffigures

\chapter*{Introduction}
\markboth{Introduction}{Introduction}
  This book was written in response to the lack of resources regarding machine learning designed for those without math backgrounds, or those without the knowledge of the mathematics ``needed'' to fully learn it.

  I have found that the machine learning tutorials, papers, and information online divide themselves into two camps, either;
  \begin{itemize}
    \item \textbf{Over Simplified:} These resources simply apply pre-existing algorithms to data sets, explaining the code behind it but not the mathematics. These are great for people in the industry who simply need an algorithm, but from the perspective of someone trying to learn are very frustrating because they gloss over a lot of material.
    \item \textbf{Over Complicated: } These resources assume a CS or Mathematics background, and therefore don't explain the semantics of the math behind the algorithms. I'm sure that these are great for scientists to read, but from the perspective of a high schooler they are dense, difficult to read, and frustrating to try and understand.
\end{itemize}

  My frustration with these resources led me bite the bullet and learn some complex math in order to better understand both types of resource. However, I decided that I didn't want other people to have to go through what I did, so writing a book seemed logical.

  \subsection{Who This Book is For:}
    This book is designed for students who haven't taken courses in higher math, programmers who want to get into the mechanics behind machine learning but don't have the math knowledge, and anyone who wants to learn more about the fascinating field that is machine learning.

  \subsection{Who This Book (probably) isn't For:}
    People who already have backgrounds in mathematics involving linear algebra, multivariate calculus, advanced statistics, etc. I go into lots of detail about the mathematics behind machine learning that would probably prove boring to most of you.

  \subsection{License}
    Throughout the course of researching, writing, typesetting and coding this book, I have made use of resources that were Creative Commons and Open Source. As a result, you are free to reuse any code examples from this book and sample the text in any way, as long as it isn't for profit. Attribution would be nice but is not required. An email including a link to your project would also be awesome, but again, not required.

    \vspace{0.5em}

In this book I have endeavored to explain every bit of code, every line of math, and every letter of vocabulary. I hope you find it informative, helpful, and interesting.
  \begin{flushright}\emph{-Aidan Sciortino, 2017}\end{flushright}

\chapter{Beginning Neural Networks}
\pagenumbering{arabic}
In this chapter we will build a single layer feed forward neural network using some simple math concepts and an older model called the perceptron. This all may sound complicating at first, but we will break it down into simple parts that can be easily digested.
\section{Objective}
This chapter is designed to serve as an introduction to the mathematics, models, history and logic behind neural networks. The chapter is divided into four sections; one serving as an introduction to the theory behind machine learning, another dedicated to the mathematics needed for this chapter, another explaining some history behind the algorithms used in this chapter, and a last dedicated to the actual code with accompanying explanations.

The chapter is not necessarily designed to be read in order. Readers who believe that they can pick up most of the mathematics and algorithms from reading the code section could skip to there, and then perhaps look back at the mathematics and algorithms sections.
\clearpage

\section{Theory}
Machine learning operates on the principle of modifying input data to return certain output data. For example, we might have the following input data:
\[
	\begin{bmatrix}
		0 & 1 & 0 \\
		1 & 0 & 0 \\
		1 & 1 & 0 \\
		0 & 1 & 1 \\
	\end{bmatrix}
\]
We expect that each row of this data will return a certain value when fed through the network; For our dataset above this data is:
\[
	\begin{bmatrix}
		0 \\
		1 \\
		1 \\
		0 \\
	\end{bmatrix}
\]
The observant reader may realize that the output data simply corresponds with the first digit of the input data. However, the neural network does not start knowing this. Instead, we begin by training it based on this data.

To predict results based on this data we apply certain \emph{weights} to each of the inputs. Weights are simply numbers that we multiply the input data by to get an output value. In our case, the ideal weights would be one, zero, and zero, because the first digit of input corresponds directly with the output, and the other two digits of input don't matter. However, we don't begin by giving our network these weights, instead we make it guess what they are and figure them out itself.

\section{Mathematics}
Machine Learning is based primarily on three fields of mathematics; Multivariate Calculus, Linear Algebra, and Statistics. Don't worry if these terms sound confusing at first. This book will break these complex fields of mathematics down into digestible chunks that anyone with basic high school math can understand.

\subsection{The Sigmoid Function}
Most machine learning algorithms require some sort of function in order to squash values towards one or zero. Some very complex functions have been developed in order to do this in the most efficient way possible, including some that are actually changed in accordance with the output of the algorithm itself.

For this chapter complex functions such as these are not needed. Instead, for demonstration purposes, I will make use of a function known as the sigmoid function, \(\sigma (x)\). Represented by the formula and graph below, this function serves to `squash' very negative numbers towards zero and very positive numbers towards one. This will prove helpful in our usecase because the weights can turn out such that they modify the output very high. This function's squashing feature can be used to get an output between zero and one despite the high value of these numbers.

\begin{figure}[h]
	\centering
	\begin{minipage}{0.5\textwidth}
		\centering
		\[\sigma (t) = \frac{1}{1+\e^{-t}}\]
	\end{minipage}\hfill
	\begin{minipage}{0.5\textwidth}
		\centering
		\begin{tikzpicture}
			\begin{axis}[plot, width=\textwidth,
				xmin=-5,xmax=5,ymin=-0.125,ymax=1.125,
				ytick={0,0.5,1},xtick = {-4,0,4},
				xticklabels={$-$,0,$+$}]
				\addplot [mark = , domain=-4.5:4.5, <->, >=latex] {1/(1+exp(-1*x))};
				\addplot [mark = , domain=-5:5, <->, >=latex, dashed] {1};
				\addplot [mark = , domain=-5:5, <->, >=latex, dashed] {0};
			\end{axis}
		\end{tikzpicture}
    \caption{Graph of $\sigma (x)$}
	\end{minipage}
\end{figure}
\subsection{The Derivative}
One of the first concepts that you are taught in pre-algebra and algebra classes is the graphing of linear equations in the \( y = mx + b\) form. As you may remember a linear equation can be broken down into parts; they $y$ representing the output value of the function, the $m$ representing the slope of the produced line, $x$ representing the input value, and $b$ representing the initial value of the function.

The derivative of a function represents the slope of the function, the $m$ value of a linear equation. By providing simple rules to calculate an equation representing the slope of both linear as well as more complex equations at any point in their domain, the derivative is very useful across higher math.

The graph of $y=x^2$ and it's derivative are shown below. As you may observe, the derivative is equal to zero at the maximums and minimums of $y=x^2$ due to the fact that the slope is equal to zero at the maximum and minimum.

\begin{figure}[h]
	\centering
	\begin{tikzpicture}
		\begin{axis}[plot,xmin=-5,xmax=5,ymin=-5,ymax=5]
			\addplot[mark = ,domain=-2:2, <->, >=latex]{x^2 + 1};
			\addlegendentry{$y=x^2$}
			\addplot[mark= , domain=-2.5:2.5, <->, >=latex, dashed]{2*x};
			\addlegendentry{$\ddx(x^2)$}
		\end{axis}
	\end{tikzpicture}
  \caption{Graph of $y=x^2$ and it's Derivative}
\end{figure}

The derivative is notated in many ways, the most common being $\ddx$. Be careful that you don't confuse the fraction-like syntax for division. While in some examples it \emph{can} behave as addition, in our case it does not. Another way that the derivative is often notated is by making a function prime. For example, if we have a function $f(x)$, then $f'(x)$ would represent it's respective derivative.

For this chapter you need to know the following derivative rules. If you already know them you can safely skip to the end of this section. \\

\begin{center}
	\vspace{-1cm}
	\begin{minipage}[b]{0.31\textwidth}
		\[\ddx(\e^x) = e^xdx\]
	\end{minipage}\hfill
	\begin{minipage}[b]{0.37\textwidth}
		\[\ddx(f(g(x))) = g'(x)f'(g(x))\]
	\end{minipage}\hfill
	\begin{minipage}[b]{0.31\textwidth}
		\[\ddx(x^a) = ax^{a-1}dx\]
	\end{minipage}
\end{center}

I will not go into the details of proving these derivatives. Very good explanations are available the Internet from reputable sources such as Khan Academy\footnote{\texttt{https://www.khanacademy.org/math/ap-calculus-ab/basic-differentiation-ab}}.
\subsection{Matrices and Vectors}
You may have noticed that the input data used in the first example looked an awful lot like a \emph{matrix} of numbers. A matrix is a grid of numbers that describes something. In our case, each row of the matrix described the input values that we put into our neural network. Matrices (plural for matrix) are notated by capital letters accompanied by a subscript describing the size of the matrix. For example,

\[
	A_{1\times2} =
	\begin{bmatrix}
		1 \\
		0 \\
	\end{bmatrix}
\]

As you can see, the subscript describes the size of the matrix much as you would notate a coordinate on the coordinate plane. It describes the number of columns (the $x$ dimension in coordinate terms) and then the number of rows (the $y$ dimension, in coordinate terms). Therefore, even if $A$ was not accompanied by a matrix showing actual numbers, the reader could understand that $A_{1\times2}$ Represents a matrix with 1 column and 2 rows.

\subsection{The Dot Product}

Machine learning is composed almost entirely of matrices and their close cousin, the vector (best described as a $1\times n$ matrix). A dot product can be used to multiply two  vectors together. For example;

\[
	\begin{bmatrix}
		a_1    \\
		a_2    \\
		\vdots \\
		a_n
	\end{bmatrix}
	\cdot
	\begin{bmatrix}
		b_1    \\
		b_2    \\
		\vdots \\
		b_n
	\end{bmatrix}
	= a_1b_1 + a_2b_2+ \dots +a_nb_n
\] %TODO Cite Khan Academy
\subsection{The Transpose}
The Transpose is another linear algebra function that manipulates matrices\footnote{Plural for Matrix}. However, instead of multiplying, dividing, adding, or subtracting, the Transpose serves to flip a matrix on it's side. For example, if $A$ represents an $m \times n$ matrix the transpose of $A$ is an $n \times m$ matrix. The transpose of a matrix is represented by superscript $\intercal$ on the variable representing the matrix. In our previous example, if $A$ is an $m \times n$ matrix then $A^\intercal$ is an $n \times m$ matrix.
\[
	\begin{align}
		A =
		\begin{bmatrix}
		_1a_1  & _2a_1  & \dots  & _ma_1  \\
		_1a_2  & _2a_2  & \dots  & _ma_2  \\
		\vdots & \vdots & \ddots & \vdots \\
		_1a_n  & _2a_n  & \dots  & _ma_n  \\
		\end{bmatrix} \\
		A^\intercal =
		\begin{bmatrix}
		_1a_1  & _1a_2  & \dots  & _1a_n  \\
		_2a_1  & _2a_2  & \dots  & _2a_n  \\
		\vdots & \vdots & \ddots & \vdots \\
		_ma_1  & _ma_2  & \dots  & _ma_n  \\
		\end{bmatrix}
	\end{align}\]
	\section{Algorithms}
	The algorithm used in the neural network built in this chapter is called the perceptron. Invented in 1958 at the Cornell Aeronautical Laboratory by a psychologist named Frank Rosenblatt\footcite{perceptronSrc}.
	First published in \emph{Psychological Review} under the title ``The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain'', the idea, although promising at first, proved unsuitable for classifying more than simple problems. This led to the study of Neural Networks being abandoned for several years, until the discovery of the multi-layer perceptron and it's capabilities (see chapter two for further info).

	The perceptron operates by taking the dot product of a set of inputs and a set of weights, and then running the result through a sigmoid function. In this case the weights are simply a set of numbers that are tuned to return a certain output when given certain inputs.

	The way this algorithm learns is called \emph{supervised learning}. This means that we feed it training data where each piece of input data is also labeled with a desired output data point. This differs from other methods of learning in that someone has to go through and label all of the data used to train the network before it can be used.

	The network is trained based on the following steps. All that the training process does is calculate the proper weights to apply to the input data.

	\begin{enumerate}
		\item Initialize Weights to Random Number\footcite{perceptron}.
		\item Pick Learning Weight between zero and one.
		\item For many iterations (More iterations returns more accurate weights but is also slower)%\footfullcite{perceptron}
		      \begin{enumerate}
		      	\item Make a prediction based on the training input and the weights
		      	\item Calculate the difference between the output value of the algorithm and the output value attached to the input data. This is called the error.
		      	\item Calculate an adjustment value for the weights by multiplying the error by the derivative of the sigmoid function for these data points
		      	\item Adjust the weights according to the product of the inputs and the adjustment value
		      \end{enumerate}
	\end{enumerate}

	The neural network that we build in this chapter is known as a single layer feed forward neural network with back-propagation. That's a mouthful at first, so let's break it down into parts to better understand it.

	\begin{itemize}
		\item \textbf{Single Layer:} This neural network consists of one layer of inputs that do calculation
		\item \textbf{Feed Forward:} Data is fed forward through the network and modified by weights
		\item \textbf{Neural Network:} Data is manipulated by a network of neurons.
		\item \textbf{Back-propagation:} The weights that manipulate the data are modified by looking at the output data and comparing it to the desired output.
	\end{itemize}

	For the more visually minded, the diagram below represents the neural network where $i_1, i_2, i_3$ are inputs, $w_1, w_2, w_3$ are weights that modify the inputs, and $o_1$ is the output.
	\begin{figure}[h]
		\centering
		\begin{tikzpicture}[node distance = 1.5cm]
			%--Input Neurons--%
			\node(in1)[neuron]{$i_1$};
			\node(in2)[neuron,right of=in1]{$i_2$};
			\node(in3)[neuron,right of=in2]{$i_3$};

			%--Ouput Neuron--%
			\node(out1)[neuron,above of=in2]{$o_1$};

			%--Synapses--%
			\draw[arrow] (in1) -- node[anchor = east]{$w_1$} (out1);
			\draw[arrow] (in2) -- node[anchor = east]{$w_2$} (out1);
			\draw[arrow] (in3) -- node[anchor = west]{$w_3$} (out1);

			%--Other Labels--%
			\node(lay1)[right of=in3]{Input Layer};
			\node(lay2)[above of=lay1]{Output Layer};
		\end{tikzpicture}
    \caption{A Diagram Representing the Layout of the Neural Network}

	\end{figure}
	\clearpage
	\section{Code}
	The code for this book is written in Java. However, the sources used have code written in python. I chose Java because I find that python automates things too much to achieve a full understanding of what is actually going on behind the scenes.

	I don't start off writing this Neural Net by diving straight in and writing the perceptron itself. Instead I start off by writing some helper methods to handle the linear algebra, sigmoid curves, etc. If you want to see a full code listing see the end of this section.

	\subsection{Dot Product}
	To begin with I wrote a function that calculates the dot product of two $1\times n$ vectors. The function accepts two single dimensional arrays and returns the dot product of them. The code is as follows:
	\begin{lstlisting}[caption={Method to Calculate Dot Product in Java}]
public double dot(double[] input, double[] weight) {// Dot product of two 1xN arrays
	double sum = 0; //Define a variable that will hold the sum of the function
	for (int i = 0; i < input.length; i++) { //Repeat over the length of the input arrays
		sum += input[i] * weight[i]; //Add to the sum the product of the corresponding data points from each array
	}
	return sum; //Return the sum as the output of the method
}
// Overloaded method to accept integer as input
public double dot(int[] input, double[] weight){
    double[] out = new double[input.length];//Define output array
	for(int i = 0; i < input.length; i++){ //Convert input array from integer type to double type
		out[i] = input[i];
	}
    return dot(out,weight); //Run the dot function with the new input array
}
	\end{lstlisting}

	All that this code does is iterate over each array and sum the product of each item together. This process can be represented mathematically by the following:
	\[
		A =
		\begin{bmatrix}
			a_1    \\
			a_2    \\
			\vdots \\
			a_n    \\
		\end{bmatrix}
		B =
		\begin{bmatrix}
			b_1    \\
			b_2    \\
			\vdots \\
			b_n    \\
		\end{bmatrix}
	\]
	\[
		A \cdot B = \sum^n_{i=1} a_ib_i = a_1b_1 + a_2b_2 + \dots + a_nb_n
	\]
	Don't worry if this math looks confusing. All it's describing is the addition of all of the products of the matrices $A$ and $B$
	\subsection{Transpose}

	After the dot product code I wrote another method to calculate the transpose of a matrix (in programming just a multidimensional array). The code looks as follows:
	\begin{lstlisting}[caption = {Method to Return Transpose of an Array in Java}]
// Returns Transpose of  array
public int[][] transpose(int[][] in) { //Take input as array of integers
	int[][] out = new int[in[0].length][in.length]; //Create an output array using the dimensions of the input array, except flipped.
	// Set out[x][y] equal to input[y][x]
	for (int y = 0; y < in.length; y++) {
		for (int x = 0; x < in[0].length; x++) {
			out[x][y] = in[y][x];
		}
	}
	return out; //Return output array as the output of the method
}
	\end{lstlisting}

	Much like the dot product code from before, this code iterates over an array using a \lstinline{for} loop, except that this time instead of taking the products of each item and then summing them this just assigns each value to a new matrix such that:
	\[
		_na_m \rightarrow {}_mb_n
	\]

	\printbibliography
\end{document}


\section{Misc}
  \subsection{Debugging Tips}
    \begin{myitemize}
      \item Stack overflow? Recursive DFS on tree that is actually a long path?
      \item Floating-point numbers
        \begin{itemize}
          \item Getting \texttt{NaN}? Make sure \texttt{acos} etc.\ are
            not getting values out of their range (perhaps
            \texttt{1+eps}).
          \item Rounding negative numbers?
          \item Outputting in scientific notation?
        \end{itemize}
      \item Wrong Answer?
        \begin{itemize}
          \item Read the problem statement again!
          \item Are multiple test cases being handled correctly?
            Try repeating the same test case many times.
          \item Integer overflow?
          \item Think very carefully about boundaries of all input parameters
          \item Try out possible edge cases:
            \begin{itemize}
              \item $n=0, n=-1, n=1, n=2^{31}-1$ or $n=-2^{31}$
              \item List is empty, or contains a single element
              \item $n$ is even, $n$ is odd
              \item Graph is empty, or contains a single vertex
              \item Graph is a multigraph (loops or multiple edges)
              \item Polygon is concave or non-simple
            \end{itemize}
          \item Is initial condition wrong for small cases?
          \item Are you sure the algorithm is correct?
          \item Explain your solution to someone.
          \item Are you using any functions that you don't completely understand? Maybe STL functions?
          \item Maybe you (or someone else) should rewrite the solution?
          \item Can the input line be empty?
        \end{itemize}
      \item Run-Time Error?
        \begin{itemize}
          \item Is it actually Memory Limit Exceeded?
        \end{itemize}
    \end{myitemize}

  \subsection{Solution Ideas}
    \begin{myitemize}
      \item Dynamic Programming
        \begin{itemize}
          \item Parsing CFGs: CYK Algorithm
          \item Drop a parameter, recover from others
          \item Swap answer and a parameter
          \item When grouping: try splitting in two
          \item $2^k$ trick
          \item When optimizing
              \begin{itemize}
                \item Convex hull optimization
                  \begin{itemize}
                    \item $\mathrm{dp}[i] = \min_{j<i}\{\mathrm{dp}[j] + b[j] \times a[i]\}$
                    \item $b[j] \geq b[j+1]$
                    \item optionally $a[i] \leq a[i+1]$
                    \item $O(n^2)$ to $O(n)$
                  \end{itemize}
                \item Divide and conquer optimization
                  \begin{itemize}
                    \item $\mathrm{dp}[i][j] = \min_{k<j}\{\mathrm{dp}[i-1][k] + C[k][j]\}$
                    \item $A[i][j] \leq A[i][j+1]$
                    \item $O(kn^2)$ to $O(kn\log{n})$
                    \item sufficient: $C[a][c] + C[b][d] \leq C[a][d] + C[b][c]$, $a\leq b\leq c\leq d$ (QI)
                  \end{itemize}
                \item Knuth optimization
                  \begin{itemize}
                    \item $\mathrm{dp}[i][j] = \min_{i<k<j}\{\mathrm{dp}[i][k] + \mathrm{dp}[k][j] + C[i][j]\}$
                    \item $A[i][j-1] \leq A[i][j] \leq A[i+1][j]$
                    \item $O(n^3)$ to $O(n^2)$
                    \item sufficient: QI and $C[b][c] \leq C[a][d]$, $a\leq b\leq c\leq d$
                  \end{itemize}
              \end{itemize}
        \end{itemize}
      \item Greedy
      \item Randomized
      \item Optimizations
        \begin{itemize}
          \item Use bitset (/64)
          \item Switch order of loops (cache locality)
        \end{itemize}
      \item Process queries offline
        \begin{itemize}
          \item Mo's algorithm
        \end{itemize}
      \item Square-root decomposition
      \item Precomputation
      \item Efficient simulation
        \begin{itemize}
          \item Mo's algorithm
          \item Sqrt decomposition
          \item Store $2^k$ jump pointers
        \end{itemize}
      \item Data structure techniques
        \begin{itemize}
          \item Sqrt buckets
          \item Store $2^k$ jump pointers
          \item $2^k$ merging trick
        \end{itemize}
      \item Counting
        \begin{itemize}
          \item Inclusion-exclusion principle
          \item Generating functions
        \end{itemize}
      \item Graphs
        \begin{itemize}
          \item Can we model the problem as a graph?
          \item Can we use any properties of the graph?
          \item Strongly connected components
          \item Cycles (or odd cycles)
          \item Bipartite (no odd cycles)
            \begin{itemize}
              \item Bipartite matching
              \item Hall's marriage theorem
              \item Stable Marriage
            \end{itemize}
          \item Cut vertex/bridge
          \item Biconnected components
          \item Degrees of vertices (odd/even)
          \item Trees
            \begin{itemize}
              \item Heavy-light decomposition
              \item Centroid decomposition
              \item Least common ancestor
              \item Centers of the tree
            \end{itemize}
          \item Eulerian path/circuit
          \item Chinese postman problem
          \item Topological sort
          \item (Min-Cost) Max Flow
          \item Min Cut
            \begin{itemize}
              \item Maximum Density Subgraph
            \end{itemize}
          \item Huffman Coding
          \item Min-Cost Arborescence
          \item Steiner Tree
          \item Kirchoff's matrix tree theorem
          \item Pr\"ufer sequences
          \item Lov\'asz Toggle
          \item Look at the DFS tree (which has no cross-edges)
          \item Is the graph a DFA or NFA?
              \begin{itemize}
                \item Is it the Synchronizing word problem?
              \end{itemize}
        \end{itemize}
      \item Mathematics
        \begin{itemize}
          \item Is the function multiplicative?
          \item Look for a pattern
          \item Permutations
            \begin{itemize}
              \item Consider the cycles of the permutation
            \end{itemize}
          \item Functions
            \begin{itemize}
              \item Sum of piecewise-linear functions is a piecewise-linear function
              \item Sum of convex (concave) functions is convex (concave)
            \end{itemize}
          \item Modular arithmetic
            \begin{itemize}
              \item Chinese Remainder Theorem
              \item Linear Congruence
            \end{itemize}
          \item Sieve
          \item System of linear equations
          \item Values too big to represent?
            \begin{itemize}
              \item Compute using the logarithm
              \item Divide everything by some large value
            \end{itemize}
          \item Linear programming
            \begin{itemize}
              \item Is the dual problem easier to solve?
            \end{itemize}
          \item Can the problem be modeled as a different combinatorial problem? Does that simplify calculations?
        \end{itemize}
      \item Logic
        \begin{itemize}
          \item 2-SAT
          \item XOR-SAT (Gauss elimination or Bipartite matching)
        \end{itemize}
      \item Meet in the middle
      \item Only work with the smaller half ($\log(n)$)
      \item Strings
        \begin{itemize}
          \item Trie (maybe over something weird, like bits)
          \item Suffix array
          \item Suffix automaton (+DP?)
          \item Aho-Corasick
          \item eerTree
          \item Work with $S+S$
        \end{itemize}
      \item Hashing
      \item Euler tour, tree to array
      \item Segment trees
        \begin{itemize}
          \item Lazy propagation
          \item Persistent
          \item Implicit
          \item Segment tree of X
        \end{itemize}
      \item Geometry
        \begin{itemize}
          \item Minkowski sum (of convex sets)
          \item Rotating calipers
          \item Sweep line (horizontally or vertically?)
          \item Sweep angle
          \item Convex hull
        \end{itemize}
      \item Fix a parameter (possibly the answer).
      \item Are there few distinct values?
      \item Binary search
      \item Sliding Window (+ Monotonic Queue)
      \item Computing a Convolution? Fast Fourier Transform
      \item Computing a 2D Convolution? FFT on each row, and then on each column
      \item Exact Cover (+ Algorithm X)
      \item Cycle-Finding
      \item What is the smallest set of values that identify the solution? The cycle structure of the permutation? The powers of primes in the factorization?
      \item Look at the complement problem
        \begin{itemize}
          \item Minimize something instead of maximizing
        \end{itemize}
      \item Immediately enforce necessary conditions. (All values greater than 0? Initialize them all to 1)
      \item Add large constant to negative numbers to make them positive
      \item Counting/Bucket sort
    \end{myitemize}

\section{Formulas}

  % \item Number of permutations of length $n$ that have no fixed
  %     points (derangements): $D_0 = 1, D_1 = 0, D_n = (n - 1)(D_{n-1}
  %     + D_{n-2})$
  % \item Number of permutations of length $n$ that have exactly $k$
  %     fixed points: $\binom{n}{k} D_{n-k}$

  \begin{itemize}[leftmargin=*]
    \item \textbf{Legendre symbol:} $\left(\frac{a}{b}\right) = a^{(b-1)/2} \pmod{b}$, $b$ odd prime.
    \item \textbf{Heron's formula:} A triangle with side lengths
      $a,b,c$ has area $\sqrt{s(s-a)(s-b)(s-c)}$ where $s =
      \frac{a+b+c}{2}$.
    \item \textbf{Pick's theorem:} A polygon on an integer grid
      strictly containing $i$ lattice points and having $b$ lattice
      points on the boundary has area $i + \frac{b}{2} - 1$. (Nothing
      similar in higher dimensions)
    \item \textbf{Euler's totient:} The number of integers less than
      $n$ that are coprime to $n$ are $n\prod_{p|n}\left(1 - \frac{1}{p}\right)$
      where each $p$ is a distinct prime factor of $n$.
    \item \textbf{König's theorem:} In any bipartite graph $G=(L\cup R,E)$, the number
      of edges in a maximum matching is equal to the number of
      vertices in a minimum vertex cover. Let $U$ be the set of
      unmatched vertices in $L$, and $Z$ be the set of vertices that
      are either in $U$ or are connected to $U$ by an alternating
      path. Then $K=(L\setminus Z)\cup(R\cap Z)$ is the minimum
      vertex cover.
    \item A minumum Steiner tree for $n$ vertices requires at most $n-2$ additional Steiner vertices.
    \item The number of vertices of a graph is equal to its minimum
      vertex cover number plus the size of a maximum independent set.
    \item \textbf{Lagrange polynomial} through points $(x_0,y_0),\ldots,(x_k,y_k)$ is $L(x) = \sum_{j=0}^k y_j \prod_{\shortstack{$\scriptscriptstyle 0\leq m \leq k$ \\ $\scriptscriptstyle m\neq j$}} \frac{x-x_m}{x_j - x_m}$
    \item \textbf{Hook length formula:} If $\lambda$ is a Young diagram and $h_{\lambda}(i,j)$ is the hook-length of cell $(i,j)$, then then the number of Young tableux $d_{\lambda} = n!/\prod h_{\lambda}(i,j)$.
    \item \textbf{Möbius inversion formula:} If $f(n) = \sum_{d|n} g(d)$, then $g(n) = \sum_{d|n} \mu(d) f(n/d)$. If $f(n) = \sum_{m=1}^n g(\lfloor n/m\rfloor)$, then $g(n) = \sum_{m=1}^n \mu(m)f(\lfloor\frac{n}{m}\rfloor)$.
    \item \#primitive pythagorean triples with hypotenuse $<n$ approx $n/(2\pi)$.
    \item \textbf{Frobenius Number:} largest number which can't be
      expressed as a linear combination of numbers $a_1,\ldots,a_n$
      with non-negative coefficients. $g(a_1,a_2) = a_1a_2-a_1-a_2$,
      $N(a_1,a_2)=(a_1-1)(a_2-1)/2$. $g(d\cdot a_1,d\cdot a_2,a_3) =
      d\cdot g(a_1,a_2,a_3) + a_3(d-1)$. An integer $x>\left(\max_i
      a_i\right)^2$ can be expressed in such a way iff.\ $x\ |\
      \mathrm{gcd}(a_1,\ldots,a_n)$.
  \end{itemize}

  \subsection{Physics}
    \begin{itemize}
      \item \textbf{Snell's law:} $\frac{\sin\theta_1}{v_1} = \frac{\sin\theta_2}{v_2}$
    \end{itemize}

  \subsection{Markov Chains}
    A Markov Chain can be represented as a weighted directed graph of
    states, where the weight of an edge represents the probability of
    transitioning over that edge in one timestep. Let $P^{(m)} = (p^{(m)}_{ij})$
    be the probability matrix of transitioning from state $i$ to state $j$
    in $m$ timesteps, and note that $P^{(1)}$ is the adjacency matrix of
    the graph. \textbf{Chapman-Kolmogorov:} $p^{(m+n)}_{ij} = \sum_{k}
    p^{(m)}_{ik} p^{(n)}_{kj}$. It follows that $P^{(m+n)} =
    P^{(m)}P^{(n)}$ and $P^{(m)} = P^m$. If $p^{(0)}$ is the initial
    probability distribution (a vector), then $p^{(0)}P^{(m)}$ is the
    probability distribution after $m$ timesteps.

    The return times of a state $i$ is $R_i = \{m\ |\ p^{(m)}_{ii} > 0 \}$,
    and $i$ is \textit{aperiodic} if $\gcd(R_i) = 1$. A MC is aperiodic if
    any of its vertices is aperiodic. A MC is \textit{irreducible} if the
    corresponding graph is strongly connected.

    A distribution $\pi$ is stationary if $\pi P = \pi$. If MC is
    irreducible then $\pi_i = 1/\mathbb{E}[T_i]$, where $T_i$ is the
    expected time between two visits at $i$. $\pi_j/\pi_i$ is the expected
    number of visits at $j$ in between two consecutive visits at $i$. A MC
    is \textit{ergodic} if $\lim_{m\to\infty} p^{(0)} P^{m} = \pi$. A MC is
    ergodic iff.\ it is irreducible and aperiodic.

    A MC for a random walk in an undirected weighted graph (unweighted
    graph can be made weighted by adding $1$-weights) has $p_{uv} =
    w_{uv}/\sum_{x} w_{ux}$. If the graph is connected, then $\pi_u =
    \sum_{x} w_{ux} / \sum_{v}\sum_{x} w_{vx}$. Such a random walk is
    aperiodic iff.\ the graph is not bipartite.

    An \textit{absorbing} MC is of the form $P = \left(\begin{matrix} Q & R
    \\ 0 & I_r \end{matrix}\right)$. Let $N = \sum_{m=0}^\infty Q^m = (I_t
    - Q)^{-1}$. Then, if starting in state $i$, the expected number of
    steps till absorption is the $i$-th entry in $N1$. If starting in state
    $i$, the probability of being absorbed in state $j$ is the $(i,j)$-th
    entry of $NR$.

    Many problems on MC can be formulated in terms of a system of
    recurrence relations, and then solved using Gaussian elimination.

  \subsection{Burnside's Lemma}
    Let $G$ be a finite group that acts on a set $X$. For each $g$ in $G$
    let $X^g$ denote the set of elements in $X$ that are fixed by $g$. Then
    the number of orbits \[ |X/G| = \frac{1}{|G|} \sum_{g\in G} |X^g| \]

    \[
        Z(S_n) = \frac{1}{n} \sum_{l=1}^n a_l Z(S_{n-l})
    \]

  \subsection{Bézout's identity}
    If $(x,y)$ is any solution to $ax+by=d$ (e.g.\ found by the Extended
    Euclidean Algorithm), then all solutions are given by \[
    \left(x+k\frac{b}{\gcd(a,b)}, y-k\frac{a}{\gcd(a,b)}\right) \]

  \subsection{Misc}
    \subsubsection{Determinants and PM}
      \begin{align*}
        det(A) &= \sum_{\sigma \in S_n}\text{sgn}(\sigma)\prod_{i = 1}^n a_{i,\sigma(i)}\\
        perm(A) &= \sum_{\sigma \in S_n} \prod_{i = 1}^n a_{i,\sigma(i)}\\
        pf(A) &= \frac{1}{2^nn!}\sum_{\sigma \in S_{2n}} \text{sgn}(\sigma)\prod_{i = 1}^n a_{\sigma(2i-1),\sigma(2i)}\\ &= \sum_{M \in \text{PM}(n)} \text{sgn}(M) \prod_{(i,j) \in M} a_{i,j}
      \end{align*}

    \subsubsection{BEST Theorem}
      Count directed Eulerian cycles. Number of OST given by
      Kirchoff's Theorem (remove r/c with root) $\#\textsc{OST}(G,r)
      \cdot \prod_{v}(d_v-1)!$

    \subsubsection{Primitive Roots}
      Only exists when $n$ is $2, 4, p^k, 2p^k$, where $p$ odd prime. Assume
      $n$ prime. Number of primitive roots $\phi(\phi(n))$
      Let $g$ be primitive root. All primitive roots are of the form $g^k$
      where $k,\phi(p)$ are coprime.\\ $k$-roots:
      $g^{i \cdot \phi(n) / k}$ for $0 \leq i < k$

    \subsubsection{Sum of primes} For any multiplicative $f$:
      \[
          S(n,p) = S(n, p-1) - f(p) \cdot (S(n/p,p-1) - S(p-1,p-1))
      \]

    \subsubsection{Floor}
      \begin{align*}
          &\left\lfloor \left\lfloor x/y \right\rfloor / z \right\rfloor = \left\lfloor x / (yz) \right\rfloor \\
          &x \% y = x - y \left\lfloor x / y \right\rfloor
      \end{align*}

\iffalse
  \clearpage
  \section*{Practice Contest Checklist}
    \begin{itemize}
      \item How many operations per second? Compare to local machine.
      \item What is the stack size?
      \item How to use printf/scanf with long long/long double?
      \item Are \texttt{\_{}\_{}int128} and \texttt{\_{}\_{}float128} available?
      \item Does MLE give RTE or MLE as a verdict? What about stack overflow?
      \item What is \texttt{RAND\_{}MAX}?
      \item How does the judge handle extra spaces (or missing newlines) in the output?
      \item Look at documentation for programming languages.
      \item Try different programming languages: C++, Java and Python.
      \item Try the submit script.
      \item Try local programs: i?python[23], factor.
      \item Try submitting with \texttt{assert(false)} and \texttt{assert(true)}.
      \item Return-value from \texttt{main}.
      \item Look for directory with sample test cases.
      \item Make sure printing works.

      \item Remove this page from the notebook.
    \end{itemize}
\fi


{\bfseries\slshape\sffamily\color{ChapterTitleColor} \chapter{Abstract Syntax Tree}} \label{chap:AST}

\section{Introduction}

The biggest internal change in Part III is by far the introduction of an additional phase between syntax analysis and code generation. See Figure~\ref{fig:simpleInterpreter2} from Chapter 1. In the new phase we construct the abstract syntax tree or AST for short.

The AST is so-called because it represents a very terse view of the source code. For example, bracketing is absent because the grouping of elements is implicit in the structure of the tree. But what is the purpose of the AST stage? There are several, the first is that we get a clean separation of syntax analysis from code generation and that the AST structure can be analysed for semantic correctness and even some basic code optimization can be carried out. For languages that are statically typed (such as Object Pascal or C), where variables have strict immutable types, semantic analysis can be very useful to eliminate errors.

In part II of this series we talked about tree structures and mentioned the different ways in which such a tree can be traversed. I am therefore going to assume that the reader knows something about tree structures but not necessarily a lot.

The source file that constructs the AST is called {\tt uConstructAST.pas}. There is also a separate file called {\tt uAST.pas} that contains all the support code to aid with construction and {\tt uASTNodeTypes} which lists the types of node we can have.

The easiest example to consider is the binary operator such as {\tt 2 + 5}. A very simple parser for this statement would be:

\begin{lstlisting}
procedure TConstructAST.parseBinaryOperator;
begin
  parseNumber;
  nextToken;
  if token = tPlus then
     begin
     nextToken;
     parseNumber;
     end
  else
     error
end;
\end{lstlisting}

We need to modify this code so that we can turn {\tt 2 + 3} into a little tree. The first change is that the procedure must return the root to the mini-tree we construct. For that reason lets create a class called {\tt TASTNode} that the function will return. We'll have more to stay about this class later on. {\tt parseNumber} should return a node that represents a number and we need to record the plus symbol we find. The method needs to return a tree that looks like the one shown in Figure~\ref{fig:TwoPlusThreeTree}. I've marked the plus node as the root of this tree.

\begin{figure}[htpb]
\centering
\includegraphics[scale=0.55]{TwoPlusThreeTree.pdf}
\caption{A little tree formed from {\tt 2 + 3}.}
\label{fig:TwoPlusThreeTree}
\end{figure}

Let's update the function to collect any nodes we create:

\begin{lstlisting}
function TConstructAST.parseBinaryOperator : TASTNode;
var : operand1, operand2 : TASTNode;
      operand : TASTNodeType;
begin
  operand1 := parseNumber;
  nextToken;
  if token = tPlus then
     begin
     operand := ntPlus;
     nextToken;
     operand2 := parseNumber;
     end
  else
     error
end;
\end{lstlisting}

We need a method that will knit the three components, {\tt operand1}, {\tt operand2} and {\tt operator} into a little tree. Let's pretend we have a method that does that, called {\tt TASTBinOp}. We might use it like this:

\begin{lstlisting}
function TConstructAST.parseBinaryOperator : TASTNode;
var : operand1, operand2 : TASTNode;
      operator : TASTNodeType;
begin
  operand1 := parseNumber;
  nextToken;
  if token = tPlus then
     begin
     operator := ntPlus;
     nextToken;
     operand2 := parseNumber;
     end
  else
     error
  result := TASTBinOp.Create (operand1, operand2, operator);
end;
\end{lstlisting}

You can probably guess from the syntax of {\tt TASTBinOp} that the first two arguments are the leaves of tree and the third argument is the node type we use to make the root node.

One problem we need to deal with is what to do with errors?. For example, what if there is no second operand? The problem is that we've already constructed the first leaf, {\tt operand1}. An easy way out would be to just raise an exception and get out quick. The problem with that strategy is we'll end up with memory leaks since we're leaving behind tree components we no longer need. We have to somehow gracefully exit if there is an error. I came up with a number of possible solutions to this:

\begin{itemize}
  \item Create two parsers, one parser only checks for syntax errors and generates no output other than success or failure and in the case of a failure the line number and nature of the error. If the first parse is successful we run the second parser which does create the AST. Since we're guaranteed there won't any errors, there won't be any chance to create a partial tree that needs cleaning up.
  \item Build the AST using a data structure, perhaps a table, that doesn't require any memory to be allocated as the parser runs. If there is syntax error we can get out and free the table.
  \item Monitor for errors as we build the tree, take care to free any partially constructed tree components in the event of an error. When an error occurs, clean up any partial tree components and return a special error node that includes information on the nature of the error including any line number information. When we come to generate the bytecode from the AST, we would need to watch out for any error nodes and respond with the error message kept in the error node.
\end{itemize}

All three solutions are viable. The first one results in a two pass system that could slow things down. The second seems a bit complicated as we'd have to use indices to represent links between nodes and tree traversals won't be as simple to implement. In the third solution, we clean up any dangling tree components if an error occurs and insert a special error node into the tree to indicate a problem. How might this translate into parsing {\tt 2 + 3} example? I am going to assume that {\tt parseNumber} and {\tt expect} will return an AST error node in the event of an error. We can check for this error node as we do the parsing.  The listing below shows what I mean.

\begin{lstlisting}
function TConstructAST.parseBinaryOperator : TASTNode;
var : operand1, operand2, operator : TASTNode;
begin
  operand1 := parseNumber;
  if operand1 = error then
     exit (operand1)

  operator := expect(plus)
  if operator = error then
     begin
     operand1.free;
     exit (operator)
     end;

  operand2 := expect(number)
  if operand2 = error then
     begin
     operand1.free;
     operator.free
     exit (operand2)
     end

  // No errors, so lets build the minitree and return it.
  result := TASTBinOp.Create (operand1, operand2, operator);
end;
\end{lstlisting}

If there is an error, {\tt parseBinaryOperator} returns an AST node that represents an error. Let me show you what the {\tt TASTError} node might looks like:

\begin{lstlisting}
TASTErrorNode = class (TASTNode)
   lineNumber, columnNumber : integer;
   errorMsg : string;
   constructor Create (errMsg : string; lineNumber, columnNumber : integer);
end;
\end{lstlisting}

Notice that the error node will store the current line number and column number in the text that is being parsed. This is important, because once the tree is built there won't be another opportunity to collect this information. The error node also has an {\tt errorMsg} field that we use to store a suitable message for the user.

Let's look at another example, this time parsing the repeat/until construct. The grammar for this is simple:

{\tt\small repeatStatement = REPEAT statementList UNTIL expression}

The AST that we'll get from this is equally simple, its just like the {\tt 2 + 3} minitree, except the root is labelled {\tt repeat} and the leaves are {\tt statementList} and {\tt condition}.

{\tt\small (repeat) -> (statementList) and (condition)}

In the following code I've left out the {\tt break} statements to make it easier to understand. Also there is no error checking at the moment. Hopefully the code that is left is self-explanatory. We first expect the {\tt repeat} keyword, then a list of statements, followed by the {\tt until} keyword, and finally the until condition. With that we construct the minitree, using {\tt TASTRepeat} and splice together the {\tt condition} and {\tt statementList} and return the repeat node to the caller. One thing you may have realized is that the left-hand side of each grammar statement will have its own AST node. This is so that during compilation we can recognise particular nodes.

\begin{lstlisting}
function TConstructAST.parseRepeatStatement: TASTNode;
begin
  expect(tRepeat);
  listOfStatements := statementList;
  expect(tUntil);
  condition := expression;

  result := TASTRepeat.Create(listOfStatements, condition);
end;
\end{lstlisting}

You might start to get the feeling that we will have quite a few different nodes representing different aspects of the Rhodus language, and you'll be right. {\tt TASTNode} will be the parent node for a wide range of descendent nodes.

Let's now add the error detection. For the first line, {\tt expect (tRepeat)}, there won't be an error because in order to have got here we must already have seen the keyword repeat.  However, errors can occur when we parse the {\tt statementList}, {\tt condition} and the {\tt tUntil}. In particular if there is an error at the condition state we must make sure to free up the partial tree we get from statementList. You'll noticed I also check for an error from {\tt expect (tUntil)}. This is in case we read the {\tt endOfStream} token. The method {\tt freeAST} is part of the AST class and can be used to free any complete or partially constructed tree.

\begin{lstlisting}
function TConstructAST.parseRepeatStatement: TASTNode;
var node : TASTNode;
begin
  expect(tRepeat);
  listOfStatements := statementList;
  if listOfStatements = error then
     exit (listofStatements);

  node := expect(tUntil);
  if node == error then
     begin
     listOfStatements.freeAST;
     exit (node);
     end;

  condition := expression;
  if condition = error then
     begin
     listOfStatements.freeAST;
     exit (condition)
     end;

  result := TASTRepeat.Create(listOfStatements, condition);
end;
\end{lstlisting}

You can imagine having to do this at every stage in the syntax analysis. I did in fact implement it this way at one point but realised that maintaining the code could be problematic. It was too easy to forget to free something and cause a memory leak and it took some time to fully debug the code. In the end I decided this was not the way to do it and ultimately I discarded the work and reverted to the first solution which involved doing two passes. One pass to check syntax and a second to construct the AST.

\begin{lstlisting}
// Base AST Node
TASTNode = class (TObject)
   nodeType : TASTNodeType;

   procedure   freeAST;
   constructor Create(nodeType: TASTNodeType);
   destructor  Destroy; override;
end;
\end{lstlisting}

%
%\section{Useful Reading}
%
%\subsection{Introductory Books}
%
%{\bf 1.} Ball, Thorsten. Writing A Compiler In Go. Thorsten Ball, 2018.
%
%{\bf 2.} Kernighan, Brian W.; Pike, Rob (1984). The Unix Programming Environment. Prentice-Hall. ISBN 0-13-937681-X.
%
%{\bf 3.} Nisan, Noam, and Shimon Schocken. The elements of computing systems: building a modern computer from first principles. MIT press, 2005.
%
%{\bf 4.} Parr, Terence. Language implementation patterns: create your own domain-specific and general programming languages. Pragmatic Bookshelf, 2009.
%
%\subsection{More Advanced Books}
%
%{\bf 1.} Jim Smith, Ravi Nair, Virtual Machines: Versatile Platforms for Systems and Processes, Morgan Kauffmann, June 2005
%
%{\bf 2.} Aho, Alfred V., Ravi Sethi, and Jeffrey D. Ullman. Compilers: Principles, Techniques and Tools (also known as The Red Dragon Book), 1986.
%
%\subsection{Source Code}
%
%{\bf 1.} Mak, Ronald. Writing compilers and interpreters: an applied approach/by Ronald Mark. 1991
%
%Note, this is the first edition, 1991. The code is in C, which I found to be understandable. The later editions that use C++ are not as clear. The issue I found is that the object orientated approach that's used tends to obscure the design principles of the interpreter and requires much study to decipher, The C version is much more straightforward.
%
%{\bf 2.} Wren: \url{https://github.com/wren-lang/wren}.\index{wren}
%
%Of the open source interpreters on GitHub, I found this to be the easiest to read. It's written by Bob Nystrom in C, the same person who is writing the web book: Crafting Interpreters \url{https://craftinginterpreters.com/}.
%
%{\bf 3.} Gravity: \index{Gravity} Another open source interpreter worth looking at is Gravity (\url{https://github.com/marcobambini/gravity}). Gravity, like Wren, is also written in C.
%
%{\bf 4.}  If you prefer Go,\index{Go} then the source code to look at is the interpreter written by Thorsten Ball (see book reference above).
%
%There are umpteen BASIC interpreters\index{BASIC} and other languages that can be studied.
%
%\bigskip\medskip

\begin{center}
\pgfornament[width = 8cm, color = cardinal]{83}
\end{center}


\documentclass[paper.tex]{subfiles}

\usepackage{tikz}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{tabularx}
\usepackage{multicol}
\usepackage{algpseudocode}
\usepackage{algorithm}

% Add vertical spacing to tables
\renewcommand{\arraystretch}{1.4}

% Begin Document
\begin{document}

\section{Conclusion}

This paper has explored two approaches to finding the Minimum Domating Set of a graph, a brute force approach and an approximation approach.
This problem is an $NP-Complete$ problem.
This follows from two facts about the problem.
First, the best known solution is a brute force approach, checking every possible solution.
Second, the solution is verifiable in polynomial time.

The brute force approach checks every possible combination of nodes of the graph to find a minimal covering.
Thus, for a graph with $n$ nodes, the amount of possible combinations is $2^n$, indicating a runtime complexity of $O(2^n)$.
Our experimental data shows that a modest 37-node graph can take nearly 45 hours to run.
This does come with the guarantee of finding the minimal solution.

The approximation approach is much faster however.
Utilizing a naive greedy approach, a 37-node graph was near instantaneous for a solution.
For all experimental data, the approximate solution ran orders of magnitude faster than the brute force approach.
The implemented approach ran with a time complexity of $O(n^2)$, much better than the brute force approach.
However, this comes with a major trade-off.
The approximation algorithm rarely finds a minimal solution, with approximate solutions often being double the size of the found minimal solution.

Whether the brute force approach or the approximate approach is the proper choice is domain specific.
Each project will thus need to weigh its factors in determining if an approximate solution is of sufficient quality when applying these techniques.

\end{document}


\documentclass[letterpaper,10pt,twoside,printwatermark=false]{pinp}

%% Some pieces required from the pandoc template
\providecommand{\tightlist}{%
  \setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}

% Use the lineno option to display guide line numbers if required.
% Note that the use of elements such as single-column equations
% may affect the guide line number alignment.

\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}

% The geometry package layout settings need to be set here...
\geometry{layoutsize={0.95588\paperwidth,0.98864\paperheight},%
          layouthoffset=0.02206\paperwidth,%
		  layoutvoffset=0.00568\paperheight}

\definecolor{pinpblue}{HTML}{185FAF}  % imagecolorpicker on blue for new R logo
\definecolor{pnasbluetext}{RGB}{101,0,0} %

\title{Poll 3 - Sampling Distribution, Confidence Intervals}

\author[a]{EPIB607 - Inferential Statistics}

  \affil[a]{Fall 2018, McGill University}

\setcounter{secnumdepth}{5}

% Please give the surname of the lead author for the running footer
\leadauthor{Bhatnagar and Hanley}

% Keywords are not mandatory, but authors are strongly encouraged to provide them. If provided, please include two to five keywords, separated by the pipe symbol, e.g:
 \keywords{  Sampling distribution |  Confidence interval |  qnorm |  pnorm  }

\begin{abstract}
This live poll was conducted on October 4, 2018.
\end{abstract}

\dates{This version was compiled on \today}
\doi{\url{https://sahirbhatnagar.com/EPIB607/}}

\pinpfootercontents{Poll 3 - October 4, 2018}

\begin{document}

% Optional adjustment to line up main text (after abstract) of first page with line numbers, when using both lineno and twocolumn options.
% You should only change this length when you've finalised the article contents.
\verticaladjustment{-2pt}

\maketitle
\thispagestyle{firststyle}
\ifthenelse{\boolean{shortarticle}}{\ifthenelse{\boolean{singlecolumn}}{\abscontentformatted}{\abscontent}}{}

% If your first paragraph (i.e. with the \dropcap) contains a list environment (quote, quotation, theorem, definition, enumerate, itemize...), the line after the list may have some extra indentation. If this is the case, add \parshape=0 to the end of the list environment.

\section{Which R function would we use to answer the following question:
what is the probability of seeing an IQ score as extreme as 130 (select
all that
apply)?}\label{which-r-function-would-we-use-to-answer-the-following-question-what-is-the-probability-of-seeing-an-iq-score-as-extreme-as-130-select-all-that-apply}

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  pnorm \textbf{(Correct)}
\item
  qnorm
\item
  dnorm
\item
  rnorm
\end{enumerate}

\section{Which R function would we use to answer the following question:
What is the 75th percentile of the IQ scores distribution (select all
that
apply)?}\label{which-r-function-would-we-use-to-answer-the-following-question-what-is-the-75th-percentile-of-the-iq-scores-distribution-select-all-that-apply}

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  pnorm
\item
  qnorm \textbf{(Correct)}
\item
  dnorm
\item
  rnorm
\end{enumerate}

\section{\texorpdfstring{The population SD is unknown and is denoted by
\(\sigma\). In an SRS of size \(n\), the standard deviation of that
sample
is}{The population SD is unknown and is denoted by \textbackslash{}sigma. In an SRS of size n, the standard deviation of that sample is}}\label{the-population-sd-is-unknown-and-is-denoted-by-sigma.-in-an-srs-of-size-n-the-standard-deviation-of-that-sample-is}

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  the sample standard deviation \textbf{(Correct)}
\item
  the sample standard deviation divided by \(\sqrt{n}\)
\item
  \(\sigma / \sqrt{n}\)
\end{enumerate}

\section{\texorpdfstring{The sampling distribution of \(\bar{y}\)
(select all that
apply):}{The sampling distribution of \textbackslash{}bar\{y\} (select all that apply):}}\label{the-sampling-distribution-of-bary-select-all-that-apply}

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  describes how the statistic \(\bar{y}\) varies in all possible SRSs of
  the same size from the same population \textbf{(Correct)}
\item
  is Normally distributed
\item
  is Normally distributed only if the population distribution is Normal
\item
  is usually unknown \textbf{(Correct)}
\item
  is centered around the population mean \(\mu\) \textbf{(Correct)}
\item
  has an SD greater than the population SD for an SRS of size \(n\)
\end{enumerate}

\section{The central limit theorem applies to one random
sample}\label{the-central-limit-theorem-applies-to-one-random-sample}

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  TRUE
\item
  FALSE \textbf{(Correct)}
\end{enumerate}

\section{\texorpdfstring{A 95\% confidence interval for the mean \(\mu\)
(select all that
apply)}{A 95\% confidence interval for the mean \textbackslash{}mu (select all that apply)}}\label{a-95-confidence-interval-for-the-mean-mu-select-all-that-apply}

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  is a random quantity \textbf{(Correct)}
\item
  tells us that in the long run, 95\% of your intervals will contain the
  mean \(\mu\) \textbf{(Correct)}
\item
  tells us that in the long run, 95\% of your intervals will contain the
  sample mean \(\bar{y}\)
\end{enumerate}

\section{\texorpdfstring{Which of the following assumptions are needed
to be able to use a formula of the form
\(\bar{y} \pm z^\star (\sigma / \sqrt{n})\) (select all that
apply)}{Which of the following assumptions are needed to be able to use a formula of the form \textbackslash{}bar\{y\} \textbackslash{}pm z\^{}\textbackslash{}star (\textbackslash{}sigma / \textbackslash{}sqrt\{n\}) (select all that apply)}}\label{which-of-the-following-assumptions-are-needed-to-be-able-to-use-a-formula-of-the-form-bary-pm-zstar-sigma-sqrtn-select-all-that-apply}

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  the population distribution must be normal
\item
  the CLT has `kicked in' \textbf{(Correct)}
\item
  we have an SRS of size \(n\) from the population of interest
  \textbf{(Correct)}
\end{enumerate}

\section{\texorpdfstring{A 95\% CI for the mean \(\mu\) is given by
\(\bar{y} \pm z^\star (\sigma / \sqrt{n})\). This can be calculated in R
using the following command (select all that
apply)}{A 95\% CI for the mean \textbackslash{}mu is given by \textbackslash{}bar\{y\} \textbackslash{}pm z\^{}\textbackslash{}star (\textbackslash{}sigma / \textbackslash{}sqrt\{n\}). This can be calculated in R using the following command (select all that apply)}}\label{a-95-ci-for-the-mean-mu-is-given-by-bary-pm-zstar-sigma-sqrtn.-this-can-be-calculated-in-r-using-the-following-command-select-all-that-apply}

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  \(\bar{y}\) + qnorm(p = c(0.025, 0.975)) \((\sigma / \sqrt{n})\)
  \textbf{(Correct)}
\item
  \(\bar{y}\) + qnorm(p = c(0.025, 0.975), mean = \(\bar{y}\), sd =
  \(\sigma\)) \((\sigma / \sqrt{n})\)
\item
  \(\bar{y}\) + qnorm(p = c(0.025, 0.975), mean = \(\bar{y}\), sd =
  \(\sigma / \sqrt{n}\))
\item
  \(\bar{y}\) + qnorm(p = c(0.025, 0.975), mean = 0, sd =
  \(\sigma / \sqrt{n}\))
\item
  qnorm(p = c(0.025, 0.975), mean = \(\bar{y}\), sd =
  \(\sigma / \sqrt{n}\)) \textbf{(Correct)}
\end{enumerate}

%\showmatmethods

\bibliography{pinp}
\bibliographystyle{jss}

\end{document}