% Like most advanced LaTeX files, this one begins with a lot of
% boilerplate. You don't need to understand (or even read) most of it.
% All you need to do is fill in your name, UMN ID, email address,
% and the number of the pset. (Search for "METADATA" to find the place
% for this.) Then, you can go straight to the "EXERCISE 1"
% section and start writing your solutions.
% The "VARIOUS USEFUL COMMANDS" section is probably worth taking a
% look at at some point.

%----------------------------------------------------------------------------------------
%	PACKAGES AND OTHER DOCUMENT CONFIGURATIONS
%----------------------------------------------------------------------------------------
\documentclass[paper=a4, fontsize=12pt]{scrartcl} % A4 paper and 12pt font size
\usepackage[T1]{fontenc} % Use 8-bit encoding that has 256 glyphs
\usepackage[english]{babel} % English language/hyphenation
\usepackage{amsmath,amsfonts,amsthm,amssymb} % Math packages
\usepackage{mathrsfs}    % More math packages
\usepackage{sectsty}  % Allows customizing section commands
\allsectionsfont{\centering \normalfont\scshape} % Make all section titles centered, the default font and small caps %remove this to left align section tites
\usepackage{hyperref} % Turns cross-references into hyperlinks,
                      % and defines \url and \href commands.
\usepackage{graphicx} % For embedding graphics files.
\usepackage{framed}   % For the "leftbar" environment used below.
\usepackage{ifthen}   % Used for the \powset command below.
\usepackage{lastpage} % for counting the number of pages
\usepackage{mathdots} % for \iddots
\usepackage[headsepline,footsepline,manualmark]{scrlayer-scrpage}
\usepackage[height=10in,a4paper,hmargin={1in,0.8in}]{geometry}
\usepackage[usenames,dvipsnames]{xcolor}
\usepackage{tikz}     % This is a powerful tool to draw vector
                      % graphics inside LaTeX. In particular, you can
                      % use it to draw graphs.
\usepackage{verbatim} % For the "verbatim" environment, in which
                      % special symbols can be used freely without
                      % confusing the compiler. (And it's typeset in
                      % a constant-width font.)
                      % Useful, e.g., for quoting code (or ASCII art).

%\numberwithin{table}{section} % Number tables within sections (i.e. 1.1, 1.2, 2.1, 2.2 instead of 1, 2, 3, 4)

\setlength\parindent{20pt} % Makes indentation for paragraphs longer.
                           % This makes paragraphs stand out more.

%----------------------------------------------------------------------------------------
%	VARIOUS USEFUL COMMANDS
%----------------------------------------------------------------------------------------
% The commands below might be convenient. For example, you probably
% prefer to write $\powset[2]{V}$ for the set of $2$-element subsets
% of $V$, rather than writing $\mathcal{P}_2(V)$.
% Notice that you can easily define your own commands like this.
% Caveat: Some of these commands need to be properly "guarded" when
% they occur in subscripts or superscripts. So you should not write
% $K_\CC$, but rather $K_{\CC}$.
\newcommand{\CC}{\mathbb{C}} % complex numbers
\newcommand{\RR}{\mathbb{R}} % real numbers
\newcommand{\QQ}{\mathbb{Q}} % rational numbers
\newcommand{\NN}{\mathbb{N}} % nonnegative integers
\newcommand{\DD}{{\mathbb{D}}} % dual numbers
\newcommand{\PP}{\mathbb{P}} % positive integers
\newcommand{\KK}{\mathbb{K}} % notation for a ring
\newcommand{\LL}{\mathbb{L}} % notation for a ring
\newcommand{\MM}{\mathbb{M}} % notation for a ring
\newcommand{\FF}{\mathbb{F}} % notation for a field
\newcommand{\Z}[1]{\mathbb{Z}/#1\mathbb{Z}} % integers modulo k
                                            % (syntax: "\Z{k}")
\newcommand{\ZZ}{\mathbb{Z}} % integers
\newcommand{\id}{\operatorname{id}} % identity map
\newcommand{\op}{\operatorname{op}} % opposite ring
\newcommand{\tildot}{\left. \widetilde{\cdot} \right.}
% I use \tildot for an alternative multiplication in one of the
% exercises below.
\newcommand{\lcm}{\operatorname{lcm}}
% Lowest common multiple. For historical reasons, LaTeX has a \gcd
% command built in, but not an \lcm command. The preceding line
% rectifies that.
\newcommand{\set}[1]{\left\{ #1 \right\}}
% $\set{...}$ compiles to {...} (set-brackets).
\newcommand{\abs}[1]{\left| #1 \right|}
% $\abs{...}$ compiles to |...| (absolute value, or size of a set).
\newcommand{\tup}[1]{\left( #1 \right)}
% $\tup{...}$ compiles to (...) (parentheses, or tuple-brackets).
\newcommand{\ive}[1]{\left[ #1 \right]}
% $\ive{...}$ compiles to [...] (Iverson bracket, aka truth value; also, set of first n integers).
\newcommand{\floor}[1]{\left\lfloor #1 \right\rfloor}
% $\floor{...}$ compiles to |_..._| (floor function).
\newcommand{\underbrack}[2]{\underbrace{#1}_{\substack{#2}}}
% $\underbrack{...1}{...2}$ yields
% $\underbrace{...1}_{\substack{...2}}$. This is useful for doing
% local rewriting transformations on mathematical expressions with
% justifications. For example, try this out:
% $ \underbrack{(a+b)^2}{= a^2 + 2ab + b^2 \\ \text{(by the binomial formula)}} $
\newcommand{\powset}[2][]{\ifthenelse{\equal{#2}{}}{\mathcal{P}\left(#1\right)}{\mathcal{P}_{#1}\left(#2\right)}}
% $\powset[k]{S}$ stands for the set of all $k$-element subsets of
% $S$. The argument $k$ is optional, and if not provided, the result
% is the whole powerset of $S$.
\newcommand{\mapeq}[1]{\underset{#1}{\equiv}}
% $\mapeq{f}$ yields $\underset{f}{\equiv}$.
% This is the "equality upon applying $f$" relation.
\newcommand{\eps}{\varepsilon}
% Just a shorthand for a commonly-used symbol.
\newcommand{\N}{\operatorname{N}} % norm of a complex number
\newcommand{\horrule}[1]{\rule{\linewidth}{#1}} % Create horizontal rule command with 1 argument of height
\newcommand{\nnn}{\nonumber\\} % Don't number this line in an "align" environment, and move on to the next line.

%----------------------------------------------------------------------------------------
%	MAKING SUMMATION SIGNS ALWAYS PUT THEIR BOUNDS ABOVE AND BELOW
%	THE SIGN
%----------------------------------------------------------------------------------------
% The following are hacks to ensure that sums (such as
% $\sum_{k=1}^n k$) always put their bounds (i.e., the $k=1$ and the
% $n$) underneath and above the sign, as opposed to on its right.
% Same for products (\prod), set unions (\bigcup) and set
% intersections (\bigcap). Remove the 8 lines below if you do not want
% this behavior.
\let\sumnonlimits\sum
\let\prodnonlimits\prod
\let\cupnonlimits\bigcup
\let\capnonlimits\bigcap
\renewcommand{\sum}{\sumnonlimits\limits}
\renewcommand{\prod}{\prodnonlimits\limits}
\renewcommand{\bigcup}{\cupnonlimits\limits}
\renewcommand{\bigcap}{\capnonlimits\limits}

%----------------------------------------------------------------------------------------
%	ENVIRONMENTS
%----------------------------------------------------------------------------------------
% The incantations below define how theorem environments
% (\begin{theorem} ... \end{theorem}) and their likes will look like.
\newtheoremstyle{plainsl}% <name>
  {8pt plus 2pt minus 4pt}% <Space above>
  {8pt plus 2pt minus 4pt}% <Space below>
  {\slshape}% <Body font>
  {0pt}% <Indent amount>
  {\bfseries}% <Theorem head font>
  {.}% <Punctuation after theorem head>
  {5pt plus 1pt minus 1pt}% <Space after theorem headi>
  {}% <Theorem head spec (can be left empty, meaning `normal')>

% Environments which make the text inside them slanted:
\theoremstyle{plainsl}
  \newtheorem{theorem}{Theorem}[section]
  \newtheorem{proposition}[theorem]{Proposition}
  \newtheorem{lemma}[theorem]{Lemma}
  \newtheorem{corollary}[theorem]{Corollary}
  \newtheorem{conjecture}[theorem]{Conjecture}
% Environments that don't:
\theoremstyle{definition}
  \newtheorem{definition}[theorem]{Definition}
  \newtheorem{example}[theorem]{Example}
  \newtheorem{exercise}[theorem]{Exercise}
  \newtheorem{examples}[theorem]{Examples}
  \newtheorem{algorithm}[theorem]{Algorithm}
  \newtheorem{question}[theorem]{Question}
 \theoremstyle{remark}
  \newtheorem{remark}[theorem]{Remark}
\newenvironment{statement}{\begin{quote}}{\end{quote}}
\newenvironment{fineprint}{\begin{small}}{\end{small}}

%----------------------------------------------------------------------------------------
%	METADATA
%----------------------------------------------------------------------------------------
\newcommand{\myname}{Darij Grinberg} % ENTER YOUR NAME HERE
\newcommand{\myid}{00000000} % ENTER YOUR UMN ID HERE
\newcommand{\mymail}{dgrinber@umn.edu} % ENTER YOUR EMAIL HERE
\newcommand{\psetnumber}{6} % ENTER THE NUMBER OF THIS PSET HERE

%----------------------------------------------------------------------------------------
%	HEADER AND FOOTER
%----------------------------------------------------------------------------------------
\ihead{Solutions to homework set \#\psetnumber} % Page header left
\ohead{page \thepage\ of \pageref{LastPage}} % Page header right
\ifoot{\myname, \myid} % left footer
\ofoot{\mymail} % right footer

%----------------------------------------------------------------------------------------
%	TITLE SECTION
%----------------------------------------------------------------------------------------
\title{
\normalfont \normalsize
\textsc{University of Minnesota, School of Mathematics} \\ [25pt] % Your university, school and/or department name(s)
\horrule{0.5pt} \\[0.4cm] % Thin top horizontal rule
\huge Math 4281: Introduction to Modern Algebra, \\
Spring 2019:
Homework \psetnumber\\% The assignment title
\horrule{2pt} \\[0.5cm] % Thick bottom horizontal rule
}
\author{\myname}

\begin{document}

\maketitle % Print the title

\begin{center} % Delete this if you want to save space!
{\large due date: \textbf{by Canvas or email till the beginning of class on Monday, 22 April 2019}.

Please solve \textbf{at most 3 of the 6 exercises}!}
\end{center}

%----------------------------------------------------------------------------------------
%	EXERCISE 1
%----------------------------------------------------------------------------------------
\horrule{0.3pt} \\[0.4cm]

\section{Exercise 1: The opposite ring}

Let $\KK$ be a ring.
We define a new binary operation $\tildot$ on $\KK$ by setting
\[
a \tildot b = ba \qquad \text{for all } a, b \in \KK .
\]
(Thus, $\tildot$ is the multiplication of $\KK$, but with the
arguments switched.)

\begin{enumerate}

\item[\textbf{(a)}] Prove that the set $\KK$, equipped with the
addition $+$, the multiplication $\tildot$, the zero $0_{\KK}$
and the unity $1_{\KK}$, is a ring.

\end{enumerate}

\noindent This new ring is called the \textit{opposite ring} of $\KK$,
and is denoted by $\KK^{\op}$.

Note that the \textbf{sets} $\KK$ and $\KK^{\op}$ are identical
(so a map from $\KK$ to $\KK$ is the same as a map from
$\KK$ to $\KK^{\op}$);
but the \textbf{rings} $\KK$ and $\KK^{\op}$ are generally
not the same
(so a ring homomorphism from $\KK$ to $\KK$ is not the same as a
ring homomorphism from $\KK$ to $\KK^{\op}$).

\begin{enumerate}

\item[\textbf{(b)}] Prove that the identity map
$\id : \KK \to \KK$ is a ring isomorphism from $\KK$ to
$\KK^{\op}$ if and only if $\KK$ is commutative.

\item[\textbf{(c)}] Now, assume that $\KK$ is the matrix ring
$\LL^{n \times n}$ for some commutative ring $\LL$ and some
$n \in \NN$.
Prove that the map
\[
\KK \to \KK^{\op}, \qquad A \mapsto A^T
\]
(where $A^T$, as usual, denotes the transpose of a matrix $A$)
is a ring isomorphism.

\end{enumerate}

[\textbf{Hint:} In \textbf{(a)}, you only have to check the ring axioms that
have to do with multiplication.
Similarly, in \textbf{(b)}, you are free to check the one axiom relating
to multiplication only.
In \textbf{(c)}, you can use \cite[Exercise 6.5]{detnotes} without proof.]

\subsection{Remark}

This exercise gives some examples of rings $\KK$ that are isomorphic
to their opposite rings $\KK^{\op}$. See
\url{https://mathoverflow.net/questions/64370/} for examples of rings
that are not.

\subsection{Solution}

[...]

%----------------------------------------------------------------------------------------
%	EXERCISE 2
%----------------------------------------------------------------------------------------
\horrule{0.3pt} \\[0.4cm]

\section{Exercise 2: More ring isomorphisms}

\subsection{Problem}

\begin{enumerate}

\item[\textbf{(a)}]
Let $\LL$ be a ring.
Let $w \in \LL$ be an invertible element.
Prove that the map
\[
\LL \to \LL , \qquad a \mapsto waw^{-1}
\]
is a ring isomorphism.

\item[\textbf{(b)}] Let $\KK$ be a ring.
Let $W$ be the $n \times n$-matrix
\[
\tup{ \ive{ i + j = n + 1 } } _{1\leq i\leq n,\ 1\leq j\leq n}
=
\begin{pmatrix}
0 & \cdots & 0 & 0 & 1\\
0 & \cdots & 0 & 1 & 0\\
0 & \cdots & 1 & 0 & 0\\
\vdots & \iddots & \vdots & \vdots & \vdots\\
1 & \cdots & 0 & 0 & 0
\end{pmatrix}
\in \KK^{n \times n}
\]
(where we are using
\href{https://en.wikipedia.org/wiki/Iverson_bracket}{the
Iverson bracket notation} again).

Prove that $W = W^{-1}$.

\item[\textbf{(c)}] Let
$A = \tup{ a_{i,j} }_{1\leq i\leq n,\ 1\leq j\leq n}
\in \KK^{n \times n}$ be any $n \times n$-matrix.
Prove that
\[
WAW^{-1}
= \tup{ a_{n+1-i,n+1-j} }_{1\leq i\leq n,\ 1\leq j\leq n}.
\]
(In other words, $WAW^{-1}$ is the $n \times n$-matrix obtained from $A$ by
reversing the order of the rows and also reversing the order of the columns.)

\end{enumerate}

\subsection{Remark}

The map
\[
\LL \to \LL , \qquad a \mapsto waw^{-1}
\]
in part \textbf{(a)} of this exercise is called \textit{conjugation by $w$}.
It is best known in the case of a matrix ring, where it corresponds to a
change of basis for an endomorphism of a vector space. When $\KK$ is a
field, the \textbf{only} ring isomorphisms $\KK^{n \times n}
\to \KK^{n \times n}$ are conjugations by invertible matrices;
this is the Noether--Skolem theorem (in one of its less general variants).

\subsection{Solution}

[...]

%----------------------------------------------------------------------------------------
%	EXERCISE 3
%----------------------------------------------------------------------------------------
\horrule{0.3pt} \\[0.4cm]

\section{Exercise 3: Entangled inverses}

Let $\KK$ be a ring.

A \textit{left inverse} of an element $x \in \KK$
is defined to be a $y \in \KK$ such that $yx = 1$.

A \textit{right inverse} of an element $x \in \KK$
is defined to be a $y \in \KK$ such that $xy = 1$.

Let $a$ and $b$ be two elements of $\KK$.
Prove the following:

\begin{enumerate}

\item[\textbf{(a)}] If $c$ is a left inverse of $1 - ab$,
then $1 + bca$ is a left inverse of $1 - ba$.

\item[\textbf{(b)}] If $c$ is a right inverse of $1 - ab$,
then $1 + bca$ is a right inverse of $1 - ba$.

\item[\textbf{(c)}] If $c$ is an inverse of $1 - ab$,
then $1 + bca$ is an inverse of $1 - ba$.

\end{enumerate}

Here and in the following, the word ``\textit{inverse}''
(unless qualified with an adjective) means
``multiplicative inverse''.

\subsection{Solution}

[...]

%----------------------------------------------------------------------------------------
%	EXERCISE 4
%----------------------------------------------------------------------------------------
\horrule{0.3pt} \\[0.4cm]

\section{Exercise 4: Composition of ring homomorphisms}

\subsection{Problem}

Let $\KK$, $\LL$ and $\MM$ be three rings.
Prove the following:

\begin{enumerate}
\item[\textbf{(a)}] If $f : \KK \to \LL$ and $g : \LL \to \MM$
are two ring homomorphisms, then $g \circ f : \KK \to \MM$
is a ring homomorphism.

\item[\textbf{(b)}] If $f : \KK \to \LL$ and $g : \LL \to \MM$
are two ring isomorphisms, then $g \circ f : \KK \to \MM$
is a ring isomorphism.
\end{enumerate}

\subsection{Solution}

[...]

%----------------------------------------------------------------------------------------
%	EXERCISE 5
%----------------------------------------------------------------------------------------
\horrule{0.3pt} \\[0.4cm]

\section{Exercise 5: Squares in finite fields I}

\subsection{Problem}

Let $\FF$ be a field.

\begin{enumerate}

\item[\textbf{(a)}] Prove that if $a, b \in \FF$ satisfy
$ab = 0$, then $a = 0$ or $b = 0$.

\item[\textbf{(b)}] Prove that if $a, b \in \FF$ satisfy
$a^2 = b^2$, then $a = b$ or $a = -b$.

\end{enumerate}

Recall that an element $\eta \in \FF$ is called a
\textit{square} if there exists some $\alpha \in \FF$
such that $\eta = \alpha^2$.

From now on, assume that $2 \cdot 1_{\FF} \neq 0_{\FF}$ (that is,
$1_{\FF} + 1_{\FF} \neq 0_{\FF}$). Note that this is
satisfied whenever $\FF = \ZZ / p$ for a prime $p > 2$ (but also for
various other finite fields), but fails when $\FF = \ZZ / 2$.

\begin{enumerate}

\item[\textbf{(c)}] Prove that $a \neq -a$ for every
nonzero $a \in \FF$.

\end{enumerate}

From now on, assume that $\FF$ is finite.

\begin{enumerate}

\item[\textbf{(d)}] Prove that the number of squares in $\FF$ is
$\dfrac{1}{2} \tup{ \abs{\FF} + 1 }$.

\item[\textbf{(e)}] Conclude that $\abs{\FF}$ is odd.

\end{enumerate}

[\textbf{Hint:} For part \textbf{(d)}, argue that each nonzero square in
$\FF$ can be written as $\alpha^2$ for exactly two $\alpha \in \FF$.]

\subsection{Solution}

[...]

%----------------------------------------------------------------------------------------
%	EXERCISE 6
%----------------------------------------------------------------------------------------
\horrule{0.3pt} \\[0.4cm]

\section{Exercise 6: The characteristic of a field}

\subsection{Problem}

Let $\FF$ be a field.
Recall that we have defined $na$ to mean
$\underbrace{a + a + \cdots + a}_{n \text{ times}}$ whenever
$n \in \NN$ and $a \in \FF$.

Assume that there exists a positive integer $n$ such that
$n \cdot 1_{\FF} = 0$.
Let $p$ be the \textbf{smallest} such $n$.

Prove that $p$ is prime.

[\textbf{Hint:}
$\tup{a \cdot 1_{\FF}} \cdot \tup{b \cdot 1_{\FF}}
= ab \cdot 1_{\FF}$
for all $a, b \in \NN$.]

\subsection{Remark}

The $p$ we just defined is called the \textit{characteristic} of the field
$\FF$ when it exists. (Otherwise, the characteristic of the field
$\FF$ is defined to be $0$.)

Thus, for each prime $p$, the finite field $\ZZ / p$, as well as the
finite field of size $p^2$ that we constructed in class,
have characteristic $p$.

\subsection{Solution}

[...]

\begin{thebibliography}{99999999}                                                                                %

% Feel free to add your sources -- or copy some from the source code
% of the class notes ( http://www.cip.ifi.lmu.de/~grinberg/t/19s/notes.tex ).

% This is the bibliography: The list of papers/books/articles/blogs/...
% cited. The syntax is: "\bibitem[name]{tag}Reference",
% where "name" is the name that will appear in the compiled
% bibliography, and "tag" is the tag by which you will refer to
% the source in the TeX file. For example, the following source
% has name "GrKnPa94" (so you will see it referenced as
% "[GrKnPa94]" in the compiled PDF) and tag "GKP" (so you
% can cite it by writing "\cite{GKP}").

% I've commented the below references out, since I'm not citing
% them above.

% \bibitem[GrKnPa94]{GKP}Ronald L. Graham, Donald E. Knuth, Oren Patashnik,
% \textit{Concrete Mathematics, Second Edition}, Addison-Wesley 1994.\\
% See \url{https://www-cs-faculty.stanford.edu/~knuth/gkp.html} for errata.

\bibitem[Grinbe19]{detnotes}Darij Grinberg,
\textit{Notes on the combinatorial fundamentals of algebra},
10 January 2019. \\
\url{http://www.cip.ifi.lmu.de/~grinberg/primes2015/sols.pdf}
\\
The numbering of theorems and formulas in this link might shift
when the project gets updated; for a ``frozen'' version whose
numbering is guaranteed to match that in the citations above, see
\url{https://github.com/darijgr/detnotes/releases/tag/2019-01-10} .

\end{thebibliography}

\end{document}


\section{Results} \label{sec:results}
In this section, I present the results of the experiments and answer the two research questions.
\subsection{RQ1 Results: CPD Performance}
\begin{sidewaystable}
% \begin{table*}
\caption{Change point detection precision, recall, and F1-score calculated for the baseline methods using three values of tolerance ($\tau$) for multiple configurations.}
\label{tab:rq1-results}
\resizebox{\columnwidth}{!}{%
\begin{tabular}{llcccccccccc}
\toprule
  \textbf{Cost Function} &
  \textbf{Search Method} &
  \textbf{Penalty} &
  \textbf{Prec.} &
  \textbf{Recall} &
  \textbf{F1} &
  \textbf{Prec.} &
  \textbf{Recall} &
  \textbf{F1} &
  \textbf{Prec.} &
  \textbf{Recall} &
  \textbf{F1} \\
                                                  &              &   & \multicolumn{3}{c}{$\tau=1$s} & \multicolumn{3}{c}{$\tau=3$s} & \multicolumn{3}{c}{$\tau=5$s} \\  \toprule
\multirow{2}{*}{\textbf{Autoregressive Model}}     & Bottom Up    & 1000 & 10.43\%    & 75.44\%    & 18.33\%   & 21.21\%    & 80.32\%    & 33.55\%   & 28.94\%    & 81.22\%    & 42.68\%   \\
                                                   & Window Based & 100  & 2.94\%     & 3.98\%     & 3.38\%    & 8.53\%     & 11.41\%    & 9.76\%    & 12.89\%    & 17.54\%    & 14.86\%   \\ \midrule
\multirow{2}{*}{\textbf{Least Absolute Deviation}} & Bottom Up    & 500  & 7.32\%     & 52.54\%    & 12.85\%   & 17.52\%    & 87.73\%    & 29.20\%   & 25.02\%    & 88.95\%    & 39.05\%   \\
                                                   & Window Based & 500  & 5.24\%     & 8.31\%     & 6.42\%    & 15.20\%    & 24.03\%    & 18.62\%   & 21.79\%    & 38.25\%    & 27.76\%   \\ \midrule
\multirow{2}{*}{\textbf{Least Squared Deviation}}  & Bottom Up    & 1000 & 7.44\%     & 85.09\%    & 13.68\%   & 16.40\%    & 89.81\%    & 27.74\%   & 24.16\%    & 90.47\%    & 38.14\%   \\
                                                   & Window Based & 500  & 3.59\%     & 6.79\%     & 4.70\%    & 10.27\%    & 16.51\%    & 12.66\%   & 16.18\%    & 26.84\%    & 20.19\%   \\ \midrule
\multirow{2}{*}{\textbf{Linear Model Change}}      & Bottom Up    & 100  & \textbf{37.59\%}    & 28.98\%    & \textbf{32.73\%}   & \textbf{45.20\%}    & 38.39\%    & \textbf{41.52\%}   & \textbf{48.07\%}    & 41.36\%    & \textbf{44.46\% }  \\
                                                   & Window Based & 500  & 6.70\%     & 4.14\%     & 5.12\%    & 20.50\%    & 13.05\%    & 15.95\%   & 38.78\%    & 26.77\%    & 31.67\%   \\ \midrule
\multirow{2}{*}{\textbf{Gaussian Process Change}}  & Bottom Up    & 100  & 3.77\%     & \textbf{92.23\%}    & 7.25\%    & 8.99\%     & \textbf{92.23\%}    & 16.39\%   & 13.53\%    & \textbf{92.23\%}    & 23.60\%   \\
                                                   & Window Based & 100  & 2.94\%     & 3.95\%     & 3.37\%    & 8.69\%     & 11.50\%    & 9.90\%    & 13.64\%    & 18.30\%    & 15.63\%   \\ \midrule
\multirow{2}{*}{\textbf{Rank-based Cost Function}} & Bottom Up    & 100  & 13.45\%    & 60.19\%    & 21.98\%   & 19.49\%    & 80.10\%    & 31.35\%   & 22.98\%    & 87.23\%    & 36.38\%   \\
                                                   & Window Based & 100  & 8.10\%     & 13.70\%    & 10.18\%   & 15.72\%    & 30.73\%    & 20.80\%   & 21.38\%    & 46.64\%    & 29.32\%   \\ \midrule
\multirow{2}{*}{\textbf{Kernelized Mean Change}}   & Bottom Up    & 100  & 4.13\%     & 3.24\%     & 3.63\%    & 12.22\%    & 8.14\%     & 9.77\%    & 15.38\%    & 10.58\%    & 12.54\%   \\
                                                   & Window Based & 100  & 2.82\%     & 3.00\%     & 2.91\%    & 10.14\%    & 8.40\%     & 9.19\%    & 13.64\%    & 12.61\%    & 13.10\%   \\ \bottomrule

\end{tabular}%
}
% \end{table*}
\end{sidewaystable}

Table~\ref{tab:rq1-results} shows the results of running CPD algorithms for various configurations (as described in \ref{sec:CPD_baseline}). For each search method and cost function pair only one of the penalty values which resulted in the highest F1 scores for all $\tau$ values is reported.

The first observation from the results is that as values of $\tau$ increases the scores get better. This was expected, since larger values relax the constraints on which detected change points are considered as a true positive. Another observation is that the bottom-up segmentation consistently outperforms the window-based segmentation method. I can also see that the linear cost function beats all the other ones, in terms of precision. The Gaussian cost function achieves much higher recall values costing it a huge loss in precision. It means this cost function results in detecting numerous change points spread across the time axis, so there is a good chance of having at least one change point predicted close to each true change point (hence the high recall), but also there are a lot of false positives, which leads to a low precision.

\begin{table}
\caption{Change point detection precision, recall, and F1-score calculated, on the test data, for the proposed model, using three values of tolerance ($\tau$) compared with the respective $\tau$'s best F1 score among baseline methods}
\label{tab:rq1-2-results}
% begin{tabular}{cccc}
\begin{tabularx}{\columnwidth}{cXXXX}
\toprule
  $\mathbf{\tau}$ &
  \textbf{Prec.} &
  \textbf{Recall} &
  \textbf{F1 score} &
  \textbf{Baseline F1}  \\ \midrule
1s & 56.77\% &	79.32\% &	66.18\% & 32.73\% \\
3s & 69.58\% &	88.88\% &	78.06\% & 41.52\% \\
5s & 79.82\% &	91.87\%	&   85.42\% & 44.46\% \\\bottomrule
\end{tabularx}
%\end{tabular}
\end{table}

Measuring the same metrics on how the proposed model performs on the test data shows better scores, almost twice the F1 score of the best performing baseline (see Table~\ref{tab:rq1-2-results}).
Please note that unlike machine learning algorithms (such as mine), CPD algorithms do not have a separate training and testing phases. This fact works in their favor (by using the entire dataset for prediction and not just the training set), but still my model outperforms them.

%Although measuring the time it takes to detect change points was not one of the comparison metrics it would not hurt to note that
In terms of execution cost, running all 42 different settings of CPD algorithms on the whole dataset took a bit over 12 hours in the cloud using 16 CPUs and 64GB of main memory. The deep learning model on the other hand takes about an hour to train (which only needs to be done once), on a smaller machine (see section~\ref{sec:machines_config}). It made predictions on the whole dataset in less than a minute.
%$(/)-1=
So to answer RQ1, my method has shown $(66.18/32.73)-1=102.20\%$ improvement in F1 score with $\tau=1s$, $(78.06/41.52)-1=88.00\%$ with $\tau=3s$, and $(85.42/44.46)-1=92.13\%$ with $\tau=5s$; almost doubling the score compared to the baselines.

\begin{rqanswer}
The proposed model, which requires less memory compared to traditional CPD algorithms, improved their best performance by up to 102\%, measured by F1 score, in less execution time.
\end{rqanswer}

\subsection{RQ2 Results: Multi-class Classification Performance}
To answer RQ2, I first compare different configurations of the baseline methods using the F1 score (harmonic mean of precision and recall) on the test data. The results are presented in Table~\ref{tab:rq2-1-results}.

\begin{table}
\caption{Precision, recall, and F1 score of ridge classifiers (linear classifiers with L2 regularization) and decision tree classifiers (DT) with different sliding window widths ($w$). For each algorithm on each $w$ several hyper-parameters were applied producing 152 different models. In this table, I only show the results of the best performing model in each group.}
\label{tab:rq2-1-results}
\resizebox{\linewidth}{!}{%
\begin{tabular}{clcclll}
\toprule
\textbf{w} &
  \multicolumn{1}{c}{\textbf{Classifier}} &
  \multicolumn{1}{c}{\textbf{\begin{tabular}[c]{@{}c@{}}Max\\ Depth\end{tabular}}} &
  \multicolumn{1}{c}{\textbf{\begin{tabular}[c]{@{}c@{}}Max\\ Features\end{tabular}}} &
  \multicolumn{1}{c}{\textbf{Prec.}} &
  \multicolumn{1}{c}{\textbf{Recall}} &
  \multicolumn{1}{c}{\textbf{F1}} \\ \midrule
3  & Ridge & -   & -          & 71.39\% & 20.73\% & 32.13\% \\
3  & DT    & -   & -          & 69.21\% & 82.36\% & 75.21\% \\ \midrule
5  & Ridge & -   & -          & 69.15\% & 21.89\% & 33.26\% \\
5  & DT    & 100 & -          & 68.37\% & \textbf{83.16\%} & 75.04\% \\ \midrule
10 & Ridge & -   & -          & 71.97\% & 24.02\% & 36.02\% \\
10 & DT    & 260 & -          & 67.94\% & 79.14\% & 73.12\% \\ \midrule
15 & Ridge & -   & -          & 76.87 & 25.90\% & 38.75\% \\
15 & DT    & -   & $\sqrt{10w}$ & 69.06\% & 80.76\% & 74.45\% \\ \midrule
20 & Ridge & -   & -          & \textbf{80.38\%} & 26.50\% & 39.86\% \\
20 & DT    & 175 & $\sqrt{10w}$ & 73.21\% & 82.16\% & \textbf{77.42\%} \\  \midrule
   & \multicolumn{3}{l}{My Approach} & \textbf{86.29\%} & \textbf{95.04\%} & \textbf{90.45\%} \\
\bottomrule
\end{tabular}%
}
\end{table}

Comparing the baseline methods with my approach (the last row) in Table~\ref{tab:rq2-1-results} shows that my model outperforms all baselines. Comparing it with the model with the best F1-score shows a $(86.29/73.21)-1=17.87\%$ improvement in precision as well as a $(95.04/82.16)-1=15.68\%$ improvement in recall that means $(90.45/77.42)-1=16.83\%$ overall improvement in F1-score.

To have a feeling of how good my predictions are in practice, Figure~\ref{fig:test_0} shows the output of my model side by side with the ground truth. The horizontal axis shows sample ID (time) and the states are color coded. As it is seen, this algorithm performs better when the state changes are farther apart. Also there are some state changes that happen quite briefly which are not detected. That is not to a great surprise since it takes some time for state changes to be reflected in the outputs and those might not have got any chance.

The classical models only see one window of the data at a time, convolutional layers on the other hand are more generalized and flexible since each filter in each layer is comparable to a sliding window. As we saw in Table~\ref{tab:rq2-1-results}, a larger window size means a higher performance. However, it gets significantly more difficult to train a model with large window sizes. In addition, convolutions can automatically learn preprocessing steps that could be beneficial such as a moving average. Each convolutional filter can learn a linear combination of its inputs. So when the convolutional layers are stacked on each other, with non-linear activation functions in between, the hypothesis space they can learn becomes quite large, probably much larger than most of the classical ML algorithms here. Also, they are still quite efficient (more efficient than baselines) due to parameter sharing and their high parallelizability.

The fact that the performance improves as the window size increases indicates the positive effect of being able to see longer-term relations in detecting the system's state. Recurrent cells (such as GRU) can capture long-term dependencies (that do not necessarily fall into one window) and learn sequences. This is one of the major differences between an RNN model and others, such as decision trees, which do not have such a notion of a ``long-term memory'' as LSTM/GRU neural networks do. All a decision tree could see is the values in a sliding window.

In terms of the training complexity (time and memory), this method is superior as well. That can largely be attributed to the use of deep learning. In baseline models, as the window size $w$ grows the training and evaluation complexity grows, up to a point that they ran out of memory -- consuming all the 47GB of main memory and swap area. This forced us to train them in the cloud. Meanwhile, as mentioned earlier, the deep learning model could be trained on a 8GB GPU in roughly an hour. (see section \ref{sec:machines_config} for the machines' specs). Also, the decision tree training was not parallelized using only one core of the CPU, while virtually all deep learning models can be heavily parallelized on a GPU/TPU. %All the reported scores for both the baseline models and my model are computed on previously unseen test data.

\begin{rqanswer}
The proposed model, which requires less than half as many CPUs and 70\% as much memory compared to the best performing classical ML model, improved their best performance by up to 17\%, measured by F1 score, in less execution time.
\end{rqanswer}

\begin{figure*}
    \centering
    \includegraphics[width=\linewidth]{ASE_files/test_0.png}
    % \Description[States in time diagram]{A figure that color-codes the true and predicted states}
    \caption{Evaluation of the model on 30 random test data. Each graph shows the states in one run of the system. The colors show the states. The top-half of each plot depicts model's prediction of the system states ($\hat{O}$) and the bottom-half shows the true labels($O$). Since the output is one-hot encoded, the item with the most probability is used as the predicted label at each point in time. X-axis is the time axis. Only the first 600 samples (2 minute of simulation) are shown to improve legibility.}
    \label{fig:test_0}
\end{figure*}

\subsection{RQ3 Results: CPD Performance on Paparazzi (replication)}
The results of applying baseline algorithms on Paparazzi dataset can be seen in table~\ref{tab:cpd_paparazzi}.
The range of CPD performance is quite varied across the techniques, for example there are scores less then 1\% (e.g. 0.39\% recall score of linear models) as well as some 100\% scores (e.g. recall scores for Gaussian models with bottom-up search method). Please note that the high variation in performance scores is not limited to recall metric.
The 100\% recalls are accompanied with a low precision; achieving that is not difficult. A method that outputs every point as a change-point will get similar results.
The setting that got most of the best numbers is Pelt algorithm using an L1 cost function and a high penalty coefficient of 1,000. However, please note that Pelt is a quite slow algorithm that could easily become infeasible to run on larger dataset, as it was the case for the first two RQs.
As a matter of fact, running Pelt took more than 14 hours on the same machine that performed all other CPD algorithms in less than 1 hour. What you see in table~\ref{tab:cpd_paparazzi} is the summary of the results of more than 23,800 experiments.

Comparing the baseline results with the neural network model's results in the last three rows of the table, you can see improvements of 47.88\%, 34.81\%, and 18.30\% in F1 scores compared to the best numbers in the baselines (for $\tau = 1, 3, 5$ seconds respectively). Although this model still performed better than the baselines, the improvement margin was higher in RQ1. (see the RQ1 results summary box and the paragraph above that). % in section~\ref{sec:results}).
However, you can see that the scores that this approach achieved are not low, but the baselines could do a better job on Paparazzi's data compared to MicroPilot's, therefore shrinking the margin of improvement.
It can be due to the fact that the new dataset is tiny compared to MP, classic machine learning algorithms can do better so a deep learning model cannot shine here as it could for MP.
As a reminder: Paparazzi dataset consists of ~300 tests, each containing up to 2,500 samples vs. ~900 tests in MP with lengths up to 18,000 samples.
It is worth mentioning that both in the first two RQs and here, the variation in baselines performances was way higher while my deep learning approach shows a more consistent performance; check the second column (recall with $\tau=1$s) for example.

\begin{rqanswer}
The proposed approach was applied on another similar software system with a smaller dataset size and showed a near 48\% improvement over the baselines (with a 94\% F1 score), confirming that it is a feasible approach even with smaller dataset size. The lower margin of improvement compared to RQ1 is attributed to the baselines performing better on this data, rather than my approach not performing good enough.
\end{rqanswer}

\begin{sidewaystable}
    \centering
\resizebox{\columnwidth}{!}{%
\begin{tabular}{llcccccccccc}
\toprule
  \textbf{Cost Function} &
  \textbf{Search Method} &
  \textbf{Penalty} &
  \textbf{Prec.} &
  \textbf{Recall} &
  \textbf{F1} &
  \textbf{Prec.} &
  \textbf{Recall} &
  \textbf{F1} &
  \textbf{Prec.} &
  \textbf{Recall} &
  \textbf{F1} \\
                                                  &              &   & \multicolumn{3}{c}{$\tau=1$s} & \multicolumn{3}{c}{$\tau=3$s} & \multicolumn{3}{c}{$\tau=5$s} \\  \toprule
\multirow{3}{*}{\textbf{Autoregressive Model}}
    & Bottom Up    & 500  & 13.65\% &    14.13\% & 18.89\% & 39.73\% &     36.40\% & 36.14\% & 57.11\% &     53.20\% & 50.04\% \\
    & Pelt         & 500  & 13.56\% &    13.85\% & 19.03\% & 39.67\% &     36.17\% & 36.03\% & 56.97\% &     52.89\% & 49.83\% \\
    & Window Based & 100  & 15.97\% &     7.59\% & 13.54\% & 45.47\% &     20.40\% & 29.56\% & 64.16\% &     30.76\% & 41.20\% \\ \midrule
\multirow{3}{*}{\textbf{Least Absolute Deviation}}
    & Window Based & 100  & 24.50\% &    15.92\% & 19.85\% & 56.39\% &     37.24\% & 44.53\% &        68.94\% &     48.20\% & 56.23\% \\
    & Pelt         & 1000 &\textbf{26.74\%}&34.08\% & 29.92\% &\textbf{60.77\%}&     77.32\% &\textbf{67.74\%}&\textbf{74.73\%}&     93.16\% &\textbf{82.61\%}\\
    & Bottom Up    & 1000 & 26.66\% &\textbf{34.99\%}& 30.35\% &        60.11\% &     78.14\% & 67.61\% &        72.85\% &     92.94\% & 81.30\% \\ \midrule
\multirow{3}{*}{\textbf{Least Squared Deviation}}
    & Window Based & 1000 & 25.01\% &    16.14\% & 20.26\% & 54.58\% &     35.77\% & 42.96\% & 68.73\% &     48.11\% & 56.18\% \\
    & Pelt         & 1000 & 21.28\% &    80.04\% & 33.49\% & 51.05\% &     99.58\% & 67.20\% & 65.36\% & \textbf{99.97\%} & 78.76\% \\
    & Bottom Up    & 1000 & 21.30\% &    81.24\% & \textbf{33.62\%}  &     50.98\% &\textbf{99.50\%}& 67.12\% &        65.24\% &\textbf{99.97\%}& 78.66\% \\ \midrule
\multirow{3}{*}{\textbf{Linear Model Change}}
    & Bottom Up    & 100  &  7.39\% &     0.39\% &  9.98\% & 27.44\% &      1.49\% & 10.27\% & 52.51\% &      2.75\% &  9.90\% \\
    & Window Based & 100  &  7.39\% &     0.39\% &  9.98\% & 27.44\% &      1.49\% & 10.27\% & 52.51\% &      2.75\% &  9.90\% \\
    & Pelt         & 100  &  7.39\% &     0.39\% &  9.98\% & 27.44\% &      1.49\% & 10.27\% & 52.51\% &      2.75\% &  9.90\% \\ \midrule
\multirow{3}{*}{\textbf{Gaussian Process Change}}
    & Window Based & 100  &  7.39\% &     0.39\% &  9.98\% & 27.44\% &      1.49\% & 10.27\% & 52.51\% &      2.75\% &  9.90\% \\
    & Pelt         & 100  &  7.39\% &     0.39\% &  9.98\% & 27.44\% &      1.49\% & 10.27\% & 52.51\% &      2.75\% &  9.90\% \\
    & Bottom Up    & 100  & 12.42\% &\textbf{100.00\%}& 22.03\% &33.15\% &\textbf{100.00\%}& 49.55\% &        49.04\% &    \textbf{100.00\%} & 65.47\% \\ \midrule
\multirow{2}{*}{\textbf{Rank-based Cost Function}}
    & Pelt         & 100  & 21.11\% &    31.85\% & 25.51\% & 52.29\% &     74.63\% & 61.13\% & 65.88\% &     89.30\% & 75.46\% \\
    & Bottom Up    & 100  & 20.55\% &    31.84\% & 25.17\% & 49.97\% &     73.22\% & 59.04\% & 64.57\% &     89.35\% & 74.61\% \\ \midrule
\multirow{3}{*}{\textbf{Kernelized Mean Change}}
    & Bottom Up    & 100  & 15.66\% &     1.90\% &  9.45\% & 44.23\% &      5.36\% & 12.95\% & 62.37\% &      7.48\% & 15.66\% \\
    & Pelt         & 100  & 14.07\% &     1.64\% &  9.77\% & 42.06\% &      4.74\% & 12.59\% & 60.92\% &      6.67\% & 14.56\% \\
    & Window Based & 100  &  8.31\% &     0.61\% &  9.97\% & 29.38\% &      2.02\% & 10.60\% & 53.34\% &      3.39\% & 10.79\% \\ \midrule \midrule
\multirow{3}{*}{\textbf{My Approach}}
    & \multirow{3}{*}{\textbf{Dataset:}}
         & \textbf{Validation}    &  44.70\% &    52.38\% & 48.24\% &   84.56\% &     90.54\% & 87.45\% &   90.53\% &     96.86\% & 93.59\% \\
    & {} & \textbf{Test (Tuning)} &  42.56\% &    48.26\% & 45.23\% &   87.11\% &     89.68\% & 88.37\% &   92.54\% &     96.48\% & 94.47\% \\
    & {} & \textbf{Training}      &  56.43\% &    62.53\% & 59.32\% &   89.64\% &     92.79\% & 91.19\% &   93.97\% &     97.93\% & 95.91\% \\
\bottomrule
\end{tabular}%
}
    \caption{Change point detection methods performance on Paparazzi dataset. Since the 100\% recalls are actually outliers, the next largest recall values are in bold face as well.}
    \label{tab:cpd_paparazzi}
\end{sidewaystable}

\subsection{RQ4 Results: Multi-class Classification Performance on Paparazzi (replication)}
In table~\ref{tab:rq2-1-results-replicate} the comparative results of the aforementioned algorithms with my method is presented.
To compare with the MP's case, we can see that in all but two settings limiting the maximum number of features did not help improve the model.
Another similar observation is that the scores do not vary much with changing the window size, and decision trees almost universally outperform linear classifiers.
The scores themselves are just around the same values as well: Ridge classifier F1 score here is in 21-29\% range, which was 32-39\% in RQ2. For decision trees it is in 67-68\% range here while it is in 73-77\% range in RQ2.

The deep learning model's performance measures (on validation data) can be found in the last row of the table.
You can see an $(82.23/68.96)-1=19.24\%$ improvement in F1 score, over the baselines. It confirms that this method can beat the best performing baselines in classifying input samples into the system's internal states. The improvement in RQ2 is 16.83\%, not very different from this. This, again provides some evidence on the generalizability of the results in the first two RQs.

\begin{table}
\caption{Precision, recall, and F1 score of ridge classifiers (linear classifiers with L2 regularization) and decision tree classifiers with different sliding window widths ($w$). In this table, I only show the results of the best performing model in each group.}
\label{tab:rq2-1-results-replicate}
\resizebox{\linewidth}{!}{%
\begin{tabular}{llcccc}
\toprule
\textbf{w} &
  \multicolumn{1}{c}{\textbf{Classifier}} &
  \multicolumn{1}{c}{\textbf{Max Features}} &
  \multicolumn{1}{c}{\textbf{Precision}} &
  \multicolumn{1}{c}{\textbf{Recall}} &
  \multicolumn{1}{c}{\textbf{F1}} \\ \midrule
 3 &  Ridge &            - &      44.82\% &   14.45\% & 21.85\% \\
 3 & Decision Tree & $\sqrt{10w}$ &      61.88\% &   73.56\% & 67.22\% \\ \midrule
 5 &  Ridge &            - &      46.62\% &   15.10\% & 22.81\% \\
 5 & Decision Tree &            - &      64.28\% &   73.00\% & 68.36\% \\ \midrule
10 &  Ridge &            - &      47.59\% &   16.29\% & 24.27\% \\
10 & Decision Tree &            - &\textbf{65.06\%}& 73.36\% &\textbf{68.96\%}\\ \midrule
15 &  Ridge &            - &      44.41\% &   17.33\% & 24.93\% \\
15 & Decision Tree &            - &      63.53\% &   74.68\%& 68.65\% \\ \midrule
20 &  Ridge &            - &      57.97\% &   19.10\% & 28.74\% \\
20 & Decision Tree &            - &      64.72\% &   73.36\% & 68.77\% \\ \midrule
25 &  Ridge &            - &      62.13\% &   19.66\% & 29.87\% \\
25 & Decision Tree & $\sqrt{10w}$ &      61.17\% &\textbf{75.18}\% & 67.45\% \\ \midrule \midrule
   & \multicolumn{2}{l}{Proposed Method} & \textbf{77.20\%} & \textbf{87.97\%} & \textbf{82.23\%} \\
\bottomrule
\end{tabular}%
}
\end{table}

%They say a picture is worth a thousand words\footnote{missing citation},
Figure~\ref{fig:paparazzi_predictions} summarizes the prediction results of the deep learning model compared to the ground truth. You can get a sense of how good does an `82.23\%' F1 score look by examining that chart. To measure this quality it in a more scientific way, it is possible to use this state model in a downstream task and see how that task is performing.
\begin{figure}
    \centering
    \includegraphics[width=\columnwidth]{RQ3-5_charts/states_chart.png}
    \caption{The states are color coded and each rectangle shows one test, only the first 1000 samples (200 seconds) are shown for more clarity. The same as figure~\ref{fig:test_0} in, the top half of each rectangle shows the model's output and the bottom half shows the true labels.}
    \label{fig:paparazzi_predictions}
\end{figure}

\begin{rqanswer}
Compared to the baselines, the proposed approach could detect the internal state of system with a 77\% precision and 88\% recall rate, showing a 19\% improvement. Similar results were seen in RQ 2 (which is closely related to this question) confirming generalizability of the findings.
\end{rqanswer}

\subsection{RQ5 Results: Hyper-parameter Tuning}
I used Tensor Board to visualize and compare the effects of different hyper-parameters on the model's performance. First, for a sanity check, I visualized 6 metrics, precision and recall in detecting change points with a the smallest and largest tolerances of $\tau=1s$ and $\tau=5s$, and precision and recall in estimating the internal state in Figure~\ref{fig:precision_recall_matrix}. A healthy linear and positive correlation is visible between virtually all pairs of these metrics (all except classification precision), it means that trying to improve one will not come at the expense of others.
\begin{figure}
    \centering
    \includegraphics[width=\columnwidth]{RQ3-5_charts/prec_recall_matrix_tuning_white_background.png}
    \caption{Matrix of scatter plots comparing precision and recall of the deep learning model on test data.}
    \label{fig:precision_recall_matrix}
\end{figure}

Filtering the data to find the commonalities of better hyper-parameters shows that the number of GRU cells has a high correlation with both CPD precision and classification precision.
First and foremost the learning rate is has the largest correlation (absolute) with the performance metrics. The best configurations all share the lowest learning rate.
Filtering the other configs out, the next most important factor was the convolutional layer counts.
The configurations that did not include the last and largest convolutional layer with kernel size of 20, performed much better than the others.
% It also shows that the kernel sizes and number of convolutional layers does not have an strong correlation with higher precision and recalls.
% The number of convolutional filters in each layer has a bigger impact on the performance than the size of the kernels does.
It makes sense because training larger kernels gets harder and harder and requires more data.
Also, based on the previous studies, especially in the field of computer vision, we can say that each filter learns one feature so it is more effective to have small filters that are easier to train and can learn multiple features of the data.
Therefore I can reduce the size of convolutional section of the model and use more recurrent cells to boost the performance without adding too many parameters.
%I performed another set of tests in a smaller search space to further optimize the hyper-parameters.

%The new search space includes all combinations of these hyper-parameters:

%\begin{enumerate}
%    \item Learning rate, options being: $3\times10^{-4}$, $1\times10^{-3}$, and $3\times10^{-3}$
%    \item Number of GRU cells, options: 128, 256, 384, and 512
%    \item Number of convolutional filters in each layer: 32, 64, and 72
%\end{enumerate}

\begin{figure}
    \centering
    \includegraphics[width=\columnwidth]{RQ3-5_charts/refined_prec_recall_white_background.png}
    \caption{Parallel coordinates view showing the effects of selected hyper-parameters on the performance measures.}
    \label{fig:precision_recall_parallel_coordinates}
\end{figure}
%The results of the second iteration of
After filtering out some of the worst configurations (such as the ones with lower than 40\% Recall (t=1s)), the better ones are shown in figure~\ref{fig:precision_recall_parallel_coordinates}.
This chart shows 3 + 8 parallel axes, 3 hyper-parameters and 8 metrics.
% To make it more legible, the results with lower than 70\% precision ($\tau = 5s$) are omitted.
The green highlighted line is the best configuration tried, with the highest area under the curve. That configuration has the best or close to the best performance in all metrics.
The configuration is to use 384 GRU cells in 1 layer, 16 filter per convolutional layer sizes from 5 to 15, and a slower learning rate of $3\times10^{-4}$.
I used this set of hyper-parameters to train the best model. The stopping criteria on training the model was for the validation loss to plateau, with a `patience' of 10 epochs. The optimizer converged in 54 epochs, after less than 3 minutes of training.
You can see the performance scores of the tuned model in tables~\ref{tab:rq2-1-results-replicate} and \ref{tab:cpd_paparazzi}.
% The scores used in the research question 3 and 4 are the evaluation results of this model on the [previously unseen] validation data.

The sensitivity of this approach to the choice of hyper parameters can be seen clearly here. I first started with the hyper-parameters that worked best for MP and the values close to them. With that set of hyper-parameters the performance was even worse than the baselines in some cases; see table~\ref{tab:results_no_tuning}.
In the second iteration, after removing the worst performing configurations and defining a search space, that set of parameters was discarded.
Even though the search space was not very large, the range of
This is to emphasize the effect and importance of fine-tuning hyper-params.

\begin{table}[]
    \centering
    \begin{tabular}{lcc}
    \toprule
\textbf{Evaluation Metric} & \textbf{Default Hyper-params} & \textbf{Tuned Hyper-params} \\\midrule
Precision ($\tau=1s$)      &  28.90\%  &  44.70\% \\
Recall ($\tau=1s$)         &  37.44\%  &  52.38\% \\
F1 ($\tau=1s$)             &  32.62\%  &  48.24\% \\ \midrule
Precision ($\tau=3s$)      &  59.34\%  &  84.56\% \\
Recall ($\tau=3s$)         &  71.52\%  &  90.54\% \\
F1 ($\tau=3s$)             &  64.87\%  &  87.45\% \\ \midrule
Precision ($\tau=5s$)      &  73.85\%  &  90.53\% \\
Recall ($\tau=5s$)         &  85.18\%  &  96.86\% \\
F1 ($\tau=5s$)             &  79.11\%  &  93.59\% \\ \midrule
State Detection Precision  &  57.35\%  &  77.20\% \\
State Detection Recall     &  78.90\%  &  87.97\% \\
State Detection F1         &  66.41\%  &  82.23\% \\
\bottomrule
    \end{tabular}
    \caption{The performance results of the model trained on paparazzi before hyper parameter tuning (i.e. using the hyper-parameters that worked best for MP on Paparazzi as well) vs. after fine-tuning hyper-params on Paparazzi's dataset.}
    \label{tab:results_no_tuning}
\end{table}

\begin{rqanswer}
The result of this tuning was a model with fewer convolutional layers and smaller kernel sizes but with more convolutional filters and recurrent layers that could perform better than the baselines. This shows having more and smaller convolutional filters is more effective in capturing local features in the data compared to having a small number of large filters (with kernels of size 20 for example which has 200 parameters). It also shows the important role that recurrent layers play in discovering long-term relations between inputs and outputs.
\end{rqanswer}

\section{Limitations and Threats to Validity} \label{sec:threats_to_validity}
One of the limitations of this approach is that it might miss an input-output invariant correlation. It can happen when the input remains constant or it changes too little to reveal its relation with certain outputs.
%In addition, as mentioned before, my approach relies on changes in signals.
We assume that during the data collection, sampling happens in regular intervals; this approach probably will have a hard time achieving high performances, working on unevenly spaced time-series data.
%Maybe the CPD baselines could perform better if they were boosted by feature engineering, but the feature engineering task is can quickly become an arduous and inefficient.
%It exists so it compensates for inability of traditional non-deep-learning methods in discovering meaningful features in the data. \cite{bengio2013deep}

Although labeling the input/output series for debugging, comprehension, ... purposes is a common task that developers do in the industry and in open source settings, performing the labeling on a large dataset can become arduos; thus making this approach less practical in some settings. However, as will be stated later, there are potential remedies to reduce the amount of required data to train a model. Also, it is worth mentioning that training this model needs to happen once. The labeling task can bee looked at as an investment; after the model is trained, it can be used in the future to automatically label the data without any need to more user input.

In terms of construct validity, I am using standard metrics to evaluate the results. However, the use of tolerance margin should be taken with caution since it is a domain-dependant variable and can change the final results. To alleviate this threats, I have used multiple margins and reported all results. In terms of internal validity threats, I reduced the threat by not implementing the CPD baselines by myself and rather reusing existing libraries. In terms of conclusion validity threats, I have used many (888) real test cases from MicroPilot's test repository and provided a proper train-validation-test split for training, tuning, and evaluation. Finally, in terms of external validity threats, this study suffers from being limited to only two case studies. However, (a) the studies are large-scale real-world systems with many test cases, and (b) they are both from industry and open source to be more representative. In the future one potential extension is to extend this work with more case studies from other domains, to increase its generalizability. Another external validity threat could be in the random split of training and validation tests. There is no straightforward way to make sure the validation dataset is 100\% different from the training data; however thee split was performed randomly as it is common in Machine Learning and there are no repetitive test cases in the dataset, so the chances are high that the validation dataset is different enough from the training data.


\documentclass[english,course]{lecture}
\usepackage{lipsum} % Only needed to generate dummy text for sample.tex
%
% First, provide some data about this document
\title{Main title of the lecture}
\subtitle{Subtitle of the lecture}
\shorttitle{Shortened title} % For headers; if undefined, the usual title will be used
\ccode{Code 7.45} % Most of these data are not compulsory
\subject{Subject of the Talk}
\author{Author's name}
\spemail{speaker@email.com}
\email{email@email.com}
\speaker{Speaker's name}
\date{02}{11}{2134}
\dateend{07}{11}{2134}
\conference{Lecture hall 7}
\place{University of Physics}
\flag{An extra line if you need it.}
\attn{Place anything here to gather your readers' attention. This could be a warning, a disclaimer, a license, or, more likely, some helpful suggestions for your readers.}
\morelink{vhbelvadi.com}
%
% And then begin your document
\begin{document}
\section{Example section}
\lecture[1 hour]{12}{06}{2017} % Mark new lectures if you like
Pick between \texttt{seminar}, \texttt{talk} or \texttt{course} (this document) to get appropriately different layouts. Seminars being shorter, for example, will not carry the contents section you see above but are otherwise mostly similar to courses. Talks are two-column layouts, which means you cannot have lecture numbers (on your right) or margin notes (see below). Use appropriate options. For more consult \url{http://vhbelvadi.com/latex-lecture-notes-class}.

\lipsum[1-2]
%
\section{Another section by itself}
\lipsum[3] Here, for example, is an \texttt{equation} environment:
\begin{equation}
\mathbf{v} = { \mathbf{s} \over t } \label{eq:velocity}
\end{equation}%
Nam nulla erat, elementum nec magna sit amet, vestibulum rhoncus augue. Praesent non fermentum nulla, quis blandit tortor.\margintext{This is some margin text and you can include such text anywhere in your notes.} Maecenas ut nisi condimentum nisi iaculis porttitor eu sed metus. Proin faucibus aliquet odio, ac lobortis tortor. Mauris porta molestie tortor blandit pretium. Nulla pulvinar id mauris ut efficitur. Donec posuere tortor a odio pellentesque tincidunt. Nulla mi nunc, accumsan nec lectus ut, euismod vulputate libero.
%
Maecenas eu hendrerit metus. Aenean consequat, ex a semper tristique, leo ipsum blandit dui, pharetra consectetur magna enim ac lectus. Pellentesque vel purus malesuada metus scelerisque aliquam. Sed finibus ex sit amet eros faucibus congue. Nulla ut dui egestas, dignissim neque ut, fringilla massa.
%
\subsection{This is a subsection by itself}
\lipsum[11]
%
And this is a nice \texttt{\$\$...\$\$} display environment:
$$
\Delta v = \int\displaylimits_{t_0}^{t_1} a \, \textrm{d}t
$$
Maecenas ut nisi condimentum nisi iaculis porttitor eu sed metus. Proin faucibus aliquet odio, ac lobortis tortor. Mauris porta molestie tortor blandit pretium. Nulla pulvinar id mauris ut efficitur. Donec posuere tortor a odio pellentesque tincidunt. Nulla mi nunc, accumsan nec lectus ut, euismod vulputate libero.
%
And finally we have the \texttt{align/align*} environment:
\begin{align}
x_f - x_i &= \bar{v}t \nonumber\\
\Rightarrow s &= \bar{v}t
\end{align}
\lipsum[6]
%
\section{Yet another section}
\subsection{And a subsection beneath it}
\lecture[1 hour]{13}{06}{2017}
\lipsum[7]
%
\subsection{And now a subsection}
\subsubsection{With a subsubsection following it}
%
Integer pharetra nulla scelerisque purus luctus iaculis. Mauris pulvinar erat non dui pretium, sed vestibulum sapien condimentum. Nam in urna quis sapien rhoncus placerat vitae sit amet odio. Vivamus finibus euismod nibh vestibulum lobortis.\margintext{These ideas were probably discussed in lecture 1 in a parallel universe.} Integer arcu tortor, vestibulum sit amet iaculis ut, ullamcorper non ante. Pellentesque consectetur nec odio quis placerat. Vestibulum vehicula massa vel euismod blandit.
%
\lipsum[8]
%
\margintext{\protect\vspace{2cm}Table \ref{tab:mori} courtesy of Mori, L.F. `Tables in \LaTeX2$\epsilon$: Packages and Methods'.}
\begin{table}[h]
\centering
\begin{tabular}{clcc}
\toprule
\multicolumn{2}{c}{$D$} & $P_u$ & $\sigma_N$\\
\multicolumn{2}{c}{(in)} & (lbs) & (psi)\\\toprule
\multirow{3}*{5} & test 1 & 285 & 38.00\\\cmidrule(l){2-4}
& test 2 & 287 & 38.27\\\cmidrule(l){2-4}
& test 3 & 230 & 30.67\\\midrule
\multirow{3}*{10} & test 1 & 430 & 28.67\\\cmidrule(l){2-4}
& test 2 & 433 & 28.87\\\cmidrule(l){2-4}
& test 3 & 431 & 28.73\\\bottomrule
\end{tabular}
\caption{A table beautified by the \protect\texttt{booktabs} package.}\label{tab:mori}
\end{table}
%
\lipsum[9]
%
\subsubsection{This subsubsection is all by itself}
\lipsum[12-14]
%
\vskip7ex
\centering
* * *
%
\end{document}%


\documentclass{article}
\usepackage{graphicx}

\begin{document}
	\title{Logic Gates}
	\author{Clement Obieke}

\maketitle
\newpage
\centering
\section*{Table Of Contents}

	\begin{enumerate}
		\item Introduction
		\item Types of Logic Gates and their Truth Tables
		\begin{itemize}
			\item AND Gate
			\item OR Gate
			\item NOT Gate
			\item NOR Gate
			\item NAND Gate
			\item XOR Gate
			\item XNOR Gate
		\end{itemize}
			\item Summary
	\end{enumerate}

\newpage
\section{Introduction}
	\raggedright
	\paragraph{}
	Logic gates were fabricated from an array of configurable switches, each consisting of a monolayer of redox-active rotaxanes sandwiched between metal electrodes~\cite{collier1999electronically}.

	A gate in a simple electronic componet that functions like a lightbulb.
	They function by combining multiple signals to produce a single signal
	based on their unique properties~\cite{okoacha2021logicgatesintro}.

\section{Types of Logic Gates}
		\paragraph{1}
		Fundamental gates are AND, OR , NOT:
		\begin{minipage}{\linewidth}
			\includegraphics[width = 0.3\linewidth]{and}
			\includegraphics[width = 0.3\linewidth]{or}
			\includegraphics[width = 0.37\linewidth]{not}
		\end{minipage}
		\paragraph{2}
		Derived Gates NAND, NOR, XOR and XNOR(derived from fundamental gates)
		\begin{minipage}{\linewidth}
			\includegraphics[width = 0.2\linewidth]{nand}
			\includegraphics[width = 0.3\linewidth, height=0.15\textheight]{nor}
			\includegraphics[width = 0.2\linewidth]{xor}
			\includegraphics[width = 0.2\linewidth]{xnor}
		\end{minipage}
		\paragraph{3}
		Universal Gates NAND and NOR gates(The fundamental gates can be realized through them).
		\begin{minipage}{\linewidth}
			\includegraphics[width = 0.36\linewidth]{nand}
			\includegraphics[width = 0.4\linewidth, height=0.23\textheight]{nor}
		\end{minipage}
referenced from~\cite{okoacha2021logicgatestypes}
\subsection{AND Gate}
	\includegraphics{and}
	\paragraph{}
		The \textbf{AND} operation produces a true output (result of 1) only for a single case when all of the input variables are 1 and a false output(result of 0) where one or more inputs are 0.
		\begin{table}[h]
			\centering
			\label{tab:table1}
			\caption{AND Logic Table}
			\begin{tabular}{|c|c|c|}
				INPUT 1 & INPUT 2 & OUTPUT\\
				\hline
				1&1&1\\
				1&0&0\\
				0&1&0\\
				0&0&0\\
			\end{tabular}
		\end{table}

\newpage
\subsection{OR Gate}
\includegraphics{or}
\paragraph{}
The \textbf{OR} operation produces a true output (result of 1)when any of the input variables are 1 and a false output(result of 0) where both inputs are 0.
\begin{table}[h]
	\centering
	\label{tab:table2}
	\caption{OR Logic Table}
	\begin{tabular}{|c|c|c|}
		INPUT 1 & INPUT 2 & OUTPUT\\
		\hline
		1&1&1\\
		1&0&1\\
		0&1&1\\
		0&0&0\\
	\end{tabular}
\end{table}

\newpage
\subsection{NOT Gate}
\includegraphics{not}
\paragraph{}
The \textbf{NOT} operation also known as a \textbf{logical inverter}, has only one input and produces the oppostite of the input signal.
\begin{table}[h]
	\centering
	\label{tab:table3}
	\caption{NOT Logic Table}
	\begin{tabular}{|c|c|}
		INPUT & OUTPUT\\
		\hline
		1&0\\
		0&1\\
	\end{tabular}
\end{table}

\newpage
\subsection{NOR Gate}
\includegraphics{nor}
\paragraph{}
The \textbf{NOR} operation produces the opposite result of an OR Gate.
\begin{table}[h]
	\centering
	\label{tab:table4}
	\caption{NOR Logic Table}
	\begin{tabular}{|c|c|c|}
		INPUT 1 & INPUT 2 & OUTPUT\\
		\hline
		1&1&0\\
		1&0&0\\
		0&1&0\\
		0&0&1\\
	\end{tabular}
\end{table}

\newpage

\subsection{NAND Gate}
\includegraphics[width=0.7\linewidth]{nand}
\paragraph{}
The \textbf{NAND} operation produces the opposite output of an AND Gate.
\begin{table}[h]
	\centering
	\label{tab:table5}
	\caption{NAND Logic Table}
	\begin{tabular}{|c|c|c|}
		INPUT 1 & INPUT 2 & OUTPUT\\
		\hline
		1&1&0\\
		1&0&1\\
		0&1&1\\
		0&0&1\\
	\end{tabular}
\end{table}

\newpage
\subsection{XOR Gate}
\includegraphics[width=0.8\linewidth]{xor}
\paragraph{}
The \textbf{XOR} operation produces a true output (result of 1) if either case of the input variables are 1 and a false output(result of 0) if both inputs are 0 or one at the same time.
\begin{table}[h]
	\centering
	\label{tab:table6}
	\caption{AND Logic Table}
	\begin{tabular}{|c|c|c|}
		INPUT 1 & INPUT 2 & OUTPUT\\
		\hline
		1&1&0\\
		1&0&1\\
		0&1&1\\
		0&0&0\\
	\end{tabular}
\end{table}

\newpage
\subsection{XNOR Gate}
\includegraphics[width=0.8\linewidth]{xnor}
\paragraph{}
The \textbf{XNOR} operation produces the opposite output of the operation XOR Gate
\begin{table}[h]
	\centering
	\label{tab:table7}
	\caption{XOR Logic Table}
	\begin{tabular}{|c|c|c|}
		INPUT 1 & INPUT 2 & OUTPUT\\
		\hline
		1&1&1\\
		1&0&0\\
		0&1&0\\
		0&0&1\\
	\end{tabular}
\end{table}
Referenced from~\cite{okoacha2021logicgatestype2}
\newpage
\section{Summary}

In a nutshell, Logic Gates are basic computer components used in the design of logical componets. They are used in various complex combinations and connections to perform operations. They are implemented in various technologies and from the foundation for computers.

\newpage

\bibliography{gst108}
\bibliographystyle{ieeetr}

\end{document}


\DIFaddbegin \chapter*{Nomenclature}

\section*{\DIFadd{Anacronyms}}
\begin{itemize}
\item	\DIFadd{AC -- Alternating Current
}\item	\DIFadd{AEMO -- Australian Energy Market Operator
}\item	\DIFadd{ARENA -- Australian Renewable Energy Agency
}\item	\DIFadd{DC -- Direct Current
}\item	\DIFadd{DER -- distributed energy resource
}\item	\DIFadd{DR -- demand response
}\item	\DIFadd{EBB -- Empirical Bernstein Bound
}\item	\DIFadd{EEBB -- Unioned Engineered EBB sampling method
}\item	\DIFadd{EV -- electric vehicle
}\item	\DIFadd{FCAS -- Frequency Control Ancillary Services
}\item	\DIFadd{FTR -- Financial Transmission Rights
}\item	\DIFadd{GHG -- greenhouse gasses
}\item	\DIFadd{GNK -- Generalized Neyman and Kohlberg Value
}\item	\DIFadd{IIA -- Independence of irrelevant alternatives axiom
}\item	\DIFadd{ISO -- Independant System Operator
}\item	\DIFadd{KKT -- Karush–Kuhn–Tucker conditions
}\item	\DIFadd{LBP -- linear bilevel program
}\item	\DIFadd{LMP -- Locational Marginal Pricing
}\item	\DIFadd{M-GNK -- modified GNK value
}\item	\DIFadd{NEL -- Australia's National electricity law
}\item	\DIFadd{NEM -- Australia's National Electricity Market
}\item	\DIFadd{NEO -- Australia's National Electricity Objective
}\item	\DIFadd{NTU -- non-transferable utility game
}\item	\DIFadd{OPF -- Optimal Power Flow problem
}\item	\DIFadd{P2P -- peer-to-peer electricity trading
}\item	\DIFadd{PEEF -- pareto-efficient envy-free allocation
}\item	\DIFadd{PV -- solar photovoltaic system
}\item	\DIFadd{RHS -- right hand side
}\item	\DIFadd{SEBB -- Stratified Empirical Bernstein Bound
}\item	\DIFadd{SEBM -- Stratified Empirical Bernstein Method
}\item	\DIFadd{SEBM* -- Stratified empirical Bernstein method with variance assistance
}\item	\DIFadd{SEBM-W -- Stratified empirical Bernstein method with replacement
}\item	\DIFadd{SECB -- Stratified Empirical Chebyshev Bound
}\item	\DIFadd{TU -- transferable utility game
}\item	\DIFadd{UC -- Unit Commitment problem
}\item	\DIFadd{VCG -- Vickrey-Clarke-Groves mechanism
}\item	\DIFadd{VPP -- virtual power plant
}\end{itemize}

\section*{\DIFadd{Symbols}}

\paragraph*{\DIFadd{Chapter \ref{sec:philosophy}}}
\begin{itemize}
\item	\DIFadd{$I$ indicator function
}\item	\DIFadd{$u_i(\cdot)$ utility of player $i$
}\item	\DIFadd{$\pi_i$ resource bundle of player $i$
}\end{itemize}

\paragraph*{\DIFadd{Chapter \ref{cha:solutions}}}
\begin{itemize}
\item	\DIFadd{$X$ set of physical outcomes
}\item	\DIFadd{$n$ number of agents
}\item	\DIFadd{$v_i$ valuation over physical outcomes for player $i$
}\item	\DIFadd{$x^*$ socially optimal physical outcome
}\item	\DIFadd{$d_i(v)$ VCG payment for player $i$, given valuation set $v$
}\item	\DIFadd{$u_i$ utility of player $i$
}\item	\DIFadd{$C_{-i}$ a pivot rule in groves mechanism
}\item	\DIFadd{$D(x)$ demand curve in marginal pricing game
}\item	\DIFadd{$S(x)$ supply curve in marginal pricing game
}\item	\DIFadd{$U_d(x)$ buyers total utility in marginal pricing game
}\item	\DIFadd{$U_s(x)$ sellers total utility in marginal pricing game
}\item	\DIFadd{$L(a,b,\lambda)$ lagrangian function, for amount bought $a$ and amount sold $b$, with lagrange multiplier $\lambda$
}\item	\DIFadd{$N$ is set of players $1$ to $n$
}\item	\DIFadd{$v: S\subseteq N \rightarrow \mathbb{R}$ is a characteristic function
}\item	\DIFadd{$C(v)$ the Core of a cooperative game with characteristic function $v$
}\item	\DIFadd{$C_\epsilon(v)$ the strong epsilon Core of a cooperative game with characteristic function $v$
}\item	\DIFadd{$\hat{v}_{i,k}$ average marginal contribution of player $i$ across coalitions of size $k$ in a cooperative game with characteristic function $v$
}\item	\DIFadd{$\varphi_i(\langle N,v\rangle)$ the Shapley Value for player $i$, for a coaltional game with player set $N$ and characteristic function $v$
}\item	\DIFadd{$\pi(N)$ the permutations of join orderings of the player set $N$
}\item	\DIFadd{$Pre^i(O)$ as the set of predecessors of player $i$'s addition in that ordering $O\in \pi(N)$
}\item	\DIFadd{$F$ is a set of potential outcomes
}\item	\DIFadd{$P=\{p_1,p_2,\dots\}$ is set of players
}\item	\DIFadd{$u_{p\in P}(f)$ for $f\in F$ is utility of outcome $f$ for player $p$
}\item	\DIFadd{$d$ is the disagreement outcome
}\item	\DIFadd{$\text{nash}(F,d)$ is the nash bargaining solution
}\item	\DIFadd{$S_i$ strategy set for player $i$
}\item	\DIFadd{$p_1(s_1,s_2)$ and $p_2(s_1,s_2)$ for $s_1\in S_1$ and $s_2\in S_2$ is payoffs for players 1 and 2 for playing strategies $s_1$ adn $s_2$
}\item	\DIFadd{$B$ set of cooperative strategies
}\item	\DIFadd{$d_i$ demand of player $i$
}\item	\DIFadd{$t_i$ threat of player $i$
}\end{itemize}

\paragraph*{\DIFadd{Chapter \ref{cha:new_solution}}}
\begin{itemize}
\item	\DIFadd{$N$ is a finite set of $n$ players
}\item	\DIFadd{$v:2^N\rightarrow \mathbb{R}$ is characteristic function
}\item	\DIFadd{$\mathbb{D}$ the set of all games of threats
}\item	\DIFadd{$\varphi_i(\langle N,v\rangle)$ the GNK value for player $i$ of player set $N$ for characteristic function $v$
}\item	\DIFadd{$A\subseteq \prod_{i\in N}A^i$ is a set of all possible joint strategies, where $A^i$ denotes the set of strategies available to player $i\in N$
}\item	\DIFadd{$u_i(a)$ is player $i$'s payoff/utility when joint strategy $a\in A$ is executed
}\item	\DIFadd{$v_o(S)$ Neymann \& Kohlberg's original value function
}\item	\DIFadd{$v_m(S)$ von Neumann and Morgenstern's historic value function
}\item	\DIFadd{$B$ set of busses in a network
}\item	\DIFadd{$p_i$ power consumption/generation at bus $i$
}\item	\DIFadd{$\theta_i$ phase angle at bus $i$
}\item	\DIFadd{$C\subseteq B\times B$ set of lines in network
}\item	\DIFadd{$b_{i,j}$ susceptance of the line between busses $i$ and $j$
}\item	\DIFadd{$p_{i,j}$ power flow from bus $i$ to $j$
}\item	\DIFadd{$h_j$ and $g_k$ (for indices $j,k$) are shorthand for equality and inequality constraints in DC OPF
}\item	\DIFadd{$u_i(p_i)$ utility for players at bus $i$ for power consumption at bus $i$, $p_i$
}\item	\DIFadd{$\mathcal{KKT}[\cdot]$ set of KKT points to the optimisation problem
}\item	\DIFadd{$\lambda_k$ is $k$th lagrange multiplier
}\item	\DIFadd{$Z_k$ activation binary variable for $k$th complementary slackness constraint
}\item	\DIFadd{$\mathbb{KKT}(\cdot)$ set of KKT points for optimisation problem under binary relaxation of complementary slackness constraint
}\item	\DIFadd{$d_i$ VCG payment to player $i$
}\item	\DIFadd{$v_{shap}(S)$ characteristic function for Shapley Value formulation of network
}\end{itemize}

\paragraph*{\DIFadd{Chapter \ref{chap:stratified_sampling_chapter}}}
\begin{itemize}
\item	\DIFadd{$N_i$ the size of strata $i$
}\item	\DIFadd{$n_i$/$m_i$ the number of samples taken from strata $i$
}\item	\DIFadd{$\sigma_i^2$ the variance of data in strata $i$
}\item	\DIFadd{$\mu$ the mean of the population
}\item	\DIFadd{$\hat{\mu}$ the estimate of the population mean using stratified sampling
}\item	\DIFadd{$a$ the data lower bound
}\item	\DIFadd{$b$ the data upper bound
}\item	\DIFadd{$D$ the data bound width $=b-a$
}\item	\DIFadd{$\mathbb{E}[\cdot]$ the expecation operator
}\item	\DIFadd{$\hat{\sigma}^2$ the sample variance estimator
}\item	\DIFadd{$\hat{\sigma}_0^2$ the average of sample squares
}\item	\DIFadd{$X$ a random variable
}\item	\DIFadd{$X_1,\dots,X_n$ random samples of $X$
}\item	\DIFadd{$Y_1,\dots,Y_n$ random samples of $X$ drawn without replacement
}\item	\DIFadd{$D_i$ data bound width of strata $i$
}\item	\DIFadd{$\hat{\sigma}^2_i$ the sample variance of strata $i$
}\item	\DIFadd{$X_{i,j}$ the $j$th random sample taken from strata $i$
}\item	\DIFadd{$H_1^n$ the RHS of Bennett's inequality, Theorem \ref{hoeffdings1}
}\item	\DIFadd{$H_2^n$ the RHS of our sample squares bound, Theorem \ref{sample_squares}
}\item	\DIFadd{$H_3^n$ the RHS of a variance bound \ref{variance2}
}\item	\DIFadd{$h$ placeholder function in Theorem \ref{ebb1}
}\item	\DIFadd{$f$ placeholder functions in Theorems \ref{ebb1} and \ref{hoeffdings_reduction}
}\item	\DIFadd{$z$ placeholder function in Theorem \ref{ebb1}
}\item	\DIFadd{$\kappa$ placeholder real variable
}\item	\DIFadd{$s$ placeholder real variable
}\item	\DIFadd{$t$ placeholder real variable
}\item	\DIFadd{$x$ placeholder real variable
}\item	\DIFadd{$y$ placeholder real variable
}\item	\DIFadd{$q$ placeholder real variable
}\item	\DIFadd{$\alpha$ placeholder parameter/variable
}\item	\DIFadd{$\beta$ placeholder parameter/variable
}\item	\DIFadd{$\gamma$ placeholder parameter/variable
}\item	\DIFadd{$x_1,x_2\dots$ population of finite data points
}\item	\DIFadd{${\Omega}_m^n$ parameters from Theorem \ref{martingale1}
}\item	\DIFadd{${\Psi}_m^n$ parameters from Theorem \ref{martingale1}
}\item	\DIFadd{$\bar{\Omega}_m^n$ parameters from Theorem \ref{martingale0}
}\item	\DIFadd{$\bar{\Psi}_m^n$ parameters from Theorem \ref{martingale0}
}\item	\DIFadd{$\chi_{i,m_i}$ the sample mean of strata $i$, after taking $m_i$ samples
}\item	\DIFadd{$\hat{\sigma}^2_i$ the biased sample variance estimator of statum $i$
}\item	\DIFadd{$\doublehat{\sigma}^2_i$ the unbiased sample variance estimator of statum $i$
}\item	\DIFadd{$\tau_i$ the weight of data in stratum $i$
}\item	\DIFadd{$\theta_i$ temporary variable for stratum $i$
}\item	\DIFadd{$\Gamma$ the gamma function
}\item	\DIFadd{$\phi(x)_{\{\alpha,\beta\}}$ the probabiliyt density function for Beta distributed data
}\item	\DIFadd{$v(S)$ characetistic function for coalition $S$
}\item	\DIFadd{$w_i$ the weight of player $i$ in coatlitional game
}\item	\DIFadd{$e^{Ma}$ the error in Maleki's method from section \ref{sec:shapley}
}\item	\DIFadd{$e^{sim}$ the error in Castro's simple sampling method from section \ref{sec:shapley}
}\item	\DIFadd{$e^{Ca}$ the error in Castro's Neyman sampling method from section \ref{sec:shapley}
}\item	\DIFadd{$e^{SEBM}$ the error in our SEBM sampling method in section \ref{sec:shapley}
}\item	\DIFadd{$\mu_{i,j}$ is the $j$th component of the mean vector from the $i$th stratum
}\item	\DIFadd{$D_{i,j}$ is the data bound width of the $j$th vector component of the $i$th stratum
}\item	\DIFadd{$\sigma_{i,j}^2$ the variance of the $j$th vector component of the $i$th stratum
}\item	\DIFadd{$X_{i,k,j}$ is the $j$th component, of the $k$th vector from stratum $i$
}\item	\DIFadd{$\chi_{i,m_i,j}$ being the $j$th component of the average of $m_i$ sampled vectors from the $i$th stratum
}\item	\DIFadd{$\doublehat{\sigma}_{i,j}^2=\frac{i}{m_i-1}\sum_{k=1}^{m_i}(X_{i,k,j}-\chi_{i,m_i,j})^2$ be the unbiased sample variance of the first $m_i$ of these random variables in the $j$th component  of the $i$th stratum.
}\item	\DIFadd{$\alpha_j$ placeholder parameter/variable with index $j$
}\item	\DIFadd{$\beta_j$ placeholder parameter/variable with index $j$
}\item	\DIFadd{$\gamma_j$ placeholder parameter/variable with index $j$
}\end{itemize}

 \DIFaddend


\subsection{Designing Output Test} % (fold)
\label{sub:designing_hello_world}

Our first programming task is to extended the classic `Hello World' program to also output some data values. We will call this program \textbf{Output Test}. This program will allow you to see all of the different concepts from this chapter in action. A description of the program is shown in Table \ref{tbl:program-creation-hello world description}, and a sample execution is shown in Figure \ref{fig:program-creation-helloworld2}.

\begin{table}[h]
\centering
\begin{tabular}{l|p{12cm}}
  \hline
  \multicolumn{2}{c}{\textbf{Program Description}} \\
  \hline
  \textbf{Name} & \emph{Output Test} \\
  \\
  \textbf{Description} & Displays the text \textbf{`Output Test Program!'} on the Terminal, followed by \\
                       & \textbf{` 1 + 1 = '} and the result of the calculation 1 + 1, and then \\
                       & \textbf{` Area of a circle with radius 3 = '} and the result of calculating this value. \\
  \hline
\end{tabular}
\caption[Output Test Description]{Description of the \emph{Output Test} program.}
\label{tbl:program-creation-hello world description}
\end{table}

\begin{figure}[h]
   \centering
   \includegraphics[width=0.8\textwidth]{./topics/program-creation/images/HelloWorld}
   \caption{Output Test executed from the Terminal}
   \label{fig:program-creation-helloworld2}
\end{figure}

To design and implement this program we need to follow a number of steps:

\begin{enumerate}
  \item Understand the problem, and get some ideas on the tasks that need to be performed.
  \item Choose the artefacts we will create and use.
  \item Map these artefacts to code.
  \item Compile and run the program.
\end{enumerate}

% \begin{enumerate}
%   \item Determine the programming artefacts that you will need for the program; covered in section \ref{sub:choosing_artefacts_hello_world}. To do this you will need to:
%   \begin{itemize}
%     \item Decide which artefacts to \textbf{create}.
%     \item Select artefacts to \textbf{use} from the Libraries.
%   \end{itemize}
%   \item Determine the sequence of instructions you will need to give the computer for it to use these artefacts to produce the outcome you are looking for.
% \end{enumerate}

% subsection designing_hello_world (end)
\clearpage
\subsection{Understanding Output Test} % (fold)
\label{sub:understanding_output_test}

The first step in creating a program is to analyse the problem, and find related material that we can use to make sure we know what the computer needs to do. You need to understand what the program needs to do before you can start to design the solution.

For this program you need to understand how to calculate the answer of the equation 1 + 1 (which is a trivial task), and also how to calculate the area of a circle \texttt{a}, given its radius \texttt{r}. A quick search of the internet and you can find the equation is $a=\pi r^2$. Using these equations you have the knowhow needed to calculate the values needed to be output.

% subsection understanding_output_test (end)
\subsection{Choosing Artefacts for Output Test} % (fold)
\label{sub:choosing_artefacts_hello_world}

Designing a program is all about making decisions around how to organise the program's instructions. This means \textbf{deciding} on which artefacts you will \textbf{create}, and which you will \textbf{use}. This chapter has already introduced the concept of \textbf{creating} a \nameref{sub:program}, and \textbf{using} \nameref{sub:procedure}s. With these tools it is possible to design simple programs that will get the computer to perform actions by running procedures in sequence.

Let us return to the design of the \emph{Output Test} program. The program will require:

\begin{itemize}
  \item The \textbf{creation} of a \nameref{sub:program}. This program will contain the instructions to write the three messages to the terminal.
  \item The \textbf{use} of a procedure to write to the terminal. Programming languages provide a standard \nameref{sub:library} that, amongst other things, contains procedures to write data to the terminal. This means you can call one of these procedures, passing it the data you want written to the Terminal, and let it take care of the details of how this is done.
\end{itemize}

\bigskip

The design for this program can be documented using a programming language neutral \textbf{pseudocode}\footnote{This is literally translated as `false code'; it looks like code, but it is not real code!}. This is a structured textural description of how the program works that is independent of the programming language used to implement it.

Listing \ref{lst:program-creation-hello-pseudo} shows the pseudocode for the \emph{Output Test} program. This represents the \textbf{program} that needs to be created, and shows the three \textbf{procedure calls} that need to be made.

\pseudocode{lst:program-creation-hello-pseudo}{Pseudocode for the Output Test program.}{./topics/program-creation/application/HelloWorld.txt}

\csection{In C the procedure to write data to the Terminal is called \texttt{printf}, see \nameref{sub:c_console_output}.}
\passection{Pascal has two procedures to write data to the Terminal, \texttt{Write} and \texttt{WriteLn}. }

% subsection choosing_artefacts (end)
\clearpage
\subsection{Writing the Code for Output Test} % (fold)
\label{sub:writing_the_code_for_output_test}

The pseudocode in Listing \ref{lst:program-creation-hello-pseudo} contains the instructions that will get the computer to perform the actions needed by the \emph{Output Test} program. These pseudocode instructions are not in a form that can be used by the computer, which can only use machine code. To make this into a program the pseudocode must be translated into source code, and then to compile this into machine code.

\sref{sec:program-creation-in-c} \nameref{sec:program-creation-in-c} and \sref{sec:program-creation-in-pas} \nameref{sec:program-creation-in-pas}, contain a description of the syntax needed to create programs using the C and Pascal programming languages. Each section outlines how to write the code so that it can be understood by a compiler. The programming languages are expressed as a number of related syntax rules. You will find the rules that you need to create a program, the rules for calling a procedure, and other related rules.

In this book the syntax rules are expressed using \textbf{syntax diagrams}. An example is shown in \fref{synt:basic-rule}. This diagram shows the syntax related to two rules, \emph{first rule}, and \emph{second rule}, and shows the four main parts of all the syntax diagrams.

\begin{enumerate}
  \item Text found at the start of a line (not contained in a box) is the name of a rule. There are two rules in Figure \ref{synt:basic-rule}: \emph{first rule}, and \emph{second rule}.
  \item Arrows show the order in which the parts of the rule are applied. They start at the rule name, and point in the direction you need to follow. Each box pointed to by an arrow represents either another rule to apply, or the text that must written.
  \item Rectangular boxes (nodes) on a line indicate points where other rules need to be applied. For example, the node  \tikz \node [nonterminal] {second rule}; within the \emph{first rule} indicates that you \textbf{must} apply the \emph{second rule} at this point.
  \item Boxes with rounded corners represent text that must be entered into the code. For example, the node \tikz \node [terminal] {write in the code}; within the \emph{first rule} indicates that you \textbf{must} write the text `\emph{write in the code}' at this point in your code.
\end{enumerate}

\examplesyntax{synt:basic-rule}{An example rule}{basic_rules}

In order to make use of these syntax diagrams, you must know what it is that you \emph{want} to write in code. The rules in \fref{synt:basic-rule} indicate that you can write either a \emph{first rule} or a \emph{second rule}. If you want to write a \emph{first rule} you find that rule in the diagram, and then follow its arrows. Reading the \emph{first rule} indicates that the \emph{second rule} must be applied first.

The \emph{second rule} tells you that you must write the text `\emph{stuff to}'. The vertical bar at the end of the line indicates the end of the \emph{second rule}. This means at this stage the code is:

\begin{verb}
  stuff to
\end{verb}

Having finished the \emph{second rule}, you can return back to finish the \emph{first rule}. This indicates that you need to write `\emph{write in the code}' in the code. This is the last part of the \emph{first rule}, and so the code needed to write a \emph{first rule} from the syntax diagram in \fref{synt:basic-rule} is shown below.

\begin{verb}
  stuff to write in the code
\end{verb}

For a more realistic example have a look at the syntax diagram in \fref{synt:example-identifier}.\footnote{The \tikz \node [meta-terminal] {...}; is used as shorthand to avoid having to list all of the characters between `A' to `Z'.} This shows the rules you need to follow to code an \nameref{sub:identifier}\footnote{Most programming languages have the same rules for identifiers.} in either C or Pascal.

\examplesyntax{synt:example-identifier}{The Syntax Rules for an Identifier}{identifier}

There are three rules in Figure \ref{synt:example-identifier}, including the \textbf{identifier} rule itself and rules for \textbf{letter} and \textbf{digit}. These rules show arrows that give you \emph{options}, and the ability to \emph{repeat} parts of the rules.

\begin{itemize}
  \item A \textbf{letter} is one alphabetic character: i.e. one of `A' to `Z' or `a' to `z'. This is an example of options in the syntax, where you follow \textbf{one} of the available arrows.
  \item A \textbf{digit} is a single number: i.e. a number between `0' and `9'.

  \item The \textbf{identifier} has a more complicated rule, with the following parts:
  \begin{enumerate}
    \item The first thing in an \nameref{sub:identifier} must be either a \emph{letter} or an underscore ( \_ ).
    \item Next you have the option of following the top line and ending the identifier, or following the downward arrow and including other letters, numbers, and underscores.
    \item Following the downward arrow you have a new option where you can choose to have either a \emph{letter}, a \emph{digit}, or an underscore as the second character in your identifier.
    \item Continuing after this option you have another option where you can return back to repeat the previous step, allowing you to have identifiers with more than one or two characters.
  \end{enumerate}
\end{itemize}

\subsubsection{Using the Syntax Diagrams} % (fold)
\label{ssub:using_the_syntax_diagrams}

Syntax diagrams can help you to map a concept to actual code that needs to be written in your source code. To use these diagrams you must first know \emph{what} it is that you want to create or use, and then you can \emph{look up} the related syntax. This is where pseudocode code comes into play. It contains a description of the things that need to be created.

Listing \ref{lst:program-creation-hello-pseudo3} shows the pseudocode for the Output Test program. This tells you what needs to be created; you need to create a \nameref{sub:program}. This means you need to find the syntax that tells you the rules of how a \emph{Program} is written in a programming language.

\pseudocode{lst:program-creation-hello-pseudo3}{Pseudocode for the \emph{Output Test} program (repeated from Listing \ref{lst:program-creation-hello-pseudo})}{./topics/program-creation/application/HelloWorld.txt}

When you write the code for the program you will need to know the actions you want the computer to perform. The pseudocode in \lref{lst:program-creation-hello-pseudo3} indicates that you need to output text and calculated values to the terminal; this can be achieved using \nameref{sub:procedure call}s. You can lookup the language syntax for a procedure call, and use this to write the required code.

% subsubsection using_the_syntax_diagrams (end)

\subsubsection{How the Syntax is Presented} % (fold)
\label{ssub:how_the_syntax_is_presented}

The C and Pascal Syntax needed to create a program are shown in Section \ref{sec:program-creation-in-c} \nameref{sec:program-creation-in-c} and Section \ref{sec:program-creation-in-pas} \nameref{sec:program-creation-in-pas}. Each part of the Syntax is presented on its own page that shows a syntax diagram followed by an example and some notes. The best way to approach this is to do the following:

\begin{enumerate}
  \item Find the page with the syntax rule you are interested in knowing about.
  \item Have a quick look at the \textbf{syntax diagram} and the rules it contains. Read each rule, and get a basic feel for how it is going to come together for your program.
  \item Read the \textbf{example} to see one way of using the rule. A syntax diagram can be used to create any number of variations of the rule. However examples show you at least one way these rules can be coded.
  \item Return to the diagram and make sure you can match each part of the example back to the rule that created it.
  \item Now look up any related rules that are not explained on this rule's page. For example, a \nameref{sub:program} will use the \nameref{sub:statement} rule to code its instructions. The actual rule for a Statement will have its own page. When you read the rules for a Program you will need to also find the page with the rules for a Statement so that you know how to code these within the program.
\end{enumerate}

As you follow this process it is also a good idea to take notes and to try to use these rules in your own programs. Have your code editor open, and see if you can follow the rules or mimic the examples to start building your own program's code. You can also try typing in some of the examples to see how they work.

Learning to program, and learning a language's syntax, takes time and practice (like juggling). Reading about these concepts is one thing, but being able to successfully apply this ideas is something different. Make sure that you practice using these concepts and syntax.

% subsubsection how_the_syntax_is_presented (end)

% subsection writing_the_code_for_output_test (end)

\subsection{Compiling and Running Output Test} % (fold)
\label{sub:compiling_and_running_output_test}

Previous sections have shown pseudocode for the \emph{Output Test} program. These steps can be coded, using either C or Pascal, into a source code file. In order to actually use these instructions you will need to \textbf{compile} the source code file. This will produce an executable file that you can run. To do this you can use the terminal and the command line compiler as shown in \cref{cha:building programs}.

\begin{itemize}
  \item On Ubuntu \textbf{Linux} you can find the Terminal in the \textbf{Accessories} folder within \textbf{Applications}. See Figure \ref{fig:program-creation-ubuntu-terminal}.
  \item On \textbf{MacOS} you can find the Terminal in the \textbf{Utilities} folder within \textbf{Applications}. See Figure \ref{fig:program-creation-macos-terminal}.
  \item On \textbf{Windows} you will need to download and install \emph{MSys2}, follow the steps in the SplashKit installer. The \emph{Msys2 Shell} is then the equivalent of Terminal on the other operating systems. You will find this in \textbf{Program Files}, \textbf{MSys2}. See Figure \ref{fig:program-creation-mingw-shell}.
\end{itemize}

\begin{figure}[p]
   \centering
   \includegraphics[width=0.6\textwidth]{./topics/program-creation/images/UbuntuTerminal}
   \caption{Terminal on Ubuntu Linux}
   \label{fig:program-creation-ubuntu-terminal}
\end{figure}

\begin{figure}[p]
   \centering
   \includegraphics[width=0.6\textwidth]{./topics/program-creation/images/MacOSTerminal}
   \caption{Terminal on MacOS}
   \label{fig:program-creation-macos-terminal}
\end{figure}

\begin{figure}[p]
   \centering
   \includegraphics[width=0.6\textwidth]{./topics/program-creation/images/MinGWShell}
   \caption{MinGW Shell (Terminal) for Windows}
   \label{fig:program-creation-mingw-shell}
\end{figure}

Once you are in the Terminal you have the ability to run a number of text based commands. These commands instruct the computer to perform actions for you.

\begin{itemize}
  \item \textbf{pwd} stands for \emph{Present Working Directory}, and shows you where you are in the file system.
  \item \textbf{ls} stands for \emph{List} and it prints out a list of the files that are in the current directory.
  \item \textbf{cd} stands for \emph{Change Directory} and it moves you to another directory.
\end{itemize}

To compile your program you need to do the following:

\begin{enumerate}
  \item Change into the directory where your code is located using the \textbf{cd} command. For example, if your code is in a \emph{Code} folder in your \emph{Documents} folder you would use:
  \begin{itemize}
    \item \textbf{Linux}: \texttt{cd /home/username/Documents/Code}
    \item \textbf{MacOS}: \texttt{cd /Users/username/Documents/Code}
    \item \textbf{Windows}: (in MinGW Shell) \texttt{cd /c/Users/username/Documents/Code}\footnote{This example moves you into the \texttt{c:{\textbackslash}Users{\textbackslash}username{\textbackslash}Documents{\textbackslash}Code} folder. You need to use the /c/ to refer to the C drive. }
  \end{itemize}
  \item Run the compiler, passing in the name of the file you want to compile. See the language specific notes below
  \item Execute the program using \texttt{./OutputTest}
\end{enumerate}

\csection{
The C compiler is called \textbf{gcc}. To compile your \emph{Output Test} program you will need to run the following: \bashcode{}{}{code/c/program-creation/compile-output-test.sh}
}

\passection{
The Pascal compiler is called \textbf{fpc}. To compile your \emph{Output Test} program you will need to run the following: \bashcode{}{}{code/pascal/program-creation/compile-output-test.sh}
}

\clearpage
Figure \ref{fig:program-creation-complete-linux-version} shows an example of the instructions needed to compile and run the C version of the \emph{Output Test} program on Linux. Figure \ref{fig:program-creation-complete-linux-version-1} shows an example of the instructions needed to compile and run the Pascal version of the \emph{Output Test} program on Linux.

\begin{figure}[h]
   \centering
   \includegraphics[width=0.62\textwidth]{./topics/program-creation/images/LinuxCompleteExample1}
   \caption{Example of compiling and running a C version of Output Test on Linux}
   \label{fig:program-creation-complete-linux-version}
\end{figure}

\begin{figure}[h]
   \centering
   \includegraphics[width=0.62\textwidth]{./topics/program-creation/images/LinuxCompleteExample}
   \caption{Example of compiling and running a Pascal version of Output Test on Linux}
   \label{fig:program-creation-complete-linux-version-1}
\end{figure}

\clearpage
Figure \ref{fig:program-creation-complete-macos-version} shows an example of the instructions needed to compile and run the C version of the \emph{Output Test} program on MacOS. Figure \ref{fig:program-creation-complete-macos-version-1} shows an example of the instructions needed to compile and run the Pascal version of the \emph{Output Test} program on MacOS.

\begin{figure}[h]
   \centering
   \includegraphics[width=0.73\textwidth]{./topics/program-creation/images/MacOSCompleteExample1}
   \caption{Example of compiling and running a C version of Output Test on MacOS}
   \label{fig:program-creation-complete-macos-version}
\end{figure}

\begin{figure}[h]
   \centering
   \includegraphics[width=0.73\textwidth]{./topics/program-creation/images/MacOSCompleteExample}
   \caption{Example of compiling and running a Pascal version of Output Test on MacOS}
   \label{fig:program-creation-complete-macos-version-1}
\end{figure}

\clearpage
Figure \ref{fig:program-creation-complete-windows-version} shows an example of the instructions needed to compile and run the C version of the \emph{Output Test} program on MacOS. Figure \ref{fig:program-creation-complete-windows-version-1} shows an example of the instructions needed to compile and run the Pascal version of the \emph{Output Test} program on MacOS.

\begin{figure}[h]
   \centering
   \includegraphics[width=0.73\textwidth]{./topics/program-creation/images/WindowsCompleteExample}
   \caption{Example of compiling and running a C version of Output Test on Windows}
   \label{fig:program-creation-complete-windows-version}
\end{figure}

\begin{figure}[h]
   \centering
   \includegraphics[width=0.73\textwidth]{./topics/program-creation/images/WindowsCompleteExample1}
   \caption{Example of compiling and running a C version of Output Test on Windows}
   \label{fig:program-creation-complete-windows-version-1}
\end{figure}

% subsection compiling_and_running_output_test (end)
\clearpage
\subsubsection{Compiler Errors} % (fold)
\label{ssub:compiler_errors}

A compiler is a very sensitive piece of software; they require that you follow the language's syntax \emph{precisely}. One small mistake, and the compiler will fail to compile your program and end with an error message. This is a \emph{good} thing! (Although it may not feel like it at first.) Why? Because when programs become complicated we need the exact details to help us, and to assure the program works the way we need it to.

Here are some handy hints related to dealing with compiler errors.

\begin{enumerate}
  \item \textbf{Know that errors are inevitable}. You will get compiler errors! As you gain more experience you will get fewer errors, but it will always be a rare event that everything is exactly as it should be the first time.
  \item \textbf{Error messages may appear cryptic at first}. In many cases the compiler's error messages can appear cryptic, you need to learn to decode the messages the compiler gives you.
  \item \textbf{Start with the \underline{first} error message, and do not move on until it is fixed}. The compiler will read your code `top down'. So the first error it outputs will be the first in the file. The problem is that the compiler will try to continue on, despite the error. This can mean that other errors further on in the output may actually be caused by the compiler being \emph{confused} due to the first error. By fixing the first error you may also fix subsequent errors.
  \item \textbf{Deal with \underline{one} error at a time}. It is easy to feel overwhelmed when you see a huge list of errors, but do not be scared off! Start at the first error, and solve them one by one. Compile after fixing each problem to see if the others still exist.
  \item \textbf{Do not add more code until the errors are fixed}. Compile your program frequently. Add small pieces of functionality, and then fix any syntax errors before adding the next piece of functionality. Coding in this way makes sure that you do not get too many errors, and reduces the code you have to search in order to fix the problem.
  \item \textbf{Read the error message \underline{carefully}}. The message will try to tell you what has gone wrong, and once you understand what the messages are trying to say you will be able find and fix the issues quickly.
  \item  \textbf{Work out what the error messages mean}. Understanding the error messages will mean that you will know what to look for, and will help you avoid these errors in the future. If your not sure what an error message means ask others developers or search for the message on the internet.
  \item \textbf{Find the \underline{line} and \underline{character number} of the error}. One important detail in the error message will be the line number of the error. This gives you a starting point to help you locate the problem. It is important to note that this is where the compiler got to when it noticed the error - this \emph{does not} mean this is where the error actually is (but it is a good place to start looking). You may need to look back one or more lines to find the actual source of the problem.
  \item \textbf{Watch for typos}. It is easy to mistype an identifier. When this happens the compiler will not know what to do (and it will complain about you trying to use `unknown' things). Make sure you check for these tiny typos when you get errors related to the compiler not being able to find an identifier.
  \item \textbf{When you get stuck \underline{ask for help}}. Compiler errors are likely to be something small, but sometimes the causes are hard to find. If you get stuck, \emph{ask for help}! Having access to other more experienced developers will be a valuable resource as you learn to program. Learning to program can be tough at times, and having someone who can help you when you get stuck will make all the difference.
\end{enumerate}

% subsubsection compiler_errors (end)