\documentclass[oneside,a4paper]{article}

% ========== Preamble (packages, definitions etc.) ==========

\usepackage[utf8]{inputenc}
\usepackage{graphicx}
\usepackage{xcolor}
\usepackage{amsmath, amsthm, amssymb}
\usepackage{csquotes}
\usepackage{hyperref}
\usepackage{listings}
\usepackage{lmodern}
\usepackage{float}
\usepackage{braket}

\setlength{\parskip}{\baselineskip}

%\newcounter{questionnum} \setcounter{questionnum}{0}
%\newcommand{\question}[1]{%
%  \refstepcounter{questionnum}%
%  \paragraph{Question~\arabic{questionnum}:}{\emph{#1}}}

\newcommand\filltoend{\leavevmode{\unskip
  \leaders\hrule height.5ex depth\dimexpr-.5ex+0.4pt\hfill\hbox{}%
  \parfillskip=0pt\endgraf}}

\newcommand{\problem}[2]{%
	\vspace{-0.7em}
	\hspace{0.02\textwidth}
	\begin{minipage}[t][][b]{0.95\textwidth}
		{\bf \hspace{-0.015\textwidth}\makebox[7.5em][l]{{#1} ~~\filltoend}}%
		\hspace{1.2mm}{\it #2}%
	\end{minipage}
}

\lstset{ % Set the default style for code listings
	numbers=left,
	numberstyle=\scriptsize,
	numbersep=8pt,
	basicstyle=\scriptsize\ttfamily,
	keywordstyle=\color{blue},
	stringstyle=\color{red},
	commentstyle=\color{green!70!black},
	breaklines=true,
	frame=single,
	language=C++,
	captionpos=b,
	tabsize=4,
	showstringspaces=false
}

\graphicspath{ {res/} }

% ========== Title page ==========

\title {
	\includegraphics[width=0.6\textwidth]{UU_logo.pdf}\\[1em]
	Cryptology \\
	Report\\[1em]
	%\\[3em]
	Quantum cryptography: Public key distribution and coin tossing
}

\author{
		Hendrik Bierlee \and
		Nodari Kankava
}

\begin{document}

\maketitle
\thispagestyle{empty} % Removes page number for front page
\pagebreak

\newcounter{qcounter}
\setcounter{qcounter}{1}
\newcommand{\question}[1]{\par\vspace{10px}\noindent\textbf{Question \theqcounter \stepcounter{qcounter}:} \emph{#1}\vspace{0.5em}\\\noindent}

% ========== Document contents ==========
\section{Introduction}

\subsection{Problem statement of key-distribution} \label{sec:key-dist-problem}
Key distribution is one of the main issues in cryptography. In a scenario where two parties want to establish a secure communication channel, first they need to agree on a shared secret key. This is an issue for symmetric cryptography protocols, where cryptographic keys has to be securely shared between sender and receiver. Easy key distribution becomes even more important when frequent key change is required or number of participants in communication increases.

\subsection{Classical solution to the public key distribution}
Public key cryptography allows secure distribution of secret keys between parties who don't share initial secret information. The Diffie–Hellman key exchange protocol~\cite{diffie1976new} allows two parties to agree on a shared secret key. This protocol is frequently used in systems that also achieve forward secrecy because of its fast key generation. The Diffie-Helman key exchange is considered secure, until the efficient algorithm to compute discrete logarithm is found.

Classic information theory assumes that communication can be eavesdropped either passively or actively. There is no protection against copying transmitted information. Communicating parties can verify with high degree of confidence that exchanged message was not modified, but they should operate under the assumption that information was most probably copied.
Another major advantage of public key cryptography is message verification and non-repudiation.

\subsection{Problem statement of the remote coin-flip problem}
The `coin-flip by telephone' problem was first described in 1981~\cite{blum1981coin}: two distrusting parties, Alice and Bob, want to play a remote game without a third party where both have an equal win-chance, and where neither can cheat.

\subsubsection{Classical solution to the remote coin-flip problem}
In the same paper, the problem was solved by constructing a one-way function $f$ which is two-to-one, meaning that one can have a pair of inputs $x$ and $y$ that map to the same value $f(x)=f(y)$.
An example of such a function is $f(x) =|1/x|$, because every allowed input $x$ has a counterpart $-x$ that $f$ maps to the same value (except for $0$, which is undefined).
Furthermore, $x$ and $y$ have some distinguishing property, for instance, $x$ is always even and $y$ is always odd. Because of the one-way nature of $f$, Alice can randomly select $x$ but it is computationally impossible for her to also find out the corresponding $y$.

After selecting $x$, Alice sends $f(x)=c_{\text{Alice}}$ to Bob (her \textit{commitment} to $x$), and from $f(x)$ Bob cannot know whether Alice used $x$ or $y$.
\footnote{
    This idea is reminiscent of a zero-knowledge proof~\cite{goldwasser1989knowledge}, where one party proves they posses certain information without revealing to the other side what that information is.
    Alice proofs to Bob she committed to one of two choices without revealing her actual choice.
}
So, Bob can proclaim a guess to Alice with a 50\% chance of getting it right.
Subsequently, Alice can claim victory or defeat and proof her original choice of $x$ by sending $x$ to Bob.
Bob can verify the claim by recognising that $x$ is in fact even, and by computing $f(x)=c_{\text{Alice}}$.
Alice does not possess $y$, so she cannot possibly cheat by sending Bob an odd value that $f$ maps to $c_{\text{Alice}}$.

A benefit to this problem is that if the messages between Alice and Bob are signed, any attempt at cheating on the part of Alice (say she sends $x'$ such that $f(x') \neq c_{\text{Alice}}$) can be proved to a judge.

\subsection{Need for new solutions}
Many cryptography methods are founded on the assumption that some procedures, such as prime factorisation, are computationally hard, i.e. $P \neq NP$.
A prime example of this is prime factorisation, or computing discrete logarithms.
However, this assumption may yet be invalidated by breakthroughs in algorithm design, or, more on topic, the introduction of quantum computing that can execute algorithms such as Shor's algorithm, which can do prime factorisation in polynomial time~\cite{bernstein2017post}.
In the next section, we will summarise the quantum methods for solving the problems, which rely on fundamental properties of physics for security.

\section{Problem solution}
Before presenting the solutions for public-key exchange and remote coin-flipping, we briefly introduce some of the fundamental properties of photons that will be used in both solutions.

In our following discussion, we assume \emph{good} but not \emph{perfect} quantum hardware (when it comes to the ability to pick out, send, receive and measure individual photons).
Perfect hardware would enable cheating the coin-flip with the EPR effect, as discussed in section~\ref{sec:cheating-epr}.

\subsection{The behaviour of polarised photons}
Light exists of individual photon particles, which can be polarised at any angle using polarising filters.
Four of these polarisation axis are important to us: two rectilinear (0 degrees, 90 degrees) and two diagonal (45 degrees, 135 degrees).
To measure the orientation of a polarised photon, one can again use polarising filters.
If the orientation (or basis) of the filter matches the orientation of the photon, the photon will pass through, and if the filter is orthogonal to the photon, the photon will not pass through.
In other words, rectilinear photons can be measured accurately 100\% of the time (deterministically) if one uses the `correct', rectilinear filter, since the difference in angle between photon and a filter at 0 degrees is always either 0 or 90 degrees.

But suppose the difference is 45 degrees, such as is always the case when you try to measure rectilinear photons with a `wrong', diagonal filter.
It happens that the probability of photons passing through this filter is exactly 50\%, so the measurement will tell you exactly nothing about the original polarisation.
Photons that did pass through now have the same polarisation as the filter, so re-measuring them tells you nothing about their original orientation.
Furthermore, we cannot simply clone photons and perform multiple measurements.
Essentially, you only get one chance to measure a photon, and even though there is theoretically infinite information in a photon (if the polarisation is on a continuous spectrum), measurement will only yield one bit of information.

\subsection{Quantum key distribution}
In the quantum key distribution protocols, a quantum channel is used to exchange random bits between two parties who haven't initially shared any secret information and then continue conversation using a classic communication channel. As soon as both parties share random bits using a quantum channel, they will be able to verify that quantum key exchange was not disturbed and be able to agree to use the shared secret bits to encrypt classic communication channel.

Unlike digital communications that can be monitored and copied, transmissions inside a quantum communication channel cannot be copied or observed without randomly and uncontrollably changing the state, therefore any disturbance will be evident. If communicating parties detect that their exchange was disturbed they will try again to securely share enough random bits using a quantum channel to have a guarantee with high probability that their communication is secure.

The protocol for a quantum key exchange is as follows:

Using a quantum channel:
\begin{enumerate}
    \item Alice chooses random bit-string, encodes each bits as polarised photons, switching randomly between rectilinear and diagonal bases and sends polarised photons to Bob.
    \item Bob independent of Alice, randomly chooses what bases to use to measure photons (rectilinear or diagonal polarisation).
\end{enumerate}

Using a classic channel (with the assumption that it could be monitored, but those messages can't be changed or altered):

\begin{enumerate}
    \setcounter{enumi}{2}
    \item Bob reveals for each photon what basis he used to measure (rectilinear or diagonal).
    \item Alice reveals which basis were correct.
    \item Alice and Bob use the correct bits to generate a key for use in classic communication channel.
\end{enumerate}

Because photons have to be measured using random choice of rectilinear or diagonal basis, any measurement by an eavesdropper will alter the message Bob gets and subsequently produce disagreement between Alice and Bob on bits that they should agree on.

\subsection{The quantum remote coin-flip protocol}
The protocol for a quantum remote coin-flip is as follows:
\begin{enumerate}
	\item
		Alice chooses at random a secret basis (rectilinear or diagonal) and encodes a secret, random bit-string of decent length as polarised photons using her chosen basis.
		She sends over the photons to Bob on a quantum channel.
	\item
		To win, Bob has to guess Alice's choice of basis.
		But when Bob receives the photons, he does not know how to measure them deterministically to find out their polarisation.
		The best he can do is switch bases randomly and independently for each photon, so that he at least ends up with a half-filled rectilinear measurement table and a half-filled diagonal measurement table, one of which is 100\% correct and the other is 50\% correct.
		With nothing to go on, Bob has to make a random guess to Alice as to the original basis.
	\item Alice reveals her basis, and sends over the original bit-string on a classical channel.
	\item Bob can verify Alice's choice, because he can confirm that the elements of the bit-string matches up 100\% with the corresponding measurement table.
\end{enumerate}

Like in the classical solution, the key is that one party has to send a commitment to their choice to the other party, without revealing the choice itself.

\section{Benefits and drawbacks of the solutions}
Here we discuss the benefits and drawbacks to the solution.

\subsection{Advantages of the quantum remote coin-flip protocol}

\subsubsection{Impossibility of cheating}
The protocol is protected against cheating by the laws of physics, instead of by computational hardness:

\begin{itemize}
	\item Given the polarised photons, Bob cannot guess the basis of the polarisation with a probability greater than $1/2$.
	\item Alice has to provide the measurement results of the table with the basis she supposedly picked: if she lies about the basis (because Bob guessed it correctly), she would have to guess the contents of the other table.
	If $n$ photons were sent, the probability that she can perfectly match the table is $\frac{1}{(n/2)^2}=\frac{4}{n^2}$.
	Getting even one measurement wrong would expose Alice as a cheater.
	\item Alice could send a mix of rectilinear and diagonal photons, but then she would not be able to verify either table.
\end{itemize}

\subsection{Common disadvantages and possible attacks for quantum key distribution and coin-tossing}
As mentioned in section~\ref{sec:key-dist-problem}, classic information theory assumes that digital communications can be passively monitored. Quantum information theory guarantees that communication cannot be passively monitored and forces the adversary to be active and detectable. However there are possible drawbacks.

\subsubsection{No non-repudiation}
One of the issues with quantum communication is that there is no way of verifying who sent the individual photons. Digital signatures cannot be created with quantum cryptography. There is no such thing as signing photons. Related to this issue is that an eavesdropper can try to suppress communications by introducing noise into a channel or just by trying to measure photons. This is a threat to availability and integrity, but not to the confidentiality.
Also in case of quantum coin-flip Alice or Bob could deny that the coin-flip happened (and the other would not be able to prove to a judge that it did happen).

\subsubsection{Generating random bit-strings}
The bit-string that Alice encodes needs to be generated by a cryptographically secure pseudo-random number generator.
In a post-quantum world, a CSPRNG that relies on prime-factorisation (such as a RSA generator) would be compromised, so it obviously should not be used in an implementation of this protocol.
%TODO ref to slides / RSA generator?
Maybe Alice should just measure rectilinear photons diagonally, instead.

\subsection{Disadvantages of the quantum key distribution}
\subsubsection{Theoretical attacks}
More sophisticated attacks involve quantum phenomena that have not yet been demonstrated as possible but exist as a theoretical thought experiment. In this case if an adversary can clone or multiply photons he can measure them in multiple bases and choose correct ones after Alice and Bob share correct bases in public channel.

\subsection{Disadvantages of the quantum remote coin-flip protocol}

\subsubsection{Need for quantum hardware} %TODO probably for general to both
Even though a version of the quantum remote coin-flip has been performed experimentally by researchers at the Laboratory for Communication and Processing of Information (LTCI) in Paris~\cite{pappa2014experimental}, there is no prediction for when quantum hardware will become available.
This is also good news, because it means that for now we can still rely on computational hardness to safeguard the classical solution.
More generally, one could state that quantum solutions will become useful on the same day that classical solutions will be compromised.

\subsubsection{Possibility of cheating}
\label{sec:cheating-epr}
In theory, Alice can cheat by using the Einstein-Podolsky-Rosen effect, which implies that pairs of polarised photons can be created which always collapse to opposite directions (regardless of in which basis they are measured).
Alice could encode the bit-string in such pairs, send one photon of each pair to Bob and keep the other.
If Bob then guesses, say, rectilinear, she can measure her photons in the diagonal basis, thus being able to match the results of Bob's diagonal table, pretending to have used diagonal encoding all along and verifying her `win'.

However, storing and measuring the twin photons with 100\% accuracy is likely impossible in practice, even with good quantum hardware.

\section{Implementation of a quantum remote coin-flip protocol simulation}
For those who can't wait until quantum computers become commercially available, the Microsoft Q\#~\cite{qsharp} project provides a quantum development kit for expressing quantum algorithms and running simulations of them on a classical machine.
With this, we implemented (a simulation of) the quantum remote coin-flip protocol~\cite{quantum-coin-toss-repo}.
The end-product is a simulation in two ways: obviously, no actual qubits were used, and secondly, since Q\# does not provide a way to simulate a quantum channel, the communication between Alice and Bob takes place in one process.

To implement the protocol, we need to translate photons, polarisation filter, etc. over to the world of quantum computing.

\subsection{Qubits and operating on them}
Photons are one way to achieve quantum behaviour, but in quantum computing, the implementation (be it photons or otherwise) is abstracted away behind the concept of qubits, which uses linear algebra as foundation.
A qubit is represented as a unit vector $\vec{q}=\begin{pmatrix} \alpha \\ \beta\end{pmatrix}$ where $\alpha, \beta \in \mathbb{C}$ and $||\vec{q}||=1$.
Here we have four examples of qubits, along with their more convenient Dirac notations:

\begin{equation}
    \label{eq:qubits}
    \begin{pmatrix} 1 \\ 0\end{pmatrix} = \ket{0}, \quad
    \begin{pmatrix} 0 \\ 1\end{pmatrix} = \ket{1}, \quad
    \begin{pmatrix} \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}}\end{pmatrix} = \ket{+}, \quad
    \begin{pmatrix} \frac{1}{\sqrt{2}} \\ \frac{-1}{\sqrt{2}}\end{pmatrix} = \ket{-}.
\end{equation}

Qubits can be manipulated using linear transformations, such as the negation operation $X$ and the Hadamard gate operation $H$:

\begin{equation}
    X = \begin{pmatrix} 0 & 1 \\ 1 & 0 \\\end{pmatrix}, \quad
    H = \begin{pmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} & \frac{-1}{\sqrt{2}} \\\end{pmatrix}.
\end{equation}

With simple linear algebra we can easily transform our qubits in equation~\ref{eq:qubits} from one into another using $X$ and $H$:

\begin{equation}
    \label{eq:operations}
    X(\ket{0}) = \ket{1},
    \quad H(\ket{0})=\ket{+},
    \quad H(\ket{1})=\ket{-}.
\end{equation}

Since qubits are (unit) vectors, they can be seen as points on the surface of a unit sphere, the so-called Bloch sphere~\cite{bloch1946nuclear}.
For our purposes, we don't need qubits with imaginary numbers, so we stay within one slice of the sphere: a unit circle.
In figure~\ref{fig:unit-circle}, we can see the effects of using $X$ and $H$ on qubits in different states.

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{microsoft-quantum-coding-unit-circle}
    \caption{The unit circle state machine, displaying the effects of $X$ and $H$ operations. Taken from the supporting slides of a Microsoft talk: \emph{`Quantum Computing for Computer Scientists'}~\cite{quantum-computing-talk}.}
    \label{fig:unit-circle}
\end{figure}

\subsection{Measuring qubits}
The $\alpha$ and $\beta$ values determine the superposition or quantum state of the qubit, but we cannot obtain these values directly.
Instead, we have to measure $\vec{q}$, and the only possible outcomes are either $0$ or $1$, ie. a classical bit (or cbit).
The result of a measurement $M$ is probabilistic, and depends on the value of $\alpha$ and $\beta$:

\begin{equation}
    \label{eq:measuring-distribution}
    P(M=0) = |\alpha|^2,
    \quad P(M=1) = |\beta|^2.
\end{equation}

Just like with photons, the basis in which we measure can be varied, so we have to specify that too: in this case using the Pauli measurements.
Two Pauli measurements are of importance: Pauli Z and Pauli X.
Measuring Pauli Z,~$\ket{0}$ always results in~$0$ and ~$\ket{1}$ always in~$1$.
However, when measuring Pauli Z of either~$\ket{-}$ or~$\ket{+}$ we get an uniformly random outcome of $0$ and $1$. This is in line with the above equation~\ref{eq:measuring-distribution}.

With Pauli X, things are turned around: measuring Pauli X of~$\ket{-}$ and~$\ket{+}$ deterministically leads to~$0$ and~$1$ (respectively), but when measuring Pauli X of~$\ket{0}$ and~$\ket{1}$ there is a 50\% chance of~$0$, 50\% chance of~$1$.
On a side-note, measuring Pauli X of~$\vec{q}$ is equivalent to first performing~$H$, and then measuring Pauli Z.

Just like in other quantum systems, no re-measurements is possible (the qubit is said to have collapsed to one of two states) and no cloning is possible (which prevents statistical analysis of a single qubit).

\subsection{From photons to qubits}
With this foundation of quantum computing, we are ready to define the quantum computing equivalent of the quantum remote coin-flip protocol:

\begin{itemize}
    \item A photon with a polarisation axis of~$\theta$ degrees can be represented as a qubit with $\alpha=\cos{\theta}$ and $\beta=\sin{\theta}$.
    Specifically, $\ket{0}$ and $\ket{1}$ are our rectilinearly polarised photons, and ~$\ket{-}$ and~$\ket{+}$ are the diagonal ones.
    \item To encode a bit from a bit-string, we can start from the $\ket{0}$ qubit and use the $X$ and $H$ operations (see equations~\ref{eq:operations}) to achieve any of the four polarisations.
    \item To measure with a rectilinear or diagonal filter, we can measure Pauli Z or Pauli X, respectively.
\end{itemize}

\subsection{Q\# code sample}
The following Q\# operation performs the encoding, from a $\ket{0}$ qubit, rectilinear/diagonal basis and 0/1 bit:

\begin{lstlisting}[caption={\texttt{Operations.qs}}]
operation Encode (qubit : Qubit, rectilinear : Bool, bit : Bool) : Unit {
    if (rectilinear && not(bit)){
        // do nothing, qubit is already |0>
        return (); // since there is no `else if` statement in Q#, we return after each if-statement
    }
    if (rectilinear && bit) {
        X(qubit);   // |1>
        return ();
    }
    if (not(rectilinear) && not(bit)) {
        H(qubit);   // |+>
        return ();
    }
    if (not(rectilinear) && bit) {
        X(qubit);   // |1>
        H(qubit);   // |->
        return ();
    }
}
\end{lstlisting}

\bibliography{tex/sources}
\bibliographystyle{acm}

\end{document}


\section{Grammar preprocessing}

The goal was to implement an algorithm that would convert a context-free grammar into a grammar in the Chomsky normal-form, the most simple choice was to develop a new version of the Grammar class.

On its initialization the class is given a file that contains the grammar, which can be in any form.

The first thing that the class does is to replace all the \textit{non-terminal} variables by an index, starting at 0 for the \textit{start symbol} and increasing by 1 for every variable, the \textit{terminals} are also put in negative so there is no chance of confusion between the variables.

Then the grammar is converted to the Chomsky normal-form, following the same process as manually explained is section 1.1.2.

Finally the converted grammar is splitted into a set of \textit{terminal} rules and a set of \textit{non-terminal} rules.
\\
\\
The final grammar is stocked and can be accessed through the following variables:

\begin{itemize}
    \item[$-$] A 3D vector of integers named `non\_terminal\_rules'.
        \begin{enumerate}
            \item The first dimension represents the \textit{non-terminal} variable, converted as an index starting at 0 for the \textit{start symbol}.
            \item Each element of the second dimension is a right hand of the variable from the first dimension.
            \item The pairs of \textit{non-terminals} are stocked in the third dimension.
        \end{enumerate}
    \item[$-$] A 2D vector of integers named `terminal\_rules' that allows access to the \textit{terminal} rules.
    \item[$-$] Since there are no \textit{non-terminal} rules for every variable, when it is the case the variable is added to the string `nt\_variables'.
    \item[$-$] The string `t\_variables' is exactly the same as the one above but for the \textit{terminal} rules.
\end{itemize}

For example if one gives the following grammar to the program:

\begin{align*}
    &S \to Ac | b\\
    &A \to aS | aB\\
    &B \to bS\\
\end{align*}

This is how the grammar will be stocked:

\begin{itemize}
    \item[$-$] `non\_terminal\_rules' =
        \begin{align*}
            &0 \to 0\text{ }0 | 2\text{ }1 | 2\text{ }3 &2 \to\\
            &1 \to 0\text{ }3                           &3 \to\\
        \end{align*}
    \item[$-$] `terminal\_rules' =
        \begin{align*}
            &0 \to  &2 \to (\\
            &1 \to  &3 \to )\\
        \end{align*}
    \item[$-$] `nt\_variables' = "0 1"
    \item[$-$] `t\_variables' = "2 3"
\end{itemize}

The symbol `$|$' indicates the separation of the second dimension of the vector.
As we can see in the variables content above the \textit{non-terminals} are converted in indices.

That preprocessing allows one to give a linear grammar to the program without bothering to convert anything by advance.

Since the conversion of a linear grammar in the Chomsky normal-form often add new rules there is no reason to think that it could in any way fasten the parsing process to automaticly convert a linear grammar instead of directly giving the algorithm a Chomsky normal-form grammar.

\section{Implementation of a linear grammar parser}

Converting a grammar to the Chomsky normal-form cannot improve the running time of the parsers but a parser that directly parses strings using a linear grammar can be faster.

To implement a linear grammar parser the easiest choice was to use the actual top-down parser as basis for the new parser.
The program now contains a LinearGrammar class which simply stocks a linear grammar in the $rules$ variable without doing any modifications to the characters, the variable $non\_terminals$ is a string linking a \textit{non-terminal} to its index in the rules.
The parser is concived in such a way that it can parse a string using a linear grammar but also a grammar in the Chomsky normal-form.
Since the algorithm is based on the top-down parser if it is given a grammar in the Chomsky normal-form it will need the same number of resursive calls as the top-down parser would.

\FloatBarrier
\begin{algorithm}
\caption{Linear grammar parser}
\label{parse}
\begin{algorithmic}[1]
    \State $table$ is a 3d array of 0
    \State $s \gets input\_string$
    \Procedure{parse}{$var, i, j$} \Comment{Recursive parser function}
        \If{$table[var, i, j] != 0$}
            \State \textbf{return} $(table[var, i, j] == 1)$
        \EndIf
        \ForAll{$right\_hand \in rules[var]$}
            \If{$right\_hand.size() == j - i \land s.size() - i \geq j - i$}
                \State $bool \text{ } is\_equal \gets true$
                \For{$k \gets 0: k < right\_hand.size()$}
                    \If{$s[i + k] \neq right\_hand[k]$}
                       \State $is\_equal \gets false$
                        \State \textbf{break}
                    \EndIf
                \EndFor
                \If{$is\_equal$}
                    \State $table[var, i, j] \gets 1$
                    \State \textbf{return} $true$
                \EndIf
            \EndIf
            \State $var\_1, var\_2, var\_3 \text{ } are \text{ } characters$
            \If{$right\_hand.size() == 1$}
                \State $var\_1 \gets right\_hand[0]$
                \If{$var\_1 \geq \text{} 'A' \land var\_1 \leq\text{}'Z'$}
                    \If{$parse(non\_terminals.find(var\_1), i, j)$}
                        \State $table[var, i, j] \gets 1$
                        \State \textbf{return} $true$
                    \EndIf
                \EndIf
            \ElsIf{$j - i \geq 2 \land right\_hand.size() == 2$}
                \State $var\_1 \gets right\_hand[0]$
                \State $var\_2 \gets right\_hand[1]$
                \If{$var\_1 \geq \text{} 'A' \land var\_1 \leq \text{} 'Z' \land var\_2 \geq \text{} 'A' \land var\_2 \leq \text{} 'Z'$}
                    \For{$k \gets i + 1: k < j$}
                        \If{$parse(non\_terminals.find(var\_1), i, k) \land parse(non\_terminals.find(var\_2), k, j)$}
                            \State $table[var, i, j] \gets 1$
                            \State \textbf{return} $true$
                        \EndIf
                    \EndFor
                \ElsIf{$(var\_1 < \text{} 'A' \lor var\_1 > \text{} 'Z') \land var\_2 \geq \text{}'A' \land var\_2 \leq \text{} 'Z'$}
                    \If{$s[i] == var\_1$}
                        \If{$parse(non\_terminals.find(var\_2), i + 1, j)$}
                            \State $table[var, i, j] \gets 1$
                            \State \textbf{return} $true$
                        \EndIf
                    \EndIf
                    \algstore{part1}
\end{algorithmic}
\end{algorithm}
\FloatBarrier
\begin{algorithm}
\begin{algorithmic}[1]
                \algrestore{part1}
                \ElsIf{$var\_1 \geq \text{} 'A' \land var\_1 \leq \text{} 'Z' \land (var\_2 < \text{}'A' \lor var\_2 > \text{} 'Z')$}
                    \If{$s[j - 1] == var\_2$}
                        \If{$parse(non\_terminals.find(var\_1), i, j - 1)$}
                            \State $table[var, i, j] \gets 1$
                            \State \textbf{return} $true$
                        \EndIf
                    \EndIf
                \EndIf
            \ElsIf{$j - i \geq 3 \land right\_hand.size() == 3$}
                \State $var\_1 \gets right\_hand[0]$
                \State $var\_2 \gets right\_hand[1]$
                \State $var\_3 \gets right\_hand[2]$
                \If{$var\_1 \geq \text{} 'A' \land var\_1 \leq \text{} 'Z' \land (var\_2 < \text{}'A' \lor var\_2 > \text{} 'Z') \land var\_3 \geq \text{} 'A' \land var\_3 \leq \text{} 'Z'$}
                    \For{$k \gets i + 1; k < j - 1$}
                        \If{$parse(non\_terminals.find(var\_1), i, k) \land var\_2 == s[k] \land parse(non\_terminals.find(var\_3), k + 1, j)$}
                            \State $table[var, i, j] \gets 1$
                            \State \textbf{return} $true$
                        \EndIf
                    \EndFor
                \ElsIf{$(var\_1 < \text{} 'A' \lor var\_1 > \text{} 'Z') \land var\_2 \geq \text{}'A' \land var\_2 \leq \text{} 'Z') \land (var\_3 < \text{} 'A' \lor var\_3 > \text{} 'Z')$}
                    \If{$var\_1 == s[i] \land parse(non\_terminals.find(var\_2), i + 1, j - 1) \land var\_3 == s[j - 1]$}
                        \State $table[var, i, j] \gets 1$
                        \State \textbf{return} $true$
                    \EndIf
                \EndIf
            \EndIf
        \EndFor
        \State $table[var, i, j] \gets 2$
        \State \textbf{return} $false$
    \EndProcedure
\end{algorithmic}
\end{algorithm}
\FloatBarrier

Since this parser allows one to parse strings using a linear grammar instead of the Chomsky normal-form version it should be more efficient, indeed the linear grammar possesses less production rules and so the parser should need less recursive calls.

On the other hand since this parser is based on the same logic as the top-down parser its complexity is the same as that last one: $O(n^3)$.
That means that this parser will be more efficient in most practical cases but the same as the top-down parser in some other cases.

\section{Experimentation and comparison with previous parsers}

For this experimentation the first used linear grammar is the following.

\begin{align*}
    &S \to Ac | b\\
    &A \to aS | aB\\
    &B \to bS\\
\end{align*}

The linear parser is given the grammar above as it appears and the other parsers are given the automatically converted grammar as showed in section 4.1.

The used pattern to obtain the following results is `$a\string^ x \text{ } b \text{ } c\string^ x$', the strings generated by this pattern always match the grammar.

\FloatBarrier
\begin{figure}[h]
\begin{tikzpicture}
\begin{groupplot}[group style={group size=2 by 4},height=0.5\textwidth,width=0.5\textwidth]
    \nextgroupplot[title=Linear parser, xlabel=string size, ylabel=recursive calls, legend pos=north west]
    \addplot coordinates {
        (5001, 5001)
        (10001, 10001)
        (15001, 15001)
        (20001, 20001)
        (25001, 25001)
        (30001, 30001)};
    \nextgroupplot[title=Linear parser, xlabel=string size, ylabel=seconds, legend pos=north west]
    \addplot coordinates {
        (5001, 0.096865)
        (10001, 0.356474)
        (15001, 0.793719)
        (20001, 1.4169)
        (25001, 2.16269)
        (30001, 3.06646)};
    \addlegendentry{Total}
    \addplot[mark=+, red] coordinates {
        (5001, 0.095975)
        (10001, 0.354843)
        (15001, 0.791188)
        (20001, 1.41313)
        (25001, 2.15709)
        (30001, 3.05987)};
    \addlegendentry{Init}
    \addplot coordinates {
        (5001, 0.000854)
        (10001, 0.001589)
        (15001, 0.002483)
        (20001, 0.003718)
        (25001, 0.005531)
        (30001, 0.006541)};
    \addlegendentry{Parsing}
    \nextgroupplot[title=`String' bottom-up parser, xlabel=string size, ylabel=iterations, legend pos=north west]
    \addplot coordinates {
        (51, 23350)
        (201, 1358400)
        (401, 10756800)
        (601, 36195200)
        (801, 85673600)
        (1001, 167192000)};
    \nextgroupplot[title=`String' bottom-up parser, xlabel=string size, ylabel=seconds, legend pos=north west]
    \addplot coordinates {
        (51, 0.000608)
        (201, 0.031996)
        (401, 0.268441)
        (601, 0.932661)
        (801, 3.07346)
        (1001, 6.4173)};
    \nextgroupplot[title=`Boolean' bottom-up parser, xlabel=string size, ylabel=iterations, legend pos=north west]
    \addplot coordinates {
        (51, 110755)
        (201, 6768005)
        (401, 53736005)
        (601, 180904005)
        (801, 428272005)
        (1001, 835840005)};
    \nextgroupplot[title=`Boolean' bottom-up parser, xlabel=string size, ylabel=seconds, legend pos=north west]
    \addplot coordinates {
        (51, 0.003579)
        (201, 0.206125)
        (401, 1.75563)
        (601, 6.04026)
        (801, 14.8514)
        (1001, 30.8098)};
    \nextgroupplot[title=Top-down parser, xlabel=string size, ylabel=recursive calls, legend pos=north west]
    \addplot coordinates {
        (51, 41126)
        (201, 2657001)
        (401, 21294001)
        (601, 71911001)
        (801, 170508001)
        (1001, 333085001)};
    \nextgroupplot[title=Top-down parser, xlabel=string size, ylabel=seconds, legend pos=north west]
    \addplot coordinates {
        (51, 0.00054)
        (201, 0.031107)
        (401, 0.248966)
        (601, 0.801141)
        (801, 1.82474)
        (1001, 3.55611)};
\end{groupplot}
\end{tikzpicture}
\caption{Results with linear grammar 1, case 1}
\end{figure}
\FloatBarrier

The results above about the `String' and `boolean' bottom-up and top-down parsers are similar to results obtained before with various grammars.

Is is interesting to see that the linear parser needs $n$ recursive calls to parse a string of size $n$ which matches this grammar, the expected running time is expected to be linear too.
However the initialization process of the memoization table takes basicaly all the running time, the actual parsing time is too small to be relevant.
This shows a limitation of the linear parser in that case, the memoization table limits the input strings sizes that can be parsed.
Even with input strings bigger than $30000$ characters the parsing time is so low that even if it seems to follow a linear behaviour it is not relevant.

The next results are obtained using a set of strings that does not match the grammar, the pattern is `$a\string^ n$'.

The `boolean' bottom-up parser having the same behaviour for a given grammar the results are exactly the same for this pattern so they are not displayed.

\FloatBarrier
\begin{figure}[h]
\begin{tikzpicture}
\begin{groupplot}[group style={group size=2 by 3},height=0.5\textwidth,width=0.5\textwidth]
    \nextgroupplot[title=Linear parser, xlabel=string size, ylabel=recursive calls, legend pos=north west]
    \addplot coordinates {
        (51, 1)
        (201, 1)
        (401, 1)
        (601, 1)
        (801, 1)
        (1001, 1)};
    \nextgroupplot[title=Linear parser, xlabel=string size, ylabel=seconds, legend pos=north west]
    \addplot coordinates {
        (51, 0.00004)
        (201, 0.000215)
        (401, 0.000726)
        (601, 0.001504)
        (801, 0.002412)
        (1001, 0.003654)};
    \addlegendentry{Total}
    \addplot[mark=+, red] coordinates {
        (51, 0.00004)
        (201, 0.000215)
        (401, 0.000726)
        (601, 0.001504)
        (801, 0.002412)
        (1001, 0.003654)};
    \addlegendentry{Init}
    \addplot coordinates {
        (51, 0.000002)
        (201, 0.000002)
        (401, 0.000002)
        (601, 0.000002)
        (801, 0.000002)
        (1001, 0.000002)};
    \addlegendentry{Parsing}
    \nextgroupplot[title=`String' bottom-up parser, xlabel=string size, ylabel=iterations, legend pos=north west]
    \addplot coordinates {
        (51, 22605)
        (201, 1355405)
        (401, 10750805)
        (601, 36186205)
        (801, 85661605)
        (1001, 167177005)};
    \nextgroupplot[title=`String' bottom-up parser, xlabel=string size, ylabel=seconds, legend pos=north west]
    \addplot coordinates {
        (51, 0.000547)
        (201, 0.032757)
        (401, 0.258234)
        (601, 1.19085)
        (801, 2.98934)
        (1001, 6.43343)};
    \nextgroupplot[title=Top-down parser, xlabel=string size, ylabel=recursive calls, legend pos=north west]
    \addplot coordinates {
        (51, 42951)
        (201, 2686801)
        (401, 21413601)
        (601, 72180401)
        (801, 170987201)
        (1001, 333834001)};
    \nextgroupplot[title=Top-down parser, xlabel=string size, ylabel=seconds, legend pos=north west]
    \addplot coordinates {
        (51, 0.000558)
        (201, 0.030692)
        (401, 0.240735)
        (601, 0.798035)
        (801, 1.8229)
        (1001, 3.54812)};
\end{groupplot}
\end{tikzpicture}
\caption{Results with linear grammar 1, case 2}
\end{figure}
\FloatBarrier

With this pattern the top-down needs a few more recursive calls than before, the change is so little that its running time is not touched by it.

With that pattern the number of recursive calls for the linear parser is always 1 since it just goes through the right-hands of `S' and then return false.
Once again the running time is high because of the initialization process, with that case the parsing time is constant and so small that it is not possible to represent it properly.

A limitation of the linear parser for this grammar in this form is the memoization table, indeed the parsing process is so short that it would be better to just use a naive version of this parser by not using the table.
This for two reason, the first obvious reason is that with this grammar the long part is to allocate the memory for the table.
The second reason is that once again with string longer than 40000 characters the used machine is not able to allocate enough memory for the table.

\subsection{Well balanced parentheses}

To really compare the efficiency of the linear parser with the other ones the best thing is to compare how they deal with the well balanced parenthese grammar.

The linear form of this grammar is the following.

\begin{align*}
    &S \to SS | (S) | ()
\end{align*}

The linear parser results will be compared with the top-down parser results in section 3.2.1.

With this grammar the linear parser behaves differently depending on the pattern of the strings.
The first considered case is the pattern `$)\string^ n$'.

\FloatBarrier
\begin{figure}[h]
\begin{tikzpicture}
\begin{groupplot}[group style={group size=2 by 2},height=0.5\textwidth,width=0.5\textwidth]
    \nextgroupplot[title=Linear parser, xlabel=string size, ylabel=recursive calls, legend pos=north west]
    \addplot coordinates {
        (100, 4951)
        (200, 19901)
        (400, 79801)
        (600, 179701)
        (800, 319601)
        (1000, 499501)};
    \nextgroupplot[title=Linear parser, xlabel=string size, ylabel=seconds, legend pos=north west]
    \addplot coordinates {
        (100, 0.000117)
        (200, 0.000346)
        (400, 0.00112)
        (600, 0.002303)
        (800, 0.004045)
        (1000, 0.006164)};
    \addlegendentry{Total}
    \addplot coordinates {
        (100, 0.000023)
        (200, 0.000083)
        (400, 0.000285)
        (600, 0.000526)
        (800, 0.001056)
        (1000, 0.001518)};
    \addlegendentry{Init}
    \addplot coordinates {
        (100, 0.000085)
        (200, 0.000252)
        (400, 0.000814)
        (600, 0.001761)
        (800, 0.002977)
        (1000, 0.004633)};
    \addlegendentry{Parsing}
    \nextgroupplot[title=Top-down parser, xlabel=string size, ylabel=recursive calls, legend pos=north west]
    \addplot coordinates {
        (100, 14851)
        (200, 59701)
        (400, 239401)
        (600, 539101)
        (800, 958801)
        (1000, 1498501)};
    \nextgroupplot[title=Top-down parser, xlabel=string size, ylabel=seconds, legend pos=north west]
    \addplot coordinates {
        (100, 0.000317)
        (200, 0.00112)
        (400, 0.004209)
        (600, 0.008897)
        (800, 0.015769)
        (1000, 0.021436)};
    \addlegendentry{Total}
    \addplot coordinates {
        (100, 0.000119)
        (200, 0.000363)
        (400, 0.00129)
        (600, 0.002816)
        (800, 0.004587)
        (1000, 0.007104)};
    \addlegendentry{Init}
    \addplot coordinates {
        (100, 0.000188)
        (200, 0.000742)
        (400, 0.002894)
        (600, 0.006056)
        (800, 0.011159)
        (1000, 0.017006)};
    \addlegendentry{Parsing}
\end{groupplot}
\end{tikzpicture}
\caption{Results with linear grammar 2, case 1}
\end{figure}
\FloatBarrier

Here the two parsers running times follow the same behaviour as their number of recursive calls, the linear parser needs less recursive calls and so has a lower running time than the top-down parser.

For the second case the same parsers are used and the used pattern is `$(\string^ n$'.

\FloatBarrier
\begin{figure}[h]
\begin{tikzpicture}
\begin{groupplot}[group style={group size=2 by 2},height=0.5\textwidth,width=0.5\textwidth]
    \nextgroupplot[title=Linear parser, xlabel=string size, ylabel=recursive calls, legend pos=north west]
    \addplot coordinates {
        (100, 87026)
        (200, 681551)
        (400, 5393101)
        (600, 18134651)
        (800, 42906201)
        (1000, 83707751)};
    \nextgroupplot[title=Linear parser, xlabel=string size, ylabel=seconds, legend pos=north west]
    \addplot coordinates {
        (100, 0.001216)
        (200, 0.007831)
        (400, 0.054993)
        (600, 0.180519)
        (800, 0.425874)
        (1000, 0.805482)};
    \addlegendentry{Total}
    \addplot coordinates {
        (100, 0.000031)
        (200, 0.000104)
        (400, 0.000292)
        (600, 0.00069)
        (800, 0.001011)
        (1000, 0.00142)};
    \addlegendentry{Init}
    \addplot[mark=+, brown] coordinates {
        (100, 0.001175)
        (200, 0.007718)
        (400, 0.054686)
        (600, 0.179794)
        (800, 0.42483)
        (1000, 0.804034)};
    \addlegendentry{Parsing}
    \nextgroupplot[title=Top-down parser, xlabel=string size, ylabel=recursive calls, legend pos=north west]
    \addplot coordinates {
        (100, 340801)
        (200, 2696601)
        (400, 21453201)
        (600, 72269801)
        (800, 171146401)
        (1000, 334083001)};
    \nextgroupplot[title=Top-down parser, xlabel=string size, ylabel=seconds, legend pos=north west]
    \addplot coordinates {
        (100, 0.004185)
        (200, 0.032133)
        (400, 0.25005)
        (600, 0.805403)
        (800, 1.97674)
        (1000, 3.68684)};
    \addlegendentry{Total}
    \addplot coordinates {
        (100, 0.000131)
        (200, 0.00039)
        (400, 0.001275)
        (600, 0.002598)
        (800, 0.005133)
        (1000, 0.0072)};
    \addlegendentry{Init}
    \addplot[mark=+, brown] coordinates {
        (100, 0.004044)
        (200, 0.031729)
        (400, 0.248752)
        (600, 0.802779)
        (800, 1.97158)
        (1000, 3.67961)};
    \addlegendentry{Parsing}
\end{groupplot}
\end{tikzpicture}
\caption{Results with linear grammar 2, case 2}
\end{figure}
\FloatBarrier

Here the initialization time is negligible before the parsing time for both parsers.

With that grammar the linear parser is more efficient than the top-down parser which was by far the most efficient compared to the other parser in section 3.2.1.

Here is the verification of the complexity of the linear parser.

\begin{align*}
    &y = 8.04034 \cdot 10^{-10} \cdot x^3
\end{align*}

\FloatBarrier
\begin{figure}[h]
\centering
\begin{tikzpicture}
\begin{groupplot}[group style={group size=1 by 1},height=0.5\textwidth,width=0.5\textwidth]
    \nextgroupplot[title=linear parser, xlabel=string size, ylabel=seconds, legend pos=north west]
    \addplot[domain=0:1000, samples=100, line width=1.5pt, green] {8.04034*(10^(-10))*(x^3)};
    \addlegendentry{Theory}
    \addplot[only marks, mark=*, mark size=2pt] coordinates {
        (100, 0.001175)
        (200, 0.007718)
        (400, 0.054686)
        (600, 0.179794)
        (800, 0.42483)
        (1000, 0.804034)};
    \addlegendentry{Real}
\end{groupplot}
\end{tikzpicture}
\caption{Checking theorical fit, linear parser}
\end{figure}
\FloatBarrier

The real points fit perfectly the theorical curve which proves that the complexity of the linear parser really is $O(n^3)$.


\documentclass[12pt]{report}
\usepackage{amsmath}
\usepackage{graphicx}
\title{Project 1:Exploring Weather Trends}
\author{Jared Thacker}

\begin{document}
\maketitle

\chapter{Report}
\section{ Outline}
In this project PostgreSQL was used to obtain both the global and local Atlanta temperature data from a database kept by Udacity.  The SQL statements to extract the Atlanta temperature data is:\\

\noindent SELECT * \\
FROM city\textunderscore data \\
WHERE country = 'United States' AND city = 'Atlanta'\\
LIMIT 100 \\

\noindent And the SQL statements to extract the global temperature data is: \\

\noindent SELECT * \\
FROM global\textunderscore data \\

\noindent Python, with the pandas and matplotlib libraries, was then used to load and visualize the data. The rolling mean method from pandas was used to calculate the moving average with a window of 5. This means that the previous 5 data points (including the current data point) were used to calculate the average at the current point in time.  A window of $n = 20$ was initially chosen but tended to smooth out the curve too much, so a window of $n = 5$ was settled upon and maintained the noise in the graph, but also captured the trend in the data well.
\begin{figure}
	\center{\includegraphics[width=\textwidth]
        {globalAndAtlanta.png}}
        \caption{\label{fig:atlantaDataAndTrend} The Atlanta temperature data and moving average to show the trend in the data.}
\end{figure}

\begin{figure}
	\center{\includegraphics[width = \textwidth]
	{globalAndAtlantaRawData.png}}
	\caption{\label{fig:globalDataAndTrend} The global temperature data and moving average to show the trend in the data.}
\end{figure}

\section{Observations}
The moving averages definitely show that the average global temperature increases as time has passed, however, it is rather unclear with the Atlanta data. In addition, the Atlanta moving average tends to dip and peak in the same place as does the global moving average; this is most easily seen in at approximately year 1825 by comparing the moving averages between the two plots. Third, the temperature in Atlanta is definitely warmer than the global average temperature; the Atlanta temperature is only lower than the global average at one point in time. Lastly, while both temperature trends increase as time passes, the increase in global temperature is more dramatic than Atlanta's as can be seen when comparing the moving averages.\\

\noindent In figure (\ref{fig:atlantaDataAndTrend}), the moving average and mean temperatures for the globe and Atlanta are plotted in the same figure and in figure (\ref{fig:globalDataAndTrend}) the raw time-series and moving averages are plotted both globally and for Atlanta. In the paragraphs that follow, visual inspection as well as the Dickey-Fuller test is used to come to conclusions about the data.\\

\noindent If we visually inspect the raw data (orange curve) for Atlanta in figure (\ref{fig:globalDataAndTrend}) there does not seem to be an upward trend (values get steadily larger) or a downward trend (values get steadily smaller). In addition, there does not seem to be significantly large differences in each value in the time-series with the exception in the late 1700s which implies a constant variance in time in this time-series. Therefore, upon visual inspection, it looks like the Atlanta data is stationary and has no trend. In figure (\ref{fig:atlantaDickeyFullerTest}), the results of the Dickey-Fuller test performed on the Atlanta data are given. Since the test statistic is smaller any of the critical values, we can conclude that the data Atlanta is stationary. This means that, at least with the data collected so far, there is no trend in the Atlanta data that shows that temperature is increasing or varying signifcantly. \\

\begin{figure}
	\center{\includegraphics[width = \textwidth]
	{atlantaDickeyFullerTest.png}}
	\caption{\label{fig:atlantaDickeyFullerTest} The results of the Dickey-Fuller test for the Atlanta data.}
\end{figure}

\noindent If we visually inspect the raw (blue curve) for the global data in figure (\ref{fig:globalDataAndTrend}) there is definitely an upward trend (values get larger). This can be seen more clearly in figure (\ref{fig:atlantaDataAndTrend}) where the values past around 1925 seem to \textit{stay} above the mean (the red horizontal line).  In addition, the variance seems to decrease as time goes on since the difference between the values seems to decrease as time goes on. Therefore, upon visual inspection, the global time-series data is non-stationary and exhibits trends, the most important of which that the average global temperature is increasing as time passes (global warming). We can confirm this by referring to the Dickey-Fuller test in figure (\ref{fig:globalDickeyFullerTest}) . The test statistic is not smaller than any of the critical values which means that we cannot reject that the time-series is non-stationary. Therefore, by visual comparison and by the Dickey-Fuller test, the global time-series has an upwards (increasing) trend and is non-stationary. Therefore, we can conclude that the average temperature of the earth is increasing.

\begin{figure}
	\center{\includegraphics[width = \textwidth]
	{globalDickeyFullerTest.png}}
	\caption{\label{fig:globalDickeyFullerTest} The results of the Dickey-Fuller test for the global data.}
\end{figure}

\end{document}


\documentclass[12pt]{article}

% packages
\usepackage{enumitem} % enumerations
\usepackage{fancyhdr} % header
\usepackage[a4paper, margin=2.5cm]{geometry} % customize page
\usepackage[colorlinks=true]{hyperref} % links
\usepackage{xcolor} % colors

% header
\pagestyle{fancy}
\fancyhead[L]{\textbf{Exercise Sheet 1: Fundamentals}}
\fancyhead[R]{Data Science 1 -- Winter 21/22}

% colors and commands
\definecolor{kitgreen}{HTML}{00876C}
\definecolor{kitblue}{HTML}{4664AA}

\hypersetup{allcolors=kitblue}

\newcommand{\code}[1]{\textcolor{kitgreen}{\texttt{#1}}}
\newcommand{\taskname}[1]{\textcolor{kitblue}{\textbf{[#1]}}}

\begin{document}

\section*{Setup}

For this exercise, you should install the packages \code{matplotlib}, \code{pandas}, \code{scikit-learn}, \code{scipy}, and \code{seaborn}.
If you don't want to install them individually, you may also use the file \code{requirements.txt} from ILIAS.
See \code{DS1\_Python\_Setup.pdf} for more information.

All these packages have detailed documentation as well as countless Stack Overflow questions, which both might help to solve the following tasks.

\section*{Task: Statistics and Plots}

The aim of this exercise is to compute descriptive statistics and create different types of plots.
We use the world-renowned \code{iris} dataset, which was not only featured in the lecture, but even has \href{https://en.wikipedia.org/wiki/Iris_flower_data_set}{its own Wikipedia entry}.

\begin{enumerate}[label=\alph*), left=0pt, itemsep=12pt]
	\item
	\taskname{Loading}
	Obtain the dataset with the function \code{load\_iris()} from the package \code{sklearn.datasets}.
	Combine the values of the properties \code{data}, \code{feature\_names}, \code{target}, and \code{target\_names} from the loaded object into one \code{pandas.DataFrame}.
	Name the target column \code{species}.
	View the resulting \code{DataFrame}.
	\item
	\taskname{Descriptive Statistics}
	Compute descriptive statistics like mean, standard deviation etc. for the numeric features of the dataset.
	You may call methods to compute individual statistics like \code{mean()} as well as a summary with \code{describe()}, both applicable to the whole \code{DataFrame} as well as single columns.
	\newline
	To summarize the target variable, count how often each \code{species} occurs.
	\item
	\taskname{Distribution Plots}
	Choose at least one of the features and create a histogram as well as a boxplot with the package \code{matplotlib.pyplot}, \code{seaborn}, or using the \code{plot()} method of \code{DataFrame}.
	Try changing the number of buckets used in the histogram.
	\item
	\taskname{Scatter Plots}
	Create a scatter plot with one feature on one axis and another feature on the other axis.
	Color the data points according to \code{species}.
	\item
	\taskname{Grouped Boxplots}
	Use \code{seaborn} to create a boxplot of one numeric feature, having a separate box for each \code{species}.
	Repeat this procedure for each feature.
	\item
	\taskname{$\chi^2$ Test}
	Conduct $\chi^2$ tests to examine the relationship between each of the numeric features and the target \code{species}.
	You may use \code{chi2\_contingency()} from \code{scipy.stats} for the tests.
	As preparation, you may use \code{crosstab()} from \code{pandas} to create contingency tables and \code{cut()} from \code{pandas} to discretize features.
	\newline
	How do you interpret the results?
	What is the relationship between the results of the statistical tests and the plots from the previous sub-task?
\end{enumerate}

\end{document}


\documentclass{article}
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage{fancyhdr}
\usepackage{extramarks}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{amsfonts}
\usepackage{tikz}
\usepackage[plain]{algorithm}
\usepackage{algpseudocode}
\usepackage{fancyvrb}
\usepackage{multirow}
\usepackage{booktabs}
\usetikzlibrary{automata,positioning}
\usepackage{subfigure}

%
% Basic Document Settings
%

\topmargin=-0.45in
\evensidemargin=0in
\oddsidemargin=0in
\textwidth=6.5in
\textheight=9.0in
\headsep=0.25in

\linespread{1.1}

\pagestyle{fancy}
\lhead{\hmwkAuthorName}
\chead{\hmwkClass\ : \hmwkTitle}
\rhead{\firstxmark}
\lfoot{\lastxmark}
\cfoot{\thepage}

\renewcommand\headrulewidth{0.4pt}
\renewcommand\footrulewidth{0.4pt}

\setlength\parindent{0pt}

%
% Create Problem Sections
%

\newcommand{\enterProblemHeader}[1]{
    \nobreak\extramarks{}{Problem \arabic{#1} continued on next page\ldots}\nobreak{}
    \nobreak\extramarks{Problem \arabic{#1} (continued)}{Problem \arabic{#1} continued on next page\ldots}\nobreak{}
}

\newcommand{\exitProblemHeader}[1]{
    \nobreak\extramarks{Problem \arabic{#1} (continued)}{Problem \arabic{#1} continued on next page\ldots}\nobreak{}
    \stepcounter{#1}
    \nobreak\extramarks{Problem \arabic{#1}}{}\nobreak{}
}

\setcounter{secnumdepth}{0}
\newcounter{partCounter}
\newcounter{homeworkProblemCounter}
\setcounter{homeworkProblemCounter}{1}
\nobreak\extramarks{Problem \arabic{homeworkProblemCounter}}{}\nobreak{}

%
% Homework Problem Environment
%
% This environment takes an optional argument. When given, it will adjust the
% problem counter. This is useful for when the problems given for your
% assignment aren't sequential. See the last 3 problems of this template for an
% example.
%
\newenvironment{homeworkProblem}[1][-1]{
    \ifnum#1>0
        \setcounter{homeworkProblemCounter}{#1}
    \fi
    \section{Problem \arabic{homeworkProblemCounter}}
    \setcounter{partCounter}{1}
    \enterProblemHeader{homeworkProblemCounter}
}{
    \exitProblemHeader{homeworkProblemCounter}
}

%
% Homework Details
%   - Title
%   - Due date
%   - Class
%   - Section/Time
%   - Instructor
%   - Author
%

\newcommand{\hmwkTitle}{Homework\ \# 4}
\newcommand{\hmwkDueDate}{\today}
\newcommand{\hmwkClass}{MATH6644 Iterative Methods}
\newcommand{\hmwkClassTime}{}
\newcommand{\hmwkClassInstructor}{}
\newcommand{\hmwkAuthorName}{Komahan Boopathy}

%
% Title Page
%

\title{
    \vspace{2in}
    \textmd{\textbf{\hmwkClass:\ \hmwkTitle}}\\
    \normalsize\vspace{0.1in}\small{\hmwkDueDate}\\
    %\vspace{0.1in}\large{\textit{\hmwkClassInstructor\ \hmwkClassTime}}
    \vspace{3in}
}

\author{\textbf{\Large\hmwkAuthorName}}
\date{}

\renewcommand{\part}[1]{\textbf{\large Part \Alph{partCounter}}\stepcounter{partCounter}\\}

%
% Various Helper Commands
%

% Useful for algorithms
\newcommand{\alg}[1]{\textsc{\bfseries \footnotesize #1}}

% For derivatives
\newcommand{\deriv}[1]{\frac{\mathrm{d}}{\mathrm{d}x} (#1)}

% For partial derivatives
\newcommand{\pderiv}[2]{\frac{\partial #1}{\partial #2}}

% Integral dx
\newcommand{\dx}{\mathrm{d}x}

% Alias for the Solution section header
\newcommand{\solution}{\textbf{\large Solution}}

% Probability commands: Expectation, Variance, Covariance, Bias
\newcommand{\E}{\mathrm{E}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\Cov}{\mathrm{Cov}}
\newcommand{\Bias}{\mathrm{Bias}}

\begin{document}
\maketitle
\thispagestyle{empty}

\pagebreak

\begin{homeworkProblem}

  \emph{Can the performance of Newton iteration be improved by a
    linear change of variables?  That is, for nonsingular $N\times N$
    matrices $A$ and $B$, can the Newton iterates for $F(x)=0$ and
    $AF(Bx)=0$ show any performance difference when started at the same
    initial iterate? What about the chord method?
  }

  \medskip

  The transformation of varaibles will change the condition number of
  the Jacobian. The problem can be easier (or difficult) to solve than
  the original problem. The convergence of iterative methods for the
  solution of linear system depnds on the condition number of the
  Jacobian matrix.

  We should get the same solution $x$ with Newton's method. The
  Newton's method is given by:
  $$
  x_{n+1} = x_{n} - F'(x_n)^{-1}F(x_n)
  $$
  Introduce a change of variables such that $x = A^{-1} y$.  Let $G(y)$ be
  the new function and $G'(x)$ be the new Jacobian. The Newton iteration
  is:
  $$
  y_{n+1} = y_{n} - G'(y_n)^{-1}G(y_n)
  $$
  Multiply by $A^{-1}$ on both sides
  $$
  A^{-1}y_{n+1} = A^{-1}y_{n} - A^{-1}G'(y_n)^{-1}G(y_n)
  $$
  Using $A^{-1}y =x$, this becomes:
  $$
  x_{n+1} = x_{n} - A^{-1} G'(y_n)^{-1}G(y_n)
  $$
  Now expaning the second term in term of the original function and
  variables:
  $$
  x_{n+1} = x_{n} - A^{-1} AF'(A^{-1}y_n)^{-1}A^{T}A^{-T}G(A^{-1}y_n)
  $$
  Using $A^{-1}y =x$, this becomes:
  $$
  x_{n+1} = x_{n} - F'(x_n)^{-1}F(x_n)
  $$ which is Newton's method in terms of the original variable $x$.

  Assume that standard assumptions hold, then Newton iterates
  converge quadratiacally such that
  $$ || x_{n+1} - x^* || \le \beta || x_{n} - x^*||^2
  $$
  where $\beta>0$ is a constant.

  If we use transmation of varaibles for $x$:
  $$ || A^{-1}y_{n+1} - A^{-1}y^* || \le \beta || A^{-1}y_{n} - A^{-1}y^*||^2
  $$ The definition of the norm of now different than before and we
  expect the performance to change. For chord method, we have linear
  order of convergence.  $$ || A^{-1}y_{n+1} - A^{-1}y^* || \le \beta
  || A^{-1}y_{n} - A^{-1}y^*|| $$

  The chord method will be affected \emph{comparatively} less by
  change of variables.

\end{homeworkProblem}

\pagebreak

\begin{homeworkProblem}

  \emph{Assume that the standard assumptions hold, that the cost of a
    function evaluation is ${\cal{O}}(N^2)$ floating-point operations,
    the cost of a Jacobian is ${\cal{O}}(N)$ function evaluations, and
    that $x_0$ is near enough to $x^*$ so that the Newton iteration
    converges quadratically to $x^*$.}

  \medskip

  \paragraph{a.}
  \emph{Estimate what is the number of iteration needed to obtain $\|e_n\| \le \epsilon \|e_0\|$, where $\epsilon$ is a small tolerance value.}

  $$
  ||e_{n}|| \le \beta ||e_{n-1}||^\alpha
  $$ where $\beta>0$ is a constant and $\alpha =2$ is the order of
  convergence of Newton's method. Newton's method will converge
  \emph{quadratically}.
  Using the above relation,
  $$   ||e_{1}|| \le \beta ||e_{0}||^2$$
  $$   ||e_{2}|| \le \beta ||e_{1}||^2 =  \beta \left( \beta ||e_{0}||^2 \right)^2 = \beta^3 ||e_0||^{2^2} $$
  $$   ||e_{3}|| \le \beta ||e_{2}||^2 = \beta \left( \beta^3 ||e_0||^{2^2} \right)^2 = \beta^{2*3+1} ||e_0||^{2^3} $$
  Therefore,
  $$ ||e_{n}|| \le \beta ||e_{n-1}||^2  = \beta^{2*n+1} ||e_0||^{2^n} $$

  This can be rearranged to get:
  $$
  \frac{||e_{n}||}{||e_{0}||} \le \beta^{2*n+1} ||e_0||^{2^n - 1} \le \epsilon
  $$
  We ignore the contribution of $\beta^{2*n+1}$ as it is not comparatively dominant (for simplicity).

  We use:
  $$
  ||e_0||^{2^n - 1} \le \epsilon
  $$
  Taking logarithm on both sides

  $$
  (2^n -1) \ln(||e_0||) \le \ln (\epsilon)
  $$
  $$
  2^n \le \frac{ \ln (\epsilon)}{ \ln (||e_0||)} + 1
  $$
  Taking logarithm again and rearranging we get:

  $$
  n \le  \frac{ \ln \left( \frac{ \ln (\epsilon)}{ \ln (||e_0||)} + 1 \right)}{\ln (2)}
  $$

  \paragraph{b.}
  \emph{What is the number of floating point operations required to
    get this accuracy?}  At each Newton's iteration $n$ there is:

  \begin{enumerate}
  \item one function evaluation: ${\cal{O}}(N^2)$
  \item one Jacobian evaluation takes N function evaluations: ${\cal{O}}(N^3)$
  \item one factorization of Jacobian is ${\cal{O}}(N^3)$  (assuming LU decomposition)
    used)
  \item forward and back-substitutions each take ${\cal{O}}(N^2)$
  \end{enumerate}

  Floating point operations per iteration: ${\cal{O}}(N^2+N^3+N^2+N^2)={\cal{O}}(3N^2+N^3)$
  Total floating point operations for Newton method : $n*{\cal{O}}(3N^2+N^3)$

\end{homeworkProblem}

\pagebreak

\begin{homeworkProblem}

  \emph{Answer the questions in the previous problem for the chord method.}

  \medskip

  \paragraph{a.}
  \emph{Estimate what is the number of iteration needed to obtain
    $\|e_n\| \le \epsilon \|e_0\|$, where $\epsilon$ is a small
    tolerance value.}

  The chord method exhibits linear convergence $\alpha=1$. Therefore,
  the error relation to the initial error is as follows:
  $$ ||e_{n}|| \le \beta ||e_{n-1}||  = \beta^{n} ||e_0|| $$
  Rearranging this:
  $$
  \frac{||e_{n}||}{||e_{0}||} \le \beta^{n} \le \epsilon
  $$
  Taking logarithm on both sides
  $$
  n  \ln(\beta) \le \ln (\epsilon)
  $$
  This results in the following estimate for the number of
  iterations:
  $$
  n \le \frac{\ln (\epsilon)}{\ln (\beta)}
  $$

  \paragraph{b.}
  \emph{What is the number of floating point operations required to
    get this accuracy?}  At each Newton's iteration $n$ there is:

  \begin{enumerate}
  \item one function evaluation: ${\cal{O}}(N^2)$
  \item one Jacobian evaluation takes N function evaluations: ${\cal{O}}(N^3)$
  \item one factorization of Jacobian is ${\cal{O}}(N^3)$  (assuming LU decomposition)
    used)
  \item forward and back-substitutions each take ${\cal{O}}(N^2)$
  \end{enumerate}

  Floating point operations per iteration: ${\cal{O}}(N^2+N^3+N^2+N^2)={\cal{O}}(3N^2+N^3)$
  Total floating point operations for Chord method : $n*{\cal{O}}(3N^2+N^3)$

\end{homeworkProblem}

\pagebreak

\begin{homeworkProblem}

  \emph{Write the program that solves single nonlinear equations with
    Newton's method, the chord method, and the secant method. For the
    secant method, use $x_{-1}=0.99x_0$. Apply your program to the
    following function/initial iterate combinations, document and
    explain your results.}
  \begin{enumerate}
  \item $f(x) = 2x^2 -5, \quad x_0=10$
  \item $f(x) = \sin x + x, \quad x_0=0.5$
  \item $f(x) = \cos x, \quad x_0=3$
  \end{enumerate}
  \medskip

  \begin{table}[H]
    \caption{Number of iterations taken to for the test case.} \medskip \centering
    \begin{tabular}{c|c|ccc}
      \toprule
     Case & Test/Method         & Newton & Chord & Secant \\
      \midrule
     a & $f(x) = 2x^2 -5$    & 6 & 8  & 64  \\\midrule
     b & $f(x) = \sin x + x$ & 3 & 3  & --   \\\midrule
     c & $f(x) = \cos x$     & 4 & --  & -- \\
      \bottomrule
    \end{tabular}
    \label{tab:number_iterations}
  \end{table}
  The stopping criterion used in $\tau_r=\tau_a=10^{-6}$.

  \begin{figure}[H]
    \centering
    \begin{minipage}{0.45\linewidth}
      \includegraphics[width=\textwidth]{f1.pdf}
    \end{minipage}
    \begin{minipage}{0.45\linewidth}
      \includegraphics[width=\textwidth]{f2.pdf}
    \end{minipage}
    \begin{minipage}{0.45\linewidth}
      \includegraphics[width=\textwidth]{f3.pdf}
    \end{minipage}
    \caption{Convergence history with different methods: case (a) (left), case (b) middle, case (c) right .}
    \label{convergence}
  \end{figure}

  Figure~\ref{convergence} shows the convergence of the three methods
  for these problems.

 Secant and chord method do not yield the solution for case
      (c). The tolerance is satisfied for secant method (shown in
      green), but the solution is not the correct solution. This tells
      that the stopping criteria is not robust. For the chord method
      (shown in blue), the the iterates keep cycling and they do not
      converge at all.

About the convergence rates:
  \begin{itemize}
    \item The Newton's method is quadratic
    \item The secant method is superlinear
    \item The chord method is linear
  \end{itemize}

\end{homeworkProblem}

\end{document}


\documentclass{scrartcl}
\usepackage[utf8]{inputenc}
\usepackage{blindtext}
\usepackage{biblatex}
\bibliography{bibliography}
\usepackage{booktabs}
\usepackage{pgfplots}
\usepackage{subcaption}

\title{Towards Paralleled N-Body Interaction}
\subject{INF560 Report}
\author{Zhengqing Liu  ~~Jiongyan Zhang}
\date{March 2022}

\begin{document}

\maketitle

\noindent \textbf{Abstract} The N-body problem has been widely studied and applied in various fields. Since the traditional Brute Force algorithm is computationally expensive and over time-consuming, the Barnes Hut quad-tree algorithm is proposed to simulate the particle force, which greatly improves the efficiency of the algorithm. Nowadays, the advent of parallel computing has given new born to these legacy algorithms. With MPI, OpenMP, and CUDA, we enforce parallelization at various degrees to experiment with the performance improvement upon Brute Force and Barnes Hut algorithms. The performance evaluation by exploiting different parallel paradigms as per the input scope are given in details.

\section{Introduction}
\noindent
N-body simulation consists of the evolution of a dynamic system of N bodies considering their mass, positions and initial speed, which is extensively used in astrophysics for studying how galaxies composed of millions of bodies behave~\cite{wiki}.
Computers have been used to simulate N-body problems since the 1960s.
Its main principle is based on Newtonian mechanics, simulating the force between each particle to calculate the direction and position at the next movement~\cite{aarseth2003gravitational}.
As the computing power increases, multiple algorithms (i.e., Barnes Hut, etc.) are used to replace this basic algorithm~\cite{trenti2008n}.
However, the development of computing power did not stop there.
The availability of multicore processors has opened a new pan for high performance computing.
Parallelization unleashes the multi-thread or multi-process capabilities of CPUs and GPUs, along with distributed storage or shared storage, to divide and conquer so that the algorithm can leverage the computing power to solve the problem.

\noindent In the rest of this paper, we target at optimizing two classic algorithms (Brute Force and Barnes Hut) with several parallel paradigms, and evaluate the performance of various modes.

\section{Brute Force}
\noindent
Brute force is a naive algorithm that consists, for each time step, of computing the gravitational force applied to each particle, and then computing its new position respectively.
The force applied to particle $p_1$ is computed by summing the force that all the other particles apply to $p_1$, which takes the complexity of $O(n^2)$.
Hence, the loops of force computation and position movement are the two essential parts to unleash the capability of parallelism.

\begin{figure}[!h]
\centering
\subcaptionbox{Original}{
\begin{tikzpicture}[scale=0.6]
\begin{axis}[
    xlabel= Particle number,
    ylabel= Time(s),
    tick align=outside,
    legend style={at={(-0.2,0.6)},anchor=north}
    ]

\addplot[smooth,mark=*,blue] plot coordinates {
    (100,0.0087)
    (500,0.3814)
    (1000,2.3094)
    (1500,6.7026)
    (2000,14.7978)
    (2500,25.0299)
};

\end{axis}
\end{tikzpicture}
}
\subcaptionbox{MPI}{
\begin{tikzpicture}[scale=0.6]

\begin{axis}[
    xlabel= Process number,
    ylabel= Time (s),
    tick align=outside,
    legend style={at={(1.2,0.9)},anchor=north}
    ]
\addplot[smooth,mark=*,green] plot coordinates {
    (1,0.3814)
    (2,0.3894)
    (3,0.2012)
    (4,0.1356)
    (5,0.1034)
    (6,0.1249)
};
\addlegendentry{500 particles}

\addplot[smooth,mark=*,purple] plot coordinates {
    (1,2.3094)
    (2,2.4071)
    (3,1.2224)
    (4,0.9334)
    (5,0.8675)
    (6,0.7857)
};

\addlegendentry{1000 particles}

\addplot[smooth,mark=*,blue] plot coordinates {
    (1,6.7026)
    (2,6.992950)
    (3,3.511393)
    (4,2.398373)
    (5,1.814117)
    (6,2.189829)
};

\addlegendentry{1500 particles}

\addplot[smooth,mark=*,cyan] plot coordinates {
    (1,14.7978)
    (2,15.114640)
    (3,7.583416)
    (4,5.188921)
    (5,3.939495)
    (6,3.930487)
};

\addlegendentry{2000 particles}

\end{axis}
\end{tikzpicture}
}
\caption{Orignal and MPI performance of Brute Force}
\label{fig:name}
\end{figure}

\subsection{MPI}
MPI (Message-Passing Interface)\footnote{https://www.mpi-forum.org/} is a message-passing library interface specification.
It addresses primarily the message-passing parallel programming model, in which data is moved from the address space of one process to that of another process through cooperative operations on each process.

\noindent To utilize the distributed-memory paradigm, we define a \textit{master} process and multiple \textit{slave} processes according to the given processes.
While slaves take force computation tasks equally, master finishes the modulo task and moves all the particles.
Collective communication, such as broadcast and gather,  facilitates the synchronization and collection.
However, since the algorithm encompasses several arguments (i.e., force, velocity, position), synchronizing all of them would require broadcast with a couple of steps.
To this end, a new MPI datatype, \textit{particle\_mpi\_t}, is declared in our implementation, thus the whole particle structures can be transmitted at once.
In case that the number of modulus task of master is not same as those of slave, gatherv is used to gather sets of particles in disparate lengths.
In addition, the global variables, \textit{max\_acc} and \textit{max\_speed}, should also be updated among all the processes.

\subsection{MPI + OpenMP}
The OpenMP (Open Multi-Processing)\footnote{https://www.openmp.org/} supports multi-platform shared-memory parallel programming in C/C++ and Fortran.
To leverage both MPI and OpenMP for N-body problem, a fine-grain mode of hybrid programming is implemented - distribution of tasks over MPI ranks and process of each subpart with OpenMP.
Optimization of loop scheduling is a huge advantage of fine granurity.
In our scenarios, each slave rank computes the forces added up to a set of particles, which are specified by the master.
By creating omp parallel region for each slave, the for-loop of computing force can speed up. In this way, only few OpenMP constructs and clauses are involved and meanwhile relatively better performances are obtained.

\noindent In addition, setting more than 4 processes, the performance is degraded along with the increment of the number of processes.
This phenomenon conforms to Amdahl's law.
Not only cannot the sequential portion be diminished, but launching and cooperating threads or processes takes costs.

\begin{center}
\begin{table}
\footnotesize
\centering
\begin{subtable}[c]{0.7\textwidth}
\centering
\begin{tabular}{cccc}
\hline
Threads& 1000 particles& 2000 particles& 3000 particles\\
\hline
1& 2.587433& 16.243086& 43.273200\\
2& 2.336531& 14.792514& 39.182745\\
3& 2.255282& 15.018447& 42.185766\\
4& 2.411418& 15.249535& 42.789421\\
5& 2.452970& 14.749308& 43.151368\\
\hline
\end{tabular}
\subcaption{2 Processes}
\end{subtable}

\begin{subtable}[c]{0.7\textwidth}
\centering
\begin{tabular}{cccc}
\hline
Threads& 1000 particles& 2000 particles& 3000 particles\\
\hline
1& 1.327676& 8.315050& 23.706969\\
2& 1.894452& 11.965878& 33.056768\\
3& 1.790665& 10.809736& 28.787725\\
4& 1.654219& 9.455634& 24.729652\\
5& 4.189442& 9.502631& 24.380480\\
\hline
\end{tabular}
\subcaption{3 Processes}
\end{subtable}

\begin{subtable}[c]{0.7\textwidth}
\centering
\begin{tabular}{cccc}
\hline
Threads& 1000 particles& 2000 particles& 3000 particles\\
\hline
1& 0.899367& 5.583197& 15.528205\\
2& 1.586002& 9.062810& 24.073398\\
3& 1.414150& 6.780683& 22.774145\\
4& 4.471012& 9.574463& 23.262464\\
5& 5.686800& 12.974246& 26.736431\\
\hline
\end{tabular}
\subcaption{4 Processes}
\end{subtable}

\caption{MPI+OMP performance on Brute Force }
\end{table}
\end{center}
\subsection{MPI + OpenMP + CUDA}
CUDA is a parallel computing platform and application programming interface that allows software to use certain types of graphics processing unit for general purpose processing.
With the acceleration of dedicated hardware, parallelism can step further.

\noindent The unit task of computing all other forces upon one particle is transferred to CUDA.
After allocating and copying memory to the device side, CUDA exploits its given blocks and threads to implement the force computation.
Due to the computed forces are added up in the various threads, the atomicity should be guaranteed to eschew the asynchronization.
Typically, the built-in \textit{atomicAdd} is constrained to integer operations, thus rewriting the atomic adding for double type is necessary.
Inspired by Marco Bertini's solution~\cite{atomicAdd}, \textit{atomicAddDouble} is implemented to successfully execute double addition.

\noindent However, the performance is degraded with the involvement of CUDA.
And we observe that with the increment of processes, the speed further slows down.
Due to that OMP has trivial effects on the MPI version, we choose 2 omp threads and analyze the performance brought by CUDA.
We choose \textit{thr\_per\_block} as 10, 50, 100 for 100, 500, 1000 particles scenarios respectively.

\begin{table}
\footnotesize
    \centering
    \begin{tabular}{cccc}
    \hline
    Processes& 100 particles& 500 particles& 1000 particles\\
    \hline
    2 (No CUDA)& 0.007055& 0.385060& 2.336531\\
    2& 0.631886& 13.081082& 78.902757\\
    3& 1.409775& 22.233090& 92.376449\\
    4& 1.477007& 22.345278& 109.797773\\
    \hline
    \end{tabular}
    \caption{Hybrid performance}
    \label{tab:my_label}
\end{table}

\section{Barnes Hut}
The Barnes Hut algorithm exploits the sparsity of particles to estimate the force on each particle. Instead of calculating the interaction force among each pair of particles, the algorithm treats the distant group of particles as one large particle and performs mechanical calculations with it~\cite{van2013barnes}. Via the approximate approach, one particle doesn't require to traverse all other particles.The core step in this algorithm is to establish a quad-tree, upon which one can gather space-neighbour particles~\cite{smith1994quad}.\\
The main idea of Barnes Hut parallelism is to make recursive structures loop. We mainly focus on the force computing part, as it is one of the most computing expensive steps and with the highest possibility to achieve parallelism~\cite{arnautovic2013parallelization}. The target is to separate the particles into several parts, which would be convenient for multiple threads or processes to handle independently. Meanwhile, we also try to use parallelism in the recursive function, aiming to finding a
decent level to assign tasks to each thread.
The original non-parallel code possesses the performance as below.\\
\begin{figure}[!h]
\centering

\begin{tikzpicture}[scale=0.6]

\begin{axis}[
    xlabel= Particle number,
    ylabel= Time (s),
    tick align=outside,
    legend style={at={(-0.2,0.6)},anchor=north}
    ]

\addplot[smooth,mark=*,blue] plot coordinates {
    (500,0.1249)
    (1000,0.3709)
    (1500,0.6494)
    (2000,1.0282)
    (2500,1.4578)
    (3000,1.7784)
};

\end{axis}
\end{tikzpicture}
\caption{Original code performance of Barnes Hut}
\label{fig:name}
\end{figure}
\\It shows that with development of the particle number, the code operation time increases with an approximate-linear model.

\subsection{MPI}
\noindent
Due to the distributed memory system and the complexity of a quad-tree structure, we cannot transfer the tree directly. Thus, we use more space for less time. The basic idea is to establish the tree in each process simultaneously. Finally, the tree in each process is the same as the initial parameters are shared. Each process could obtain a certain quantity of tasks in correspondence to the process number. Based on 1000 particles, we add the required process gradually and the result is showed in Figure 3.

\subsection{OpenMP}
\noindent
OpenMP provides a shared memory, thus the tree establishment in each thread is not required. We can just simply apply OpenMP for-loop with the dynamic mode. The parallelism mainly focuses on the force computing process. We try to increase the threads to examine the efficiency of OpenMP in this algorithm:
\begin{figure}[!h]
\centering
\subcaptionbox{MPI}{
\begin{tikzpicture}[scale=0.6]

\begin{axis}[
    xlabel= Thread number,
    ylabel= Time (s),
    tick align=outside,
    legend style={at={(1.2,0.9)},anchor=north}
    ]
\addplot[smooth,mark=*,blue] plot coordinates {
    (1,0.128103)
    (2,0.117752)
    (3,0.102021)
    (4,0.094662)
    (5,0.088839)
    (6,0.086950)
};

\addlegendentry{500 particles}

\addplot[smooth,mark=*,yellow] plot coordinates {
    (1,0.367194)
    (2,0.329829)
    (3,0.296141)
    (4,0.274345)
    (5,0.250986)
    (6,0.242753)
};
\addlegendentry{1000 particles}

\addplot[smooth,mark=*,red] plot coordinates {
    (1,1.048104)
    (2,0.968597)
    (3,0.857459)
    (4,0.782523)
    (5,0.743836)
    (6,0.720818)
};

\addlegendentry{2000 particles}

\addplot[smooth,mark=*,green] plot coordinates {
    (1,2.970725)
    (2,2.539754)
    (3,2.306051)
    (4,2.134788)
    (5,2.033596)
    (6,1.973217)
};

\addlegendentry{4000 particles}
%\addlegendentry{case 1}

\end{axis}
\end{tikzpicture}
}
\subcaptionbox{OMP}{
\begin{tikzpicture}[scale=0.6]

\begin{axis}[
    xlabel= Process number,
    ylabel= Time (s),
    tick align=outside,
    legend style={at={(1.2,0.9)},anchor=north}
    ]
\addplot[smooth,mark=*,green] plot coordinates {
    (1,0.127127)
    (2,0.097437)
    (3,0.087053)
    (4,0.078695)
    (5,0.077142)
    (6,0.117528)
};
\addlegendentry{500 particles}

\addplot[smooth,mark=*,purple] plot coordinates {
    (1,0.373572)
    (2,0.279013)
    (3,0.250259)
    (4,0.235290)
    (5,0.229734)
    (6,0.221141)
};

\addlegendentry{1000 particles}

\addplot[smooth,mark=*,blue] plot coordinates {
    (1,1.062833)
    (2,0.768592)
    (3,0.675868)
    (4,0.635460)
    (5,0.618210)
    (6,0.652331)
};

\addlegendentry{2000 particles}

\addplot[smooth,mark=*,cyan] plot coordinates {
    (1,2.980229)
    (2,2.203253)
    (3,1.977129)
    (4,1.887603)
    (5,1.931117)
    (6,2.158590)
};

\addlegendentry{4000 particles}

\end{axis}
\end{tikzpicture}
}
\caption{MPI and OMP performance in Barnes Hut}
\label{fig:name}
\end{figure}
\\The operating time decreases with the thread number increasing. However, it is clear that the speedup decreases, mainly because of the bus contention.

\subsection{Hybrid of MPI and OpenMP}
\noindent
The hybrid mode of MPI and OpenMP could provide a further choice for parallelism. The combination of two tools comes from the previous two sections. The top level design provides an MPI infrastructure, which follows the each-tree-each-process principle in 3.1 to create an environment for labour division. Inside each process, the for-loop could be parallelized by OpenMP.
\begin{figure}[!h]
\centering
\subcaptionbox{500 particles}
{
\begin{tikzpicture}[scale=0.6]

\begin{axis}[
    xlabel= Thread number,
    ylabel= Time (s),
    tick align=outside,
    legend style={at={(0.7,0.9)},anchor=north}
    ]

\addplot[smooth,mark=*,blue] plot coordinates {
    (1,0.144559)
    (2,0.142259)
    (3,0.130231)
    (4,0.132497)
    (5,0.135561)
    (6,0.138098)
};

\addlegendentry{One process}

\addplot[smooth,mark=*,green] plot coordinates {
    (1,0.102542)
    (2,0.115353)
    (3,0.110164)
    (4,0.104497)
    (5,0.109342)
    (6,0.124713)
};
\addlegendentry{Two processes}

\end{axis}
\end{tikzpicture}
}
\subcaptionbox{1000 particles}
{
\centering
\begin{tikzpicture}[scale=0.6]

\begin{axis}[
    xlabel= Thread number,
    ylabel= Time (s),
    tick align=outside,
    legend style={at={(0.7,0.9)},anchor=north}
    ]

\addplot[smooth,mark=*,blue] plot coordinates {
    (1,0.379917)
    (2,0.388298)
    (3,0.347317)
    (4,0.350786)
    (5,0.365727)
    (6,0.368501)
};

\addlegendentry{One process}

\addplot[smooth,mark=*,green] plot coordinates {
    (1,0.284265)
    (2,0.289831)
    (3,0.278172)
    (4,0.279449)
    (5,0.288258)
    (6,0.296005)
};
\addlegendentry{Two processes}

\end{axis}
\end{tikzpicture}
}
\subcaptionbox{2000 particles}
{
\begin{tikzpicture}[scale=0.6]

\begin{axis}[
    xlabel= Thread number,
    ylabel= Time (s),
    tick align=outside,
    legend style={at={(0.7,0.9)},anchor=north}
    ]

\addplot[smooth,mark=*,blue] plot coordinates {
    (1,1.138275)
    (2,1.155056)
    (3,1.029712)
    (4,1.041404)
    (5,1.048535)
    (6,1.056574)
};

\addlegendentry{One process}

\addplot[smooth,mark=*,green] plot coordinates {
    (1,0.832576)
    (2,0.925001)
    (3,0.793431)
    (4,0.799537)
    (5,0.831406)
    (6,0.845097)
};
\addlegendentry{Two processes}

\end{axis}
\end{tikzpicture}
}
\subcaptionbox{4000 particles}
{
\begin{tikzpicture}[scale=0.6]

\begin{axis}[
    xlabel= Thread number,
    ylabel= Time (s),
    tick align=outside,
    legend style={at={(0.7,0.9)},anchor=north}
    ]

\addplot[smooth,mark=*,blue] plot coordinates {
    (1,3.188606)
    (2,3.242111)
    (3,2.877078)
    (4,2.903363)
    (5,2.940451)
    (6,2.992418)
};

\addlegendentry{One process}

\addplot[smooth,mark=*,green] plot coordinates {
    (1,2.360512)
    (2,2.619732)
    (3,2.240612)
    (4,2.248531)
    (5,2.305573)
    (6,2.329586)
};
\addlegendentry{Two processes}

\end{axis}
\end{tikzpicture}
}
\caption{Hybrid performance of Barnes Hut}
\label{fig:name}
\end{figure}
\\We notice that there would be a time rising when applying the two threads. The possible reason is the additional thread cost.\\
\begin{figure}[!h]
\centering

\begin{tikzpicture}[scale=0.8]

\begin{axis}[
    xlabel=Thread number,
    ylabel=Time (s),
    tick align=outside,
    legend style={at={(1.2,0.9)},anchor=north}
    ]

\addplot[smooth,mark=*,green] plot coordinates {
    (1,0.093081)
    (2,0.092248)
    (3,0.107316)
    (4,0.111796)
    (5,2.511965)
    (6,5.710171)
};

\addlegendentry{500 particles}

\addplot[smooth,mark=triangle,cyan] plot coordinates {
    (1,0.257285)
    (2,0.242022)
    (3,0.247034)
    (4,0.322218)
    (5,3.591909)
    (6,7.811057)
};
\addlegendentry{1000 particles}

\addplot[smooth,mark=triangle,red] plot coordinates {
    (1,0.755377)
    (2,0.704551)
    (3,0.873846)
    (4,0.871475)
    (5,5.479085)
    (6,11.445225)
};
\addlegendentry{2000 particles}

\addplot[smooth,mark=*,blue] plot coordinates {
    (1,2.136210)
    (2,2.033896)
    (3,2.469436)
    (4,2.461708)
    (5,9.864239)
    (6,16.961032)
};

\addlegendentry{4000 particles}

\end{axis}
\end{tikzpicture}
\caption{Abnormal trend in three processes}
\label{fig:name}
\end{figure}
We can then find that the increment of processes and threads contributes to the program's acceleration. However, the abnormal trend appears, i.e., there would be a strange rise with three-process. The context switching of threads probably counts a main reason as the threads in use may exceed the limitation of the device.
\subsection{CUDA}
\noindent
Unlike MPI and OpenMP, the CUDA programming aims to utilizing numerous threads of GPU. The largest problem is how to transfer the tree structure from host to device. To simplify this problem, we try to establish the quad-tree on the device side directly. During the experiments, there are two problems: the stack space in GPU is relatively less, which could not support a normal establishment process; and, the computation of the normal-level amount of particles would be quite slower. Table 3 shows the efficiency of the algorithm based on 1000 and 2000 particles and that more threads lead to accelerate the program.
\begin{center}
\begin{table}
\footnotesize
\begin{center}
\begin{tabular}{ccc}
\hline
Thread number& 1000 particles& 2000 particles\\
\hline
20& 16.026382& 45.959570\\
1000& 14.313604& 40.887589\\
Demo (non-parallel)&0.398095&1.095418\\
\hline
\end{tabular}
\end{center}
\caption{CUDA performance on Barnes Hut}
\end{table}
\end{center}
As the Barnes Hut is not independently computation-intensive, while requiring more space than the normal algorithm like Brute Force, GPU can not have an obvious speed-up on the program; instead, the cost greatly worsens the algorithm outcome.

\subsection{Non-recursive algorithms}
\noindent
Non-recursive algorithms aims to solve the performance problem in CUDA. The main idea is to revise the recursive function to the non-recursive function. To fit the logic of Barnes Hut, Breadth-First Search is utilized to develop the non-recursive algorithm. A queue structure provides a container for the while-loop iteration search in the quad-tree.\\
The problem still exists. First, the algorithm could not facilitate the performance in CUDA (with similar time consumption); second, the heap overflow problem occurs.
\subsection{Other trials}
\noindent
\textbf{(a) Linear quad-tree}\\
The linear quad-tree is a strategy, which utilizes a 1D array where each index represents the Morton code of one 2D space cell. This structure is easy to be implemented and understood, which suits the particle separation and device or process transfer. However, one great drawback is that this method requires a large space to generate redundant space for the linear structure. \\
After the experiment, the compiler error indicates that the heap space is used out. There are two main problems: first, the requested length number has already exceeded the limitation of the INT type; second, we can notice that based on 1000 particles, the depth is around 15~20, which requires about $4^{14}\sim4^{19}$ cells for the array, which may be far more than the device burden limitation.\\
\textbf{(b) OMP task in recursion}\\
Though the recursion structure is not suitable for parallelism, OpenMP provides an \textit{omp task} function, which may take convenience for recursion parallelism. However, in our experiment, the time consumption would slightly rise if combined with \textit{omp task}. After granting each task operating condition (based on the node depth, forbidding deep omp tasks), we found in a more shadow level, less the time. We assume that the for-loop has already taken all the threads thus it might be a performance lose with the task.

\section{Conclusion}
In this work, we target at paralleling N-body simulation (Brute Force and Barnes Hut) with MPI, OpenMP and CUDA.
In the Brute Force, MPI largely speeds up the computation, OpenMP provides slight enhancement in a fine-grain mode, while CUDA slows down the task at several magnitude of orders.
In the Barnes Hut, the crucial point is to parallelize force computing as well. In face of a complex and huge tree structure, we can try to share the tree structure in the memory space or build the same tree structure, using space to acquire a more efficient parallel algorithm. Meanwhile, the control of thread overhead also greatly affects the performance of the program. Besides, the CUDA model is not suitable for such a non-computation-intensive task, which largely degrades performance.
In both models, we observe the performance impairment when processes and threads are over some certain bounds.
Launching paralleled regions, communication among processes, copying memory to dedicated hardware all contributes to such degradation.

\def \articleBIBLIOGRAPHY{bibliography.bib}
\printbibliography

\end{document}


\documentclass{article}

\usepackage{graphicx}
\usepackage{hyperref}
\usepackage{multirow}

\begin{document}

\title{Data description}
%\author{Author's Name}

\maketitle

The following document contain a short description of the datasets used for the data competitions described in the paper.

\section{Melbourne properties price dataset}

The dataset `\textit{Melbourne properties price}' contains sales prices of different types of residential properties in Melbourne area along with other characteristics of the properties.

The variables and their descriptions prevented in table \ref{tab:Melb_price}:

\begin{table}[h]
	\begin{center}
		\renewcommand{\arraystretch}{1.1}
		\begin{tabular}{p{2cm} p{13cm}} \hline
			Variable   				& Description       \\\hline
			price 		& Price house sold for in AUS dollars\\
			suburb 		& Suburb: different areas of Melbourne\\
			land\_size  & Area of the land in square meters \\
			house\_size & Area of the house or the apartment in square meters\\
			nbeds 		& Number of bedrooms\\
			nbaths 		& Number of bathrooms/toilets\\
			ncars 		& How many cars fit in the carport/garage\\
			result 		& S - property sold; SP - property sold prior; PI - property passed in; VB - vendor bid; SA - sold after auction\\
			agent 		& Selling agent\\
			property\_type & h house; u appartment; t town-house or unit \\
			nvisits 	& Number of visitors before the selling\\
			rating 		& The (rounded) average rating by the visitors, 0 to 10, 0-``awful'' to 10-``unbelivebly lovely'' \\
			year 		& YYYY Year sold \\
			month 		& MM Month sold\\
			day 		& DD day sold\\
			id 			& Unique id for each house \\\hline

	\end{tabular}
	\caption{Melbourne properties price dataset: variable description.}
\end{center}
\label{tab:Melb_price}
\end{table}

The data were collected between Feb 2, 2013 and Dec 17, 2016. The training set ans the test set contained 75,366 and 25,122 observations, respectively.
The associated errors were calculated as absolute errors: $\frac{1}{N} \sum_{i=1}^{N} |y_i - \hat{y}_i|$.

\section{Spam dataset}
The `\textit{Spam}' data contains few thousands emails that were classified by their
owners as either spam or non-spam (3066 in the training set and additional 1533 in the testing set). Each email message also includes many other characteristics.

The variables and their descriptions prevented in table \ref{tab:spam_var}:

\begin{table}[h]
	\begin{center}
		\renewcommand{\arraystretch}{1.2}
		\begin{tabular}{p{4cm} l} \hline
		Variable   				& Description       \\\hline
		numRecipients		    & The number of people in the To: or Cc: lines\\
		Domain      		    & Domain name of the sender's email address\\
		replyToSameAddress      & TRUE/FALSE\\
		Weekday 				& Day of the week\\
		Hour 					& Hour of the day\\
		percentLowerCase 		& \% of lower case letters in the subject line\\
		numDigits 				& Number of digits in the sender's email address\\
		percentNonLetters 		& \% of non-letters in the sender's email address\\
		Size 					& Size of the email in kb\\
		numLinks				& Integer number of links included in the email\\
		localSender 			& Indicator - was the sender in the local domain as the\\
		receiver
		credit, porn, sucker,
		pharm, prescription,
		drugs, save, sex, dis-
		creet, free, sell, sale,
		asseenon, discount		& Binary variables computed by the presence of
		certain key words \\
		newsletter 				& Indicator - is this email basically a newsletter\\
		Spam 					& Indicator - classification of the email as spam or not by the receiver\\
		id 						& Unique id for each email \\\hline

		\end{tabular}
		\caption{Spam dataset: variable description.}
	\end{center}
	\label{tab:spam_var}
\end{table}

The mean consequential error (MCE) was selected as the method for prediction accuracy measurement. The mean/average of the "consequential error", where all errors are equally bad is given by:
$$ MCE = \frac{1}{N} \sum_{y_i \neq \hat{y}_i} 1$$

%\begin{figure}
%    \centering
%    \includegraphics[width=3.0in]{myfigure}
%    \caption{Simulation Results}
%    \label{simulationfigure}
%\end{figure}

\end{document}