\begin{center}
{{\Large \sc High-Performance Computing}}
\end{center}
\rule{\textwidth}{1pt}
\begin{description}
\item[Student name and id:] Marie Mørk \{s112770\} Anja Liljedahl Christensen\{s162876\}
 Andreas Vedel Jantzen \{s162858\}
 Anders Launer Bæk \{s160159\}
\item[Team name:] matmult\_MAAA
\item[Hand-in:] OpenMP - Poisson Problem
\end{description}
\rule{\textwidth}{1pt}

\section{Summary}
In this project, the Possion problem is solved numerically using two different methods; Jacobi and Gauss-Seidel. Initially, the methods are implemented in sequential code. This showed that the Gauss-Seidel method needed less iterations for the solution to converge, but at the same time that the Jacobi method was executed faster, due to a higher number of flops/s. The Jacobi method was then parallelized in different versions to investigate the scaling behaviour of the algorithms. Parallelizing not only the algorithm but also the initialization of the matrices resulted in the overall best performance. Different ways of implementing parallelization in the Gauss-Seidel method are explained but not implemented.

\section{Statement of the problem}
The goal of the project is to determine the steady state heat distribution in a square room with constant heat capacity, a source (radiator), and constant boundary conditions at the walls by solving the Poisson problem numerically. Two different methods are considered for solving the problem; the Jacobi method and the Gauss-Seidel method. Furthermore, the performance of the Jacobi method is to be improved by parallelizing the program using OpenMP. The scaling behaviour of the different versions of the parallelized Jacobi will be compared with that of the Mandelbrot problem. The Gauss-Seidel method cannot be parallelized in the same way as the Jacobi method, but different ways of implementing parallelization using this method will be explained and discussed.\\
% In short, the problem is divided into 4 sub-problems:
% \begin{itemize}
%     \item Sequential Jacobi method
%     \item Sequential Gauss-Seidel method
%     \item OpenMP Jacobi
%     \item OpenMP Gauss-Seidel
% \end{itemize}

\section{Hardware and software}
Specifications of the test environment are listed below:
\begin{itemize}
\item CPU information
\begin{itemize}
\item Vendor ID:             GenuineIntel
\item CPU family:            6
\item Model:                 79
\item Model name:            Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
\item CPU MHz:               1297.656
\item L1 (d / i) cache:             32K
\item L2 cache:              256K
\item L3 cache:              30720K
\end{itemize}
\item Compiler settings%\textcolor{red}{and Matrix implementations}
\begin{itemize}
\item The SunCC compiler have been applied with and without compiler optimization using the compiler flags \texttt{-fast -xopenmp} and \texttt{-xopenmp=noopt} respectively. See uploaded \texttt{makefile} for further information.
\end{itemize}
\item Highlights from the batch script. See \texttt{master\_sh.sh} for further information.
\begin{itemize}
\item \texttt{\#BSUB -R "rusage[mem=2048MB] span[hosts=1]"} and \texttt{\#BSUB -n 24} which dedicate the whole host.
\item \texttt{export OMP\_WAIT\_POLICY=active}
\end{itemize}
\end{itemize}

\section{Theory}
The Poisson problem is given by eq. \ref{eq:Poisson}. The goal is to estimate the heat distribution of a room with a radiator.
\begin{equation}
    -f(x,y)
   = \frac{\partial^2 u}{\partial x^2}
      + \frac{\partial^2 u}{\partial y^2}, \qquad (x,y) \in \Omega
\label{eq:Poisson}
\end{equation}
The heat distribution will be estimated using two methods: The Jacobi method given in eq. \ref{eq:jac} and the Gauss-Seidel method given in eq. \ref{eq:gauss}.

\begin{equation}
u_{i,j}^{k+1} = \frac{1}{4} (u_{i-1,j}^{k} +
u_{i+1,j}^{k} + u_{i,j-1}^{k} + u_{i,j+1}^{k} + \Delta^2 f(i,j))
\label{eq:jac}
\end{equation}

\begin{equation}
u_{i,j}^{k+1} = \frac{1}{4} (u_{i-1,j}^{k+1} +
u_{i+1,j}^{k} + u_{i,j-1}^{k+1} + u_{i,j+1}^{k} + \Delta^2 f(i,j))
\label{eq:gauss}
\end{equation}

The main difference between eq. \ref{eq:jac} and eq. \ref{eq:gauss} is that the Gauss-Seidel method uses two updated elements in the update of the center element. The Jacobi method only uses the values from the previous iteration, which makes this method simple to parallelize. \\

One of the most important measures of scalability is Amdahl's law.
It defines the ideal speed-up $S(P)$ for an increase in number of threads when running a given algorithm. Amdahl's law is defined as:
\begin{equation}
    S(P) = \frac{T(1)}{T(P)} = \frac{1}{f/P+1-f}.
    \label{eq:SPp}
\end{equation}

where $S(P)$ is the speed-up as a function of the number of threads $P$. $T(\bullet )$ is the execution time for the given number of threads and $f$ the parallel fraction of the program.

Equation \ref{eq:SPmax} determines the limit of the theoretical speed-up for a given $f$.
\begin{equation}
S_{max}(f) = \frac{1}{1-f}
\label{eq:SPmax}
\end{equation}

It is possible to achieve a parallel fraction close to $1$ in the Poisson problem, hence the expectation is that a parallel implementation should scale perfectly.
In reality though, there are numerous things to consider in order to ensure that the program scales properly.
Amdahl's law neglects the cost of communication, bookkeeping, synchronization, initializing, closing threads etc. The costs influence the properties of the scaling, recall the plot on slide 29 in \texttt{intro\_para2018\_ho.pdf}, found on Inside.

There are other approaches to increase the speed-up in parallel applications executed on multiple sockets. One approach is to initialize the matrices in parallel which "places" matrix elements efficiently according to the assigned threads.
The native \texttt{numactl} command can be used to control the policies for the processes and shared memory. This approach has not been implemented in the project but could be considered as future work. This could speed-up the computation even more. \texttt{NUMA} is a memory placement policy that specifies on what and how many threads the computation has to be processed on.

The number of floating point operations per second (flops/s) is used as a measure of performance when comparing the different implementations. The total number of flops needed to solve the Poisson problem is calculated as the product of the iteration in each of the nested loops times the number of floating point operations performed in the innermost loop.
This is multiplied by the number of iterations needed in order for the solution to converge (or the maximum number of iterations).
The total number of floating point operations is divided by the time needed to execute the code. The equation becomes:

\begin{equation}
flops/s =  \frac{I \cdot N^2 \cdot O}{T}
\label{eq:flops_time}
\end{equation}
where $I$ is the number of iterations, $N$ is the dimension of the grid (excluding the boarders), $O=10$ is the number of floating point operations in the innermost loop and $T$ the execution time.

\section{Algorithms}
This section introduces the main algorithms used to solve the Poisson problem. Additional algorithms can be found in the appendix.

\subsection{Sequential code - Jacobi method}
The first function is a sequential implementation of the Jacobi method, see algorithm \ref{alg:jac}. The first input is the problem size, $N$, excluding the borders of the grid. Other inputs are the grid spacing, $\Delta$, the threshold of the convergence, the maximum number of iterations as well as the matrices $f$, $u$ and $u_{old}$. In the implementation, the approximation of $u$ at iteration number $k$ is saved in the matrix $u$. As the calculation of $u$ at timestep $k$ is dependent on the approximation of $u$ at time step $k-1$, the pointers are swapped between $u$ and $u_{old}$ just before the calculations at each iteration commence.

\lstinputlisting[label={alg:jac},caption={Algo. \texttt{jac\_seq\_con()}.},style=CStyle, linerange={28-52-}]{code_2/func.c}

\subsection{Sequential code - Gauss-Seidel method}
The sequential implementation of the Gauss-Seidel method takes the same inputs as the Jacobi method except for $u_{old}$, which is not needed. The code is listed in algorithm \ref{alg:gauss}.

\lstinputlisting[label={alg:gauss},caption={Algo. \texttt{gauss\_con()}.},style=CStyle, linerange={228-246}]{code_2/func.c}

\subsection{OpenMP Jacobi}
A simple OpenMP version of the Jacobi function is implemented in algorithm \ref{alg:jac_pm}. In this version, the parallel region is set to be around the two inner loops, which loops over the index of the matrices. It is implemented such that the variables $f$, $u$, $u_{old}$, $N$, threads and $d$ are shared, while $i$ and $j$ are private. Each time the innermost loop is reaches, an addition is made to $d$. This results in dataracing, because all threads needs to make an addition to the same shared variable. \\
In the second version can be seen in the appendix as algorithm \ref{alg:app_jac_mp_v23}. The difference between the first and second version is that the parallel region has been moved such that it covers the whole while loop. This is an advantage, because the team of parallel threads are created once in the function instead of using fork and join for each iteration. In order to ensure that the threads does not work on different iterations, $k$, at the same time, $k$ is set to be firstprivate. This means that each of the threads gets the same value of k and only one thread updates it in the end of the loop. In the second version, the dataracing of version 1 has been avoided by adding the command \texttt{\#pragma omp for reduction (+ : d)} before the loops over the matrix indices. \\
In the third version, the function in itself is not changed. Instead, the main file is altered such that matrix initializations are parallelized.

\lstinputlisting[label={alg:jac_pm},caption={Algo. \texttt{jac\_mp()}.},style=CStyle, linerange={54-89}]{code_2/func.c}

The performance of the OpenMP Jacobi is to be compared with the scaling of the Mandelbrot problem. An implementation of the Mandelbrot problem has been given in the week exercise, see algorithm  \ref{alg:app_mandel} in the appendix. \\

\subsection{OpenMP Gauss-Seidel}

The Gauss-Seidel method (eq. \ref{eq:gauss}) is more difficult to parallelize than the Jacobi method.
It is because of the tightly bound nature (the method is coupled to itself by recursion) of the algorithm. Due to this, an OpenMP Gauss-Seidel is not implemented as a part of this project. Instead, different ways of implementing parallelization in Gauss-Seidel will be discussed in this section.\\

The article \citetitle{C2} by \citeauthor{C2} proposes several methods for solving fully coupled systems.
The \textbf{Row Parallel Algorithm} is a brute force parallelization of the Gauss-Seidel method. It takes the inner loop and calculates every row by itself and combines them in the end.
The \textbf{Column Parallel Algorithm} works in a similar way as the \textbf{Row Parallel Algorithm}, only that it is then the columns that are considered one at a time. Both of these methods reduces the number of syncronizations to be perfromed to $N$ in the case of an $N \times N$ matrix \cite{C2}.

The \textbf{Block-Column Parallel Algorithm} is based on the CUDA library. CUDA only works within thread blocks, hence the system matrix will be divided into $g$ submatrices (or subproblems) each with its own constraints.
The subproblems can be computed in parallel because the calculations are bounded by its own constraints.
For every subproblem, the matrix is divided into two blocks; one covering the diagonal of the full original matrix and another with the off-diagonal elements. Every subgroup computes the block with the diagonal elements first and then the off-diagonal. A barrier is introduced in between the calculation of diagonal and off-diagonal blocks.
The subgroups only need $2g$ synchronizations which reduces the dependence of global synchronizations that restricts scalability of the algorithm \cite{C2}.

The \textbf{Atomic Update Counter Parallel Algorithm} does, contrary to the above mentioned, get rid of global synchronization.
This method stores a shared integer which counts the number of diagonal blocks computed.
The counter and the revision of the Gauss-Seidel method is presented in the article. Specific hardware is also part of the \textbf{Atomic Update Counter Parallel Algorithm}, because it has to produce an automated-updated counter at the same time as allocating enough memory. This eliminates all synchronizations of the Gauss-Seidel methods \cite{C2}. \\

The following methods are derived from the original sequential Gauss-Seidel method from the article \citetitle{C3} by \citeauthor{C3}, \cite{C3}.
The \textbf{Computation Synchronization} method recalculates the values of $u_{i,j}$ at random.
The \textbf{Computation Deadlock} method restricts the access to the grid nodes and is hereby forced to parallelize which decreases the computation time.
The \textbf{Interdeterminacy Elimination} method makes the storing of results and current iterations in independent matrices.
The \textbf{Computation Load Balancing} method is as the name suggests a method for balancing the computation load among the used processors. The computation time of the program will hereby depend on the thread with the longest computation time \cite{C3}.

\newpage

\section{Results}
In the following sections, performance and scaling results from the implemented functions are presented. The functions are tested for different problem sizes and number of threads and optimal choices discussed.

\subsection{Sequential code - Jacobi and Gauss-Seidel method}
First off, the Poisson problem is solved using sequential code and two different methods; Jacobi and Gauss-Seidel. Both programs are tested using matrix sizes $N = 128$ and $N = 256$ and are set to run until they converge. The threshold of the convergence is set to $d = 10 ^{-10}$. In figure \ref{fig:test} the function estimates of $u(x,y)$ based on both methods are shown as heat maps. The estimates obtained are visually similar.

\begin{figure}[!th]
\centering
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.9\linewidth]{data_2/plots/gaus_con.pdf}
  \caption{Gauss-Seidel}
  \label{fig:}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=.9\linewidth]{data_2/plots/jac_con.pdf}
  \caption{Jacobi}
  \label{fig:}
\end{subfigure}
\caption{Estimate of the function $u(x,y)$. The problem size of the illustrated heats are $N=256$. See table \ref{tab:seq_par} for further evaluation details.}
\label{fig:test}
\end{figure}

In table \ref{tab:seq_par}, details of the iterations of the two functions for problem sizes $N = 128$ and $256$ are shown. As expected, the memory usage in the Gauss-Seidel method is 2/3 of the Jacobi. This is due to the fact that three matrices are used in the Jacobi method, namely \texttt{f, u} and \texttt{u\_old}, whereas only \texttt{f} and \texttt{u} are needed in Gauss-Seidel. The number of iterations shows how many iterations are needed in order for $u(x,y)$ to converge. For both problem sizes, the Jacobi method needs almost twice as many iterations to reach convergence. In spite of this, the execution time is much larger in the case of Gauss-Seidel because the iterations are coupled, why the compiler optimizer fails to reduce runtime. This also means that fewer iterations and \texttt{mflops} are performed pr. second, as seen in the table. All together this information shows that if parallelization and smart memory allocation is implemented, there is a possibility of a larger performance gain for Gauss-Seidel than for Jacobi. On the other hand, Jacobi is easier parallelized as a result of the uncoupled iterations.

\begin{table}[!th]
\centering
\begin{tabular}{ll|rrrrr}
&& Memory & Iterations & Duration & Iterations/s & mflops/s \\ \hline
\multirow{2}{*}{128}&\texttt{gauss\_con()}&262144&51225&4.79&10704&1754\\
&\texttt{jac\_seq\_con()}&393216&101831&1.76&57717&9456\\\hline
\multirow{2}{*}{256}&\texttt{gauss\_con()}&1048580&194176&73.38&2646&1734\\
&\texttt{jac\_seq\_con()}&1572860&390976&25.80&15153&9931\\
\end{tabular}
\caption{Comparison of the Jacobi method and Gauss-Seidel method.}
\label{tab:seq_par}
\end{table}

\subsection{OpenMP Jacobi}

The Jacobi method can be parallelized and run on multiple cores with shared memory using OpenMP. In the optimal setting the speed-up of increasing the number of cores is 1, i.e. using 2 cores instead of 1 should make the program run twice as fast. As discussed in the theory section OpenMP has several constructs which can be used to optimize the parallelization and ensure correctness. In the following sections, parallelized versions of the Jacobi method are implemented and continuously improved. For this purpose the number of iterations are fixed such that $k$ runs from 1 to \texttt{max\_iter} (with \texttt{max\_iter} = 5000) in order to best compare the efficiency of the different parallelized versions. Each version is run on 1, 4, 8, 16, and 24 threads for 4 different problem sizes $N \in \{256, 512, 1024, 2048\}$.

The first simple version will be used as a baseline.

\subsubsection{Version 1}

The first simple version has 2 parallel constructsm see algorithm \ref{alg:jac}: The first one contains a single-construct to make one of the threads extract the number of threads running the program. The second is nested within the while loop and contains a loop-construct such that the iterations from the outer for-loop (i.e. the rows) are distributed among the threads.

Figure \ref{fig:jac_comp_measure_v1} shows how increasing the number of threads for different problem sizes increases the number of flops per second. The gain is especially evident for $N = 512$ and $N = 1024$, while for $N = 2048$ there is no efficiency improvement when increasing the number of threads beyond $8$. This is explained by the fact that when $N$ is this size the problem does not fit within the 3 caches, hence
memory has to be fetched from the RAM which is significantly slower and hereby slows down the process.

In figure \ref{fig:scaling_mp_jac_v1} the speed-up for each problem size is compared to the theoretical speed-up. For problem sizes $256$, $512$, and $1024$ there is a clear correlation between problem size and speed-up, i.e. as the problem size increases so does the speed-up. This, however, is not the case for $N=2048$ which is as before most likely linked to the amount of needed memory. Since there are two sockets each running $12$ threads, they are forced to communicate when the number of threads exceeds 12 which could potentially slow down the process when the problem is large enough.

\begin{figure}[H]
\centering
\begin{tikzpicture}[scale = 0.85]
\begin{axis}[
width=17cm, height=8cm,     % size of the image
grid = major,
grid style={dashed, gray!30},
xmode=log,log basis x=10,
ymin=0,    % start the diagram at this
ymax=155000, % end   the diagram at this
axis background/.style={fill=white},
ylabel=Mflops/s,
xlabel=Memory/kbytes,
legend pos=north east]
\addplot[mark=*, green] table [x=mem, y=mflops, col sep=comma]{data_2/plots/program_jac_mp_ncore_1.txt};
\addlegendentry{1 thread}
\addplot[mark=*, orange] table [x=mem, y=mflops, col sep=comma]{data_2/plots/program_jac_mp_ncore_4.txt};
\addlegendentry{4 threads}
\addplot[mark=*, blue] table [x=mem, y=mflops, col sep=comma]{data_2/plots/program_jac_mp_ncore_8.txt};
\addlegendentry{8 threads}
\addplot[mark=*, red] table [x=mem, y=mflops, col sep=comma]{data_2/plots/program_jac_mp_ncore_16.txt};
\addlegendentry{16 threads}
\addplot[mark=*, black] table [x=mem, y=mflops, col sep=comma]{data_2/plots/program_jac_mp_ncore_24.txt};
\addlegendentry{24 threads}
\end{axis}
\end{tikzpicture}
\caption{Jacobi version 1.}
\label{fig:jac_comp_measure_v1}
\end{figure}

\begin{figure}[H]
\centering
\begin{tikzpicture}[scale = 0.85]
\begin{axis}[
width=9cm, height=9cm,
grid = major,
grid style={dashed, gray!30},
xmin=0,
ymin=0,
ymax=24,
xmax=24,% end   the diagram at this
/pgfplots/xtick={0,2,...,24},
/pgfplots/ytick={0,2,...,24},
axis background/.style={fill=white},
ylabel=S(P),
xlabel=P,
legend pos=north west]
\addplot[green, domain=0:25, no markers,samples=100] {x};
\addlegendentry{Theo.}
\addplot[mark=*, orange] table [x=ncore, y=SP, col sep=comma]{data_2/plots/program_jac_mp_SP_256.txt};
\addlegendentry{256}
\addplot[mark=*, blue] table [x=ncore, y=SP, col sep=comma]{data_2/plots/program_jac_mp_SP_512.txt};
\addlegendentry{512}
\addplot[mark=*, red] table [x=ncore, y=SP, col sep=comma]{data_2/plots/program_jac_mp_SP_1024.txt};
\addlegendentry{1024}
\addplot[mark=*, black] table [x=ncore, y=SP, col sep=comma]{data_2/plots/program_jac_mp_SP_2048.txt};
\addlegendentry{2048}
\end{axis}
\end{tikzpicture}
\caption{Scaling Jacobi version 1.}
\label{fig:scaling_mp_jac_v1}
\end{figure}

\subsubsection{Version 2}

For version 2 the second parallel construct is moved out to include the while-loop as well, seen in algorithm \ref{alg:app_jac_mp_v23}. In the previous version the master thread distributed the iterations among the threads each time the while-loop counter $k$ was incremented by 1. Hence, there was a fork and a join of the threads for each $k$ from 1 to \texttt{max\_iter} which resulted in a lot of waiting time in between the actual computations. In this version the iterations are only distributed among the threads once which should decrease the waiting time. Now the pointers have to be swapped in parallel but this should only be done by one of the threads to avoid data racing. Therefore it is also crucial that the threads use the same iteration $k$ by declaring the variable as a \texttt{firstprivate}.

Figure \ref{fig:jac_comp_measure_v2} shows how the performance is now increased for each combination of problem size and number of threads (apart from 1 thread). The improvement is the most evident when using 24 threads, while it seems there is no improvement when using 1 thread. This makes sense since there is nothing running in parallel when using only 1 thread.

In figure \ref{fig:scaling_mp_jac_v2} it is hard to detect any improvement for problem sizes 256, 512, and 1024, while the speed-up for $N = 2048$ is a little better beyond 12 threads.

\begin{figure}[H]
\centering
\begin{tikzpicture}[scale = 0.85]
\begin{axis}[
width=17cm, height=8cm,     % size of the image
grid = major,
grid style={dashed, gray!30},
xmode=log,log basis x=10,
ymin=0,    % start the diagram at this
ymax=155000, % end   the diagram at this
axis background/.style={fill=white},
ylabel=Mflops/s,
xlabel=Memory/kbytes,
legend pos=north east]
\addplot[mark=*, green] table [x=mem, y=mflops, col sep=comma]{data_2/plots/program_jac_mp_v2_ncore_1.txt};
\addlegendentry{1 thread}
\addplot[mark=*, orange] table [x=mem, y=mflops, col sep=comma]{data_2/plots/program_jac_mp_v2_ncore_4.txt};
\addlegendentry{4 threads}
\addplot[mark=*, blue] table [x=mem, y=mflops, col sep=comma]{data_2/plots/program_jac_mp_v2_ncore_8.txt};
\addlegendentry{8 threads}
\addplot[mark=*, red] table [x=mem, y=mflops, col sep=comma]{data_2/plots/program_jac_mp_v2_ncore_16.txt};
\addlegendentry{16 threads}
\addplot[mark=*, black] table [x=mem, y=mflops, col sep=comma]{data_2/plots/program_jac_mp_v2_ncore_24.txt};
\addlegendentry{24 threads}
\end{axis}
\end{tikzpicture}
\caption{Jacobi version 2.}
\label{fig:jac_comp_measure_v2}
\end{figure}

\begin{figure}[H]
\centering
\begin{tikzpicture}[scale = 0.85]
\begin{axis}[
width=9cm, height=9cm,
grid = major,
grid style={dashed, gray!30},
xmin=0,
ymin=0,
ymax=24,
xmax=24,% end   the diagram at this
/pgfplots/xtick={0,2,...,24},
/pgfplots/ytick={0,2,...,24},
axis background/.style={fill=white},
ylabel=S(P),
xlabel=P,
legend pos=north west]
\addplot[green, domain=0:25, no markers,samples=100] {x};
\addlegendentry{Theo.}
\addplot[mark=*, orange] table [x=ncore, y=SP, col sep=comma]{data_2/plots/program_jac_mp_v2_SP_256.txt};
\addlegendentry{256}
\addplot[mark=*, blue] table [x=ncore, y=SP, col sep=comma]{data_2/plots/program_jac_mp_v2_SP_512.txt};
\addlegendentry{512}
\addplot[mark=*, red] table [x=ncore, y=SP, col sep=comma]{data_2/plots/program_jac_mp_v2_SP_1024.txt};
\addlegendentry{1025}
\addplot[mark=*, black] table [x=ncore, y=SP, col sep=comma]{data_2/plots/program_jac_mp_v2_SP_2048.txt};
\addlegendentry{2048}
\end{axis}
\end{tikzpicture}
\caption{Scaling Jacobi version 2.}
\label{fig:scaling_mp_jac_v2}
\end{figure}

\subsubsection{Version 3}

For the final version the Jacobi method itself is the same as version 2, but this time the initialization of the matrices is also parallelized. Hence, the data is pre-distributed among the threads to make it more accessible when running the actual program.

In figure \ref{fig:jac_comp_measure_v3} it is once again obvious that the larger of the problem sizes are the ones to exploit the further parallelization the most. Especially $N = 2048$ utilizes the parallelized initialization of the matrices, which makes sense given the large memory consumption.

By considering figure \ref{fig:scaling_mp_jac_v3} it is clear that the problem sizes $256$ and $512$ experience the best improvement of speed-up, while $N=1024$ seems to be unchanged.

\begin{figure}[H]
\centering
\begin{tikzpicture}[scale = 0.85]
\begin{axis}[
width=17cm, height=8cm,     % size of the image
grid = major,
grid style={dashed, gray!30},
xmode=log,log basis x=10,
ymin=0,    % start the diagram at this
ymax=155000, % end   the diagram at this
axis background/.style={fill=white},
ylabel=Mflops/s,
xlabel=Memory/kbytes,
legend pos=north east]
\addplot[mark=*, green] table [x=mem, y=mflops, col sep=comma]{data_2/plots/program_jac_mp_v3_ncore_1.txt};
\addlegendentry{1 thread}
\addplot[mark=*, orange] table [x=mem, y=mflops, col sep=comma]{data_2/plots/program_jac_mp_v3_ncore_4.txt};
\addlegendentry{4 threads}
\addplot[mark=*, blue] table [x=mem, y=mflops, col sep=comma]{data_2/plots/program_jac_mp_v3_ncore_8.txt};
\addlegendentry{8 threads}
\addplot[mark=*, red] table [x=mem, y=mflops, col sep=comma]{data_2/plots/program_jac_mp_v3_ncore_16.txt};
\addlegendentry{16 threads}
\addplot[mark=*, black] table [x=mem, y=mflops, col sep=comma]{data_2/plots/program_jac_mp_v3_ncore_24.txt};
\addlegendentry{24 threads}
\end{axis}
\end{tikzpicture}
\caption{Jacobi version 3.}
\label{fig:jac_comp_measure_v3}
\end{figure}

\begin{figure}[H]
\centering
\begin{tikzpicture}[scale = 0.85]
\begin{axis}[
width=9cm, height=9cm,
grid = major,
grid style={dashed, gray!30},
xmin=0,
ymin=0,
ymax=24,
xmax=24,% end   the diagram at this
/pgfplots/xtick={0,2,...,24},
/pgfplots/ytick={0,2,...,24},
axis background/.style={fill=white},
ylabel=S(P),
xlabel=P,
legend pos=north west]
\addplot[green, domain=0:25, no markers,samples=100] {x};
\addlegendentry{Theo.}
\addplot[mark=*, orange] table [x=ncore, y=SP, col sep=comma]{data_2/plots/program_jac_mp_v3_SP_256.txt};
\addlegendentry{256}
\addplot[mark=*, blue] table [x=ncore, y=SP, col sep=comma]{data_2/plots/program_jac_mp_v3_SP_512.txt};
\addlegendentry{512}
\addplot[mark=*, red] table [x=ncore, y=SP, col sep=comma]{data_2/plots/program_jac_mp_v3_SP_1024.txt};
\addlegendentry{1024}
\addplot[mark=*, black] table [x=ncore, y=SP, col sep=comma]{data_2/plots/program_jac_mp_v3_SP_2048.txt};
\addlegendentry{2048}
\end{axis}
\end{tikzpicture}
\caption{Scaling Jacobi version 3.}
\label{fig:scaling_mp_jac_v3}
\end{figure}

%DET ER YDERST fedt at se 2048 hvor P er 1,2 og 4

\subsubsection{Comparison}

In figure \ref{fig:jac_comp_measure_v123} the efficiency of the 3 different OpenMP Jacobi methods are compared for 4 different problem sizes (the same as before) using 8 and 16 cores. Version 1 is outperformed by version 2 and 3 using both 8 and 16 threads. With 8 threads version 2 and 3 are practically indistinguishable, while using 16 threads the best performing version alternates between 2 and 3 depending on the problem size.

\begin{figure}[H]
\centering
\begin{tikzpicture}[scale = 1]
\begin{axis}[
width=17cm, height=8cm,     % size of the image
grid = major,
grid style={dashed, gray!30},
xmode=log,log basis x=10,
ymin=0,    % start the diagram at this
ymax=155000, % end   the diagram at this
axis background/.style={fill=white},
ylabel=Mflops/s,
xlabel=Memory/kbytes,
legend pos=north east]
\addplot[mark=*, purple] table [x=mem, y=mflops, col sep=comma]{data_2/plots/program_jac_mp_ncore_8.txt};
\addlegendentry{v1\_8}
\addplot[mark=*, orange] table [x=mem, y=mflops, col sep=comma]{data_2/plots/program_jac_mp_v2_ncore_8.txt};
\addlegendentry{v2\_8}
\addplot[mark=*, green] table [x=mem, y=mflops, col sep=comma]{data_2/plots/program_jac_mp_v3_ncore_8.txt};
\addlegendentry{v3\_8}

\addplot[mark=*, red] table [x=mem, y=mflops, col sep=comma]{data_2/plots/program_jac_mp_ncore_16.txt};
\addlegendentry{v1\_16}
\addplot[mark=*, black] table [x=mem, y=mflops, col sep=comma]{data_2/plots/program_jac_mp_v2_ncore_16.txt};
\addlegendentry{v2\_16}
\addplot[mark=*, blue] table [x=mem, y=mflops, col sep=comma]{data_2/plots/program_jac_mp_v3_ncore_16.txt};
\addlegendentry{v3\_16}

\end{axis}
\end{tikzpicture}
\caption{Jacobi version 1, 2 and 3 for 8 and 16 threads.}
\label{fig:jac_comp_measure_v123}
\end{figure}

\subsection{Mandelbrot}

The scaling behaviour of the Jacobi method and of the Mandelbrot algorithm is given in figure \ref{fig:scaling_comp_opt} and in figure \ref{fig:scaling_non_comp_opt} with and without compiler optimization flags repectively.

The four methods have been evaluated for $N=1024$ and with $ max_{iter} = 5000$.

\begin{figure}[H]
\begin{subfigure}{.5\textwidth}
\centering
\begin{tikzpicture}[scale = 1]
\begin{axis}[
width=8cm, height=8cm,
grid = major,
grid style={dashed, gray!30},
xmin=0,
ymin=0,
ymax=24,
xmax=24,% end   the diagram at this
/pgfplots/xtick={0,2,...,24},
/pgfplots/ytick={0,2,...,24},
axis background/.style={fill=white},
ylabel=S(P),
xlabel=P,
legend pos=north west]
\addplot[green, domain=0:25, no markers,samples=100] {x};
\addlegendentry{Theo.}
\addplot[mark=*, black] table [x=ncore, y=SP, col sep=comma]{data_2/plots/program_mandel.txt};
\addlegendentry{Mandel}
\addplot[mark=*, red] table [x=ncore, y=SP, col sep=comma]{data_2/plots/program_jac_mp_SP.txt};
\addlegendentry{Jacobi\_v1}
\addplot[mark=*, blue] table [x=ncore, y=SP, col sep=comma]{data_2/plots/program_jac_mp_v2_SP.txt};
\addlegendentry{Jacobi\_v2}
\addplot[mark=*, orange] table [x=ncore, y=SP, col sep=comma]{data_2/plots/program_jac_mp_v3_SP.txt};
\addlegendentry{Jacobi\_v3}
\end{axis}
\end{tikzpicture}
\caption{Optimized}
\label{fig:scaling_comp_opt}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
\centering
\begin{tikzpicture}[scale = 1]
\begin{axis}[
width=8cm, height=8cm,
grid = major,
grid style={dashed, gray!30},
xmin=0,
ymin=0,
ymax=24,
xmax=24,% end   the diagram at this
/pgfplots/xtick={0,2,...,24},
/pgfplots/ytick={0,2,...,24},
axis background/.style={fill=white},
ylabel=S(P),
xlabel=P,
legend pos=north west]
\addplot[green, domain=0:25, no markers,samples=100] {x};
\addlegendentry{Theo.}
\addplot[mark=*, black] table [x=ncore, y=SP, col sep=comma]{data_2/plots/program_mandel_non.txt};
\addlegendentry{Mandel\_non}
\addplot[mark=*, red] table [x=ncore, y=SP, col sep=comma]{data_2/plots/program_jac_mp_SP_non.txt};
\addlegendentry{Jacobi\_v1\_non}
\addplot[mark=*, blue] table [x=ncore, y=SP, col sep=comma]{data_2/plots/program_jac_mp_v2_SP_non.txt};
\addlegendentry{Jacobi\_v2\_non}
\addplot[mark=*, orange] table [x=ncore, y=SP, col sep=comma]{data_2/plots/program_jac_mp_v3_SP_non.txt};
\addlegendentry{Jacobi\_v3\_non}
\end{axis}
\end{tikzpicture}
\caption{Non optimized}
\label{fig:scaling_non_comp_opt}
\end{subfigure}
\caption{The plots shows the scaling as function of number of threads P with following compiler flags: \texttt{-fast -xopenmp} and \texttt{-xopenmp=noopt} for \ref{fig:scaling_comp_opt} and \ref{fig:scaling_non_comp_opt} respectively.}
%\label{fig:test}
\end{figure}

There is a slight difference between the three versions of the Jacobi method when not using the mentioned compiler optimization flags, see figure \ref{fig:scaling_non_comp_opt}. The three versions follows each other for a given $P$ except for $P=24$.

It can be concluded that the current implementation of the Mandelbrot algorithm (algo. \ref{alg:app_mandel}) does not scale as the implementations of the Jacobi method, see table \ref{tab:avergae_f} where $f$ have been derived from eq. \ref{eq:SPp}.
An explanation for the scaling of the Mandelbrot is the unbalanced work load in the while loop for the assigned threads. A solution to this could be dynamic scheduling of the work balance.

Table \ref{tab:avergae_f} reports the average values of $f$ for the curves in figure \ref{fig:scaling_comp_opt} and figure \ref{fig:scaling_non_comp_opt}. $f$ have been derived from eq. \ref{eq:SPp}.

\begin{table}[H]
\centering
\begin{tabular}{l|cc}
Algo. & $f_{opt}$ & $f_{non}$  \\\hline
\texttt{manddel()} & 0.730 & 0.726\\
\texttt{jac\_mp()} & 0.976 & 0.972\\
\texttt{jac\_mp\_v2()} & 0.974 & 0.974\\
\texttt{jac\_mp\_v3()} & 0.976 & 0.980\\
\end{tabular}
\caption{Average value of $f$ for each algorithm. The values in the second column corresponds to figure \ref{fig:scaling_comp_opt} and the third column corresponds to figure \ref{fig:scaling_non_comp_opt}. }
\label{tab:avergae_f}
\end{table}
The max speed-up limit for \texttt{jac\_mp\_v3()} is $S_{max}(0.976)=\left\lfloor  41 \right\rfloor$. The max speed-up limit for \texttt{mandel()} is $S_{max}(0.730)=\left\lfloor  3\right\rfloor$, see eq. \ref{eq:SPmax}.

\section{Conclusion}
The sequential codes for the Jacobi and Gauss-Seidel methods showed that while the Gauss-Seidel method needs less iterations for the solution to converge, it is still much slower than Jacobi. This is because more floating point operations are performed pr. second in Jacobi. From the analysis of the performance of the sequential programs it was found that parallelization and smarter data allocation could enhance the performance of both methods, but that the possible gain might be higher for using Gauss-Seidel, due to convergence being reached after fewer iterations and the fact that this method uses less memory.

As previously discussed the parallelized third version of the Jacobi method clearly improved the efficiency (number of flops per second) compared to the first and second version. The improvement of the speed-up is not that evident for the problem sizes 256, 512, and 1024, but when the problem doesn't fit within the first three caches and has to be stored in the RAM, there is some gain in speed-up when optimizing the parallelization – both with respect to the Jacobi method itself and the initialization of the matrices.

For future work it could be beneficial to implement the \texttt{NUMA} control policy.


\documentclass[a4paper,11pt]{article}
\usepackage{graphicx}
\usepackage{enumerate}
\usepackage{amsmath}
\usepackage{mathtools}
\DeclarePairedDelimiter{\ceil}{\lceil}{\rceil}

\usepackage{listings}
\usepackage {tikz}
\usetikzlibrary {positioning}
%\usepackage {xcolor}
\definecolor {processblue}{cmyk}{0.96,0,0,0}

\begin{document}

\begin{flushright}

\vspace{1.1cm}

{\bf\Huge Problem Set 3}

\rule{0.25\linewidth}{0.5pt}

\vspace{0.5cm}
%Put Authors
Justin Ely
\linebreak
\newline
%Put Author's affiliations
\footnotesize{Foundations of Algorithms \\}
% Date here below
July 5, 2016 \\
612-240-0924
\end{flushright}

\noindent\rule{\linewidth}{1.0pt}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{1a)}
With weights of $[5, 9, 10, 2]$ and $k=10$, the number of
trucks used will be 4, where the minimum possible would be 3
given a more efficient algorithm.

\section*{1b)}
After playing with this for a while, and listening to office hours,
I still wasn't able to come up with an adequate solution.  The office
hours recording referenced variables that were seemingly undefined, and
some of the assumptions made didn't seem to fit in the problem definition.

Nevertheless, I tried a bunch of scenarioes and always found the best and
worst cases to be within a factor of 2 of each-other, so it does seem to be
the correct result.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{2a)}
The probability of any new node to link to a specific existing node
is $P_l = \frac{1}{k-1}$.  With a node $v_j$, where only nodes added
after j can possibly have linked to it, the expected number of linked
nodes is going to be the sum of the probabilites of each new node:
\begin{eqnarray}
N_l &=& \sum_{k=j+1}^{n} \frac{1}{k-1}\\
&=& \sum_{k=j}^{n-1} \frac{1}{k} \\
&=& \int_{0}^{n-1} \frac{1}{k} - \int_{0}^{j} \frac{1}{k} \\
&=& ln(n-1) - ln(j)\\
&=& ln(\frac{n-1}{j}) \\
&=& \theta(ln(\frac{n}{j}))
\end{eqnarray}

\section*{2b)}
The probability of each new node to not link to a specific existing
node is $P = 1-\frac{1}{k-1}$.  Thus, the expectation value is given by:

\begin{eqnarray}
E &=& \prod_{k=j+1}^{n} (1-\frac{1}{k-1}) \\
&=& \frac{j-1}{j} \frac{j}{j+1} \frac{j+1}{j+2} ... \frac{n-2}{n-1} \\
&=& \frac{j-1}{n-1}
\end{eqnarray}

Thus the total number of unlinked nodes is:

\begin{eqnarray}
N &=& \sum_{j=1}^{n} \frac{j-1}{n-1} \\
&=& \frac{1}{n} \sum_{j=1}^{n} j-1 \\
&=& \frac{1}{n-1} \frac{n(n-1)}{2} \\
&=& \frac{n}{2}
\end{eqnarray}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{3a)}
Pop operations run in $O(1)$ time, and in worst case MultiPopA, MultiPopB,
and Transfer will need to exhaust the full stack.  This gives worst case
running times of:

\begin{itemize}
  \item MultiPopA: $O(n)$
  \item MultiPopB: $O(m)$
  \item Transfer: $O(n)$
\end{itemize}

\section*{3b)}
Define $\phi(n, m) = n + m$ and $\hat{c_{i}} = \phi(D_{i+1}) - \phi(D_{i})$, thus:

\subsection*{PushA}
\begin{eqnarray}
\hat{c_{i}} &=& c_{i} + \phi(D_{i+1}) - \phi(D_{i}) \\
&=& 1 + (n+1) + m - n + m \\
&=& 1 + 1 \\
&=& 2
\end{eqnarray}

\subsection*{PushB}
\begin{eqnarray}
\hat{c_{i}} &=& c_{i} + \phi(D_{i+1}) - \phi(D_{i}) \\
&=& 1 + n + (m+1) - n + m \\
&=& 1 + 1 \\
&=& 2
\end{eqnarray}

\subsection*{MultiPopA}
\begin{eqnarray}
\hat{c_{i}} &=& c_{i} + \phi(D_{i+1}) - \phi(D_{i}) \\
&=& k + (n-k) + m - n + m \\
&=& k - k \\
&=& 0
\end{eqnarray}

\subsection*{MultiPopB}
\begin{eqnarray}
\hat{c_{i}} &=& c_{i} + \phi(D_{i+1}) - \phi(D_{i}) \\
&=& k + n + (m-k) - n + m \\
&=& k - k \\
&=& 0
\end{eqnarray}

\subsection*{Transfer}
\begin{eqnarray}
\hat{c_{i}} &=& c_{i} + \phi(D_{i+1}) - \phi(D_{i}) \\
&=& 2k + (n-k) + (m+k) - n + m \\
&=& 2k + 0 \\
&=& 2k
\end{eqnarray}

We see from the amortized cost of each operation that they each run in constant time: $O(1)$.

\end{document}


\section{Results}

\begin{table}
    \caption{Comparison of MAE and STDAE for original and modified pictures.}
    \label{surveytable}
\begin{tabular}{|p{0.2\linewidth}|p{0.1\linewidth}|p{0.1\linewidth}|p{0.1\linewidth}|p{0.1\linewidth}|}
    \hline
    \multicolumn{5}{|p{0.7\linewidth}|}{\centering Mean Average Error (MAE) and standard deviation of average error (STDAE) for each picture.} \\
    \hline
    \multirow{2}{0.2\linewidth}{Image name} & \multicolumn{2}{|p{0.2\linewidth}|}{Original picture} & \multicolumn{2}{|p{0.2\linewidth}|}{Our picture} \\\cline{2-5}
    & MAE & STDAE & MAE & STDAE \\
    \hline
    img0 & 6.685 & 1.039 & 5.218 & 1.166 \\
    img1 & 1.939 & 0.345 & 0.929 & 0.759 \\
    img2 & 5.491 & 0.663 & 5.397 & 0.493 \\
    img3 & 4.723 & 1.218 & 4.737 & 1.717 \\
    img4 & 5.115 & 0.858 & 3.628 & 0.723 \\
    img5 & 10.407 & 0.922 & 10.174 & 0.985 \\
    img6 & 10.647 & 1.219 & 8.937 & 1.693 \\
    img7 & 6.163 & 0.943 & 5.457 & 0.973 \\
    img8 & 2.746 & 0.659 & 1.906 & 0.904 \\
    img9 & 3.712 & 0.651 & 3.383 & 0.555 \\
    img10 & 7.029 & 0.170 & 5.676 & 0.692 \\
    img11 & 10.983 & 0.296 & 9.500 & 0.985 \\
    img12 & 4.966 & 0.417 & 3.972 & 0.985 \\
    img13 & 7.726 & 0.813 & 6.325 & 1.586 \\
    img14 & 3.738 & 0.603 & 3.000 & 1.008 \\
    \hline
\end{tabular}
\end{table}

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{figures/survey_combined.png}
    \caption{Combined histogram of mean absolute errors of objects estimation.}
    \Description{Distributions of original and modified images are comparable but modified images are slightly better recognized.}
    \label{survey:combined}
\end{figure}

We asked people to estimate the number of objects on randomly chosen 15 pictures. We didn't record the exact number of users, but observations over IP addresses of review submissions show that about 30 different persons participated in our review producing 1930 estimations over 30 different pictures. In \autoref{surveytable} we collected mean average error and standard deviation of error of users' estimations. For all pictures except img3, we noticed that error of predictions lowered, and users started to notice more objects on the picture. However, despite the stable decline, results don't change much in absolute values. We connect this with the dataset being rather clean in terms of additional objects and different backgrounds. Since the used dataset provides a simple background (as table, floor, or carpet), reviewers are not distracted by additional obstacles that would be less noticeable on processed images. Besides that, we think that due to surveyors being unaware of key changes in the picture and not being trained, collected results could be worse than for those who understand how highlighting would emphasize the objects.

On \autoref{survey:combined} you can see the histogram of absolute errors combined for all pictures. This histogram also shows a slight improvement in recognizing the objects on the pictures, with a mean absolute error being shifted to zero compared with original images. The more detailed histograms of each picture are available in \autoref{survey:separate} in papers appendices. We noticed that the histograms support our conclusions about the changes and show stable improvement for every image except img3.

The worse performance on img3 is closely connected with the used segmentation model. After investigating the reasons, we noticed that the segmentation model constantly highlights straight parallel lines as a possible book that leads to invalid estimations. We suggest that adjusting the model and its settings would provide better results due to the reduced number of false-positive segmented objects. In addition, we intentionally didn't finetune the model to the dataset in order to measure the performance in the wild, but in real life, such finetuning could be introduced for certain backgrounds, classes, or depending on person perception.


\section{Observations}

This experiment aims to obtain an optical spectrum of Earthshine and test for
the presence of biomarker features,\eg O$_2$, O$_3$, H$_2$O, and the presence of
the vegetation red edge through fits of several models' components to the
observed spectrum. The observations were performed on the {\bf Mt. Stony Brook
14-inch telescope}, with the {\bf DADOS optical spectrograph} \cite{dados}, and
the {\bf SBIG STL-402 CCD camera} \cite{CCD}, on the date of April 17th, from
1:30 am to 5:30 am. The sky was clean and the temperature was around
55$^{\circ}$ F. The overall cloud cover from the archival satellite imagery and
the Earth seen from the moon, can be seen in the figures \ref{sat} and
\ref{frommoon},  in the appendix.

\subsection{Setting up the Spectrometer}
A spectrometer  splits the incoming light
of Earthshine by wavelength, having previously been focused by the optical
section of the instrument. The intensity of light at each wavelength is
measured and recorded by the detector, and it is then  plotted  against light
intensity, which is analyzed to find a number of
features. In order to pick up the key features of oxygen, water and vegetation
red edge, we investigate the visible and near infrared sections of
the electromagnetic spectrum, with light of wavelengths between  500 to
800 nm.

Earthshine is brightest when most of the Earth as seen from the Moon is
illuminated, {\it i.e.}, when the Moon is only a thin crescent. However, if the
Moon is too close to the Sun, there are  difficulties  separating
the glow of Earthshine from the bright twilight sky \cite{woolf_etal02}.
We select  a  night of waxing crescent moon when  the angular separation from
the Sun to the moon (from the moonrise to the sunrise) is suitable for the
experiment. This means that   the moon is sufficiently high above the
horizon ($>20^{\circ}$), yet sufficiently close to the Sun
($<90^{\circ}$) so that the Earthshine signal is bright.  The observations
cease at sunrise, since when the Sun reaches about 5$^{\circ}$ below the
horizon, the sky will  be too bright  to measure Earthshine.

We set the optical wavelength range of the spectrograph to cover our chosen set
of Earthshine absorption by molecular O$_2$, O$_3$, H$_2$O. This was done before
mounting the spectrograph on the telescope with the help of the Neon light
source. We looked up the  wavelengths of the strongest Neon gas transitions
in the optical, adjusting the wavelength range of the spectrograph. We use the
DADOS spectrograph with long integration times. To maximize the spectral
resolution the narrowest width slit $ 25 \mu m$ is to be used. This gives a
spectral resolution of $\lambda / \Delta \lambda \sim 500$.

Setting  the telescope tracking rate to lunar ({\texttt Autostar II}
keypad), we obtain sequences of spectra of the {\bf bright} (moonshine) and {\bf dark} (earthshine) sides
of the facing Moon, each of them together with  the adjacent {\bf sky} (in the
adjacent slit). We also take
calibration exposures of the Neon light source and  darks with duration matched
to the duration of all of the science exposures.

\subsection{Estimative of Exposure Times}

The Earthshine  usually has low S/N (signal to background) ratio because the
observations are obtained with Moon low above the horizon, and consequently a
high airmass and low Earthshine fluxes with respect to the sky. Moreover, the
detector can be quickly saturated when recording the spectrum of the sunlit Moon
crescent. When it comes to the vegetation signal, the Earthshine data reduction
becomes even more difficult: past works \cite{seager_etal05} have
shown that it  is only a few percent (less than $5\%$) above
the continuum. Two reasons for this are pointed in the literature: (i) the variable amplitude, induced by a variable cloud cover and the Earth phase, (ii) the strong astmospheric bands which need to be removed to access the surface reflectance.

Assuming that the signal is limited to photons, the signal to background ratio
for each of the sciences can be written as
\begin{equation}
 S/N = \sqrt{N_{\gamma}t},
\end{equation}
where $t$ is the time of integration ans $N_{\gamma}$ the number of photon
counts.

First, we make an estimate of the exposure time for the dark and
the bright side of the waxing crescent moon, supposing that their S/N are
similar. Considering that the {\it full moon} has magnitude
$m_{full} = -12.7$ and the {\it new moon} has magnitude $m_{new}= -2.5$, the
ratio of {\it photon fluxes} for the dark and bright side of the moon \cite{CCD}
is
\begin{eqnarray}
 R_{d/b} &\sim&\frac{F_{dark}}{F_{bright}}, \nonumber \\
&\sim& 0.1 \times 10^{(m_{full} - m_{new}) / 2.5}, \nonumber \\
&\sim& 10^{3}. \nonumber
\end{eqnarray}

We can then derive the exposures times for each side,
\begin{eqnarray}
 t_{bright} \times N_{\gamma \, bright} &\sim& t_{dark} \times N_{\gamma \,
dark} \times R^{-1}_{d/b}.
\nonumber
\end{eqnarray}

The last result means that if we keep the number of counts constant for the dark
and bright sides, we should aim for calibration dark exposure times of
around 1000
larger than those for bright, \eg we should have at least 90 minutes of net
observation of the dark side for a 5 seconds exposure of the bright side.
However, due the observational constrains, we ended up collecting only 1/4 of
this value, as it is shown in the Table \ref{looo} of the
appendix.

\subsection{Steps of Data Acquisition}

The data acquisition was  performed by the following steps, based on the
exposure times in the Table \ref{looo} in the appendix:
\begin{enumerate}
\item We record the spectrum of the Neon lamp on all the slits.

 \item We record a non-saturated spectrum of a {\it  bright stellar point
source} (Altair) at {\it two distinct positions on the slit}. We trace their
spectrum, which gives the direction along which we extract the spectrum of  the
Earthshine.

\item We track the {\it dark side of the Moon},  positioning the {\it dark limb}
in one slit and the adjacent on the other and recording the spectra.

\item We measure  the {\it illuminated limb of the moon } in the same way,
with a much smaller time to avoid saturation.

\item We repeat the above three procedures while the Sun is still not close
enough to the Moon.
\item   We obtain sets of dark frames for all the  exposure times of
our sciences.
\end{enumerate}


% This is a sample document using the University of Minnesota, Morris, Computer Science
% Senior Seminar modification of the ACM sig-alternate style. Much of this content is taken
% directly from the ACM sample document illustrating the use of the sig-alternate class. Certain
% parts that we never use have been removed to simplify the example, and a few additional
% components have been added.

% See https://github.com/UMM-CSci/Senior_seminar_templates for more info and to make
% suggestions and corrections.

\documentclass{sig-alternate}
\usepackage{color}
\usepackage[colorinlistoftodos]{todonotes}
\usepackage{comment}
\usepackage{amsmath}
\usepackage{hyperref}
\usepackage[linesnumbered,ruled,vlined]{algorithm2e}
\hypersetup{
    colorlinks=true,
    linkcolor=black,
    filecolor=black,
    urlcolor=cyan,
    citecolor=black
}

%%%%% Uncomment the following line and comment out the previous one
%%%%% to remove all comments
%%%%% NOTE: comments still occupy a line even if invisible;
%%%%% Don't write them as a separate paragraph
%\newcommand{\mycomment}[1]{}

\begin{document}

% --- Author Metadata here ---
%%% REMEMBER TO CHANGE THE SEMESTER AND YEAR AS NEEDED
\conferenceinfo{UMM CSci Senior Seminar Conference, April 2019}{Morris, MN}

\title{Wing Design Using SAIL}

\numberofauthors{1}

\author{
% The command \alignauthor (no curly braces needed) should
% precede each author name, affiliation/snail-mail address and
% e-mail address. Additionally, tag each line of
% affiliation/address with \affaddr, and tag the
% e-mail address with \email.
\alignauthor
Leonid Scott\\
	\affaddr{Division of Science and Mathematics}\\
	\affaddr{University of Minnesota, Morris}\\
	\affaddr{Morris, Minnesota, USA 56267}\\
	\email{scot0530@morris.umn.edu}
}

\maketitle
\begin{abstract}
In engineering spaces where modeling is difficult, engineers seek a variety of well performing solutions in order to concentrate resources on promising areas of the problem space.
We call this process \textit{illumination}.
Gaire et al have designed an algorithm specifically for illumination of problem spaces where the underlying model is computationally expensive.
This algorithm, Surrogate Assisted Illumination (SAIL) uses an \textit{evolutionary algorithm} called MAP-Elites to do illumination.
However, SAIL introduces a Gaussian process to simulate the computationally expensive model, and bayesian optimization for quality control of the Gaussian process.
SAIL has demonstrated potential for finding a variety of well performing solutions when applied to the design of aerodynamic hulls for \textit{velomobiles}.
We will walk through each of the components of the SAIL algorithm, and the results of the velomobile experiment.

\begin{comment}
This paper takes a deep dive into Gaier et al's paper, \textit{Data-Efficient Design Exploration through Surrogate-Assisted Illumination}.
While many evolutionary algorithms attempt to find the optimal solution to a problem space, Gaier et al develops an evolutionary algorithm that \textit{illuminates} the problem space.
This concept is particularly attractive in aerospace engineering as it allows engineers to properly survey the possible landscape of wing designs before investing into one particular concept heavily.

Gaier et al build off of the work of a previous algorithm called \textit{MAP-Elites}.
MAP-Elites is designed to provide a number of high performing solutions that represent diverse regions of performance from the problem space.
Experiments in Monet et al show MAP-Elites to be effective in creating this diverse set of solutions.
However, MAP-Elites has been shown to be extremely computationally expensive.
Moreover, MAP-Elites relies on a static model that does not change during an evolutionary run.

Gaier et al set out to improve MAP-Elites in the context of problem spaces where:
 \begin{enumerate}
   \item Running a high fidelity model is too computationally expensive to run per individual created.
   \item Running each generation comes at a significant computational cost and accuracy cost.
 \end{enumerate}
Gaier et al's algorithm, SAIL (Surrogate Assisted Illumination), uses a surrogate model to approximate the fitness of a solution instead of high fidelity model.
In its current implementation, SAIL uses a Gaussian process as a surrogate model. In order to guess where to select new individuals, SAIL uses Bayesian Optimization (BO).
BO allows SAIL to reduce the number of generations it needs to acquire a strong, diverse set of solutions that represent a problem space well.

This paper conducts two different experiments to test SAIL's viability in the context of aerospace engineering.
The first test attempts to design a set of two dimensional airfoils. A second test was added to design the three dimensional shape of a velomobile.
This second test was deemed important as the model to test it was much more complicated than its two dimensional counterpart.
This difficult model would stress SAIL's Surrogate Assisted model to a greater extent.
\end{comment}
\end{abstract}

\keywords{Evolutionary Computation, MAP-Elites, Gausian Processes, Bayesian Optimization}

\section{Introduction}
\label{sec:introduction}

Fluid dynamics stands as one of the most difficult problem spaces to model and thus design in.
The equations that govern fluid flow, known as the \textit{Navier-Stokes equations}, are a set of of partial differential equations with no known solution.
As a result, aerospace engineers must use extraordinary computational power to approximate Navier-Stokes results.
High fidelity models of aerodynamic devices can take hours to simulate and still impart noticeable error.
Given the difficulty in modeling fluid flow, designing wing shapes is an extraordinary challenge.

Optimization tools aid in this challenge by helping engineers at the end of the design cycle by refining a design to what is known as a \textit{local optima}.
In difficult problem spaces such as fluid dynamics, there is no guarantee that an optimizer has found the best possible solution across the entire problem space.
However, the optimizer can be very confident that it has found the best solution in one small region.
We call the optimal solution across the entire problem space a \textit{global optima}, and an optimal solution in a subsection of the problem space a \textit{local optima}.

In 2014, Autodesk, a producer of modelling and optimizing software found that their software was being used in a peculiar way by engineers.
Instead of using optimizing tools at the end of the design process to refine designs, engineers used them at the beginning to explore the space of possible designs\cite{Bradner:2014:PTD:2664323.2664349}.
Instead of using these tools to find, with great accuracy, one local optima, they were used to find several local optima throughout the problem space.
By finding several local optima, designers could see what tradeoffs are inherent in the problem space, and hone in on regions of interest.
This process is known as \textit{illumination}.

Over the course of several years, a research group consisting of Adam Gaier, Alexander Asteroth, and Jean-Baptiste Mouret have developed a purpose built algorithm to illuminate problem spaces, particularly when models are computationally expensive.
This algorithm, known as \textit{Surrogate-Assisted Illumination} (SAIL) uses a thus far reliable method for exploring problem spaces known as evolutionary computation, albeit in a very specialized form.
The goal of SAIL is to produce a series of high performing solutions, across the problem space, to challenging engineering problems.

\section{Evolutionary Algorithms}
\label{sec:evolutionaryAlgorithms}

SAIL uses a type of \emph{evolutionary algorithm} (EA) to explore and exploit the problem space.
Evolutionary algorithms are stochastic algorithms (algorithms including randomness) inspired by biological evolution.
The premise of an evolutionary algorithm is that a population of potential solutions is tested against a model of the problem and assigned a \textit{fitness score}.
The individuals with the best fitness scores move onto the next generation and produce ``offspring''.
Over many generations the performance against the model will improve until a target performance is obtained, or the algorithm reaches a fixed number of generations.

\begin{figure*}[th]
\centering
\psfig{file=CycleOfLife.PNG,width=\textwidth}
\caption{Process of moving from an individual genome to a fitness score.}
\label{fig:genome-to-fitness}
\end{figure*}

\section{SAIL Architecture}
\label{sec:SAILBuildUp}
SAIL is made up of three major components:
MAP-Elites is the evolutionary algorithm that illuminates problem spaces.
Gaussian processes approximate the computationally expensive model.
Finally, bayesian optimization enforces quality control of the Gaussian process.
This section will explain each of these components in depth before constructing Surrogate Assisted Illumination.

\subsection{MAP-Elites}
\label{sec:MAP-Elites}

MAP-Elites is an evolutionary algorithm developed in April 2015 by Jean-Baptiste Mouret and Jeff Clune~\cite{Mouret:2015}.
The purpose of MAP-Elites is to \textit{illuminate} the problem space.
That is, to produce a series of high performing solutions that represent different trade offs and insights into the problem space.
Before moving into the details of MAP-Elites, it is important to understand the terminology surrounding individuals in MAP-Elites (See Figure \ref{fig:genome-to-fitness}).

\subsubsection{Genotypes}
\label{sec:genotypes}

An important aspect in the application of any evolutionary algorithm is the way that individuals are represented.
In engineering contexts, there is a need to represent a physical object in all of its complexities in a compact and understandable form.
In evolutionary algorithms, this representation is referred to as an \textit{encoding} or a \textit{genotype} interchangeably.
For example, a common way to represent a two dimensional \textit{foil} (wing shape) is with a format called the NACA 4 Digit Series \cite{wiki:NACAairfoil}.
NACA 4 digit foils contain three numbers refering to important geometric features of the foil shape, but also directly relate to foil performance.

\begin{figure}[htb]
\centering
\psfig{file=NACA4-Desc.PNG,width = 3in}
\caption{Description of NACA 4 Foil.}
\label{fig:NACA4}
\end{figure}

Figure \ref{fig:NACA4} describes these three values geometrically:
$M$ refers to how ``arched'' the foil is.
$P$ describes where the most ``arched'' part of the foil exists.
Finally, $T$, refers to the thickness of the foil at its thickest point.
Figure \ref{fig:genome-to-fitness} shows a \textit{solutions space}, the space of all genomes.
In this example, we hold thickness constant so the solution space is in two dimensions.
With this compact genotype for a 2D foil, it is possible to represent a large range of complex shapes with only three numbers.

\subsubsection{Phenotypes}
\label{sec:phenotypes}

Once we have a representation of an individual, we can start deriving its \textit{behaviors}, or \textit{phenotypes}.
In this example, those behaviors are the \emph{lift} and \emph{drag} of the foil in a certain fluid condition as well as \emph{cross-sectional area} (the space between the top and bottom lines in Figure \ref{fig:NACA4}).
In Figure \ref{fig:genome-to-fitness}, the \textit{feature space}, the space of all possible phenotypes, will exist in three dimensions: lift, drag, and cross-sectional area.
The function that takes in an individual and returns its phenotype is known as the \textit{behavior function}, $b(x)$.

\subsubsection{Fitness}
\label{sec:fitness}

MAP-Elites requires some sort of specific score so that it can definitively tell that one foil is ``better'' than another.
The \textit{fitness function} $f(x)$, takes an individual and its behaviors and returns a score quantifying how well it accomplishes our specified goals.
When developing a foil shape, we most certainly care about lift and drag, but for weight and structural reasons, we might also care about the foil's cross-sectional area.
A fitness function that encompasses these behaviors into a single score might look like this:
$$f(x) = a*\textit{Lift}(x) - b *\textit{Drag}(x) - c*\textit{Area}(x)$$
where $a$, $b$, and $c$ are constants defined by the engineer based on which factors are more important than others.
Heree, $f(x)$ is setup such that a \textit{higher} fitness score means a foil is better at achieving our goals, but it doesn't have to be that way.
Fitness functions can be setup such that a good score is a: small score, a small absolute value, etc...

\subsubsection{MAP-Elites}

MAP-Elites works by discretizing the feature space into a set of bins.
Each bin in the grid represents tradeoffs between features.
For example, the feature space in Figure \ref{fig:FeatureSpace} is split into three sections per dimension, with three dimensions, this results in 27 bins in the feature space.
The highlighted box indicates individuals with high lift, medium drag, and medium cross sectional area.

Each generation in MAP-Elites begins by creating an individual from the search space and computing its behaviors using the behavior function.
The results of the behavior function will place it in a bin of the feature space.
If there is already an individual in that bin, the fitness function will be computed on both individuals, and the individual with the superior fitness will end that generation occupying the bin.
If there is no individual in that bin, the new individual simply occupies it.

In the initial generation (generation 0), a randomly generated population of individuals is created from the search space.
Those individuals then compete for spots in the feature space.
The individuals who occupy a bin at the end of a generation are known as \textit{elites}.
During each subsequent generation, a new individual is randomly generated.
That new individual will compete with the elites from the previous generation for a bin in the feature space.

\begin{figure}[tb]
\centering
\psfig{file=DiscretizedSpace.PNG,width = 3in}
\caption{How MAP-Elites disrectizes a feature space. Here, each of three dimensions is split into three sections, yielding 27 ``bins''.}
\label{fig:FeatureSpace}
\end{figure}

By default, MAP-Elites creates new individuals by choosing a random elite and ``tweaking'' it.
This process is called \textit{mutation}.
Because small changes in the genome of an individual can result in large changes in behavior, it is common for the mutated individual to compete in a different bin than the parent.

The designer can decide how MAP-Elites terminates.
Termination can happen  after:
A fixed number of generations or computational cost is exceeded.
One or many elites have a fitness over a given threshold.
Alternatively, after a certain number of bins are filled.

The last point alludes to an important aspect of MAP-Elites: not all bins can always be filled.
In our example, physics does not allow a foil to have high lift, zero drag, and minimal cross-sectional area.
For this reason, the highest lift, lowest drag, lowest cross-sectional area bin might not be filled.
In addition to finding good solutions for a problem, MAP-Elites also offers insights on just where the physics of a problem caps performance of solutions.

\label{MAPElitesSub}

\subsection{Gaussian Process}
\label{gaussianProcess}
Like many evolutionary algorithms, MAP-Elites evaluates the model's fitness and behavior functions several times per generation.
Considering a single MAP-Elites run can involve hundreds of thousands, or even millions of generations, the model needs to be computationally inexpensive.
However, in engineering contexts like fluid dynamics, computing lift and drag of a foil in high fidelity can take hours.
In these cases, the model is prohibitively computationally expensive for use in traditional evolutionary algorithms.

SAIL avoids computing the model directly most of the time by utilizing \textit{surrogate models} to approximate the model.
Surrogate models strategically execute the model in only limited points of the problem space.
They then use this information to extrapolate what other parts of the model should behave like.
Gaier et al have chosen to use \textit{Gaussian processes} (GPs) as a surrogate model because they require very few queries from the model in order to start extrapolating from the problem space.
In addition, GP's include information about how confident they are about the extrapolation of a certain point in the problem space.

%\begin{figure}[htb]
%\centering
%\psfig{file=GP-T3.PNG,width = 3in}
%\label{fig:GPT3}
%\caption{The plot of a Gaussian processes trying to model a %function $f(x)$ with three observed values (There is an observed %value on the far right).}
%\end{figure}

Space constraints limit us from diving into the statistics driving Gaussian processes.
Instead, we will walk through how they are used.\footnote{If you would like to learn the mechanics behind Gaussian processes, Dr. Nando de Freitas has a fantastic set of recorded lectures:
\url{https://youtu.be/4vGiHC35j9s}
}
Figure \ref{fig:BO5} shows a Gaussian process that is trying to model a target function shown in black.
The GP has four observed points to build its model from (shown as red dots in figure \ref{fig:BO5}).
With these observations, the GP can begin to extrapolate what a value of $f(x_{*})$ might be for a given $x_{*}$.
The GP will return a prediction, and a confidence in its extrapolation of $f(x_{*})$.

If we extrapolate a large number of points, to the degree in which they blend together to look like lines, we get Figure \ref{fig:BO5}.
The GP prediction for each of these extrapolations form the blue dashed line.
We call this dashed line, the \textit{mean prediction} of the Gaussian process.
The confidence at each extrapolation blends into two lines:
upper and lower confidence intervals.
These confidence bounds define the top and bottom of the GP prediction the figure.
The confidence bounds in Figure \ref{fig:BO5} are drawn such that we are 95\% certain that the true value of $f(x_{*})$ is within these bounds.
Moreover, for each extrapolated $x_{*}$, we are most confident that the true value of $f(x_{*})$ is on the mean prediction line.
From there, the likelihood of the true value of $f(x_{*})$ drops off with distance from the mean prediction line.

Areas that have low confidence, and a wide spread are said to have \textit{high variance}.
The variance contracts around our observed points, and expands as we get farther away from the observed points.
This leads to an important feature of Gaussian processes:
\textit{Where there is data, we are confident, where there is no data, we are less confident}.

\subsection{Bayesian Optimization}
\label{bayesianOptimization}

The fact that Gaussian processes are confident where there is data, and less confident where there is not, allows us to ask an important question:
\textit{If we were to add another point to our Gaussian process, where would we do it?}
If our objective is to maximize the function modeled by our Gaussian processes in Figure \ref{fig:BO5}, then our new point can help us do that in one of two ways:
First, we could try to pick a point where our prediction is maximized,near $x=5$; this is called \textit{exploiting} our problem space.
Second, there are two areas between $x=0$ to $x=2$ and $x=5$ to $x=7$ with high variance.
Selecting a point in this area would improve the overall accuracy of our model, and there might just be a maxima to our function in that region.
Selecting a point for this rational is called \textit{exploring} the problem space.

\begin{figure}[htb]
\centering
\psfig{file=t-4.png,width = 3in}
\label{fig:t4}
\end{figure}

\begin{figure}[htb]
\centering
\psfig{file=t-5.png,width = 3in}
\caption{A Gaussian processes modelling a function $f(x)$ with four points (top image), and five points on the bottom.
One of the observed values is on the far right.
For both GP's, the UCB acquisition function is computed in purple.
Taken from \cite{rand:BayesianOptimization}}
\label{fig:BO5}
\end{figure}

The degree to which we should explore or exploit the problem space is a difficult question.
\textit{Bayesian optimization} attempts to balance these objectives by providing a rule for computing the utility of observing another point.
There are many different ways of computing utility with bayesian optimization;
Gair et al decided to use the \textit{Upper Confidence Bound Optimizer}:

\[ UCB(x) = \mu(x) + k\sigma(x) \]

The utility of observing a new value from the model, the $UCB(x)$ at a point $x$, is the linear combination of the the mean prediction at a point, $\mu(x)$, and some constant $k$, times the variance, $\sigma(x)$, at some point.
Varying $k$ tunes the model to favor exploration vs exploitation in different quantities. We call the resulting function the \textit{acquisition function}.

Figure \ref{fig:BO5} shows how a bayesian optimization is applied to a Gaussian processes.
The UCB, in purple, is computed along the x axis.
The top image shows the GP with four observations.
At this stage, the model is not doing too well at approximating the true function.
Maximizing the acquisition function gives us a point with both a high approximated value and a high variance.

We select that point, re-compute the Gaussian processes and rebuild the acquisition function.
The updated GP and acquisition function is shown on the bottom image.
Again, UCB balances exploration vs exploitation and finds a point with a high predicted value, and high certainty.
Using bayesian optimization allows the GP to model the underlying function with very few points, and make significant progress with each observation.

\subsection{SAIL Algorithm}
\label{SAILAlgorithm}

Understanding Gaussian processes and bayesian optimization allows us to construct Surrogate Assisted Illumination (SAIL).
The pseudocode of SAIL is shown in Algorithm \ref{SAIL}.
SAIL consists of three phases:
1) Creation of the Gaussian processes,
2) The production of the acquisition map, and
3) The production of the prediction map.

%\begin{figure}[htb]
%\centering
%\psfig{file=SAILpcode.PNG,width = 4in}
%\caption{}
%\label{fig:SAILpcode}
%\end{figure}

\begin{algorithm}[h]
\DontPrintSemicolon
\SetAlgoLined
    \SetKw{thing}{1) Create Gaussian Process Model}
    \thing{}

    $\mathcal{X}\leftarrow Rand_{1:G}$

    %\tcp{Initialize with G random individuals}

    $\mathcal{P}\leftarrow PE(\mathcal{X})$ %\tcp{Precisely evaluate individuals}

    $\mathcal{G}\mathcal{P}\leftarrow Gaussian_process(\mathcal{X},\mathcal{P})$
    %\tcp{Train GP}

    \SetKw{secondStage}{2) Produce Acquisition Map}
    \secondStage{}

    \While{precise evaluation budget not exhausted}{
        $acquisition()\leftarrow UCB(\mathcal{G}\mathcal{P}(x))$

        %\tcp{Create acquisition function}

        $(\mathcal{X}_{acq}, \mathcal{P}_{acq}) = $ MAP-Elites $(acquisition(), \mathcal{X})$

        %\tcp{Illuminate acquisition function}

        $x\leftarrow \mathcal{X}_{acq}(Rand)$

        %\tcp{Select individuals from acquisition map}

        $\mathcal{X}\leftarrow \mathcal{X} \cup x, \mathcal{P}\leftarrow\mathcal{P} \cup PE(x)$

        $\mathcal{GP}\leftarrow Gaussian_process(\mathcal{X},\mathcal{P})$
    }

    \SetKw{thirdStage}{3) Produce Prediction Map}
    \thirdStage{}

    $prediction()\leftarrow mean(\mathcal{GP}(x))$

    $(\mathcal{X}_{pred},\mathcal{P}_{pred})=$ MAP-Elites $(prediction(),\mathcal{X})$

\caption{Surrogate Assisted Illumination (SAIL)\label{SAIL}}
\end{algorithm}

The first stage of the algorithm consists of sampling the problem space and building a Gaussian processes that will model fitness.
Recalling the foil terminology used in section 3.1, SAIL would create random individuals with \textit{M} and \textit{P} values, compute the precise model for each of them, and use those results to build a Gaussian process.

The second stage of SAIL trains the Gaussian processes by producing an \textit{acquisition map}.
Here, an acquisition function is computed from the Gaussian process using the \textit{Upper Confidence Bound} (UCB).
It is important to note that even though UCB is a linear combination of components, these components are in no way simple shapes.
For this reason, the acquisition function will not have an intuitive shape either.
Moreover, the acquisition function reflects the dimensionality of the feature space.
In a high dimensional feature space, the aquisition function will be a complicated high dimensional function.
Considering this, finding optima in the acquisition function can become a challenge.
MAP-Elites is perfectly suited to find not just the global optima of the acquisition function, but illuminate the acquisition function across the entire feature space.
The resulting set of high utility individuals is called the \textit{acquisition map}.

SAIL takes a random set of individuals from the acquisition map, and runs the computationally expensive model on them.
Now there is a larger set of observed values that can be used to retrain the Gaussian processes.
The loop will enter its next iteration, repeating the process of creating the acquisition function, illuminating the acquisition function to create an acquisition map, selecting individuals from the acquisition map for precise evaluation, and rebuilding the Gaussian process with the new observations.
This processes will continue until a fixed computational budget is reached.

Through extensive iteration, the Gaussian process solidifies into a robust model that accurately describes the underlying function.
The goal of SAIL is to illuminate the underlying function.
This task is fulfilled in the third stage of SAIL by illuminating the mean prediction of the Gaussian processes using MAP-Elites.
The result of this illumination is a set of well performing solutions describing the various optima of the problem space.
We call this resulting set of solutions the prediction map.

\section{3D Foil Experiment: Velomobile Experiment}
\label{3DFoilExperiment}

Gaier et al included two experiments in their paper.
Due to space constraints, we will only visit the more ambitious of the two:
The application of SAIL to design three dimensional shapes of aerodynamic hulls for \textit{velomobiles}.
Velomobiles, shown in Figure \ref{fig:Velomobile}, are fully human powered ``bicycles''.
The cyclists in a velomobile sit in a \textit{recumbent position}, like one would sit in a paddle boat, and are enclosed in an aerodynamic hull.
The hull allows engineers to carefully craft the airflow around the velomobile to reduce drag.
Beause of this reduced drag, velomobiles hold several world speed records for human powered devices.

\begin{figure}[htb]
\centering
\psfig{file=Velomobile.PNG,width = 3in}
\caption{Milan Velomobile. Taken from \cite{Gaier:2018}}
\label{fig:Velomobile}
\end{figure}

Figure \ref{fig:Velomobile} shows how velomobiles have unintuitive hull shapes.
The strange contours to these hulls provide an interesting problem space to illuminate.
Moreover, the fluid dynamics software that simulates how air moves around the hull shape has high computational cost, stressing the need for a \textit{data efficient} approach to illumination.
In short, the illumination of velomobile hulls is a good testing ground for SAIL.

Gaier et al encountered several difficulties with this experiment.
Most prevalently, the data efficient nature of this problem rules out use of MAP-Elites as a control.
With MAP-Elite's heavy reliance on the computationally expensive model, illuminating these shapes would take hundreds, if not thousands of hours.
Previous experiments in \cite{Gaier:2018} demonstrate SAIL's ability to create near optimal solutions in fluid dynamic contexts, so the focus shifted from testing SAIL's ability to compete against other illumination algorithms, to focusing on how different velomobile representations affect performance of the SAIL algorithm.
Two representations, or encodings, of velomobile shape, are used in this experiment: a parameterized, and deformed encoding.

\subsection{Parameterized Encodings}
The parameterized encoding uses a set of aerofoil shapes to create a three dimensional velomobile hull as seen in Figure~\ref{fig:ParameterizedEncoding}.

\begin{figure}[htb]
\centering
\psfig{file=ParameterizedEncoding.PNG,width = 3in}
\caption{Parameterized Encoding. Taken from \cite{Gaier:2018}}
\label{fig:ParameterizedEncoding}
\end{figure}

There are four foil shapes in the parameterized encoding.
The top foil shape, a symmetrical foil, defines the hull at its widest point from above.
The mid foil shape represents the foil at it's centerline.
The ridge foil represents the hull where the cyclist's knees protrude upward.
Finally, the rib foil affects how aggressively curved the surface of the hull will be.
All in all, this encoding will contain 16 parameters.
An advantage of an encoding like this is that each of the 16 parameters directly change the aerodynamic properties of their respective foil shapes.

\subsection{Deformation Encodings}
Where the parameterized encoding represent key aspects of the shape directly relating to performance, deformed encodings take another approach.
Deformation encodings ``decouple the complexity of a the design from the complexity of the representation''\cite{Gaier:2018}.
That is, these encodings allow SAIL to find key features of foil performance on its own.
Figure \ref{fig:Deformation} shows an example of a deformation encoding:

\begin{figure}[htb]
\centering
\psfig{file=Deformation.PNG,width = 3in}
\caption{Deformed Encoding. Taken from \cite{Gaier:2018}}
\label{fig:Deformation}
\end{figure}

Deformation encodings start with a base design, and surround it with a \textit{lattice}, or grid, of \textit{control points}.
These control points can be shifted in a particular axis.
As a control point is moved, the base design will \textit{deform} to adjust to the shifted control point.
Gaier et al allow  16 different points to be altered, each in only one axis.

\subsection{Experimental Setup}
The goal of this experiment is to see how two features, curvature and volume, impact the drag of a velomobile shape.
It is known that an increase in volume leads to an increase in drag, but a designer might want to explore how volume impacts drag to accommodate shifting requirements regarding volume (the pedaling mechanism requiring more space).
Moreover, designs with less curvature generally have less drag, but the thin carbon fiber might require additional (heavy) reinforcement to prevent flutter at high speed.

SAIL started with 200 velomobile shapes to train the GP. During each iteration of the illumination phase, 10 individuals were chosen from the acquisition map to improve the GP.
There were a total of 100 illumination iterations leading to $200 + 10*100=1200$ precise evaluations during the illumination phase.
The feature map was discretized into a 25 x 25 map, leading to 625 bins.

Fitness was computed as the drag force on the velomobile as it travels at 20 m/s (44.7 mph).
A low drag force will represent a better velomobile.

\subsection{Results: Design Performance}
Figure \ref{fig:DesignPerformance} shows the prediction maps for both the deformed and parameterized encodings as well as a comparison against the two.
In the left two images, a lighter colour indicates more drag, and a cooler colour indicates less.
We are aiming to minimize drag, so darker regions are preferred.

\begin{figure}[htb]
\centering
\psfig{file=DesignPerformance.png,width = 3in}
\caption{(Left) Prediction maps from deformed and perameterized encodings. (Right) Comparison of the two encodings. Taken from \cite{Gaier:2018}}
\label{fig:DesignPerformance}
\end{figure}

The deformation encoding has a set of empty bins in the high volume, high curvature section of its prediction map.
This is because the deformed encoding is too tightly constrained to create high volume, high curvature solutions.
It's clear that drag does increase with volume.
Encoding constraints aside, curvature does not seem to impact on drag.

The right side of Figure \ref{fig:DesignPerformance} shows the deformation encodings' successes against the parameterized encoding.
Lighter colour highlights regions where deformation has a better fitness than the parameterized encoding;
cooler colour indicates the converse.
There is only colour where there is a significant change in fitness between the two encodings.
Deformation does better in the high volume, low curvature region of the feature space, where as the parameterized encoding does better around the edges of the feature space.

\subsection{Results: Model Accuracy}
After each run, the individuals in the prediction map where precisely evaluated in order to record errors in the final GP model.
Figure \ref{fig:ModelAccuracy} shows that roughly 60\% of the drag calculations for individual's where within 5\% of the computationally expensive model.
The left side of Figure \ref{fig:ModelAccuracy} shows the deformed encoding has slightly lower error than the parameterized encoding.
However, it's unclear why.

\begin{figure}[htb]
\centering
\psfig{file=ModelAccuracy.PNG,width = 3in}
\caption{GP error for both encodings. From \cite{Gaier:2018}}
\label{fig:ModelAccuracy}
\end{figure}

\subsection{Results: Design Exploration}
The two encodings lead to very different looking solutions that occupy the same bins in the prediction map at the end of the SAIL run.
Figure \ref{fig:3DShapes} shows two sets of similarly performing solutions (one of each encoding) that occupy the same bin.
Across the prediction map, the parameterized encoding is taller than its deformation analog.

\begin{figure}[htb]
\centering
\psfig{file=3DShapes.PNG,width = 3in}
\caption{Two sets of similarly performing solutions that highlights the geometric tendencies of each encoding.
The left side includes two similarly performing low volume velombiles.
The right includes two similar high volume designs.
The red indicates how aerodynamic pressure is distributed over the velomobile shape.
Areas of greater intensity of red indicate higher aerodynamic pressure.
The numbers above the figures refer to drag force of the individual in \textit{newtons}.
Taken from \cite{Gaier:2018}}
\label{fig:3DShapes}
\end{figure}

Gaier et al suspect that parameterized encodings are taller than deformed encodings because they have more freedom to pinch the nose of the velomobile than deformed encodings can.
A pinched nose reduces frontal area, thereby reducing pressure on the nose of the velomobile.
While deformed designs lack this geometric freedom, they make up for it by smoothing out the shape along the length of the entire velomobile, especially where the ridges meet the hull.

These experiments show how various encodings can reach similarly performing solutions from different directions.
Moreover, they showcase SAIL's ability to create a variety of high performing solutions independent of encoding.

\section{Conclusions}
\label{sec:conclusions}
The velomobile experiment demonstrates several of SAIL's capabilities.
They show the potential of SAIL as a powerful algorithm for illuminating problem spaces in computationally difficult problem spaces.
Moreover, SAIL could be used for testing the limits of various encodings.
Before committing to a specific encoding, SAIL can show whether or not that encoding is capable of reaching far areas of the feature space.
Moreover, SAIL can give an engineer insights as to how each individual point of variation in the encoding contributes to performance.

A particular advantage of SAIL is that it returns, not only a set of high performing solutions, but also the Gaussian proccess model.
This model can be used to gain additional insights into the problem space

Gaier et al mention that a potential bottleneck for SAIL is the behavior function.
For example, computing curvature for each velomobile precisely would be too computationally expensive.
The authors used a simplified model to compute curvature that was significantly faster.
Gaier et al propose the expanded use of surrogate models in the behavior function in cases where the behavior function is too computationally expensive to run for every individual in a SAIL run.

\section*{Acknowledgments}
There are many people who deserve acknowledgment for the help they have provided me during this senior seminar. Thank you to Andrew Kroska, Islamzhan Saliyev, and Kirbie Dramdahl for their gracious time, and review of this paper.
Thank you to my advisor, Dr. Nic McPhee, for his continued mentorship and time throughout this semester and my college career.
Thank you to the Computer Science faculty at University of Minnesota Morris.
Finally, thank you to Alexa Barta for her continued support.
\label{sec:acknowledgments}

% The following two commands are all you need in the
% initial runs of your .tex file to
% produce the bibliography for the citations in your paper.
\bibliographystyle{abbrv}
% sample_paper.bib is the name of the BibTex file containing the
% bibliography entries. Note that you *don't* include the .bib ending here.
\bibliography{sample_paper}
% You must have a proper ".bib" file
%  and remember to run:
% latex bibtex latex latex
% to resolve all references

\end{document}