\section{Water Models}\label{sec:water_models}

% Navier stokes, Different level of water depth, Spatial, Spectral domain,
% eulerian, lagrangian frameworks, talk about their visual accuracy, SLProject,
% 0 a.d.

% Talk about the different depth of water

We start this section by presenting the physically accurate description of
fluids, \nameref{subsec:navier_stokes}, in \autoref{subsec:navier_stokes}. Water
models are separated into groups depending on the water depth: deep,
intermediate or shallow. This classification takes its root from oceanography
\autocite{darles2011survey}. We will present the deep water group in
\autoref{subsec:deep_water} and the shallow one in
\autoref{subsec:shallow_water} (we include the intermediate models directly into
the shallow group). For each of them we will present selected relevant
approaches from the literature. These two groups describe only the
\textit{shape} of the water. In \autoref{subsec:ocean_details} we discuss the
methods to produce realistic water surfaces. This includes caustics, foam, light
scattering and more. Finally in \autoref{subsec:candidate_apps} we talk about
two applications in need of a real-time water solution. Readers who desire to
get a broader picture about water rendering should read the survey from
\citeauthor{darles2011survey} \autocite{darles2011survey}.

\subsection{Navier-Stokes Equations}\label{subsec:navier_stokes}

% explain that they describe a fluid but they don't have a solution yet and it's
% one of the biggest open problems in math. Computational Fluid Dynamics CFD is
% the field of finding numerical approximations to the equations
% The process of solving problems in fluid dynamics numerically on a computer is
% called CFD

Navier-Stokes Equations (or NSE) describe the motion of a fluid in any dimension
$\mathbb{R}^n$. They are named after Sir George Gabriel Stokes\footnote{Best
known for \textit{Stokes's law}, the expression of the frictional force of a
fluid exerted onto a sphere.}, an Irish mathematician and physicist and Henri
Navier, a French engineer, mathematician and economist. Both worked on, and
enhanced the Euler equations. Navier proposed in 1820 to add a term to them for
the heat dissipation. In 1845 Stokes also suggested introducing a term for the
energy dissipation in form of heat \autocite{gallagher2010autour}. The equations
are shown below only for the curious reader as we will not further discuss them.
The Navier-Stokes equation of an incompressible fluid flow is:

\begin{equation}\label{eq:nse}
    \frac{\partial \textbf{u}}{\partial t} + \textbf{u}\cdot \nabla \textbf{u}
     = - \frac{\nabla P }{\rho} + \nu {\nabla}^2 \textbf{u}
\end{equation}
%
where $\nu$ is the kinematic viscosity, $\textbf{u}$ is the velocity of the
fluid parcel, $P$ is the pressure, $\rho$ is the fluid density and $t$ the
time \autocite{weisstein2018navier}.

Solving the Navier-Stokes equations in three dimensions remains an open problem
in mathematics and physics. It is one of the seven ``Millennium Problems''
defined by the Clay Mathematics Institute.

We felt the need to mention them here because all the water models found in the
literature are somehow linked to them. Computational Fluid Dynamics (CFD) is the
field of finding numerical approximations to those equations.

\subsection{Deep Water Models}\label{subsec:deep_water}
% What is the spatial domain?  present some papers:
% - Johanson
% What is the spectral domain?  present some models in the spectral and spatial
% domains from different papers that look great:
% - Tessendorf
% - GPU Gems 2 Effective Water Simulation from Physical Models
% - Using Vertex Texture Displacement for Realistic Water Rendering
Deep water models can further be classified into two groups: spatial and
spectral \autocite{darles2011survey}. To describe the water surface the former
uses a sum of sines and cosines and the latter an inverse fast Fourier transform
(IFFT)\footnote{The FFT approach allows to use data from a buoy or from aerial
water photographs.}. Unfortunately both have downsides: spatial models produce
water having a too rounded shape and spectral models are difficult to control
\autocite{darles2011survey}. In order to benefit from both, a hybrid solution
should be chosen \autocite{darles2011survey}. We present two models: one for
each group.

%\subsubsection{Projected Grid Concept}
\subsubsection{Gerstner Waves}\label{subsub:gerstner}
%\subsubsection{Vertex Texture Displacement}

Observing the ocean from the atmosphere, as seen in figure
\autoref{fig:barren_island}, a slightly repetitive pattern is noticeable. This
resembles strongly to a hevily disturbed combination of sines and cosines. The
combination of such curves, also called \textit{Gerstner
Waves}\footnote{Gerstner Waves where first described in
\autocite{gerstner1802theorie} and provide a solution to a certain type of
Euler equations.}, was proposed in \autocite{max1981vectorized} for their
ray-tracing procedural model. The height $y = h(x,z,t)$ for each point $(x,z)$
at time $t$ is

\begin{equation}\label{eq:gerstner_max}
    h(x, z, t) = -y_0 + \sum_{i = 1}^{N_w} A_i \cos(k_{i_x}x + k_{i_y}y -
    \omega_i t)
\end{equation}
%
where $N_w$ is the total number of waves, $A_i$ is the amplitude of wave $i$,
$(k_{i_x}, k_{i_y})$ the wave vector and $\omega_i$ the angular frequency
\autocite{max1981vectorized,darles2011survey}. \autoref{fig:max_raytraced}
depicts the sum of four such waves rendered in 1981.

\begin{figure}[hbt!]
    \centering
    \begin{subfigure}[hbt]{\textwidth}
        \centering
        \includegraphics[height=5cm]{barren_island_far.jpg}
        \subcaption{Far view of the Barren
        Island}\label{subfig:barren_island_far}
    \end{subfigure}\\%
    %
    \begin{subfigure}[hbt]{\textwidth}
        \centering
        \includegraphics[height=5cm]{barren_island_close.jpg}
        \subcaption{Close view of the Barren
        Island}\label{subfig:barren_island_close}
    \end{subfigure}
    \caption{The Barren Island captured by the Earth Observing-1 satellite in
        2007. The darker waves in \autoref{subfig:barren_island_far} are
        deep internal waves (source:
        \url{earthobservatory.nasa.gov}).}\label{fig:barren_island}
\end{figure}

\begin{figure}[hbt]
    \centering
    \includegraphics[height=5cm]{max1981raytracing.jpg}
    \caption{Ray-tracing of four waves using \autoref{eq:gerstner_max}. Source:
    \autocite{max1981vectorized}}\label{fig:max_raytraced}
\end{figure}

Another form of recommended Gerstner Waves, described in
\autocite[Chapter~1]{fernando2004gpu}, displaces also the $x$ and $z$
directions. $\textbf{P}$ is the position in space, and $\textbf{N}$ is its
corresponding normal. See \autoref{eq:gerstner_position} and
\autoref{eq:gerstner_normal}.

\begin{equation}\label{eq:gerstner_position}
    \textbf{P}(x,y,t)=\begin{bmatrix}x\\ y\\ z\end{bmatrix}=\begin{bmatrix} x +
    \sum_{i}(Q_i A_i\cdot\textbf{D}_i.x\cdot \cos(\omega_i \textbf{D}_i\cdot
    (x,y) + \varphi_i t))\\ y + \sum_{i}(Q_i A_i\cdot\textbf{D}_i.y\cdot
    \cos(\omega_i \textbf{D}_i\cdot (x,y) + \varphi_i t))\\ \sum_{i}(A_i\cdot
\sin(\omega_i \textbf{D}_i\cdot (x,y) + \varphi_i t))\\\end{bmatrix}
\end{equation}

\begin{equation}\label{eq:gerstner_normal}
    \textbf{N}(x,y,t)=\begin{bmatrix}x\\ y\\ z\end{bmatrix}=\begin{bmatrix}
    - \sum_{i}(\textbf{D}_i.x\cdot WA \cdot C(\cdot))\\
    - \sum_{i}(\textbf{D}_i.y\cdot WA \cdot C(\cdot))\\
    1-\sum_{i}(Q_i \cdot WA \cdot S(\cdot))\\\end{bmatrix}
\end{equation}
%
With the substitutions $WA = \omega_i A_i,\ S(\cdot) = \sin(\omega_i
\textbf{D}_i \cdot \textbf{P} + \varphi_i t),\ C(\cdot) = \cos(\omega_i
\textbf{D}_i \cdot \textbf{P} + \varphi_i t)$.

The authors derive the binormal and tangent vectors from the position. They can
be used with the normal vector from \autoref{eq:gerstner_normal} to compute the
transformation from the tangent space into the local space.

\begin{equation}\label{eq:gerstner_binormal}
    \textbf{B}(x,y,t)=\begin{bmatrix}x\\ y\\ z\end{bmatrix}=\begin{bmatrix}
    1 - \sum_{i}(Q_i \cdot \textbf{D}_i.x^2 \cdot WA \cdot S(\cdot))\\
    - \sum_{i}(Q_i \cdot \textbf{D}_i.x \cdot \textbf{D}_i.y\cdot WA \cdot
    S(\cdot))\\
    \sum_{i}(\textbf{D}_i.x \cdot WA \cdot C(\cdot))\\\end{bmatrix}
\end{equation}

\begin{equation}\label{eq:gerstner_tangent}
    \textbf{T}(x,y,t)=\begin{bmatrix}x\\ y\\ z\end{bmatrix}=\begin{bmatrix}
    - \sum_{i}(Q_i \cdot \textbf{D}_i.x \cdot \textbf{D}_i.y\cdot WA \cdot
    S(\cdot))\\
    1 - \sum_{i}(Q_i \cdot \textbf{D}_i.y^2 \cdot WA \cdot S(\cdot))\\
    \sum_{i}(\textbf{D}_i.y \cdot WA \cdot C(\cdot))\\\end{bmatrix}
\end{equation}

The Gerstner wave approach is easy to implement and understand but produces too
rounded shapes. On one hand computing only four waves produces a too repetitive
result. On the other hand adding more of them reduces the performance. To obtain
a reasonable result it should be combined with another model.

\subsubsection{Fast Fourier Transform (FFT)}\label{subsub:fft}

The fast Fourier transform approach was introduced in 1987 by
\citeauthor{mastin1987fourier} \autocite{mastin1987fourier}. The representation
of the wave height $h$ given a horizontal position $\textbf{x} = (x,z)$ and the
time $t$ is given by

\begin{equation}
    h(\textbf{x}, t) = \sum_{\textbf{k}}^{} \tilde{h}(\textbf{k},
    t)e^{i\textbf{k}\cdot\textbf{x}}
\end{equation}
%
where $\textbf{k}$ is the wave vector and $\tilde{h}(\textbf{k}, t)$ is the
amplitude and phase of the $\textbf{k}$th wave at time $t$. The wave vector
$\textbf{k}$ is a two dimensional vector equal to $(k_x, k_z) = (2\pi n/s, 2\pi
m / s)$. $n$ and $m$ are bounded by the resolution $r$ of the grid in the
following manner: $-r/2 \leq n,m \leq r/2$. $s$ is the dimension of the animated
area \autocite{jensen2001deep,darles2011survey}.

In order to compute $\tilde{h}$ its value at time $t=0$ is needed, which is
denoted by $\tilde{h}_0(\textbf{k})$. Then $\tilde{h}(\textbf{k}, t)$ can be
expressed as

\begin{equation}
    \tilde{h}(\textbf{k}, t) = \tilde{h}_0(\textbf{k}) e^{i\omega(k)t} +
    \tilde{h}_0^*(\textbf{k}) e^{i\omega(k)t}
\end{equation}
%
where $\tilde{h}_0^*(\textbf{k})$ is the complex conjugate s.t.
$\tilde{h}(\textbf{-k})=\tilde{h}^*(\textbf{k})$ and $\omega(k)=gk$ with $g$
being the gravitational constant.

\citeauthor{mastin1987fourier} take for $\tilde{h}_0$ the following:

\begin{equation}
    \tilde{h}_0(\textbf{k}) = \frac{\alpha g^2}{{(2\pi)}^4 f^5}
    e^{-\frac{5}{4}{(\frac{f_m}{f})}^4}, \quad f_m=\frac{0.13}{u_{10}}
\end{equation}
%
$f$ is the frequency in Hz, $f_m$ is the peak frequency of the wind speed
$u_{10}$, measured at ten meters above the ocean surface, $\alpha=0.0081$ is
Phillips' constant and $g$ is again the gravitational constant.

\citeauthor{tessendorf2001simulating} proposed another amplitude function
$\tilde{h}_0$,

\begin{equation}\label{eq:tessendorf_h0}
    \tilde{h}_0(\textbf{k}) = \frac{1}{\sqrt{2}}(\xi_r +
    i\xi_i)\sqrt{P_h(\textbf{k})}
\end{equation}
%
where $\xi_r$ and $\xi_i$ are constants and $P_h(\textbf{k})$, the spectrum, is:

\begin{equation}
    P_h(\textbf{k}) = A\frac{e^{\frac{-1}{{(kL)}^2}}}{k^4}
    |\textbf{k}\cdot\textbf{w}|
\end{equation}
%
$A$ is a constant, $L=V^2/g$, with $V$ being the wind speed, $g$ the
gravitational constant and finally $\textbf{w}$ the wind direction.

As seen in figure \autoref{subfig:fft_round}, the method only produces rounded
waves. In order to get the choppy effect of a stormy ocean, the position can be
displaced locally in the horizontal direction. This can be done with

\begin{equation}
    \textbf{x} = \textbf{x} + \lambda \textbf{D}(\textbf{x}, t), \quad
    \textbf{D} = \sum_{\textbf{k}}^{} -i\frac{\textbf{k}}{k}
    \tilde{h}(\textbf{k},t) e^{i\textbf{k}\cdot\textbf{x}}
\end{equation}

\begin{figure}[hbt!]
    \centering
    \begin{subfigure}[hbt]{0.45\textwidth}
        \centering
        \includegraphics[height=4cm]{fft_rounded.png}
        \subcaption{Without local displacement}\label{subfig:fft_round}
    \end{subfigure}~%
    %
    \begin{subfigure}[hbt]{0.45\textwidth}
        \centering
        \includegraphics[height=4cm]{fft_sharp.png}
        \subcaption{With local displacement}\label{subfig:fft_sharp}
    \end{subfigure}
    \caption{Result from the use of the fast Fourier transform to compute the
    ocean's shape. Source:
    \autocite{tessendorf2001simulating}.}\label{fig:fft_comparison}
\end{figure}

Using this method, it is possible to compute two height fields of different
resolution. The first one can be used as a displacement map and the second one
as a bump map. As already mentioned before, the frequency spectrum is hard to
control for an artist and the gird needs to be large to display most of the
frequencies \autocite{gonzalez2016rendering}.

%\subsubsection{Hybrid}

\subsection{Shallow Water Models}\label{subsec:shallow_water}

Shallow water models try to discretize the Navier-Stokes equations (described in
\autoref{eq:nse}) in two ways \autocite{darles2011survey}. The first one uses a
two or three dimensional grid to represent the fluid moving from cell to cell.
It is called the Eulerian approach. The second one makes the assumption that
particles transport a small fluid volume. This one is called the Lagrangian
approach. Both share similar problems and advantages
\autocite{darles2011survey}. The two can reproduce many different phenomena but
are constrained by the grid size and the particle amount. They also become
expensive when simulating finer details such as foam and spray as they need a
lot of cells or particles \autocite{darles2011survey}.

We will present only one hybrid method because a derivation of it has been used
for an industry production as described in \cref{subsub:uncharted}.

\subsubsection{Wave Particles}\label{subsub:wave_particles}
% Combines the Eulerian approach for the shape and the Lagrangian approach for
% interaction with the water.
% Also include object interaction. The waves are generated with a height field.
% The object interaction is done with vertical cells. Depending whether the
% object pushes into the water or pulling out of the water they generate a wave
% with a negative, respectively positive amplitude moving away from the object.
% Moving away only if the object is not fully submerged. If it is they move
% towards the object.
% Each face of the object is handled differently.
% Only focuses on surface waves.
% Does not generate waves.

The \textit{Wave Particles} method as described in \autocite{yuksel2007wave} is
a hybrid shallow water model. It uses an Eulerian approach to compute the shape
of the water and a Lagrangian one for the interaction with objects. Each
\textit{particle} is subdivided into three smaller ones as the wave expands. If
the \textit{particle} is too small it dies and is not displayed. When a
\textit{particle} encounters a fixed boundary it is reflected back. To compute
the water volume moved by an object, \citeauthor{yuksel2007wave} find the volume
displaced by each face of the object. This is done with the help of a low
resolution silhouette in form of a texture. They also compute the buoyant, drag
and lift forces acting on the object \autocite{yuksel2007wave}. The result can
be seen in \cref{subfig:wave_particle_1,subfig:wave_particle_2}.

\begin{figure}[hbt!]
    \centering
    \begin{subfigure}[hbt]{\textwidth}
        \centering
        \includegraphics[height=5cm]{wave_particle_1.jpg}
        \subcaption{Pool test scene, source
        \autocite{yuksel2007wave}}\label{subfig:wave_particle_1}
    \end{subfigure}\\%
    %
    \begin{subfigure}[hbt]{\textwidth}
        \centering
        \includegraphics[height=5cm]{wave_particle_2.jpg}
        \subcaption{Boat armada test scene, source
        \autocite{yuksel2007wave}}\label{subfig:wave_particle_2}
    \end{subfigure}
    \caption{Result of the demonstration application from
    \autocite{yuksel2007wave}}\label{fig:wave_particle}
\end{figure}

\subsection{Rendering of Ocean Details}\label{subsec:ocean_details}
% Modeling waves and surf 1986
% Nishita et al. [NSTN93] Check the ocean color part
% Also check On Modeling and Rendering Ocean Scenes
% A Procedural Model for Interactive Animation of Breaking Ocean Waves
% Towards Real-Time Visual Simulation of Water Surfaces
% Realistic Water Volumes in Real-Time
% Rendering Natural Waters
% GPU-based Ocean Rendering

% Talk about reflection, refraction, foam, splash, caustics, color of the water

Up until now we only presented methods which describe the shape of the water.
The most important property of water is its reflection and refraction. We will
discuss those two in \cref{subsub:reflection_refraction}. Closely linked to them
are caustics (\cref{subsub:caustics}). Another one which can be observed is
foam, presented in \cref{subsub:foam}. Finally the color of the water varies
depending on the environment as explained in \cref{subsub:caustics}.

\subsubsection{Reflection and Refraction}\label{subsub:reflection_refraction}

Reflection can occur on any kind of surface. The most common one encountered
every day is the mirror. The law of reflection states that the angle of
incidence between the normal and an incident ray is equal to the angle between
the normal and the reflected ray. The reflection vector can be computed using
\autoref{eq:reflection}. A visual representation can be seen in
\autoref{fig:reflection}.

\begin{equation}\label{eq:reflection}
    \textbf{r}_{i} = \textbf{l} - 2(\textbf{l} \cdot
    \textbf{n})\textbf{n}
\end{equation}

Refraction depends on the two media the incident ray interacts with, as shown in
\autoref{fig:reflection_refraction}. It follows Snell's law from
\autoref{eq:snell}.

\begin{equation}\label{eq:snell}
    \frac{\sin(\theta_i)}{\sin(\theta_t)} ={} \frac{n_t}{n_i}
\end{equation}

To get the refraction vector, we need to use \autoref{eq:snell} and split up
$\textbf{t}$ into its tangent and normal part. The result is shown in
\autoref{eq:refraction}.

\begin{equation}\label{eq:refraction}
    \textbf{t} = \frac{1}{n} \textbf{l} +
    \left(\frac{1}{n}\cos(\theta_i) + \sqrt{1 - \frac{1}{n^2} \left(1 -
    \cos^2(\theta_i)\right)}\right) \textbf{n}
\end{equation}

Other surfaces than perfect mirrors do not entirely reflect the incoming ray.
Only one part is reflected and the other refracted. The Fresnel equations give
the ratio between the reflected and refracted light. As they are too expensive
to compute, Schlick proposed the following approximation:

\begin{equation}\label{eq:schlick_cst}
    F_0 = \frac{n - 1}{n + 1}^2
\end{equation}
\begin{equation}\label{eq:schlick}
    F(\theta_i) = F_0 + (1 - F_0) {(1 - \cos(\theta_i))}^5
\end{equation}
%
where $n$ is the refractive index of the material. Here we assume that the
incident ray comes from the air medium.

\begin{figure}[hbt]
    \centering
    \includegraphics[height=5cm]{reflection.pdf}
    \caption{Computation of the reflection vector $\boldsymbol{r_i}$. Source:
    \autocite{RTR3}.}\label{fig:reflection}
\end{figure}

\begin{figure}[hbt]
    \centering
    \includegraphics[height=5cm]{reflection_refraction.pdf}
    \caption{Reflection and refraction vectors $\boldsymbol{r_i}$ and
    $\boldsymbol{t}$ computed from the incoming light vector $\boldsymbol{l}$.
    Source: \autocite{RTR3}.}\label{fig:reflection_refraction}
\end{figure}

\subsubsection{Caustics}\label{subsub:caustics}

Caustics only appear in shallow water and can be observed for example in rivers,
pools, near hulls or at the beach. A caustic is formed by reflected and
refracted light rays originating from a curved surface. For real-time
applications, approximations have to be made. In
\autocite[Chapter~2]{fernando2004gpu}, Guardado and Sánchez-Crespo propose a
technique based on photon mapping using a simplified backward Monte Carlo ray
tracing method. This technique is displayed in \autoref{algo:caustics}.

\begin{algorithm}
    \small
    \DontPrintSemicolon{}
    Paint the ocean floor.\\
    For each vertex in the fine mesh:\\
    \Indp{}
        Send a vertical ray.\\
        Collide the ray with the ocean's mesh.\\
        Compute the refracted ray using Snell's Law in reverse.\\
        Use the refracted ray to compute texture coordinates for the ``Sun''
        map.\\
        Apply texture coordinates to vertices in the finer mesh.\\
    \Indm{}
    Render the ocean surface.\\
    \caption{\small{Pseudocode for real-time water caustics rendering
    \autocite[Chapter~2]{fernando2004gpu}}}\label{algo:caustics}
\end{algorithm}

Other techniques exist \autocite{darles2011survey} to compute caustics such as
in \autocite{arvo1986backward} where they are stored into an illumination map.
\citeauthor{iwasaki2001efficient} treat light rays as
``beams''\autocite{heckbert1984beam,watt1990light}. They use the Z-buffer and
stencil buffer to compute the intersection between ``illumination volumes'' and
objects. However, we think that these techniques are too costly for a
real-time application and should be simplified and computed beforehand.

\subsubsection{Foam}\label{subsub:foam}

Foam is only formed\footnote{The other kind of foam that can be observed at
beaches is caused by the decaying of algal blooms or by pollutants and
detergents \autocite{noaa2017what}.} when the wind velocity exceeds 13km/h
\autocite{munk1947critical, darles2011survey}. In
\autocite{jensen2001deep,jeschke2003procedural} the authors use the height of
the vertex and the slope formed with the surrounding vertices to fade the foam
in and out.

Particle systems also exist to represent foam but they are too expensive to
render as many of them are needed to get realistic results
\autocite{darles2011survey}.

\subsubsection{Water Color}\label{subsub:water_color}

The color of the water depends on organic matter present in it, such as
phytoplanktons, chlorophyll concentration and turbidity.
\autocite{darles2011survey}. In \autocite{gonzato2000modelling} the authors use
a color lookup table taken from \autocite{ivanoff1972introduction} which
contains the color of the seas and oceans based on the angle between the sun and
the horizon.  \citeauthor{premovze2001rendering} use a more complicated
approach: they compute absorption and backscattering coefficients based on a
table containing the wave length of different water types
\autocite{premovze2001rendering}.  Unfortunately this method was used in a Monte
Carlo path tracer and we doubt that it can be used like this in a real-time
application.

\subsection{Methods Used in Industry Productions}\label{subsec:methods_industry}

% FFT, Projected Grid, LOD, talk about world of warships, anno, and oceans on a
% shoestring and uncharted. Talk also about some available shaders in the Unity
% store, the default one in Unity, and in Unreal Engine, also in Cryengine.

Many different water models are used in industry productions. One that has been
applied extensively in movies (such as ``Titanic'', ``13 days'', ``The World Is
Not Enough'') and most of the games is the one from Tessendorf
\autocite{tessendorf2001simulating} as seen in \cref{subsub:fft}. During our
research we came across two solutions which where presented at the Game
Developer Conference or at the SIGGRAPH conference. They are implemented in
Uncharted and World of Warships.

\subsubsection{Uncharted}\label{subsub:uncharted}

A modified version of the \textit{Wave Particle} approach from
\cref{subsub:wave_particles} has been used in \textit{Uncharted 3} (2011) and
\textit{4} (2016) to render rapids and the ocean. Developers chose this
technique because it is more intuitive for artists to control, it presents no
tiling artifacts, it is deterministic in time and fast to compute
\autocite{gonzalez2012water}. For the level of detail they use a ``Continuous
Distant-Dependent Level of Detail'' from \autocite{strugar2009continuous}. This
technique stitches quads of different size together. The quads become bigger the
further away they are from the camera. The result can be seen in
\autoref{fig:uncharted}.

\begin{figure}[hbt]
    \centering
    \includegraphics[height=5cm]{uncharted_ocean.png}
    \caption{View of the ocean from \textit{Uncharted 4}. Source:
    \autocite{gonzalez2016rendering}}\label{fig:uncharted}
\end{figure}

\subsubsection{World of Warships}\label{subsub:world_of_warships}

The developers at \textit{Wargaming} choose to superpose three different height
maps, as described in \autocite[Chapter~18]{pharr2005gpu}, for their
\textit{World of Warships} (2015) game \autocite{kryachko2016sea}.  For the calm
zones they use an additional one. To approximate the geometry they use the
``Projected Grid'' concept from \autocite{claes2004real}, which works as
follows:

\begin{algorithm}
    \small
    \DontPrintSemicolon{}
    Create a regular discredited plane in post-perspective space that is
    orthogonal towards the viewer.\\
    Project this plane onto a plane in the world-space.\\
    Displace the world-space vertices according to the height field.\\
    Render the projected plane.\\
    \caption{\small{Projected Grid. Source:
    \autocite{claes2004real}}}\label{algo:projected_grid}
\end{algorithm}

They took this approach because it gives them an efficient polygon distribution
over the screen \autocite{kryachko2016sea}. However, this method has several
drawbacks: it causes flickering, instability in motion, is not very accurate and
does not handle water deformations very well. By raising the triangle density up
to three to four times and redistributing them in the vertical direction, they
solved the low accuracy problem. To fix the flickering and instability in motion
issue, the developers use a modified per pixel ray casting algorithm. It
computes an improved version of Newton's method to solve the ray cast equation.
Finally they manage to create realistic water deformations with the help of six
times anti-aliasing (6xAA) \autocite{kryachko2016sea}.

\begin{figure}[hbt]
    \centering
    \includegraphics[height=5cm]{wow_water.jpg}
    \caption{Rendered ocean in \textit{World of Warships}. Source:
    \autocite{kryachko2016sea}}\label{fig:wow_sea}
\end{figure}

%\subsubsectjon{Undisclosed Game}\label{subsub:undisclosed_game}

\subsection{Candidate Applications for Embedding}\label{subsec:candidate_apps}

We now present two candidate applications which need a real-time water model:
\textit{SLProject} and \textit{0 A.D.} Although both have already one,
in the former application the model does not work and in the latter it is
outdated.

\subsubsection{SLProject}

\textit{SLProject} is a Scene Library developed and maintained by the cpvrLab at
the Bern University of Applied Sciences (BFH). It is free and open
source\footnote{Licensed under the GNU-GPL license.} and runs on Windows, MacOS,
Linux, Android and iOS\@. The project is a showcase for 3D computer graphics and
image processing topics such as real-time rendering, raytracing and feature
tracking. It is coded in C++ and OpenGL ES to ensure complete platform
independence \autocite{hudritch2017slproject, slproject2017doxygen}.

The project has a real-time water rendering demo but it is unfortunately broken
(see open ticket, \#36 ``Fix water
shader''\footnote{\url{https://github.com/cpvrlab/SLProject/issues/36}}). The
water plane is modeled by a simple multiplication of the sine and cosine
functions and is illuminated 50\% by a point light and 50\% by a cube texture.

\subsubsection{0 A.D.}

\textit{0 A.D.} is a real-time strategy game representing the 500 B.C to 500 A.D
era. It is developed by Wildfire Games, an independent game development studio.
The game is free, open source\footnote{Licensed under the GNU-GPL and CC BY-SA
license.} and runs on Windows, MacOS and Linux.

The development of the game began in 2000 when three modding groups whished to
create a free game engine. Up until 2009 the source code was accessible only to
members of Wildfire Games. Anyone interested in participating could make an
application and pass an interview. However, due to the fading interest of a
mod\footnote{Shorthand for the term \textit{modification}} which used 0 A.D.'s
game engine and the lack of programmers, they opened up the code in July 2009.
From then on many contributions have been made, notably one redesigning entirely
the code base, making new contributions easier. As of today the project is still
actively maintained, with the latest alpha-release (number 22 code named
``Venustas'') dating from July 26,
2017 \autocite{wildfire0adproject,wildfire0adstory}.

The game engine, Pyrogenesis, is heavily oriented towards modding. The core
engine is coded in C++ and Javascript is used for the game's logic, like the
artificial intelligence or the random generation of maps. This means that when
\textit{0 A.D.} is run, the game engine is started with a given mod. A mod
contains all the graphical elements, 3D models, sound, shaders, scenarios, map
generators and artificial intelligence.

The documentation for developers provides good build instructions, coding
conventions and codebase descriptions. Unfortunately we found that it is
difficult for a newcomer to understand the game engine architecture. It is less
documented and scattered accross multiple pages which are sometimes outadated.
The Doxygen documentation is minimalistic and unlike the one from
\textit{SLProject}, does not help to understand the architecture.

The current water implementation has been created in 2006 by a student form the
University of Waterloo, Canada, as a course project \autocite{zaharia2006cs}.
Over the years it has been slightly modified and optimized but appears visually
outdated (see \autoref{fig:0ad_water} on page~\pageref{fig:0ad_water}) compared
to the one used in commercial RTS games like Anno 1800 (see
\autoref{fig:anno}) or Age of Empires~3. There is an open ticket, \#48
``Advanced Water
Rendering''\footnote{\url{https://trac.wildfiregames.com/ticket/48}}, which was
opened twelve years ago and modified three years ago but is still not closed.
We have contacted the developers and they would gladly appreciate a contribution
that would close the issue.

\begin{figure}[hbt!]
    \centering
    \begin{subfigure}[hbt]{\textwidth}
        \centering
        \includegraphics[height=5cm]{figures/0ad_water-specular.jpg}
        \subcaption{Close view of the water. Source:
        \url{play0ad.com}}\label{subfig:0ad_water_close}
    \end{subfigure}\\%
    %
    \begin{subfigure}[hbt]{\textwidth}
        \centering
        \includegraphics[height=5cm]{figures/0ad_cycladic_arcgipelago_6.jpg}
        \subcaption{Zoomed out view. Source:
        \url{play0ad.com}}\label{subfig:0ad_water_far}
    \end{subfigure}
    \caption{\textit{0 A.D.}'s water rendering. Notice on the lower left part
        of \autoref{subfig:0ad_water_close} the repetition of the waves.
        In \autoref{subfig:0ad_water_far} it is even more
    apparent.}\label{fig:0ad_water}
\end{figure}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Carlos Santos             %
% ECE 351-51                %
% Lab 7                     %
% 03/24/2020                %
%                           %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\documentclass[12pt]{article}

% Language and font encoding
\usepackage[english]{babel}
\usepackage[utf8x]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{caption}
\usepackage{float}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{rotating}
\usepackage{setspace}

% Sets page size and margins
\usepackage[a4paper,top=3cm,bottom=2cm,left=3cm,right=3cm,marginparwidth=1.75cm]{geometry}

% Useful packages
\usepackage[colorinlistoftodos]{todonotes}
\usepackage[colorlinks=true, allcolors=blue]{hyperref}
\usepackage{listings}
\usepackage{gensymb}
\usepackage{graphicx} %package to manage images
\usepackage{listings}
\usepackage{color}

\definecolor{dkgreen}{rgb}{0,0.6,0}
\definecolor{gray}{rgb}{0.5,0.5,0.5}
\definecolor{mauve}{rgb}{0.58,0,0.82}

\lstset{frame=tb,
  language=Python,
  aboveskip=3mm,
  belowskip=3mm,
  showstringspaces=false,
  columns=flexible,
  basicstyle={\small\ttfamily},
  numbers=none,
  numberstyle=\tiny\color{gray},
  keywordstyle=\color{blue},
  commentstyle=\color{dkgreen},
  stringstyle=\color{mauve},
  breaklines=true,
  breakatwhitespace=true,
  tabsize=3
}

%Line Spacing
\setstretch{1.5}

%Info for Title Page
\title{ECE 351 Lab 7 Report \\ Section 52}
\date{March 24, 2020}
\author{Carlos Santos}
\begin{document}

\maketitle
\tableofcontents

%Make a Title Page
\vspace{\fill}
\maketitle
\vspace{\fill}
\clearpage

%Introduction
\section{Introduction}
This lab is meant for us to become familiar with Laplace-domain block diagrams. Especially being able to find both open-loop and closed-loop response of the system.

%Equations
\section{Equations}

    \begin{equation}
        G(s) = \frac{s+9}{(s^2 - 6s - 16)(s+4)}
    \end{equation}

    \begin{equation}
        A(s) = \frac{s+4}{s^2 + 4s + 3}
    \end{equation}

    \begin{equation}
       B(s) = s^2 + 26s + 168
    \end{equation}

    Open-loop:
    \begin{equation}
        H(s) = A(s) * G(s)
    \end{equation}

    Closed-loop:
    \begin{equation}
        H(s) = G(s)A(s) / 1 + G(s)B(s)s
    \end{equation}

%Methodology
\section{Methodology}

First of this lab was successful I got all the code to work properly.
\begin{enumerate}
    \item The first thing I did was to represent each equation in python. I used the following code for all three equations. I modified it a bit accordinaly to each equation. For equation 3 we only had a numerator.:
        \begin{lstlisting}
            # Equation G(s) Transfer Function
            Gnum = [1,9] # numerator
            Gden = [1,-2,-40,-64] # denomerator
        \end{lstlisting}
    \item Next, I verified each equations poles and zeros. To do that I used a special scipy.signal function called tf2zpk(). This function can be used like this:
    \begin{lstlisting}
        # Equation G(s) Transfer Function
        rG,pG,kG = sig.tf2zpk(Gnum,Gden)
    \end{lstlisting} It will return 3 array items.
    \item After finding the poles the next thing I did was to find the open-loop and closed-loop transfer functions for the system. An open-loop transfer function is the shortest path from X(s) to Y(s) not including any feedback loops. A closed-loop transfer function is the path from X(s) to Y(s) including feedback loops. (See Figure 3, Equation 4 and 5).
    \item Lastly, I plotted the closed-loop and open-loop functions using the pyplot python library from matplotlib.

\end{enumerate}

%Results
\section{Results}

\begin{figure}[H]
\caption{Closed-loop Response}
\centering
\includegraphics[width=.8\textwidth]{closed_loop_response.png}
\end{figure}

\begin{figure}[H]
\caption{Open-loop Response}
\centering
\includegraphics[width=.8\textwidth]{open_loop_response.png}
\end{figure}

\begin{figure}[H]
\caption{Block Diagram for lab 7}
\centering
\includegraphics[width=.8\textwidth]{block_diagram_for_lab_7.png}
\end{figure}

Code for non-proper functions:
\begin{lstlisting}
# Equation B(s)
rB = np.roots(Bnum)
print('Equation B(s)')
print('rB:',rB)
print('\n')
\end{lstlisting}

\begin{enumerate}
    \item Section 3.3.4: Considering the expression found in Task 3, is the open-loop response stable? Explain why
or why not?

    \begin{enumerate}
        \item No, it is not stable. We can look at the graph and see that it shoots off into infinity as "t" approaches 2. We also know it's not stable because the denominator contains negative values.
    \end{enumerate}

    \item  Section 3.3.6: Does your result from Task 5 support your answer from Task 4? Explain how?
    \begin{enumerate}
        \item Yes, it does. Because if we graph the convolution we see it shooting off into infinity as "t" approaches 2.
    \end{enumerate}
    \item Section 4.3.3: Using the closed-loop transfer function from Task 1, is the closed-loop response stable?
Explain why or why not>
    \begin{enumerate}
        \item Yes, it is stable. I can tell because it has all positive denominator values. I also see that from the plot.
    \end{enumerate}

    \item Section 4.3.5: Does your result from Task 4 support your answer from Task 3? Explain how?
    \begin{enumerate}
        \item It does support it. We can see by graphing the convolution that the system becomes stable as "t" approaches 2. It appears to begin to level off.
    \end{enumerate}

\end{enumerate}

%Questions
\section{Questions}
 \begin{enumerate}
     \item In Part 1 Task 5, why does convolving the factored terms using scipy.signal.convolve()
result in the expanded form of the numerator and denominator? Would this work with your
user-defined convolution function from Lab 3? Why or why not?

    \begin{enumerate}
        \item Convolving results in the expanded form of the neumerator and denominator because we are essentially multiplying two functions together. Thus result usually in a higher order equation. I don't think this would work with my user-defined function from lab 3. If I recall right it was designed to convolve ramp and step functions.
    \end{enumerate}

    \item Discuss the difference between the open-loop and closed-loop systems from Part 1 and Part 2.
How does stability differ for each case, and why?

    \begin{enumerate}
        \item An open-loop function is the transfer function without any corrections from the system. Thus most the time it is unstable. The closed-loop functions is the transfer function with corrections otherwise known as feedback loops included. The feedback loops help stabilize the system.
    \end{enumerate}

    \item What is the difference between scipy.signal.residue() used in Lab 6 and
scipy.signal.tf2zpk() used in this lab?

    \begin{enumerate}
        \item The residue() function can handle repeated roots and the tf2zpk() function can't. The tf2zpk() function also can't handle 0's to well.
    \end{enumerate}

    \item Is it possible for an open-loop system to be stable? What about for a closed-loop system to
be unstable? Explain how or how not for each.

    \begin{enumerate}
        \item Yes it is possible for an open-loop system to be stable. The example I can think of is a car driving down a highway. As soon as the gas peddle is let off it will come to a stop eventually without any correction or feedback. It is also possible for a closed-loop system to become unstable due to over correction from feedback loops. An example would be the same car doing down the highway but this time we slam on the brakes hard causing the car to stop quicker and probably not in a safe way. The brake system (feedback loop) plus the friction will cause the car to come to a stop but will increase the chances of it ending badly.
    \end{enumerate}

    \item Leave any feedback on the clarity/usefulness of the purpose, deliverables, and expectations
for this lab.

    \begin{enumerate}
        \item N/A. This lab went well.
    \end{enumerate}

 \end{enumerate}

%Conclusion
\section{Conclusion}

%Appendix
\section{Appendix}

\end{document}


\documentclass[simplex.tex]{subfiles}
% NO NEED TO INPUT PREAMBLES HERE
% packages are inherited; you can compile this on its own
\definecolor{jgreen}{HTML}{197300}
\definecolor{jblue}{HTML}{0000cd}
\definecolor{jred}{HTML}{cc0000}

\begin{document}
\onlyinsubfile{
\thispagestyle{empty}

%%%% Table of Contents
\tableofcontents

%%%% Publications
\bibliographystyle{IEEEtran}
\begin{spacing}{0.5}
\section*{Publications, Presentations, and Talks}
%\vspace{-20pt}
\nocite{*}
{\footnotesize	\bibliography{simplex}}
\end{spacing}
%%%% End Publications
}

\subsection{meda}

Matrix Exploratory Data Analysis (meda) is a package being developed to
allow for easy generation of modern summary statistics effective for
high-dimensional data analysis.

\begin{compactitem}
  \item Source code: \href{https://github.com/neurodata/meda}{https://github.com/neurodata/meda}
  \item Example output generated from Fisher's Iris data is here:
    \href{http://docs.neurodata.io/meda}{http://docs.neurodata.io/meda}
\end{compactitem}

The goal of this package is to realize the following checklist: Given a new set of n samples of vectors in $\mathbb{R}^d$

\begin{compactenum}
  \item histogram of feature types (binary, integer, non-negative, character, string etc.)
  \item \# NaNs per row? Per column? Infs per row? Per column? "Zero" variance rows? columns?
  \item Heat map of raw data that fits on screen (k-means++ to select 1000 samples, CUR to select 100 dimensions)
  \item 1st moment statistics
  \begin{compactenum}
    \item mean (line plot + heatmap)
    \item median (line plot + heatmap)
  \end{compactenum}
  \item 2nd moment statistics
  \begin{compactenum}
    \item correlation matrix (heatmap)
    \item matrix of energy distances (heatmap)
  \end{compactenum}
  \item density estimate
  \begin{compactenum}
    \item 1D marginals (Violin + jittered scatter plot of each dimension,  if n > 1000 or d>10, density heatmaps)
    \item 2D marginals (Pairs plots for top ~8 dimensions, if n*d>8000, 2D heatmaps)
  \end{compactenum}
  \item Outlier plot
  \item cluster analysis (IDT++)
  \begin{compactenum}
    \item BIC curves
    \item mean line plot
    \item covariance matrix heatmaps
  \end{compactenum}
  \item spectral analysis
  \begin{compactenum}
    \item cumulative variance (with elbows) of data matrix
    \item eigenvectors (pairs plot + heatmap)
  \end{compactenum}
\end{compactenum}

\begin{compactitem}
\item To rescale the data in case of differently scaled features, we will implement the following options:
\begin{compactitem}
  \item raw
  \item linear options
  \begin{compactitem}
    \item linear squash between 0 \& 1
    \item mean subtract and standard deviation divide
    \item median subtract and median absolute deviation divide
    \item make unit norm
  \end{compactitem}
  \item nonlinear
  \begin{compactitem}
    \item rank
    \item sigmoid squash
  \end{compactitem}
\end{compactitem}

\item To robustify in the face of outliers, we will utilize
 \href{http://projecteuclid.org/euclid.bj/1438777595}{Geometric median and robust estimation in Banach spaces}

\item { if features have categories}
\begin{compactenum}
  \item sort by category
  \item color code labels by category
\end{compactenum}

\item { if points have categories}:
   label points in scatter plots by symbol
\end{compactitem}

\vspace{12pt}

For point 6 (a) in the above checklist we have developed functionality
in \verb+meda+ to plot 1-dimensional heatmaps.  The 1D heatmap is
a different representation of a histogram, using color to denote count
instead of bin height, see figure~\ref{fig:meda}.

\begin{figure}[!h]
\begin{cframed}
\centering
\includegraphics[width=0.95\textwidth]{../../figs/201701-meda-1dheat.pdf}
\caption{A one dimensional heatmap for each of the $\log_{10}$
  transformed feature columns of the Kristina15 synaptome dataset.
  Colors correspond to the count of data points falling in to each bin.
  Scott's binning streategy is used which, in this case, yields 105 bins
  of equal width $w = 0.05$.
  Label colors and groups starting from the bottom:
  \textcolor{jgreen}{Synap1\_F0 - Synapo\_F0 (green, excitatory)},
  \textcolor{jred}{gad\_F0 - GABABR\_F0 (red, inhibatory)},
  \textcolor{jblue}{Vglut3\_F0 - DAPI\_F0 (blue, other)}.
  }
\label{fig:meda}
\end{cframed}
\end{figure}

\end{document}


\documentclass{article}
\usepackage[utf8]{inputenc}

\title{MAT257 Notes}
\author{Jad Elkhaleq Ghalayini}
\date{November 26 2018}

\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{mathtools}
\usepackage{enumitem}
\usepackage{graphicx}
\usepackage{cancel}

\usepackage[margin=1in]{geometry}

\newtheorem{theorem}{Theorem}
\newtheorem{lemma}{Lemma}
\newtheorem{definition}{Definition}
\newtheorem*{corollary}{Corollary}
\newtheorem{exercise}{Exercise}
\newtheorem{claim}{Claim}

\DeclareMathOperator{\Int}{Int}
\DeclareMathOperator{\grad}{grad}
\DeclareMathOperator{\Ker}{Ker}
\DeclareMathOperator{\Ima}{Im}

\newcommand{\reals}[0]{\mathbb{R}}
\newcommand{\nats}[0]{\mathbb{N}}
\newcommand{\ints}[0]{\mathbb{Z}}
\newcommand{\rationals}[0]{\mathbb{Q}}
\newcommand{\brac}[1]{\left(#1\right)}
\newcommand{\sbrac}[1]{\left[#1\right]}
\newcommand{\mc}[1]{\mathcal{#1}}
\newcommand{\eval}[3]{\left.#3\right|_{#1}^{#2}}
\newcommand{\ip}[2]{\left\langle#1,#2\right\rangle}
\newcommand{\prt}[2]{\frac{\partial #1}{\partial #2}}
\newcommand{\mb}[1]{\mathbf{#1}}
\newcommand{\hlfspc}[0]{\mathbb{H}}

\begin{document}

\maketitle

\section{Integration}

In the couple of weeks left in the term, we are going to develop the theory of integration. We said what the integral of a closed rectangle on \(\reals^n\) was last time, in complete analogy with first year calculus:
\begin{definition}
  If \(f: A \to \reals\) is a bounded function on a closed rectangle \(A \subset \reals^n\), \(f\) is \underline{integrable} on \(A\) if, where \(\mc{P}\) is the set of partitions of \(A\),
  \[\sup_{P \in \mc{P}} L(f, P) = \inf_{P \in \mc{P}} U(f, P)\]
  then the common value is called the integral of \(f\) over \(A\), written
  \[\int_Af\]
\end{definition}
Throughout this lecture we'll see that most of the properties you learned in first year calculus will carry over to this more general case. One thing we'll be carrying over today is that instead of expressing things in terms of \(\sup\) and \(\inf\), we can instead express things in terms of \(\epsilon\):
\begin{lemma}
  If \(f: A \to \reals\) is a bounded function on a closed rectangle \(A\) then \(f\) is integrable on \(A\) if and only if for every \(\epsilon > 0\), there is apartition \(P\) of \(A\) such that
  \[U(f, P) - L(f, P) < \epsilon\]
\end{lemma}
The proof is exactly like first year, and one direction should be immediate:
\begin{proof}
\begin{itemize}
  \item \(\impliedby\): if there's a partition \(P\) for every \(\epsilon\), then there's no ``room'' between the \(\sup\) and the \(\inf\)
  \item \(\implies\): if \(\sup = \inf\), then there are partitions \(P, P'\) such that
  \[U(f, P) - \inf_{P \in \mc{P}} U(f, P) < \frac{\epsilon}{2} \land \sup_{P \in \mc{P}}L(f, P) - L(f, P') < \frac{\epsilon}{2} \implies U(f, P'') - L(f, P'') < \epsilon\]
  where \(P''\) is a refinement of \(P\) and \(P'\).
\end{itemize}
\end{proof}
Just like in first year. And also, the kind of first calculations of itnegrals that you've done in first year work out the same way here. For example,
\begin{enumerate}

  \item If \(f = c\) is a constant, then \(f\) is integrable and
  \[\int_Af = cv(A)\]
  where \(v(A)\) denotes the volume of \(A\). This is because, for any subrectangle \(S\) of \(P\),
  \[m_S(f) = c = M_S(f) \implies L(f, P) = \sum m_s(f)v(S) = c\sum v(S) = cV(A)\]
  with an analogous calculation for \(U(f, P)\)

  \item Let \(f: [0, 1] \times [0, 1] \to \reals\). Then
  \[f(x, y) = \left\{\begin{array}{cc}
    0 & \text{if } x \in \rationals \\
    1 & \text{otherwise}
  \end{array}\right.\]
  is not integrable, since for any partition \(P\),
  \(m_S(f) = 0 \land M_S(f) = 1 \implies L(f, P) = 0 \land U(f, P) = 1\)

\end{enumerate}

\subsection{Basic Properties of the Integral}

Suppose we have two integrable functions
\[f, g: A \to \reals\]
Then
\begin{enumerate}

  \item \(f + g\) is integrable and
  \[\int_A(f + g) = \int_Af + \int_Ag\]

  \item \[f \leq g \implies \int_Af \leq \int_Ag\]

  \item \(f \cdot g\) is integrable

  \item \(|f|\) is integrable and \[
  \int_A|f| \geq \left|s22\int_Af\right| \geq \int_Af\]
  It's enough to show that \(|f|\) is integrable

  \item Exercise: any increasing function is integrable

\end{enumerate}

\end{document}


\newcommand{\lsql}[1]{\lstinline[language=SQL,prebreak=]!#1!}

\lab{SQL and Relational Databases}{SQL}
\objective{Understand concepts of a relational database and the fundamentals of the SQL language via SQLite.}
\label{lab:sql_rdb}

When working with large amounts of data, it is important to be able to quickly find and retrieve interesting information.
Fortunately, there is a way to handle such massive amounts of data in a reasonably efficient way: a database.
A database is simply a structured repository of data, and it allows us to store and retrieve information very quickly.
It is managed by a \emph{database management system}, or DBMS.
The DBMS is software that allows users to interact directly with the database.

\section*{Relational Databases}
A \emph{relational database} is a paradigm for organizing data inside of a database.
In this paradigm, the data are broken down into tuples of information.
These tuples are then grouped into tables, or \emph{relations}, each of which is simply a set of tuples.
Each table has a \emph{schema} that defines the attributes of the tuples within the table.
If we fix an order to the attributes in the schema, we can think of each attribute as a column
of the table, and each tuple as a row of the table. See Figure \ref{fig:relation} for an illustration of
these ideas.

As an example, suppose we have demographic data for a large number of individuals.
If we are interested in the gender and age of the individuals, we might make a table
with schema (Name, Gender, Age). This table would consist of several 3-tuples, such as
(Jane Doe, F, 20). Alternatively, we can view this table as having three columns
and as many rows as there are individuals within our data set.
We might also create a table with schema (Name, Employment Status, Income, Education).

In the relational paradigm, there must be at least one attribute in each schema that can act as a \emph{primary key}.
This can uniquely identify each tuple of the table.
It is common to use an ID number or other such unique information for the primary key.
In our example above, the ``Name" attribute acted as a primary key. However, this attribute only works as a primary
key provided no two individuals within the data set have the same name.
\begin{figure}
\centering
\includegraphics[width=\textwidth]{rdb_table.pdf}
\caption{Elements of a relation.}
\label{fig:relation}
\end{figure}

One important feature of a database is the \emph{transaction}, which is a conceptual protocol for
interacting with the database.
Most relational databases are transactional databases.
The best way to conceptualize this is imagine that your database is like a bank.
Your connection to the database is analogous to the bank teller.
When you make one or more deposits and withdrawals, you are making a transaction.
A database transaction should have certain properties to protect the integrity of the data.
These properties are described in detail in \li{en.wikipedia.org/wiki/ACID}

\section*{Introduction to SQL}
Most common DBMSs use a variant of the SQL language to interact with the database.
SQL is an acronym for \emph{Structured Query Language}, and may be pronounced like the word ``sequel" or by saying
the letters ``s", ``q", and ``l" separately.
While SQL is not generally portable across different DBMSs, we will focus on the parts of SQL that are relatively common.
In particular, we will base our discussion on the SQLite database management system, a very popular DBMS.

SQL consists of blocks of code called statements.
Each statement is made up of clauses which may or may not require predicates.
Predicates specify conditions that can limit the effect of a clause.

\begin{info}
SQL commands are often written in all caps to help distinguish them from the other parts of the query.
It is only a matter of style.
SQLite, along with most other database managers, is case insensitive.
In Python's SQL interface, the semicolon is also not needed.
However, most other database systems will require it, so it's a good idea to conform in Python.
\end{info}

Let's look at an example SQL statement:
\begin{lstlisting}[language=SQL]
SELECT * FROM table WHERE id=3+1 AND name='Bob';
\end{lstlisting}
This statement includes a SELECT clause and a WHERE clause.
The WHERE clause contains two predicates: \texttt{id=3+1} and \texttt{name='Bob'}.
These two predicates limit the effect of the SELECT clause because any resulting tuples in the table must satisfy both conditions.
This entire statement is classified as a query since it does not modify the database in any way.

SQL has several classes of statements.
The two main classes we will cover in this lab are schema (Table \ref{table:sql-schema}) and data manipulation (Table \ref{table:sql-data_manip}).
We will give you a simplified description of each command and its syntax.
You are encouraged to look up the full syntax outside of this lab.

\begin{table}
\begin{tabular}{|l|l|}
\hline
Keyword & Syntax \\
\hline
\lsql{CREATE TABLE} & \lsql{CREATE TABLE <table> (<col1> <type>, <col2> <type>, ...);} \\
\lsql{DROP TABLE} & \lsql{DROP TABLE <table>;} \\
\lsql{CREATE INDEX} & \lsql{CREATE INDEX <name> ON <table> (<col>);} \\
\lsql{DROP INDEX} & \lsql{DROP INDEX <name>;} \\
\hline
\end{tabular}
\caption{The SQL Schema commands}
\label{table:sql-schema}
\end{table}

\begin{table}
\begin{tabular}{|l|l|}
\hline
Keyword & Syntax \\
\hline
\lsql{INSERT INTO} & \lsql{INSERT INTO <table> <attributes> VALUES (<value1>, <value2>, ...);} \\
\lsql{UPDATE} & \lsql{UPDATE <table> SET (<col1>=<val1>, <col2>=<val2>, ...) WHERE <condition>;} \\
\lsql{DELETE} & \lsql{DELETE FROM <table> WHERE <condition>;} \\
\lsql{SELECT} & \lsql{SELECT <attributes> FROM <table> WHERE <condition>;} \\
\hline
\end{tabular}
\caption{The SQL Data Manipulation commands}
\label{table:sql-data_manip}
\end{table}

\section*{SQL in Python}
Python has built-in support for SQLite databases using the standard library.
Let's open a database called \texttt{test1}.
\begin{lstlisting}
import sqlite3 as sql
db = sql.connect("test1")
\end{lstlisting}
The \li{connect()} function is used to connect to a database.
If it does not already exist, then a new database will be created using the string passed as the argument for the name.
The new database was created as a file in the current working directory.

To execute SQL commands, we need to get a cursor object from the database.
\begin{lstlisting}
cur = db.cursor()
\end{lstlisting}
The cursor object has several useful methods (Table \ref{table:cursormethods}).
\begin{table}
\begin{tabular}{|l|l|}
\hline
Method & Description \\
\hline
\li{execute} & Execute a single SQL statement \\
\li{executemany} & Execute a single SQL statement over a sequence \\
\li{executescript} & Execute a SQL script (multiple SQL commands) \\
\li{close} & Closes the cursor object \\
\hline
\end{tabular}
\caption{Cursor object methods}
\label{table:cursormethods}
\end{table}

Before creating a table, we need to  understand how SQLite stores information in a database.
SQLite uses five native data types (a simplified system from other SQL database managers).
Table \ref{table:typemap}, gives a mapping between Python and SQLite native types.
\begin{table}
\begin{tabular}{|l|l|}
\hline
Python Type & SQLite Type \\
\hline
\li{None} & \lsql{NULL} \\
\li{int} & \lsql{INTEGER} \\
\li{long} & \lsql{INTEGER} \\
\li{float} & \lsql{REAL} \\
\li{str} & \lsql{TEXT} \\
\li{buffer} & \lsql{BLOB} \\
\hline
\end{tabular}
\caption{Python and SQLite types mapping}
\label{table:typemap}
\end{table}

\subsection*{Creating and Dropping Tables}
Let's create a table.
\begin{lstlisting}
cur.execute('CREATE TABLE student_information (StudentID INTEGER NOT NULL, Name TEXT, SSN INTEGER, MajorCode INTEGER);')
\end{lstlisting}
This will create the empty table in Table \ref{table:student_information}.
\begin{table}
\begin{tabular}{|l|l|l|l|}
\hline
StudentID & Name & SSN & MajorCode \\
\hline
\end{tabular}
\caption{student\_information}
\label{table:student_information}
\end{table}

The arguments in parentheses are the column names followed by the data type that entries in that column will be,
and together these form the schema of the table.
The \lsql{INTEGER} data type in SQLite is a 1, 2, 3, 4, 6, or 8 byte integer depending on the value.
The \lsql{NOT NULL} command is a \emph{constraint} on the StudentID column.  It requires that all records in the table have a student ID.

\begin{info}
SQLite does not enforce types on columns.
Just like Python, SQLite is dynamically typed.
However, most other database systems strictly enforce column types.
It is a good idea to conform to the column types specified in the schema.
\end{info}

Note that each command we execute returns the same cursor object.  This object is equipped with a method that allows us to look
at any results of the previous command.  The result is formally known as the \emph{result set}.
If you use \li{cur.fetchall()}, you will see an empty list.
That is because the create table command does not return a result set.

Now we want to build a relation between students, the courses they've had, and their grades in those courses.
\begin{lstlisting}
cur.execute('CREATE TABLE student_classes (StudentID INT NOT NULL, CourseID INT, Grade TEXT);')
\end{lstlisting}

\begin{problem}
In this problem you will create two new tables.
The first table will be called MajorInfo and have a column called MajorCode and MajorName.
MajorCode is an integer and MajorName is a string.

The second table will be called CourseInfo and have columns called CourseID and CourseName, also integers and strings, respectively.
\label{prob:new_tables}
\end{problem}

We can also destroy tables using the \lsql{DROP TABLE} command.
\begin{lstlisting}
cur.execute("CREATE TABLE test_table (id int, name text);")
\end{lstlisting}
We can delete the table by dropping it.
\begin{lstlisting}
cur.execute("DROP TABLE test_table;")
\end{lstlisting}
If a table doesn't exist, an exception will be raised.
We can tell the database to drop the table only if it really exists by using \lsql{DROP TABLE IF EXISTS test_table;}.

\subsection*{Inserting and Removing Data}
Let's insert some data into our new tables.
We can add rows to tables using the \lsql{INSERT INTO} command.
\begin{lstlisting}
cur.execute("INSERT INTO student_information VALUES(55, 'John Smith', 372897382, 2);")
\end{lstlisting}
After running this statement, we will have the table in Table \ref{table:student_information1}.
\begin{table}
\begin{tabular}{|l|l|l|l|}
\hline
StudentID & Name & SSN & MajorCode \\
\hline
55 & John Smith & 372897382 & 2 \\
\hline
\end{tabular}
\caption{student\_information}
\label{table:student_information1}
\end{table}

Note that SQLite will assume that values match sequentially with the schema of the table.
We can also specify the schema of the table to use in the mapping of the values.
\begin{lstlisting}
cur.execute("INSERT INTO student_information(MajorCode, Name, SSN, StudentID)  VALUES(55, 'John Smith', 372897382, 2);")
\end{lstlisting}
This will map the value 55 to MajorCode and the value 2 to StudentID.  This may be useful sometimes.

It can quickly become tedious to insert large amounts of data into a table, one row at a time.
We can automate the process somewhat by using the \li{executemany} method of the cursor object.
To insert several rows into a table using a single command, we can do the following:
\begin{lstlisting}
cur.executemany("INSERT INTO student_information VALUES (?, ?, ?, ?);", rows)
\end{lstlisting}
In the code above, we assume that \li{rows} is a Python list of tuples, each tuple containing the data for one
row.

We may remove rows from a table using the \lsql{DELETE FROM} command.
\begin{lstlisting}
cur.execute("DELETE FROM student_information WHERE MajorCode=55;")
\end{lstlisting}

\begin{warn}
\emph{\textbf{Never}} use Python's string operations to construct a SQL query.
It is extremely insecure and is an easy target for a well known type of database called a SQL injection attack.

Parameter substitution can be used to construct dynamic queries.
In the simplest way, it involves using a `?' character whenever you want to use a value and providing a
sequence of values as a second argument to \li{execute()}.
\begin{lstlisting}
statement = "INSERT INTO student_information VALUES(?, ?, ?, ?);"
values = (55, 'John Smith', 372897382, 2)
cur.execute(statement, values)
\end{lstlisting}
\end{warn}

\subsection*{Updating Rows of a Table}
We can modify records in a table by using the \lsql{UPDATE} command.
\begin{lstlisting}
cur.execute("UPDATE student_information SET MajorCode=2, StudentID=55, Name='Jonathan Smith' WHERE StudentID=2;")
\end{lstlisting}

\begin{info}
When updating a table, having a sufficient \lsql{WHERE} clause is essential.  Any record that matches the criteria will be modified.
If we omitted the \lsql{WHERE} clause, every record in the table would be set to the values given in the example.
\end{info}

\begin{problem}
The ICD is a large collection of codes used to classify any diagnosis that a doctor would make.
When someone goes to the hospital or doctors office, their visit will be recorded using these codes.
Insurance companies, the government, and researchers find this data useful.
The data file provided to you, \li{icd9.csv}, has simulated health histories for one million persons.
Each line has columns for identification number, gender, and age, followed by ICD-9 codes of various quantities.
Note that the codes for each individual are written in a single string, each code separated by semicolons.
Create a new database with a single table to store all the simulated data.
Yourn table should have four columns, one each for id number, gender, age, and codes.

Because of the volume of data, it is highly recommended you use the \li{executemany()} method of the cursor.
It will be about twice as fast as using an \li{execute()} for each line of the CSV file.
Recall the \li{csv} package in Python. To read a CSV file into a list of tuples, where each tuple consists
of the delimited values of a particular line in the file, one can use the following code as a guideline:
\begin{lstlisting}
import csv
with open('filename', 'rb') as csvfile:
    rows = [row for row in csv.reader(csvfile, delimiter=',')]
\end{lstlisting}
\label{prob:icd9tables}
\end{problem}

\begin{problem}
Create the following tables in the same database you created in Problem \ref{prob:icd9tables}.  You may do so however you think is best.

\begin{table}[H]
\begin{tabular}{|l|l|l|}
\hline
StudentID & Name & MajorCode \\
\hline
401767594 & Michelle Fernandez & 1 \\
678665086 & Gilbert Chapman & 1 \\
553725811 & Roberta Cook & 2 \\
886308195 & Rene Cross & 3 \\
103066521 & Cameron Kim & 4 \\
821568627 & Mercedes Hall & 3 \\
206208438 & Kristopher Tran & 2 \\
341324754 & Cassandra Holland & 1 \\
262019426 & Alfonso Phelps & 3 \\
622665098 & Sammy Burke &2 \\
\hline
\end{tabular}
\caption{students}
\end{table}

\begin{table}[H]
\begin{tabular}{|l|l|}
\hline
ID & Name \\
\hline
1 & Math \\
2 & Science \\
3 & Writing \\
4 & Art \\
\hline
\end{tabular}
\caption{majors}
\end{table}

\begin{table}[H]
\begin{tabular}{|l|l|l|}
\hline
StudentID & ClassID & Grade \\
\hline
401767594 & 4 & C \\
401767594 & 3 & B- \\
678665086 & 4 & A+ \\
678665086 & 3 & A+ \\
553725811 & 2 & C \\
678665086 & 1 & B \\
886308195 & 1 & A \\
103066521 & 2 & C \\
103066521 & 3 & C- \\
821568627 & 4 & D \\
821568627 & 2 & A+ \\
821568627 & 1 & B \\
206208438 & 2 & A \\
206208438 & 1 & C+ \\
341324754 & 2 & D- \\
341324754 & 1 & A- \\
103066521 & 4 & A \\
262019426 & 2 & B \\
262019426 & 3 & C \\
622665098 & 1 & A \\
622665098 & 2 & A- \\
\hline
\end{tabular}
\caption{grades}
\end{table}

\begin{table}[H]
\begin{tabular}{|l|l|}
\hline
ClassID & Name \\
\hline
1 & Calculus \\
2 & English \\
3 & Pottery \\
4 & History \\
\hline
\end{tabular}
\caption{classes}
\end{table}
\label{prob:sampletables}
\end{problem}

\subsection*{Selecting Data From Tables}
The process of retrieving data from a table in a database is accomplished by the \lsql{SELECT} statement.
The \lsql{SELECT} statement can be thought of as a very high level set description.
For example, to view the contents of an entire table, we simply need to unconditionally select its contents.
\begin{lstlisting}[language=SQL]
SELECT * FROM students;
\end{lstlisting}
This is equivalent to the following set (where $x$ is a row).
\[ \setconstruct{x}{x \in \text{classes}} \]

We can also select specific columns.
\begin{lstlisting}[language=SQL]
SELECT StudentID, Name FROM students;
\end{lstlisting}
Or we can impose conditions on the selected rows.
\begin{lstlisting}[language=SQL]
SELECT StudentID, Name FROM students WHERE MajorCode=1;
\end{lstlisting}
This query results in the following table (Table \ref{table:selectmath}) where the contents are all the students that are math majors.
\begin{table}
\begin{tabular}{|l|l|}
\hline
StudentID & Name \\
\hline
401767594 & Michelle Fernandez \\
678665086 & Gilbert Chapman \\
341324754 & Cassandra Holland \\
\hline
\end{tabular}
\caption{Selected students who are math majors.}
\label{table:selectmath}
\end{table}

Select statements return a \emph{result set}.
This is an iterable object.
Each row in the object is represented as a tuple of values.
\begin{lstlisting}
cur.execute('SELECT StudentID, Name FROM students WHERE MajorCode=1;')
for student in cur:
    print student
\end{lstlisting}
We can also use the fetch methods of the returned cursor to extract rows from the result set (Table \ref{table:fetching}).
\begin{table}
\begin{tabular}{|l|l|}
\hline
Method & Description \\
\hline
\li{fetchone()} & Return a single row from the result set \\
\li{fetchmany(n)} & Return the next $n$ rows from the result set \\
\li{fetchall()} & Return the entire result set \\
\hline
\end{tabular}
\caption{Fetch methods of a cursor.}
\label{table:fetching}
\end{table}

\begin{problem}
From the ICD9 table you created in Problem \ref{prob:icd9tables}, how many men between the ages of 25 and 35 are there?  How many women between those same ages?
\label{prob:youngfreqcodes}
\end{problem}

\begin{comment}
From the ICD9 table you created in Problem \ref{prob:icd9tables}, how many men between the ages of 25 and 35 are there?  How many women between those same ages?

What code is the most frequent code for men between age 25 and 35?
What code is the most frequent for women between  the same age?
\end{comment}

\subsection*{When an Error Occurs}
It is important to be able to recover from errors gracefully, especially when working a database.
Data integrity in a database is often a critical need.
When an error occurs, we need to undo the changes that triggered the error.
Fortunately, \li{sqlite3} reports a variety of errors.
These errors and when they are raised is explained in PEP249 (\url{http://legacy.python.org/dev/peps/pep-0249/}).
\begin{description}
\item[Error] The base class for errors thrown by \li{sqlite3}.  All other errors inherit from this class.  Catching this error will catch any error raised.
\item[InterfaceError] Raised when there is a problem with the interface to the database rather than the database itself.
\item[DatabaseError] Raised when there is an error with the database itself.
\item[DataError] Subclass of DatabaseError.  Raised when there are errors in the processed data (division by zero, value out of range, etc.).
\item[OperationalError] Subclass of DatabaseError. Raised for errors related to the database that are not the fault of the programmer.  For example, an unexpected disconnect, failure to process a transaction, a memory allocation error during a transaction, etc.
\item[IntegrityError] Subclass of DatabaseError.  Raised when the relational integrity of the database is compromised.
\item[InternalError] Subclass of DatabaseError. Raised when there is an internal error such as an invalid cursor, out-of-sync transaction, etc.
\item[ProgrammingError] Subclass of DatabaseError.  Raised for programming errors.
\item[NotSupportedError] Subclass of DatabaseError.  Raised when a method is called that is not supported by the database.
\end{description}

The way to gracefully recover from errors is to catch them and handle them accordingly.  For example, if any error occurs, with the interface or the database, we immediately rollback the transaction.  If no error occurs, commit.  We could use if-statements or we could use a try-except block.
\begin{lstlisting}
try:
    <code>
    db.commit()
except sql.Error:
    db.rollback()
\end{lstlisting}
Note that rolling back is not needed if we are just performing queries.
If we don't change any of the data in the database, there is no need to roll anything back.  However, even with queries, there is the potential for errors.  You must design your code to handle these errors gracefully.

\subsection*{Ending the SQL Session}
Once we are finished performing SQL statements and interacting with the database, we need to commit our changes and safely close the connection
to the database.
This can be done by calling methods on the database connection object.
\begin{lstlisting}
db.commit()     #save changes made in the transaction
db.close()      #safely close the database
\end{lstlisting}

A database connection is automatically closed in Python when the connection object is garbage-collected.  However, it is nice to be safe and explicit in closing a database connection using the \li{close()} method.

\let\undefined\lsql


\chapter{Results}\label{cha:results}

In this chapter, the results of the experiments are reported, as described in \Cref{cha:methods}. Some sentences and formulations are adapted from the project report \textit{Multi-Layer Perceptrons for Branching in Mixed-Integer Linear Programming} (2020), as this thesis shares methods with the report.

\section{Data Set}\label{sec:datagen}

Problem instances are generated as described in \Cref{ssec:probleminstances}. In total, $2000$ training instances, $500$ validation instances, and $ 500 $ test instances are created. These are of size $100 \times 500$ for the combinatorial auction problems and $100 \times 500$ for the set covering problems, as stated in \Cref{sec:dataset}. This information is shown in \Cref{tab:instances}.

The generated problem instances are then solved as specified in \Cref{ssec:expertsolutiongeneration}. This results in a total of 50000 training samples, 10000 validation samples, and 10000 test samples for the combinatorial auctions problems, and the same for the set covering problems. This is also shown in \Cref{tab:instances}.

The division into train, validation and test sets is done as is standard in \gls{ML} development \cite{goodfellow2016deep}. The train set will be used to optimize the model parameters, the validation set will be used to monitor the model's ability to generalize to unseen samples, and the test set will be used to evaluate the model.

A notable difference between the problem classes is the number of constraints. The combinatorial auction problems average about 200 constraints, while the set covering problems have 500 constraints.

$250$ problem instances are generated for evaluating the efficiency of the models. %In addition, as is done in \cite{gasse2019exact}, additional larger problems are generated in order to evaluate the models' ability to perform on more difficult problems. This includes $50$ medium-sized instances and $20$ large-sized instances. These problems are shown in \Cref{tab:samp_transf}.

\begin{scriptsize}
\begin{table}[ht]
	\centering
	\begin{tabular}{llccc}
		\toprule
		  && \multicolumn{1}{c}{Train} & \multicolumn{1}{c}{Validate} & \multicolumn{1}{c}{Test}\\
		  \cmidrule(lr){3-3} \cmidrule(lr){4-4} \cmidrule(lr){5-5}
		  \multirow{3}{*}{Auctions} & Dimensions & $100 \times 500$ & $100 \times 500$ & $100 \times 500$ \\
		% \midrule
		& Instances & 2000 &  500  & 250\\
		& Samples & 50000 &  10000  & 10000 \\
		\addlinespace
		\multirow{3}{*}{Setcover} & Dimensions & $500\times1000$ & $500\times1000$ & $500\times1000$\\
		% \midrule
		 & Instances & 2000 &  500  & 500 \\
		 & Samples & 50000 &  10000  & 50000 \\
		% \addlinespace
		\bottomrule
	\end{tabular}
	\caption{Data set, dimensions and number of problem instances for each problem class.}\label{tab:instances}
\end{table}
\end{scriptsize}

\begin{comment}
  \begin{scriptsize}
\begin{table}[ht]
	\centering
	\begin{tabular}{llccc}
		\toprule
		&& \multicolumn{1}{c}{Small} & \multicolumn{1}{c}{Medium} & \multicolumn{1}{c}{Large}\\
		  \cmidrule(lr){3-3} \cmidrule(lr){4-4} \cmidrule(lr){5-5}
		  \multirow{2}{*}{Auctions} & Dimensions & $100 \times 500$ & $200 \times 1000$ & $300 \times 1500$ \\
		% \midrule
		& Instances & 250 &  50  & 20\\
		\addlinespace
		\multirow{2}{*}{Setcover} & Dimensions & $500\times1000$ & $1000\times1000$ & $2000\times1000$\\
		% \midrule
		 & Instances & 250 &  50  & 20 \\
		% \addlinespace
		\bottomrule
	\end{tabular}
	\caption{Data set, dimensions and number of problem instances for each problem class.}\label{tab:samp_transf}
\end{table}
\end{scriptsize}

\end{comment}

\section{Training}\label{sec:res_training}

The training was performed as described in \Cref{sec:trainingprotocol}.
The training graph for MLP2 on the combinatorial auction data set is shown in \Cref{fig:training_mlp2}, and the training graph for GNN1 on the set covering data set is shown in \Cref{fig:training_gnn1}. The training and validation loss quickly converges before flattening out. There is no discernible difference between training and validation loss. Loss graphs for the other models are not presented, as they are very similar, and are therefore considered to be of little interest.
%

\begin{figure}[ht]
    \begin{tikzpicture}
      \begin{axis}[
        mlineplot,
        ylabel={Logloss},
        xlabel={Epoch},
        width=\textwidth,
        height=7cm,
	    ymin=2.95,   ymax=3.1,
	    xtick=data,
	    legend style={at={(0.01,0.9)},anchor=west},
	    cycle list name=black white,
	    minor xtick={1, 2, ..., 55},
        xtick={0, 5, ..., 55},
        ytick={2.9, 2.95, ..., 4.4},
      ]
        \addplot plot coordinates{(1, 4.095) (2, 3.315) (3, 3.015) (4, 3.008) (5, 3.003) (6, 3.0) (7, 2.995) (8, 2.991) (9, 2.987) (10, 2.985) (11, 2.984) (12, 2.982) (13, 2.982) (14, 2.981) (15, 2.98) (16, 2.98) (17, 2.979) (18, 2.979) (19, 2.979) (20, 2.978) (21, 2.978) (22, 2.978) (23, 2.978) (24, 2.977) (25, 2.977) (26, 2.977) (27, 2.976) (28, 2.976) (29, 2.975) (30, 2.976) (31, 2.976) (32, 2.975) (33, 2.975) (34, 2.974) (35, 2.975) (36, 2.974) (37, 2.974) (38, 2.974) (39, 2.974) (40, 2.974) (41, 2.973) (42, 2.973) (43, 2.973) (44, 2.973) (45, 2.974) (46, 2.973) (47, 2.97) (48, 2.97) (49, 2.969) (50, 2.969) (51, 2.969) (52, 2.969) (53, 2.969) (54, 2.969) (55, 2.969)};
        \addplot plot coordinates{(1, 4.095) (2, 3.031) (3, 3.019) (4, 3.011) (5, 3.011) (6, 3.005) (7, 3.002) (8, 3.004) (9, 3.009) (10, 3.002) (11, 2.995) (12, 3.002) (13, 2.997) (14, 2.996) (15, 2.997) (16, 2.996) (17, 2.998) (18, 2.995) (19, 2.996) (20, 2.996) (21, 2.993) (22, 2.996) (23, 2.997) (24, 2.994) (25, 2.994) (26, 2.996) (27, 2.996) (28, 2.991) (29, 2.995) (30, 2.999) (31, 2.995) (32, 2.995) (33, 2.994) (34, 2.995) (35, 2.99) (36, 2.991) (37, 2.995) (38, 2.991) (39, 2.994) (40, 2.992) (41, 2.994) (42, 2.993) (43, 2.991) (44, 2.991) (45, 2.992) (46, 2.991) (47, 2.991) (48, 2.991) (49, 2.991) (50, 2.992) (51, 2.99) (52, 2.992) (53, 2.991) (54, 2.99) (55, 2.994)};
        \legend{Training loss,Validation loss}
      \end{axis}
    \end{tikzpicture}
    \caption{Training graph for MLP2 on the combinatorial auctions data set.}
    \label{fig:training_mlp2}
\end{figure}

\begin{figure}[ht]
    \centering
    \begin{tikzpicture}
      \begin{axis}[
        mlineplot,
        ylabel={Logloss},
        xlabel={Epoch},
        width=\textwidth,
        height=7cm,
	    ymin=2.4,   ymax=3.4,
	    xtick=data,
	    legend style={at={(0.01,0.9)},anchor=west},
	    cycle list name=black white,
	    minor xtick={1, 2, ..., 62},
        xtick={0, 5, ..., 62},
        ytick={2.4, 2.6, ..., 4.4},
      ]
        \addplot plot coordinates{(1, 4.44) (2, 3.328) (3, 3.081) (4, 3.026) (5, 2.989) (6, 2.957) (7, 2.922) (8, 2.894) (9, 2.878) (10, 2.864) (11, 2.853) (12, 2.838) (13, 2.828) (14, 2.818) (15, 2.795) (16, 2.785) (17, 2.773) (18, 2.765) (19, 2.748) (20, 2.745) (21, 2.739) (22, 2.728) (23, 2.726) (24, 2.722) (25, 2.716) (26, 2.72) (27, 2.715) (28, 2.708) (29, 2.702) (30, 2.703) (31, 2.7) (32, 2.698) (33, 2.687) (34, 2.688) (35, 2.688) (36, 2.688) (37, 2.687) (38, 2.681) (39, 2.677) (40, 2.603) (41, 2.59) (42, 2.584) (43, 2.583) (44, 2.579) (45, 2.578) (46, 2.574) (47, 2.571) (48, 2.571) (49, 2.568) (50, 2.567) (51, 2.566) (52, 2.565) (53, 2.563) (54, 2.562) (55, 2.563) (56, 2.559) (57, 2.558) (58, 2.557) (59, 2.533) (60, 2.531) (61, 2.53)};
        \addplot plot coordinates{(1, 4.44) (2, 3.168) (3, 3.049) (4, 3.019) (5, 2.984) (6, 2.984) (7, 2.913) (8, 2.984) (9, 2.909) (10, 2.857) (11, 2.933) (12, 2.852) (13, 2.853) (14, 2.825) (15, 2.783) (16, 2.768) (17, 2.816) (18, 2.793) (19, 2.77) (20, 2.754) (21, 2.751) (22, 2.756) (23, 2.712) (24, 2.754) (25, 2.779) (26, 2.735) (27, 2.793) (28, 2.695) (29, 2.732) (30, 2.744) (31, 2.787) (32, 2.764) (33, 2.698) (34, 2.816) (35, 2.749) (36, 2.801) (37, 2.731) (38, 2.718) (39, 2.739) (40, 2.68) (41, 2.667) (42, 2.655) (43, 2.652) (44, 2.669) (45, 2.66) (46, 2.69) (47, 2.641) (48, 2.66) (49, 2.657) (50, 2.664) (51, 2.654) (52, 2.675) (53, 2.754) (54, 2.661) (55, 2.653) (56, 2.66) (57, 2.662) (58, 2.653) (59, 2.652) (60, 2.649) (61, 2.652) (62, 2.641)};
        \legend{Training loss,Validation loss}
      \end{axis}
    \end{tikzpicture}
    \caption{Training graph for GNN1 on the set covering data set.}
    \label{fig:training_gnn1}
\end{figure}

\section{Accuracy}\label{accuracy}\label{sec:res_accuracy}

\begin{scriptsize}
\begin{table}[ht]
	\centering
	\begin{tabular}{lrrrrrrr}
		\toprule
		& \multicolumn{3}{c}{Combinatorial Auction} & \multicolumn{3}{c}{Set Covering}\\
		\cmidrule(lr){2-4} \cmidrule(lr){5-7}
		Model & acc@1 & acc@5 & acc@10 & acc@1 & acc@5 & acc@10 \\
		\midrule
		random & 15.8 \% & 34.5 \% & 44.6 \% & 14.6 \% & 32.6 \% & 41.2 \%\\
		MLP1 & 46.2 \% & 75.1 \% & 87.0 \% & 40.2 \% & 63.6 \% & 73.3 \%\\
		MLP2 & 50.1 \% & 78.9 \% & 89.9 \% & 49.1 \% & 75.5 \% & 88.1 \%\\
		MLP3 & 50.2 \% & 79.3 \% & 90.1 \% & 49.4 \% & 75.7 \% & 88.3 \%\\
		GNN1 & 53.0 \% & 87.5 \% & 95.9 \% & 57.9 \% & 89.4 \% & 97.0 \%\\
		GNN2 & 53.0 \% & 87.1 \% & 96.0 \% & 58.8 \% & 89.9 \% & 97.2 \%\\
		\addlinespace
		\bottomrule
	\end{tabular}
	\caption{Top-k accuracy scores for combinatorial auctions and set covering on the test set, representing the priority of the selected variable by the strong branching score evaluation.}\label{tab:accs}
\end{table}
\end{scriptsize}

% CAUCTIONS
% MLP1 0.30329361 0.43040384 0.5133234  0.57414763 0.62793777 0.66972857
%0.70506455 0.7397385  0.76903343 0.79443893
%[0.50099305 0.61717974 0.69298246 0.7469381  0.78930818 0.82000993
%0.84640847 0.86701423 0.88339954 0.89920556]
% 0.50281364 0.61924859 0.69546508 0.75       0.79319762 0.82116849
% 0.84789805 0.8683383  0.8857994  0.90143992]
% [0.52962595 0.68884475 0.7744952  0.8326713  0.87520689 0.90292949
% 0.92303873 0.938762   0.95076134 0.95945051]
% [0.52962595 0.68636213 0.7744952  0.83325058 0.87065541 0.90052963
% 0.92171466 0.93776895 0.95092685 0.96027805]

% SETCOVER
% MLP1 [20.4 30.2 36.8 41.8 45.7 48.8 51.8 54.5 56.6 58.5] %
% MLP2 [0.4604 0.5681 0.6453 0.7026 0.7424 0.7804 0.8085 0.8315 0.853  0.8719]
% MLP3 [0.4616 0.5721 0.652  0.7054 0.7472 0.7834 0.8127 0.8363 0.8568 0.8744]
% GNN1 [0.5693 0.7178 0.7961 0.8507 0.8862 0.9108 0.9317 0.9486 0.9584 0.9682]
% GNN2 [0.5679 0.7139 0.7955 0.8477 0.8848 0.9101 0.9336 0.9496 0.961  0.9688]
The accuracy of the models is measured by the top-k accuracy, as specified in \Cref{ssec:benchmarking}.
The top-k accuracy scores for k equal to 1, 5, and 10 for the models along with the benchmark accuracy for random variable selection are shown in \Cref{tab:accs}. Top-k accuracy for this application is defined as the percentage of selected branching variables within the top k variables as determined by the strong branching evaluation.
% https://scikit-learn.org/dev/modules/model_evaluation.html#top-k-accuracy-score

Comparison with the random variable selection policy shows considerable improvement in favor of the trained models. Accuracy decreases with the ablations, although there is little difference between GNN2 and GNN1, and MLP3 and MLP2, respectively. MLP1 performs considerably poorer than the other models, indicating that a non-linear relationship between the variable features and the variable score is beneficial. There is also a noteworthy decrease in accuracy after removing the graph convolutions, indicating that the module aids in prediction.

A comparison of the differences between models of the two problem classes shows that the decrease in accuracy from GNN1 to MLP3 is around three times larger for the set covering problems. This indicates that the addition of different problem classes in the evaluation of branching policies is necessary.

Note that the GNN2 model should have similar accuracy to the corresponding model in Gasse et al. (2019) \cite{gasse2019exact}, however, the models have around $ 7 \%$ lower accuracy in this implementation. The cause for this is unknown, and the analysis of the models will not be affected by this.

\section{Efficiency}\label{sec:res_efficiency}

Eight branching policies were compared on the problem data set. The branching policies are full strong branching, pseudo-cost branching, reliability pseudo-cost branching, and the five learned models. The results for the combinatorial auctions problems are shown in \Cref{tab:results1_cauction} and for the set covering problems in \Cref{tab:results1_set}. \textit{Time} is the mean solution time, \textit{nodes} is the mean number of nodes in the solution graphs (calculated in accordance with the findings of Gamrath et al. (2018) \cite{gamrath2018measuring}), \textit{time/node} is the average time per node, calculated by dividing the two means, and \textit{parameters} is the number of (trainable) parameters of the model. The solution times and number of nodes are also provided with their respective standard deviations. For evaluating the time per node, it is worth noting that the average number of candidate variables may vary between methods. In other works relating to node and time measurements when comparing branching policies, a variant of the shifted geometric mean is used. This is not done in this thesis, see \Cref{cha:dists} for a short discussion on this.
The branching policy with the shortest mean solution time is marked bold for the \gls{GPU} and \gls{CPU} times.

The solution time and number of nodes for the classical branching policies is as expected: \gls{FSB} has the lowest number of nodes but the processing time per node is too high to be competitive, \gls{PC} has a competitive solution time as the branching variable selection is performed very quickly, while \gls{RPC} is a midpoint between the two algorithms.

For the learned models, the results show that the ablation from GNN2 to GNN1 does not particularly reduce the time per node, neither does the removal of hidden layers between the models MLP3 and MLP2. Time per node is considerably reduced after the graph convolutions are removed and reduced again when the model's hidden layers are removed. This is consistent over both the \gls{GPU} and \gls{CPU} variants.

\Cref{fig:cauctions_bar} and \Cref{fig:set_bar} shows the results for the results from \Cref{tab:results1_cauction} and \Cref{tab:results1_set}, respectively, including $95 \%$ confidence intervals calculated under the assumption that the solution times are normally distributed. See \cref{cha:dists} for a brief discussion on this assumption.

For the set covering problems, the learned models are less efficient than \gls{PC} and \gls{RPC}, which is inconsistent with the results from Gasse et al. (2020) \cite{gasse2019exact}. This will not be given too much attention, as this is irrelevant to the comparative analysis of the ablations.
Note also that the time per node is not equal between the two problem classes --- the average number of candidate variables per node will vary between problems.

% \usepackage{booktabs}
%time  25.89 \pm  19.78  | nb nodes      80 \pm   66 | time/node   321.9 ms | completed   250
% MLP2 CUDA: nb nodes     550 +- 1013 |  time  13.37 +-  20.60 | 100
\begin{scriptsize}
\begin{table}[ht]
	\centering
	\begin{tabular}{lrrrrrrr}
	    \toprule
		Model & Time [s] & Nodes  & Time/node [ms] & parameters \\
		\midrule
		FSB & $4.08 \pm 2.11$ & $8 \pm 8 $  & $462.6$ & --- \\
		PC  & $1.96 \pm 0.84$ & $355 \pm 358$ & $5.5$ & ---\\
		RPC & $2.73 \pm 1.18$ & $16 \pm 30$ & $ 171.1$ & ---\\
		\addlinespace
		MLP1\textsubscript{g} & $  1.58 \pm   0.46$ &  $ 146 \pm  149$ &  $  10.8$  & 19\\
		MLP2\textsubscript{g} & $  \mathbf{1.56 \pm   0.43}$ &  $ 119 \pm  119$ &  $  13.1$ & 1344\\
		MLP3\textsubscript{g} & $  1.57 \pm   0.44$ &  $ 118 \pm  115$ &  $  13.2$ & 9702\\
		GNN1\textsubscript{g} & $  1.58 \pm   0.42$ &  $  92 \pm   78$ &  $  17.2$ & 52083\\
		GNN2\textsubscript{g} & $  1.59 \pm   0.43$ &  $  91 \pm   80$ &  $  17.5$ & 64562\\
		\addlinespace
		MLP1\textsubscript{c} & $  3.14 \pm   1.53$ &  $ 147 \pm  148$ &  $  21.2$  & 19\\
		MLP2\textsubscript{c} & $  3.07 \pm   1.35$ &  $ 120 \pm  118$ &  $  25.4$ & 1344\\
		MLP3\textsubscript{c} & $  3.08 \pm   1.32$ &  $ 118 \pm  112$ &  $  26.0$ & 9702\\
		GNN1\textsubscript{c} & $  5.34 \pm   3.42$ &  $  97 \pm   86$ &  $  54.8$ & 52083\\
		GNN2\textsubscript{c} & $  5.61 \pm   3.65$ &  $  94 \pm   84$ &  $  59.3$ & 64562\\
		\bottomrule
	\end{tabular}
	\caption{Combinatorial auction solving time for classical and learned methods. Subscripts denote whether the model is run on the \gls{GPU} or \gls{CPU}.}\label{tab:results1_cauction}
\end{table}

\begin{table}[ht]
	\centering
	\begin{tabular}{lrrrrrrr}
		\toprule
		Model & Time [s] & Nodes  & Time/node [ms] & parameters \\
		\midrule
		FSB & $26.23 \pm  33.79$ & $31 \pm 45$ & $845.6$ & ---\\
		PC  & $7.78 \pm 9.50$ & $749 \pm 1634$ & $10.4$ & ---\\
		RPC & $10.68 \pm 8.01$ & $295 \pm 895$ & $36.2$ & ---\\
		\addlinespace
		MLP1\textsubscript{g} & $ 16.28 \pm  24.29$ &  $ 621 \pm 1097$ &  $  26.2$ & 19\\
		MLP2\textsubscript{g} & $ 15.75 \pm  21.89$ &  $ 531 \pm  917$ &  $  29.6$ & 1344\\
		MLP3\textsubscript{g} & $ 15.83 \pm  22.25$ &  $ 523 \pm  899$ &  $  30.3$ & 9702\\
		GNN1\textsubscript{g} & $ 11.95 \pm  12.94$ &  $ 268 \pm  384$ &  $  44.6$  & 52083\\
		GNN2\textsubscript{g} & $ \mathbf{11.71 \pm  12.92}$ &  $ 266 \pm  382$ &  $  44.0$ & 64562\\
		\addlinespace
		MLP1\textsubscript{c} & $ 17.57 \pm  20.96$ &  $ 609 \pm 1019$ &  $  28.8$ & 19\\
		MLP2\textsubscript{c} & $ 17.39 \pm  21.31$ &  $ 535 \pm  929$ &  $  32.5$ & 1344\\
		MLP3\textsubscript{c} & $ 17.38 \pm  21.35$ &  $ 517 \pm  889$ &  $  33.6$ & 9702\\
		GNN1\textsubscript{c} & $ 60.06 \pm  81.62$ &  $ 265 \pm  382$ &  $ 226.1$ & 52083\\
		GNN2\textsubscript{c} & $ 63.11 \pm  87.58$ &  $ 268 \pm  392$ &  $ 235.1$ & 64562\\
		\bottomrule
	\end{tabular}
	\caption{Set covering solving time for classical and learned methods. Subscripts denote whether the model is run on the \gls{GPU} or \gls{CPU}.}\label{tab:results1_set}
\end{table}
\end{scriptsize}
%time  26.23 \pm  33.79  | nb nodes      31 \pm   45 | time/node   845.6 ms
%[05-10 21:02:43.02] Device:  cpu
%[05-10 21:02:43.03] ------ setcover, small ------
%[05-10 21:47:49.72]  nb nodes    1303 +- 2308 |  time  23.85 +-  34.68 | completed   250
%[05-10 21:47:49.72] End of evaluation.
%(ecole) �� ~/git/branch2learn python branch2learn/04_evaluate.py -m mlp1 -g 0
%[05-10 22:42:42.31] Model:   mlp1
%[05-10 22:42:42.31] Device:  cuda
%[05-10 22:42:42.33] ------ setcover, small ------
%[05-10 23:27:49.03]  nb nodes    1322 +- 2346 |  time  20.62 +-  33.04 | completed   250
% 04-27 20:24:04.15] Policy:   fsb
%[04-27 22:08:43.84]  nb nodes      77 +-   64 |  time  25.02 +-  19.25 | completed   250
\begin{figure}[ht]
    \centering
    \begin{tikzpicture}
    \begin{axis}[
        width=\linewidth,
        axis x line*=bottom,
        axis y line*=left,
    	ylabel={Solution time [s]},
    	xlabel={Branching policy/model},
    	ymajorgrids=true,
    	ybar=5pt,% configures `bar shift'
    	bar width=9pt,
        xticklabels={SB,PC,RPC,MLP1,MLP2,MLP3,%
		GNN1,GNN2},
		xtick=data,
		%enlarge x limits=0.15,
        %enlarge y limits=0.05,
        %restrict y to domain=0:6
        ymin=0.0,   ymax=6.0,
	    legend style={at={(0.01,0.9)},anchor=west,legend columns=-1}
    ]
    \addplot[color=black,fill=black!35,error bars/.cd,y dir=both,y explicit,]
    coordinates{
       (1,4.08) +-(0.0,0.262)
       (2,1.96) +-(0.0,0.1043)
       (3,2.73) +-(0.0,0.146)
       (4,3.14) +- (0.0, 0.19)
       (5,3.07) +- (0.0, 0.17)
       (6,3.08) +- (0.0, 0.16)
       (7,5.34) +- (0.0, 0.43)
       (8,5.61) +- (0.0, 0.46)
    };
    %negativ
    \addplot[color=black,fill=black!10,error bars/.cd,y dir=both,y explicit,]
    coordinates {
      %(1,0.9365) +- (0.00587,0.00587)
      %(2,0.1435) +- (0.01737,0.01737)
      (4,1.58) +- (0.0, 0.06)
      (5,1.56) +- (0.0, 0.05)
      (6,1.57) +- (0.0, 0.05)
      (7,1.58) +- (0.0, 0.05)
      (8,1.59) +- (0.0, 0.05)
};
    \legend{CPU,GPU}
    \end{axis}
    \end{tikzpicture}
    \caption{Combinatorial Auction test problem mean solution time when run on the GPU and CPU.}
    \label{fig:cauctions_bar}
\end{figure}

\begin{figure}[ht]
    \centering
    \begin{tikzpicture}
    \begin{axis}[
        width=\linewidth,
        axis x line*=bottom,
        axis y line*=left,
    	ylabel={Solution time [s]},
    	xlabel={Branching policy/model},
    	ymajorgrids=true,
    	ybar=5pt,% configures `bar shift'
    	bar width=9pt,
        xticklabels={SB,PC,RPC,MLP1,MLP2,MLP3,%
		GNN1,GNN2},
		xtick=data,
		%enlarge x limits=0.15,
        %enlarge y limits=0.05,
        %restrict y to domain=0:6
        ymin=0.0,   ymax=80.0,
	    legend style={at={(0.01,0.9)},anchor=west,legend columns=-1}
    ]
    \addplot[color=black,fill=black!35,error bars/.cd,y dir=both,y explicit,]
    coordinates{
       (1,26.23) +-(0.0, 4.19)
       (2,7.78) +-(0.0, 1.18)
       (3,10.68) +-(0.0,0.993)
       (4,17.57) +- (0.0, 2.62)
       (5,17.39) +- (0.0, 2.66)
       (6,17.38) +- (0.0, 2.66)
       (7,60.06) +- (0.0, 10.19)
       (8,63.11) +- (0.0, 10.93)
    };
    %negativ
    \addplot[color=black,fill=black!10,error bars/.cd,y dir=both,y explicit,]
    coordinates {
      %(1,0.9365) +- (0.00587,0.00587)
      %(2,0.1435) +- (0.01737,0.01737)
      (4,16.28) +- (0.0, 3.03)
      (5,15.75) +- (0.0, 2.73)
      (6,15.83) +- (0.0, 2.78)
      (7,11.95) +- (0.0, 1.61)
      (8,11.71) +- (0.0, 1.61)
};
    \legend{CPU,GPU}
    \end{axis}
    \end{tikzpicture}
    \caption{Set covering test problem mean solution time when run on the GPU and CPU.}
    \label{fig:set_bar}
\end{figure}

\begin{comment}

\begin{figure}
    \centering
    \begin{tikzpicture}

      \def\MarkSize{.75em}
      \protected\def\ToWest#1{%
        \llap{#1\kern\MarkSize}\phantom{#1}%
      }
      \protected\def\ToSouth#1{%
        \sbox0{#1}%
        \smash{%
          \rlap{%
            \kern-.5\dimexpr\wd0 + \MarkSize\relax
            \lower\dimexpr.375em+\ht0\relax\copy0 %
          }%
        }%
        \hphantom{#1}%
      }
    \begin{axis}[ xlabel={CPU time}, ylabel={GPU time}, width=10cm, height=10cm, ymin=1.0, ymax=5.5, xmin=1.0,   xmax=5.5, ymajorgrids=true,
    xmajorgrids=true,]v
    \addplot[scatter,mark=*,only marks, point meta=x,nodes near coords*={\data},
    visualization depends on={value \thisrow{dataname} \as \data},]
    table [x=x,y=y]{
    x       y       dataname
    1.84    1.78    MLP1
    1.60    3.12    \ToSouth{MLP2}
    1.62    3.24    MLP3
    1.65    5.06    \ToSouth{GNN1}
    1.63    5.10    GNN2
    };
    \end{axis}
    \end{tikzpicture}
    \caption{Combinatorial Auction small problem mean solution time GPU and CPU.}
    \label{fig:gpu_cpu}
\end{figure}
\end{comment}

%In Gasse et al. (2019) \cite{gasse2019exact} and Gupta et al. (2020) \cite{gupta2020hybrid}, the models are also evaluated by their performance on larger problem sizes, as a method of measuring the models' ability to generalize.
%This is a time-consuming process, and therefore only a select number of models is used: MLP1, MLP2, and GNN1, as these represent the variations that are shown in \Cref{fig:cauctions_bar} and \Cref{fig:set_bar}.

%The result is shown in \Cref{tab:results_trans_cauction}, where \textit{time} and \textit{nodes} is measured as in \Cref{tab:results1_cauction} and \Cref{tab:results1_set}.
%\textit{completed} is the number of problems where optimality was achieved within the time limit of 45 minutes and is only applicable for the large problem size.

%GNN1 and MLP2 perform competitively on the \gls{GPU}, while the \gls{MLP} suffers from the consequences of poor variable branching choices, leading to very poor performance.

% including transfer learning

\begin{comment}

\begin{scriptsize}
\begin{table}[ht]
	\centering
	\begin{tabular}{lrrrrr}
	    \toprule
		&  \multicolumn{2}{c}{Medium} & \multicolumn{3}{c}{Large}\\ \cmidrule(lr){2-3} \cmidrule(lr){4-6}
		Model & Time (s) & Nodes & Time (s) & Nodes & Completed\\
		\midrule
		FSB &  $158.03$ & $139 $ & 0 & 0 & 0 / 20\\
		PC  &  $26.40 $ & $5429 $ & $410.24 $ & $54923 $ & 20 / 20\\
		RPC &  $20.60 $ & $1692 $ & $\mathbf{214.30} $ & $19069 $ & 20 / 20\\
		\addlinespace
		MLP1\textsubscript{g}  & $260.18$ & $73895$ & $2490.55$ & $424168$ & 6 / 20\\
		MLP2\textsubscript{g} &  $19.20$ & $2875$ & $470.39$ & $53314$ & 20 / 20\\
		GNN1\textsubscript{g} & $\mathbf{15.76}$ & $1628$ & $273.89$ & $25879$ & 20 / 20\\
		\addlinespace
		MLP1\textsubscript{c} &  $667.74 $ & $74891 $ & $9556.22 $ & $463156 $ & 1 / 20\\
		MLP2\textsubscript{c} &  $60.87 $ & $2871$ & $1791.31 $ & $53933 $ & 15 / 20\\
		GNN1\textsubscript{c} & $118.54 $ & $1701 $ & $2197.60 $ & $26470 $ & 14 / 20\\
		\bottomrule
	\end{tabular}
	\caption{Combinatorial auction solving efficiency on larger problem sets.}\label{tab:results_trans_cauction}
\end{table}
\end{scriptsize}
\end{comment}
% time 158.03 \pm 227.58  | nb nodes     139 \pm  187
%MLP1 cpu
%04-24 11:37:37.76] ------ cauctions, large ------
%[04-25 02:04:00.40]  nb nodes     +- 93142 |  time  +- | completed    1
%
% include feature importance

% include solution hists nnodes