\chapter{Experimental design and Data exploration}
\label{ch:ExpDesign}

Ideally, you would like to design experiments (manipulations and/or
observations) that are appropriate for the question you want to answer.
However, you still need to explore you data to determine what kind of
statistical tests would be appropriate because: (a) Your experiments or
observations may not go as planned, and (b) You might have somebody
else's data to analyse (very likely in your UG projects). Building on
the previous chapter, in this chapter we you learn how to use R to
explore your data and determine appropriate statistical tests. By the
time you have worked through this chapter, you should be able to

\begin{compactitem}
	\item Provided sufficient information is available, be able to judge
	whether the sampling design used to generate a particular dataset was
	appropriate
	\item Determine if your sample sizes are adequate, especially for a
	specific statistical test qualitatively (Y1) or quantitatively using
	power analysis (Y2).
	\item Calculate basic statistical measures on your data to determine
	its properties
\end{compactitem}

We are going to start off in Y1 with the simplest of scenarios for
statistical testing --- that you want to determine whether a sample, or
a pair of samples meet some expectation (hypothesis) or not.

First, some conceptual preliminaries.

\section{Some statistical parlance}
The following terms are important for you to get familiar with:
\begin{description}

		\item[(Statistical) Population] A statistical population is a {\it
		complete set} of items that share at least one {\it attribute} of
		interest. This attribute of interest is the target of your
		statistical analysis. For example, if we are interested in studying
		the weight of year-old cod in the Oceans, the population
		consists of {\it all} year-old cod, but more specifically, the
		weight measurements of all the individuals of the cod population is
		what we want to analyse.

		\item[(Statistical) Distribution] A statistical distribution is a
		mathematical description (expressed as a mathematical equation) of
		the properties of a population of interest. Theoreticians have come
		up with a bunch of distributions (e.g., Gaussian or Normal,
		Poisson, Binomial, etc.) that are appropriate for different kinds
		of data. Figuring out which distribution best describes a
		population of interest is one of the first steps in a statistical
		analysis. In reality, of course, even collecting and measuring all
		the individuals of a population may not be sufficient to
		characterize its statistical properties --- imagine the situation
		where the cod population has declined to a few hundred individuals
		(not an impossibility in the future!).

		\item[(Data or Population) Sample] A data {\it sample} is a set of
		measurements of the attribute of interest collected from a {\it
		statistical population} by a defined procedure ({\it sampling
		methodology}). In the cod example above, this would be the weight
		of every individual of a {\it subset} of the year-old Cod
		population.

		\item[(Statistical) Parameter]  A statistical parameter is a
		measure of some attribute of the {\it theoretical} statistical
		distribution that is support to represent your population. An
		example would be the average weight of yearling cod. In
		practice, this is not measurable because the population is much too
		large or incompletely inaccessible/invisible --- imagine measuring
		the weight of every year-old cod individual in the oceans!

		\item[Statistic] A statistic (singular) is an {\it estimate} of a
		statistical parameter of the population of interest, obtained by
		calculating the {\it measure} for a {\it sample} (e.g.,  the
		average or mean weight of individuals in a sample of one-year old
		cod). This is also your {\it descriptive statistic}. Therefore, a
		{\it Statistic} is to a {\it Statistical Parameter} what a {\it
		Sample} is to the {\it Statistical Population}. For example, the
		average of a sample of cod weights is a statistic that {\it
		estimates} the theoretical ``real'' average of the weights of the
		entire one-year Cod population, which is its statistical parameter.

		\item[Hypothesis] A Hypothesis is an (hopefully) informed {\it
		postulate} about an attribute of your population of interest. For
		example, you may hypothesize that the one-year old cod population's
		mean weight has declined over the last two decades (it has!). You
		will want to confront your main hypothesis with a ``null''
		Hypothesis, to minimize the risk of making a ``type I'' error. A
		type I error is the probability of accepting an alternative (or
		main) hypothesis (and rejecting the null hypothesis) that is not
		really valid (e.g., the yearling cods have actually not declined in
		weight). This is a big NO NO from a scientific and philosophical
		standpoint. The rate or probability of the type I error is denoted
		by the Greek letter $\alpha$, and equals the {\it significance
		level} of a statistical test.

\end{description}

\section{Descriptive Statistics}

The fundamental statistics that describe a sample (or a population)
are, firstly, the mean, or average value of a sample, typically denoted
by a $\bar{x}$:

\begin{equation}
	\bar{x} =  \frac{\sum\limits_{i=1}^n x_i}{n} = \frac{x_{1} + x_{2} + \dots +x_{n}}{n}
\end{equation}

That is, it is the sum of all the values in a sample divided by the
number, $n$, of items in the sample. Thus it is a measure of the {\it
central-tendency} of the sample and population.

Second, the standard deviation ($s$) is:

\begin{equation}
	s =  \sqrt{\frac{\sum\limits_{i=1}^n (\bar{x} - x_{i})^2}{n-1}} =
	\sqrt{\frac{(\bar{x} - x_{1})^{2} + (\bar{x} - x_{2})^2 + \dots + (\bar{x}
	- x_{n})^{2}}{n-1}}
\end{equation}

That is, the the square root of the sum of squares ("SS") of the
differences between each item in the sample and the mean, divided by
the {\it degrees of freedom, "df"} remaining in the data set ($
n-1$). df is the sample size, $n$, minus the number of statistical
parameters estimated from the data set. This is to reduce the {\it
bias} in your {\it estimate} of the statistic, as you are calculating
it from the sample, and not the whole theoretical population.

Thus, the formula for $s$ above has $n-1$ in its denominator because,
to to work out the standard deviation, you must have already estimated
the mean ($\bar{x}$) from the same data set. This removes 1 degree of
freedom. Also, note that the sample variance, $s^2$ is the square of
standard deviation. Or, in other words, the standard deviation is the
square-root of the variance!

The above two statistics (mean and sd) are particularly meaningful when
the sample and population have a symmetric distribution (e.g., normal
or gaussian). When the distribution is not symmetric (that is, it is
{\it skewed}), another statistic, the {\it median} becomes important.
This is the middle value in the ordered set of data, that is exactly
50\% of the data live below and 50\% lie above the median. In skewed
distributions, the median is a better measure of the
{\it central-tendency} of the sample and population.

Other descriptive statistics you should keep in mind are the range
(difference between the largest and smallest values), and the quartiles
(values lying in the data divided into the intervals $[{1\over4}, {1\over
2},{3\over 4}, 1]$ or at 1\% intervals (percentiles). Box-plots, which
you have seen, represent a number of these statistics in one figure.

\subsection{Descriptive statistic functions in R}

\begin{tabular}{p{3.5cm} p{9.5cm}}
	{\tt mean(x)} & Compute mean (of a vector or matrix)\\
	{\tt sd(x)} & Standard deviation\\
	{\tt var(x)} & Variance\\
	{\tt median(x)} & Median\\
	{\tt quantile(x,0.05)} & Compute the 0.05 quantile\\
	{\tt range(x)} & Range of the data\\
	{\tt min(x)} & Minimum\\
	{\tt max(x)} & Maximum\\
	{\tt sum(x)} & Sum all elements\\
\end{tabular}\\

\section{Data types and distributions}
You will typically encounter or sample the following main types of
data:

\begin{description}

		\item[Continuous numeric] This is the {\tt  numeric} or {\tt real}
		data type in R, and as far as you are concerned, these data
		typically will be made up of (mathematically) real numbers such as
		human height or weight. These may be unbounded (any value between
		negative infinity to positive infinity), or bounded (e.g., between
		or zero and some upper positive number) like human weight.

    \item[Discrete numeric] This is the {\tt  integer} data type in R,
    and consist of (mathematically) integer (whole) numbers such as
    counts of individuals in a population, e.g., The number of bacteria
    in a ml of pond water.

		\item[Percentage (proportion)] Percentage data is a particular kind
		of numeric data that is strictly bounded between 0 and 100. The
		fact that you can never get samples of percentages that exceed
		these bounds makes such data tricky to analyse.

    \item [Categorical] These are typically stored as the {\tt
		 character} data type in R. Categorical data are discrete,
		 typically expressed as a one of a fixed number of {\it levels} of a
		 {\it factor}. For example, the the factor "Type.of.feeding.interaction"
		 from the predator-prey dataset you have seen previously had five
		 levels: "insectivorous", "piscivorous", "planktivorous",
		 "predacious", and "predacious/piscivorous".

     \item [Binary (presence/absence) data] A special type of
     categorical data are binary, where only two categories or states
     are possible: (1, 0) (or "present", "absent"), e.g., a disease
     symptom. These may be stored as {\tt  integer} or {\tt
     character} in R.
\end{description}

While designing experiments or exploring data obtained by somebody
else, you need to keep in mind that each type will typically be best
represented a particular {\it statistical distribution}. For example,
continuous numeric data are {\it often} normally distributed. On the
other hand, count data are likely to be distributed according to the
Poisson distribution.

If you are lucky, you will mostly have to deal with data that are
continuous or discrete numeric, which are the most straightforward to
analyse using Linear models (more on that in subsequent chapters).
However, some of the most interesting and important problems in biology
involve proportion (percentage), categorical and binary data (e.g.,
Presence or absence of a disease symptom).

For example, think about what type of data, and what type of
distribution, a sample of the following is likely to be:
\begin{compactitem}
    \item Wavelength of light
    \item Temperature
    \item Egg clutch size
    \item Rate of a reaction
    \item Eye-colour
    \item Score in Scrabble
    \item UG Degree class
    \item Ground-cover of grass in a quadrat
    \item Winning side in chess
\end{compactitem}

\subsection{Sampling from distributions in R}

You can generate samples form many distributions in R (and handy thing
to know). In particular, the following are important:

\begin{tabular}{p{5.4cm} p{9cm}}
    {\tt rnorm(10, m=0, sd=1)} & Draw 10 normal random numbers with mean 0 and s.d. 1\\
    {\tt dnorm(x, m=0, sd=1)} & Density function\\
    {\tt qnorm(x, m=0, sd=1)} & Cumulative density function\\
    {\tt runif(20, min=0, max=2)} & Twenty random numbers from uniform
        [0,2]\\
    {\tt rpois(20, lambda=10)} & Twenty random numbers from
        Poisson($\lambda$)\\
\end{tabular}\\

\section{Two basic rules of experimental design and sampling}

In general, while designing experiments, and sampling from a {\it
population}, there are two key (and simple) rules:
\begin{enumerate}
    \item {\bf The more you sample, the more your sample's distribution
    will look like the population distribution} (obviously!)
    \item {\bf The more you sample, the closer will your sample
    statistic be to the population's statistical parameter} (the
    central limit theorem)
\end{enumerate}

Let's have a quick look at rule 1 using R (open R and {\tt setwd} to
{\tt Code}):
\begin{lstlisting}
# Draw 5 normal random nos w/ mean 0 and s.d. 1:
> MySample5 <- rnorm(5, m=0, sd=1)
> MySample10 <- rnorm(10, m=0, sd=1)
> MySample20 <- rnorm(20, m=0, sd=1)
> MySample40 <- rnorm(40, m=0, sd=1)
> MySample80 <- rnorm(80, m=0, sd=1)
> MySample160 <- rnorm(160, m=0, sd=1)
\end{lstlisting}
Now let's visualize these "samples":
\begin{lstlisting}
> par(mfcol = c(2,3)) #initialize multi-paneled plot
> par(mfg = c(1,1)); hist(MySample5, col = rgb(1,1,0), main = 'n = 5')
> par(mfg = c(1,2)); hist(MySample10, col = rgb(1,1,0), main = 'n = 10')
> par(mfg = c(1,3)); hist(MySample20, col = rgb(1,1,0), main = 'n = 20')
> par(mfg = c(2,1)); hist(MySample40, col = rgb(1,1,0), main = 'n = 40')
> par(mfg = c(2,2)); hist(MySample80, col = rgb(1,1,0), main = 'n = 80')
> par(mfg = c(2,3)); hist(MySample160, col = rgb(1,1,0), main = 'n = 160')
\end{lstlisting}
The second rule above states that if I was to repeat even $n
= 5$ sufficient number of times, I would get a good {\it estimate} of
mean (= 0) and standard deviation (= 1) of the normal distribution we
sampled from.

\section{A data exploration case study}

As a case study, we will use data from a paper looking at the
relationship between genome size and body size across species of
dragonflies and damselflies
(\href{http://en.wikipedia.org/wiki/Odonata}{Odonata}):

\begin{quote}
Ardila-Garcia, AM \& Gregory, TR (2009) `An exploration of genome size
diversity in dragonflies and damselflies (Insecta: Odonata)' Journal of
Zoology, 278, 163 - 173
\end{quote}

You will work with the script file {\tt  ExpDesign.R}, which performs
exploratory analyses on the data in {\tt GenomeSize.csv}. Let's go
through the code block by block.

\begin{compactitem}[$\quad\star$]
    \item Get the script {\tt ExpDesign.R} from the Bitbucket
    repository and put it in your own {\tt Code} directory.
		\item Also get {\tt GenomeSize.csv}
		\item Open the script {\tt ExpDesign.R} in RStudio (or some other
		text editor).
		\item Use the shift and arrow keys to select the code in block (2),
		including the comments. Now use the keyboard short cut (look back
		at the R Chapters if you don't know how!) to run the highlighted block
		of code.
\end{compactitem}

This first line (block (1)) reads in the data, as you have learned previously.
\begin{compactitem}[$\quad\star$]
    \item Now the code in block (2) line by line
    of code.
\end{compactitem}
Have a good look at the data. There are three factors
(categorical variables): Suborder, splitting the species into
dragonflies (Anisoptera) and damselflies (Zygoptera); Family, splitting
the species further into 9 taxonomic families; and Species, giving the
latin binomial for each species in the table. The remaining columns are
measurements of genome size (in picograms) and measurements of body
size and morphology (in grams, mm and mm$^2$). There are two columns
ending with an N that show the sample size from which the observations
for each species are taken and a column ending SE showing standard
errors.

One thing you should see in the output from {\tt head} or {\tt str} is
that there are some observations marked as {\tt NA} -- this is the way R
shows {\it missing data}. It is important to check how much missing
data there is an dataset, so we'll use another function that includes
this information. Many R functions refuse to use variables containing
missing data --- this is just R being careful and you can add {\tt
na.rm=TRUE} into most functions to avoid this problem.

\begin{compactitem}[$\quad\star$]
\item Run the {\tt summary} line from the script window (block 3).
\end{compactitem}
Look at the output. There is a column for each variable: for
factors, it provides a short table of the number of observations in
each level and for continuous variables, it provides some simple
summary statistics about the distribution (range,  quartiles, mean and
median), and the number of missing values

\subsection{Visualise distributions of the variables}
The {\tt summary} function shows us the basic distribution (range,
quartiles, mean and median) of a continuous variable, but this is
easier to interpret if we visualise it. We'll look at two ways:
\begin{description}
    \item [Histogram] In the simplest form, this shows the number of
    observations of the variable falling into a set of bins spanning
    the range of the variable. The option {\tt breaks} allows you to
    change the number of bins.
    \item [Density plot] Rather than showing blocks of counts, the
    density plot shows a continuous smooth line. This is a {\it
    smoothed} estimate of the how frequently data is observed across
    the range of values and the {\it bandwidth} ({\tt bw=0.1}) controls
    the degree of the smoothing.
\end{description}

\begin{compactitem}[$\quad\star$]
\item Go to block (4) of the script and run each line separately,
looking at the output.
\item In the editor, change the values of {\tt breaks} and {\tt bw} ---
for example {\tt breaks=5} and {\tt bw=0.05} --- and re-run these lines
to see how this affects the graph. Basically, with both types of graph
you can look at the data too coarsely or too finely.
\item The graphs you've just created look at genome size. Add a copy of
those two lines of code in the script and change them to look at the
variable {\tt TotalLength}. You will need to alter the {\tt density}
function to ignore missing values ({\tt na.rm=TRUE}) and to play around
with the bandwidth. You should get something like this:
\end{compactitem}
%
\begin{center}
\includegraphics[width=0.45\textwidth]{histTL1}
\includegraphics[width=0.45\textwidth]{histTL2}
\end{center}

\subsection{Take a quick look at effects of certain factors}
R has a special way of describing a model that defines the response
variable and the explanatory variables ("factors"). This is called a
'formula' and is used to define linear models (more on these in a later
chapters). The same structure is used in many plotting functions and
will put the response variable on the $y$ axis and the explanatory
variable on the $x$ axis. The structure is "response variable $\sim$
explanatory variables". We will look at multiple explanatory variables
in a later practical but an example with one explantory variable
(factor) is:

	{\tt Genome Size $\sim$ Suborder}

This formula tells R to model genome size `as a function of'
($\sim$) the suborders of Odonata. In a plot function, the
result will be to plot genome size as a function of the suborders.

\subsection{Compare distribution of the variable across levels of a
factor}

Although looking at the distribution of variables is a good first step,
we often want to compare distributions. In this case, we might want to
know how genome size varies between dragonflies and damselflies. The
first way we will look at is using boxplots --- these show the median
and the 25\% and 75\% quantiles as a box, with whiskers extending to
the minimum and maximum. More extreme outliers are plotted
independently as points. The {\tt plot} function in R automatically
generates a boxplot when the explanatory variable is a factor.

\begin{compactitem}[$\quad\star$]
\item Go to block 5 of the script and run the first line, looking at
genome size between the two suborders.
\item Duplicate and alter this line to look at the same plot for total
length. You should get a plot like this:
\end{compactitem}

\begin{center}
    \includegraphics[width=0.55\textwidth]{bxpTL}
\end{center}

Although histograms are great for one variable, plotting two histograms
on top of one another rarely works well because the overlapping bars
are hard to interpret (recall the predator-prey body size example).
Density plots don't have this problem but it takes a bit more code to
create the plot.

\begin{compactitem}[$\quad\star$]
\item block 6 of the script uses the {\tt subset} function to create
two new data frames separating the data for dragonflies and
damselflies. Run the first two lines of this block. Remember that the
arrow symbol ({\tt <-}) is used to save the output of a function into a
new object in R --- if you use {\tt ls()} in the console, you will see
the two new data frames.
\item In the console, use {\tt str} and {\tt summary} to explore these
two new dataframes.
\end{compactitem}
Now that we've got the data separated we can go about plotting the two
curves.
\begin{compactitem}[$\quad\star$]
\item Run the next two lines of code in block 6. The first draws the
plot for damselflies and the second adds a line for the dragonflies.
\item Duplicate these last two lines of code and edit them to generate
a similar plot for total body length. You will need to edit the code to
change the range of the $x$ and $y$ axes ({\tt xlim} and {\tt ylim}) to
get both curves to fit neatly on to the graph. It should look like
this:
\end{compactitem}

\begin{center}
    \includegraphics[width=0.45\textwidth]{densTL}
\end{center}

\subsection{Explore further by scatter-plotting two variables}

Once we've looked at the distribution of variables, the next thing is
to look at the relationships between continuous variables using
scatterplots. The {\tt plot} function in R automatically generates a
scatterplot when the explanatory variable is continuous, so we can use
the same syntax and structure as for the boxplot above.

\begin{compactitem}[$\quad\star$]
\item Go to block (7) of the script and run the first plot command to
plot body weight as a function of genome size.
\item Create a new line of code to plot forewing area as a function of
genome size. It should look like this:
\end{compactitem}

\begin{center}
\includegraphics[width=0.5\textwidth]{scatWing}
\end{center}
The scatterplot seems to show a weak relationship between
genome size and morphology. But maybe dragonflies and damselflies show
different relationships, and we can't distinguish between them! To
explore this possibility, we need to plot the two orders using
different colours or plot characters. In the next code block, we want
customize the plots to show different types of points for each
suborder. It is done by using {\it indexing} (covered in the first R Chapter in SilBioComp.pdf).

\begin{compactitem}[$\quad\star$]
\item Run the first two lines of code in block 8. There are two levels
of suborder and these two lines set up a colour and a plot symbol that
will be used for each one.
\item Run the next line which shows the structure of the factor {\tt
Suborder}. You can see that there are two levels, with Anisoptera first
and then Zygoptera. You can also see that these are stored as numeric
values: 1 refers to the first level and 2 will refer to the second. We
can use these as {\it indices} to pair the colours and plot symbols to
each suborder. These are set in the {\tt plot} function using the
options {\tt col=} and {\tt pch=}, which stands for "plot character".
\item Run the next plot command  to see the resulting plot --- each
point gets the appropriate colour and symbol for its group.
\end{compactitem}

There are a lot of built in colours and plot symbols in R, so the next
thing to experiment with is changing these to your own versions.

\begin{compactitem}[$\quad\star$]
\item In the console, type in the function {\tt colors()}. You'll see a
long list of options to choose from, so pick two to replace red and
blue in the script window.
\item The options for the plot symbols are shown below. Pick two to
replace the current symbol choices.
\item Rerun the {\tt plot} function and see what you get!
\end{compactitem}

\begin{center}
    \includegraphics[width=0.55\textwidth]{pch}
\end{center}

\subsection{Saving the exploratory graphics}
The file `GenomeSize.pdf' in the practical folder was created using the
next block of code using the approach you learned previously.
The function {\tt pdf} opens a new empty pdf file  which can then be
used to plot graphs. You can set the width and the height of the page
size in the pdf but note that this is set in {\it inches}. When you
have finished creating a plot, the function {\tt dev.off} closes the
pdf file and makes it readable.

\begin{compactitem}[$\quad\star$]
\item Open 'GenomeSize.pdf' in a PDF reader. It uses the original
colours and plot symbols. Close the file and then delete it from the
folder.
\item Now go back to the script in R and select and run all the code in
block (9)
\item Go back to the {\tt Results} folder. The pdf file should have
been recreated --- open it and it should now use your choice of colours
and symbols.
\end{compactitem}

\subsection{Saving data}

One last thing you can do is to save the data and variables in R format
--- the original data, two subsets of the data and the two sets of
colours and symbols. We can recreate the subsets easily, so we'll just
save the data and your colour sets.

\begin{compactitem}[$\quad\star$]
\item Go to the script window and run the final line in block (10)
\item Still in the script window, choose `File $\triangleright$ Save'
to save your changes to the script file.
\item Quit from R {\tt q()}.
\end{compactitem}


\chapter{Methods}\label{cha:methods}

In this chapter, the selected methods for the experiments are presented and discussed.
An understanding of the theoretical groundwork of the project from \Cref{cha:background} is assumed. A substantial part of the methods are taken directly or indirectly from the source code of Gasse et al. (2019) \cite{gasse2019exact}, Gupta et al. (2020) \cite{gupta2020hybrid}, and the \gls{Ecole} source code \cite{prouvost2020ecole}. Some sentences and formulations are adapted from the project report \textit{Multi-Layer Perceptrons for Branching in Mixed-Integer Linear Programming} (2020), as this thesis shares methods with the report.

\section{Dataset}\label{sec:dataset}

This section presents the selected data set and the process of generating trainable and testable data for the models.
This will include a presentation of the problem instances, the generation of the expert solutions as well as the features that will be the input of the \gls{ML} models.

\subsection{Problem Instances}\label{ssec:probleminstances}

In order to evaluate the methods presented in this project to the previous advances in the field, artificially generated \gls{MILP} problems found in Gasse et al. (2019) \cite{gasse2019exact} are used to train and evaluate the models. These problems are also the standard implemented in \gls{Ecole}.
The problems are expressed as pure binary programs. The results of the algorithm are, however, generalizable to general \gls{MILP} problems, as it extends the general \gls{BnB} algorithm \cite{gasse2019exact}.

In the mentioned articles, four problem classes are used: set covering, combinatorial auctions, capacitated facility location, and maximum independent set. In this thesis, only the first two will be used due to problems in the \gls{Ecole} framework that have since been resolved.

%Problems are generated in three sizes: small, medium, and large. Small instances are used for training, while medium and large problem sizes are reserved for evaluating the extended algorithm's ability to generalize on more challenging problems. These problem sizes represent what current algorithms can solve within seconds (for the small instances) and multiple minutes (for the large instances).

\textbf{Set Covering}

The set covering problem is implemented in \gls{Ecole} as described in Balas \& Ho (1980) \cite{balas1980set} \footnote{The available version of this paper contains nearly illegible equations, the article Minoux (1987) \cite{minoux1987class} has therefore been used to supplement.}. The general problem includes a set of vertices $V_j$ and a matrix of costs $c_j$ for activating an edge between any two vertices. The activation of an edge is modeled with the binary variable $x_j$ and the characteristic vector of subset $V_j$ denoted $\textbf{A}_j$. The binary linear problem of covering all vertices at minimum cost can be expressed as \cite{minoux1987class}:
\begin{align}
    \min &\sum_{i=1}^N c_i x_i \\ s.t. \; &\sum_{i=1}^N \mathbf{A}_i x_i \leq \mathbf{1} \\
    \mathbf{x} &\in \mathbb{B}^N
\end{align}
The problem is therefore a pure binary linear problem with only inequality constraints. The set covering problem is a well-known class and is one of Karp's 21 \gls{NP}-complete problems \cite{karp1972reducibility}.

\textbf{Combinatorial Auction}

The combinatorial auction problem is the problem with the shortest solution time in the mentioned articles \cite{gasse2019exact,gupta2020hybrid}. The problem is implemented by Gasse et al. (2019) \cite{gasse2019exact}, and is based on the formulation presented in Leyton-Brown et al. (2020) \cite{brown2000towards}. The version is specifically the \textit{arbitrary} formulation from the original article \cite{brown2000towards}. In this formulation, $N$ is the number of combinations of items, $p_i$ is the price of an item, $\mathcal{P}$ is the set of all items, and $a_{i,j} \in \mathbb{B}$ represents whether item $j$ belongs to combination $i$. With this, the problem can be stated as \cite{abrache2007combinatorial}:
\begin{align}
    \max &\sum_{i=1}^N p_i x_i \\ s.t. \; &\sum_{i=1}^N \mathbf{a}_{i,j} x_i \leq 1 \,, \quad \forall \, j \in \mathcal{P}\\ \mathbf{x} &\in \mathbb{B}^N
\end{align}

The combinatorial auctions problem is based on multi-object auctions where bidders place monetary bids on bundles (combinations) of goods, and the optimization problem is to find the bids that maximize the profit of the auctioneer \cite{brown2000towards}. The problems are considered realistic and economically oriented, and are applicable to five broad domains in which optimization is important: proximity in space, paths in space, arbitrary relationships, temporal matching, and temporal scheduling \cite{brown2000towards}. The problems are \gls{NP}-hard \cite{dong2012combinatorial}.

%\subsubsection{Capacitated Facility Location}

%\subsubsection{Maximum Independent Set}

%Barabasi-Albert graph.
%\footnote{In Gasse et al. (2019) the graph is reported to be of the type Erd\"{o}s-R\'{e}nyi, however the implementation is in fact a Barabasi-Albert graph.}

\subsection{Expert Solution Generation}\label{ssec:expertsolutiongeneration}

The generated problems are solved using the full strong branching policy, as explained in \Cref{ssec:branchingpolicy}. To not interfere with branching policies, the application of cutting planes to the problem is restricted to the root node, before any branching decisions are made. This means that the Branch-and-bound algorithm used in these experiments could be called a \textit{cut-and-branch} algorithm, as is discussed in \Cref{ssec:inequalities}.
However, without loss of generality and for simplicity, the algorithm will be referred to as \textit{branch-and-bound} in this thesis despite the initial application of cutting planes.

\begin{algorithm}[H]
    \SetAlgoLined
    \KwResult{Variable selection samples}
    \For{$episode\in EPISODES$}{
        \While{episode not solved}{
            $\rho\leftarrow rand()$\;
            \eIf{$\rho \leq 0.05$}{
                $samples\leftarrow node\_ state$\;
                $scores\leftarrow SB\_scores$\;
                Save scores and state\;
                Branch on SB score\;
                }{
                Branch on pseudo-cost scores\;
            }
        }
    }

    \caption{\label{alg:datacol} Data collection algorithm}
\end{algorithm}

The algorithm for data collection is shown in \Cref{alg:datacol} and is adapted from Gasse et al. (2019) \cite{gasse2019exact}. During training, 5 \% of the branching variable decisions are done by the strong branching expert, compared to 5 \% in Gasse et al. (2019) \cite{gasse2019exact} and 100 \% in Gupta et al. (2020) \cite{gupta2020hybrid}. This method of generating expert samples has been criticized by Sun et al. (2021) \cite{sun2021improving}, among other reasons because strengths of \gls{SB} cannot be learned through variable selection only. The reader is referred to this article for further reading, in addition to the thesis Ross (2013) \cite{ross2013interactive} for more on expert imitation learning data.

For every node, the \gls{SB} policy assigns a score to every possible branching variable. The best variable to branch on according to \gls{SB} is saved explicitly and then branched on. The best candidate variable is used for training, while the \gls{SB} scores are used for evaluating the learned policy. The placement of the selected variable compared to the candidate variables will be used to generate an accuracy score, to evaluate whether the algorithm is able to select among the top variables to branch on.

\subsection{Branching Variable Features}

At nodes where the strong branching policy is used, available features for every possible branching variable is saved. The features, as explained in \Cref{ssec:obs}, are divided into the groups variable features, constraint features and edge features. These features are presented in
 \Cref{tab:feats}. For a further explanation on the relevant features, see \Cref{ssec:obs}.
%
\renewcommand{\arraystretch}{1.5}
\begin{table}
	\centering
	\begin{tabular}{p{0.1\linewidth}p{0.2\linewidth}p{0.5\linewidth}}%{lll}
		\toprule
		  \textbf{Tensor} & \textbf{Feature} & \textbf{Description} \\
		  \toprule
		  \multirow{13}{*}{\textbf{V} (19)} & objective & Objective coefficient, normalized.  \\
		  & type (4) & Type (binary, integer, impl. integer, continuous) as a one-hot encoding. \\
		  & has\_lb & Lower bound indicator. \\
		  & has\_ub & Upper bound indicator \\
		  & reduced\_cost & Reduced cost, normalized. \\
		  & sol\_value & Solution value. \\
          & sol\_frac  & Solution value fractionality.\\
		  & sol\_is\_at\_lb & Solution value equals lower bound.\\
		  & sol\_is\_at\_ub & Solution value equals upper bound.\\
		  & scaled\_age & LP age, normalized. \\
          & inc\_val & Value in incumbent.\\
          & avg\_inc\_val & Average value in incumbents.\\
		  & basis\_status (4) & Simplex basis status (lower, basic, upper, zero) as a one-hot encoding.\\
		  \midrule
		  \multirow{5}{*}{\textbf{C} (5)} & obj\_cos\_sim & Cosine similarity with objective. \\
		  & bias & Bias value, normalized with constraint coefficients. \\
		  & is\_tight & Tightness indicator in LP solution. \\
		  & dualsol\_val & Dual solution value, normalized. \\
		  & scaled\_age & LP age, normalized with total number of LPs. \\
		  \midrule
		  \multirow{1}{*}{\textbf{E} (1)} & coef & Constraint coefficient, normalized per constraint. \\
		  %Static Features & 18 & Khalil et al. \cite{khalil2016learning} \\
		  %Dynamic Features & 54 & Khalil et al. \cite{khalil2016learning} \\
		% \addlinespace
		\bottomrule
	\end{tabular}
	\caption{Features of each variable in the data, adapted from Gasse et al. (2019) \cite{gasse2019exact}.}\label{tab:feats}
\end{table}
%
%The features of each node are saved as a custom python-class derived from the Data-class of pyTorch Geometric, and given in the source code of \textit{\gls{Ecole}}.

The underlying assumption is that these features are sufficiently correlated with the optimal variable to branch on, as it is given by the Strong Branching algorithm.
% math stuff here?

\section{Models}\label{sec:models}

This section presents the \gls{ML} models used in the experiments. These models are \textit{iterative ablations} of the model presented in Gasse et al. (2019) \cite{gasse2019exact}, meaning each model is an increasingly simplified version of the original network.

\subsection{Original Graph Convolutional Neural Network}\label{ssec:models_gcnn}

First, the nature of the Gasse GCNN has to be formulated. The central component is the graph convolution, which is modelled as two half-convolutions. This means that the convolution is divided into to subsequent passes --- one from variable to constraints and one from constraints to variables \cite{gasse2019exact}.

The two half-convolutions can be expressed as \cite{gasse2019exact}:
\begin{equation}
    \mathbf{c}_i \leftarrow \mathbf{f}_{\mathcal{C}}\left( \mathbf{c}_i \; \sum_j^{(i,j) \in \mathcal{E}}\mathbf{g}_{\mathcal{C}} (\mathbf{c}_i, \mathbf{v}_j, \mathbf{e}_{i,j})\right), \qquad
    \mathbf{v}_j \leftarrow \mathbf{f}_{\mathcal{V}}\left( \mathbf{c}_j \; \sum_j^{(i,j) \in \mathcal{E}}\mathbf{g}_{\mathcal{V}} (\mathbf{c}_i, \mathbf{v}_j, \mathbf{e}_{i,j})\right)
\end{equation}
for all $i \in \mathcal{C}, j \in \mathcal{V}$. $\mathbf{f}_{\mathcal{C}},\mathbf{f}_{\mathcal{V}},\mathbf{g}_{\mathcal{C}},\mathbf{g}_{\mathcal{V}}$ are 2-layer perceptrons.

The addition of the graph convolution operator results in the nodes of the bipartite variable-constraint graph containing information about its neighbors

Gasse et al. (2019) \cite{gasse2019exact} also notes that it is common to normalize convolutional layers by the number of neighbors, however, they conclude that the loss of expressiveness by including this normalization is not beneficial. Therefore they include the affine normalization layer called a \textit{prenorm} layer. The layer is applied after the convolution, and is expressed as $ \mathbf{x} \leftarrow (\mathbf{x}-\bm{\beta})/\bm{\sigma}$.

In the original implementation, the prenorm layer's coefficients ($\bm{\beta}, \bm{\sigma}$) are approximated and fixed in a pre-training phase, while the implementation in this thesis uses the new PyTorch library-component \verb|torch.nn.LayerNorm()|, where the coefficients are approximated continuously.

After the normalizations, the features are transformed to \textit{embeddings} of a uniform size of 64 via linear neural network layers.

An illustration of the Gasse \gls{GCNN} in given in \Cref{fig:gnn2}. $n$ is the number of candidate variables, $m$ is the number of constraints, $d=19$ is the number of variable features, $e=1$ is the number of edge features, $c=5$ is the number of constraint features. The two convolutions are marked and the network layers are represented with a rectangle for each layer. The size of the hidden layers is consistently equal to 64. After the convolutions, a 2-layer perceptron transforms from the variable features (with information gained after the convolutions) to the single output. The dimensions for the tensors are marked in the figure.
\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{img/gnn2.png}
    \caption{The graph convolutional neural network as specified in Gasse et al. (2019) \cite{gasse2019exact}.}
    \label{fig:gnn2}
\end{figure}

\subsection{Network Topologies}

As stated, the \gls{ML}-models used in the experiments are generated by iteratively ablating (removing components) from the original network. This results in models that are \gls{GCNN}s and models that are pure \gls{MLP}s, which will be compared for accuracy and efficiency on both \gls{GPU} and \gls{CPU}.

The ablated models are chosen based on three basic assumptions that are assumed to be true based on the vast literature in deep learning\footnote{These assumptions are separate from the three research questions presented in \Cref{sec:int_questions}, and will not be covered in the same detail.} \cite{goodfellow2016deep}. The assumptions are stated as:

\begin{enumerate}[label=(\roman*)]
    \item  It is assumed that increasing the capacity of the network correlates to increasing the number of computations \cite{goodfellow2016deep}.
    \item  It is assumed that the variation in complexity will lead to differences in both accuracy and forward pass computation time \cite{goodfellow2016deep}.
    \item  It is  assumed that the addition of the graph convolution will be a significant factor in the accuracy and efficiency of the models, particularly on the \gls{CPU}, based on the results in the appendix of Gupta et al. (2020) \cite{gupta2020hybrid}.
\end{enumerate}

Considering this, the experiments will be performed with a total of five models, meaning the original model and four models stemming from the iterative ablation of this model. Of these, two models will contain the graph convolution operation, and three will be multi-layer perceptrons that only use the variable data.

Results in the specialization project this thesis is based on showed consistent and significant changes in the accuracy/efficiency trade-off were found with a similar selection of \gls{MLP} models, although these models relied on the extended variable feature set developed in Khalil et al. (2016) \cite{khalil2016learning}.

\textit{GNN2}, illustrated in \Cref{fig:gnn1}, is identical (except for the minor, commented details), to the original Gasse \gls{GCNN}.

\textit{GNN1}, illustrated in \Cref{fig:gnn1}, is the model resulting from the first iteration of the ablation, which has reduced the number of layers in the embeddings and the output module. The graph convolutions are therefore unchanged.

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{img/gnn1.png}
    \caption{Topology of the ablated model GNN1.}
    \label{fig:gnn1}
\end{figure}

\textit{\gls{MLP}3} is illustrated in \Cref{fig:topo_mlp3}. The second ablation removes the convolutions in their entirety, resulting in an \gls{MLP} model consisting of the prenorm layer and embedding of the variable features with the output module. This equates to a four-layer \gls{MLP}

\textit{\gls{MLP}2} is illustrated in \Cref{fig:topo_mlp1}. The third ablation removes layers from the MLP3 model, yielding a network with lower capacity but fewer operations needed for evaluation. The two-layer \gls{MLP} also results in a minimal network for representing non-linear relations between the features and the scores.

\textit{\gls{MLP}1} is illustrated in \Cref{fig:topo_mlp2}. The fourth ablation removes all hidden layers, resulting in an \gls{MLP} model consisting of the prenorm layer and embedding of the variable features with the output module. This is equivalent to finding the optimal linear combination of input parameters\footnote{To be exact, MLP1 is not a Multi-Layer Perceptron, however, the name is chosen to be consistent with the other models. The central difference between the models is whether they contain graph convolutions or not (given by the acronym) and the number of parameters (given by the number).}. A forward pass is equivalent to a single matrix multiplication.

\begin{figure}
    \centering
    \includegraphics[width=0.8\linewidth]{img/mlp3.png}
    \caption{Topology of the larger sized MLP3 feed-forward network.}
    \label{fig:topo_mlp3}
\end{figure}

\begin{figure}
    \centering
    \includegraphics[width=0.55\linewidth]{img/mlp2.png}
    \caption{Topology of the medium sized MLP2 feed-forward network.}
    \label{fig:topo_mlp2}
\end{figure}

\begin{figure}
    \centering
    \includegraphics[width=0.45\linewidth]{img/mlp1.png}
    \caption{Topology of the smaller sized MLP1 feed-forward network.}
    \label{fig:topo_mlp1}
\end{figure}

\subsection{Network hyperparameters}

All hyperparameters except for the amount and size of the hidden layers is consistent for all models, in order to make a fairer judgment of the accuracy/efficiency trade-off. The activation function is the \textit{recti-linear unit} (ReLU), expressed as $y = \max \{ 0, x\}$. This non-linear activation function is the least computationally expensive of the common activation functions. This is consistent with Gasse et al. (2019) \cite{gasse2019exact} and Gupta et al. (2020) \cite{gupta2020hybrid}.

\section{Training Protocol}\label{sec:trainingprotocol}

In this section, the training protocol is presented, which explains how the parameters of the models are learned.

\subsection{Loss function}

The problem is expressed as a two-class classification problem, where the classes are \textit{optimal variable} and \textit{sub-optimal variable}, respectively. The first class contains only the top variable from the strong branching evaluation. The loss function in binary classification is typically chosen as the binary cross-entropy loss function \cite{goodfellow2016deep}, calculated as:
\begin{align}
    \bm{\mathcal{L}}(\bm{\theta}) &= - \frac{1}{|\mathcal{D}|}\sum_{(\bm{x}, y) \in \mathcal{D}} \left( y_i \cdot \log( \hat y_i) + (1-y_i) \cdot \log(1 - \hat y_i) \right),
\end{align}
where $\mathcal{D}$ is the data set, $\hat y_i$ is the predicted class based on the sample features $x$, and $y_i$ is the classification by the Strong Branching expert.
This means that for each branching decision, $n$ candidate variables will be given a score analogous to the probability of that variable being the best variable according to the strong branching expert policy.

\subsection{Training method}

The parameters of the \gls{MLP} are trained via mini-batch gradient descent \cite{goodfellow2016deep}, and is performed using
 the Adam optimizer \cite{kingma2017adam}. The Adam optimizer performs gradient descent with adaptive moments in both the first and second order \cite{kingma2017adam}.
 The batch size (number of problems processed at once) is set to 32. The learning rate is reduced upon plateaus\footnote{\textit{Plateaus} are multiple epochs without reduction in the target loss.} in validation loss
 by multiplying with $ 0.2 $. This is a well-known method to ensure convergence to a reasonably good local minimum \cite{goodfellow2016deep}.

 Further, the training has implemented an \textit{early stopping scheme}, to ensure that the models have converged, regardless of model complexity. If the validation loss does not decrease 16 epochs in a row, the model that resulted in the lowest validation loss is saved and the training process is stopped. This is because it is considered unlikely for the optimizer to improve the model after that amount of epochs without improvement.

 All of these methods are implemented in order to give the fairest evaluation of the models, as individual hyperparameter optimization for each model would be too time-consuming to be performed.

\subsection{Computer Hardware}

Training of the models is performed on an NVIDIA RTX2070 SUPER GPU (8GB VRAM). The \gls{BnB} solution efficiency evaluations are performed with an Intel i5-2500K \gls{CPU} (4 cores, 6 threads) running at 3.31 GHz. This is less expensive hardware compared to Gasse et al. (2019) \cite{gasse2019exact} and Gupta et al. (2020) \cite{gupta2020hybrid}, particularly the CPU. Variation in processing power is assumed to have an insignificant effect on the relative solution times of the methods, however, \gls{CPU}s with internal acceleration methods for matrix operations might give varying results \cite{vanhoucke2011improving}. It is assumed that the results on the chosen hardware yield a more pessimistic performance overall for the \gls{ML} based methods, and will therefore not be given too much attention in the thesis. Analysis of these differences in hardware and processing performance for both the classical and learned methods on both \gls{CPU} and \gls{GPU} is of interest, but far beyond the scope of this thesis.

\section{Comparison Method}

The method in which the learned models are compared to each other and the classical branching policies is presented in this section.

\subsection{Classical Branching Policies}

The accuracy and efficiency of the algorithm with the learned branching policy are evaluated against three branching policies native to the \gls{SCIP} optimization suite, and explained in  more detail in \Cref{ssec:branchingpolicy}.

First is the full strong branching (\gls{FSB}) policy \cite{applegate1995finding}, the slow but accurate expert policy that performs strong branching at every node for every variable. This algorithm is not commonly used because of the computationally heavy variable decision process \cite{achterberg2004branching}.

Pseudo-cost branching (\gls{PC}) is the fast but inaccurate branching policy \cite{gamrath2018measuring} that chooses the variable that maximizes the lower bound improvement according to the results of previous branching decisions \cite{benchiou1971experiments}.% This method does not compute anything for the candidate variables and relies only on previous results.

Reliability pseudo-cost branching (\gls{RPC}) \cite{achterberg2004branching} combines \gls{FSB} and \gls{PC} by performing \gls{SB} on variables with low confidence in the pseudo-costs \cite{gamrath2018measuring}. \gls{RPC} is the default branching policy in the \gls{SCIP} \gls{BnB} solver.

The classical variable selection policies will be implemented by configuring the priority of the branching policies in \gls{SCIP}.

\subsection{Benchmarking}\label{ssec:benchmarking}

The learned branching policies are evaluated for both \textit{accuracy} and \textit{efficiency}.

The accuracy of the policies is evaluated against the expert decisions of the full strong branching algorithm. This is done by evaluating the average \textit{top-k accuracy} over an unseen test set. Top-k accuracy measures whether the chosen branching variable was within the top k choices of the strong branching algorithm, based on the score given to each candidate variable by the algorithm. Only the learned models are measured in this regard.

The efficiency of the policies is evaluated by exchanging the branching policy of the optimization solver and testing on the data set, as described in \Cref{sec:dataset}. The counting of nodes and time is done with the built in capabilities of \gls{Ecole}. %For the largest problems, solution time is limited to $ 45 $ minutes, and the number of problems the algorithm does not solve will be reported.
%The execution time is compared using python's \verb|time.proc_time()| function, which returns the \gls{CPU} process time \cite{rossum2009python}.

The number of nodes processed by the algorithm will also be reported. Fewer nodes are not necessarily indicative of a better algorithm, however, this will be an important comparison, as it gives a proxy measurement of the accuracy/efficiency trade-off for the algorithms.

\section{Software}

A number of comprehensive software libraries are required to perform the experiments in this thesis, this section presents the three most essential.

\subsection{SCIP}

\textit{Solving Constraint Integer Programs }(\gls{SCIP})
\cite{achterberg2009scip}, is the \gls{CO} solver that lays the foundation for the data generation and model evaluation for the methods. It is considered the fastest non-commercial solver for \gls{MIP} and \gls{MINLP} problems \cite{gamrath2020scip}. It is written in C with wrappers for C++. The software is maintained by the\textit{ Zuse Institute Berlin} (ZIB) group.

The \textit{SCIP Optimization Suite} contains, as of version 7, the simplex-based linear programming solver SoPlex \cite{wunderling1996soplex}, the automated decomposition solver GCG \cite{gamrath2010gcg}, the parallelization framework UG \cite{shinano2018ug} and the \gls{MILP} pre solving library PaPILO \cite{gamrath2020scip}.

In the experiments in this thesis, the \gls{LP} solver and node selection algorithms are the most relevant, as most of the more complex capabilities of \gls{SCIP} are deactivated in order to make the results as fair and reproducible as possible.

\subsection{PyTorch}

\textit{PyTorch} is the Python library that the machine learning models are written in. The library provides, most notably, dynamic eager execution, automatic differentiation, and \gls{GPU} acceleration \cite{paszke2019pytorch}.

Necessary for constructing and training the graph convolutional neural networks, the \textit{PyTorch Geometric} library was used \cite{fey2019pytorchgeometric}. The library facilitates deep learning on data with irregular structures, such as graphs. This includes methods for sparse data GPU acceleration and efficient mini-batch handling \cite{fey2019pytorchgeometric}.

\subsection{Ecole}\label{ssec:ecole}

To facilitate and standardize the development of \gls{ML}-based improvements of \gls{CO} algorithms, \textit{Extensible Combinatorial Optimization Learning Environments} (\gls{Ecole}) was developed \cite{prouvost2020ecole}, with the first version published in the last quarter of 2020. \gls{Ecole} aims to mimic the \textit{openAI Gym} framework \cite{brockman2016openai}, a popular framework for developing \gls{RL} models. It is a python library written largely in efficient C++.
The library is based on the open-source solver \gls{SCIP} \cite{achterberg2009scip}.

It also provides problem \textit{generators} for the four problem classes presented in this thesis. As of version 0.5, these generators are not completely correct, and in the problem generators from Gupta et al. (2020) \cite{gupta2020hybrid} were therefore used. These problems appear to be resolved in version 0.6.

The theoretical basis for \gls{Ecole} is the \gls{PO-MDP} formulation for \gls{Ecole}, given in \Cref{sec:back_mdp}.
%The relevant functions and their \gls{PO-MDP} counterparts can be presented as follows \cite{prouvost2020ecole}:
%\begin{align*}
%    RewardFunction &\equiv \mathcal{R}\\
%ObservationFunction &\equiv \mathcal{O}\\
%reset\_dynamics() &\equiv p_\textit{init}(s_0)\\
%step\_dynamics() &\equiv p_\textit{trans}(s_{t+1}|s_t,a_t)\\
%reset() &\equiv s_0 \sim p_\textit{init}(s_0), r_0=R(s_0), o_0=O(s_0)\\
%step() &\equiv s_{t+1} \sim p_\textit{trans}(s_{t+1}|a_t,s_t)\\ r_t&=R(s_t)\\ o_t&=O(s_t)
%\end{align*}

\subsection{Development Environment}

The project is run on only open-source software, in order to support and further develop the commonly available tools for machine learning and mathematical programming. For reproducibility, the libraries and their respective versions are listed here, as well as in the publicly available code repository.

The project code is written in
\verb|python 3.8.5| \cite{rossum2009python}, and uses
\verb|pytorch 1.7.1| \cite{paszke2019pytorch} for the machine learning models.

For \gls{GPU} accelerated training,
\verb|CUDA 10.1| and
\verb|cudatoolkit 11.2| \cite{kirk2008nvidia} is used.

The optimization problems are solved using the \verb|SCIP 7.0.2| Optimization Suite \cite{gamrath2020scip}, using the
\verb|SoPlex 4.0.1| \cite{wunderling1996soplex} linear programming solver to solve the relaxed problems. The Python interface
\verb|PySCIPOpt 3.0.4| \cite{maher2016pyscipopt} is used in and alongside \gls{Ecole} to facilitate communication with \gls{SCIP}.

The computer is running the \verb|Ubuntu 20.04.2| Linux distribution.

\subsection{Code Repository}

As stated, the code for reproducing the experiments is given in a publicly available code repository, found at \url{https://github.com/Sandbergo/branch2learn}. It is intended to be similar to the source code for Gasse et al. (2019) \cite{gasse2019exact}, and to be extensible in order to facilitate further work.

A presentation of the code is given in \verb|README.md|, with installation instructions in \verb|INSTALLATION.md|. Installation requires some manual installation but is mostly done using \verb|Anaconda|. The eponymous subdirectory \verb|branch2learn| contains python scripts for running the respective processes of training and evaluating the models: \verb|00_generate_instances.py|, \verb|01_generate_data.py|, \verb|02_train.py|,
\verb|03_test.py|, \\
\verb|04_evaluate.py|, and \verb|05_evaluate_standard.py|. In addition, the directories for the models and utility functions are found there.

The directory \verb|scripts| contains bash-scripts for reproducing all experiments used in this thesis.


%
% This is a borrowed LaTeX template file for lecture notes for CS267,
% Applications of Parallel Computing, UCBerkeley EECS Department.
% Now being used for CMU's 10725 Fall 2012 Optimization course
% taught by Geoff Gordon and Ryan Tibshirani.  When preparing
% LaTeX notes for this class, please use this template.
%
% To familiarize yourself with this template, the body contains
% some examples of its use.  Look them over.  Then you can
% run LaTeX on this file.  After you have LaTeXed this file then
% you can look over the result either by printing it out with
% dvips or using xdvi. "pdflatex template.tex" should also work.
%

\documentclass[twoside]{article}
\setlength{\oddsidemargin}{0.25 in}
\setlength{\evensidemargin}{-0.25 in}
\setlength{\topmargin}{-0.6 in}
\setlength{\textwidth}{6.5 in}
\setlength{\textheight}{8.5 in}
\setlength{\headsep}{0.75 in}
\setlength{\parindent}{0 in}
\setlength{\parskip}{0.1 in}

%
% ADD PACKAGES here:
%

\usepackage{amsmath,amsfonts,graphicx}

%
% The following commands set up the lecnum (lecture number)
% counter and make various numbering schemes work relative
% to the lecture number.
%
\newcounter{lecnum}
\renewcommand{\thepage}{\thelecnum-\arabic{page}}
\renewcommand{\thesection}{\thelecnum.\arabic{section}}
\renewcommand{\theequation}{\thelecnum.\arabic{equation}}
\renewcommand{\thefigure}{\thelecnum.\arabic{figure}}
\renewcommand{\thetable}{\thelecnum.\arabic{table}}

%
% The following macro is used to generate the header.
%
\newcommand{\lecture}[4]{
   \pagestyle{myheadings}
   \thispagestyle{plain}
   \newpage
   \setcounter{lecnum}{#1}
   \setcounter{page}{1}
   \noindent
   \begin{center}
   \framebox{
      \vbox{\vspace{2mm}
    \hbox to 6.28in { {\bf EE302 - Feedback Systems
	\hfill Spring 2018} }
       \vspace{4mm}
       \hbox to 6.28in { {\Large \hfill Lecture #1 \hfill} }
       \vspace{2mm}
       \hbox to 6.28in { {\it Lecturer: #2 \hfill } }
      \vspace{2mm}}
   }
   \end{center}
   \markboth{Lecture #1}{Lecture #1}

   \vspace*{4mm}
}
%
% Convention for citations is authors' initials followed by the year.
% For example, to cite a paper by Leighton and Maggs you would type
% \cite{LM89}, and to cite a paper by Strassen you would type \cite{S69}.
% (To avoid bibliography problems, for now we redefine the \cite command.)
% Also commands that create a suitable format for the reference list.
\renewcommand{\cite}[1]{[#1]}
\def\beginrefs{\begin{list}%
        {[\arabic{equation}]}{\usecounter{equation}
         \setlength{\leftmargin}{2.0truecm}\setlength{\labelsep}{0.4truecm}%
         \setlength{\labelwidth}{1.6truecm}}}
\def\endrefs{\end{list}}
\def\bibentry#1{\item[\hbox{[#1]}]}

%Use this command for a figure; it puts a figure in wherever you want it.
%usage: \fig{NUMBER}{SPACE-IN-INCHES}{CAPTION}
\newcommand{\fig}[3]{
			\vspace{#2}
			\begin{center}
			Figure \thelecnum.#1:~#3
			\end{center}
	}
% Use these for theorems, lemmas, proofs, etc.
\newtheorem{theorem}{Theorem}[lecnum]
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{definition}[theorem]{Definition}
\newenvironment{proof}{{\bf Proof:}}{\hfill\rule{2mm}{2mm}}

% **** IF YOU WANT TO DEFINE ADDITIONAL MACROS FOR YOURSELF, PUT THEM HERE:

\begin{document}

% Lecture Details
\lecture{9}{Asst. Prof. M. Mert Ankarali}
....

\section{Steady-Sate Response Analysis}

Fundamental concept that we need to perform
stead-state response analysis of a control
system is the final value theorem. Given a continuous time signal
$x(t)$ and its Laplace transform $X(s)$, if $x(t)$ is convergent signal,
final value theorem states that
%
\begin{align*}
\lim_{t\to \infty} x(t) &= \lim_{s \to 0} \left[ s X(s) \right]
\\
x_{ss} &= \lim_{s \to 0} \left[ s X(s) \right]
\end{align*}

\subsection{Tracking Performance}

The most important steady-state performance
condition for a control system is the tracking performance
under steady-state conditions. Let's consider the
following fundamental feedback topology.

    \begin{center}
\begin{minipage}[h]{0.8\linewidth}
    \begin{center}
      \includegraphics[width=\textwidth]{feedback}
    \end{center}
\end{minipage}
    \end{center}

In order to achieve a good tracking performance, obviously
the error signal $e(t)$ need to be small. Accordingly, steady-state
tracking performance is determined by the steady-state error
of the closed-loop system, that we can compute using final
value theorem as
%
\begin{align*}
	e_{ss} = \lim_{t\to \infty} e(t) &= \lim_{s \to 0} \left[ s e(s) \right]
\end{align*}
%
Let's compute $E(s)/R(s)$, i.e. transfer function from the reference input to the
error signal,
%
\begin{align*}
E(s) &= R(s) - E(s) G(s) H(s)  ,
\\
\frac{E(s)}{R(s)} &= \frac{1}{1 + G(s) H(s) }
\end{align*}
%
Note that $G(s) H(s)$ is the  transfer function from the error
signal $E(s)$ to the signal which is fed to the negative terminal of
the main difference operator, i.e. $F(s)$. This transfer function is
called the feed-forward or open-loop  pulse transfer function of the
closed-loop control system. For this system,
%
\begin{align*}
\frac{F(s)}{E(s)} = G_{OL} = G(s) H(s)
\end{align*}
%
Then $E(s)$ can be written as
%
\begin{align*}
E(s) = R(s) \frac{1}{1 + G_{OL} (s) }
\end{align*}
%
It is obvious that first requirement on steady-state error
performance is that closed-loop system have to be stable.
Now let's analyze specific but fundamental input scenarios.

\subsection*{Unit-Step Input}

We know that $r(t) = h(t)$ and $R(s) = \frac{1}{s}$ then
we have
%
\begin{align*}
e_{ss} &= \lim_{s \to 0} \left[ s R(s) \frac{1}{1
         + G_{OL} (s) } \right]
\\
&= \lim_{s \to 0} \left[ s \frac{1}{s} \frac{1}{1
         + G_{OL} (s) } \right]
\\
e_{ss} &= \frac{1}{1 + \lim_{s \to 0} G_{OL} (s) }
\end{align*}
%
If the DC gain of the system (also called static error constant) is
constant, i.e. $\lim_{s \to 0}G_{OL}(s) = K_{DC}$ then the steady state error can be
computed as
%
\begin{align*}
e_{ss} &= \frac{1}{1 + K_{DC}}
\end{align*}
%
It is obvious that
%
\begin{align*}
e_{ss} &\neq 0 \quad \mathrm{if} \quad |K_{DC}| < \infty
\\
e_{ss} &\to 0 \quad \mathrm{if} \quad K_{DC} \to \infty
\end{align*}
%
At this point, it could be helpful to introduce the concept of system \textit{type},
to generalize the steady-state error analysis.

\textbf{Definition:} Let's write the open-loop transfer function of a closed-loop
system in the following standard form
%
\begin{align*}
G_{OL}(s) = \frac{K}{s^N} \frac{b_0 s^m + \cdots + b_{m-1} s + 1}{a_0 s^n + \cdots + a_{n-1} s + 1}
\end{align*}
%
The closed-loop system is called as \textbf{Type N} system, where
$N$ is the $\#$ if integrators in the open-loop transfer function (OLTF).

Based on these results, we can have the following conclusions
regarding steady-state error for unit-step input
%
\begin{itemize}
\item If $G_{OL} (0) = K_P$ , $| K_{P} | < \infty$, then
\begin{align*}
e_{ss} =1/(1 + K_{P})
\end{align*}
%
These are \textbf{Type 0} (or \textbf{Type $N \leq 0$})
systems. We observe a bounded steady-state error and it is possible to reduce the error by increasing the static gain
constant $K_P$.
%
\item  If $G_{OL} (0) = \infty$, then $e_{ss} = 0$. In other words, for
  \textbf{Type $N > 0$} systems, the steady-state error is perfectly zero .
\end{itemize}

Now let's summarize the steady-state error conditions
%
\begin{itemize}
\item Type $N \leq 0$: $e_{ss} =  \frac{1}{1 + K_{P}}$
\item Type $N > 0$: $e_{ss} = 0$
\end{itemize}

\subsection*{Unit-Ramp Input}
%
We know that $r(t) = t h(t)$ and $R(s) = \frac{1}{(s^2}$ then
we have

\begin{align*}
e_{ss} &= \lim_{s \to 0} \left[ s R(s) \frac{1}{1
         + G_{OL} (s) } \right]
\\
&= \lim_{s \to 0} \left[ s \frac{1}{s^2} \frac{1}{1
         + G_{OL} (s) } \right]
\\
 &= \frac{1}{lim_{s \to 0} s^{-1} G_{OL} (s) }
\\
e_{ss} &= \frac{1}{lim_{s \to 0} \frac{K}{s^{N-1}} \frac{b_0 s^m + \cdots + b_{m-1} s + 1}{a_0 s^n + \cdots + a_{n-1} s + 1} }
\end{align*}
%
Based on this result we can have the following steady-state
error conditions for the unit-ramp input based on the type
condition of the system
%
\begin{itemize}
\item Type $N < 1$: $e_{ss} \to  \infty$
\item Type $N = 1$: $e_{ss} = \frac{1}{K_v}$
\item Type $N > 1$: $e_{ss} = 0$
\end{itemize}
%
where $K_v$ is called the velocity error constant.

\subsection*{Unit-Quadratic (Acceleration) Input}
%
We know that $r(t) = \frac{1}{2} t^2 h(t)$ and $R(s) = \frac{1}{s^3}$ then
we have
\begin{align*}
e_{ss} &= \lim_{s \to 0} \left[ s R(s) \frac{1}{1
         + G_{OL} (s) } \right]
\\
&= \lim_{s \to 0} \left[ s \frac{1}{s^3} \frac{1}{1
         + G_{OL} (s) } \right]
\\
 &= \frac{1}{lim_{s \to 0} s^{-2} G_{OL} (s) }
\\
e_{ss} &= \frac{1}{lim_{s \to 0} \frac{K}{s^{N-2}} \frac{b_0 s^m + \cdots + b_{m-1} s + 1}{a_0 s^n + \cdots + a_{n-1} s + 1} }
\end{align*}
%
Based on this result we can have the following steady-state
error conditions for the unit-ramp input based on the type
condition of the system
%
\begin{itemize}
\item Type $N < 2$: $e_{ss} \to  \infty$
\item Type $N = 2$: $e_{ss} = \frac{1}{K_{a}}$
\item Type $N > 2$: $e_{ss} = 0$
\end{itemize}
%
where $K_a$ is called the acceleration (parabolic) error constant.

\paragraph{Example 1:} Compute the $G_{OL}(s)$ for the following closed-loop
system and define its \textbf{Type}. After that, compute the steady-state errors to unit-step, unit-ramp, a
and unit-quadratic inputs.

\begin{center}
\begin{minipage}[h]{0.75\linewidth}
    \begin{center}
      \includegraphics[width=\textwidth]{example}
    \end{center}
\end{minipage}
\end{center}

\paragraph{Solution:}
%
\begin{align*}
&G_{OL}(s) =  \frac{K_P}{s ( s + K_D)}
\\
&\mathrm{Type} \ 1  \quad, \ K_v = \frac{K_P}{K_D}
\end{align*}
%
Then the steady-state errors are computed as
\begin{itemize}
\item Unit-step: $e_{ss} = 0$
\item Unit-ramp: $e_{ss} = \frac{K_D}{K_P}$
\item Unit-acceleration: $e_{ss} = \infty$
\end{itemize}

\paragraph{Example 2:} Compute the $G_{OL}(s)$ for the following closed-loop
system and define its \textbf{Type}. After that, compute the steady-state errors to unit-step, unit-ramp, a
and unit-quadratic inputs.

\begin{center}
\begin{minipage}[h]{0.75\linewidth}
    \begin{center}
      \includegraphics[width=\textwidth]{example2}
    \end{center}
\end{minipage}
\end{center}

\paragraph{Solution:}
%
\begin{align*}
&G_{OL}(s) =  \frac{K_P + K_D s}{s^2}
\\
&\mathrm{Type} \ 2  \quad, \ K_a = K_P
\end{align*}
%
Then the steady-state errors are computed as
\begin{itemize}
\item Unit-step: $e_{ss} = 0$
\item Unit-ramp: $e_{ss} = 0$
\item Unit-acceleration: $e_{ss} = \frac{1}{K_a}$
\end{itemize}

\paragraph{Example 3:} Compute the $G_{OL}(s)$ for the following closed-loop
system and define its \textbf{Type}. After that, compute the steady-state errors to unit-step, unit-ramp, a
and unit-quadratic inputs.

\begin{center}
\begin{minipage}[h]{0.6\linewidth}
    \begin{center}
      \includegraphics[width=\textwidth]{example3}
    \end{center}
\end{minipage}
\end{center}

\paragraph{Solution:}
%
\begin{align*}
&G_{OL}(s) =  \frac{s}{(s+1) (s+10)}
\\
&\mathrm{Type} \ \mathbf{-1}  \quad, \ K_P = 0
\end{align*}
%
Then the steady-state errors are computed as
\begin{itemize}
\item Unit-step: $e_{ss} = 1$
\item Unit-ramp: $e_{ss} = \infty$
\item Unit-acceleration: $e_{ss} = \infty$
\end{itemize}

\paragraph{Example 4:} Compute the steady-state error to unit-step input
for the following system.

\begin{center}
\begin{minipage}[h]{0.75\linewidth}
    \begin{center}
      \includegraphics[width=\textwidth]{marginal}
    \end{center}
\end{minipage}
\end{center}

\paragraph{Bad Solution:}
%
\begin{align*}
&G_{OL}(s) =  \frac{K_p}{s^2}
\\
&\mathrm{Type} \ \mathbf{2}
\\
&e_{ss} = 0 \quad ??????
\end{align*}
%
\paragraph{Good Solution:}
%
Let's compte $Y(s)$ and then $y(t)$,
%
\begin{align*}
  Y(s) &= \frac{\frac{K_p}{s^2}}{1 + \frac{K_p}{s^2}} R(s) =
         \frac{K_p}{s (s^2 + K_p)}
\\
 y(t) &= 1 - \cos (K t) \ t > 0
\end{align*}
%
Error function takes the form $e(t) = \cos (K t)$ which does not
have a limit, i.e., there is no $e_{ss}$. If closed-loop transfer
function has poles on imaginary axis then, we can not apply
final value theorem.

\paragraph{Example 5:} Compute the steady-state error to unit-step input
for the following system when $K_P = 1$.

\begin{center}
\begin{minipage}[h]{0.75\linewidth}
    \begin{center}
      \includegraphics[width=\textwidth]{unstable}
    \end{center}
\end{minipage}
\end{center}

\paragraph{Good Solution :)} Let's check if $y(t)$ is a convergent
signal
%
%
\begin{align*}
  Y(s) &= \frac{\frac{1}{s (s-2)}}{1 + \frac{1}{s (s-2)}} R(s) =
         \frac{1}{s (s^2 - 2s + 1) }
\\
 y(t) &= t e^t - e^t + 1 \ t > 0
\end{align*}
%
Error function takes the form $e(t) = e^t - t e^t$, thus
%
\begin{align*}
 e_{ss} = | \lim_{t \to \infty} e(t) | = \infty
\end{align*}
%
In conclusion, If closed-loop transfer function has poles on imaginary
axis or open right half-plane then, we can not apply final value theorem.

\newpage

\subsection{Stead-State Response to Disturbances}

When analyzing the steady-state response of a system in addition to
the desired response to the reference input, it is also important
to analyze the response to unwanted disturbances and noises.

Let's analyze the steady-state performance of the following
topology which is perturbed by a disturbance input, $d(t)$.

\begin{center}
\begin{minipage}[h]{0.75\linewidth}
    \begin{center}
      \includegraphics[width=\textwidth]{disturb}
    \end{center}
\end{minipage}
\end{center}

In order to analyze the response to the disturbance $d(t)$, we assume
$r(t) = 0$ (which is just fine due to the linearity). Let's first find
the pulse transfer function from $D(s)$ to $Y(s)$.
%
\begin{align*}
T_D(s) &= \frac{Y(s)}{D(s)} =  \frac{G(s)}{1 + C(s) G(s) H(s)}
\\
&= \frac{G(s)}{1 + G_{OL}(s)}
\end{align*}
%
Note that $Y(s)$ depends on both $G_{OL}(s)$ (OLTF) and $G(s)$ (Plant TF).
If one wants to generalize the stead-state disturbance rejection performance,
he/she needs to analyze the conditions for both $G_{OL}(s)$ and $G(s)$. Moreover,
for a different topology and type of disturbance, we can have very different conditions.
For this reason, in order to analyze steady-state disturbance/noise rejection
performance, it is better to use fundamentals and apply final value theorem.

\newpage

\paragraph{Example 6:} The following closed-loop system
is affected by a disturbance input $d(t)$. Compute the
steady-state performance/response to a unit step disturbance
input.

\begin{center}
\begin{minipage}[h]{0.8\linewidth}
    \begin{center}
      \includegraphics[width=\textwidth]{example4}
    \end{center}
\end{minipage}
\end{center}

\paragraph{Solution:} Lets compute $Y(s)/D(s)$
%
\begin{align*}
Y(s) &= \left( D(s) - Y(s) K_P \right) \frac{1}{s(s + K_D)}
\\
Y(s) \left[ 1 + \frac{K_P}{s(s + K_D)}  \right] &= D(s) \frac{1}{s(s + K_D)}
\\
\frac{Y(s)}{D(s)} &= \frac{\frac{1}{s(s + K_D)} }{\frac{s^2 + K_D s + K_P}{s(s + K_D)}}
= \frac{1}{s^2 + K_D s + K_P}
\end{align*}
%
 Now let's compute $y_{ss}$,
 %
 \begin{align*}
y_{ss} &= \lim_{s \to 0} [ s Y(s) ] = \lim_{s \to 0} \left[ s D(s) \frac{1}{s^2 + K_D s + K_P} \right]
\\
&= \lim_{s \to 0} \left[ s \frac{1}{s} \frac{1}{s^2 + K_D s + K_P} \right]
\\
&= \frac{1}{K_P}
 \end{align*}
%
We can see that even if same system has 0 steady-state error when the reference
signal is step-like input, the error under unit-step disturbance is not zero, i.e., $y_{ss} = 1 / K_P$.
One can improve the disturbance rejection performance by increasing the $K_P$ gain.

% **** This ENDS THE EXAMPLES. DON'T DELETE THE FOLLOWING LINE:
\end{document}


\section{Measurement Procedue}

\setenumerate[2]{label= (\arabic*)}

\subsection{Spring constant}

\begin{enumerate}
\item Adjust the Jolly balance to vertical and attach the spring Add a 20g
  preload and adjust knob $I_1$ and $I_2$ to make sure the mirror can move
  freely through the tube. Check if the balance parallel to the spring.
\item Adjust knob $G$ to make the three lines in the tube coincide.
\item Reading the reading on the scale, add mass from 1 to 6 and record $L_i$ in
  order.
\item Estimate the spring constant $k_1$ by fitting.
\item Repeat the measurements with spring 2. Calculate $k_2$.
\item Remove the preload and repeat the measurements for spring 1 and 2
  connected. Calculate $k_3$.
\end{enumerate}

\subsection{Relation between oscillation period $T$ and the mass of the oscillator $M$}

\begin{enumerate}
\item Adjust the air track so that it's horizontal.

\begin{enumerate}
\item Turn on the air track and check whether there's blocked holes.
\item Place the cart on the track. Adjust the track with the knob on the side
  with a single one.
\end{enumerate}

\item Horizontal air track

\begin{enumerate}
\item Attach springs to the sides of the cart and set up the I-shape shutter.
  Make sure that the photoelectric gate is at the equilibrium position.
\item Set the timer into ``T'' mode. Add weight in order and release it with a
  caliper. Record the corresponding time intervals for 10 periods.
\item Analyze the relation between $M$ and $T$ by plotting a graph.
\end{enumerate}

\item Inclined air track

\begin{enumerate}
\item Add three plastic plates under the air track every time. Repeat the steps
  in 2.
\item Analyze the relation between $M$ and $T$ by plotting a graph.
\end{enumerate}

\end{enumerate}

\subsection{Relation between the oscillation period and the amplitude}

\begin{enumerate}

\item Keep the mass of the cart unchanged and change the amplitude (choose 6
  different values). The amplitude is about 5.0/ 10.0/ 15.0/ ... /30.0 cm.
\item Apply linear fit to the data and comment on the relation between the
  oscillation period $T$ and the amplitude $A$ based on the correlation
  coefficient $\gamma$.
\end{enumerate}

\subsection{Relation between the maximum speed and the amplitude}

\begin{enumerate}
\item Keep the mass of the cart unchanged and change the amplitude from $5.0\
  to\ 30.0 cm$ with gap of $5.0cm$.
\item Measure $v_{\max}$ for different amplitudes.
\end{enumerate}

\subsection{Mass measurement}

\begin{enumerate}
\item Adjust the electronic balance every time before you use it. The level
  bubble should be in the center of the circle.
\item Add weights according to a fixed order. Weigh the cart with the I-shape
  shutter and with the U-shape shutter. Measure the mass of spring 1 and spring
  2.
\item Record the data only after the circular symbol on the scales display
  disappears.
\end{enumerate}

\section{Caution}

\begin{itemize}
\item Do not stretch the spring over its elastic limit, otherwise the spring
  will not return to its original shape.
\item When using the Jolly balance, the mirror should be moving freely in the
  glass tube without any friction. When adding weight, hold the tray steady to
  avoid errors due to vibrations.
\item Please use tweezers to move the weights around.
\item  Make sure that no air holes on the air tracks are blocked.
\item Avoid scratching the cart. Do not move the cart when pressed against the
  air track.
\end{itemize}


\documentclass[11pt,letterpaper]{article}

\usepackage[utf8]{inputenc}
\usepackage[spanish,es-nodecimaldot]{babel}
\usepackage{fullpage} % Package to use full page
\usepackage{parskip} % Package to tweak paragraph skipping
\usepackage{tikz} % Package for drawing
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{hyperref}

\title{Práctica 1}
\author{Candela Castellanos Pablo Antonio\\
\and Hurtado Gutiérrez Marco Antonio\\
\and Hernández Austria Hugo Alexis\\}
\date{18 de febrero}

\begin{document}

\maketitle

\section{Pruebas}

\begin{enumerate}
\item
\begin{itemize}
\item
Hurtado Gutiérrez Marco Antonio
\item
Laptop Dell Inspiron 14-3451
\item
Procesador: Intel Pentium CPU N3540  @ 2.16GHz x 4
\item
Memoria RAM: 3.7GB
\item
Disco duro: 476.6 GB
\item
Sistema operativo: Linux Mint 1 Cinnamon 64-b
\item
Versión del Kernel:4.40-62-generic
\end{itemize}

\item
\begin{itemize}
\item
Hernández Austria Hugo Alexis
\item
Laptop Lenovo Yoga 300
\item
Procesador: Intel Pentium N3700 @ 2.40GHz (4 núcleos)
\item
Memoria RAM: 4096 MB Memoria cache: 2 MB
\item
Disco duro: 500 GB Seagate 5400 rpm
\item
Sistema operativo: elementary 0.4
\item
Versión del Kernel: 4.40-62-generic
\end{itemize}

\item
\begin{itemize}
\item
Candela Castellanos Pablo Antonio
\item
Laptop Dell Inspiron 1011
\item
Intel Atom CPU N270 @ 1.60GHz (2 nucleos)
\item
Memoria RAM: 1 GB
\item
Disco duro: 476.6 Gb
\item
Sistema operativo: 32 bits
\item
Distribucion linux : Ubuntu 14.04
\item
Versión del Kernel:4.40-62-generic
\end{itemize}
\end{enumerate}

\section{Exploring the derivative using Sage}

The definition of the limit of $f(x)$ at $x=a$ denoted as $f'(a)$ is:

\begin{equation}
f'(a) = \lim_{h\to0}\frac{f(a+h)-f(a)}{h}
\end{equation}

The following code can be used in sage to give the above limit:

\begin{verbatim}
def illustrate(f, a):
    """
    Function to take a function and illustrate the limiting definition of a derivative at a given point.
    """
    lst = []
    for h in srange(.01, 3, .01):
        lst.append([h,(f(a+h)-f(a))/h])
    return list_plot(lst, axes_labels=['$x$','$\\frac{f(%.02f+h)-f(%.02f)}{h}$' % (a,a)])
\end{verbatim}

\begin{figure}[!htbp]
\begin{center}
\includegraphics[width=8cm]{sage1.png}
\end{center}
\caption{The derivative of $f(x)=1-x^2$ at $x=.5$ converging to -1 as $h\to0$.}
\end{figure}

If we want to plot the tangent at a point $\alpha$ to a function we can use the following:

\begin{align}
y=&ax+b&&\text{(definition of a straight line)}\nonumber\\
  &f'(a)x+b&&\text{(definition of the derivative)}\nonumber\\
  &f'(a)x+f(a)-f'(a)a&&\text{(we know that the line intersects $f$ at $(a,f(a))$}\nonumber
\end{align}

We can combine this with the approach of the previous piece of code to see how the tangential line converges as the limiting definition of the derivative converges:

\begin{verbatim}
def convergetangentialline(f, a, x1, x2, nbrofplots=50, epsilon=.1):
    """
    Function to make a tangential line converge
    """
    clrs = rainbow(nbrofplots)
    k = 0
    h = epsilon
    p = plot(f, x, x1, x2)
    while k < nbrofplots:
        tangent(x) = fdash(f, a, h) * x + f(a) - fdash(f, a, h) * a
        p += plot(tangent(x), x, x1, x2, color=clrs[k])
        h += epsilon
        k += 1
    return p
\end{verbatim}

The plot shown in Figure \ref{lines} shows how the lines shown converge to the actual tangent to $1-x^2$ as $x=2$ (the red line is the `closest' curve).

\begin{figure}[!htbp]
\begin{center}
\includegraphics[width=8cm]{sage0.png}
\end{center}
\caption{Lines converging to the tangent curve as $h\to0$.}\label{lines}
\end{figure}

Note here that the last plot is given using the \textbf{real} definition of the derivative and not the approximation.

\section{Conclusions}

In this report I have explored the limiting definition of the limit showing how as $h\to 0$ we can visualise the derivative of a function. The code involved \url{https://sage.maths.cf.ac.uk/home/pub/18/} uses the differentiation capabilities of Sage but also the plotting abilities.

There are various other aspects that could be explored such as symbolic differentiation rules. For example:

$$\frac{dx^n}{dx}=(n+1)x^{n}\text{ if }x\ne-1$$

Furthermore it is interesting to not that there exists some functions that \textbf{are not} differentiable at a point such as the function $f(x)=\sin(1/x)$ which is not differentiable at $x=0$. A plot of this function is shown in Figure \ref{notdiff}.

\begin{figure}[!htbp]
\begin{center}
\includegraphics[width=8cm]{sage2.png}
\end{center}
\caption{None differentiable function at $x=0$.}\label{notdiff}
\end{figure}

\bibliographystyle{plain}
\bibliography{bibliography.bib}
\end{document}


\documentclass[a4paper]{article}

% Language and Font Encoding
\usepackage[english]{babel}
%\usepackage[utf8x]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{listings}
\usepackage{float}
\usepackage{caption}         % For captions
\usepackage{fancyhdr}        % For header and footer
\usepackage{rotating}
\usepackage{amssymb}
%\usepackage{varwidth}
\usepackage[colorlinks=true, allcolors=blue]{hyperref}

% Page size and margins
\usepackage[a4paper,top=3cm,bottom=2cm,left=2cm,right=2cm,marginparwidth=1.75cm]{geometry}

% Useful Packages
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{amsthm}
%\usepackage{bbm}
\usepackage{siunitx}
\usepackage{cite}
\DeclareMathOperator*{\argmax}{argmax}
\DeclareMathOperator*{\argmin}{argmin}
\DeclareMathOperator*{\softmax}{softmax}

\newcommand{\squeezeup}{\vspace{-2.5mm}}
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}

\pagestyle{fancy}
\lhead{Appendix: Linear Algebra}
\rhead{}

\begin{document}

\title{Optimization on Manifolds, Appendix: Linear Algebra}
\author{Ibrahim Akbar}
\maketitle

\section{Definitions}

This is an abbreviation of the material provided in the Appendix A1 of \textit{Optimization on Manifolds} and will provide definitions of various terms that appear throughout the notes. For proofs and further insight into the concepts refer to the Linear Algebra notes. Note that the conventions used between separate texts may not be the same. I will make sure to be as explicit as possible unless it is obvious from context.\\

\noindent Let $A,B$ be $n\times n$ matrices, the \textbf{commutator} of $A,B$ is: $[A,B] = AB - BA$.\\
\noindent The commutator of two symmetric/skew-symmetric matrices is symmetric and skew-symmetric for a symmetric and skew-symmetric matrix.\\

\noindent The \textbf{trace} of $A\in\mathbb{R}^{n\times p}$ is
$$
tr(A) = \sum_{i=1}^{\min{n,p}} = (A)_{ii}
$$

\noindent Properties of Trace
\begin{itemize}
\item $tr(A) = tr(A^{T}$
\item $tr(AB) = tr(BA)$
\item $tr([A,B]) = 0$
\item $B^{T} = -B\Rightarrow tr(B) = 0$
\item $(A^{T} = A \land B^{T} = -B)\Rightarrow tr(AB) = 0$
\end{itemize}

\noindent Let $A$ be an $n\times n$ matrix. $A$ is \textbf{invertible} if there exists a matrix, $B$, such that $AB = I_{n}$.\\

\noindent A matrix $A$ is \textbf{orthonormal} if $A^{T}A = I$. If $A$ is a square orthonormal matrix then it is termed \textbf{orthogonal}.\\

\noindent A \textbf{vector space}, $\mathcal{V}$, is a collection over a field, $\mathbb{F}$ that is endowed with the operators,  $+:V\times V\rightarrow\mathbb{V}$ and $\cdot:\mathbb{F}\times\mathcal{V}\rightarrow\mathcal{V}$.\\
Note any \textit{n}-dimensional real vector space $\mathcal{E}$ is isomorphic to $\mathbb{R}^{n}$. A diffeomorphism requires determining a basis for $\mathcal{E}$ but may be computationally intractable, given it is a continuous space.
Therefore the material considers abstract vector spaces and only finite-dimensional vector spaces over $\mathbb{R}$. To understand this statement better refer to the \textit{Morphisms and Functions} Appendix.\\

\noindent A \textbf{norm} is an operator that maps from the vector space to the real field, $\norm{\cdot}:\mathcal{V}\times\mathcal{V}\rightarrow\mathbb{R}$.\\

\noindent Properties of Norm
\begin{itemize}
\item $\norm{\mathbf{v}} = 0 \iff \mathbf{v} = 0 \hspace{2mm} \forall\mathbf{v}\in\mathcal{V}$
\item $\norm{\alpha\mathbf{v}} = |\alpha|\norm{\mathbf{v}}$
\item $\norm{\mathbf{u+v}} \leq \norm{\mathbf{u}} + \norm{\mathbf{v}}$
\end{itemize}

\noindent This mapping allows for the definition of a \textbf{normed vector space}, $\mathcal{V}$, which is endowed with a norm.\\

\noindent A \textbf{linear transformation}, $\mathcal{T}:\mathcal{V}\rightarrow{U}$, is linear if $\mathcal{T}(\alpha\mathbf{v}_{1}+\beta\mathbf{v}_{2}) = \alpha T(\mathbf{v}_{1})+\beta T(\mathbf{v}_{2})\hspace{2mm}\forall \mathbf{v}_{1},\mathbf{v}_{2}\in\mathcal{V}$.\\

\noindent We can define the set $\mathcal{L}(\mathcal{E};\mathcal{F})$ as all operators $A:\mathcal{E}\rightarrow\mathcal{F}$ which is also a vector space. Furthermore, if we let $\mathcal{E}, \mathcal{F}, \mathcal{G}$ be normed vector spaces we can define norms as:
$\norm{\cdot}_{\mathcal{L}(\mathcal{E};\mathcal{F})},\norm{\cdot}_{\mathcal{L}(\mathcal{E};\mathcal{G})},\norm{\cdot}_{\mathcal{L}(\mathcal{F};\mathcal{G})}$.\\
These are \textbf{mutually consistent} if for all $A\in\mathcal{L}(\mathcal{E};\mathcal{F})$ and $B\in\mathcal{L}(\mathcal{F};\mathcal{G})$:
$$
\norm{B\circ A}_{\mathcal{L}(\mathcal{E};\mathcal{G})} \leq \norm{A}_{\mathcal{L}(\mathcal{E};\mathcal{F})}\norm{B}_{\mathcal{L}(\mathcal{F};\mathcal{G})}
$$
A \textbf{consistent} or \textbf{submultiplicative norm} is mutually consistent with itself.\\

\noindent Furthermore the \textbf{operator norm} or \textbf{induced norm} of $A\in\mathcal{L}(\mathcal{E};\mathcal{F})$ is
$$
\norm{A}_{\mathcal{L}(\mathcal{E};\mathcal{F})} = \sup_{x\in\mathcal{E}}\frac{\norm{Ax}}{\norm{x}}
$$

\noindent Operator norms are mutually consistent.\\

\noindent A \textbf{bilinear} operator is a mapping $A:\mathcal{E}\times\mathcal{F}\rightarrow\mathcal{G}$ over normed vector spaces such that it is linear operator with respect to both spaces $\mathcal{E},\mathcal{F}$. We will denote the space of such operators as $\mathcal{L}(\mathcal{E},\mathcal{F};\mathcal{G})$ or $\mathcal{L}_{2}(\mathcal{E};\mathcal{F}$ if the two input spaces are identical. A simple example of a bilinear operator would be matrix multiplication. A bilinear operator is denoted as \textbf{symmetric} if $A[x,y] = A[y,x]$.\\

\noindent A symmetric bilinear operator $A\in\mathcal{L}_{2}(\mathcal{E};\mathbb{R})$ is \textbf{positive-definite} if $A[\mathbf{x,x}] > 0\hspace{2mm}\forall\mathbf{x}\in\mathcal{E}$.\\

\noindent A \textbf{Euclidean space} is a finite-dimensional vector space endowed with an inner product (bilinear, symmetric, positive definite operator) $\langle\cdot,\cdot\rangle$.\\

\noindent An \textbf{orthonormal basis} of an n-dimensional Euclidean space $\mathcal{E}$ is a sequence $(e_{1},e_{2},\ldots,e_{n})$ of elements such that,
$$
\langle e_{i},e_{j}\rangle =
\begin{cases}
1 & i = j\\
0 & i\neq j\\
\end{cases}
$$

\noindent An operator $\mathcal{T}:\mathcal{E}\rightarrow\mathcal{E}$ is \textbf{symmetric} if $\langle \mathcal{T}[x],y\rangle = \langle x,\mathcal{T}[y]\rangle$.\\

\noindent Given an operator $\mathcal{T}:\mathcal{E}\rightarrow\mathcal{F}$ maps between two Euclidean spaces, the \textbf{adjoint} of T, denoted $\mathcal{T}^{*}:\mathcal{F}\rightarrow\mathcal{E}$ is such that $\langle \mathcal{T}[x],y\rangle = \langle x, \mathcal{T}^{*}[y]\rangle\hspace{2mm}\forall x\in\mathcal{E}\hspace{2mm}y\in\mathcal{F}$.\\

\noindent The \textbf{kernel} of an operator is $ker(\mathcal{T}) := \{x\in\mathcal{E}\mid\mathcal{T}[x] = 0\}$.\\

\noindent The \textbf{range} of an operator is $range(\mathcal{T}) := \{\mathcal{T}[x]\mid x\in\mathcal{E}\}$.\\

\noindent The \textbf{orthogonal compliment} of a subspace, $\mathcal{S}\subset\mathcal{E}$ is denoted $\mathcal{S}^{\perp} := \{x\mid\langle x,y\rangle = 0\hspace{2mm}y\in\mathcal{S}\}$.\\

\noindent Given that $x\in\mathcal{E}$ it can be uniquely decomposed as $x = x_{1} + x_{2}$ where $x_{1}\in\mathcal{S}$ and $x_{2}\in\mathcal{S}^{\perp}$. $x_{1},x_{2}$ are \textbf{orthogonal projections} of $x$ into their respective space. Let $\Pi_{\mathcal{S}}(x)$ denote the orthogonal projection of $x$ onto $\mathcal{S}$.\\

\noindent The \textbf{Moore-Penrose Inverse} also known as the pseudo-inverse is used to determine the orthogonal projection that satisfies the least squares problem.
$$
T^{\dagger} = (T^{*}T)^{-1}T^{*}y
$$

\noindent The \textbf{Euclidean norm} on a Euclidean space, $\mathcal{E}$, is $\norm{x}_{2} := \sqrt{\langle x,x\rangle}$, whereas on $\mathbb{R}^{n}$ it is $\norm{x}_{2} := \sqrt{x^{T}x}$.\\

\noindent Given an inner product space $\mathbb{R}^{n\times p}$ that has the inner product $\langle X,Y\rangle := tr(X^{T}Y)$ the Euclidean norm is the \textbf{Forbenius norm} denoted by $\norm{A}_{F} := \bigg(\sum_{i,j}(A)_{i,j}^{2}\bigg)^{\frac{1}{2}}$.\\

\noindent An operator norm on $\mathbb{R}^{n\times n}$ with $\mathbb{R}^{n}$ has the Euclidean norm is known as the \textbf{spectral norm}
$$
\norm{A}_{2} := \sqrt{\lambda_{max}(A^{T}A)}
$$

\end{document}


\documentclass{parcfd}

\usepackage{graphicx}
\usepackage{subcaption}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{enumitem}

\graphicspath{{figures/}}

\title{REPLICATION STUDIES AND REPRODUCIBILITY CHECKLIST FOR COMPUTATIONAL FLUID DYNAMICS}

\author{Olivier MESNARD$^{*}$ and Lorena A. BARBA$^{*}$}

\heading{Lorena A. Barba and Olivier Mesnard}

\address{$^{*}$ Dept.\ of Mechanical \& Aerospace Engineering, George Washington University,
20052 Washington DC, USA\\
e-mail: labarba@gwu.edu, web page: http://lorenabarba.com}

\keywords{Reproducibility, Replicability, Immersed boundary method, containerization, Reproducibility checklist}

\abstract{We conducted replication of previously published studies to highlight sources of non-reproducibility in CFD. Our objective is to develop guidance on the computational artifacts and the level of detail needed for reproducibility and replication to be possible. We propose a reproducibility checklist tailored for CFD, which could be used by journals and conferences like ParCFD.}

\begin{document}
%\maketitle

\section{INTRODUCTION}

We rely on computer simulations and data models to create new knowledge.
But how much evidence do we need to provide to be confident of the new findings claimed in a publication or conference talk?
Published studies rarely come with the underlying software and computational steps that produced the results.
Software can have bugs and articles can have typos.
When researchers do not make their code and data available, there is little hope of knowing how data were processed to verify the results.

In 2019, the National Academies of Sciences, Engineering, and Medicine released a consensus report \cite{nasem_2019} with findings and recommendations to improve rigor and transparency in research.
The report also gives definitions, intended to apply across all fields of science:

\begin{description}
    \item[Reproducibility] is obtaining consistent results using the same input data; computational steps, methods, and code; and conditions of analysis.
    \item[Replicability] is obtaining consistent results across studies aimed at answering the same scientific question, each of which had obtained its own data. Two studies may be considered to have replicated if they obtain consistent results given the level of uncertainty inherent in the system under study.
\end{description}

We have already published on our efforts to replicate our own results and the many obstacles we faced \cite{mesnard_barba_2017}.
We now explore the effort it takes to replicate the computational results from other research groups.

Our objectives are (1) investigate causes of non-reproducibility in computational fluid dynamics applications, particularly in HPC situations; (2) conduct replication of previously published computational results, looking to identify sources of non-replicability; (3) develop guidance on what computational artifacts and what level of detail need to be shared for reproducibility and replication to be possible; (4) design workflows for computational research to guarantee, as much as possible, that findings are both reproducible and replicable.

\section{REPLICATION STUDIES}

We attempted to replicate two computational articles: one presents a new immersed boundary method, the other a study on the unsteady wake dynamics of pitching-rolling wings.
The computational code, input data, and conditions of analysis used to produce the published results are not available, thus we consider the studies to be not reproducible.
But, can we replicate the scientific findings? And can we do it in a reproducible way?

\subsection{Reproducible workflow}

Our open-source software, PetIBM, solves the incompressible Navier-Stokes equations via a projection method on structured Cartesian grids, with an immersed boundary method.
It uses PETSc to run in parallel, and AmgX to run on GPU devices.
We follow an open-source development model on GitHub.
PetIBM is licensed under BSD 3-clause, archived in Zenodo, and published in the Journal of Open Source Software \cite{chuang_et_al_2018}.

To capture the computational environment, we use container technologies.
Previously, we used Docker to achieve a reproducible workflow on Microsoft Azure \cite{mesnard_barba_2020} public cloud.
Now, we adopt a similar reproducible workflow but on a University-managed HPC cluster.
Docker is not available at most HPC centers for security concerns; the Docker daemon requires root privileges that users do not have on shared production clusters.
Instead, HPC centers can use the container technology of Singularity, which was designed from the ground up to prevent escalation of user privileges.
Singularity images used to produce our results are made publicly available on the registry SingularityHub.
Researchers interested in reproducing our results can pull these images to create a faithfully reproduced computational environment.

\subsection{First replication study}

Li \& co-authors \cite{li_et_al_2016} modified the immersed boundary projection method, decoupling the solution variables through an additional block-LU decomposition.
This results in a classical pressure Poisson system.

We implemented the algorithms presented by Li et al.\ in a PetIBM application, and made it open source.
Our replication study highlights differences in the numerical results when re-implementing the ``same'' algorithms (Figure \ref{fig:cylinder2dRe40:pressure_coefficient}).

\begin{figure}
    \centering
    \includegraphics[width=0.8\linewidth]{cylinder2dRe40_pressure_coefficient.png}
    \caption{\small Distribution of the pressure coefficient along the surface of a stationary cylinder at Reynolds number $40$. With PetIBM, we used two different resolutions to discretize the immersed boundary: (left) $500$ equally spaced markers as in the original study \cite{li_et_al_2016} and (right) $189$ markers to maintain a similar resolution between the Lagrangian surface mesh and Eulerian grid.}
    \label{fig:cylinder2dRe40:pressure_coefficient}
\end{figure}

\subsection{Second replication study}

Li \& Dong \cite{li_dong_2016} conducted a parametric study of the rolling-pitching motion of a 3D elliptical plate, describing its wake topology and propulsive performance.
They used an in-house research code that implements a different immersed-boundary method (sharp-interface technique with a ghost-cell methodology) that those in PetIBM (diffused-interface method through regularized delta functions).

We have implemented the three-dimensional rolling and pitching kinematics (Figure \ref{fig:rollingpitching}) in a PetIBM application, and made it open source.
Our goal is to replicate the scientific findings with our code base and highlight possible differences in the results that may come from the use of different immersed-boundary techniques.

\begin{figure}
    \centering
    \begin{subfigure}{0.4\textwidth}
        \includegraphics[width=\linewidth]{rollingpitching_perspective.png}
        \caption{}
        \label{fig:rollingpitching:perspective}
    \end{subfigure}
    \hspace*{1cm}
    \begin{minipage}{0.3\textwidth}
        \begin{subfigure}{\linewidth}
            \includegraphics[width=\linewidth]{rollingpitching_side_view.png}
            \caption{}
            \label{fig:rollingpitching:side_view}
        \end{subfigure}
        \vspace*{0cm}
        \begin{subfigure}{\linewidth}
            \includegraphics[width=\linewidth]{rollingpitching_top_view.png}
            \caption{}
            \label{fig:rollingpitching:top_view}
        \end{subfigure}
    \end{minipage}
    \caption{\small Wake topology of a pitching and rolling circular plate after four flapping cycles at Reynolds number $Re = 200$ and Strouhal number $St = 0.6$. Flow structures are visualized using the Q-criterion for $Q = 1$ (colored by the streamwise vorticity) and $Q = 6$ (gray). (a) Perspective view, (b) side view, (c) top view.}
    \label{fig:rollingpitching}
\end{figure}

\section{HOW ABOUT A REPRODUCIBILITY CHECKLIST FOR CFD?}

On the basis of these efforts, we are developing ``Reproducibility Checklists'' for adoption by journals wanting to improve the reproducibility of their articles, by incorporating this aspect in the peer-review criteria.
Other fields already started implementing such reproducibility checklists.
At the NeuroIPS-2019 conference, authors were asked to answer questions from a ``Machine Learning Reproducibility Checklist'' during submission.

Such reproducibility checklist for CFD might include these categories: Numerical analysis; Computational environment; Software management; Data management; Computational runtime.
We propose to order the items in the list in increasing levels of reproducibility, starting with the minimum requirements, acknowledging that some authors may not be able to comply with all of them.
For example, the ``Software Management'' category might ask if the following are satisfied:

\begin{small}
\begin{itemize}[noitemsep]
    \item[$\square$] the software is under version-control.
    \item[$\square$] the software is shared under an open-source license.
    \item[$\square$] the software version used in the manuscript is deposited in a data archival service (figshare, Zenodo, etc.).
    \item[$\square$] the code base is developed following a standard branching model, using an issue tracker, pull requests, code review.
    \item[$\square$] the software was reviewed and published in the Journal of Open Source Software
\end{itemize}
\end{small}

\begin{thebibliography}{99}
    \bibitem{nasem_2019} {National Academies of Sciences, Engineering, and Medecine.} \textit{Reproducibility and Replicability in Science.} Washington, DC: The National Academies Press (2019).
    \bibitem{mesnard_barba_2017} Mesnard, O. and Barba, L.A. Reproducible and replicable computational fluid dynamics: it's harder than you think. \textit{Computing in Science \& Engineering} (2017) \textbf{19}(4):44--55.
    \bibitem{chuang_et_al_2018} Chuang, P.Y. and Mesnard O. and Krishnan A. and Barba, L.A. PetIBM: toolbox and applications of the immersed-boundary method on distributed-memory architectures. \textit{Journal of Open Source Software.} (2018) \textbf{3}(25).
    \bibitem{mesnard_barba_2020} Mesnard, O. and Barba, L.A. Reproducible Workflow on a Public Cloud for Computational Fluid Dynamics. \textit{Computing in Science \& Engineering} (2020) \textbf{22}(1):102--116.
    \bibitem{li_et_al_2016} Li, R.Y. and Xie, C.M. and Huang, W.X. and Xu, C.X. An efficient immersed boundary projection method for flow over complex/moving boundaries. \textit{Computers and Fluids} (2016) \textbf{140}:122--135.
    \bibitem{li_dong_2016} Li, C. and Dong, H. Three-dimensional wake topology and propulsive performance of low-aspect-ratio pitching-rolling plates. \textit{Physics of Fluids} (2016) \textbf{28}, 071901.
\end{thebibliography}

\end{document}