\documentclass[11pt]{article}
\usepackage{amsmath, amsfonts, amsthm, amssymb}  % Some math symbols
\usepackage{enumerate}
\usepackage{fullpage}

\usepackage[x11names, rgb]{xcolor}
\usepackage{tikz}
\usepackage[colorlinks=true, urlcolor=blue]{hyperref}
\usepackage{graphicx}
\usetikzlibrary{snakes,arrows,shapes}

\usepackage{listings}
\usepackage{array}
\usepackage{mathtools}
\setlength{\parindent}{0pt}
\setlength{\parskip}{5pt plus 1pt}
\pagestyle{empty}

\def\indented#1{\list{}{}\item[]}
\let\indented=\endlist

\newcounter{questionCounter}
\newenvironment{question}[2][\arabic{questionCounter}]{%
    \addtocounter{questionCounter}{1}%
    \setcounter{partCounter}{0}%
    \vspace{.25in} \hrule \vspace{0.5em}%
        \noindent{\bf #2}%
    \vspace{0.8em} \hrule \vspace{.10in}%
}{}

\newcounter{partCounter}[questionCounter]
\renewenvironment{part}[1][\alph{partCounter}]{%
    \addtocounter{partCounter}{1}%
    \vspace{.10in}%
    \begin{indented}%
       {\bf (#1)} %
}{\end{indented}}

%%%%%%%%%%%%%%%%% Identifying Information %%%%%%%%%%%%%%%%%
%% This is here, so that you can make your homework look %%
%% pretty when you compile it.                           %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newcommand{\myhwname}{Homework 4}
\newcommand{\mysection}{CS Fundamentals: Calculator Review [due Tuesday]}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}
\begin{center}
    {\Large \myhwname} \\
    \mysection \\
    \today
\end{center}

%%%%%%%%%%%%%%%%% PROBLEM 1: Probability Review %%%%%%%%%%%%%%%%%%%%%%%%%
\section{Review: Calculator App [Expected Duration: 15 - 60 min]}
\textbf{If you have any questions about the directions or any blocks you have not used before, let me know via email or text!}\\\\
Although we will not be working on anything on Sketch for this hw, I would \textbf{strongly recommend installing the \href{https://scratch.mit.edu/scratch2download/}{offline version of Scratch 2}} for future projects.\\
\noindent\makebox[\linewidth]{\rule{\paperwidth}{0.4pt}}\\
\begin{center}
  \includegraphics[width=3.5in]{preview.png}
 \end{center}
In the last lesson, we built together a very cool calculator together.
We built this cool calculator by using something called variables.
For homework, I want you to describe what is happening in each block shown below and also describe in general what variables are used for.\\\\
I want you to use the \textbf{word} bank below to complete this assignment.
Note that \textbf{the words in the word bank is used exactly once... all words are used}.\\\\
Just in case, it is confusing \dots\\
digits = numbers from 0 to 9 on the calculator\\
variables = the orange-y numbers\\
operators = the add, subtract, multiply, and equals.\\\\
If you have any questions regarding the terms in the word bank, \textbf{please} let me know!\\\\
Do your best and have fun!\\
\noindent\makebox[\linewidth]{\rule{\paperwidth}{0.4pt}}\\
\begin{center}
\textbf{Word Bank:}
\begin{tabular}{|c|c|c|c|}
    \hline
    + & 0 & addition & brain \\
    \hline
    digits & division & equal & operation \\
    \hline
    operators & operators & prev & previously \\
    \hline
    remember & result & subtraction & values \\
    \hline
    variable & variable & variable & variables \\
    \hline
\end{tabular}
\end{center}
\noindent\makebox[\linewidth]{\rule{\paperwidth}{0.4pt}}
\begin{enumerate}
\item \textbf{The Variables}
\begin{center}
  \includegraphics[width=2.5in]{vars.png}
 \end{center}
 % $\rule{2.5cm}{0.15mm}$
\begin{enumerate}[a.]
\item Each orange-y bubbles above are [variables].
\item ops is short for [operators] such as addition, [subtraction], and multiplication.
\item prev is short for [previously] pressed digit.
\item result is the result of using a combination of digits, operators, and [equal] sign.
\item In general, we use variables to store [values] we might need later into the computer's [brain].
\end{enumerate}
\end{enumerate}
\noindent\makebox[\linewidth]{\rule{\paperwidth}{0.4pt}}
\begin{enumerate}
\item \textbf{The Digits}
\begin{center}
  \includegraphics[width=2.5in]{numbers.png}
 \end{center}
\begin{enumerate}[a.]
\item All [digits] have similar code as the one above.
\item This specific code is for number [0].
\item Here we set the [variable] result to 0.
\end{enumerate}
\end{enumerate}
\noindent\makebox[\linewidth]{\rule{\paperwidth}{0.4pt}}
\begin{enumerate}
\item \textbf{The Operators}
\begin{center}
  \includegraphics[width=2.5in]{ops.png}
 \end{center}
\begin{enumerate}[a.]
\item All [operators] have similar code as the one above.
\item This specific code is for the operator [+].
\item Here we set the ops [variable] to + so that when equal sign is pressed we know which [operation] to use.
\item We set the prev [variable] to result because we want to [remember] the number pressed before pressing an operator.
\end{enumerate}
\end{enumerate}
\noindent\makebox[\linewidth]{\rule{\paperwidth}{0.4pt}}
\begin{enumerate}
\item \textbf{The Equals}
\begin{center}
  \includegraphics[width=2.5in]{equals.png}
 \end{center}
\begin{enumerate}[a.]
\item This equal sign only considers [addition] and subtraction.
\item We need to use ``If then else'' yellow block because ops variable may contain values for other operators like multiplication, [division], and exponents.
\item In the orange blocks within the ``if then else'' yellow block, we always have the value of the variable [prev] in the first slot of the green block and the value of the variable [result].
\end{enumerate}
\end{enumerate}
\end{document}


% !TeX spellcheck = en_GB
% !TeX encoding = UTF-8
\chapter{Background and Theory}
\label{ch:theory}
\epigraph{"Math is like water. It has a lot of difficult theories, of course, but its basic logic is very simple."}{- Haruki Murakami, IQ84}

This chapter is a summary of all the fundamental aspects of machine learning. More specifically, the crucial elements for an understanding of the prototype-functionality. First of all, the historical development of machine learning will be covered briefly. As soon as the reader has an understanding of the historical development of machine learning aspects and algorithms of machine learning, which are relevant to talk about the topic of this thesis are discussed. In the final part of this chapter, two different approaches of explainable artificial intelligence are introduced. Furthermore, a specific kind
explainable artificial intelligence is mentioned, such that the reader can understand how predictions can become transparent.

\section{History of Machine Learning}
\label{sec:history}

An example of machine learning is to take a lot of images and try to recognize cats (patterns) on them. The machine learning algorithm would be able to predict whether it is a cat or not. In this example, machine learning can be seen as

\begin{quote}\textit{"[...] a set of methods which can automatically detect patterns in data, and then use the uncovered patterns to predict unseen data or to perform other kinds of decision making under uncertainty" \cite[p. 1]{Murphy2012}.} \end{quote}

But there is more than one definition which describes machine learning. Arthur Samuel was the first researcher who used this term and defined it as "Machine Learning is a field of study that gives computers the ability to learn without being explicitly programmed" \cite{Samuel1959SomeSI}. The computer scientist Mitchel provided a more formal definition of machine learning, which is quoted in hundreds of papers\footnote{998 results on google scholar while searching for the original meaning from Mitchel (date: 24 Jan 2020)}. In his book from 1997, he says "A computer program is said to learn from experience E concerning some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E" \cite{Mitchell97}.\\

\begin{figure}[!htp]
	\centering
	\fbox{\includegraphics[width=1\linewidth]{photo/04_history_of_machine_learning}}
	\caption{Determining critical technologies within the domains of artificial intelligence(extended representation by Dmitri Gross according to a presentation from Michael Copeland) \cite{Gross2017} \cite{COPELAND2017}}
	\label{fig:04_history_of_machine_learning}
\end{figure}

According to the science magazine, the most influential computer scientist, in the field of pattern recognition, Michael Jordan wants to give another modern definition. His objective is to give a neutral statement, which defines the word artificial intelligence. He says:

\begin{quote}
	\textit{"It is one of today's rapidly growing technical fields, lying at the intersection of computer science and statistics, and at the core of artificial intelligence and data science" \cite{Bohannon2016} \cite{Jordan255}}
\end{quote}

Furthermore, with this definition, Jordan wants to connect the terms of machine learning and statistics. In his opinion, these two terms are the essential components of artificial intelligence \cite{MichaelJordan2018}. Machine learning has changed over the last decades. Machine learning became popular in the eighties, as a result of the increasing use of personal computers. Furthermore, deep learning gained on popularity until 2010, caused by the development of more powerful graphical processor units. This statement is criticized as well. Mainly because of the unclearly defined terminology. Anyways it is useful to get an overview of the field and the historical evolvement over the last years (\hyperref[fig:04_history_of_machine_learning]{Figure \ref{fig:04_history_of_machine_learning}}).

\begin{figure}[htp]
	\centering
	\fbox{\includegraphics[width=0.4\linewidth]{photo/05_deep_learning_sucess_with_alpha_go.jpeg}}
	\caption{The Netflix documentary Alpha Go is part of the hype around deep learning methods. Critics claim until the problem of transparency is not solved this results are not crucial because these models can not be used in a real-world application \cite{Thetruec1:online}}
	\label{fig:05_deep_learning_sucess_with_alpha_go}
\end{figure}

Alpha Go, which is developed by the Google Research Laboratory, is a good example of the popularity of deep learning methods. It shows how an enormous amount of time and hardware resources can lead to success as the artificial player has been the world ranked number one in the board game Go. But this success is criticized as well, because it is not a real-world application and gots pushed by a Netflix documentary, scientist claim that it is just an overhyped topic as long as the model could not explain its decisions (\hyperref[fig:05_deep_learning_sucess_with_alpha_go]{Figure \ref{fig:05_deep_learning_sucess_with_alpha_go}}).

\section{Suprvised Machine Learning for Upload Filters}
\label{sec:supervised_learning}
By doing the following steps, a so called cost function (\hyperref[def:cost_function]{Definition \ref{def:cost_function}}) will be optimized to get a prediction on unseen data, as defined by Mitchel (\hyperref[sec:history]{Section \ref{sec:history}}). An example application is an upload filter which determines if a dog or a cat is on an image:

\begin{enumerate}
	\itemsep-0.8em
	\item Eliminate empty entries and statistically irrelevant data e.g. duplicates
	\item Rescale the input data in the required form
	\item Split the Data into a test and a training set
	\item Initialize the parameters of the model
	\item Learn the parameters for the model by minimizing the cost function (\hyperref[def:cost_function]{Definition \ref{def:cost_function}})
	\item Use the learned parameters to make predictions (\hyperref[def:mapping_function]{Definition \ref{def:mapping_function}})
	\item Test the ability to predict patterns on unseen data correctly
\end{enumerate}

In the following, these steps are described in greater detail. The example of an Instagram upload filter is a classification task which can be solved by a supervised machine learning algorithm. It is designed to learn by example, \cite[p. 3 - 4]{Murphy2012}. Examples of cats and suitable counterexamples are assumed to be given before a supervised machine learning algorithms get developed.   (\hyperref[fig:06_example_dog_vs_cat_dataset]{Figure \ref{fig:06_example_dog_vs_cat_dataset}}).\\\\

\begin{figure}[htp]
	\fbox{\includegraphics[width=1\linewidth]{photo/06_example_dog_vs_cat_dataset}}
	\caption{Examples sampled from the the ImaegNet data set. The pattern on these images are similar to the patterns which the prototype shall recognise \cite{Building13:online}.}
	\label{fig:06_example_dog_vs_cat_dataset}
\end{figure}

The name supervised learning comes from the idea that training this type of algorithm is like having a supervisor which observes the whole learning process, e.g. recognize cats on images after having seen enough samples of the same distribution. At the same time, a "trainer" leads the algorithm to the correct result \cite[p. 103]{Goodfellow-et-al-2016} \cite[p. 3]{Murphy2012}. Technically spoken, these input data is called training data. First, training data consists of input data, for example, values that represent an image. Second, training data consists of output data, for example, the class which an image belongs to. During training, the algorithm will search for patterns in the input data that correlate with the desired outputs. After training, the supervised learning algorithm will feed with new unseen examples such that its ability to predict unseen data can be tested. So, the algorithm determines which labels correspondent to the unseen examples, based on optimized parameters. Such an algorithm can be expressed as follows \cite[p. 3]{Murphy2012}:

\begin{definition}[label=def:mapping_function]{Mapping Function}
	\begin{align*}
		f(X) = \hat{y}
	\end{align*}
	where \\\\
	\( f() \) = Mapping function which assigns the labels to a given input. \\
	\( X \)   = A matrix of input value where each column in \( X \) stands for a single example. \\
	\( \hat{y}\) = The determined labels for each column in \( X \) represented by a vector.\\
\end{definition}

Where \(\hat{y}\) is the predicted output, which is determined by a mapping function \(f\) that assigns a label to a single value or a set of multiple-input values donated by \(X\), the function connects an input features to a probability to belong to a certain class. The logic behind this function is also called machine learning model \cite[p. 3]{Murphy2012}.\\

Before a machine learning algorithm gets trained, the training data has to be prepared. This step is called preprocessing. An example of this is a machine learning algorithm which detects cats on images. Therefore training data is required to be preprocessed. This means the data has to be prepared such it fits into the mapping function. In the example of an machine learning algorithm which detects cat and dogs, this means the images transformed such that they are represented by a matrix. Within this matrix, every column stands for an example. Furthermore, the corresponding labels shall be in the form of a vector. Therefore, every value within this vector corresponds to a column in the input matrix. The vector represents the labels by zeros and ones. For example, a zero at the first position in the output vector means that the first column in the input matrix shows a cat \cite{brownlee2019deep}.

When talking about the training of a machine learning algorithm, the optimization of a function is always meant by that. Further, the function to be optimized is the mapping function which assigns an output value to a given input. Therefore the parameters of the function will be optimized during the training process, such that the output values match the actual output values as closely as possible. In other words, such that the real values as close as possible to the predicted values  (\hyperref[def:cost_function]{Definition \ref{def:cost_function}}). Usually, that is an optimization problem and is solved with different techniques. The technique also determines the required form of the input and output data, as suggested in step two. If we used a logistic regression to classify cats, a structure as can be seen in \hyperref[def:structure_for_linear_logistic_regression]{Definition \ref{def:structure_for_linear_logistic_regression}} is necessary. The logistic regression is a model which is applied to determine the probability of a certain class or event exists such as pass/fail, win/lose, alive/dead or healthy/sick. It can be extended to predict several classes of events such as determining whether an image contains a cat, dog, lion, etc.

\begin{definition}[label=def:structure_for_linear_logistic_regression]{required input format for a machine learning program with logical regression}
	\begin{align*}
		& X \in \mathbb{R}^{n\times m}  \\
		& y \in \mathbb{R}^{m}
	\end{align*}
	where \\\\
	\( X \) = A matrix of input values. Each column in stands for a single example. \\
	\( y \) = A vector with labels. Each value belongs to a column of the input matrix\\
	\( n \) = Total amount of input features\\
	\( m \) = Total amount of examples\\
	\( \mathbb{R} \) = All values are in real number space \\
\end{definition}

A pixel image of a cat, as you can see on the left-hand side in \hyperref[fig:07_image_vector_representation.png]{Figure \ref{fig:07_image_vector_representation.png}}, is represented as three matrices (\hyperref[fig:07_image_vector_representation.png]{Figure \ref{fig:07_image_vector_representation.png}, centre}). With the help of mathematical functions from the field of linear algebra, the images are transformed from the matrix form into a vector form (\hyperref[fig:07_image_vector_representation.png]{Figure \ref{fig:07_image_vector_representation.png}, right}) \cite[p. 276]{brownlee2019deep}. Even if it is not necessary, it is recommended to normalize the values. Otherwise, the calculations could become very slow and very memory intensive. \cite[p. 57]{Goodfellow-et-al-2016}.

\begin{figure}[htp]
	\fbox{\includegraphics[width=1\linewidth]{photo/07_image_vector_representation.png}}
	\caption{An image is represented by three matrices. Each for one colour channel (red, green and blue). After transforming it with linear algebra it becomes a vector.}
	\label{fig:07_image_vector_representation.png}
\end{figure}

The objective is to optimize the cost function (\hyperref[def:cost_function]{Definition \ref{def:cost_function}}) for many examples. This is why the input gets feed into a matrix (\hyperref[def:structure_for_linear_logistic_regression]{Definition \ref{def:structure_for_linear_logistic_regression}}). After transforming an image, we get a vector, so it is mandatory to store every single vector in the column of a matrix. Matrix representations are one of the reasons why machine learning algorithms are much more efficient. Nowadays, memory is cheaper than computing power. Without matrix multiplication, more loops are required. Therefore, loops require a relatively large amount of computing power, especially for extensive data. Matrix multiplication requires a lot of memory but requires fewer loops and therefore, less computing power. Due to the extreme amounts of data that are processed during the calculation of machine learning algorithms, these effects are multiplied by each other, which makes an efficient computation even more important \cite{AndrewNG}.\\

The next step in designing a machine learning algorithm is to create a specific function which can be optimized. As a first step  a linear function which looks like the following can be used:

\begin{definition}[label=lf]{A Linear Function as a vector and matrix representation to compute multiple input values at once}
	\begin{align*}
		z = W X + b
	\end{align*}
	where \\\\
	\(z\) = The results of the linear function\\
	\(W\) = A matrix with parameters assigned to every row of the input matrix \( X \)\\
	\(X\) = A matrix of input value where each column in \( X \) stands for a single example. \\
	\(b\) = Intercept added in a linear equation called bias parameter.\\
\end{definition}

Next step in order to get the results is to pass the results \( z \) to a sigmoid function. Every function which is used after calculating the linear function itself is a so-called activation function. Activation functions are mathematical equations to determine the output of a neural network. Furthermore, an activation function can be used for different tasks. In the pretend example the sigmoid function   (\hyperref[def:sf]{Definition \ref{def:sf}})  maps a value between 0 and 1 to each output. In such a way, that this value can be interpreted as a probability (\hyperref[def:sf]{Definition \ref{def:sf}}). Finally, a label can be assigned to the input by using a rule-based approach. For example, it could be the case that if the probability is greater than 50\% that a certain case will occur, a positive label will also be assigned (the probability that a cat is on the image is greater than 50\%, then the image has the label cat).\\

\begin{definition}[label=def:sf]{Sigmoid Function e.g. for using it to get probabilities}
	\begin{align*}
		\hat{y} = sigmoid(z)
	\end{align*}
	where \\\\
	\( \hat{y}\) = The determined labels for each column in \( X \) represented by a vector.\\
	\( z \)  = The results of the linear function\\
\end{definition}

The next step for the example of an Instagram upload filter is to get a criterion, which can be used to optimize the parameters. In other words, to quantify the success of the training process. In order to get such a criterion, a so-called error value is formed. This is calculated from the actual label (\(y\) )and the predicted label (\( \hat{y} \)). How exactly this calculation is done is determined by the loss function. For example the loss function could be defined by the cross-entropy function (\hyperref[def:cost_function]{Definition \ref{def:sf}}), which compares the given values with the calculated values. Every function in machine learning that could be used to compute such an error can be used as a loss function as well. Finally, all error values of the individual examples are summed up and divided by the total number of examples. The function which will be used to do so is therefore called error function and denoted by \(\mathcal{J}\). \(\mathcal{J}\) maps a single value to all errors. Finally, the error function \(\mathcal{J}\) gets optimized during the training process such that the parameters (\( W \) and \( b \)) adjusted such that the error function \(\mathcal{J}\) reaches a global minimum. While in the first step, the so-called forward propagation, the error values and calculations are calculated, in the second step, the weights of the function will be optimized. This step uses the chain rule to form the gradient of the weights and adjust them according to this gradient. In the literature, the second step is commonly called backward propagation. This step is explained in greater detail in \hyperref[sec:gd]{Section (\ref{sec:gd})}.

\begin{definition}[label=def:cost_function]{Cost Function which can be optimized}
	\begin{align*}
		\mathcal{J} = \frac{1}{m} \sum_{i=1}^m \mathcal{L}(\hat{y}^{i}, y^{i})
	\end{align*}
	where \\\\
	\(m\)  = Number of examples e.g. number of cat and no cat images \\
	\(\mathcal{L}\) = Cross-Entropy function \\
	\(\hat{y}\textsuperscript{i}\) = Calculated label for the \(i\)th example\\
	\(y\textsuperscript{i}\)     = Correct label for the \(i\)th  example\\
	\(    \mathcal{J} \) = Cost Function which can be optimized s. t. it is at its global minimum
\end{definition}

The cost function (\hyperref[def:cost_function]{Definition \ref{def:cost_function}}) should not be optimized as close as possible to the training data. A function which fits perfectly is called over-fitted.

Overfitting means models perform well on the training data but do not generalize well for new data. It happens when the model is too complex relative to the amount and noisiness of the training data. If the accuracy on predicting patterns within the training set is high, but the accuracy on predicting patterns on images from the test set is low, the machine learning algorithm is likely overfitted. So, it is time to take corrective measure \cite[p. 110]{Goodfellow-et-al-2016}. Anyway, the goal of supervised learning as in the Instagram upload filter example is to achieve high performance on unseen data. To do that the data has to be divided into a test and a training set. The training set is used like described before (preprocessing the data and approximate the cost function to its global minimum). In contrast, the test data set would only be preprocessed and then used to make predictions about unseen cat images. So it is possible to test the neural network of its ability to predict unseen data \hyperref[sec:history]{Section (\ref{sec:history}}).\\\\

In other words, every step is necessary to train a machine-learning algorithm to predict if it is a cat or not. In  \hyperref[fig:08_process_of_pretictions_for_a_cat_image]{Figure \ref{fig:08_process_of_pretictions_for_a_cat_image}} is the process of predicting an \(\times64 \) image visualized. Therefore the notations are explained in \hyperref[def:label]{Definition \ref{def:label}}.

\begin{definition}[label=def:label]{Cost Function which can be optimized}
	\(x\textsuperscript{th} \) = exists for each pixel of the \(i\)th example \\
	\( W\textsubscript{x} \) = \(x\)th parameter assigned to each pixel of \(x\textsuperscript{th}\) \\
	\( W a + b\) = the linear function which computes a first output (\hyperref[lf]{Definition \ref{lf}}) \\
	\( \sigma \) = the sigmoid function to get the labels  (\hyperref[def:sf]{Definition \ref{def:sf}})
\end{definition}

\begin{figure}[htp]
	\fbox{\includegraphics[width=1\linewidth]{photo/08_process_of_pretictions_for_a_cat_image.png}}
	\caption{The image visualizes the whole process of using an image (1) and putting this into a single neuron (4). First, the neuron calculates a linear function and uses the result as input to a sigmoid function (becomes an interpretable value between zero and one). The network can make a prediction (5) while using a decision boundary (e.g. if the probability is higher as 0.5 it is a cat) if the forecast is not correct, the parameters of the function (2, 3) getting adjusted as long as the function is optimized. Finally, we get an approximation that fits all training examples as close as possible.}
	\label{fig:08_process_of_pretictions_for_a_cat_image}
\end{figure}

An algorithm that works like this can be considered as a simple neural network \cite{Britz2015}. The term neural came from the fact that at first scientists tried to recreate the functionality of neurons in the human brain. Besides the early beginning, neural networks have not much in common with the brain because actually, it seems the brain is much more complicated as it seems before \cite{Kriesel2007NeuralNetworks}. However, it is called a network because usually different neurons working together. In the simple scenario from \hyperref[fig:08_process_of_pretictions_for_a_cat_image]{Figure \ref{fig:08_process_of_pretictions_for_a_cat_image}} we had just one neuron (\hyperref[fig:09_example_of_a_single_neuron]{Figure \ref{fig:09_example_of_a_single_neuron}})

\begin{figure}[htp]
	\centering
	\fbox{\includegraphics[width=0.4\linewidth]{photo/09_example_of_a_single_neuron}}
	\caption{An example for a single neuron. First, a linear output will be calculated by a simple linear function with the parameters \(W\) and \(b\). Afterwards the output will be normalized to get probabilities.}
	\label{fig:09_example_of_a_single_neuron}
\end{figure}

For the Instagram upload filter, such a neural network\footnote{The achieved accuracy is not always precisely the same, because the optimization just reaches a local minimum rather than a global minimum. In practice, this change is hardly relevant. Besides, the results can differ depending on the implementation. The results presented here were obtained with the programming language Python using frameworks from the field of data science.} achieves a test accuracy of 70\% (\hyperref[lst:acc]{Listing \ref{lst:acc}}).\\\\

\begin{lstlisting}[captionpos=b,label={lst:acc},language=Python, caption=Test accuracy is 70\% after iteration 2000 times and using 209 examples with 12287 features (\(64\times64 \) pixels). This is not state of the art but very good if considering that this is a linear classifier on a high dimensional feature space.]
Cost after iteration 0: 0.693147
Cost after iteration 1000: 0.214820
...
Cost after iteration 1800: 0.146542
Cost after iteration 1900: 0.140872
train accuracy: 99.04306220095694 %
test accuracy: 70.0 %
\end{lstlisting}

It is crucial to achieving a higher accuracy before creating a transparent neural network which is easy to understand. Within the next section, machine learning algorithms will be introduced to get higher accuracy.

\subsection{Deep Learning Neural Networks}
\label{subsec:deep_learning}
\hyperref[fig:10_process_of_pretictions_for_a_cat_image_two_layers]{Figure \ref{fig:10_process_of_pretictions_for_a_cat_image_two_layers}} shows how the same network as in \hyperref[fig:08_process_of_pretictions_for_a_cat_image]{Figure \ref{fig:08_process_of_pretictions_for_a_cat_image}} would look like if we would use two layers. Therefore the notations are explained in
\hyperref[def:label2]{Definition \ref{def:label2}}.
\begin{definition}[label=def:label2]{Cost Function which can be optimized}
	\(x\textsubscript{th} \) = exists for each pixel of the \(i\)th example \\
	\(a\textsubscript{th} \) = a so called neuron which gets the results from the first layer \\
	\( WX+b\) = the linear function which computes a first output (\hyperref[lf]{Definition \ref{lf}}) \\
	\( \sigma \) = the sigmoid function to get labels  (\hyperref[def:sf]{Definition \ref{def:sf}})
\end{definition}

\begin{figure}[htp]
	\centering
	\fbox{\includegraphics[width=0.8\linewidth]{photo/10_process_of_pretictions_for_a_cat_image_two_layers.png}}
	\caption{As before in the \hyperref[fig:08_process_of_pretictions_for_a_cat_image]{Figure \ref{fig:08_process_of_pretictions_for_a_cat_image}} The image visualizes the whole process of using a neural network to recognise patterns of cats and dogs. Instead of just on layer now the machine learning algorithm uses two layers.}
	\label{fig:10_process_of_pretictions_for_a_cat_image_two_layers}
\end{figure}

On the one hand, more layers resulting in an increasing amount of parameters which the algorithm has to optimize \cite[p. 21]{Goodfellow-et-al-2016}. For example, to find the global minimum of a cost function in our Instagram upload filter example, it would take significantly more time and requires more examples. And even then it is not sure if such a global minimum exists. On the other hand, a neural network with more layers would outstand a neural network with just one layer by far. \hyperref[lst:acc2l]{Listing \ref{lst:acc2l}} shows that the algorithm detects up to 80\% of the images. So, the deep neuronal network, compared with a single layer neural network, has increased by 10\%.

The results could be better if more examples were used as input data. The required time and computing capacity would be comparatively low with such an implementation. That is why the neural network is just used with the same amount of examples  \cite[p.167]{Goodfellow-et-al-2016} \cite[p.995 - 997]{Murphy2012}.

In general, the term deep within deep neural networks is related to the number of layers. Andrew Ng suggests that neural networks with more than one layer should be called deep \cite{AndrewNG}. Furthermore, every layer which is not the output and not the output layer itself would be called the hidden layer. Whereas the output layer counts to the total amount of layers, the input layer is usually not considered as a layer. There is not an exact definition of what a layer exactly is. Most sources are excluding the input and including the output layer. Steps in which an activation function activates the neurons do not count as independent layers. They belong to the previous layer \cite{AndrewNG} \cite{Kriesel2007NeuralNetworks} \cite{Goodfellow-et-al-2016}.

A problem with the neural network, which is not deep is its capability of training with larger images. So, the network was built to recognize patterns on images with 4096 input features ((\( 64 \times 64\) pixels). Suppose the input is a (\( 300 \times 300\) RGB image, the first layer of the network has 90000 neurons, and each one is fully connected to the input. That means that each neurone in the previous layer is connected to each node in the following layer. The number of parameters which the training process has to optimize would be calculated with the following formula \cite{Vasudev2019}\cite{DBLP:journals/corr/TraskGR15}.

\begin{definition}[label=cn]{Calculate the total number of neurons in a neuronal network}
	\begin{align*}
		& W\textsubscript{ff} = F\textsubscript{-1}\times F \\
		& B\textsubscript{ff} = F \\
		& P\textsubscript{ff} = W\textsubscript{ff}\times +  B\textsubscript{ff}
	\end{align*}
	where \\\\
	\(W\textsubscript{ff}\) = Number of weights of a FC Layer which is connected to an FC Layer \\
	\(B\textsubscript{ff}\) = Number of biases of a FC Layer which is connected to an FC Layer \\
	\(P\textsubscript{ff}\) = Number of parameters of a FC Layer which is connected to an FC Layer \\
	F = Number of neurons in the FC Layer\\
	\(F\textsubscript{-1}\) = Number of neurons in the previous FC Layer
\end{definition}

In the equation above, \( F_{-1} \times F \) is the total number of weights (connections between layers) from neurons of the previous fully connected layer, to the neurons of the current fully connected layer.  The total number of biases is the same as the number of neurons (F). So, for the example given above (\( 300 \times 300\) pixels, 100 Layers in the first hidden layer, RGB and fully connected) the calculation would look like this:\\
\begin{align*}
	& W\textsubscript{ff} =  270000 \times 100 = 27000000  \\
	& B\textsubscript{ff} = 100 \\
	& P\textsubscript{ff} = W\textsubscript{ff} +  B\textsubscript{ff} = 27000000 + 100 =27000100    \\
	& F = 100 \\
	& F\textsubscript{-1} = 270000  \text{ (300}  \times \text{300 pixels}  \times \text{3 colour channels)}\\
\end{align*}

If the image gets even bigger and the amount of layers increases, the weights can not be optimized with standard hardware, because the number of parameters is too big. As a guideline, there are about 14000000000 parameters where it is still possible to train without special hardware (e.g. on your own computer). Everything above this is difficult to realize on a personal computer without any hardware adjustments. Besides, the models could not predict in any case in a pleasant amount of time \cite{DBLP:journals/corr/TraskGR15}. Instead of the introduced neural network, there are particular implementations of deep neural networks that can calculate a forecast more efficiently.\\

The objective of the next section is to determine how to get excellent results in a reasonable time, even if the input images are greater or equal to \( 300 \times 300\) pixels.

\begin{lstlisting}[captionpos=b,label={lst:acc2l}, float=tb,language=Python, caption=Test accuracy is 80\% after iteration 2400 times and using 209 examples with 12288 features (\(64\times64 \) pixels). This is not state of the art but very good if considering that this is an algorihm wich is not spezialised to recoginize images.]
...
Cost after iteration 2300: 0.100897
Cost after iteration 2400: 0.092878
train accuracy: 98.5645933014 %
test accuracy: 80.0 %
\end{lstlisting}

\subsection{Convolutional Neural Networks}
\label{subsec:cnn}

The most popular deep learning models leveraged for computer vision problems are convolutional neural networks. To present the convolutional neural networks, the example given at the beginning of this chapter will be changed. Instead of cats, the network shall now recognize ten different digits which are simulated with the hands as you can see in \hyperref[fig:signs]{Figure \ref{fig:signs}}, where:

\begin{itemize}
	\itemsep-0.8em
	\item \textbf{y} is a vector with the length of the possible outputs, e.g. five-digit imitating means the length of a single output vector would be five \\
	\item and each element of \textbf{y} is a binary value which determines if the input belongs to a class on the \(i\textsuperscript{th}\) position within the vector\\
\end{itemize}

\begin{figure}[htp]
	\centering
	\fbox{\includegraphics[width=0.9\linewidth]{photo/11_numbers_by_hand_gestures}}
	\caption{Examples of gestures which are imitating digits}
	\label{fig:signs}
\end{figure}

This is a more realistic case of an upload filter because similar to this, the "Hitlergruß" gesture or another forbidden gesticulation can be identified. Even if the strength of a convolutional neural network is to recognize images with high dimensions, the dimensions will be low as in the example before. The calculation with inputs of higher dimensions takes to long such that it can be trained along with this thesis. \cite{DBLP:journals/corr/TraskGR15}\\

Before I used a so-called densely fully connected neural network (Figure \ref{fig:10_process_of_pretictions_for_a_cat_image_two_layers}). The network consists of a certain amount of neurons which are arranged in different layers (Figure \ref{fig:08_process_of_pretictions_for_a_cat_image}). Each neuron is fully connected with the neurons in the previous layer. To see how complex such a network can be,  in Figure \ref{fig:signs} such a network can be seen. The Figure shows that such a network can become complex. Consider that each connection is represented by a weight. This weight has to be initialized and adjusted. If it comes to understand when and why which weight was adjusted to ensure transparency, it becomes clear that such a network can be a confusing issue.

\begin{figure}[htp]
	\centering
	\fbox{\includegraphics[width=0.85\linewidth]{photo/12_example_fully_connected_network}}
	\caption{Examples of a fully connected neuronal network}
	\label{fig:signs}
\end{figure}

More neurons would result in a high amount of parameters if the image has too many dimensions \cite[p. 324]{Goodfellow-et-al-2016}. An alternative approach would be to use the mathematical operation of convolution \footnote{Technically I skip the narrowing operation, so this would be a cross-correlation instead of convolution. Still, as in literature and by convention, I call this a convolutional operation anyway\cite{AndrewNG}}. The convolution makes it possible to detect patterns like edges which could be used to classify an image. If the images have edges which are similar to the kind of edges from another image, the probability is high that it is in the same class. For example, if the algorithms detect a set of edges which are typically for cat ears, the algorithm will probably consider this image as a cat. Other possibilities of extracted patterns are shown in Figure \ref{fig:edge_detec}.\\\\
\begin{figure}[htp]
	\centering
	\fbox{\includegraphics[width=0.85\linewidth]{photo/13_example_feature_extraction}}
	\caption{Patterns identified while using convolutional neural networks \cite{SimpleIn5:online}}
	\label{fig:edge_detec}
\end{figure}

Convolution operations are widely spread in computer vision algorithms, so it is not unique for convolutional neural networks. It is a mathematical operation where a small matrix of numbers is passed through the matrix image representation. This matrix is a so-called filter or kernel. Every colour channel would need its filter. The name filter comes from the fact that it filters specific features from the input image, e.g. horizontal and vertical images. In Figure \ref{fig:edge_detec}, where

\begin{itemize}
	\item \textbf{*} is the mathematical operation of convolution
	\item \textbf{X} represents an image with a horizontal edge in the middle of the image
	\item \textbf{A} = the output matrix (feature map) which shows where the edge is\footnote{because this images are very small the dimensions of the edge which is shown in A are not accurate and unprofessionally. While using matrices of higher dimensions the proportions would be more accurate}
	\item \textbf{f} = a filter for detecting horizontal edges,
\end{itemize}

you can see how a vertical edge, which is represented by a
\( 6 \times 6\) matrix \(X\) is filtered and represented by a matrix while using another \( 3 \times 3\) matrix as a filter.

\begin{figure}[htp]
	\centering
	\fbox{\includegraphics[width=0.85\linewidth]{photo/14_vertical_edge_detection}}
	\caption{Using Convolution for edge detection}
	\label{fig:edge_detec}
\end{figure}

For a convolution operation, a filter would be placed in front of a selected pixel. Afterwards, each value from the filter has to be multiplied with the corresponding values from the image. Finally, the sum would be placed in the right place in the original matrix, as shown in Figure \ref{fig:conv}.

\begin{figure}[htp]
	\centering
	\fbox{\includegraphics[width=0.85\linewidth]{photo/15_example_convolution_operation.png}}
	\caption{Convolution Operation over a matrix}
	\label{fig:conv}
\end{figure}

\begin{figure}[htp]
	\centering
	\fbox{\includegraphics[width=0.85\linewidth]{photo/16_example_horizontal_edges_detected.png}}
	\caption{Convolution Operation on hand gestures, using a filter for horizontal edges and projecting them on the original image (without changing the size of it)}
	\label{fig:horizontal}
\end{figure}

\begin{figure}[htp]
	\centering
	\fbox{\includegraphics[width=0.85\linewidth]{photo/17_example_verticall_edges_detected.png}}
	\caption{Convolution Operation on hand gestures, using a filter for vertical edges and projecting them on the original image (without changing the size of it)}
	\label{fig:vert}
\end{figure}

\begin{figure}[htp]
	\centering
	\fbox{\includegraphics[width=0.85\linewidth]{photo/18_example_verticall_horizontal_edges_detected}}
	\caption{Convolution Operation on hand gestures, using a filter for horizontal and vertical edges and projecting them on the original image (without changing the size of it but while using normalization to make the edges more clear)}
	\label{fig:edges}
\end{figure}

The algorithms use a specific filter matrix, to detect of edges of every kind (e.g. vertical and horizontal edges). Two filters are used to identify at first horizontal edges (Figure \ref{fig:horizontal})and second vertical edges (Figure \ref{fig:vert}). Finally, both edges get projected to the original image and colour values were normalized to show the vertical and horizontal edges more clearly (\ref{fig:edges}). To figure out which filters shall be used to detect patterns is challenging because there are almost endless opportunities. That is where the neural network comes into part. Instead of setting the filter values manually, they are the parameters of a neural network. These parameters can now be optimized by using the cost function of the network (Definition \ref{def:cost_function}). That means the total amount of parameters comes no longer from the size of an image it comes now from the filter size \cite{AndrewNG})\\\\

Figure \ref{fig:archi} shows the architecture of such a network which is used to identify filter values  and finally helps
to classify more efficiently.

\begin{figure}[htp]
	\centering
	\fbox{\includegraphics[width=0.85\linewidth]{photo/19_example_cnn_architecture.png}}
	\caption{A typical architecture of convolutional neural network made from different building blocks}
	\label{fig:archi}
\end{figure}

As can be seen in the image, a convolutional neuronal network contains more than just convolution operations. One example is the RelU which is an activation function and is used to standardize values between the layers. The softmax activation function is used to get probabilities for each possible class. Afterwards, a decision boundary can be decided whether the image belongs to a class or not. The pooling layer reduces the height and width of the input. It helps minimize computations, as well as it helps to make feature detectors more invariant to its position in the input data. Typically a pooling layer is one of the two following types:

\begin{itemize}
	\item Max-pooling uses another matrix which slides over the input and stores the max value of the window in the output.
	\item Average-pooling uses another matrix which slides over the input and stores the average value in the output.
\end{itemize}

In the Figure \ref{fig:avg} \ref{fig:max} you can see how this would be done, where:
\begin{itemize}
	\item \textbf{Stride} is the value which determines for how many pixels the filter get shifted each time.
\end{itemize}

\begin{figure}[htp]
	\centering
	\fbox{\includegraphics[width=0.85\linewidth]{photo/20_max_pooling}}
	\caption{Maximum pooling, or max pooling, is a pooling operation that calculates the maximum, or largest, value in each patch of each feature map. The results are down sampled or pooled feature maps that highlight the most present feature in the patch, not the average presence of the feature in the case of average pooling \cite{}.}
	\label{fig:max}
\end{figure}
\begin{figure}[htp]
	\centering
	\fbox{\includegraphics[width=0.85\linewidth]{photo/21_average_pooling.png}}
	\caption{Average pooling involves calculating the average for each patch of the feature map. This means that each 2×2 square of the feature map is down sampled to the average value in the square.}
	\label{fig:avg}
\end{figure}

This kind of network and a dataset of ten different classes achieve accuracy close to 80\%\footnote{The result could be even better if more training data and a deeper network would be used}.

\begin{lstlisting}[captionpos=b,label={lst:cnn}, float=tb,language=Python, caption=Test accuracy is 80\% after iterations 2400 times and using 209 examples with 12288 features (\(64\times64 \) pixels). That accuracy is not state of the art but very good if considering that this is an algorithm which is not specialized to recognize images.]
Cost after epoch 0 =    1.917929
Cost after epoch 5 =    1.506757
Train Accuracy =    0.940741
Test Accuracy =    0.783333
\end{lstlisting}

The state of the art results in image recognition, e.g. on the cifar10\footnote{CIFAR-10  is an established computer-vision dataset used for object recognition. It is a subset of the 80 million tiny images dataset and consists of 60,000 \( 32 \times 32\) colour images containing one of 10 object classes, with 6000 images per class. Alex Krizhevsky, Vinod Nair collected it, and Geoffrey Hinton.} dataset, are impressive examples of how this technique has increased the performance of machine learning in the last years. They achieve results of 90\% and can be computed within a pleasant amount of time.\\

However, because of their non-linear structure, convolutional neural network algorithms with outstanding overall performance are usually seen as a black box. That means no information is provided about what exactly lead the networks to their conclusion. Transferred to the Instagram upload filter example, this would imply that 10\% of the users are blocked more falsely during the uploading.  The user will not be informed about why the algorithm came to its decision.\\

The goal of the next section is to present the theory of different methods of how a convolutional neural network behave more like a white-box model. In other words, a decision support system from which the user can be informed of why the algorithm came to a specific decision.

\section{Explainable Artificial Intelligence}
Most deep learning models today are black-box. That means no counteraction is set to make them transparent. Within the last years, techniques have arisen, which help to make those models more transparent. That means for a given prediction, how important is each input feature value to that prediction? Before two of these techniques are get explained in detail, it will be demonstrated one more time why this is so important.\\

Since deep learning has become more and more successful, science is also increasingly concerned with how to make deep learning models more transparent. For business leaders, machine learning engineers and the users who are confronted by the results, it is essential to know why a decision was made. Unless we were not able to learn from these models and they will still be a black box \cite{MichaelJordan2018} \cite{Kuang2017}.\\

Figure \ref{fig:xai-old-vs-new} shows the difference between a traditional machine learning model and an explainable artificial intelligence model. Whereas the traditional approach focuses on the output itself (e.g. this is a cat) the explainable approach combines some features to an explanation which can be understood by humans (e.g. has fur, whiskers and claws).\\

\begin{figure}[htp]
	\centering
	\fbox{\includegraphics[width=0.85\linewidth]{photo/22_what_is_xai}}
	\caption{This is a cat, but how do you know? Explainable artificial intelligence seeks to develop systems that can translate complex algorithmic decision-making into language humans can understand.\cite{Robinson2017}}
	\label{fig:xai-old-vs-new}
\end{figure}

Managers with a high responsibility for there department find it challenging to use these complex algorithms. Although they are aware that the results could increase the success of their company, they are afraid of using these models, because they are not transparent. If something goes wrong, they can hardly explain why the algorithm made the wrong decision. The results created by deep learning in image recognition could be beneficial for the healthcare sector as well, but if it is unclear why decisions were made, doctors will not give these approaches any chance. As well as the manager's doctors are responsible for their decisions, and if they get supported by an algorithm, they must understand the algorithm in the first place. Otherwise, they could not trust them, because they would not be able to distinguish between a right and a wrong decision.\\

A lack of transparency is responsible for the rejection of deep learning models in the finance industry as well. Especially since the last big financial crisis in 2009, banks and financial companies must make their decisions transparent to different stakeholders. And as already introduced in the introduction, small business owners could be affected by semitransparent models as well. If they run their business with the help of social media, it would harm them if their uploads get rejected automatically, without even knowing why this decision was made.\\

More specifically, a small shop owner wants to upload an image which shows him, introducing a new product. The algorithm was trained to detect the "Hitlergruß", and so he predicted the user performs this gesture. So, explainable artificial intelligence would help to explain automatically why this prediction would be made. Thus, the shop owner could change the crucial factors immediately.\\\\

The following sections will focus on two techniques which could be used in an upload filter. This could help the shop owner from the example given in the paragraph above. These two techniques are \cite{Kuang2017}:

\begin{enumerate}
	\item Shapely Additive exPlanations (SHAP) and
	\item Integrated Gradients (IG)
\end{enumerate}

The Shapley Additive exPlanations and the Integrated Gradients (IG) belong to two different categories for explainable artificial intelligence.

\begin{enumerate}
	\item Shapley-value-based algorithms and
	\item gradient-based algorithms
\end{enumerate}

Two factors can easily distinguish these models. Firstly, the assumptions and secondly, the underlying mathematical principles used in the models. Before it is explained how they differ, the fundamentals of each category will be explained.\\

The two fundamentals of the shapely-value-based algorithms and gradient-based algorithms are \cite{MichaelJordan2018} \cite{Kuang2017}:

\begin{enumerate}
	\item Shapley Values
	\item The Gradient
\end{enumerate}

\subsection*{Shapley Values}

Let assume that the algorithm behind an upload filter predicts the price for a painting. For a particular painting, it predicts 300000€, and the prediction will be explained through shapely values. The art has an age of 50 years, is part of a private collection, is not mentioned in literature and is in good shape (Figure \ref{fig:shapley_values_example_01}).\\

\begin{figure}[!htp]
	\centering
	\fbox{\includegraphics[width=0.5\linewidth]{photo/23_example_shapley_values_1}}
	\caption{A concrete visualization of the features from painting for which the hapley values will be calculated.The painting has an age of 50 years, is part of a private collection, is not mentioned in literature and and is in a good shape.}
	\label{fig:shapley_values_example_01}
\end{figure}

The average prediction for all paintings is 310000€. Now the question is how much has each feature value contributed to the prediction compared to the average prediction? For that, quantitative criteria will be used.

For a linear regression model, the answer is quite simple. The attribute of each feature is the weight of the feature times the feature value. It works because linear regression is a linear model. So, to make more complex models (e.g. deep learning, convolutional neural networks) transparent, a different solution is required. The Shapley value, coined by Shapley (1953), is a method for assigning payouts to players depending on their contribution to the total payout. Players cooperate in a coalition and receive a certain profit from this cooperation \cite{aas2019explaining} \cite{Lundberg} \cite{ScottM}.

Because this is not a real game, the game is the prediction task for a single instance of the dataset. Therefore the gain is the actual prediction for this instance, minus the average prediction for all instances. The players are the feature values of the instance that collaborate to receive the gain (predict the price for the painting). In the example, the feature values "Not mentioned in the literature, "painting is in good shape", "painting is 50 years old" and "painting is part of a private collection" worked together to achieve the prediction of 300000€. Our goal is to explain the difference between the actual prediction (300000€) and the average prediction (310000€): a difference of -10000€.

An answer looks like the following:

\begin{itemize}
	\item "Not mentioned in literature" feature contributed  30000€
	\item "50 years old" feature contributed 10000€
	\item "Part of a private collection" feature contributed 0€
	\item "Paining is in good shape" feature contributed 50000€
\end{itemize}

The contributions add up to 10000€, the final prediction minus the average predicted apartment price.

How do we calculate The Shapley value for one feature gets calculated the follows. The Shapley value is the average minimal contribution of a feature value among all possible coalitions.\\

\begin{figure}[!htp]
	\centering
	\fbox{\includegraphics[width=0.5\linewidth]{photo/24_example_shapley_values_2}}
	\caption{One sample in order to calculate the contribution of "is in a good shape" feature value to the prediction when added to the coalition of age, mentioned in literature and within a private collection }
	\label{fig:shapley_values_example_01}
\end{figure}

In the following  we evaluate the contribution of the "Painting is in good shape" feature value when it is added to a coalition of "Not mentioned in literature" and "50 years old"."Not mentioned in literature", "Painting is in good shape", and 50-years old" feature values are from the given predicted instance (painting with a worth of 300000€). The feature value for whether a painting is or is not in a private collection came from a random choice among all possible paintings which are in the same distribution. So, the value "in private collection" was replaced from yes by no. If the price of the painting gets now predicted again (with this combination), the predicted price is 310000€. In a second step, the "in good shape" feature value get doped from the coalition by replacing it with a random value among all other possible paintings which can be predicted. In the example, it was not in good shape, but it could have been good shape again. The prediction of the painting price for the coalition of "mentioned in literature" and "50 years old" is 320000€. The contribution of "In Good Shape" was:

\begin{quote}
	\centering
	310,000€ - 320000€ = -10000€
\end{quote}

This estimate depends on the values of the randomly drawn painting that served as a “donor” for the "shape" and "private" feature values. We will get better estimates if we repeat this sampling step and average the contributions. The result is even better if the sampling step gets repeated and calculate and calculate the average of all contributions. Furthermore, it is done repeatedly computing all coalitions. Because of the repeatedly computing for every combination, the calculation time increases exponentially. One solution to keep the computation time small is to calculate contributions for only a few samples of the possible coalitions.\\

If we estimate the Shapley values for all feature values, we get the complete distribution of the prediction (minus the average) among the feature values.

\subsection*{SHalpey Additive exPlanations (shap-software-library)}
If Shapley values are a method which maps a contribution to each player, a Shapley value-based explanation method will aim to get estimated shapely values which are close to the precisely calculable ones. It is achieved because the Shapley Additive Explanations are randomly dropping out some of the features. Potentially any combination of features can be left out. As long as you look at each feature individually, there is an almost infinite number of combinations. A calculation would take too long. As a counteraction, pairs of features are put together and in the following considered as one player. The individual success of a player is then estimated. Let us look again at the example of the shop owner. If a group of pixels does not contribute to the overall result of the game, it will not change even if you drop out the whole group \cite{ScottM} \cite{molnar2019} \cite{Lundberg}.\\

\begin{figure}[!htp]
	\centering
	\fbox{\includegraphics[width=0.8\linewidth]{photo/43_shap_library_example.png}}
	\caption{The header of the SHAP Github Repoitory shows an example of a \gls{mla} is explained through the library. It takes a model with age, sex, bp and bmi as feature examples and maps an attribute to this values. Each values showsthe average marginal contribution of a feature value across all possible coalitions of features.}
	\label{fig:shap_library_example}
\end{figure}

Figure \ref{fig:shap_library_example} shows an easy example of how the shape library explains the feature importance of a machine learning model. The colours help to get an intuitive understanding of which features are more important, so red stands for significant contribution to the overall prediction, and blue is less important to it.

\subsection*{Gradient and Gradient Decent}
\label{sec:gd}
I have to give some theory about the gradient and the gradient descent if it comes to an explainable artificial intelligence method which is built on top of the gradient and the gradient descent.\\

Gradient descent is possibly the most used optimization algorithm in the field of deep learning and machine learning. If there would be a cost function as introduced in the section about machine learning, the goal is to find the optimum for every weight in the model. So, the cost value would be as small as possible as long as the value is not adapting to a specific set of examples. The gradient descent is an algorithm that optimizes the cost value by making changes to the weights. The difference is determined by the gradient, which shows the direction of the deepest decent. In other words, it is a value which determines a direction which minimizes the cost in as few backpropagation steps as possible. Whereas in the so-called forward propagation the actual cost is calculated the backpropagation uses the chain rule to get the gradient at first and at second adjusts the weights by subtracting the gradient multiplied by the learning rate from them \cite{Goodfellow-et-al-2016}.\\

This method is quite old, and one of the reasons why this method got popular in the last years was the increase in computation power. Until a few years, a new generation of computational processing units and graphical procession units are fast enough to work on such tasks efficiently. \\

Figure shows\ref{fig:gd}, the graph of a cost function in three-dimensional space. The line shows the line from a starting point by going down to a local minimum. It is crucial to know that gradient descent can not distinguish between a local and a global minimum. Therefore it would need different steps of gradient descent. In practice, this is not important because it works for most cases just fine enough. For a  field where very high accuracy is a must, this kind of machine learning would be the wrong approach anyway. Therefore a logic-based attempt would lead to a state of the art result \cite{Russell} \cite{conf/mkm/KohlhaseKMT17}.

\begin{figure}[htp]
	\centering
	\fbox{\includegraphics[width=0.65\linewidth]{photo/25_gradident_decent}}
	\caption{Shows the path to a local minimum while using the parameters gradients to compute the path \cite{Gradient47:online}}
	\label{fig:gd}
\end{figure}

\subsection{Integated Gradient}
\label{subsec:ig}

A gradient-based explanation method tries to explain a given prediction by using the gradient of the output concerning the input features. The gradient is not only used here to optimize the weights of the parameters but also to see which features have been adjusted the most and are therefore most important. \\

The goal is to know how a decision was made. So it can be concluded afterwards why this decision was made, so the gradient is used again. As already mentioned, the gradient points in the direction of the next local minimum for each parameter. If it is observable which parameters are adjusted more than others, a statement which parameters are more important than others can be made. Since each parameter is assigned to an input value, it can be concluded how important this value is to reach the desired minimum as fast as possible. An efficient implementation of a deep learning approach is essential to save computing power. An exact calculation costs an enormous amount of resources, which is why the gradients are not calculated in this step but estimated \cite{molnar2019} \cite{TjoaGuan} \cite{Mukund}.\\

Figure \ref{fig:ig} shows three paths between a baseline (r1 , r2) and an input (s1, s2). Path P2, used by integrated gradients, simultaneously moves all features from off to on. Path P1 moves along the edges, turning features on in sequence. Other paths like P1 along different edges correspond to different sequences. SHAP computes the expected attribution over all such edge paths like P1.\\

\begin{figure}[htp]
	\centering
	\fbox{\includegraphics[width=0.65\linewidth]{photo/26_integradet_gradient.png}}
	\caption{Shows different paths between a baseline and an input to compare it the Integrated Gradient (P2) with the SHalpey Additive exPlanations\cite{TjoaGuan}}
	\label{fig:ig}
\end{figure}

The integrated gradient approach tries to estimate Aumann-Shapley values which are as close as possible to the calculated values. The integrated gradient works by assuming there is a straight line, from the actual input (e.g. the image of a shop owner which tires of uploading it to Instagram) to a specific baseline input (e.g. a black image). The gradient of the prediction concerning input features would be integrated along this path \cite{Mukund}.\\
The integrated gradient is also a method where the input varies along a straight line between the baseline and the input. At the same time, the prediction moves from uncertainty to certainty (the final result, e.g. for 98\% the image shows the "Hiterlergruß". At each point on this path, the gradient is used to attribute the change in the prediction probability back to the input features. So, integrated gradient aggregates these gradients along the paths integral \cite{mudrakarta-etal-2018-model}.
Figure \ref{fig:igiu} visualizes who this method would look like after applying it to a few examples.

\begin{enumerate}
	\item Choose any image as a baseline (e.g. black image with each pixel 0)
	\item Make images brighter as long as they become the input again  (Figure \ref{fig:step2})
	\item Compare the final output (as moving from certainty to uncertainty) to the path of the images (from the black baseline to the input, Figure \ref{fig:step3})
	\item We want to know then the slope of the score vs intensity graph doesn't remain stagnant (Interesting Gradients)
	\item The Input images get changed such that the interesting gradients can be seen in there (Figure \ref{fig:step5})
\end{enumerate}

\begin{figure}[!htp]
	\centering
	\fbox{\includegraphics[width=0.85\linewidth]{photo/27_integradet_gradient_step_1}}
	\caption{Three different steps along the path from the baseline to the input}
	\label{fig:step2}
\end{figure}

\begin{figure}[!htp]
	\centering
	\fbox{\includegraphics[width=0.85\linewidth]{photo/28_integradet_gradient_step_2}}
	\caption{compression from the two paths (along the result and along the baseline to output)}
	\label{fig:step3}
\end{figure}

\begin{figure}[!htp]
	\centering
	\fbox{\includegraphics[width=0.85\linewidth]{photo/29_integradet_gradient_step_3}}
	\caption{result of the Integrated Gradient}
	\label{fig:step5}
\end{figure}

As the Integrated Gradient approximates Aumann-Sapley values the function which estimates the values must be a piecewise differentiable function of the input features.\\

Because the method should make sensible feature attributions a sufficient baseline is crucial. For example, if a black image is selected as a baseline, the integrated gradient would not choose attribute importance to a completely black pixel in an actual image. If black pixels are not necessary, e.g. because just the frame of the shop owners image is black and the rest of the image is only bright, this will work perfectly fine. But in general, the baseline value should both have a near-zero prediction, and also faithfully represent a complete absence of signal.\\

\begin{figure}[htp]
	\centering
	\fbox{\includegraphics[width=0.85\linewidth]{photo/30_integradet_gradient_step_4}}
	\caption{The integrated gradient applied to sample of images and visualized to demonstrate which pixels are important (brighter) and which pixels are less important (darker)}
	\label{fig:igiu}
\end{figure}

\subsection{Expected Gradient Approach}
\label{subsec:expected_gradient_approach}
The expected Gradient Approach combines the two methods which were described above. It is useful because the two approaches above generate noisy data. While in the two methods above, one single example is used as a reference value. This approach allows using the whole dataset. It makes the resulting values easier to interpret.

It tries to combine a multitude of ideas from integrated Gradients, SHapley Additive exPlanations (SHAP) and libraries such as SHAP-Library implement some brilliant approximations and samplings. To cover up all their work would fill an entire mater thesis with ease so this part is left out. Important to understand is the result which is provided by software library such that it can be interpreted later on the course of this thesis.

An intuitive way to understand the expected gradient values is following illustration: The feature values are chefs who are entering a hotel kitchen step by step in random order. All chefs (feature values) in the room contributing to the lunch (= contribute to the prediction, which is predicted by a machine learning model). The expected gradient values are equal in the same grade given for the meal which got prepared in the kitchen. This value is the average change in the grading. That mentioned change is defined as follow: Grade revived by a combination of cooks which are already in the kitchen if the actual chef is entering. More precisely, the average of all different combinations of chefs which are in the kitchen, while the specific chef comes into the kitchen.

Figure \ref{fig:gradient_imagenet_plot} shows how this can be visualized on an overlay on image predictions. Especially, it can be seen which regions on the image are responsible for the predicted probability of being part of a class (red) or not (blue).

\begin{figure}[htp]
	\centering
	\fbox{\includegraphics[width=0.8\linewidth]{photo/31_gradient_imagenet_plot}}
	\caption{The shap applied to sample of images and visualized to demonstrate which pixels are important (red) and which pixels are less important (blue) \cite{shaoshan91:online}}
	\label{fig:gradient_imagenet_plot}
\end{figure}


\section{Ultimate RISCs}

 In 1956 W. van der Poel  showed how a single instruction was computationally sufficient for all programming needs.
  Describing the instruction set of an existing computer, he
 demonstrated by a process of elimination that  only   one instruction  was actually needed \cite{poel:urisc}.
 The instruction was to subtract a value in memory from an accumulator, rewriting the result back to the memory location.
 If the number subtracted was bigger than the accumulator then the next instruction was skipped.
Bearing no resemblance to common instructions, this Reverse Subtract instruction would be inherently inefficient. This is because of the large number of instructions needed to perform any useful operation.

My project was originally intended to be an implementation of a computer to execute this instruction.
However, I changed my plans on reading an article in the
June 1988 edition of Computer Architecture News \cite{jones:urisc}.
This article described how a computer could be built using an instruction to move
the contents of one memory location to another address.
Active elements within the computer's memory provide
program control and mathematical functions.
This `MOVE' instruction was  far more intuitive, and more efficient  for a computer to support,  so I decided to design and build such a computer.
It offered the following advantages over the older design:-
\begin{itemize}
\item the article provided a much clearer description of the computer
\item machine code programs should be easier to produce
\item the performance of such a computer should be faster due to the ability to
support operations other than subtraction.
 \item it is a  flexible architecture, permitting much further expansion.
\end{itemize}

{\samepage
During the year  research  revealed the existence of three previous implementations of a single instruction computer. Two were based on the van der Poel instruction, while the third was MOVE based ---even though it predated the article I used.
\begin{enumerate}

\item Negev University, 1976 \cite{tabak:risc}\\
		A MOVE based instruction with four index registers was
		implemented in an eight bit wide computer.

\item Manchester University, 1987\\
		A final year CS undergraduate built a computer based upon
		the reverse subtract instruction.

\item Philips Research labs, 1987 \cite{slav:oisc}\\
		A team of people implemented van der Poel's  computer,
		with a PL/0 interpreter to execute pascal code.
		 The performance of this was claimed to be significantly slower
		 than a M68000 based system.

\end{enumerate}
}

A major factor separating my design from these is the fact that
the architecture was specified formally. This provides:-
\begin{itemize}
\item
A specification for software developers to use.
\item
A  simulation of the computer
\item
The ability to verify the hardware against the specification.
\end{itemize}
A fellow student has produced a compiler for the
Ultimate RISC.
This means the final product  consists of a
computer interfaced to a host, which can cross-compile Pascal programs
 onto my computer.  The design process therefore included
 consultations with this other student, to enable
the machine and the compiler to work together.
While more akin to
how an actual computer would be produced than a hardware project
alone, it  meant  all design decisions were  the result of
heated  discussion.


% Part: model-theory
% Chapter: models-of-arithmetic
% Section: non-standard-models

\documentclass[../../../include/open-logic-section]{subfiles}

\begin{document}

\olfileid{mod}{mar}{nst}
\section{Non-Standard Models}

\begin{explain}
We call a !!{structure} for $\Lang{L_A}$ standard if it is isomorphic
to~$\Struct{N}$. If a !!{structure} isn't isomorphic to~$\Struct{N}$,
it is called non-standard.
\end{explain}

\begin{defn}
A !!{structure}~$\Struct{M}$ for $\Lang{L_A}$ is \emph{non-standard}
if it is not isomorphic to~$\Struct{N}$. The !!{element}s $x \in
\Domain{M}$ which are equal to $\Value{\num{n}}{M}$ for some $n \in
\Nat$ are called \emph{standard numbers} (of $\Struct{M}$), and those
not, \emph{non-standard numbers}.
\end{defn}

\begin{explain}
By \olref[stm]{prop:standard-domain}, any standard !!{structure}
for~$\Lang{L_A}$ contains only standard !!{element}s. Consequently, a
non-standard !!{structure} must contain at least one non-standard
element. In fact, the existence of a non-standard !!{element}
guarantees that the !!{structure} is non-standard.
\end{explain}

\begin{prop}
If !!a{structure}~$\Struct{M}$ for $\Lang{L_A}$ contains a
non-standard number, $\Struct{M}$ is non-standard.
\end{prop}

\begin{proof}
Suppose not, i.e., suppose $\Struct{M}$ standard but contains a
non-standard number~$x$. Let $g\colon \Nat \to \Domain{M}$ be an
isomorphism. It is easy to see (by induction on~$n$) that
$g(\Value{\num{n}}{N}) = \Value{\num{n}}{M}$. In other words, $g$ maps
standard numbers of~$\Struct{N}$ to standard numbers
of~$\Struct{M}$. If $\Struct{M}$ contains a non-standard number, $g$
cannot be !!{surjective}, contrary to hypothesis.
\end{proof}

\begin{prob}
Recall that $\Th{Q}$ contains the axioms
\begin{align*}
& \lforall[x][\lforall[y][(\eq[x'][y'] \lif \eq[x][y])]] \tag{$!Q_1$}\\
& \lforall[x][\eq/[\Obj 0][x']] \tag{$!Q_2$}\\
& \lforall[x][(\eq[x][\Obj 0] \lor \lexists[y][\eq[x][y']])] \tag{$!Q_3$}
\end{align*}
Give !!{structure}s~$\Struct{M_1}$, $\Struct{M_2}$, $\Struct{M_3}$ such that
\begin{enumerate}
\item $\Sat{M_1}{!Q_1}$, $\Sat{M_1}{!Q_2}$, $\Sat/{M_1}{!Q_3}$;
\item $\Sat{M_2}{!Q_1}$, $\Sat/{M_2}{!Q_2}$, $\Sat{M_2}{!Q_3}$; and
\item $\Sat/{M_3}{!Q_1}$, $\Sat{M_3}{!Q_2}$, $\Sat{M_3}{!Q_3}$;
\end{enumerate}
Obviously, you just have to specify~$\Assign{\Obj{0}}{M_i}$ and
$\Assign{\prime}{M_i}$ for each.
\end{prob}

\begin{explain}
It is easy enough to specify non-standard !!{structure}s for
$\Lang{L_A}$. For instance, take the structure with !!{domain}~$\Int$
and interpret all non-logical symbols as usual. Since negative numbers
are not values of $\num{n}$ for any~$n$, this structure is
non-standard. Of course, it will not be a \emph{model} of arithmetic
in the sense that it makes the same sentences true
as~$\Struct{N}$. For instance, $\lforall[x][\eq/[x'][\Obj{0}]]$ is
false.  However, we can prove that non-standard models of arithmetic
exist easily enough, using the compactness theorem.
\end{explain}

\begin{prop}
Let $\Th{TA} = \Setabs{!A}{\Sat{N}{!A}}$ be the theory
of~$\Struct{N}$. $\Th{TA}$ has !!a{enumerable} non-standard model.
\end{prop}

\begin{proof}
Expand $\Lang{L_A}$ by a new !!{constant}~$c$ and consider the set of
!!{sentence}s
\[
\Gamma = \Th{TA} \cup \{\eq/[c][\num{0}], \eq/[c][\num{1}],
\eq/[c][\num{2}], \dots\}
\]
Any model~$\Struct{M^c}$ of~$\Gamma$ would contain !!a{element}~$x =
\Assign{c}{M}$ which is non-standard, since $x \neq
\Value{\num{n}}{M}$ for all $n \in \Nat$. Also, obviously,
$\Sat{M^c}{\Th{TA}}$, since $\Th{TA} \subseteq \Gamma$. If we turn
$\Struct{M^c}$ into a !!{structure}~$\Struct{M}$ for $\Lang{L_A}$
simply by forgetting about~$c$, its domain still contains the
non-standard~$x$, and also~$\Sat{M}{\Th{TA}}$. The latter is
guaranteed since $c$ does not occur in~$\Th{TA}$. So, it suffices to
show that $\Gamma$ has a model.

We use the compactness theorem to show that~$\Gamma$ has a model. If
every finite subset of~$\Gamma$ is satisfiable, so
is~$\Gamma$. Consider any finite subset $\Gamma_0 \subseteq
\Gamma$. $\Gamma_0$ includes some !!{sentence}s of~$\Th{TA}$ and some
of the form~$\eq/[c][\num{n}]$, but only finitely many. Suppose $k$ is
the largest number so that $\eq/[c][\num{k}] \in \Gamma_0$. Define
$\Struct{N_k}$ by expanding~$\Struct{N}$ to include the
interpretation~$\Assign{c}{N_k} = k+1$. $\Sat{N_k}{\Gamma_0}$: if $!A
\in \Th{TA}$, $\Sat{N_k}{!A}$ since $\Struct{N_k}$ is just
like~$\Struct{N}$ in all respects except~$c$, and $c$ does not occur
in~$!A$. And $\Sat{N_k}{\eq/[c][\num{n}]}$, since $n \le k$, and
$\Value{c}{N_k} = k+1$. Thus, every finite subset of~$\Gamma$ is
satisfiable.
\end{proof}

\end{document}


\documentclass{classrep}
\usepackage[utf8]{inputenc}
\frenchspacing

\usepackage{graphicx}
\usepackage[usenames,dvipsnames]{color}
\usepackage[hidelinks]{hyperref}
\usepackage{lmodern}
\usepackage{placeins}
\usepackage{url}
\usepackage{amsmath, amssymb, mathtools}
\usepackage{listings}
\usepackage{fancyhdr, lastpage}
\usepackage{amsfonts}

\pagestyle{fancyplain}
\fancyhf{}
\renewcommand{\headrulewidth}{0pt}
\cfoot{\thepage\ / \pageref*{LastPage}}

%--------------------------------------------------------------------------------------%
\studycycle{Applied Information Technology, 2 cycle}
\coursesemester{II}

\coursename{Soft Computing Laboratory}
\courseyear{2021/2022}

\courseteacher{dr inż. Kamil Stokfiszewski}
\coursegroup{Wednesday, 8:30}

\author{%
    \studentinfo[239671@edu.p.lodz.pl]{Jan Karwowski}{239671}\\
    \studentinfo[239676@edu.p.lodz.pl]{Kamil Kowalewski}{239676}\\
}

\title{Assignment 4.: Simple Genetic Algorithm}

\begin{document}
    \maketitle
    \thispagestyle{fancyplain}

    \tableofcontents
    \newpage

    \section{Main goal} \label{main_goal} {
        The main goal of this task is to prepare implementation of Simple Genetic
        Algorithm (SGA). The algorithm have to optimize function shown below:
        \begin{equation}
            f(x)=(e^x sin(10 \pi x)+1)/x+5
        \end{equation}
        on the $[0.5, 2.5] \subset \mathbb{R}$. This function is presented on fig.
        \ref{fig:fitness_function}.

        \begin{figure}[!htbp]
            \centering
            \includegraphics[width=0.95\textwidth]{img/fitness_function.jpg}
            \caption{Fitness function}
            \label{fig:fitness_function}
        \end{figure}
        \FloatBarrier
    }

    \section{Theoretical background} \label{theory} {
        Simple Genetic Algorithm (SGA) is a very basic and first implementation of genetic
        algorithm, which a general concept of algorithm based on the law of natural selection. It
        was created to find optimal solution of very compound problems, i.e. optimize complicated,
        multimodal functions of many variables. Since this particular version of genetic algorithm
        is ,,standardized'', there is no need to introduce more details. It should be enough to
        mark, that our implementation is possibly consistent with the original one. Any
        changes and variations are described below.
    }

    \section{Implementation} \label{implementation} {
        SGA was implemented in Python programming language, using common \emph{numpy} library.
        Fitness function and optimization range are hardcoded and taken from the task description.
        Program accepts 5 parameters:
        \begin{enumerate}
            \item solution accuracy as integer number of decimal digits (p)
            \item number of populations - number of SGA iterations (N)
            \item population size (M)
            \item crossover probability (pc)
            \item mutation probability (pm)
        \end{enumerate}

        The main function starts from computing required number of bits, necessary to encode all
        possible solutions with given accuracy and range. This number should be lower then $64$ -
        the max possible integer size. In fact, all the solutions are stored as $64$-bits integer
        values and all genetic operators are implemented using bitwise operators.
    }

    \section{Experiments and results} \label{results} {
        Two series of experiments were conducted. The first one explores impact of particular
        parameters on SGA's solution finding ability. In scope of this series for each set of
        parameters algorithm was run $50$ times and successful runs were counted. When the best
        solution's fitness from the very last population is greater then $10$ (see fig.
        \ref{fig:fitness_function}, then such a run is marked as successful, i.e. SGA found global
        optimum of the fitness function. Results of this series are presented in table
        \ref{tab:series_1_results}.

        \begin{table}[!htbp]
            \centering
            \caption{Results of first series of experiments}
            \begin{tabular}{|c|c|c|c|c|c|c|}
                \hline
                # & p & N & M & pc & pm & successful runs \\ \hline
                1 & 3 & 100 & 50 & 0.5 & 0.2 & 24 \\
                2 & 3 & 100 & 80 & 0.5 & 0.2 & 25 \\
                3 & 3 & 100 & 200 & 0.5 & 0.2 & 36 \\
                4 & 3 & 200 & 50 & 0.5 & 0.2 & 23 \\
                5 & 3 & 30 & 50 & 0.5 & 0.2 & 23 \\
                6 & 5 & 100 & 50 & 0.5 & 0.2 & 16 \\
                7 & 3 & 100 & 50 & 0.7 & 0.2 & 19 \\
                8 & 3 & 100 & 50 & 0 & 1 & 30 \\
                9 & 3 & 100 & 50 & 0 & 1 & 17 \\
                10 & 3 & 100 & 50 & 0.9 & 0.05 & 24 \\ \hline
            \end{tabular}
            \label{tab:series_1_results}
        \end{table}
        \FloatBarrier

        The second series of experiments explores SGA's behaviours during single run. Two
        experiments/runs was conducted and each of them is presented on a single chart, which
        presents best solutions' fitness (vertical axis) for next populations/iterations (horizontal
        axis) - fig. \ref{fig:low_mutation_probability} and \ref{fig:high_mutation_probability}.

        \begin{figure}[!htbp]
            \centering
            \includegraphics[width=1.1\textwidth]{img/low_mutation_probability.jpg}
            \caption{Low mutation probability}
            \label{fig:low_mutation_probability}
        \end{figure}

        \begin{figure}[!htbp]
            \centering
            \includegraphics[width=1.1\textwidth]{img/high_mutation_probability.jpg}
            \caption{High mutation probability}
            \label{fig:high_mutation_probability}
        \end{figure}
        \FloatBarrier
    }

    \section{Summary and conclusions} \label{summary} {
        In table \ref{tab:series_1_results} we can see results of experiments, where impact of
        particular parameters on SGA behaviour was explored. The very first conclusion, or rather a
        \emph{remark}, is that these results are not very representative and should NOT be
        considered as important argument, when selecting best SGA hyperparameters. This is caused by
        strong stochastic nature of this algorithm, especially in context of this particular problem
        (fitness function). I.e. there is no strict rules, when this algorithm is able to find
        global optimum. On the other hand some observations can be made.

        First of all, when population size increase, then SGA can easier find global optimum. We can
        see it in rows 2. and 3. in compare to row 1., which is a reference. Another conclusion is,
        that changing number of iterations (number of populations) is not very important. This
        conclusion is related to this particular problem, and may be not very general. But in fact,
        in row 4. and 5. we can see no special difference, when compared to row 1. It could be
        caused by the fact, that this problem (fitness function) is relatively simple and global
        optimum may be found in the on of the first iterations. In row 6. required accuracy was
        increased and it noticeably degradates SGA - it seems to be intuitive behaviour. In the last
        rows (7, 8, 9, 10) probabilities of crossover and mutation was changed. We noticed that
        changing these values may have positive and negative impact as well, completely
        randomly in most cases.

        It is worth mentioning, that increasing mutation probability make searching process less
        stable, and it can produce completely different results, sometimes very good and sometimes
        very bad - it's just a random search. When mutation probability is greater, then searching
        process is more stable, it used to stay in good solutions, when it find some of them. We can
        see it in the results of second series of experiments in fig.
        \ref{fig:low_mutation_probability} and fig. \ref{fig:high_mutation_probability}.
    }

    \begin{thebibliography}{0}
        % @formatter:off
        \bibitem{instruction}{Labolatory instruction, URL: https://ftims.edu.p.lodz.pl/pluginfile.php/75442/\\mod\_resource/content/1/soft\_comp\_lab\_04\_SGA.pdf}
        % @formatter:on
    \end{thebibliography}

\end{document}


\documentclass[10pt]{report}

\usepackage{subcaption} % for subfigures
\usepackage{amsthm} % for QED
%\usepackage{algpseudocode} % for pseudo-code
\usepackage{mathtools} % for delimiter

\usepackage{listings} % for code
\lstset
{
	language=Matlab,
	frame=single,
	basicstyle=\footnotesize,
	captionpos=b,
	numbers=left,
	stepnumber=1,
	showstringspaces=false,
	tabsize=4,
	breaklines=true,
	breakatwhitespace=false,
}

\usepackage{float} % for figure [H]
\usepackage{booktabs} % for tabular
\usepackage{caption} % for \caption*
\usepackage[export]{adjustbox} % for valign=t
\usepackage{array} % for column type m
\usepackage{verbatim}
\usepackage{graphicx}
\graphicspath{ {imgs/} }
\usepackage{fancyhdr}
\usepackage{amssymb}
\usepackage{amsmath}

%%%%%% Pagination
\setlength{\topmargin}{-.3 in}
\setlength{\oddsidemargin}{0in}
\setlength{\evensidemargin}{0in}
\setlength{\textheight}{9.in}
\setlength{\textwidth}{6.5in}

%Title page
\newcommand{\hwTitle}{Homework \#6}
\newcommand{\hwCourse}{Numerical Differential Equations/Computational Mathematics II}
\newcommand{\hmwkClassInstructor}{Professor Shuwang Li}

\title{
	\vspace{2in}
	\textmd{\textbf{\hwCourse\\\hwTitle}}\\
	\vspace{0.3in}\large{\textit{\hmwkClassInstructor}}
	\vspace{3in}
}

%\title{Homework 1}
\author{\textbf{Zhihao Ai}}
\date{}

%Header setting.
\pagestyle{fancy}
\fancyhead[L]{Zhihao Ai}
\fancyhead[C]{Math 478}
\fancyhead[R]{Homework 6}
%%%%%%

%Global setting.
%\everymath{\displaystyle}
\setlength\parindent{0pt}

%Custom general commands.
\newcommand{\ds}{\displaystyle}
\newcommand{\ts}{\textstyle}
\newcommand{\f}[1] {f\left(#1\right)}
\newcommand{\eva}[2] {\left. #1 \right|_{#2}}
\newcommand{\dintt}[4] {\int_{#1}^{#2} #3 d#4}

\newcolumntype{N}{ >$ c <$}
\newcolumntype{M}[1]{>{\centering\arraybackslash $}m{#1}<{$}}

\newcommand{\abs}[1] {\left| #1 \right|}

\DeclarePairedDelimiter\autoparen{(}{)}
\newcommand{\pa}[1]{\autoparen*{#1}}
\DeclarePairedDelimiter\autodvert{\Vert}{\Vert}
\DeclarePairedDelimiter{\floor}{\lfloor}{\rfloor}
\newcommand{\norm}[1]{\autodvert*{#1}}

\begin{document}

\maketitle

\section*{Question 1}
% Fundamental concepts on error control..
\begin{enumerate}
	\item
	(Problem 6.1) Find the error constants for the Adams-Bashforth method (2.7) and for Adams-Moulton methods with $s = 2, 3$.

	3-step Adams-Bashforth: $y_{n+3} = y_{n+2} + h\pa{ \frac{23}{12}f(t_{n+2}, y_{n+2}) - \frac{4}{3}f(t_{n+1}, y_{n+1}) + \frac{5}{12}f(t_{n}, y_{n})}$
	\[
	\rho(w) = w^3 - w^2,\quad \sigma(w) = \frac{23}{12}w^2 - \frac{4}{3}w + \frac{5}{12}
	\]
	Let $x=w-1$,
	\begin{align*}
	\rho(w) - \sigma(w) \ln{w}
	&= x(x+1)^2 - \pa{\frac{23}{12}x^2 + \frac{5}{2}x + 1} \pa{x-\frac{1}{2}x^2 + \frac{1}{3}x^3 + \cdots}\\
	&= \frac{3}{8}x^4 + O(x^5)
	\end{align*}
	Therefore the error constant for three-step Adams-Bashforth is $\frac{3}{8}$.

	2-step Adams-Moulton: $y_{n+2} = y_{n+1} + h\pa{ \frac{5}{12}f(t_{n+2}, y_{n+2}) + \frac{2}{3}f(t_{n+1}, y_{n+1}) - \frac{1}{12}f(t_{n}, y_{n})}$
	\[
	\rho(w) = w^2 - w,\quad \sigma(w) = \frac{5}{12}w^2 + \frac{2}{3}w - \frac{1}{12}
	\]
	Let $x=w-1$,
	\begin{align*}
	\rho(w) - \sigma(w) \ln{w}
	&= x(x+1) - \pa{\frac{5}{12}x^2 + \frac{3}{2}x + 1} \pa{x-\frac{1}{2}x^2 + \frac{1}{3}x^3 + \cdots}\\
	&= -\frac{1}{24}x^4 + O(x^5)
	\end{align*}
	Therefore the error constant for three-step Adams-Bashforth is $-\frac{1}{24}$.

	3-step Adams-Moulton: $y_{n+3} = y_{n+2} + h( \frac{3}{8}f(t_{n+3}, y_{n+3}) + \frac{19}{24}f(t_{n+2}, y_{n+2}) - \frac{5}{24}f(t_{n+1}, y_{n+1})$\\
	$ + \frac{1}{24}f(t_{n}, y_{n})$
	\[
	\rho(w) = w^3 - w^2,\quad \sigma(w) = \frac{3}{8}w^3 + \frac{19}{24}w^2 - \frac{5}{24}w + \frac{1}{24}
	\]
	Let $x=w-1$,
	\begin{align*}
	\rho(w) - \sigma(w) \ln{w}
	&= (x^3 + 2x^2 + x) - \pa{\frac{3}{8}x^3 + \frac{23}{12}x^2 + \frac{5}{2}x + 1} \pa{x-\frac{1}{2}x^2 + \frac{1}{3}x^3 + \cdots}\\
	&= -\frac{19}{720}x^5 + O(x^6)
	\end{align*}
	Therefore the error constant for three-step Adams-Moulton is $-\frac{19}{720}$.

	\item
	(Problem 6.2) Prove that the error constant of the $s$-step backward differentiation formula is $-\beta/(s + 1)$, where $\beta$ was defined in (2.14).

	For a BDF, we have
	\[
	\rho(w) = \beta \sum_{m=1}^{s} \frac{1}{m} w^{s-m} (w-1)^m,\quad \sigma(w) = \beta w^s
	\]
	Let $x = w - 1$,
	\begin{align*}
		\rho(w) - \sigma(w) \ln{w}
		&= \beta \sum_{m=1}^{s} \frac{1}{m} w^{s-m} (w-1)^m - \beta w^s \ln{w}\\
		&= \beta \pa{\sum_{m=1}^{s} \frac{1}{m} (x+1)^{s-m} x^m - (x+1)^s (x-\frac{1}{2}x^2 + \frac{1}{3}x^3 - \frac{1}{4}x^4 + \cdots)}
	\end{align*}
	Since $s$-step BDF is of order $s$, the terms $x^k, k=1,2,\dots,s$ on the RHS cancel out. The error constant equals the coefficient for $x^{s+1}$, which is
	\[
	-\sum_{k=0}^{s}\binom{s}{k} \frac{(-1)^k}{k+1} = -\frac{1}{s+1}
	\]
	Then we have
	\[
	\rho(w) - \sigma(w) \ln{w} = -\frac{\beta}{s+1}(w-1)^{s+1} + O(\abs{w-1}^{s+2})
	\]
	Therefore the error constant of the $s$-step backward differentiation formula is $-\beta/(s + 1)$.

	\item
	(Problem 6.4) Prove that the embedded RK pair
	\[
	\renewcommand{\arraystretch}{1.2}
	\begin{array}{c|ccc}
	0 &&&\\
	\frac{1}{2} & \frac{1}{2} &&\\
	1 & -1 & 2 &\\ \hline
	& 0 & 1 &\\ \hline
	& \frac{1}{6} & \frac{2}{3} & \frac{1}{6}
	\end{array}
	\]
	combines a second-order and a third-order method.

	For the tableau of RK2,
	\begin{align*}
		y(t_{n+1})
		&= y(t_n + h)\\
		&= y(t_n) + hy'(t_n) + \frac{1}{2}h^2y''(t_n) + O(h^3)\\
		&= y(t_n) + hf + \frac{1}{2}h^2(f_t + f_y f) + O(h^3)\\
		y_{n+1}
		&= y_n + hf\pa{t_n + \frac{1}{2}h, y_n + \frac{1}{2}hf}\\
		&= y(t_n) + h(f + \frac{1}{2}hf_t + \frac{1}{2}hf_yf + O(h^2))\\
		&= y(t_n) + hf + \frac{1}{2}h^2(f_t + f_y f) + O(h^3)
	\end{align*}
	Therefore the RK2 method is second-order.
	For the tableau of RK3, consider $y'=f(y)$ and we have
	\begin{align*}
	y(t_{n+1})
	&= y(t_n + h)\\
	&= y(t_n) + hy'(t_n) + \frac{1}{2}h^2y''(t_n) + \frac{1}{6}h^3y'''(t_n) + O(h^4)\\
	&= y(t_n) + hf + \frac{1}{2}h^2f_y f + \frac{1}{6}h^3\pa{f_{yy}f^2 + f_y^2f} + O(h^4)\\
	k_1
	&= f\\
	k_2
	&= f(y_n + \frac{1}{2}hf)\\
	&= f + \frac{1}{2}h f_y f + \frac{1}{8}h^2 f_{yy} f^2 + O(h^3)\\
	k_3
	&= f\pa{y_n - hf + 2h\pa{f + \frac{1}{2}h f_y f + \frac{1}{8}h^2 f_{yy} f^2 + O(h^3)}}\\
	&= f\pa{y_n + hf + h^2f_yf + O(h^3)}\\
	&= f+hf_yf + h^2 f_y^2 f + \frac{1}{2}h^2 f_{yy}f^2 + O(h^3)\\
	y_{n+1}
	&= y_n + h\pa{\frac{1}{6}f + \frac{2}{3}\pa{f + \frac{1}{2}h f_y f + \frac{1}{8}h^2 f_{yy} f^2 + O(h^3)} + \frac{1}{6}\pa{f+hf_yf + h^2 f_y^2 f + \frac{1}{2}h^2 f_{yy}f^2 + O(h^3)}}\\
	&= y(t_n) + hf + \frac{1}{2}h^2f_y f + \frac{1}{6}h^3\pa{f_{yy}f^2 + f_y^2f} + O(h^4)
	\end{align*}
	Therefore the RK3 method is third-order. Hence the embedded RK pair combines a second-order and a third-order method.
\end{enumerate}

\section*{Computer Assignment}
Consider a linear BVP: $y'' = y - x$ on $0<x<1$ and $y(0)=0, y(1)=0$. The exact solution to this problem is $y(x) = x + \frac{e}{e^2-1}(e^{-x}-e^x)$.
\begin{enumerate}
	\item [(a)]
	Implement the linear shooting method to solve the problem using 10 subintervals. Compute the numerical errors for each node point.
	\lstinputlisting{shooting.m}
	\begin{table}[H]
		\centering
		\begin{tabular}{*{4}{N}}
			\toprule
			x & u(x) & y(x) & \abs{u(x)-y(x)}\\ \midrule
			0.0 & 0.000000 & 0.000000 & 0.000000\\
			0.1 & 0.017353 & 0.014766 & 0.002587\\
			0.2 & 0.033704 & 0.028680 & 0.005025\\
			0.3 & 0.048040 & 0.040878 & 0.007162\\
			0.4 & 0.059328 & 0.050483 & 0.008845\\
			0.5 & 0.066505 & 0.056591 & 0.009915\\
			0.6 & 0.068467 & 0.058260 & 0.010207\\
			0.7 & 0.064057 & 0.054507 & 0.009550\\
			0.8 & 0.052055 & 0.044295 & 0.007760\\
			0.9 & 0.031164 & 0.026518 & 0.004646\\
			1.0 & 0.000000 & 0.000000 & 0.000000\\
			\bottomrule
		\end{tabular}
		\caption*{Numerical errors for shooting method}
	\end{table}

	\item [(b)]
	Implement the finite difference method to solve the problem using 10 subintervals. Compute the numerical errors for each node point.
	\lstinputlisting{fd.m}
	\begin{table}[H]
		\centering
		\begin{tabular}{*{4}{N}}
			\toprule
			x & u(x) & y(x) & \abs{u(x)-y(x)}\\ \midrule
			0.0 & 0.000000 & 0.000000 & 0.000000\\
			0.1 & 0.014755 & 0.014766 & 0.000011\\
			0.2 & 0.028658 & 0.028680 & 0.000021\\
			0.3 & 0.040848 & 0.040878 & 0.000031\\
			0.4 & 0.050446 & 0.050483 & 0.000038\\
			0.5 & 0.056548 & 0.056591 & 0.000043\\
			0.6 & 0.058216 & 0.058260 & 0.000044\\
			0.7 & 0.054466 & 0.054507 & 0.000042\\
			0.8 & 0.044260 & 0.044295 & 0.000034\\
			0.9 & 0.026498 & 0.026518 & 0.000021\\
			1.0 & 0.000000 & 0.000000 & 0.000000\\
			\bottomrule
		\end{tabular}
		\caption*{Numerical errors for finite difference method}
	\end{table}

	\item [(c)]
	Comment on what you get in (a) and (b).

	The numerical errors of the finite difference method are much smaller than those of the shooting method. Besides, in this case, less computation is required for finite difference method than that for shooting method.

	\item [(d)]
	Suppose we are solving a nonlinear BVP, what changes are you going to make to the routines you developed in (a) and (b)?

	For shooting method, we will need to use iterations like secant method or Newton's method to get the correct initial slope. Finite difference method also needs a great amount of iterations to solve the equations.
\end{enumerate}

\end{document}