\documentclass[12pt,english]{article}
\usepackage{mathpazo}
\renewcommand{\familydefault}{\rmdefault}
\usepackage[T1]{fontenc}
\usepackage[latin9]{inputenc}
\usepackage{geometry}
\geometry{verbose,tmargin=1in,bmargin=1in,lmargin=1in,rmargin=1in}
\usepackage{color}
\usepackage[dvipsnames]{xcolor}
\definecolor{oucrimson}{RGB}{132.,22.,23.}
\usepackage{babel}
\usepackage{booktabs}
\usepackage[authoryear]{natbib}

% Useful Packages
%\usepackage{bookmark} % For speedier bookmarks
\usepackage{amsthm}   % For detailed theorems
\usepackage{amssymb}  % For fancy math symbols
\usepackage{amsmath}  % For awesome equations/equation arrays
\usepackage{array}    % For tubular tables
\usepackage{longtable}% For long tables
\usepackage[flushleft]{threeparttable} % For three-part tables
\usepackage{multicol} % For multi-column cells
\usepackage{graphicx} % For shiny pictures
\usepackage{subfig}   % For sub-shiny pictures
\usepackage{enumerate}% For cusomtizable lists
\usepackage{pstricks,pst-node,pst-tree,pst-plot} % For trees

\usepackage[unicode=true,pdfusetitle,bookmarks=true,bookmarksnumbered=false,bookmarksopen=false,breaklinks=true,pdfborder={0 0 0},pdfborderstyle={},backref=false,colorlinks=true]{hyperref}
\hypersetup{citecolor=oucrimson,filecolor=oucrimson,linkcolor=oucrimson,urlcolor=oucrimson}
\usepackage{breakurl}

\begin{document}

\title{Introduction to Optimization}
\author{Econ 5253: Data Science for Economists\\ Prof. Tyler Ransom\\ University of Oklahoma}
\date{Spring 2020}
\maketitle

\section{What is ``optimization''?}

The word ``optimization'' generally means ``the action of making the best or most effective use of a situation or resource'' (Google's dictionary). In data science, this could refer to two different things:
\begin{enumerate}
    \item Streamlining some sort of automated process
    \begin{enumerate}
        \item e.g. ``We \emph{optimized} the web scraping script so that it now runs in half the time and has 40\% fewer lines of code.''
    \end{enumerate}
    \item Finding the optimum of an objective function (or ``selecting the best element from some set of available alternatives,'' according to Wikipedia)
    \begin{enumerate}
        \item e.g. ``Our preferred model estimate is the set of parameters that \emph{optimizes} the sum of the squared distance between the data and the model.'' {[}in this case, ``optimize'' would mean ``minimize''{]}
    \end{enumerate}
\end{enumerate}
The \textbf{second} of these two definitions is what you should think of when you hear ``optimization'' in a data science context.

\section{Ways to optimize}
There are many different ways we can optimize a particular objective function. The two most common ways---which we will discuss today---are known as \textbf{minimum distance estimation} and \textbf{maximum likelihood estimation}. These are two different but related ways of expressing the objective function.

In any optimization problem, the data scientist must also choose an \emph{algorithm} with which to optimize, if the optimum cannot be expressed in a math formula. The three most popular optimization algorithms are:
\begin{enumerate}
    \item gradient descent
    \item stochastic gradient descent
    \item L-BFGS
\end{enumerate}
We'll talk about these algorithms a bit later in the course.

\section{Math: the classical linear model}

Let's suppose we have the classical linear model:

\begin{equation}
y_{i}=x_{i}^{\prime}\beta+\varepsilon_{i}\label{eq:1}
\end{equation}
where $y_{i}$ is the outcome for observation (equiv. training example) $i$, $x_{i}$ are the covariates (equiv. features) of observation $i$, $\beta$ are parameters to be estimated, and $\varepsilon_{i}$ is the error term.

We can rewrite (\ref{eq:1}) in vector form:
\begin{equation}
y=X\beta+\varepsilon\label{eq:2}
\end{equation}
where now $y$ is a vector (i.e. column of data), $X$ is a matrix (i.e. a table of data), $\beta$ are exactly the same as before, and $\varepsilon$ is an (unobservable) error term vector.

\subsection{Optimization method 1: least squares}

We can define our objective function so that we minimize the (squared) distance between $X\beta$ and $y$. In this sense, minimizing the distance will give us the best fit (i.e. optimum) of the model given the data. In formal terms, we want to
\begin{align}
 & \min_{\beta}\sum_{i}\left(y_{i}-x_{i}^{\prime}\beta\right)^{2}\nonumber \\
= & \min_{\beta}\sum_{i}\varepsilon_{i}^{2}\nonumber \\
= & \min_{\beta}\varepsilon^{\prime}\varepsilon\label{eq:3}\\
= & \min_{\beta}\left(y-X\beta\right)^{\prime}\left(y-X\beta\right)\nonumber
\end{align}
Note that the four equations above are all equivalent. We can conserve on notation by utilizing vector and matrix notation (e.g. $\varepsilon^{\prime}\varepsilon$ is the dot product of the $\varepsilon$ vector with itself).\footnote{A related objective function to the one above is to minimize the \emph{absolute value} of the distance between $y$ and $X\beta$. This model is known as \emph{least absolute deviations} and the parameter estimates at the optimum correspond to the \emph{median regression} model rather than the OLS regression model (the linear prediction of which passes through the mean).}

\subsubsection{Using calculus to solve for the least squares estimator}

We can solve this optimization problem using calculus:

\begin{align*}
 & \min_{\beta}\left(y-X\beta\right)^{\prime}\left(y-X\beta\right)\\
    = & \min_{\beta}\left(y^{\prime}y-\beta^{\prime}X^{\prime}y-y^{\prime}X\beta+\beta^{\prime}X^{\prime}X\beta\right)
\end{align*}
How do we use calculus to find an optimum? Take first-order conditions and set them equal to zero:
\begin{align}
\left[\beta\right]: & -\frac{\partial}{\partial\beta}\beta^{\prime}X^{\prime}y-\frac{\partial}{\partial\beta}y^{\prime}X\beta+\frac{\partial}{\partial\beta}\beta^{\prime}X^{\prime}X\beta\label{eq:4}\\
0= & -X^{\prime}y-X^{\prime}y+2\left(X^{\prime}X\right)\hat{\beta}\nonumber \\
0= & -2X^{\prime}y+2\left(X^{\prime}X\right)\hat{\beta}\nonumber \\
X^{\prime}y= & \left(X^{\prime}X\right)\hat{\beta}\nonumber \\
\hat{\beta}= & \left(X^{\prime}X\right)^{-1}X^{\prime}y\nonumber
\end{align}
which is known as the \emph{OLS estimator}.

\subsubsection{Checking second-order conditions}

How do we know if our optimum is a minimum or a maximum? We need to check that the second-order conditions are satisfied. For a \emph{minimum}, we need the second derivative to be \emph{positive}. Taking the derivative of our first-order condition with respect to beta, we get:
\begin{align}
0 & <\frac{\partial^{2}}{\partial\beta\partial\beta^{\prime}}\left(y-X\beta\right)^{\prime}\left(y-X\beta\right)\label{eq:5}\\
0 & <\frac{\partial}{\partial\beta^{\prime}}\left[-X^{\prime}y-X^{\prime}y+2\left(X^{\prime}X\right)\hat{\beta}\right]\nonumber \\
0 & <2\left(X^{\prime}X\right)\nonumber \\
\left(X^{\prime}X\right) & >0\nonumber
\end{align}
This condition is satisfied so long as the following conditions hold:
\begin{enumerate}
\item $X$ has more rows than columns (i.e. $N\geq K$, where $N$ is the number of observations and $K$ is the number of columns in $X$)
\item $X$ cannot have perfectly collinear columns
\end{enumerate}
In linear algebra parlance, if these conditions are met, then we say that the matrix $\left(X^{\prime}X\right)$ has \emph{full rank},
or that $\left(X^{\prime}X\right)$ is \emph{nonsingular}.

\subsection{Optimization method 2: maximum likelihood}

Another option for finding the optimum of our linear model is by using what is called maximum likelihood. Let's review what maximum likelihood is and then derive it for this simple linear model.

\subsubsection{Maximum likelihood estimation of the mean and variance of the normal distribution}

Suppose we have $N$ observations of a random variable, call it $X$, which we know to be independently and identically distributed (\emph{iid}) according to the Normal distribution. This means that, for any given observation $x_{i}$, we will have the following probability density function:
\begin{equation}
f\left(x_{i};\mu,\sigma\right)=\frac{1}{\sqrt{2\pi\sigma^{2}}}\exp\left(-\frac{\left(x_{i}-\mu\right)^{2}}{2\sigma^{2}}\right)\label{eq:6}
\end{equation}
where we put $\mu,\sigma$ on the right hand side of the semicolon to denote that we are interested in estimating these parameters from our $N$ observations of variable $X$.

Why do we care about the probability density function? We want to use our data to find what the mean and variance would be if we assume that our collected data follows a normal distribution.

\paragraph{The likelihood function}

We define what is called the \emph{likelihood function}, which in plain terms is the likelihood that\textemdash given our data and assumption of normal distribution of that data\textemdash the mean is equal to $\mu$ and the standard deviation is equal to $\sigma$.
\begin{align}
\mathcal{L}\left(\mu,\sigma;x_{1},x_{2},\ldots,x_{N}\right)= & \prod_{i=1}^{N}f\left(x_{i};\mu,\sigma\right)\label{eq:7}\\
= & f\left(x_{1};\mu,\sigma\right)f\left(x_{2};\mu,\sigma\right)\cdots f\left(x_{N};\mu,\sigma\right)\nonumber
\end{align}
where here $\prod$ is the Greek capital $\pi$, which is known as the \emph{product operator}. $\prod$ is the multiplication analog of the summation operator $\sum$. We can do a little bit of simplifying:
\begin{align}
\mathcal{L}\left(\mu,\sigma;x_{1},x_{2},\ldots,x_{N}\right)= & \prod_{i=1}^{N}f\left(x_{i};\mu,\sigma\right)\nonumber \\
= & f\left(x_{1};\mu,\sigma\right)f\left(x_{2};\mu,\sigma\right)\cdots f\left(x_{N};\mu,\sigma\right)\nonumber \\
= & \left[\frac{1}{\sqrt{2\pi\sigma^{2}}}\exp\left(-\frac{\left(x_{1}-\mu\right)^{2}}{2\sigma^{2}}\right)\right]\times\cdots\times\left[\frac{1}{\sqrt{2\pi\sigma^{2}}}\exp\left(-\frac{\left(x_{N}-\mu\right)^{2}}{2\sigma^{2}}\right)\right]\label{eq:8}
\end{align}

\begin{figure}[ht!]
\caption{Illustration of a likelihood function}
\label{fig:mle}
\centering
\includegraphics[width=0.5\textwidth]{../Graphics/350px-Apa_fig3}

{\raggedright {\footnotesize Source: \url{reliawiki.org}}}
\end{figure}

If you recall properties of exponents, we can simplify (\ref{eq:8}) as follows:
\begin{align}
\mathcal{L}\left(\mu,\sigma;x_{1},x_{2},\ldots,x_{N}\right)= & \left(\frac{1}{\sqrt{2\pi\sigma^{2}}}\right)^{N}\exp\left(-\frac{\sum_{i=1}^{N}\left(x_{i}-\mu\right)^{2}}{2\sigma^{2}}\right)\nonumber \\
= & \left(\frac{1}{\left(\sqrt{2\pi\sigma^{2}}\right)^{N}}\right)\exp\left(-\frac{\sum_{i=1}^{N}\left(x_{i}-\mu\right)^{2}}{2\sigma^{2}}\right)\label{eq:9}
\end{align}

What does the likelihood function look like? See Figure \ref{fig:mle} for an example where there are two parameters (e.g. $\mu$ and $\sigma^{2}$ in the example above).

\paragraph{The log likelihood function}

Recall that, when ordering a set, one can take a \emph{monotonic transformation} of the set and preserve the ordering. It turns out that we can apply such a transformation to (\ref{eq:9}) by taking the natural logarithm ($\ln$) of both sides. This leaves us with:
\begin{align}
\ln\mathcal{L}\left(\mu,\sigma;x_{1},x_{2},\ldots,x_{N}\right)= & \ln\left[\left(\frac{1}{\left(\sqrt{2\pi\sigma^{2}}\right)^{N}}\right)\exp\left(-\frac{\sum_{i=1}^{N}\left(x_{i}-\mu\right)^{2}}{2\sigma^{2}}\right)\right]\nonumber \\
\ell\left(\mu,\sigma;x_{1},x_{2},\ldots,x_{N}\right)= & \ln\left(\frac{1}{\left(\sqrt{2\pi\sigma^{2}}\right)^{N}}\right)+\ln\left(\exp\left(-\frac{\sum_{i=1}^{N}\left(x_{i}-\mu\right)^{2}}{2\sigma^{2}}\right)\right)\nonumber \\
= & \underbrace{\ln\left(1\right)}_{=0}-\ln\left(\left(2\pi\sigma^{2}\right)^{N/2}\right)-\frac{1}{2\sigma^{2}}\sum_{i=1}^{N}\left(x_{i}-\mu\right)^{2}\nonumber \\
= & -\frac{N}{2}\ln\left(2\pi\sigma^{2}\right)-\frac{1}{2\sigma^{2}}\sum_{i=1}^{N}\left(x_{i}-\mu\right)^{2}\label{eq:10}
\end{align}

Equation (\ref{eq:10}) is known as the \emph{log likelihood function}. Our goal is find the estimated $\mu$ and $\sigma$ that maximize this function.\footnote{Because of monotonicity, the maximum of the likelihood function will be the same as the maximum of the log likelihood function.}

\paragraph{Finding the maximum of the log likelihood function}

To find the maximum likelihood estimates (MLEs) of the log likelihood function, we take the first derivative of $\ell\left(\mu,\sigma;x_{1},\ldots,x_{N}\right)$ with respect to $\mu$ and $\sigma$:
\begin{align}
\frac{\partial}{\partial\mu}\ell\left(\mu,\sigma;x_{1},x_{2},\ldots,x_{N}\right)= & \frac{\partial}{\partial\mu}\left[-\frac{N}{2}\ln\left(2\pi\sigma^{2}\right)-\frac{1}{2\sigma^{2}}\sum_{i=1}^{N}\left(x_{i}-\mu\right)^{2}\right]\nonumber \\
= & -\frac{1}{2\sigma^{2}}\frac{\partial}{\partial\mu}\sum_{i=1}^{N}\left(x_{i}-\mu\right)^{2}\,\,\,\,\ldots\,\,\,\text{(need\,to\,use\,the\,chain\,rule)}\nonumber \\
0= & -\frac{2\cdot\left(-1\right)}{2\sigma^{2}}\sum_{i=1}^{N}\left(x_{i}-\hat{\mu}\right)\label{eq:11}\\
0= & \frac{1}{\sigma^{2}}\left(\sum_{i=1}^{N}x_{i}-N\hat{\mu}\right)\nonumber \\
\hat{\mu}= & \frac{1}{N}\sum_{i=1}^{N}x_{i}\nonumber
\end{align}

Now with respect to $\sigma^{2}$ (it's easier than with respect to $\sigma$):

\begin{align}
\frac{\partial}{\partial\sigma^{2}}\ell\left(\mu,\sigma;x_{1},x_{2},\ldots,x_{N}\right)= & \frac{\partial}{\partial\sigma^{2}}\left[-\frac{N}{2}\ln\left(2\pi\sigma^{2}\right)-\frac{1}{2\sigma^{2}}\sum_{i=1}^{N}\left(x_{i}-\mu\right)^{2}\right] \nonumber \\
= & \frac{\partial}{\partial\sigma^{2}}\left[-\frac{N}{2}\ln\left(2\pi\right)-\frac{N}{2}\ln\left(\sigma^{2}\right)-\frac{1}{2\sigma^{2}}\sum_{i=1}^{N}\left(x_{i}-\mu\right)^{2}\right] \nonumber \\
= & -\frac{N}{2\sigma^{2}}-\frac{\partial}{\partial\sigma^{2}}\left[\frac{1}{2\sigma^{2}}\sum_{i=1}^{N}\left(x_{i}-\mu\right)^{2}\right] \nonumber \\
0= & -\frac{N}{2\hat{\sigma}^{2}}+\frac{1}{2\left(\hat{\sigma}^{2}\right)^{2}}\sum_{i=1}^{N}\left(x_{i}-\mu\right)^{2}\\
0= & -\frac{N}{2\hat{\sigma}^{2}}+\frac{1}{2\hat{\sigma}^{4}}\sum_{i=1}^{N}\left(x_{i}-\mu\right)^{2} \nonumber \\
0= & \frac{1}{2\hat{\sigma}^{2}}\left[\frac{1}{2\hat{\sigma}^{2}}\sum_{i=1}^{N}\left(x_{i}-\mu\right)^{2}-N\right] \nonumber \\
\hat{\sigma}^{2}= & \frac{1}{N}\sum_{i=1}^{N}\left(x_{i}-\mu\right)^{2} \nonumber
\end{align}
So our Maximum Likelihood Estimate (MLE) is
\begin{align*}
\hat{\mu}= & \frac{1}{N}\sum_{i=1}^{N}x_{i}\\
\hat{\sigma}^{2}= & \frac{1}{N}\sum_{i=1}^{N}\left(x_{i}-\mu\right)^{2}
\end{align*}
and those formulas should look very familiar to you.

\paragraph{Second-order conditions}

For a maximum, we want the second order conditions to be \emph{negative} to ensure that we indeed have a maximum. It turns out that these conditions are satisfied for the MLE of the normal distribution.

\subsection{MLE optimization for linear regression}
Going back to the linear model, let's now assume that $\varepsilon \overset{iid}{\sim}N\left(0,\sigma^2\right)$. We can write down a likelihood function for this model, which is going to look similar to the likelihood function we wrote before, but with a slight tweak---instead of targeting $\mu$ as a parameter, we'll be targeting $\beta$:

\begin{align*}
    \mathcal{L} =& \prod_{i} f\left(\varepsilon_{i}\right) \\
    =& \prod_{i} \frac{1}{\sqrt{2\pi \sigma^{2}}} \exp \left(-\frac{\varepsilon_{i}^{2}}{2\sigma^{2}}\right) \\
    =& \left(\frac{1}{\sqrt{2\pi \sigma^{2}}}\right)^{n} \exp \left(-\frac{\sum_{i}\varepsilon_{i}^{2}}{2\sigma^{2}}\right) \\
    =& \left(\frac{1}{\sqrt{2\pi \sigma^{2}}}\right)^{n} \exp \left(-\frac{\varepsilon^{\prime}\varepsilon}{2\sigma^{2}}\right)\,\,\,,\,\text{and,\,taking\,logs,} \\
    \ell\left(y,X;\beta,\sigma\right)    =& -\frac{n}{2}\ln\left(2\pi\right) - n \ln \left(\sigma\right) - \frac{1}{2\sigma^{2}}\left(y-X\beta\right)^{\prime}\left(y-X\beta\right)
\end{align*}

Our objective is to maximize the likelihood above with respect to $\beta$ and $\sigma$:
\begin{align*}
    & \max_{\beta,\sigma} -\frac{n}{2}\ln\left(2\pi\right) - n \ln \left(\sigma\right) - \frac{1}{2\sigma^{2}}\left(y-X\beta\right)^{\prime}\left(y-X\beta\right) \\
    \frac{\partial\ell}{\partial\beta} =& \frac{1}{2\sigma^{2}}\frac{\partial}{\partial\beta}\left(y-X\beta\right)^{\prime}\left(y-X\beta\right) \\
    0 =& \frac{1}{2\sigma^{2}}\left[-2X^{\prime}y +2\left(X^{\prime}X\right)\hat{\beta}\right] \\
    \hat{\beta} =& \left(X^{\prime}X\right)^{-1}X^{\prime}y \\
    \frac{\partial\ell}{\partial\sigma} =& -\frac{n}{\sigma} + \frac{\left(y-X\beta\right)^{\prime}\left(y-X\beta\right)}{\sigma^{3}} \\
    0 =& -\frac{n}{\hat{\sigma}} +  \frac{\left(y-X\beta\right)^{\prime}\left(y-X\beta\right)}{\hat{\sigma}^{3}} \\
    \frac{n}{\hat{\sigma}} =& \frac{\left(y-X\beta\right)^{\prime}\left(y-X\beta\right)}{\hat{\sigma}^{3}} \\
    \hat{\sigma}^{2} =& \frac{\left(y-X\beta\right)^{\prime}\left(y-X\beta\right)}{n}
\end{align*}

Second-order conditions for a maximum require that the matrix below be \emph{negative definite}\footnote{Negative definite is how we tell that a matrix is ``negative.''}:

\begin{align}
    \label{eq:hess}
    \left[ {\begin{array}{cc}
                \frac{\partial^{2}\ell}{\partial\beta\partial\beta^{\prime}} & \frac{\partial^{2}\ell}{\partial\beta\partial\sigma}  \\
                \frac{\partial^{2}\ell}{\partial\sigma\partial\beta^{\prime}} & \frac{\partial^{2}\ell}{\partial\sigma^{2}}  \\
                  \end{array} } \right]
                  =&
    \left[ {\begin{array}{cc}
                - \frac{X^{\prime}X}{\hat{\sigma}^{2}} &  \frac{X^{\prime}X\hat{\beta} - X^{\prime}y}{\hat{\sigma}^{3}}  \\
                \frac{\hat{\beta}^{\prime}X^{\prime}X - y^{\prime}X}{\hat{\sigma}^{3}}  & \frac{n}{\hat{\sigma}^{2}} - \frac{3\left(y-X\hat{\beta}\right)^{\prime}\left(y-X\hat{\beta}\right)}{\hat{\sigma}^{4}}  \\
                  \end{array} } \right]
              \end{align}

\subsection{Vocabulary}
\begin{description}
    \item[closed-form~solution] A solution to an optimization problem that can be expressed as a mathematical formula. For example, $\hat{\beta} = \left(X^{\prime}X\right)^{-1}X^{\prime}y$ is a closed-form solution to the parameters of the classical linear model. This solution holds for either ordinary least squares (OLS) minimization or maximum likelihood estimaiton (MLE)

        Most optimization problems do \emph{not} have a closed-form solution, in which case we will need a computer to iteratively ``guess and check'' different candidate parameters to find the minimum or maximum of our objective function.

        We will talk about these in the next couple of classes.

    \item[gradient~vector] The gradient vector is the vector of first derivatives of the objective function. For MLE problems, it is the vector

        \begin{align*}
    \left[ {\begin{array}{c}
                \frac{\partial\ell}{\partial\beta}  \\
                \frac{\partial\ell}{\partial\sigma} \\
                  \end{array} } \right]
                  =&
    \left[ {\begin{array}{c}
\frac{1}{2\sigma^{2}}\left[-2X^{\prime}y +2\left(X^{\prime}X\right)\hat{\beta}\right] \\
-\frac{n}{\hat{\sigma}} +  \frac{\left(y-X\beta\right)^{\prime}\left(y-X\beta\right)}{\hat{\sigma}^{3}} \\
                  \end{array} } \right]
        \end{align*}

\item[hessian~matrix] The hessian matrix is the matrix of second derivatives of the objective function. For MLE problems, it is the matrix listed in \eqref{eq:hess}.

\end{description}

Why do we need to know these three vocabulary words? Because we need to understand how the computer finds the optimum of our objective function. For the 99\% of problems that don't have a closed-form solution, the computer tries to find the values of the parameters that set the gradient vector equal to zero. Depending on the algorithm, it will also check that the hessian matrix is negative definite (if we are maximizing) or positive definite (if we are minimizing).

\section{Math: logistic regression}
The above examples optimized the classical linear model, which is a model where the dependent variable is continuous and the optimization has a closed-form solution.

Now let's look at a case where the dependent variable $y$ is binary and (without loss of generality) takes on values 0 and 1.

The appropriate statistical distribution to model a 0/1 outcome is known as the \emph{Bernoulli distribution}. In this distribution, $y=1$ with probability $p$ and $y=0$ with probability $1-p$. You can think of a Bernoulli distribution as modeling a weighted coin flip, where heads occurs with probabiliy $p$.

The probability density function (pdf) of the Bernoulli distribution is
\[
    f\left(y;p\right) = p^{y}\left(1-p\right)^{1-y}
\]

Now let's return to our classical linear model, but where $y$ is now binary instead of continuous. As before, we have (in matrix form)
\[
y = X\beta + \varepsilon
\]

Now we want to think about the conditions under which $y=1$ (which occurs with probability $p$).
\begin{align*}
    \Pr\left(y=1\right) =& \Pr\left(y>0\right) \\
                        =& \Pr\left(X\beta + \varepsilon > 0\right) \\
                        =& \Pr\left(X\beta > -\varepsilon \right) \\
                        =& \Pr\left(-\varepsilon < X\beta \right) \\
    =& \Pr\left(\varepsilon < X\beta \right)\,\,\,\text{if}\,\,\varepsilon\,\,\text{has\,a\,symmetric\,distribution} \\
    =& F\left(X\beta\right)
\end{align*}
where $F\left(\cdot\right)$ denotes the cumulative distribution function of $\varepsilon$.

\subsection{MLE of the logistic regression model}
If we assume that $\varepsilon$ is drawn from the logistic distribution (instead of the normal distribution as in \eqref{eq:8}), then we get
\begin{align*}
    F\left(x\right) =& \frac{1}{1+e^{-x}} \\
                    =& \frac{e^{x}}{1+e^{x}} \\
\end{align*}

This is known as the \textbf{logistic probability function}, or the \textbf{sigmoid function}.

If we go back to the previous set of equations and think about our Bernoulli parameter $p$, we have that $p=\Pr\left(y=1\right)=F\left(X\beta\right)=\frac{\exp\left(X\beta\right)}{1+\exp\left(X\beta\right)}$.

\subsubsection{Logit likelihood function}
Combining the above results, we get our likelihood function:
\begin{align}
    \label{eq:logit}
    \mathcal{L}\left(y,X;\beta\right) =& \prod_{i}p_{i}^{y_{i}}\left(1-p_{i}\right)^{1-y_{i}} \nonumber \\
                                      =& \prod_{i}\left(\frac{\exp\left(X\beta\right)}{1+\exp\left(X\beta\right)}\right)^{y_{i}}\left(1-\frac{\exp\left(X\beta\right)}{1+\exp\left(X\beta\right)}\right)^{1-y_{i}} \nonumber \\
                                      =& \prod_{i}\left(\frac{\exp\left(X\beta\right)}{1+\exp\left(X\beta\right)}\right)^{y_{i}}\left(\frac{1}{1+\exp\left(X\beta\right)}\right)^{1-y_{i}} \nonumber \\
    \ell\left(y,X;\beta\right) =& \sum_{i}y_{i}\ln\left(\frac{\exp\left(X\beta\right)}{1+\exp\left(X\beta\right)}\right)+\left(1-y_{i}\right)\ln\left(\frac{1}{1+\exp\left(X\beta\right)}\right)
\end{align}

\subsubsection{Logit first order conditions}
The first order conditions of \eqref{eq:logit} are:
\begin{align}
    \label{eq:logitFOC}
    \frac{\partial\ell}{\partial\beta} =& \frac{\partial\ell}{\partial\beta}y\left[X\beta - \ln\left(1+\exp\left(X\beta\right)\right)\right] - \left(1-y\right)\left[\ln\left(1+\exp\left(X\beta\right)\right)\right] \nonumber \\
0 =& \frac{\partial\ell}{\partial\beta}y\left[X\beta\right] - \ln\left(1+\exp\left(X\beta\right)\right) \nonumber \\
    0 =& X^{\prime}y-\left[\frac{1}{1+\exp\left(X\beta\right)}X\exp\left(X\beta\right)\right] \nonumber \\
    0 =& X^{\prime}y-\left[X\frac{\exp\left(X\beta\right)}{1+\exp\left(X\beta\right)}\right] \nonumber \\
    0 =& X^{\prime}y-X^{\prime}p \nonumber \\
    0 =& X^{\prime}\left(y-p\right) \nonumber \\
    0 =& X^{\prime}\left(y-\frac{\exp\left(X\beta\right)}{1+\exp\left(X\beta\right)}\right)
\end{align}

Now we need to solve for $\beta$:
\[
    X^{\prime}y=X^{\prime}\left(\frac{\exp\left(X\beta\right)}{1+\exp\left(X\beta\right)}\right)
\]
But this does not have a closed-form solution! Hence, we need to use numerical methods such as gradient descent or L-BFGS to iteratively ``guess and check'' various values of the $\beta$ vector that solve \eqref{eq:logitFOC}.

\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% MAIN BODY
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Approach}

It was essential to realise the requirements of this project before tackling the code. This meant ensuring a consistent understanding between group members of the process we would take, as well as the project specifications. As a result, we conducted some initial research and data creation before the implementation began.

\subsection{Data Creation}

As we would be dealing with large data-sets, we looked at creating a standard format which would reduce some of the difficulties of processing input data. We decided to use CSV files as they are a widely used standard with minimal file-size. The data was generated through a custom MATLAB script that could generate random continuous values from a given probability distribution. We assumed the format of the input data would contain these attributes:
\begin{itemize}
    \setlength\itemsep{0pt}
    \item Clean data (no other characters, legitimate values, decimal notation etc.)
    \item No unnecessary gaps between data points
    \item The size of the data set was a multiple of the work group size.
\end{itemize}
The first two assumptions are valid as the input data is pre-processed so can be cleaned at the CSV-generation step (out of scope). The final assumption is valid as if the data set were not a multiple of the work group size, the final data points would be processed in sequential due to the overhead of creating another buffer. The test data was created by assuming normal and logistic probability distributions, and randomly generating data from these. We replicated 3 processes, temperature ($\mu$(mean)=20°C), voltage ($\mu$=230V) and air pressure ($\mu$=1020hPa), each with 5 significant figures. These provide realistic data sets that might be expected in practice such as streams of data from power meters, barometers and thermometers which would create millions of data points.

\subsection{Statistics}
Descriptive statistics are the representative scalar values that describe a data set and its estimated probability distribution. This includes measures of location (mode, mean, median, etc), measures of spread (standard deviation, variance) and measures of shape (skewness and kurtosis). We have discarded co-variance statistics for multivariate datasets as data from multiple variables is out of scope. We can calculate all of these values from some minimal set of meta-statistics. These aggregate-statistics are the sample size, first 4 orders of statistical moments, min/max and a histogram. The first 4 statistical moments can be used to calculate the mean, standard deviation/variance, skewness and kurtosis respectively. The histogram can calculate the mode (and second-most-frequent, least-frequent, etc) as well as rank statistics (median, upper quartile, lower quartile, etc). Thus if we find efficient algorithms for calculating these meta-statistics, we can obtain any of the required statistics.

\paragraph{Minimum and Maximum}are calculated by just comparing the current Min/Max against a new candidate. This can trivially be implemented as an online algorithm (for sequential) or parallel reduction (in OpenCL).

\paragraph{The first 4 statistical moments}have well-defined and numerically-stable algorithms \cite{Pebay2008} for both online \cite{Choi2010} and parallel \cite{Chan1979} implementations that require only 1 floating point division per update. These algorithms do not suffer from the underflow/overflow that naive algorithms do as there is no summation occurring. The only case where underflow/overflow could occur is when the range of the dataset is very large, which is unlikely in the case of natural processes. In the case of reducing a vector where the length is a power of 2, we can simplify the equation to require no floating point divisions, as such we have implemented this algorithm in our OpenCL kernel. The first order moment $M_1$ is equal to the mean. Variance (square of standard deviation) is the 2nd moment divided by 2: $\sigma^2 = \frac{M_2}{2}$. The skewness can be calculated as $\frac{\sqrt{n}M_3}{M_2^1.5}$. Finally kurtosis is $\frac{nM_4}{M_2^2}$. These values along with min/max are sufficient to describe a dataset and estimate its probability distribution.

\paragraph{The histogram}requires a finite number of bins, so inherently will lose accuracy due to larger representation error. However, if we know the properties of the probability distribution (such as the range, mean, dispersion, shape as calculated above)) we can use non-uniform bin sizes to minimise the errors to the point where double-precision error and sampling error have similar magnitude. Smaller bin sizes can be used closer to the peaks of the dataset where a finer distinction will be required. Finding the mode (or kth-frequent) value can be done by calculating the max (or kth-rank) statistic of the histogram. Finding the median (or k-th rank) can be found by summing along the histogram till you reach the bin that contains half of the sample size. Computing the histogram requires significant local (work-group) memory, so becomes infeasible in the case of GPU acceleration using OpenCL as memory overflows to the host and communication costs outweigh any potential speedup. The histogram calculation is more suited towards an OpenMP approach. Due to poor initial results in OpenCL and time constraints, this meta-statistic was not implemented.

% PRE-IMPLEMENTATION

\subsection{Development setup}
Before we started implementing our project, we first needed to think about how our development process would function. Our process would need to include both the approach we would need to take for coding the project, as well as how we could ensure that our system was both functioning correctly, and correctly implemented. These reasons provided us with the motivation to use technologies such as version control systems and continuous integration.

\subsubsection{Version Control}
We used version control as it would allow all group members to be able to concurrently develop the project and be able to implement different functionality in parallel. Version control also provided a code review process for this project, which helped ensure correct functionality of the developing project.

\subsubsection{Continuous Integration/Testing}
Being able to ensure correctness of implementation was a significant goal of the project. We chose Travis-CI as it would help provide checks that would ensure this correctness, by running tests anytime a branch is pushed to or pull requested. Continuous integration also allowed for the master branch to remain in an unbroken state so that each group member could work off of it.
Tests were used both on Travis-CI and locally to ensure that results on parallel and sequential were within our accuracy goal.


\documentclass[11pt,a4paper]{article}
\usepackage[utf8]{inputenc}
\usepackage[english]{babel}
\usepackage[left=2.5cm, right=2.5cm, top=2.5cm, bottom=2.5cm]{geometry}
\usepackage{hyperref}
\usepackage{graphicx}
\usepackage{times}
\usepackage[bottom]{footmisc}
\usepackage{listings}
\usepackage{bm}
\usepackage{amsmath}
\usepackage{algpseudocode}
\usepackage{algorithm}
\usepackage{titlesec}
\usepackage{booktabs}
\usepackage{amsmath}

\titleformat{\subsubsection}[runin]{}{}{}{}[]

\title{\textbf{Advanced Machine Learning -- Homework 1}}
\author{Anna Berger, Ashwin Sadananda Bhat (AML25)}

\begin{document}
	\maketitle

	In the following exercises we use the shortened notation: $ P(C = c_0) = P(c_0).$
	\section*{Exercise 1}

	\subsection*{Part a}

	From the Bayesian network given in the exercise we can write the joint distribution of five variables as follows:

	$$ P(C,D,I,G,S) = P(C) \cdot  P(D|C)\cdot  P(I|D) \cdot  P(G|D,I) \cdot  P(S|I) $$

	In order to find $P(G)$, we marginalise over all other variables:

	$$ P(G) = \sum_{C, D, I, S} P(C) \cdot  P(D|C)\cdot P(I|D)\cdot P(G|D,I)\cdot P(S|I) $$

	Rearranging the summations, we get the following equation which is easier to compute:
	$$ P(G) = \sum_{D, I} P(G|D,I)\cdot P(I|D) \sum_{S} P(S | I)  \sum_{C} P(C) \cdot  P(D|C) $$

	First, we compute $ \phi_1(D) =  \sum_{C} P(C) \cdot  P(D|C): $

	\begin{itemize}
		\item $ \phi_1(d_0) = P(d_0 | c_0) P(c_0) + P(d_0 | c_1) P(c_1) = 0.1 \cdot 0.3 + 0.7 \cdot 0.7 = 0.52 $
		\item $ \phi_1(d_1) = P(d_1 | c_0) P(c_0) + P(d_1 | c_1) P(c_1) = 0.9 \cdot 0.3 + 0.3 \cdot 0.7 = 0.48 $
	\end{itemize}

	$$ \phi_1(D)  = (0.52, 0.48)$$

	Secondly, we will make use of the fact that  $\phi_2(I) = \sum_{S} P(S | I) = (1, 1). $

	Then we compute $\phi_2(D, I) = P(I|D) \cdot \phi_1(D)$

	\begin{itemize}
		\item $\phi_2(d_0, i_0) = P(i_0 | d_0) \cdot \phi_1(d_0) = 0.6 \cdot 0.52 = 0.312 $
		\item $\phi_2(d_0, i_1) = P(i_1 | d_0) \cdot \phi_1(d_0) = 0.4 \cdot 0.52 = 0.208 $
		\item $\phi_2(d_1, i_0) = P(i_0 | d_1) \cdot \phi_1(d_1) = 0.9 \cdot 0.48 = 0.432 $
		\item $\phi_2(d_1, i_1) = P(i_1 | d_1) \cdot \phi_1(d_1) = 0.1 \cdot 0.48 = 0.048 $
	\end{itemize}

	$$ \phi_2(D, I) =  \left(\begin{smallmatrix} 0.312 & 0.208 \\ \\ 0.43 2 & 0.048 \end{smallmatrix} \right) $$

	Finally, we are left to compute $ P(G) =  \sum_{D, I} P(G|D,I) \phi_2(D, I):$
	\begin{itemize}
		\item $P(g_0) = P(g_0 | i_0, d_0) \phi_2(i_0, d_0) + P(g_0 | i_1, d_0) \phi_2(i_1, d_0) + P(g_0 | i_0, d_1) \phi_2(i_0, d_1) + P(g_0 | i_1, d_1) \phi_2(i_1, d_1) = $
		$ = 0.3 \cdot 0.312 + 0.4 \cdot 0.208 + 0.4 \cdot 0.432 + 0.6 \cdot 0.048 = 0.3784 $
		\item $P(g_1) = P(g_1 | i_0, d_0) \phi_2(i_0, d_0) + P(g_1 | i_1, d_0) \phi_2(i_1, d_0) + P(g_1 | i_0, d_1) \phi_2(i_0, d_1) + P(g_1 | i_1, d_1) \phi_2(i_1, d_1) = $
		$ = 0.2 \cdot 0.312 + 0.2 \cdot 0.208 + 0.3 \cdot 0.432 + 0.3 \cdot 0.048 = 0.248 $
		\item $P(g_2) = P(g_2 | i_0, d_0) \phi_2(i_0, d_0) + P(g_2 | i_1, d_0) \phi_2(i_1, d_0) + P(g_2 | i_0, d_1) \phi_2(i_0, d_1) + P(g_2 | i_1, d_1) \phi_2(i_1, d_1) = $
		$ = 0.5 \cdot 0.312 + 0.4 \cdot 0.208 + 0.3 \cdot 0.432 + 0.1 \cdot 0.048 = 0.3736 $
	\end{itemize}

	All in all, we obtain $ P(G) = (0.3784, 0.248, 0.3736)$
	\subsection*{Part b}

	Again we use the equation for the joint distribution from the Bayesian network given in the exercise:

	$$ P(C,D,I,G,S) = P(C)\cdot P(D|C)\cdot P(I|D)\cdot P(G|D,I)\cdot P(S|I) $$

	Then we apply the Bayes rule and marginalise out  $P(S , c_0):$
	$$  P(S|c_0) = \frac{P(S, c_0)}{P( c_0)} =  \frac{\sum_{D,I,G} P(c_0,D,I,G,S)}{P(c_0)} $$

	$$ P(S|c_0) =  \frac{\sum_{D,I,G} P(c_0)\cdot P(D|C)\cdot P(I|D)\cdot P(G|D,I)\cdot P(S|I)}{P(c_0)} $$

	$$ P(S|c_0)  = \sum_{D,I,G} P(D|C)\cdot P(I|D)\cdot P(G|D,I)\cdot P(S|I) $$

	Rearranging the summations, we obtain the equation handy to compute:
	$$ P(S|c_0)  = \sum_{D} P(D|C) \cdot  \sum_{I}P(I|D)\cdot P(S|I) \cdot \sum_{G}P(G|D,I)  \label{eq:1.1} $$

	As $ \sum_{G}P(G|D,I) = 1$, then we are left to compute:

	$$ P(S|c_0) = \sum_{D} P(D|C) \cdot  \sum_{I}P(I|D)\cdot P(S|I)$$

	First we calculate $ \phi_I(S,D) = \sum_{I}P(I|D)\cdot P(S|I)$:
	\begin{itemize}
		\item $ \phi_I(s_0, d_0) = P(i_0|d_0) \cdot  P(s_0|i_0) + P(i_1|d_0) \cdot  P(s_0|i_1) $
		$ = 0.6\cdot 0.2 + 0.4\cdot 0.7 = 0.4 $
		\item $ \phi_I(s_0, d_1) = P(i_0|d_1) \cdot  P(s_0|i_0) + P(i_1|d_1) \cdot  P(s_0|i_1) $
		$ = 0.9\cdot 0.2 + 0.1\cdot 0.7 = 0.25 $
		\item $ \phi_I(s_1, d_0) = P(i_0|d_0) \cdot  P(s_1|i_0) + P(i_1|d_0) \cdot P(s_1|i_1) $
		$ = 0.6\cdot 0.8 + 0.4\cdot 0.3 = 0.6 $
		\item $ \phi_I(s_0,d_1) = P(i_0|d_1) \cdot  P(s_0|i_0) + P(i_1|d_1) \cdot P(s_0|i_1) $
		$ = 0.9\cdot 0.8 + 0.1\cdot 0.3 = 0.75 $
	\end{itemize}

	$$  \phi_I(S,D) =  \left(\begin{smallmatrix} 0.4 & 0.25 \\ \\ 0.6 & 0.75 \end{smallmatrix} \right) $$

	Then we are left to compute $ P(S|c_0) = \sum_{D}P(D|c_0) \cdot  \phi_I(S,D) :$
	\begin{itemize}
		\item $ P(s_0|c_0) = P(d_0|c_0)\cdot \phi_I(s_0,d_0) + P(d_1|c_0)\cdot \phi_I(s_0,d_1) = 0.1\cdot 0.4 + 0.9\cdot 0.25 = 0.265 $
		\item $ P(s_1|c_0) = P(d_0|c_0)\cdot \phi_I(s_1,d_0) + P(d_1|c_0)\cdot \phi_I(s_1,d_1)  = 0.1\cdot 0.6 + 0.9\cdot 0.75  = 0.735 $
	\end{itemize}

	Therefore, $$ P(S | c_0) = (0.265, 0.735). $$

	\subsection*{Part c}
	Again we use the equation for the joint distribution from the Bayesian network given in the exercise:
	$$ P(C,D,I,G,S) = P(C)\cdot P(D|C)\cdot P(I|D)\cdot P(G|D,I)\cdot P(S|I) $$

	First we apply the Bayes rule to compute the conditional probability distribution:
	$$ P(D | g_2) = \frac{P(D, g_2)}{P(g_2)} $$

	In \textbf{Part a} we have already computed $P(g_2) = 0.3736. $

	We marginalise over all other variables to obtain $P(D , g_2)$ and group sums:

	$$ P(D, g_2) = \sum_{C,S,I}P(C,D,I, g_2,S) = \sum_{C}P(C)\cdot P(D|C)\cdot \sum_{I}P(I|D) \cdot P(g_2|D,I) \cdot \sum_{S}P(S|I)$$

	We make use of the fact that $ \sum_{S}P(S|I) = (1, 1)$, then we are left to compute:
	$$ P(D,g_2)  = \sum_{C}P(C)\cdot P(D|C)\cdot \sum_{I}P(I|D) \cdot P(g_2|D,I)$$

	First, we calculate $ f_C(D) = \sum_{C}P(C) \cdot P(D|C): $
	\begin{itemize}
		\item $f_C(d_0) = P(c_0) \cdot P(d_0|c_0) + P(c_1) \cdot P(d_0|c_1) = 0.3 \cdot 0.1+0.7 \cdot 0.7 = 0.52$
		\item $f_C(d_1) = P(c_0) \cdot P(d_1|c_0) + P(c_1) \cdot P(d_1|c_1) = 0.3 \cdot 0.9 + 0.7 \cdot 0.3 = 0.48$
	\end{itemize}

	Then we compute $f_I(D) =  \sum_{I}P(I|D)\cdot P(g_2|D,I): $
	\begin{itemize}
		\item $f_I(d_0) = P(i_0|d_0) \cdot P(g_2|d_0,i_0) + P(i_1|d_0) \cdot P(g_2|d_0,i_1) = 0.6 \cdot 0.5 + 0.4 \cdot 0.4 = 0.46$
		\item $f_I(d_1) = P(i_0|d_1) \cdot P(g_2|d_1,i_0) + P(i_1|d_1) \cdot P(g_2|d_1,i_1)  = 0.9 \cdot 0.3 + 0.1 \cdot 0.1 = 0.28$
	\end{itemize}

	Having computed $f_I(D)$ and $f_C(D)$, we can now compute $P(D,g_2)$ as $$P(D,g_2) = f_I(D) \cdot f_C(D): $$

	\begin{itemize}
		\item $P(d_0,g_2) = f_I(d_0) \cdot f_C(d_0) = 0.46  \cdot 0.52 = 0.2392 $
		\item $P(d_1,g_2) = f_I(d_1) \cdot f_C(d_1)  = 0.28  \cdot 0.48 = 0.1344 $
	\end{itemize}

	Now, we can do the final computation $ P(D | g_2) = \frac{P(D, g_2)}{P(g_2)} $ :
	\begin{itemize}
		\item $P(d_0|g_2) = \frac{P(d_0,g_2)}{P(g_2)} \approx 0.64 $
		\item $P(d_1|g_2) = \frac{P(d_1,g_2)}{P(g_2)} \approx 0.36 $
	\end{itemize}

	Therefore, $$P(D | g_2) = (0.64, 0.36).$$

	\newpage
	\section*{Exercise 2}

	In Exercise 2 we use the factor graph from the Figure \ref{fig:fgraph_2}. Although it has cycles in it, we can use it as it doesn't create any difficulties for approach with direct marginalization. However, for Exercise 3 this graph is inapplicable: as it has a cycle and the sum-product algorithm won't work properly with it.

	\begin{figure}[H]
		\centering{\includegraphics[height =10cm]{figures/fgraph_2.jpg}}
		\caption{Factor graph for Exercise 2}
		\label{fig:fgraph_2}
	\end{figure}

	\subsection*{Part a:}
	From the graph we can obtain the following equation for factors:
	$$\Phi(C,D,I,G,S) = \Phi(C) \cdot \Phi(C,D) \cdot \Phi(D,I) \cdot \Phi(D,I,G) \cdot \Phi(I,S)$$

	In order to compute the factor $\Phi(G)$ we marginalise over all variables except $G$:
	$$\Phi(G) = \sum_{C,D,I,S}\Phi(C,D,I,G,S)$$
	$$  \Phi(G)= \sum_{D, I} \Phi(D,I) \Phi(D,I,G)  \sum_{C}\Phi(C) \Phi(C,D) \cdot \sum_{S} \Phi(I,S)$$

	First we compute  $ f_S(I) = \sum_{S}\Phi(I,S):$
	\begin{itemize}
		\item $ f_S(i_0) = \Phi(i_0,s_0) + \Phi(i_0,s_1) = 8+32 = 40 $
		\item $ f_S(i_1) = \Phi(i_1,s_0) + \Phi(i_1,s_1) = 24+16 = 40 $
	\end{itemize}

	Then we define $f_C(D) = \sum_{C}\Phi(C) \cdot \Phi(C,D):$
	\begin{itemize}
		\item $ f_C(d_0) = \Phi(c_0) \cdot \Phi(c_0,d_0) + \Phi(c_1) \cdot \Phi(c_1
		,d_0) = 3 \cdot 1 + 10 \cdot 7 = 73 $
		\item $ f_C(d_1) = \Phi(c_0) \cdot \Phi(c_0,d_1) + \Phi(c_1) \cdot \Phi(c_1
		,d_1) = 3 \cdot 9 + 10 \cdot 3 = 57 $
	\end{itemize}

	After that we compute $ f_I(G, D) = \sum_I \Phi(D, I) \cdot \Phi(D, I, G) \cdot f_S(I): $
	\begin{itemize}
		\item $ f_I(g_0, d_0) = \Phi(d_0,i_0) \cdot \Phi(g_0,i_0,d_0) \cdot f_s(i_0) + \Phi(d_0,i_1) \cdot \Phi(g_0,i_1,d_0) \cdot f_s(i_1) = $ \\
		$ = 14 \cdot 6 \cdot 40 + 6 \cdot 12 \cdot 40 = 6240 $
		\item $ f_I(g_0, d_1) = \Phi(d_1,i_0) \cdot \Phi(g_0,i_0,d_1) \cdot f_s(i_0) + \Phi(d_1,i_1) \cdot \Phi(g_0,i_1,d_1) \cdot f_s(i_1) $ \\
		$ = 18 \cdot 12 \cdot 40 + 2 \cdot 18 \cdot 40 = 10080 $
		\item $ f_I(g_1, d_0) = \Phi(d_0,i_0) \cdot \Phi(g_1,i_0,d_0) \cdot f_s(i_0) + \Phi(d_0,i_1) \cdot \Phi(g_1,i_1,d_0) \cdot f_s(i_1) $ \\
		$ = 14 \cdot 3 \cdot 40 + 6 \cdot 9 \cdot 40 = 3840 $
		\item $ f_I(g_1, d_1) = \Phi(d_1,i_0) \cdot \Phi(g_1,i_0,d_1) \cdot f_s(i_0) + \Phi(d_1,i_1) \cdot \Phi(g_1,i_1,d_1) \cdot f_s(i_1) $ \\
		$ = 18 \cdot 6 \cdot 40 + 2 \cdot 9 \cdot 40 = 5040 $
		\item $ f_I(g_2, d_0) = \Phi(d_0,i_0) \cdot \Phi(g_2,i_0,d_0) \cdot f_s(i_0) + \Phi(d_0,i_1) \cdot \Phi(g_2,i_1,d_0) \cdot f_s(i_1) $ \\
		$ = 14 \cdot 21 \cdot 40 + 6 \cdot 9 \cdot 40 = 13920 $
		\item $ f_I(g_2, d_1) = \Phi(d_1,i_0) \cdot \Phi(g_2,i_0,d_1) \cdot f_s(i_0) + \Phi(d_1,i_1) \cdot \Phi(g_2,i_1,d_1) \cdot f_s(i_1) $ \\
		$ = 18 \cdot 12 \cdot 40 + 2 \cdot 3 \cdot 40 = 8880 $
	\end{itemize}

	Now we can calculate the $\Phi(G) = \sum_D f_C(D) \cdot f_I(g_0, D): $
	\begin{itemize}
		\item $\Phi(g_0) = f_C(d_0) \cdot f_I(g_0, d_0) + f_C(d_1) \cdot f_I(g_0, d_1)  = 73 \cdot 6240 + 57 \cdot 10080 = 1030080 $
		\item $\Phi(g_1) = f_C(d_0) \cdot f_I(g_1, d_0) + f_C(d_1) \cdot f_I(g_1, d_1)  = 73 \cdot 3840 + 57 \cdot 5040 = 567600 $
		\item $\Phi(g_2) = f_C(d_0) \cdot f_I(g_2, d_0) + f_C(d_1) \cdot f_I(g_2, d_1)= 73 \cdot 13920 + 57 \cdot 8880 = 1522320 $
	\end{itemize}

	Therefore, $$ \Phi(G) = (1030080, 567600, 1522320). $$

	The corresponding probability distribution is then given by:
	$$ P(G) = \left(\frac{1030080}{1030080 + 567600 + 1522320}, \frac{567600}{1030080 + 567600 + 1522320}, \frac{1522320}{1030080 + 567600 + 1522320}\right). $$

	$$ P(G) \approx (0.33, 0.18, 0.49). $$

	\subsection*{Part b}
	$$ \Phi(S | c_1) = \Phi(S, c_1).$$
	Again we marginalise over all variables except S and C:
	$$ \Phi(S, c_1) = \sum_{D, I, G} \Phi(c_1) \cdot \Phi(c_1, D) \cdot \Phi(I, D) \cdot \Phi(G, I, D) \cdot \Phi(S, I)  $$

	We rearrange the sums to make the computations simpler:
	$$ \Phi(S, c_1) = \Phi(c_1) \sum_I \Phi(S, I) \sum_D \Phi(c_1, D) \cdot \Phi(I, D) \sum_G \Phi(G, D, I) $$

	First we compute  $ f_G(D, I) = \sum_G \Phi(G, D, I):$
	\begin{itemize}
		\item $ f_G(d_0, i_0) = \Phi(g_0, d_0, i_0) + \Phi(g_1, d_0, i_0) + \Phi(g_2, d_0, i_0) = 6 + 3 + 21 = 30$
		\item $ f_G(d_0, i_1) = \Phi(g_0, d_0, i_1) + \Phi(g_1, d_0, i_1) + \Phi(g_2, d_0, i_1) = 12 + 9 + 9 = 30$
		\item $ f_G(d_1, i_0) = \Phi(g_0, d_1, i_0) + \Phi(g_1, d_1, i_0) + \Phi(g_2, d_1, i_0) = 12 + 6 + 12 = 30$
		\item $ f_G(d_1, i_1) = \Phi(g_0, d_1, i_1) + \Phi(g_1, d_1, i_1) + \Phi(g_2, d_1, i_1) = 18 + 9 + 3 = 30$
	\end{itemize}

	Then we obtain $f_D(I) = \sum_D \Phi(c_1, D) \cdot \Phi(I, D) \cdot f_G(D, I):$
	\begin{itemize}
		\item $f_D(i_0) = \Phi(c_1, d_0) \cdot \Phi(i_0, d_0) \cdot f_G(d_0, i_0) + \Phi(c_1, d_1) \cdot \Phi(i_0, d_1) \cdot f_G(d_0, i_1) = $ \\
		$ = 7 \cdot 14 \cdot 30 + 3 \cdot 18 \cdot 30 = 4560 $
		\item $f_D(i_1) = \Phi(c_1, d_0) \cdot \Phi(i_1, d_0) \cdot f_G(d_0, i_1) + \Phi(c_1, d_1) \cdot \Phi(i_1, d_1) \cdot f_G(d_1, i_1) = $ \\
		$ = 7 \cdot 6 \cdot 30 + 3 \cdot 2 \cdot 30 = 1440 $
	\end{itemize}

	Finally, we compute $\Phi(S, c_1) = \Phi(c_1)  \sum_I \Phi(S, I) f_D(I): $
	\begin{itemize}
		\item $ \Phi(s_0, c_1) = \Phi(c_1) ( \Phi(s_0, i_0) f_D(i_0) + \Phi(s_0, i_1) f_D(i_1))  = $ \\
		$= 10 (4560 \cdot 8 + 1440 \cdot 24) = 710400$
		\item $ \Phi(s_1, c_1) = \Phi(c_1) ( \Phi(s_1, i_0) f_D(i_0) + \Phi(s_1, i_1) f_D(i_1))  = $ \\
		$= 10 (4560 \cdot 32 + 1440 \cdot 16) = 1689600$
	\end{itemize}

	Therefore, $$ \Phi(S | c_1) \approx (710400, 1689600). $$

	\subsection*{Part c}
	$$ \Phi(D, g_2) = \Phi(D, g_2).$$
	Marginalising over all variables except D and G, we get:
	$$ \Phi(D, g_2) = \sum_{C,I,S}\Phi(C) \cdot \Phi(C,D) \cdot \Phi(D,I) \cdot \Phi(D,I,g_2) \cdot \Phi(S, I) $$
	We rearrange the sums to make computations easier again:
	$$ \Phi(D, g_2) = \sum_{C}\Phi(C) \cdot \Phi(C,D) \sum_{I}\cdot \Phi(D,I) \cdot \Phi(D,I,g_2) \cdot \sum_{S}\Phi(S, I) $$

	First we compute $f_S(I) = \sum_{S}\Phi(S,I) $:
	\begin{itemize}
		\item $ f_S(i_0) = \Phi(s_0, i_0) +\Phi(s_1, i_0) = 8 + 32 = 40$
		\item $ f_S(i_1) = \Phi(s_0, i_1) +\Phi(s_1, i_1) = 24 + 16 = 40$
	\end{itemize}

	Now, we compute $f_I(D) = \sum_{I} \Phi(D,I) \cdot \Phi(D,I,g_2) \cdot f_S(I)$
	\begin{itemize}
		\item $ f_I(d_0) = \Phi(d_0, i_0) \cdot \Phi(d_0, i_0, g_2) \cdot f_S(i_0) +  \Phi(d_0, i_1) \cdot \Phi(d_0, i_1, g_2) \cdot f_S(i_1) = 14 \cdot 21 \cdot 40 + 6 \cdot 9 \cdot 40 = 13920 $
		\item $ f_I(d_1) = \Phi(d_1, i_0) \cdot \Phi(d_1, i_0, g_2) \cdot f_S(i_0) + \Phi(d_1, i_1) \cdot \Phi(d_1, i_1, g_2) \cdot f_S(i_1) = 18 \cdot 12 \cdot 40 + 2 \cdot 3 \cdot 40 = 8880 $
	\end{itemize}

	Then, we calculate $f_C(D) = \sum_{C} \Phi(C) \cdot \Phi(C,D)$:
	\begin{itemize}
		\item $ f_C(d_0) = \Phi(c_0) \cdot \Phi(c_0, d_0) + \Phi(c_1) \cdot \Phi(c_1, d_0) = 3 \cdot 1 + 10 \cdot 7 = 73 $
		\item $ f_C(d_1) = \Phi(c_0) \cdot \Phi(c_0, d_1) + \Phi(c_1) \cdot \Phi(c_1, d_1) = 3 \cdot 9 + 10 \cdot 3 = 57 $
	\end{itemize}

	Finally, we calculate $ \Phi(D, g_2) $ as:
	$$\Phi(D,  g_2) = \sum_{D}f_C(D) \cdot f_I(D)$$
	$$\Phi(d_0, g_2) = f_C(d_0) \cdot f_I(d_0) = 73 \cdot 13290 = 1016160 $$
	$$\Phi(d_1, g_2) = f_C(d_1) \cdot f_I(d_1) = 57 \cdot 8880 = 506160 $$

	Therefore,

	$$ \Phi(D \ | \ g_2) = (1016160, 506160). $$

	The corresponding probability distribution, $P(D | g_2)$ is given by:
	$$ P(D | g_2) = \left(\frac{\Phi(d_0 | g_2)}{\sum_{D}\Phi(D | g_2)}, \frac{\Phi(d_1 | g_2)}{\sum_{D}\Phi(D | g_2)}\right) = \left(\frac{1016160}{1016160+506160},\frac{506160}{1016160+506160}\right) $$

	So,
	$$ P(D | g_2) \approx (0.67,0.33) $$

\section*{Exercise 3}

\subsection*{Part a}

The corresponding factor graph and the messages which are discussed below are shown on the Figure \ref{fig:fgraph_3a}.

\begin{figure}[H]
	\centering{\includegraphics[width = 10cm]{figures/fgraph_3a.pdf}}
	\caption{Factor graph and messages for Exercise 3, Part a}
	\label{fig:fgraph_3a}
\end{figure}

First, we compute the factor $\Phi_5 = \Phi_{D, I} \cdot \Phi_{G, D, I}$ and obtain the following table (Table \ref{tab:f5}) for it:

\begin{table}[H]
	\centering
	\begin{tabular}{l|rrr}
		\toprule
		$\Phi_5$ & $g_0$ & $g_1$ & $g_2$ \\
		\midrule
		$d_0, i_0$ & 84 & 42 & 294 \\
		$d_0, i_1$ & 72 & 54 & 54 \\
		$d_1, i_0$ & 216 & 108 & 216 \\
		$d_1, i_1$ & 36 & 18 & 6 \\
		\bottomrule
	\end{tabular}
	\caption{Factor $\Phi_5(G, D, I) =  \Phi_{D, I} \cdot \Phi_{G, D, I}$}
	\label{tab:f5}
\end{table}

Now we compute the factor $\Phi(G)$ using sum-product algorithm. We view the node $G$ as the root of the factor graph and initiate messages from the leaves. The messages from the leaves are propagated along every link of the network, using the rules of sum-product algorithm described in Bishop 8.4.4, until the root node has receive messages from all other neighbours.

The sequence of messages, propagated through the network, is as follows (you can see them in the Figure \ref{fig:fgraph_3a} as well):
\begin{itemize}
	\item $ \mu_1(C) = [3, 10] $
	\item $ \mu_2(S) = [1, 1] $
	\item $ \mu_3(C) = [3, 10] $
	\item $ \mu_4(I) = [1 \cdot \Phi(s_0, i_0) + 1 \cdot \Phi(s_1, i_0), 1 \cdot \Phi(s_0, i_1) + 1 \cdot \Phi(s_1, i_1)] = [40, 40] $
	\item $ \mu_5(D) = [3 \cdot \Phi(d_0, c_0) + 10 \cdot \Phi(d_0, c_1), 3 \cdot \Phi(d_1, c_0) + 10 \cdot \Phi(d_1, c_1)] = $ \\
	$ = [3 \cdot 1 + 10 \cdot 7, 3 \cdot 9 + 10 \cdot 3] = [73, 57] $
	\item $ \mu_6(D) = [73, 57]$
	\item $ \mu_7(I) = [73 \cdot \Phi(i_0, d_0) + 57 \cdot \Phi(i_0, d_1), 73 \cdot \Phi(i_1, d_0+ 57 \cdot \Phi(i_1, d_1)] = $ \\
	$ = [73 \cdot 14 + 57 \cdot 18, 73 \cdot 6 + 57 \cdot 2] = [2048, 552] $
	\item $ \mu_8(G) = [\sum_{I, D} \Phi_5(I, D, g_0) \cdot \mu_6(D) \cdot \mu_7(I) , \sum_{I, D} \Phi_5(I, D, g_1) \cdot \mu_6(D) \cdot \mu_7(I), \sum_{I, D} \Phi_5(I, D, g_2) \cdot \mu_6(D) \cdot \mu_7(I)] = [84 \cdot 73 \cdot 40 + 72 \cdot 73 \cdot 40 + 216 \cdot 57 \cdot 40 + 36 \cdot 57 \cdot 40, 42 \cdot 73 \cdot 40 + 54 \cdot 73 \cdot 40 + 108 \cdot 57 \cdot 40 + 18 \cdot 57 \cdot 40, 294 \cdot 73 \cdot 40 + 54 \cdot 73 \cdot 40 + 216 \cdot 57 \cdot 40 + 6 \cdot 57 \cdot 40] = [1030080, 567600, 1522320]$
\end{itemize}

	Therefore, $$ \Phi(G) = \mu_8(G) = (1030080, 567600, 1522320). $$

	The corresponding probability distribution is then given by:
	$$ P(G) = \left(\frac{1030080}{1030080 + 567600 + 1522320}, \frac{567600}{1030080 + 567600 + 1522320}, \frac{1522320}{1030080 + 567600 + 1522320}\right). $$

	$$ P(G) \approx (0.33, 0.18, 0.49). $$

\subsection*{Part b}
	To compute the conditional factor $\Phi(S \ | \ c_1)$, we introduce one extra factor connecting to the node $C$, which is one for observed value of the variable and zero otherwise: $\Phi_{c_1} = [0, 1]. $

	The corresponding factor graph and propagated messages which are discussed below are shown on the Figure \ref{fig:fgraph_3b}.

	\begin{figure}[H]
		\centering{\includegraphics[width = 10cm]{figures/fgraph_3b.pdf}}
		\caption{Factor graph and messages for Exercise 3, Part b}
		\label{fig:fgraph_3b}
	\end{figure}

	Now we compute the factor $\Phi(S \ | \ c_1)$ using sum-product algorithm. We view the node $S$ as the root of the factor graph and initiate messages from the leaves. The messages from the leaves are propagated along every link of the network, using the rules of sum-product algorithm described in Bishop 8.4.4, until the root node has receive messages from all other neighbours.

	The sequence of messages, propagated through the network, is as follows (you can see them in the Figure \ref{fig:fgraph_3b} as well):
	\begin{itemize}
		\item $ \mu_1(C) = [3, 10] $
		\item $ \mu_2(C) = [0, 1] $
		\item $ \mu_3(G) = [1, 1] $
		\item $ \mu_4(C) = [0, 10] $
		\item $ \mu_5(D) = [0 \cdot \Phi(d_0, c_0) + 10 \cdot \Phi(d_0, c_1), 0 \cdot \Phi(d_1, c_0) + 10 \cdot \Phi(d_1, c_1)] = [10 \cdot 7, 10 \cdot 3] = [70, 30] $
		\item $ \mu_6(D) = [70, 30]$
		\item $ \mu_7(I) = \left[\sum_{G, D}\Phi_5 (G, i_0, D) \cdot \mu_3(G) \mu_6(D), \sum_{G, D}\Phi_5 (G, i_1, D) \cdot \mu_3(G) \mu_6(D) \right] = $ \\
		$ = [84 \cdot 1 \cdot 70 + 42 \cdot 1 \cdot 70+ 294 \cdot 1 \cdot 70 + 216 \cdot 1 \cdot 30 + 108 \cdot 1 \cdot 30 + 216 \cdot 1 \cdot 30, 72 \cdot 1 \cdot 70 + 54 \cdot 1 \cdot 70+ 54 \cdot 1 \cdot 70 + 36 \cdot 1 \cdot 30 + 18 \cdot 1 \cdot 30 + 6 \cdot 1 \cdot 30] = [45600, 14400] $
		\item $ \mu_8(I) = \mu_7(I)  = [45600, 14400]$
		\item $ \mu_9(S) = [\mu_8(i_0) \cdot \Phi(i_0, s_0) + \mu_8(i_1) \cdot \Phi(i_1, s_0), \mu_8(i_0) \cdot \Phi(i_0, s_1) + \mu_8(i_1) \cdot \Phi(i_1, s_1)] = [45600 \cdot 8 + 14400 \cdot 24, 45600 \cdot 32 + 14400 \cdot 16] = [710400, 1689600]$
	\end{itemize}

	Therefore, $$ \Phi(S | c_1) = (710400, 1689600). $$

\subsection*{Part c}
	To compute the conditional factor $\Phi(D \ | \ g_2)$, we introduce one extra factor connecting to the node $G$, which is one for observed value of the variable and zero otherwise: $\Phi_{g_2} = [0, 0, 1]. $

	The corresponding factor graph and propagated messages which are discussed below are shown on the Figure \ref{fig:fgraph_3c}.

	\begin{figure}[H]
		\centering{\includegraphics[width = 10cm]{figures/fgraph_3c.pdf}}
		\caption{Factor graph and messages for Exercise 3, Part c}
		\label{fig:fgraph_3c}
	\end{figure}

	Now we compute the factor $\Phi(D \ | \ g_2)$ using sum-product algorithm. We view the node $D$ as the root of the factor graph and initiate messages from the leaves. The messages from the leaves are propagated along every link of the network, using the rules of sum-product algorithm described in Bishop 8.4.4, until the root node has receive messages from all other neighbours.

	The sequence of messages, propagated through the network, is as follows (you can see them in the Figure \ref{fig:fgraph_3c} as well):
	\begin{itemize}
		\item $ \mu_1(C) = [3, 10] $
		\item $ \mu_2(G) = [0, 0, 1] $
		\item $ \mu_3(S) = [1, 1] $
		\item $ \mu_4(C) = [3, 10] $
		\item $ \mu_5(G) = [0, 0, 1]$
		\item $ \mu_6(I) = [\Phi(i_0, s_0) \cdot \mu_3(s_0) +  \Phi(i_0, s_1) \cdot \mu_3(s_1), \Phi(i_1, s_0) \cdot \mu_3(s_0) +  \Phi(i_1, s_1) \cdot \mu_3(s_1)] = $\\ $ = [8 \cdot 1 + 32 \cdot 1, 24 \cdot 1 + 16 \cdot 1] = [40, 40]$
		\item $ \mu_7(I) = \mu_6(I) = [40, 40]$
		\item $ \mu_8(D) = [\Phi(d_0, c_0) \mu_4(c_0) +  \Phi(d_0, c_1) \mu_4(c_1), \Phi(d_1, c_0) \mu_4(c_0) +  \Phi(d_1, c_1) \mu_4(c_1)] = $ \\
		$ [1 \cdot 3 + 7 \cdot 10, 9 \cdot 3 + 3 \cdot 10] = [73, 57]$
		\item $ \mu_9(D) = \left[\sum_{G, I}\Phi_5(G, I, d_0) \cdot \mu_5(G) \mu_7(I),  \sum_{G, I}\Phi_5(G, I, d_1) \cdot \mu_5(G) \mu_7(I) \right] = [13920, 8880].$

	\end{itemize}

	$$ \Phi(D, g_2) = \mu_8(D) \cdot \mu_9(D) = [13920 \cdot 73, 8880 \cdot 57] = [1016160, 506160] $$

	The corresponding probability distribution, $P(D | g_2)$ is given by:
	$$ P(D | g_2) = \left(\frac{\Phi(d_0 | g_2)}{\sum_{D}\Phi(D | g_2)}, \frac{\Phi(d_1 | g_2)}{\sum_{D}\Phi(D | g_2)}\right) = \left(\frac{1016160}{1016160+506160},\frac{506160}{1016160+506160}\right) $$

	So,
	$$ P(D | g_2) \approx (0.67,0.33) $$

\end{document}


\section{Tuning shared memory code}

\chapterDescription
  {
    Around 30 minutes.
  }
  {
    A working shared memory code.
  }

In this section, we discuss some aspects of shared memory parallelisation and
give some hints for proper shared memory parallelisation.

\subsection{Profiling}

Like any other optimisation, I do recommend to start with some profiling. This
should comprise the usage of proper profilers such as VTune Amplifier, but it
also should comprise at least one run where you
enable Peano's profiling outputs as discussed in
Section \ref{section:performance-analysis}.

Once you have obtained a program output with Peano's performance analysis
enabled, please pass this output to Peano's performance analysis scripts:
\begin{code}
python .../src/peano/performanceanalysis/performanceanalysisroutines.py
\end{code}
\noindent
Invoking the script without any argument displays some usage messages.

\begin{remark}
 Profiling Peano's shared memory behaviour induces a significant runtime
 overhead. Furthermore, it generates lots of code. As a result, Peano does some
 in-situ data accumulation. In return, it means that you have to run the code
 for a decent time and on decent problem sizes to get meaningful results.
 Otherwise, the accumulation does average out too many details.
\end{remark}

\begin{center}
 \includegraphics[width=0.6\textwidth]{67_shared-memory-tuning/trace-concurrency.pdf}
\end{center}

\noindent
Peano's shared memory profiling yields four different types of data.
These data stem from very small time sampling intervals. While the data is
sampled in small intervals, Peano dumps the aggregated data to the terminal only
every few seconds. As a result, data is noisy/smoothed out.
The data sets are coloured and we typically distinguish actual concurrency
behaviour to behaviour that could have been obtained if we had split up all
parallel codes into tiny little atomic tasks:

\begin{enumerate}
  \item The ``measured'' concurrency level is the cpu time burnt divided by the
  real time. Usually, this should be around the number of cores/threads you use
  as your system charges you for every physical thread.
  \item The maximum concurrency is the maximum number of threads that could have
  been idle over a small time interval. Peano works in terms of tasks. If the
  number of tasks exceeds the number of threads, the observed concurrency is
  smaller than the derived maximum concurrency.
  \item The time averaged concurrency weights the individual subsamples averaged
  into one data dump with the underlying time spans, i.e.~it is something alike
  $\frac{\int _T f dt }{|T|}$.
  \item The number of background tasks tries to estimate how many background
  tasks are available at any given time. Peano does not realise a mechanism
  tracking the real number of background tasks, so this is a lower bound. There
  might be more tasks still pending in your system.
\end{enumerate}

\subsection{The measured concurrency level equals two}

\begin{smell}
 The measured concurrency level equals two no matter how many cores we try to
 use. Peano runs on a machine with more than one core.
\end{smell}

\noindent
As all cores today offer hyperthreading, we've ran into this smell when the
operating system/scheduler did mask/pin our code onto exactly one core.
The code then uses the hyperthread, but cannot really scale beyond the core.
This is for example done by IBM's load balancer if you don't set the OpenMP
threads correctly---even if you don't use OpenMP but TBBs.

\subsection{Low concurrency levels}

\begin{smell}
 The (theoretical) concurrency level (black line) is very small. As a result,
 the obtained concurrency level is small, too.
\end{smell}

\noindent
There are multiple ways to tackle such a behaviour. Their ultimate goal is to
increase the concurrency of the code. However, you have to be careful: A
brilliant concurrency does not automatically mean that the code runs faster as
threads might be too small. So any optimisation here deserves a careful tracking
of total runtimes.

\begin{enumerate}
  \item Whenever Peano runs into a set of independent events such as
  \texttt{touchVertexFirstTime}, it does not automatically trigger them in
  parallel. Instead it first computes the cardinality of this set and then asks
  an oracle whether to process this set in parallel. The oracle not only says
  yes/no but also identifies well-suited grain sizes, i.e.~the minimal subset
  size. If you observe a low concurrency, it is a straightforward strategy to
  play around with the settings of the dummy oracle (if you use this one;
  cmp.~Section \ref{section:shared-memory:preparation}) or to use another oracle
  that is able to learn and to adopt the grain sizes automatically. You might also
  decide to write your own oracle.
  \item Whenever Peano runs into a set of (potentially) concurrent events, it
  evaluates the corresponding mapping specifications which concurrency
  constraints have to hold. Basically, they define in a colouring language
  (similar to red-black Gau\ss-Seidel) which events may run in parallel. If an
  adapter merges multiple events, the most restrictive constraint system
  dominates. If your overall concurrency level is too small, it is thus worth to
  study whether these constraint systems can be relaxed.
  \item Finally, you might decide to parallelise your own code within the
  mappings.
\end{enumerate}

\begin{remark}
  The mapping specifications are \textbf{not} static, i.e.~they can evaluate the
  mapping's state. In several projects, I found that some very restrictive
  colouring schemes are mandatory (in classic multigrid, I need often for
  example a $7^d$ colouring to ensure that no two vertices restrict their
  residual at the same time to a coarse vertex). However, they are not mandatory
  all the time, i.e.~I can make the specification query return a less
  restrictive pattern from time to time which speeds up the code significantly.
\end{remark}

\noindent
If you program your own code, you might want to use Peano's
\texttt{peano::grid::datatraversal::} \texttt{TaskSet} to split up your code
into multiple tasks. It then is independent of TBB/OpenMP. Furthermore, there's a macro
\texttt{pfor} in the \texttt{tarch/multicore} directory that allows you to write
parallel for loops that are again independent of the underlying shared memory
parallelisation scheme. A d-dimensional extension exists, too.

All of these helper functions again require a grain size which determines
whether and with which cardinality problems are to be split up.
You can use Peano's oracle mechanism here.
Whenever you ask the oracle for a grain size, you have to pass it the code
location.
Usually such a flag indicates some part of the grid management.
However, there's also a set of user-defined markers.
Use those (you have to bookkeep which ones you use where) to integrate into
Peano's oracle mechanism.

Here is an example from the ExaHyPE project where we use the tenth user-defined
marker:
\begin{code}
#include "tarch/multicore/Loop.h"

[...]

const int ProblemSize = ...;
auto grainSize = peano::datatraversal::autotuning::Oracle::getInstance().parallelise(
 ProblemSize,
 peano::datatraversal::autotuning::MethodTrace::UserDefined9
);
// equals for(int i=0; i<ProblemSize; i++)
pfor(i, 0, ProblemSize, grainSize.getGrainSize())
  [...]
endpfor
grainSize.parallelSectionHasTerminated();
\end{code}

\subsection{Low time-averaged concurrency}

To identify/hunt down this issue, I strongly recommend that you switch off all
of your background tasks, i.e.~tasks you fire and later check for completion.
Tasks running independently in the background are a powerful tool to exploit
many cores, but they also tend to make performance data more difficult to
digest.
Furthermore, they help to exploit the cores but the core grid stuff usually
should already scale all alone without them, i.e.~they cannot compensate for a
poor grid concurrency.

\begin{smell}
  The time averaged concurrency is very small though both the maximum
  concurrency level and the cpu occupation (cpu time by real time) both exceed
  the core count. The code is not really fast.
\end{smell}

\noindent
The plot we are speaking about here are the blue lines. If they are small but
the black line approaches the number of cores, we have to assume that the code
consists of a lot of small tasks that are computationally very cheap.
While the code exploits the cores to process work concurrently, it spends quite
some time in administrating these tasks (scheduling) and running through serial
code parts that are sprinkled in between all the parallel code parts.
Furthermore, it might do quite some data consistency operations.

There is no siler bullet in this case, i.e.~if you can't increase the arithmetic
intensity significantly, you basically have to run through a series of trial and
error ideas without any guarantee of success:

\begin{enumerate}
  \item Peano internally realises a BSP-style data flow: Whenever the grid
  traversal runs into some section of the code that shall run in parallel, it
  copies all the mapping objects and triggers the parallel traversal. Once this
  traversal has terminated, it merges all the mappings again. Obviously, this is
  unneccessary overhead if the mappings' states do not change. In this case, we
  could omit the copying and the merging. To tell Peano to skip them, please set
  the last entry in the corresponding event specifications to \texttt{false}:
  \begin{code}
peano::MappingSpecification
myprobject::mappings::MyMaping::ascendSpecification(int level) {
 return peano::MappingSpecification(
  peano::MappingSpecification::Nop,
  peano::MappingSpecification::AvoidCoarseGridRaces,
  false // This false eliminates all the copying
 );
}
  \end{code}
  In most of my own experiments, manipulating this flag howeve has only a very
  limited impact; often, it even backfires.
  \item Peano internally linearises the whole spacetree and, when it runs
  through the tree, reads on one data stream and pipes out the data stream for
  the subsequent iteration. There is an alternative storage scheme that is
  applies if a whole region of the grid remains stationary, is regular and the
  shared memory parallelisation tells the code to do so\footnote{The  design
  here is slightly flawed---a decision which storage format to choose has
  basically nothing to do with shared memory parallelisation. We hijack the
  shared memory decision making nevertheless as this storage decision starts to
  have its major influence once shared memory parallelisation is switched on.}.
  If you have a rather regular, starionary grid, you can doublecheck which
  storage format is chosen by letting \texttt{peano::grid} plot its stuff. You
  should obtain messages alike
  \begin{code}
  regular grid of height 5 used nonpersistently/persistently=0/1 times
  \end{code}
  and the index left of the \texttt{/} should be high(er). It depends on the
  type of shared memory oracle whether you can manually enforce a persistent
  storage. Please study the constructor's arguments.
\end{enumerate}

\begin{center}
 \includegraphics[width=0.8\textwidth]{67_shared-memory-tuning/vtune.png}
 \\
 {\footnotesize
  Typically VTune snapshot that uncovers very high spinning timings.
 }
\end{center}

\begin{smell}
  The time averaged concurrency is very small though the code seems to exploit
  all cores. VTune complains about a lot of spinning.
\end{smell}

\noindent
There are basically two things you can do:

\begin{enumerate}
  \item You can increase the grain sizes delivered by the shared memory oracles.
  If you use autotuning, this should automatically be done.
  \item If you use background tasks, try to fuse multiple of these tasks into
  a single one.
\end{enumerate}

\begin{center}
 \includegraphics[width=0.8\textwidth]{67_shared-memory-tuning/concurrency.png}
 \\
 {\footnotesize
  Profile obtained through Peano's built-in profiling on a two-core laptop that
  shows that many background tasks would be available if there were more cores.
 }
\end{center}

\subsection{Multiscale algorithms}

Peano tries to identify regular subregions in your grid automatically and
optimises those regions particularly aggressive.
If you run a multiscale solver, it can, in theory, run the \texttt{enterCell}
events on a level $\ell $ in parallel to the \texttt{touchVertexFirstTime}
events on level $\ell +1$.
To make this work three things have to hold:

\begin{enumerate}
  \item The two events have to be independent of each other. This is something
  you have to check. Peano cannot know this. The events are independent, if
  \texttt{touchVertexFirstTime} does not access the coarse grid level data
  handed in. There are no constraints on \texttt{enterCell}, \linebreak
  i.e.~\texttt{enterCell} may access coarser data.
  \item The event \texttt{descend} has to be set to \texttt{Nop}, i.e.~it has to
  be empty.
  \item The event specification for \texttt{touchVertexFirstTime} has to inform
  Peano that there are no inter-grid dependencies.
\end{enumerate}

\noindent
A similar arguing holds the other way round for \texttt{Ascend},
\texttt{touchVertexLastTime} and \texttt{leaveCell}.


% file: Analysis.tex
% Analysis, in unconventional ``grande'' format; fitting a widescreen format
%
% github        : ernestyalumni
% linkedin      : ernestyalumni
% wordpress.com : ernestyalumni
%
% This code is open-source, governed by the Creative Common license.  Use of this code is governed by the Caltech Honor Code: ``No member of the Caltech community shall take unfair advantage of any other member of the Caltech community.''
%

\documentclass[10pt]{amsart}
\pdfoutput=1
%\usepackage{mathtools,amssymb,lipsum,caption}
\usepackage{mathtools,amssymb,caption}

\usepackage{graphicx}
\usepackage{hyperref}
\usepackage[utf8]{inputenc}
\usepackage{listings}
\usepackage[table]{xcolor}
\usepackage{pdfpages}
%\usepackage[version=3]{mhchem}
%\usepackage{mhchem}

\usepackage{tikz}
\usetikzlibrary{matrix,arrows,backgrounds} % background for framed option

\usepackage{multicol}

\hypersetup{colorlinks=true,citecolor=[rgb]{0,0.4,0}}

\oddsidemargin=15pt
\evensidemargin=5pt
\hoffset-45pt
\voffset-55pt
\topmargin=-4pt
\headsep=5pt
\textwidth=1120pt
\textheight=595pt
\paperwidth=1200pt
\paperheight=700pt
\footskip=40pt

\newtheorem{theorem}{Theorem}
\newtheorem{corollary}{Corollary}
%\newtheorem*{main}{Main Theorem}
\newtheorem{lemma}{Lemma}
\newtheorem{proposition}{Proposition}

\newtheorem{definition}{Definition}
\newtheorem{remark}{Remark}

\newenvironment{claim}[1]{\par\noindent\underline{Claim:}\space#1}{}
\newenvironment{claimproof}[1]{\par\noindent\underline{Proof:}\space#1}{\hfill $\blacksquare$}

%This defines a new command \questionhead which takes one argument and
%prints out Question #. with some space.
\newcommand{\questionhead}[1]
  {\bigskip\bigskip
   \noindent{\small\bf Question #1.}
   \bigskip}

\newcommand{\problemhead}[1]
  {
   \noindent{\small\bf Problem #1.}
   }

\newcommand{\exercisehead}[1]
  { \smallskip
   \noindent{\small\bf Exercise #1.}
  }

\newcommand{\solutionhead}[1]
  {
   \noindent{\small\bf Solution #1.}
   }

\title{Analysis Dump}
\author{Ernest Yeung \href{mailto:ernestyalumni@gmail.com}{ernestyalumni@gmail.com}}
\date{20 Nov 2017}
\keywords{Analysis, Functional Analysis}
\begin{document}

\definecolor{darkgreen}{rgb}{0,0.4,0}
\lstset{language=Python,
 frame=bottomline,
 basicstyle=\scriptsize,
 identifierstyle=\color{blue},
 keywordstyle=\bfseries,
 commentstyle=\color{darkgreen},
 stringstyle=\color{red},
 }
%\lstlistoflistings

\maketitle

From the beginning of 2016, I decided to cease all explicit crowdfunding for any of my materials on physics, math.  I failed to raise \emph{any} funds from previous crowdfunding efforts.  I decided that if I was going to live in \emph{abundance}, I must lose a scarcity attitude.  I am committed to keeping all of my material \textbf{open-sourced}.  I give all my stuff \emph{for free}.

In the beginning of 2017, I received a very generous donation from a reader from Norway who found these notes useful, through \emph{PayPal}.  If you find these notes useful, feel free to donate directly and easily through \href{https://www.paypal.com/cgi-bin/webscr?cmd=_donations&business=ernestsaveschristmas%2bpaypal%40gmail%2ecom&lc=US&item_name=ernestyalumni&currency_code=USD&bn=PP%2dDonationsBF%3abtn_donateCC_LG%2egif%3aNonHosted}{PayPal}, which won't go through a 3rd. party such as indiegogo, kickstarter, patreon.  Otherwise, under the \emph{open-source MIT license}, feel free to copy, edit, paste, make your own versions, share, use as you wish.

\noindent gmail        : ernestyalumni \\
linkedin     : ernestyalumni \\
twitter      : ernestyalumni \\

\begin{multicols*}{2}

\setcounter{tocdepth}{1}
\tableofcontents

\begin{abstract}
Everything about Analysis, real analysis, complex analysis, functional analysis, Fourier series, Fourier transforms, Fourier analysis

\end{abstract}

\part{Fourier Analysis}

\section{Fourier transform}

cf. Ch. IX of Reed and Simon \cite{ReSi1980}, from pp. 318

\begin{definition}[Schwartz space]
Showing Reed and Simon \cite{ReSi1980}'s notation and wikipedia's notation (that'll be used here), respectively
\[
	\mathcal{S}(\mathbb{R}^n) \equiv S(\mathbb{R}^n) = \text{ Schwartz space of $C^{\infty}$ functions of rapid decrease, i.e. }
	\]
	\begin{equation}
	S(\mathbb{R}^n) = \lbrace f\in C^{\infty}(\mathbb{R}^n) | \| f\|_{\alpha, \beta} < \infty, \, \forall \, \alpha, \beta \in \mathbb{Z}^n_+ \rbrace
	\end{equation}
	where $\alpha , \beta$ are multiindices, $C^{\infty}(\mathbb{R}^n)$ is set of smooth functions from $\mathbb{R}^n$ to $\mathbb{C}$, and
	\begin{equation}
	\| f\|_{\alpha,\beta} = \sup_{\mathbf{x} \in \mathbb{R}^n }{ | \mathbf{x}^{\alpha} D^{\beta} f(\mathbf{x}) | }
	\end{equation}
cf. wikipedia definition of Schwartz space
\end{definition}

cf. IX.1 The Fourier transform on $S(\mathbb{R}^n)$ and $S'(\mathbb{R}^n)$, convolutions of Reed and Simon \cite{ReSi1980}

\begin{definition}
	Suppose $f\in S(\mathbb{R}^n)$, \\
	\textbf{Fourier transform} of $f$, $\widehat{f}$, give by
	\begin{equation}
	\widehat{f}(\mathbf{\lambda}) = \frac{1}{(2\pi)^{n/2)}} \int_{\mathbb{R}^n} e^{-i \mathbf{x}\cdot \mathbf{\lambda} } f(\mathbf{x})d\mathbf{x}
	\end{equation}
	where $\mathbf{x} \cdot \mathbf{\lambda}  =\sum_{i=1}^n x_i \lambda_i $

	Inverse Fourier transform of $f$, $\check{f}$,
	\[
	\check{f}(\lambda) = \frac{1}{(2\pi)^{n/2}} \int_{\mathbb{R}^n} e^{i\mathbf{x}\cdot \mathbf{\lambda} } f(\mathbf{x}) d\mathbf{x}
	\]
	Reed and Simon \cite{ReSi1980} mentions this notation and I will use it more here
	\[
	\widehat{f} \equiv \mathcal{F} f
	\]
\end{definition}

Standard multiindex notation:
\[
\alpha = \langle \alpha_1, \dots \alpha_n \rangle
\]
$n$-tuple of nonnegative integers, $\alpha \in \mathbb{Z}_+^n$

$I_+^n \equiv $ collection of all multiindices \\
Define:
\[
\begin{gathered}
	|\alpha | := \sum_{i=1}^n \alpha_i \\
	\mathbf{x}^{\alpha} := x_1^{\alpha_1} x_2^{\alpha_2} \dots  x_n^{\alpha_n} \\
	D^{\alpha} := \frac{\partial^{ |n |} }{ \partial x_1^{\alpha_1} \partial x_2^{\alpha_2} \dots \partial x_n^{\alpha_n} }
x^2 := \sum_{i=1}^n x_i^2
\end{gathered}
\]

\begin{lemma}
	$ \widehat{ \, } $, $\check{\, } \equiv \mathcal{F}, \mathcal{F}^{-1}$ are cont. linear transformations of $S(\mathbb{R}^n)$ into $S(\mathbb{R}^n)$.

	Furthermore, if $\alpha, \beta \in I_+^n$, then
	\begin{equation}
	((i\lambda)^{\alpha} D^{\beta}\widehat{f})(\lambda) = \widehat{ D^{\alpha}((-ix)^{\beta}f(x)) }
	\end{equation}
	cf. (IX.1) of Reed and Simon \cite{ReSi1980}.
\end{lemma}

\begin{proof}
$\widehat{ \, } \equiv \mathcal{F}$ clearly linear (since $\int$ linear), \\
Since
\[
\begin{gathered}
	(\lambda^{\alpha} D^{\beta} \widehat{f} )(\lambda) = \lambda^{\alpha} D^{\beta} \frac{1}{(2\pi)^{n/2}} \int_{\mathbb{R}^n } e^{-i\mathbf{x}\cdot \mathbf{\lambda}} f(\mathbf{x})d\mathbf{x} = \frac{1}{ (2\pi)^{n/2}} \int_{\mathbb{R}^n} \lambda^a (-i\mathbf{x})^{\beta} e^{-i\mathbf{\lambda} \cdot \mathbf{x} } f(\mathbf{x}) d\mathbf{x} = \\
	= \frac{1}{ (2\pi)^{n/2}} \int_{\mathbb{R}^n} \frac{1}{(-i)^a} (D_x^a e^{-i\mathbf{\lambda} \cdot \mathbf{x}}) (-i\mathbf{x})^{\beta} f(\mathbf{x}) d\mathbf{x} = 0 + \frac{(-i)^{\alpha} }{ (2\pi)^{n/2}} \int_{\mathbb{R}^n} e^{-i \mathbf{\lambda} \cdot \mathbf{x} } D_x^{\alpha} ((-i\mathbf{x})^{\beta}f(\mathbf{x}))d\mathbf{x}
\end{gathered}
\]
Last step is just integration by parts and using given $\| f \|_{\alpha, \beta} < \infty$ property.

Conclude $\| \widehat{f} \|_{\alpha, \beta} = \sup_{\mathbf{\lambda} } | \mathbf{\lambda}^{\alpha} (D^{\beta} \widehat{f})(\lambda) | \leq \frac{1}{ (2\pi)^{n/2}} \int | D_{\mathbf{x}}^{\alpha}(\mathbf{x}^{\beta} f) | d\mathbf{x} < \infty$.

Thus,
\[
\mathcal{F}: S(\mathbb{R}^n) \to S(\mathbb{R}^n)
\]
and
\[
((i\mathbf{\lambda})^{\alpha} D^{\beta}\widehat{f})(\lambda)  = \widehat{ D^{\alpha}((-ix)^{\beta}f(x)) }
\]

If $k$ large enough, $\int  (1+x^2)^{-k} d\mathbf{x} < \infty$ (Clearly $\int \frac{1}{(1+x^2)}= \arctan{x} \xrightarrow{ \infty, -\infty} \frac{\pi}{2} - (-\frac{\pi}{2}) = \pi < \infty$), so
\[
\| \widehat{f} \|_{\alpha, \beta} \leq \frac{1}{(2\pi)^{n/2}} \int_{\mathbb{R}^n} \frac{ (1+x^2)^{-k} }{ (1+x^2 )^{-k} } |D_x^{\alpha} (\mathbf{x}^{\beta} f) | d\mathbf{x} \leq \frac{1}{(2\pi)^{n/2}} (\int_{\mathbb{R}^n } (1+x^2)^{-k} d\mathbf{x}) \sup_{\mathbf{x}} |(1+x^2)^k D_x^{\alpha} (\mathbf{x}^{\beta} f)|
\]
By Leibnitz rule, $(fg)^{(n)}(x) = \sum_{k=0}^n \binom{n}{k} f^{(n-k)}(\mathbf{x}) g^{(k)}(\mathbf{x}) $, $\exists \, $ constants $c_j$, multiindices $\alpha_j \beta_j \in I_+^n$, s.t.
\[
\| \widehat{f} \|_{\alpha, \beta} \leq \sum_{i=1}^M c_j \| f\|_{\alpha_j,\beta_j}
\]
where $\| f\|_{\alpha_j , \beta_j} = \sup_{\mathbf{x} \in \mathbb{R}^n } | \mathbf{x}^{\alpha} D^{\beta} f(\mathbf{x}) |$, which we recall, was used.

Thus $\| \widehat{f} \|_{\alpha,\beta}$ bounded, and by, as Reed and Simon \cite{ReSi1980} said, Thm. V.4, therefore cont.  But I think that reference is incorrect.  I looked up possible theorems online, and possibly it's, \\
since $\widehat{f}$ bounded and has closed graph $(\mathbf{\lambda}, \widehat{f}(\mathbf{\lambda}))$, then $f$ cont.

Likewise for $\check{f}$

	\end{proof}

cf. Thm. IX.1. of Reed and Simon (1980)\cite{ReSi1980}
\begin{theorem}[(Fourier inverse thm.)]
	Fourier transform $\mathcal{F}$ is linear, bicont., bijection: $\mathcal{F} : S(\mathbb{R}^n ) \to S(\mathbb{R}^n) $, and $\mathcal{F}^{-1} = \check{\, }$.
\end{theorem}
\begin{proof}
Prove $\mathcal{F} \mathcal{F}^{-1} f = \mathcal{F}^{-1} \mathcal{F} f = f$ for $f$ contained in dense set $C^{\infty}(\mathbb{R}^n)$.

Let $C_{\epsilon}$ be cube of volume $\left( \frac{2}{\epsilon}\right)^n$ centered at $0\in \mathbb{R}^n$.

Choose $\epsilon$ small enough s.t. support of $f$ is contained in $C_{\epsilon}$.

Let $K_{\epsilon} := \lbrace \mathbf{k} \in \mathbb{R}^n | \forall \, k_i / \pi \epsilon \text{is an integer } \rbrace$, then
\[
f(x) = \sum_{\mathbf{k} \in K_{\epsilon}}    ((\frac{1}{2} \epsilon)^{n/2} e^{i\mathbf{k}\cdot \mathbf{x}}, f) (\frac{1}{2} \epsilon)^{n/2} e^{-i \mathbf{k} \cdot \mathbf{x}}
\]
where $(\cdot,\cdot )$ is the inner product.

The expression immediately above for $f(x)$ is just the Fourier series of $f$, which converges uniformly in $C_{\epsilon}$, to $f$, since $f$ cont. diff. (Thm. II.8 of Reed and Simon (1980) \cite{ReSi1980}).  Recall this theorem says:  \\
Suppose $f(x)$ periodic of period $2\pi$ and is cont. diff.  Then functions $\sum_{-M}^{M} c_n e^{inx} \xrightarrow{M\to \infty} f(x)$ uniformly converges.

\begin{equation}
f(x) = \sum_{\mathbf{k} \in K_{\epsilon} } \frac{\widehat{f}(\mathbf{k}) e^{i\mathbf{k}\cdot \mathbf{x}} }{ (2\pi)^{n/2} } (\pi \epsilon)^n
\end{equation}
cf. (IX.2) of Reed and Simon (1980)\cite{ReSi1980}.

Since $\mathbb{R}^n $ is the disjoint union of cubes of volume $(\pi \epsilon)^n$ centered around pts. in $K_{\epsilon}$, (indeed, $K_{\epsilon} = \lbrace \mathbf{k} \in \mathbb{R}^n | k_i/\pi \epsilon \in \mathbb{Z} \, \forall \, i=1,2,\dots n\rbrace$) then
\[
\sum_{\mathbf{k} \in K_{\epsilon} } \frac{\widehat{f}(\mathbf{k}) e^{i \mathbf{k}\cdot \mathbf{x} } }{ (2\pi )^{n/2} } (\pi \epsilon)^n
\]
is just Riemann sum for integral of function
\[
\widehat{f}(\mathbf{k}) e^{i\mathbf{k}\cdot \mathbf{x} } /(2\pi)^{n/2}
\]
By lemma, $\widehat{f}(\mathbf{k})e^{i\mathbf{k}\cdot \mathbf{x} } \in S(\mathbb{R}^n)$, so Riemann sums converge to integral.  Thus
\[
\mathcal{F}^{-1}\mathcal{F}f = f
\]

	\end{proof}

\end{multicols*}

\begin{thebibliography}{9}

\bibitem{ReSi1980}
Michael Reed and Barry Simon. \textbf{Functional Analysis (Methods of Modern Mathematical Physics, Vol. 1)}.  Academic Press.  1980.

\end{thebibliography}

\end{document}


\section{OpenMP Implementation}
\label{sec:omp}

For the OpenMP implementation used, 3 steps are required in each iteration:

\begin{enumerate}
	\item Bucket Fill
	\begin{itemize}
		\item[-] For $P$ threads and $B$ buckets, each thread will handle $B/P$ buckets
		\item[-] Each thread iterates entire array, inserting only on its own buckets
	\end{itemize}

	\item Master thread computes offset for each bucket on the result array

	\item Each thread copies its buckets to global array
\end{enumerate}

Given that radix sort is a non-comparative sorting algorithm, and it consists mostly on memory operations, it can be easily seen that it is completely memory bound. Because of that, no good results should be expected from this shared-memory implementation. Since memory is already the bottleneck, no benefit is taken from asking more from it.


\clearpage
\subsection{Type (recap)} % (fold)
\label{sub:type_recap_}

This chapter is all about types, so its is important to have a good understanding of what a type is. A type is a specification for a class of data, describing how it is stored and interpreted.

\begin{figure}[h]
   \centering
   \includegraphics[width=\textwidth]{./topics/type-decl/diagrams/TypeRecap}
   \caption{Type defines the size and interpretation if values in your code}
   \label{fig:type-recap}
\end{figure}

\mynote{
\begin{itemize}
  \item A Type is a kind of \texttt{artefact}, describing the format and interpretation of values.
  \item Types specify the following:
  \begin{itemize}
    \item The size (number of bits) needed to store values of this type.
    \item How the bits of the type are interpreted.
    \item The operations that can be performed on values of this type.
  \end{itemize}
  \item You can think of a Type as a blueprint, specifying data layout.
  \item The interpretation of a value depends on its Type, for example \texttt{10100101} is \~{N} if the value is a Character type, but the same value would be \texttt{165} if it is an Integer type.
  \item You can create your own types...
\end{itemize}
}

% subsection type_recap_ (end)


\documentclass[a4paper,11pt]{article}
\pdfoutput=1 % if your are submitting a pdflatex (i.e. if you have
             % images in pdf, png or jpg format)

\usepackage{jheppub} % for details on the use of the package, please
                     % see the JHEP-author-manual

\usepackage[T1]{fontenc} % if needed

\title{\boldmath Time Series Analysis Basics}

%% %simple case: 2 authors, same institution
%% \author{A. Uthor}
%% \author{and A. Nother Author}
%% \affiliation{Institution,\\Address, Country}

% more complex case: 4 authors, 3 institutions, 2 footnotes
\author[a,b,1]{Elliot Golias,\note{Corresponding author.}}
%\author[c]{S. Econd,}
%\author[a,2]{T. Hird\note{Also at Some University.}}
%\author[a,2]{and Fourth}

% The "\note" macro will give a warning: "Ignoring empty anchor..."
% you can safely ignore it.

\affiliation[a]{Case Western Reserve University,\\some-street, Country}
%\affiliation[b]{Another University,\\different-address, Country}
%\affiliation[c]{A School for Advanced Studies,\\some-location, Country}

% e-mail addresses: one for each author, in the same order as the authors
\emailAdd{elliotgolias@case.edu}

\abstract{Abstract...}

\begin{document}
\maketitle
\flushbottom

\section{Introduction to Time Series}
%%%%%%%%%%%%%%%%%%%%%%%%%
A time series is simply a collection of data where each element in the dataset has a corresponding time associated with it.

A \textit{statistically stationary} time series is a

%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Random Walks}
%%%%%%%%%%%%%%%%%%%%%%%
%
A \textit{random walk} is a state process $\{x_k\}$ satisfying
%
\begin{align}
	x_k = x_{k-1} + \omega_k,
\end{align}
%
where $\{\omega_k\}$ are iid Gaussian $\mathcal{N}(0, 1)$ and constitute a discrete white noise time series.

%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Autoregressive Models}
%%%%%%%%%%%%%%%%%%%%%%%%%
%
An \textit{autoregressive model} is a model consisting of a dependent variable that is dependent on one or more of previous values of itself. In particular, an autoregressive model of order $p$, denoted $AR(p)$, is defined by the equation
%
\begin{align}
X_t = c + \sum^p_{i=1} \phi_i X_{t-i} + \omega_t,
\end{align}
%
where $\phi_1, \dots, \phi_p$ are the parameters of the model, $c$ is a constant, and $\omega_t$ is white noise.

%%%%%%%%%%%%%%%%%%%%%%%%
\section{Moving Average Models}
%%%%%%%%%%%%%%%%%%%%%%%%
%
A \textit{moving-average model} is very similar to an autoregressive model, but, instead of the process being dependent on only previous values of itself, it is also dependent on a linear combination of past values of the white noise. In particular, we have
%
\begin{align}
	x_t = \omega_t + \beta_1 \omega_{t-1} + \dots \beta_p \omega_{t-p}.
\end{align}
%
%%%%%%%%%%%%%%%%%%%%%%%
\section{Autoregressive Moving Average Models}
%%%%%%%%%%%%%%%%%%%%%%
%
\begin{align}
	x_t &= \alpha_1 x_{t-1} + \alpha_2 x_{t-2} + \dots + \omega_t + \beta_1 \omega_{t-1} + \beta_2 \omega_{t-2} \dots \beta_q \omega_{t-q} \nonumber \\
	&= \sum_{i=1}^p \alpha_i x_{t-i} + \omega_t + \sum_{i=1}^q \beta_i \omega_{t-i}
\end{align}
%
%\appendix

%\section{Some title}

\end{document}


\documentclass[12pt]{article}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{geometry}
\usepackage{enumerate}
\usepackage{natbib}
\usepackage{float}%稳定图片位置
\usepackage{graphicx}%画图
\usepackage[english]{babel}
\usepackage{a4wide}
\usepackage{indentfirst}%缩进
\usepackage{enumerate}%加序号
\usepackage{multirow}%合并行
\title{\large UM-SJTU JOINT INSTITUTE\\PHYSICS LABTORATORY\\(VE216)\\\ \\\ \\\ \\\ \\\ \\\ \\\ \\\ \\\ \\\ \\\
LABTORATORY REPORT\\\ \\\ LAB 1\\\ LTI SYSTEMS \\\ \\\ \\\ \\\ \\\ }
\author{Name: Pan Chongdan\\ID: 516370910121\\Name Xiang Zhiyuan\\ID: 516370910126}
\date{Date: \today}

\begin{document}
\maketitle
\newpage
\section{Objectives}
\begin{itemize}
\item Become familiar with the laboratory equipment: power supply, signal generator, digital oscilloscope,
computer data acquisition system (Scope Connect).
\item Review basic concepts of linear time-invariant systems.
\item Illustrate several possible ways to determine the impulse response of a physical system from measured
data.
\item Use linearity, time-invariance and impulse response to compute the output of an LTI system when
the input is a step, a pulse, or a more complicated signal.
\item Measure the output response of the series RC circuit for a variety of inputs, including a step, a
combination of a step and a ramp, and a sinusoid
\item Compare results to those computed as part of pre-lab assignment
\end{itemize}
\section{Theoretical Background}
An RC circuit is used so that the computations are easy and physically meaningful. The same procedures
can be applied to much more complicated systems.
\subsection{RC Circuit}
The RC circuit shown below is an example of a simple LTI system. Of course, there are many other LTI
systems that do not involve circuits at all.
\begin{figure}[H]
\centering
\includegraphics[scale=0.5]{p1.jpg}
\caption{RC Circuit}
\end{figure}
We will take the system input to be the voltage $V_{in}(t)$, while the system output is the voltage, $V_{out}(t)$,
dropped across the capacitor. Notice that these voltages, in general, will be functions of time,$t$.
\subsection{When is a Linear Circuit a Linear System?}
Using Kirchoffs current and voltage laws, one can easily derive a differential equation model of the RC-circuit
in Figure 1, namely
\begin{equation}
RC\frac{\mathbf{d}V_{out}(t)}{\mathbf{d}t}+V_{out}(t)=V_{in}(t)
\end{equation}
Appealing to basic knowledge of ODEs from a sophomore level math course, the total solution is seen to be
\begin{equation}
V_{out}(t)=V_0e^{-t/RC}+\int_0^t\frac{1}{RC}e^{-(t-\tau)/RC}V_{in}(\tau)d\tau,\tau>0
\end{equation}
where the initial condition at time zero is $V_{out}(0) = V_0$. It is very easy to verify that $V_{out}(t)$ is a linear
function $V_{in}(t)$ if, and only if, $V_{out}(0)=V_0=0$, that is, the initial voltage on the capacitor has to be zero.
This point is emphasized because you will have to assure this in the laboratory by either waiting for the
charge to decay on the capacitor or by shorting the capacitor with a wire.
\subsection{Impulse Response}
The impulse response, $h(t)$, of an LTI system is, by definition, the output response when the input of the
system is a delta function, $\delta(t)$. Of course, the delta function is a mathematical idealization. In practice,
$h(t)$ can be well approximated by the response of the system when the input is a pulse of very short duration
(compared with the response time of the system) and unit area, such as
$p_{\bigtriangleup}=\frac{1}{\bigtriangleup}((u(t)-u(u-\bigtriangleup)))$ for $\bigtriangleup>0$ sufficiently small.
\par Note that in order to keep the area of the pulse equal to unity, the amplitude has to increase as the pulse
duration gets shorter. Often, this is a problem in a practical system as a large voltage pulse may fry an
amplifier, for example. One way to get around this is to use linearity and realize that if the input is scaled
by "$b$", then the output will be scaled by "$b$" as well. Consequently, if the measured response is divided by
"$b$", an approximation of the impulse response is obtained.
\subsection{Step Response}
For any LTI system the output can be expressed as
\begin{equation}
y(t)=x(t)*h(t)=\int_{-\infty}^\infty x(\tau)h(t-\tau)d\tau=\int_{-\infty}^\infty x(t-\tau)h(\tau)d\tau
\end{equation}
where $x(t)$ denotes the input and $*$ denotes the convolution operation. The output resulting when the input
is a unit step function, $x(t)=u(t)$, is called the unit step response. Simple manipulation leads to
\begin{equation}
y_{step}(t)=u(t)*h(t)=\int_{-\infty}^\infty h(\tau)u(t-\tau)d\tau=\int_{-\infty}^t h(\tau)u(t-\tau)d\tau+\int_t^\infty h(\tau)u(t-\tau)d\tau
\end{equation}
which, because the unit step function is equal to 1 for $t-\tau>0$ and equal to 0 for $t-\tau<0$ step response simplifies to
\begin{equation}
y_{step}(t)=\int_{-\infty}^th(\tau)d\tau
\end{equation}
By taking the derivative of $y_{step}(t)$ with respect to t, we obtain, by the fundamental theorem of calculus,
\begin{equation}
\frac{\mathbf{d}y_{step}(t)}{\mathbf{d}t}=\frac{\mathbf{d}}{\mathbf{d}t}\int_{-\infty}^th(\tau)d\tau=h(t)
\end{equation}
Thus the impulse response can be computed from the unit step response by calculating the derivative of the
step response with respect to time. This is a useful observation because it is sometimes easier to apply a
step input to a physical system than it is to apply (an approximation of) an impulse.
\par We now have two ways of determining the impulse response from data. A third way will be hinted at a
little later.
\section{Experimental Procedure}
\subsection{Step}
\begin{itemize}
\item Function generator: Utility $\to$ Output Setup $\to$ Load $\to$ High Z
\item Oscillator: Trigger Menu $\to$
$\begin{cases}
1.\text{Trigger Mode} \to \text{Basic}\\2.\text{Edge Trigger (Rising Edge)}\\\text{3.Trigger Settings} \to \text{DC Coupling}
\end{cases}$
\item RC Circuit: $R=1K\Omega$, and $C=1\mu F$
\end{itemize}
\subsection{Part 1: Step Response}
\begin{itemize}
\item Function generator:
\\Square Wave \qquad Vpp: 1V \qquad Frequency: 100Hz
\item Oscillator:
CH1: 200mV/div \qquad CH2: 200mV/div \qquad Time:2ms
\end{itemize}
\subsection{Part 2: Pulse Response}
\begin{itemize}
\item Function generator: Pulse frequency: 100Hz
\\1. Width: 1ms \qquad  A:100mV
\\2. Width: 0.5ms\qquad A:200mV
\end{itemize}
\subsection{Part 3: Ramp Response}
\begin{itemize}
\item Function generator:
\\Ramp \qquad Vpp:100mV \qquad frequency:100Hz
\end{itemize}
\subsection{Part 4: Sine Response}
\begin{itemize}
\item Function generator: 10 Vpp
\item Get the result of $V_{out}/V_{in}$, time shift and phase shift when the frequency is at 50, 500, 5000 respectively
\item Compare the results with ideal case
\end{itemize}
\section{Experimental Results}
\subsection{Part 1:Step Response}
\begin{figure}[H]
\centering
\includegraphics[scale=0.2]{p2.jpg}
\caption{Step Response}
\end{figure}
\subsection{Part 2:Pulse Response}
\begin{figure}[H]
\centering
\includegraphics[scale=0.2]{p3.jpg}
\caption{Pulse Response 1}
\includegraphics[scale=0.2]{p4.jpg}
\caption{Pulse Response 2}
\end{figure}
\subsection{Part 3:Ramp Response}
\begin{figure}[H]
\centering
\includegraphics[scale=0.2]{p5.jpg}
\caption{Ramp Response}
\end{figure}
\subsection{Part 4:Sine Response}
\begin{table}[H]
\centering
\begin{tabular}{|c|c|c|c|}
\hline
Frequency (Hz) & Vout/Vin & Time Shift & Phase Shift \\ \hline
50             &0.9626          &1.02[ms]            &$-18.8^o$             \\ \hline
500            &0.3689          &410[$\mu$s]            & $-73^o$            \\ \hline
5k             &0.0359          &45[$\mu$s]            & $-82^o$            \\ \hline
\end{tabular}
\end{table}
\section{Error Analysis and Discussion}
\subsection{Error Analysis}
\subsubsection{Part 1:Step Response}
In our first part of the experiment, the square wave function can be formed by infinite step function, so we compare our figure with the step response, whose figure looks like this
\begin{figure}[H]
\centering
\includegraphics[scale=1]{p6.jpg}
\caption{Step Response}
\end{figure}
As we can see, this figures is quite similar to the one in experimental result, which means we got the right results in our first part.
\subsubsection{Part 2:Pulse Response}
In our second part, we dealt with the pulse train function, and compared our figure with the figure of pulse response
\begin{figure}[H]
\centering
\includegraphics[scale=1]{p7.jpg}
\caption{Pulse Response}
\end{figure}
Also, its quite similar to the figure we got in experimental results, which means we got the right result in second part.
\subsubsection{Part 3:Ramp Response}
In the third part, compare our figure to the ramp response
\begin{figure}[H]
\centering
\includegraphics[scale=1]{p8.jpg}
\caption{Ramp Response}
\end{figure}
Obviously, our figure is quite same to the standard ramp response, so we succeed in getting the true results.
\subsubsection{Sine Response}
In part 4 we focused on the sine response, and figured out the values of $V_{out}/V_{in}$, Time Shift and Phase Shift at different Frequency(50Hz, 500Hz and 5kHz)
\begin{table}[H]
\centering
\begin{tabular}{|c|c|c|c|}
\hline
Frequency (Hz) & Vout/Vin & Time Shift & Phase Shift \\ \hline
50             &0.9626          &1.02[ms]            &$-18.8^o$             \\ \hline
500            &0.3689          &410[$\mu$s]            & $-73^o$            \\ \hline
5k             &0.0359          &45[$\mu$s]            & $-82^o$            \\ \hline
\end{tabular}
\caption{Experimental Results}
\end{table}
\qquad \\
\begin{table}[H]
\centering
\begin{tabular}{|c|c|c|c|}
\hline
Frequency (Hz) & Vout/Vin & Time Shift & Phase Shift \\ \hline
50             &0.9540          &0.9689[ms]            &$-17.4^o$             \\ \hline
500            &0.3033          &402[$\mu$s]            & $-72.3^o$            \\ \hline
5k             &0.0318          &49[$\mu$s]            & $-88.2^o$            \\ \hline
\end{tabular}
\caption{Theoretical Results Calculated in Prelab Exercise}
\end{table}
\qquad \\
\begin{table}[H]
\centering
\begin{tabular}{|c|c|c|c|}
\hline
Frequency (Hz) & Vout/Vin & Time Shift & Phase Shift \\ \hline
50             &0.0086          &0.0511[ms]            &$1.4^o$             \\ \hline
500            &0.0656          &8[$\mu$s]            & $0.7^o$            \\ \hline
5k             &0.0041          &4[$\mu$s]            & $6.2^o$            \\ \hline
\end{tabular}
\caption{Absolute Error}
\end{table}
\qquad \\
\begin{table}[H]
\centering
\begin{tabular}{|c|c|c|c|}
\hline
Frequency (Hz) & Vout/Vin & Time Shift & Phase Shift \\ \hline
50             &0.90\%          &5.27\%            &8.05\%             \\ \hline
500            &21.62\%          &1.99\%            &0.97\%            \\ \hline
5k             &1.29\%          &8.16\%          &7.03\%            \\ \hline
\end{tabular}
\caption{Relative Error}

\end{table}
As shown on the tables above, compare to the theoretical results we got before, they are quite close to each other, the relative error is below 8\%.
\subsection{Discussion of Error}
Though we did a good job and got the similar results comparing to the theoretical results, but there still are 4 errors in Part 4, we think there are 2 reasons.
\begin{enumerate}
\item First, is the unavoidable error, which is caused by the instruments we used. In the experiment, we were supposed to use the resistor with 1000$\Omega$, but actually, when we measured the resistor, the actual value is 985$\Omega$, and since in the ideal results, we assumed that $R$ is constant and equals to 1000$\Omega$, so obviously this will cause the errors, and it’s unavoidable since we can’t find a ideal resistor. Also, the connecting clamp might also have resistance, however, we ignore them in our ideal results.
\item Second, is the avoidable error, which is caused by ourselves. Though we called it the “avoidable error”, but actually it’s hard for us to avoid it, but we can try to decrease it. In the experiment, when we were reading the data from screen, actually the data is “shaking”, it was jumping up and down in a small range. And we just randomly chose one value in this range. This would cause some errors. To reduce the influence of this kind of errors, maybe next time we should choose the value which is exactly at the middle of the range, and repeat this experiment three or four times, choose the average value as the final answer.
\end{enumerate}
\section{Conclusion}
\subsection{Prelab}
By doing the prelab assignments, we take a review of the basic concepts of linear timeinvariant systems, like using the convolution to calculate the output signal. And also, we reviewed some knowledge about RC circuit we learned last year in Ve 215. And we got the ideal output when the input is a step, a pulse, or a more complicated siganl.
\subsection{During Lab}
We used and became familar with the laboratory equipments, like the power supply, signal generator and digital oscilloscope. By measuring the data and getting the figures by ourselves, we had a deeper understanding of the concepts in LTI system and RC circuit.
\subsection{After Lab}
we comparing the ideal results with the figures and data we got from the lab, and find they are quite similar to each other, which confirmed the concepts we learned before. Also, after we analyzing the errors, we got to know how to improve our steps next time.
All in all, this lab is quite successful, we got the results we want, we reviewed the previous knowledge and learned new knowledge at the same time.

\end{document}