% Alek Westover writeup draft

\documentclass[twocolumn, twoside, 11pt]{article}
\author{\vspace{-2ex}Alek Westover\vspace{-2ex}}
\title{\vspace{-8ex}Cache Efficient Parallel Partition\vspace{-2ex}}
\date{\vspace{-2ex}June 26, 2019\vspace{-2ex}}

\renewcommand{\thesubsection}{\thesection.\alph{subsection}}
\usepackage{graphicx}
\usepackage{listings}
\usepackage{amsfonts}

\usepackage{amsmath}
% defines a new function
\def\polylog{\operatorname{polylog}}
\def\pred{\operatorname{pred}}
\def\E{\operatorname{\mathbb{E}}}
\def\Pr{\operatorname{Pr}}

\usepackage{mathtools}
\DeclarePairedDelimiter{\floor}{\lfloor}{\rfloor}
% \floor*{\frac{x}{y}} actually does a nice looking floor!
\DeclarePairedDelimiter{\paren}{(}{)}
% paren*{ } gives the correct size
\newcommand{\defn}[1]       {{\textit{\textbf{\boldmath #1}}}}

\renewcommand{\paragraph}[1]{\vspace{0.09in}\noindent{\bf \boldmath #1.}}

\usepackage{amsthm}
\usepackage{amssymb}
\usepackage{algorithm, algpseudocode}
% \usepackage[noend]{algpseudocode} -- this would get rid of the endifs, maybe worth thinking about for code concision

%%%% Theorem stuff
\newtheorem{theorem}{Theorem}
\newtheorem{lemma}{Lemma}
\newtheorem{proposition}{Proposition}

\begin{document}
\maketitle
\begin{abstract}
% do I need to mention that it is "in-place" or is that part of being cache efficient?
	We present a cache-efficient algorithm for parallel partition that has work $O(n)$ and span $O(\log n \log\log n)$.
	This imediately gives us a cache-efficient quicksort algorithm with work $O(n\log n)$ and span $O(\log^2 n \log\log n)$.

	We also present an exceptionally simple cache-efficient algorithm for parallel partition that has work $O(n)$ and span $O(\log^2 n)$. Despite its slightly worse span, we show that this algorithm performs well in practice.
\end{abstract}

\section{Introduction}
\paragraph{Serial Partition Problem}
We breifly describe the serial partition problem and the solution which is used extensively in our parallel partition algorithms.
The serial partition problem takes in an array, and then reorders the array according to a decider function and returns the number of predecessors in the array.
Upon completion of the serial partition algorithm, which runs fully in-place, if $k$ is the number of predecessors in the array then for all $i < k$, $A[i]$ is a predecessor, and for all $i \ge k$, $A[i]$ is a successor.
In the case of the decider function "$A[i] \leq \text{pivot value}$", as is the case in the partition used for quicksort to sort an array in ascending order, this corresponds to placing all elements with values less than or equal to the pivot value before the elements with values greater than or equal to the pivot value.
Thus, by recursively sorting the predecessors and the successors, the entire array is sorted.
This is because the predecessor portion of the array is sorted, the successor portion of the array is sorted, and the predecessors occur before the successors, so the whole array is sorted.
The serial partition algorithm works as follows:

% \begin{samepage}
\begin{enumerate}
	\item{Initialize \defn{low} to point at the beginning of the array, and initialize \defn{high} to point at the end of the array}
	\item{Increment low until $A[\text{low}]$ is a successor}
	\item{Decrement high until $A[\text{high}]$ is a predecessor}
	\item{Swap values $A[\text{low}]$ and $A[\text{high}]$ in the array}
	\item{Repeat steps 3-5 until $\text{high} \geq \text{low}$ which means that all elements in the array have been processed}
	\item{If $A[\text{low}]$ is a predecessor increment $A[\text{low}]$ by $1$ so that $A[\text{low}]$ is the first successor in $A$, which is now partitioned}
\end{enumerate}
% \end{samepage}

See Algorithm \ref{alg:serialPartition} for a pseudocode implementation of this algorithm.
Note that the serial partition described makes a single pass over the $n$ elements in the array meaning that it incurs $O(n)$ cache misses, and also it has running time (i.e. work) $O(n)$.

\vspace{5mm}
\begin{samepage}
\begin{algorithm}
	\caption{Serial Partition}
	\label{alg:serialPartition}
	\begin{algorithmic}
		\State $\text{low} \gets 0; \text{high} \gets n-1$
		\While{$\text{low} < \text{high}$}
			\While{$A[\text{low}] \leq \text{pivotValue}$}
				\State $\text{low} \gets \text{low}+1$
			\EndWhile
			\While{$A[\text{high}] > \text{pivotValue}$}
				\State $high \gets high-1$
			\EndWhile
			\State Swap $A[\text{low}]$ and $A[\text{high}]$
		\EndWhile
		\If{$A[\text{low}] \leq \text{pivotValue}$}
			\State $low \gets low+1$
		\EndIf
	\end{algorithmic}
\end{algorithm}
\end{samepage}

\paragraph{Further Preliminaries}
We introduce some key ideas in analysis of the algorithms presented in this article.
Many of the algorithms we present have worst case performances that are highly unlikely to be realized.
For instance, if in quicksort a random selection of a pivot value results in only 1 element being labelled a successor, then the recursive subproblems are very unevenly sized.
If the size of the problems is $n, n-1, n-2,\ldots,2$ then quicksort performs at its worst case performance, which is $O(n^2)$.
This however is very unlikely.
We can formalize this with the notion of saying that the event does not happen \defn{with high probability}.
An event is said to happen with high probability in $n$ if the probability of the event occurring is $1-n^{-c}$ for some $c>0$.
Intuitively, if we can show that our algorithm has a certain span with high probability, then we are confident that the algorithm achieves this span.

We define throughout our analysis $\mu$ to be the fraction of predecessors in the array $A$.
In implementing the partition algorithm we do not have control over the pivot value, it is an input to our problem.
The pivot value together with the array $A$ and the decider function determine the fraction $\mu$.
Throughout our analysis we assume that $\mu$ is a constant not dependent on $n$.

\paragraph{Factors to Consider}
We now describe the desirable characteristics of a parallel partition algorithm.
The \defn{work} of an algorithm, $T_1$, is the running time of the algorithm in serial.
The \defn{span} of an algorithm, $T_\infty$ is the running time of the algorithm when run on an infinite number of processors.
The running time of the algorithm on $p$ processors, $T_p$, is determined by its work and span.
This is formally expressed in Brent's theorem which asserts that
$$\frac{T_1}{p} \le T_p \le \frac{T_1}{p} + T_\infty.$$
Thus optimizing the work and span of an algorithm will optimize its performance on $p$ processors.
Another important consideration is auxiliary memory usage.
It is desirable to be as close to in-place as possible because a user might not have any extra space that our algorithm can use if the array to be partitioned takes up nearly all of their machine's memory.
Finally, it is important to analyze whether or not the algorithm is cache efficient, which dependends on if the algorithm is in-place, because this determines the \defn{memory bandwidth bound} on the algorithm.
A \defn{cache miss} is when the algorithm must access a value that is not in the cache already.
Doing this takes time, and the memory banks on a computer can only service a limited number of memory access attempts per interval of time, so if there are lots of requests to the memory bank happening then the algorithm slows down, and we say it is memory bandwidth bound.
For the partition problem each element must be read at least once, so there are at least $n$ cache misses.
We consider all of these properties in our development of a parallel partition algorithm, and our algorithm does well according to all of these metrics.

\paragraph{Prior Work}
Prior work has been done on the parallel partition problem.
William Kuszmaul developed an in-place parallel partition algorithm.
Unfortunately, despite theoretical guarantees, his algorithm was outperformed by an optimized but higher span algorithm.
The problem with this parallel partition algorithm in terms of speed in practice is its memory bandwidth bound.
The reason for the memory bandwidth bound is the number of cache misses that the algorithm incurs.
Thus reducing the number of cache misses is a promising place to look in order to speed up parallel partition, which motivates our algorithm.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% INSERTING MY THING HERE
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Note: recopy stuff not integrated in after Bill edits my thing!!! This is mostly just in here atm so I can delete stuff from my old writeup

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% THIS SECTION GOES WITH THE PARAGRAPH ON OUR CACHE MODEL (that paragraph gets prepended to this)
\section{Cache Efficient Parallel Partition for Random Inputs}
We now present an algorithm for parallel partiition developed by Francis and Pannan, which we call the \defn{Strided Algorithm}.
The Strided Algorithm is thought to use very few passes over the array, although we could not find any accurate theoretical analysis of the algorithm.
Having a low number of cache misses is desirable because it makes the algorithm run much faster, overcoming the problem of memory bandwidth bound.
The Strided Algorithm does not achieve a small number of cache misses for all inputs, but it does achieve this when used on some inputs, for example random inputs.
We introduce our model of cache misses and then prove that the Strided Algorithm incurs very few cache misses for random inputs.

\paragraph{Strided Algorithm Description}
Now we describe the algorithm.
The algorithm is given an input array $A[0],A[1],\ldots,A[n-1]$ where for all $i$, $A[i] \in \{0, 1\}$.
The values in the array $A$ were determined independently, and randomly with an equal probability of being either $0$ or $1$.
The Strided Algorithm first logically partitions $A$ into parts $P_0, P_1, \ldots, P_{t-1}$ where
\begin{equation*}
	P_i = \{A[i], A[i+t], \ldots, A[i+(\frac{n}{t}-1)t]\}.
\end{equation*}
Note that we do not need to expend any extra memory to store each part $P_i$ because the part is determined by a formula that can be used to access elements in $P_i$.
The length of $P_i \approx n/t$. This is approximate because if $n \mod t \not\equiv 0$ then some of the $P_i$s have $1$ less element than others.\\
Define $x_i$ to be the number of predecessors in $P_i$.
The algorithm performs a serial partition of each $P_i$ in parallel (on all the $P_is$ at once).
Note that this serial partition is slightly different than a typical serial partition because the elements in $P_i$ are not adjacent in the array $A$, so to go from one value in $P_i$ to the next we must add $t$ rather than simply $1$, and in decrementing we decrement by $t$ instead of by $1$.
Partitioning $P_i$ makes the first $x_i$ elements of $P_i$ predecessors.
Define $v_i = t\cdot x_i+i \approx t\cdot x_i$.
The index $v_i$ is the location in $A$ of the first successor in group $P_i$, because by construction $P_i$ consists of elements that are $t$ apart in the array, so by scaling the index relative to the part $P_i$ by $t$ (and shifting by $i$ so that the first element is aligned) we get the index into $A$.
Define $v_{min} = \min_i{v_i}$ and $v_{max} = \max_i {v_i}$.
Note that after the partitioning all elements in the array $A$ with index less than $v_{min}$ are predecessors, and all elements after and including $v_{max}$ are successors, so to finish the partitioning of the array $A$ we only need to partition a subarray of size $v_{max}-v_{min}$.
Note that we cannot recursively apply the Strided Algorithm to solve the sub problem, because it would behave very poorly on the subarray, and also because, as we demonstrate, the size of the subarray should be very small relative to $n$.
Also note that if we can show that $v_{max} - v_{min}$ is small relative to $n$ then the partition algorithm uses very few passes over the array.

\paragraph{Strided Algorithm Analysis}
We now prove the following theorem about the Strided Algorithm's performance:
\begin{theorem}
Given an array $A[0], A[1], \ldots, A[n-1]$ where each $A[i]$ is chosen randomly independently to be either $0$ or $1$, and a decider function that labels each element with value $0$ a predecessor, and each element with the value $1$ a successor, the Strided Algorithm partitions the array in span that is, with high probability in n, $O(\frac{n}{t}+\sqrt{n\cdot t\cdot\log n})$ where $t$ is a parameter of our choice indicating the number of parts that we break $A$ into, incurring $n\cdot(1+o(1))$ cache misses.
\label{thm:stridedAlg}
\end{theorem}

To do this, we first prove the following lemma:
\begin{lemma}
With high probability in $n$, the size of the recursive subproblem, $v_{max} - v_{min}$, is $O(\sqrt{t\cdot n\log n}).$
\label{lem:epsSmallFP}
\end{lemma}

\begin{proof}[Proof of Lemma \ref{lem:epsSmallFP}]
$ $\newline
\paragraph{A Single $P_i$}
To prove this, we use Chernoff Bounds, which give bounds on the probability of a deviation of a random variable from its mean (expected value) $\mu$ by more than a certain amount.
We first bound the number of predecessors $x_i$ in each part $P_i$, which allows us to bound $v_i$, the index in the array $A$ of the first successor in $P_i$ after $P_i$ has been partitioned.

The Multiplicative Chernoff bound bounds the probability of a random variable such as $x_i$ deviating by more than $\delta \cdot \mu$ from its mean.
Specifically it says,
$$P(x_i>\mu(1+\delta)) = O(e^{-\mu \cdot \delta^2/3}).$$
We want to know what value of $\delta$ makes $x_i\leq\mu(1+\delta)$ with high probability.
To find the smallest $\delta$ satisfying this, we must solve
$$O(e^{-\mu \cdot \delta^2/3}) = n^{-c},$$
where the right hand side comes from the definition of an event being false with high probability, and the left hand side comes from the probability guaranteed by the Chernoff bound.
Solving, we obtain
$$\log e^{-\mu \cdot \delta^2/3} = \log n^{-c},$$
which upon isolating $\delta$ yields,
$$\delta = O\paren*{\sqrt{\frac{\log n}{\mu}}}.$$
Because $x_i$ is the number of predecessors in $P_i$, and $|P_i| \approx \frac{n}{t}$, the expectation of $x_i$ is $\mu = \frac{n}{2t}$ because the values in A were generated independently at random, with $\frac{1}{2}$ probability of being predecessors.
Thus we can assert that
$$\delta = O\paren*{\sqrt{\frac{t\log n}{n}}}.$$
By a nearly identical argument with the Chernoff bound for the probability of $x_i$ deviating below the mean $\mu$ by more than $\mu \cdot \delta$ we get the same result: that deviation by more than $\mu \cdot \delta$ does not happen with high probability.

Thus we have that with high probability
$$|x_i - \mu| = O\paren*{\sqrt{\frac{t\log n}{n}}} \cdot \mu.$$
Once again we can substitute in $\mu=\frac{n}{2t}$ to get,
$$|x_i - \mu| = O\paren*{\sqrt{\frac{t\log n}{n}}} \cdot \frac{n}{2t}.$$
This simplifies to,
$$\Big|x_i - \frac{n}{2t}\Big| = O\paren*{\sqrt{\frac{n\log n}{t}}}.$$
To convert this index to an index into the array A rather than an index in $P_i$, we multiply by $t$ (the gap between the indices into the array A of elements in $P_i$) obtaining,
$$\Big|t\cdot x_i - \frac{n}{2}\Big| = O(\sqrt{t\cdot n\log n}).$$
Equivalently, we can use our name $v_i$ for the index into the array $A$ to write this as
$$\Big|v_i - \frac{n}{2}\Big| = O(\sqrt{t\cdot n\log n}).$$
We now need to move the discussion from just looking at $P_i$ to looking at all $P_i$s.

\paragraph{Union Bound over all $P_i$}
Now, we apply the union bound for with high probability events.
It states that the union of a set of events that are false with high probability is still false with high probability.
Using the union bound allows us to go from saying
\begin{multline*}
\text{for all $i$, with high probability} \\
	|v_i - \frac{n}{2}| = O(\sqrt{t\cdot n\log n}),
\end{multline*}
to saying
\begin{multline*}
	\text{with high probability, for all $i$} \\
	|v_i - \frac{n}{2}| = O(\sqrt{t\cdot n\log n}).
\end{multline*}
This establishes that, with high probability,
$$v_{max}-v_{min} = O(\sqrt{t\cdot n\log n}).$$
\end{proof}
Using this lemma we can now prove Theorem \ref{thm:stridedAlg}.

\begin{proof}[Proof of Theorem \ref{thm:stridedAlg}]
$ $\\
\paragraph{Span}
There are 2 contributions to the span of our algorithm.
First, there is a contribution of $\frac{n}{t}$ for performing serial partitions on each $P_i$ in parallel because serial partition has a running time that is on the order of the input array, and $|P_i| = \frac{n}{t}$.
Second, there is a contribution from performing a serial partition on the subarray of size $v_{min} - v_{max}$ which is $O(\sqrt{t\cdot n\log n})$ by Lemma \ref{lem:epsSmallFP}.
Thus the total span is
$$T_\infty = O(n/t+\sqrt{t\cdot n \log n}).$$ \\
We can choose $t$ in order to minimize span.
To minimize span, ignore the log factor and set
$$n/t = \sqrt{t\cdot n} \implies t=n^{1/3}.$$
This yields the minimum span, because if either term could be decreased, then the function would not be at a minimum, so the terms must be equal.
This yields a span of $O(n^{2/3}).$

\paragraph{Cache Misses}
The number of cache misses in a serial partition is the input size, so the total number of cache misses in this program is $t \cdot \frac{n}{t}$ from the serial partitioning of the $P_i$s in parallel, plus $O(\sqrt{t\cdot n\log n})$ for when the algorithm performs a serial partition on the subarray. So the total number of cache misses is $$n + O(\sqrt{t\cdot n\log n}).$$
To show that the second term in this sum becomes insignificant compared to $n$ as $n$ grows, we take
$$\lim_{n\to\infty}\frac{\sqrt{t\cdot n \log n}}{n} = \sqrt{\lim_{n\to\infty}\frac{t\cdot \log n}{n}}.$$
We can't choose $t>n$, and making $t = o(\frac{n}{\log n})$ is a reasonable constraint which guarantees that,
$$\lim_{n\to\infty}\frac{t}{\paren*{\frac{n}{\log n}}} = 0.$$
Thus, the algorithm only incurs $n+o(n) = n(1+o(1))$ cache misses.
\end{proof}

\paragraph{Spatial locality}
The Strided Algorithm as described above is good because of the low number of cache misses, but it is not good in that there is no spatial locality in referencing elements of each $P_i$.
A way to increase spatial cache friendliness of the algorithm is to redefine the parts $P_i$s using blocks of adjacent elements instead of individual elements each separated by $t$ indices for each $P_i$.
Let $b$ be the \defn{block size}. \\
To compute $v_i$, the index into $A$ of the first successor in the partitioned $P_i$, we must now compute which block the index $x_i$ into $P_i$ falls in, denoted by $w_i$ and where within the block $x_i$ falls, denoted $m_i$ as follows:
$$w_i = \floor*{ \frac{x_i}{b} },\,\, m_i=x_i \mod b.$$
Once we have $w_i$ and $m_i$ we compute $v_i$ as follows:
$$v_i = t\cdot b\cdot w_i + (i-1)b+m_i.$$
But we don't need that much detail.
More simply this is,
$$v_i = O(t\cdot b \cdot w_i) = O\paren*{x_i\cdot t}.$$
Where the absorption into the big-O notation is reasonable because $m_i < b,\,\, i<t$ so the other terms get absorbed in the big-O notation, and we can also drop the floor and cancel the $b$.
Note that this is the same expression we got for the conversion from $x_i$ to $v_i$ as last time (in big-O notation).
So we once again get the same bound on $v_{max}-v_{min}$, which doesn't actually depend on $b$, that is, with high probability
$$v_{max}-v_{min} = O(\sqrt{t\cdot n\log n}).$$

\paragraph{Summary}
It is not satisfactory that this algorithm does not have theoretical guarantees for arbitrary arrays.
We use the ideas of this algorithm to create a new algorithm that has have theoretical guarantees on arbitrary inputs.

\section{Key Algorithmic Ideas}
We add randomization to the Strided Algorithm to make a new algorithm that has theoretical guarantees for arbitrary inputs while remaining cache efficient and in-place.
To accomplish this the algorithm forms groups $G_0,G_1,\ldots,G_{g-1}$ which are each a collection of indices to parts $P_j$.
We will refer to the set that is the union of all parts $P_j$ where $j \in G_y$ as $U_y$. That is,
$$U_y = \bigcup_{j\in G_y}P_j.$$
Then the algorithm partitions each $U_y$ in serial.
Define $\mu_y$ to be the fraction of $U_y$ that is composed of predecessors.
It is important that $|\mu_y - \mu|$ is small for all $y$ with high probability in $n$ so that the algorithm will have a small subproblem to recurse on.
We present 3 versions of the algorithm which are differentiated by their methods of forming groups.

\paragraph{Version 1}
In this version of the algorithm, we form groups $G_0,G_1,\ldots,G_{g-1}$ of indices for certain $P_js$, which are defined in the same way that they were defined in the Strided algorithm, as follows:
\begin{itemize}
	\item Permute the numbers $0,\ldots, t-1$ into $g$ groups of size $O(\polylog n)$.
	\item Sort each group $G_y$ of indices in serial.
	\item For every group $G_y$ perform a serial partition on $U_y$ the union of all $P_j$ where $j \in G_y$. To accomplish this, the algorithm must find the next or previous element in a group starting from the index of some other element in the group. This is accomplished by storing both the index into whichever part $P_j$ the algorithm is currently on, and the index into the group $G_y$ specifying which part $P_j$ the algorithm is currently on. Then to, for example, find the next element the algorithm increments through the part $P_j$ that it is currently in, until it reaches the end of the part, after which the algorithm increments its counter into $G_y$ and starts at the beginning of the next part $P_j$.
	\item Partition $A[v_{min}], \ldots, A[v_{max}-1]$ where $v_{min}=\min{v_j}, v_{max}=\max{v_j}$, and $v_j$ is defined, as in the Strided algorithm, to be the index of the first successor in $P_j$. As before, this guarantees that all elements of $A$ with indices $k<v_{min}$ are predecessors, and all elements with indices $k\geq v_{max}$ are successors.
\end{itemize}
This algorithm has the desirable property that, because we randomly group $O(\polylog n)$ $P_j$s together, the fraction $\mu_y$ of $U_y$ that is composed of predecessors should cluster closely around the fraction $\mu$ of $A$ that is composed of predecessors for all $y$.
Also, we can make the algorithm have small span by performing some of the steps in parallel.
The permutation step can be done in parallel.
The algorithm can also perform the serial partition of all collections $U_y$ in parallel.
The unsatisfactory property of this version of the algorithm is the auxiliary memory that is required by the algorithm.
The algorithm stores $g$ groups of size $O(\polylog n)$ where $g$ is chosen to be $O(n / \polylog n)$, so the algorithm uses $O(n / \polylog n)$ extra space and is not in-place.

\paragraph{Version 2}
In this version of the algorithm we again assign the indices of parts $P_0,\ldots, P_{t-1}$ to groups $G_0,\ldots, G_{g-1}$.
However, in this version the groups are not independent of one another.
Because of this dependence, the algorithm only needs $O(\polylog n)$ auxiliary space to store all of the groups.
The algorithm creates an array $X$ of size $O(\polylog n)$ and sets each $X[i]$ to be an integer chosen uniformly at random from $[0,g-1]$.
The value $X[i]$ is used to determine a unique index $j \in [i\cdot g, (i+1)\cdot g-1]$ for each group indicating which part $P_j$ belongs to the group $G_y$ from this set of indices.
The index for the part from this set of indices that belongs to $G_y$ is $$i \cdot g + ((X[i]+y)\bmod g).$$
This method of assigning parts to groups assigns all $P_j$s to exactly 1 group.
Having independent random variables for each group is not necessary because we apply a union bound over all groups in proving that $|\mu - \mu_y|$ is small for all $y$ with high probability in $n$, and the union bound still applies for random variables that are not independent.
This version of the algorithm is an improvement over the previous algorithm because it does not use excessive auxiliary memory and it has small span, similarly to that of the previous algorithm.
An undesirable property of this algorithm however is that it has some unnecessary overhead.

\paragraph{Version 3}
In this version of the algorithm we eliminate the use of parts $P_j$ in the algorithm.
The set $U_y$ is a union of parts $P_j$ which each contain multiple elements.
The motivation for having $P_j$ contain multiple elements was that in the Strided Algorithm the $P_j$s were the only place where we combined elements into a set to partition in serial in the algorithm.
We need to have collections of elements to make each collection have approximately the same fraction of predecessors as the entire array $A$.
Now however, we are combining elements on 2 levels: we group elements into $P_j$s, and we group $P_j$s together into $G_y$s.
We only need one source of grouping elements though, so we can set $|P_j| = 1$ and the collections $U_0,\ldots,U_{g-1}$ will still contain a collection of elements.
Thus the groups can still be expected to have close to the same fraction $\mu_y$ of predecessors in them as the fraction $\mu$ in the entire array $A$.
Setting $|P_j|=1$ makes $P_j=\{A[j]\}$, demonstrating that $P_j$ is no longer needed because it refers to a single element of $A$.
When the cache block size is $b$ instead of $1$ it is convenient to still have parts $P_j$s and say that each is composed of a single cache block.
Thus, we now refer to $P_j$ as a block, and it is defined to contain elements $A[b\cdot j],\ldots, A[b\cdot (j+1) - 1]$.
Changing the role of the $P_j$s is desirable because it adds simplicity to the algorithm and reduces overhead without sacrificing any desirable characteristics of the algorithm.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% INTEGRATE THIS PARAGRAPH INTO THE SECTION CONTAINING IT
\paragraph{Algorithm Concept}
We now present the \defn{Cache-Efficient Partial-Partition Algorithm} which, like the parallel step of the Strided Algorithm, partitions subsets of the array in parallel such that partitioning the subsets and then a small subarray results in a fully partitioned array.
However, unlike the parallel step of the Strided Algorithm, the Cache-Efficient Partial-Partition Algorithm has high probability guarantees on the size of the unpartitioned subarray for arbitrary inputs.
The Strided Algorithm has guarantees on some inputs, for instance randomly ordered inputs, but the Cache-Efficient Partial-Partition Algorithm uses randomization in the algorithm to obviate the need for any specific type of input.

Interestingly, because the Cache-Efficient Partial-Partition Algorithm has guarantees on arbitrary inputs, the Cache-Efficient Partial-Partition Algorithm can be used to partition the subarray that it generates, unlike in the Strided Algorithm where recursing with the parallel step of the Strided Algorithm would not successfully partition the subarray.
This is because the subarray generated in the parallel step of the Strided Algorithm is non-random in a problematic way: subsequences composed of every $t$-th element in the subarray would already be partitioned, so the parallel step of the Strided Algorithm would not change the array as it partitions these subsequences that are already partitioned, and thus could not finish the partitioning of the array.

The Cache-Efficient Partial-Partition Algorithm forms collections $U_y$, which are similar to the Strided Algorithm's $P_j$s, and performs a serial partition of each $U_y$ in parallel.
We cannot explicitly store each $U_y$, because this would require $O(n)$ memory (or $O(n/b)$ if we use indices of blocks of the array) which would make the algorithm not in-place, and eliminate the algorithm's desirable cache behavior.
However, we can represent all $U_y$s with very little space by making each $U_y$ random, but not independent of other $U_{y'}$s.

Our construction of $U_y$s makes it so that the fraction of predecessors in each $U_y$ will cluster closely around the fraction of predecessors in $A$.
The elements in each $U_y$ are spread out in $A$ so that each section of a certain size in the array will contain the same number of elements from $U_y$.
Define $v_y$ to be the index of the first successor in $U_y$--this is similar to the definition of the splitting position in the Strided Algorithm.
Because of this uniformity in how the elements of $U_y$ are spread out, and the guarantee that the fraction of successors in each $U_y$ will cluster closely around the fraction of successors in $A$, the index $v_y$ of the first successor in collection $U_y$ will be close to all other indices $v_{y'}$.
This means that after each $U_y$ is partitioned, $A$ will be partially partitioned, in the sense that $A[i]$ is a predecessor for all $i < \min_y{v_y}$, and $A[i]$ is a successor for all $i \ge \max_y{v_y}$.
Furthermore, the size of the unpartitioned subarray $A[\min_y{v_y}],\ldots,A[\max_y{v_y}-1]$ will be very small relative to $n$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%5

\section{Cache-Efficient Parallel Partition for Arbitrary Inputs}
\paragraph{Algorithm Description}
We now describe the algorithm in more detail.
Let $b$ be the size of a cache line.
Let $X$ be an array such that $X[i] \in \{0,1,\ldots,g-1\}$ determines which chunk of size $b$ from the array belongs to any group $G_y$ from section $i$ of size $g\cdot b$ in the array.
Let $s = |X|$.
Then $g$, the number of groups $G_y$ the algorithm will form, is determined by $g=\frac{n}{b\cdot s}$
\footnote{If $n \not\equiv 0 \mod b\cdot s$, then we define $g$ as $\Big\lfloor \frac{n}{b\cdot s}\Big\rfloor $.
This means that $g\cdot b \cdot s < n$.
We handle the array of size $n-g\cdot b \cdot s$ separately from the array of size $g\cdot b \cdot s$ that is the focus of our analysis.
Thus for simplicity in the analysis we say that $n=g\cdot b\cdot s$.
It is straightforward to deal with the $n-g\cdot b\cdot s$ elements.
This is done by swapping the extra elements with the first $n-g\cdot b\cdot s$ successors, and then increasing the size of the subproblem to include these new elements.
Because $n-g\cdot b\cdot s$ is very small, this step is not relevant to the analysis, but this is an important detail to handle in order to make the algorithm work. }.
The algorithm performs the following procedure:

\begin{itemize}
	\item Logically partition the array $A$ into blocks $P_j$ each of $b$ adjacent elements, i.e. $${P_j = \{A[b\cdot j], A[b\cdot j + 1], \ldots, A[b\cdot (j+1)-1]\}}.$$
	\item Form the array $X$, where each element $X[i]$ is an integer chosen randomly at uniform from $[0, g-1].$
	\item The values in $X$ determine $g$ groups $G_y$ each of $s$ indices to $P_j$s. The index for block $i$ of group $G_y$ is determined by
		$$G_y[i] = (X[i] + y) \bmod g + i\cdot g.$$
		Note that we do not need to store the indices of the members of each group even though the groups are random, because the constituent blocks of a group are determined by the group index and the array $X$.
		This means that as long as $|X|$ is made small, which it will be, we do not use significant extra space in creating this array.
	\item Define $U_y$ to be the union of all blocks that belong to group $G_y$. That is,
			$$U_y = \bigcup_{j\in G_y} P_j.$$
		The algorithm performs a serial partition on each $U_y$ in parallel.
	\item Define $v_y$ to be the index of the first successor in $U_y$, and define $v_{\text{min}}=\min_y{v_y}$,  $v_{\text{max}}=\max_y{v_y}$
		\footnote{When calculating $v_{\text{max}}$ and $v_{\text{min}}$ we do not want to store the $v_y$s because this would require $O(g)$ memory.
In order to avoid storing the $v_y$, we compute $v_{\text{max}}$ and $v_{\text{min}}$ while partitioning each $U_y$ in parallel.
To facilitate this, we implement a parallel for loop with a recursive divide-and-conquer strategy.
We have a function that takes in a subset of the $U_y$, partitions these $U_y$ in parallel, and then returns the minimum and maximum $v_y$ among the $U_y$ that the function partitioned.
This function operates by divide-and-conquer, i.e. it spawns processes that in parallel recursively apply this function to 2 disjoint, approximately equal sized, subsets of the subset of $U_y$s that the function was given, and then the function computes its $v_{\text{min}}, v_{\text{max}}$ as the $\min$ of the $v_{\text{min}}$s reported by the subprocesses and the $\max$ of the $v_{\text{max}}$s reported by the subprocesses.}.
Note that the array is partially partitioned, i.e. $A[i]$ is a predecessor for all $i < v_{\text{min}}$, and $A[i]$ is a successor for all $i \ge v_\text{max}$.
\end{itemize}

%% note: g > s should hold
\paragraph{Algorithm Analysis}
We now prove the following general proposition about the Cache-Efficient Partial-Partition Algorithm:
\begin{proposition}
	\label{prop:generalResult}
	Let $A$ be an array of size $n$;
	Let $\mu$ be the fraction of predecessors in $A$;
	Let $b$, the size of a cache line, be $O(\operatorname{polylog}{n})$;
	Let $\epsilon \in (0, 1)$ be a failure probability;
	Let $\delta \in (0, 1)$;
	Let $s \in \Theta\left(\frac{\log (n/\epsilon)}{\delta^2}\right)$ satisfying $s>\frac{\log(2n) - \log(b\epsilon)}{2\delta^2}$.

	The Cache-Efficient Partial-Partition Algorithm, when executed on array $A$:
	achieves work $O(n)$;
	achieves span $O\paren*{b \cdot s}$;
	incurs $\frac{s+n}{b} + O(1)$ cache misses (assuming the array $X$ is pinned to cache);
	and leaves an unpartitioned subarray $A[v_{\text{min}}],\ldots, A[v_{\text{max}}-1]$ which, with probability $1-\epsilon$, has size $$v_{\text{max}}-v_{\text{min}} < 4 n \delta.$$
\end{proposition}
\begin{proof}
	The algorithm accesses each element of the array once, resulting in work $O(n)$.
	Recall that by the Resource Augmentation Theorem we can assume small arrays and values that are repeatedly used to be pinned to cache.
	The contribution to the algorithm's cache misses are then:
	$n/b$ from accessing each block of $A$, $s/b$ for instantiating the array $X$, and $O(1)$ for performing various other tasks that require a constant number of cache misses.
	This results in the total number of cache misses being $$\frac{n+s}{b}+O(1).$$
	The algorithm has span $$O(n/g) = O(b\cdot s) $$
	because of the $g$ serial partitions of groups that it performs in parallel on arrays of size $b\cdot s$.
	To show that with probability $1-\epsilon$ the size of the unpartitioned subarray is bounded above by $4n\delta$, we consider the fraction of elements in each collection $U_y$ that are predecessors, which we denote by $\mu_y$.
	Note that each $\mu_y$ is the average of $s$ independent random variables, where random variable $i$ is the fraction of $P_{G_y[i]}$ that is composed of predecessors, a random variable in $[0, 1]$.
	The groups are constructed in a symmetric way such that for each group, the probability of the group getting any specific block is the same as the probability of any other group getting that block.
	Because of this symmetry, $\E[\mu_y]$--a characteristic on the group--is the same for all $y$.
	Furthermore, $\E[\mu_y] = \mu$ for all $y \in \{0,\ldots,g-1\}$ because the average of the fraction of predecessors in each group is the fraction of predecessors in the entire array.
	Thus we can apply Hoeffding's inequality (i.e. A Chernoff Bound for a random variable on $[0,1]$ rather than on $\{0,1\}$) to each $\mu_y$ to show that they are concentrated around their shared expected value $\mu$, i.e.
	$$\Pr[|\mu_y - \mu| \geq \delta] < 2\exp(-2s\delta^2). $$
	Substituting in $s$ which was chosen $s > \frac{\log (2n) -\log (b\epsilon)}{2\delta^2}$, we find that for all $y \in \{0,\ldots, g-1\}$,
	$$\Pr[|\mu_y - \mu| \geq \delta] < 2\exp\Big({-2} \frac{\log (2n/(b\epsilon))}{2\delta^2} \delta^2\Big) = \frac{\epsilon}{n/b} < \frac{\epsilon}{g}. $$
	We use this bound on the probability of any individual group $U_y$ failing to meet the condition $|\mu-\mu_y| < \delta$ to bound the probability that at least one of the groups $U_0, \ldots, U_{g-1}$ fails to meet the condition.
	Note that the probability of at least one group failing is:
	$$\Pr\Big[\bigvee_{y=0}^{g-1} |\mu_y - \mu| \geq \delta\Big].$$
	This is bounded by
	$$\Pr\Big[\bigvee_{y=0}^{g-1} |\mu_y - \mu| \geq \delta\Big] \leq \sum_{y=0}^{g-1} \Pr[|\mu_y - \mu| \geq \delta] < \epsilon.$$
Thus the event occurs with probability bounded above by $\epsilon$, the specified failure probability.
To complete the proof we will show that the occurrence of the event that all $y$ simultaneously satisfy $|\mu - \mu_y| < \delta$ implies that the size of the unpartitioned subarray is bounded above by $4n\delta$.

	Intuitively, because the elements in each group are distributed evenly throughout the array, $\mu_y\cdot n$ approximately determines $v_y$, so we can bound $v_y-\mu\cdot n$ for all $y$ using our bound on $|\mu-\mu_y|$.
	The index for the block of group $G_y$ that contains $v_y$ is $$G_y[\mu_y\cdot s]= (X[\mu_y \cdot s] + y) \bmod g + \mu_y \cdot s\cdot g.$$
	Because the entries of $X$ are random, we chose to bound $G_y[\mu_y\cdot s]$ without referencing $X$, only referencing parameters and variables of the problem.
	This bound is,
	$$ \mu_y \cdot s\cdot g \leq G_y[\mu_y\cdot s] \leq  \mu_y \cdot s\cdot g + g-1.$$
	Then, because block $P_{G_y[i]} = \{A[b\cdot i], \ldots, A[b\cdot i + b-1]\}$, we can bound $v_y$ as $$\mu_y\cdot s\cdot g\cdot b \leq v_y \leq (\mu_y \cdot s\cdot g + g-1)\cdot b + b-1.$$
	Note that $n=s\cdot g\cdot b$.
	Rearranging, we have
	$$0 \leq v_y - \mu_y\cdot n < g\cdot b.$$
	We desire to bound $v_y - \mu\cdot n$, which can be done using our bound on $|\mu_y - \mu|$ as follows:
	$$ - \delta\cdot n \leq v_y - \mu\cdot n < \delta\cdot n + g\cdot b.$$
	Thus, the concentration of all $v_y$s implies that with probability $1-\epsilon$,
	$$v_{\text{max}}-v_{\text{min}}<n(2\delta) + g\cdot b.$$
	We express this in terms of the algorithm's parameters, using the fact that $g\cdot b = n/s$,
	$$v_{\text{max}}-v_{\text{min}}<n\paren*{2\delta + \frac{2\delta^2}{\log (2n) - \log(b\epsilon)}} < 4n\cdot\delta.$$
\end{proof}
We can use this result to analyze two Parallel full Partition algorithms that use the Cache-Efficient Partial-Partition Algorithm.

\begin{enumerate}
	\item The \defn{Cache-Efficient Full-Partition Algorithm} algorithm achieves the best theoretical span and cache efficiency for appropriate parameter choice.
	\item The \defn{Grouped Partition Algorithm} has slightly higher span, but has a very small number of cache misses, and is exceptionally simple to implement.
\end{enumerate}

Both algorithms start by using the Cache-Efficient Partial-Partition algorithm with parameters $\epsilon = 1/n^c$ for $c$ of our choice (i.e. with high probability in $n$) and parameter $\delta$ which can be toggled to achieve a trade off between span and cache misses.
The algorithms then partition the unpartitioned subarray with their own methods.

The Cache-Efficient Full-Partition Algorithm uses the algorithm given by Theorem \ref{thminplace} which has span $O(\log\log n \log n)$ to partition the subarray.
This approach is motivated by the observation that once the problem size has been reduced cache behavior is less important, so we can switch to the algorithm that optimizes span rather than cache misses to solve the subproblem.

The Grouped Partition Algorithm partitions the subarray by repeatedly applying the Cache-Efficient Partial-Partition algorithm.
The recursive Cache-Efficient Partial-Partition algorithms use the same parameter $\epsilon$ as the top-level, in order to still guarantee success with high probability in $n$, and use $\delta = 1/8$ in order to at least halve the size of the problem size at each iteration.
% Using this higher value of $\delta$ is ok because the lower levels of recursion are small, so it doesn't matter a lot what you do with them.

We now establish the following theorem about the Cache-Efficient Full-Partition Algorithm algorithm in general and a corollary about an interesting setting of its parameters.
\begin{theorem}
	\label{thm:fullPartition}
	The Cache-Efficient Full-Partition Algorithm algorithm using parameter $\delta\in(0,1)$ satisfying $\delta \ge \Omega(n^{-1/4})$: has work $O(n)$, and with high probability in $n$, achieves span $$O\paren*{\log n \log\log n +\frac{b\log n}{\delta^2}},$$
and incurs fewer than
$$(n+O(n\delta))/b$$
cache misses.
\end{theorem}

%% note: b/loglogn > 1 in reality, but the constant factors make it so that delta is still < 1
An interesting corollary of the above theorem concerns what happens when we chose $\delta$ to optimize span.
This can be done with an extreme setting of $\delta$.
This is interesting because it shows that it is possible to achieve low span along with a small number of cache misses.
\begin{corollary}[Corollary of Theorem \ref{thm:fullPartition}]
	\label{cor:fullPartition}
	The Cache-Efficient Full-Partition Algorithm algorithm using $\delta = \Theta\big(\sqrt{b/\log\log n}\big)$, where $b$ is chosen to satisfy $\sqrt{b/\log\log n} = o(1)$: achieves work $O(n)$, and with high probability in $n$, achieves span $O(\log n \log\log n)$ and incurs fewer than $(n+o(n))/b$ cache misses.\\\\
\end{corollary}

\begin{proof}[Proof of Theorem \ref{thm:fullPartition}]
	Using Proposition \ref{prop:generalResult}, we find the relevant statistics for the step of this algorithm where we apply the Cache-Efficient Partial-Partition Algorithm.
	Note that by our choice of $\epsilon$, $s=O\left(\frac{\log n}{\delta^2}\right)$.
	The top layer has work $O(n)$, span $O\paren*{\frac{b\log n}{\delta^2}},$
	and incurs fewer than
	$$\frac{n}{b}+O\paren*{\frac{\log n}{b\delta^2}}+O(1)$$
	cache misses.
	The subproblem has size less than $4n\delta$, so using the algorithm described by Theorem \ref{thminplace}, we achieve span
	$$O(\log n\delta \log\log n\delta) = O(\log n \log\log n)$$
	and work $O(n\delta) \le O(n)$ in solving the subproblem.
	Thus the total work is $O(n)$, and the total span is
	$$O\paren*{\log n \log\log n +\frac{b\log n}{\delta^2}}.$$
	In solving the subproblem the number of cache misses that we incur is linearly dependent on the size of the subarray, so the total number of cache misses is
	$$\frac{n}{b}+O\paren*{\frac{\log n}{b\delta^2} + \frac{n\delta}{b}}+O(1).$$
	Note that $\delta > \Omega(n^{-1/4})$ implies
	$$O\left(  \frac{\log n}{\delta^2} \right) < O\left(\frac{n^{1/8}}{\delta^2}\right) < O\left(n^{1/2+1/8}\right) < O\left(n^{3/4}\right) < O(n\delta) $$
	so the number of cache misses simplifies to
	$$(n+O(n\delta))/b + O(1).$$
	Recall that $b \le O(\operatorname{polylog}{n})$, so $n\delta / b \ge \Omega(1)$, which makes the total number of cache misses
	$$(n+O(n\delta))/b.$$

\end{proof}

\begin{proof}[Proof of Corollary \ref{cor:fullPartition}] We use $\delta = \sqrt{b/\log\log n}$ in the result proved in Theorem \ref{thm:fullPartition}. \\
	First note that the assumptions of Theorem \ref{thm:fullPartition} are satisfied because $$O(\sqrt{b/\log\log n}) > \Omega(n^{-1/4}).$$
	The algorithm achieves work $O(n)$.
	With high probability in $n$ the algorithm achieves span
	$$O\paren*{\log n \log\log n +\frac{b\log n}{\delta^2}} = O(\log n\log\log n).$$
	With high probability in $n$ the algorithm incurs fewer than
	$$(n+O(n\delta))/b = (n+O(n\sqrt{b/\log\log n}))/b$$
	cache misses.
	By assumption $\sqrt{b/\log\log n} = o(1)$, so this reduces to
	$$(n+o(n))/b$$
	cache misses.
	%% explain why span and cache behavior are very good? (kind of already did before, but reiterating probably doesn't hurt...)
\end{proof}

%%% Should make the distinction between n and the array size of subproblems more clear?
%%% n always refers to the size of the top level
%%% 2^{-i} delta n refers to an upper bound on the size of a subproblem
\begin{theorem}
	\label{thm:groupedPartitionAlg}
	The Grouped Partition Algorithm using parameter $\delta_0\in(0,1)$ satisfying $\delta_0 \ge \Omega(n^{-1/4})$: achieves work $O(n)$, and with high probability in $n$, attains span
	$$O\left(b\left(\log^2 n + \frac{\log n}{\delta_0^2}\right)\right)$$
	and incurs $(n+O(n \delta_0))/b$ cache misses.
\end{theorem}

\begin{corollary}[Corollary of Theorem \ref{thm:groupedPartitionAlg}]
	\label{cor:groupedPartitionAlg}
	The Grouped Partition Algorithm using parameter $\delta_0=1/\sqrt{\log n}$:
	achieves work $O(n)$, and with high probability in $n$, attains span $O(b\log^2 n)$ and incurs $(n+o(n))/b$ cache misses.
\end{corollary}

\begin{proof}[Proof of Theorem \ref{thm:groupedPartitionAlg}]
	By Theorem \ref{prop:generalResult}, the top level of the algorithm has work $O(n)$, span $O\Big(b\frac{\log n}{\delta_0^2}\Big),$
	and incurs $$\frac{s+n}{b} + O(1)$$ cache misses.
	On the top layer of recursion the algorithm reduces the problem size by at least a factor of $4\delta$.
	On lower layers of recursion with high probability in $n$ the algorithm reduces the problem size by at least a factor of $1/2$ at each iteration by our choice of $\delta'=1/8$ as the parameter for lower levels of the Cache-Efficient Partial-Partition algorithm.
	The size of the problem on lower levels thus is bounded above by
	$$2^2n\delta_0, 2^1n\delta_0,2^0n\delta_0, 2^{-1}n\delta_0, \ldots$$
	This sequence of sizes terminates, but the sum of the terms in the infinite sequence with terms of the form $\{2^{2-i}n\delta\}_{i=0}^{\infty}$ is an upper bound for the sum of the sizes of the lower levels of recursion.
	 The bound is
	 $$\sum_{i=0}^{\infty}2^{2-i}n\delta_0 = 8n\delta_0.$$
	This means that the total work of the algorithm is bounded above by $O(8n\delta_0) + O(n)$, so the total work is $O(n)$.

	In order to compute the span of the algorithm, we apply Proposition \ref{prop:generalResult} to the subarray, which has size bounded above by $2^{2-i}n\delta_0$, and has parameter $\delta'=1/8$.
	Thus, level $i$ contributes
	$$O(b\cdot s)=O\Big(b\frac{\log(2^{-i}n\delta_0/b)}{(\delta')^2}\Big)=O(b(\log(n\delta_0/b)-i))$$ to the span.
	Now note that because the algorithm at least cuts the problem size in half each time we will reduce the problem size to the base case in at most $\log (4n\delta_0)$ levels.
	Thus, to get the total span on the lower levels of recursion, we must add together $O(\log(n\delta_0))$ terms of the form $O(b(\log (n\delta_0/b) - i))$.
	This makes the total span on lower levels of recursion $O(b\log^2 (n\delta_0))=O(b\log^2 n)$, and the total span for the algorithm
	$$O\left(b\left(\log^2 n + \frac{\log n}{\delta_0^2}\right)\right).$$
	To compute the total number of cache misses of the algorithm, we add together $(n+s)/b+O(1)$ for the top level, and then at most
	$$\sum_{0 \leq i<\log 4n\delta}\frac{1}{b}\cdot O\paren*{2^{2-i}n\delta_0 + \frac{\log 2^{2-i}n\delta_0/b}{(1/8)^2}}$$
	for lower levels.
	The first term is the sum of a geometric series and thus is $O(n\delta_0)/b$.
	We can bound the second term by $O(\log^2 n)/b$ by taking a larger term at each index in the sum.
	Then the total number of cache misses for the algorithm is
	$$\frac{1}{b}\left(n+\frac{\log n}{\delta_0^2 }\right) + O(1) + O(\log^2 n) / b + O(n\delta_0) / b = (n+O(n\delta_0))/b.$$
\end{proof}

\begin{proof}[Proof of Corollary \ref{cor:groupedPartitionAlg}]
	We use $\delta_0 = 1/\sqrt{\log n}$ in the result proved in Theorem \ref{thm:groupedPartitionAlg}.
	First note that the assumptions of Theorem \ref{thm:groupedPartitionAlg} are satisfied because
	$$\delta_0=1/\sqrt{\log n} \ge \Omega(n^{-1/4}).$$
	The algorithm has work $O(n)$.
	With high probability in $n$ the algorithm has span
	$$O\left(b\left(\log^2 n + \frac{\log n}{\delta_0^2}\right)\right) =  O(\log^2 n),$$
	and incurs
	$$(n+O(n\delta_0))/b = (n+O(n/\sqrt{\log n}))/b = (n+o(n))/b$$
	cache misses.
\end{proof}

% In addition to these desirable theoretical guarantees, these algorithms performs well in practice, as shown in the experimental section.
% Also, because the Cache-Efficient Partial-Partition Algorithm achieves an optimal number of cache misses up to lower order terms, its bandwidth bound is optimal.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% START OF STUFF THAT SHOULD GET ADDED BACK IN maybe integrate some of this stuff back into the above
\section{Analysis of Grouped Partition}
In this section we give a more detailed description of our final algorithm, which we call the \defn{Grouped Partition Algorithm} and prove some key results regarding its performance and number of caches misses.

\paragraph{Algorithm Overview}
We logically divide the array $A = A[0],A[1],\ldots, A[n-1]$ into blocks $P_j$ each of $b$ adjacent elements, where $b$ is the \defn{block size}.
That is,
$$P_j = \{A[b\cdot j], A[b\cdot j + 1], \ldots, A[b\cdot (j+1)-1]\}.$$
Note that the $P_j$s are defined in a different way then in the Strided algorithm when $P_j$ contained elements spaced throughout the array.
This is equivalent to setting $|P_j| = 1$ for the $P_j$s in the Strided Algorithm and then making $P_j$ a collection of cache blocks.
Define \defn{$\pred(j)$} to be the number of predecessors in $P_j$.
There are $n/b$ blocks $P_j$.

The algorithm will form $g$ groups of these $P_j$s by the following procedure.
The algorithm generates a random array $X=X[0],X[1],\ldots,X[s-1]$ where each element in $X$ is an integer chosen randomly at uniform from $[0,g-1]$.
The values in $X$ determine groups $G_0, G_1, \ldots, G_{g-1}$ of $P_j$s.
The first group is
\begin{multline*}
	G_0 = \{X[0], X[1]+g, X[2]+2\cdot g,\\
	\ldots,X[s-1]+ (s-1)\cdot g\}.
\end{multline*}
This means that group $G_0$ is a collection of indices for specific parts $P_j$, indicating that these parts belong to group $G_0$. Similarly, group $G_y$ is defined as
\begin{multline*}
	G_y = \{(X[0]+y) \bmod g, (X[1]+y) \bmod g + g, \\
	\ldots, (X[s-1]+y) \bmod g + (s-1)\cdot g \}.
\end{multline*}
Intuitively this means that, for group $G_y$, on each chunk of size $g$ of the array, take $X[j]$ and add the group's index $i$ (wrapping around if there is overflow beyond the number of groups) to get to the index of the $P_j$ that belongs to group $G_y$ from this chunk of the array.
Note that we do not need to store the indices of each $P_j$ that belongs to a group because $X$ and the group index is enough information to determine which $P_j$s belong to a group.
We call our algorithm in-place because $s$ (recall $s = |X|$) is made $O(\polylog(n))$.

Define $U_y$ to be the union of all parts that belong to group $G_y$. That is,
$$U_y = \bigcup_{j\in G_y} P_j.$$
Define $\mu_y$ to be the number of predecessors in $U_y$ divided by the number of elements in $U_y$. This is analogous to the definition of $\mu$: the number of predecessors in $A$ divided by $n$.

Once the algorithm has generated $X$, it performs a serial partition on each collection $U_y$ in parallel.
The algorithm performs the partitions in parallel by splitting the tasks using a recursive method of calling the function repeatedly.
While performing the serial partitions of each $U_y$, the algorithm computes the indices $v_{max}, v_{min}$ where $v_{max}=\max{v_y}, v_{min}=\min v_y$ and $v_y$ is the index in $A$ of the first successor in $U_y$.
This is roughly outlined in Figure \ref{fig:recursiveParallelLoop}.
\begin{figure*}
	\caption{Recursive Spawning Implementation of Parallel For Loop}
	\label{fig:recursiveParallelLoop}
	\textbf{Input:} first group index, number of groups to process\\
	\textbf{Output: $v_{min}, v_{max}$ for $U_y,\ldots, U_{y+n-1}$}
	\begin{algorithmic}[1]
		\Procedure{partitionGroups}{$y, n$}
			\If{$n=1$}
				\State perform serial partition of $U_y$
				\State\Return{$v_y,v_y$}
			\Else
				\State leftvs $\gets$ \Call{partitionGroups}{$y,n/2$}
				\State rightvs $\gets$ \Call{partitionGroups}{$y+n/2,n/2+n\&2$}
				\State\Return{$\min(\text{leftvs.$v_{min}$}, \text{rightvs.$v_{min}$})$, $\max(\text{leftvs.$v_{max}$}, \text{rightvs.$v_{max}$})$}
			\EndIf
		\EndProcedure
	\end{algorithmic}
\end{figure*}

Then, because for all $y$
$$v_{min}\leq v_{y} \leq v_{max},$$
all elements of $A$ with index less than $v_{min}$ are predecessors and all elements of $A$ with index greater or equal to $v_{max}$ are successors.
Thus, by recursing on the subarray $A[v_{min}],\ldots,A[v_{max}-1]$, we complete the partitioning of the array.
We recursively apply the Grouped Partition algorithm to the subproblem.
The base case for the recursion is that when the algorithm can no longer make a substantial number of groups, in which case it partitions the input array in serial.

See Figure \ref{alg:parallelPartition} for a pseudocode implementation of the algorithm.

%%%%%%%% ALEK ALEK ALEK: show recursive spawning rather than parallel for loop?
\begin{figure*}
	\caption{Parallel Partition}
	\label{alg:parallelPartition}
	\begin{algorithmic}[1]
		\If{$g<2$}
			\State serialPartition A
		\Else
			\For{$i \in \{0,1,\ldots,s-1\}$}
				\State $X[i] \gets$ a random integer from $[0,g-1]$
			\EndFor
			\State -- We implement this parallel for-loop with a sequence of recursive spawns, and which facilitates computing $v_{min}$, $v_{max}$ without storing $v_y$s
			\ForAll{$ y \in \{0, 1,\ldots,g-1\}$ in parallel}
				\State -- Now we perform a serial partition on $U_y$

				\State -- Initialize ALowIdx to be the index of the first element in $U_y$
				\State $\text{ALowIdx} \gets ((X[0]+y) \bmod g)\cdot b$

				\State -- Initialize AHighIdx to be the index of the last element in $U_y$
				\State $\text{AHighIdx} \gets n - g\cdot b + ((X[s-1]+y) \bmod g)\cdot b + b-1$

				\While{ALowIdx $<$ AHighIdx}
					\While{A[ALowIdx] $\leq$ pivotValue}
						\State ALowIdx $\gets$ ALowIdx$+1$
						\If{ALowIdx on block boundary}
							\State -- We perform a block increment
							\State $i \gets $ \# of block increments so far (including this one)
							\State -- Increase ALowIdx to start of block $i$ of $G_y$
							\State $\text{ALowIdx} \gets ((X[i] + y)\bmod g) \cdot b + i\cdot b\cdot g$
						\EndIf
					\EndWhile
					\While{A[high] $>$ pivotValue}
						\State AHighIdx $\gets$ AHighIdx$-1$
						\If{AHighIdx on block boundary}
							\State -- We perform a block decrement
							\State $i \gets $ \# of block decrements so far (including this one)
							\State -- Decrease AHighIdx to end of block $s-1-i$ of $G_y$
							\State $\text{AHighIdx} \gets ((X[s-1-i] + y)\bmod g) \cdot b + i\cdot b\cdot g + b - 1$
						\EndIf
					\EndWhile

					\State Swap $A[\text{ALowIdx}]$ and $A[\text{AHighIdx}]$
				\EndWhile
			\EndFor
			\State Recurse on $A[v_{min}],\ldots,A[v_{max}-1]$
		\EndIf
	\end{algorithmic}
\end{figure*}

% Now we prove the following lemma to simultaneously bound $|\mu - \mu_y|$ for all $y$.
% \begin{lemma}
% Given $n, \mu$, For our choice of $b, \delta$, if $s=\Theta(\frac{\log n}{\delta^2})$ and $g = \frac{n}{b\cdot s}$ then the Grouped Partition Algorithm forms $g$ such that, with high probability in $n$, for all $y$, $|\mu - \mu_y| < \delta$.
% \label{lem:groupedAlgDelta}
% \end{lemma}

\begin{proof}[Proof of Lemma \ref{lem:groupedAlgDelta}]
$ $\\
First we use Hoeffding's inequality (i.e. Chernoff Bounds for random variables on $[0,1]$ rather than on $\{0,1\}$) to bound the probability that $|\mu - \mu_y| \geq \delta$.
We can express $\mu_y$ as the sum of the number of predecessors in all the blocks of the group $G_y$ divided by $|G_y|\cdot|P_j|$.
Note that $G_y$ is composed of $s$ blocks of $b$ elements each, so $|G_y|\cdot|P_j| = s\cdot b$.
Thus we can express $\mu_y$ as \\
\begin{equation}
	\begin{split}
	\mu_y = \frac{1}{|G_y|\cdot|P_j|}\sum_{j \in G_y} \pred(j) \\
	= \frac{1}{b \cdot s}\sum_{i=0}^{s-1}\pred(G_y[i]).
	\end{split}
	\label{eqn:emui}
\end{equation}
We are interested in calculating that the expected value of $\mu_y$, or equivalently calculating the average fraction of predecessors in group $G_y$ over all allowable assignments of blocks to the group.
This is,
$$\E(\mu_y) = \E\paren*{\frac{1}{s\cdot b} \sum_{i=0}^{s-1}\pred(G_y[i])}.$$
By linearity of expectation this is
$$\E(\mu_y) = \frac{1}{s\cdot b} \sum_{i=0}^{s-1}\E(\pred(G_y[i])).$$
Because $G_y[i]$ is an integer random variable which assumes an index uniformly at random from $[g \cdot i, g \cdot (i + 1) - 1]$ we can expand the expectation to:
$$\E(\mu_y) = \frac{1}{s\cdot b}\sum_{i=0}^{s-1}\sum_{k=0}^{g-1}\frac{1}{g}\pred(k+i\cdot g).$$
	But the argument to $\pred$ assumes all possible values from $[0, s\cdot g -1]$, so we can rewrite the sum as
$$\E(\mu_y) = \frac{1}{b\cdot s \cdot g}\sum_{j=0}^{s\cdot g-1}\pred(j) = \frac{n \cdot \mu}{b \cdot s \cdot g} = \mu.$$

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% END OF STUFF THAT MIGHT GET ADDED BACK IN

\section{Systems Performance}

\paragraph{Handling Non-divisible Array Size}
In the analysis above we have assumed that $n$ is divisible by our parameters $g, b, s$.
However, this simplifying assumption is incredibly restrictive and unnecessary.
In our algorithm $b$ and $s$ parameters, and then $g$ is determined as $$g = \floor* { \frac{n}{b\cdot s} }.$$
If $n \bmod (b\cdot s) \not\equiv 0$, then $g\cdot b \cdot s < n$.
To deal with this problem, we first apply our partial partition algorithm to ${A[0],\ldots,A[g\cdot s\cdot b -1]}$.
After doing this there is a subarray ${A[v_{min}],\ldots, A[v_{max}-1]}$ left to partition, and there is also the part of the array ${A[g\cdot s\cdot b],\ldots, A[n-1]}$ that we have ignored.
The algorithm deals with the tail of the array by either swapping the tail with a portion of the array of equal size which is the earliest successors in the array, or, in the rare case that there are more elements in the tail than successors--this may realistically happen on a very low layer of recursion when solving the partition problem with repeated applications of our algorithm, or if the pivot value is chosen adversarially such that there are very few successors--then the entire section of the array which is composed of successors is swapped with the last portion of the tail which is of the same length as the length of the section of the array composed of successors and appropriately incrementing $v_{max}$.
This step adds very little to the size of the recursive subproblem because $$n - b\cdot s \floor*{\frac{n}{b\cdot s}} < b\cdot s.$$
This does not affect the algorithm's work, span, or cache-efficiency.

\paragraph{Optimizations}
The Grouped Partition algorithm must perform a serial partition on each group $G_y$.
The serial partition is complicated by the fact that the groups do not contain a single contiguous portion of the array, but rather they contain $s$ blocks $P_j$ of size $b$ each.
In the serial partition the algorithm must repeatedly find the next or previous element in the group.
Although the algorithm could utilize modular arithmetic and floor division of a counter to find the index into the array $A$, division and modulo are very expensive operations to perform, so we use the better approach of checking proceeding within a block while checking if we have hit the boundary of the block, and then at that point jumping to the next block.
It is possible with this method to only use addition and subtraction and bit shifting by $\log b$, which is numerically equivalent to multiplication by $b$ but faster and possible because we choose $b$ to be a power of $2$.
This makes the determination of next and previous elements in the serial partition step much faster.
Our algorithm spends most of its time on the serial partition step at the top level of recursion.
This makes it so that the algorithm gets high parallelism because by increasing the number of groups that the algorithm can process in parallel the amount of time that the algorithm must spend on the time consuming step of serial partitioning is reduced.
The overhead introduced in the serial partition step of the algorithm slows it down initially, but is overcome when the number of processors increases.

\paragraph{Performance}
This new algorithm is very cache efficient, and operates very close to the memory bandwidth bound. Our implementation of the algorithm is available on github: https://github.com/awestover/Parallel-Partition.

\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%2345678901234567890123456789012345678901234567890123456789012345678901234567890
%        1         2         3         4         5         6         7         8

\documentclass[letterpaper, 10 pt, conference]{ieeeconf}  % Comment this line out if you need a4paper

%\documentclass[a4paper, 10pt, conference]{ieeeconf}      % Use this line for a4 paper

\IEEEoverridecommandlockouts                              % This command is only needed if
                                                          % you want to use the \thanks command

\overrideIEEEmargins                                      % Needed to meet printer requirements.

% See the \addtolength command later in the file to balance the column lengths
% on the last page of the document

% The following packages can be found on http:\\www.ctan.org
\usepackage{graphics} % for pdf, bitmapped graphics files
\usepackage{epsfig} % for postscript graphics files
%\usepackage{mathptmx} % assumes new font selection scheme installed
%\usepackage{times} % assumes new font selection scheme installed
\usepackage{amsmath} % assumes amsmath package installed
\usepackage{amssymb}  % assumes amsmath package installed
\usepackage{url}
\usepackage{subfigure}
% Rafael
\usepackage{xspace}
\usepackage{pgfplots}
\usepackage{graphicx}
\usepackage{epstopdf}
\usepackage{cite}

% axded by Rafael
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\def\marhes{{\sc Marhes}\xspace}
\def\octor{{OctoRoACH}\xspace}
\def\octors{{OctoRoACHes}\xspace}
\def\turtle{{TurtleBot}\xspace}
\def\turtles{{TurtleBots}\xspace}
\newcommand{\ie}{{\it i.e.},\xspace}
\newcommand{\eg}{{\it e.g.},\xspace}
\newcommand{\cf}{{\it c.f.},\xspace}
\newcommand{\el}{{\it et al.},\xspace}
\newcommand{\etc}{\text{etc.}}

\providecommand{\norm}[1]{\left\lVert#1\right\rVert}

%% Math defs
% The set of reals, integers, etc.
\renewcommand{\Re}{\mathbb{R}}
\newcommand{\Ze}{\mathbb {Z}}
\newcommand{\Pe}{\mathbb {P}}
\newcommand{\I}{\mathcal{I}}

\newtheorem{definition}{Definition}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\title{\LARGE \bf
Homework 1
}

\author{Luis A. Valbuena Reyes% <-this % stops a speeeeace
%\thanks{*This work was not supported by the MAST project.}% <-this % stops a space
\thanks{Luis Valbuena is with the Department of Electrical and Computer Engineering,
        University of New Mexico, Albuquerque, NM, 87131-0001, {\tt\small \{lavalbuenar@unm.edu}}%
}

\begin{document}

\newtheorem{theoremMyThesis}{Theorem}
\newtheorem{corollary}{Corollary}
\newtheorem{problemStatement}{Problem Statement}

\maketitle
\thispagestyle{empty}
\pagestyle{empty}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{abstract}

In this document, we present the solution for homework 1. We are required to use to function \emph{svmtrain} and \emph{svmpredict}
from the package LIBSVM. Trying to follow the guidelines for writing papers, we develop each the requirements of the assignment on Sections \ref{sec:theory}, \ref{sec:experiments}, and \ref{sec:conclusion}.
\end{abstract}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{INTRODUCTION}
\label{sec:Intro}

This assignment was about how the support vector machine technique behave under different scenarios.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Theory}
\label{sec:theory}

\subsection{Construction of a classifier with the model parameters}
\label{sec:TheoryConstructionClassifierModelParameters}

Taking care of conducting matrix operations with compatible dimensions\footnote{The dimensions of the quantities
in the code are not compatible with the matrix multiplications presented on some of the slides}, we construct the
simplest machine:

\begin{equation*}
    \hat{\mathbf{y}} = \mathbf{Xw},
\end{equation*}
where it is suggested to calculate $\mathbf{w}$ from the parameters $\alpha_{i}$ and the data, $\mathbf{X}$. That is:

\begin{equation*}
   \mathbf{w} = \mathbf{X}^{T} \mathbf{\alpha} \quad \text{with } \mathbf{\alpha} = (\mathbf{XX}^{T})^{-1}\mathbf{y},
\end{equation*}
where $y$ are the labels associated with the samples in $X$. The matrix $(XX^{T})^{-1}$ is problematic because it is ill
conditioned. Then, using the reduced single value decomposition $X = \mathbf{U}\mathbf{\Sigma} \mathbf{V}^{T}$:

\begin{equation*}
\begin{aligned}
    \alpha &= (\mathbf{XX}^{T})^{-1}\mathbf{y}, \\
                 &= (\mathbf{U} \mathbf{\Sigma} \mathbf{V}^{T}(\mathbf{U}\mathbf{\Sigma} \mathbf{V}^{T})^{T})^{-1}\mathbf{y},\\
                 &= (\mathbf{U} \mathbf{\Sigma} \mathbf{V}^{T} \mathbf{V} \mathbf{\Sigma}^{T} \mathbf{U}^{T})^{-1}\mathbf{y},\\
                 &= (\mathbf{U} \mathbf{\Sigma}^{2} \mathbf{U}^{T})^{-1}\mathbf{y},\\
                 &= \mathbf{U}\mathbf{\Sigma}^{-2} \mathbf{U}^{T} \mathbf{y}
\end{aligned}
\end{equation*}
where matrices $ \mathbf{U}$ and $ \mathbf{V}$ are unitary. The machine is then:
\begin{equation*}
    \hat{ \mathbf{y}} = \mathbf{ Xw}=  \mathbf{X X}^{T} \mathbf{U \Sigma}^{-2}  \mathbf{U}^{T}  \mathbf{y}.
\end{equation*}

Then, for a single sample $x_{i}$, we have $ \hat{ \mathbf{y}}_{i} = x_{i}\mathbf{w}$. The results are presented on Section  \ref{sec:ExperimentConstructionClassifierModelParameters}.

On the other hand, we have a tool called support vector machines, which finds a balance between complexity and regularization. On the minimization of the mean-squared error, we are increasing the complexity of the solution, which can lead to an over-fitting of the learner: the machine will be learning the noise of the data along. Then given a linear classifier of the form $f(\mathbf{X}) = \mathbf{w}^{T}\mathbf{X} + b$ we minimize the error measure and also the magnitude of $ \mathbf{w}$, which in turn minimizes the complexity. The algorithm is minimize

\begin{equation*}
 \| \mathbf{w} \| + C \sum_{i} \xi_{i}, \text{ subject to } y_{i}(\mathbf{w}^{T}x_{i} + b ) \geq 1 - \xi_{i}.
\end{equation*}

The region between the planes $y_{i}(\mathbf{w}^{T}x_{i} + b ) \geq 1 $ is called the margin and the only data that takes place in the support vector algorithm are
the samples that are inside the margin and the samples that are completely misclassified with respect to the classifier, see Fig.~\ref{fig:SupportVectorImage}

\begin{figure}[thpb]
      \centering
      \framebox{\includegraphics[scale=.25]{SolutionMaterial/SupportVectorImage.png}}
      \caption{Support Vector machine schematics. Taken from the class notes. }
      \label{fig:SupportVectorImage}
\end{figure}

\subsection{Graphical representation of an SVM}
\label{sec:TheoryGraphicalRepresentationSVM}

The function \emph{svmtrain} has the option of displaying the data with the separating line and the support vectors. To draw
the two margin lines, we created a meshgrid, evaluated the support vector machine given by @svmtrain@ with the margin
offset $\pm1$ and plot the contour where $f(X) = 0$. The results are presented on Section \ref{sec:ExperimentGraphicalRepresentationSVM}.

 \subsection{Estimating Structural Risk}
 \label{sec:TheoryEstimatingStructuralRisk}

 In \cite{Burges1998}, for a group of $l$ predictors, the empirical risk (error) is given as:

 \begin{equation}
  R_{emp}(\alpha) = \frac{1}{2l} \sum_{i=1}^{l} \mid y_{i} - f(\mathbf{x}_{i},\alpha) \mid,
  \label{eq:Error}
 \end{equation}

 where $\mathbf{x}_{i}$ and $y_{i}$ are the data from the training set. The instructor advised  to use the same expression using $\mathbf{x}_{i}$ and $y_{i}$ from
 the test set to calculate the actual risk. The structural risk is the difference between the actual risk and empirical risk.  The results are presented on Section  \ref{sec:ExperimentEstimatingStructuralRisk}.

\subsection{SVM for Regression}
\label{sec:TheorySVMRegression}

This experiment requires us to simulate the linear process
\begin{equation*}
\begin{aligned}
   y[n] &= 0.03y[n-1] - 0.01y[n-2] \\ &+ 3x[n] - 0.5x[n-1] + 0.2x[n-2],
\end{aligned}
\end{equation*}
where $x[n]$, $x[n-1]$ and $x[n-2]$ are the inputs. Preliminarily, it was assumed that $x[n]$, $x[n-1]$ and $x[n-2]$
were the attributes and $y[n]$ the label, but note that $y[n-1]$ and $y[n-2]$ also take part on  $y[n]$. Therefore, it
is required to precompute the kernel used by the support vector machine algorithm to specify  what are the labels
and the attributes. The input is $N(0,1)$ and the signal $y[n]$ is disturbed by additive noise of the form $e[n] ~ N(0, 0.01)$.
The idea of this theoretical example is that we only have $x[n]$ and  $y[n]+e[n]$ and we reconstruct $y[n]$. Let $o[n] = y[n]+e[n]$.
Based on the advice given by the instructor, the new attributes and labels are of the form:

\begin{equation*}
 Z_{i} = \begin{bmatrix} x[i-2] \\ x[i-1] \\ x[i]\\ o[i-2] \\ o[i-1] \end{bmatrix},\quad Y_{i} = o[i],
\end{equation*}
for $i \geq 2$. Then $\mathbf{Z} = [  Z_{2} \quad Z_{3} \quad \hdots \quad  Z_{N}]$ and $\mathbf{Y} = [  o_{2} \quad o_{3} \quad \hdots \quad  o_{N}]^{T}$.
The precomputed kernel has the form $\mathbf{K} = \gamma \mathbf{I} + \mathbf{Z}^{T}\mathbf{Z}$.

The results are presented on Section  \ref{sec:ExperimentSVMRegression}. The RMS error is calculated between the output $y[n]$ and the estimated $\hat{o}[n]$ signal given by the calculated support vector machine.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Experiments}
\label{sec:experiments}

This is the output of the command  @svmtrain@ given on the sample code, see Fig.~\ref{fig:NormalExecution}

%\begin{figure}[thpb]
%      \centering
%      \framebox{\includegraphics[scale=.35]{SolutionMaterial/Comparizon.eps}}
%      \cprotect  \caption{Output of  @svmtrain@. }
%      \label{fig:NormalExecutionsvmtrain}
%\end{figure}

\subsection{Construction of a classifier with the model parameters}
\label{sec:ExperimentConstructionClassifierModelParameters}

The comparizon requested can be appretiated on Fig.~\ref{fig:NormalExecution} and Fig.~\ref{fig:Comparizon}.

\begin{figure}[ht!]
 \begin{center}
  	\subfigure[\label{fig:NormalExecution}]{\includegraphics[width=0.42\textwidth]{SolutionMaterial/NormalExecution.eps}}
 	\subfigure[\label{fig:Comparizon}]{\includegraphics[width=0.42\textwidth]{SolutionMaterial/Comparizon.eps}}
         \caption{Output of \emph{svmtrain}. (a), Normal Execution. (b) The constructed machine.}
 \end{center}
\end{figure}

This time, the samples that the constructed machine classifies as $1$  are assigned a red circle marker, while the
others are assigned a blue square marker. The stepped black\footnote{This line is a contour line for which the classifier is zero.
The step feature of the line is due to the mesh grid used.} line on  Fig.~\ref{fig:Comparizon} is the black line on Fig.~\ref{fig:NormalExecution},
and the cyan line is a contour line of the plane perpendicular to $ \mathbf{w}$ as $x_{i}\mathbf{w}=0$. Note that these planes
are not parallel. Also note that for this particular experiment,  the sample in the region $1 \leq x \leq1.5$ and $0.4 \leq y \leq 0.6$
is misclassified by the constructed machine.

\subsection{Graphical representation of an SVM}
\label{sec:ExperimentGraphicalRepresentationSVM}

We chose $\sigma = \{0.05, 1, 1.4\}$, the plots of the separating line, the two
margin lines and the support vectors are presented in Fig.~\ref{fig:svm_sigma_5}, Fig.~\ref{fig:svm_sigma_100}, and
Fig.~\ref{fig:svm_sigma_140}.

\begin{figure}[ht!]
 \begin{center}
  	\subfigure[\label{fig:svm_sigma_5}]{\includegraphics[width=0.36\textwidth]{SolutionMaterial/svm_sigma_5.eps}}
 	\subfigure[\label{fig:svm_sigma_100}]{\includegraphics[width=0.36\textwidth]{SolutionMaterial/svm_sigma_100.eps}}
 	\subfigure[\label{fig:svm_sigma_140}]{\includegraphics[width=0.36\textwidth]{SolutionMaterial/svm_sigma_140.eps}}
         \caption{Output of \emph{svmtrain} with different values of $\sigma$. (a), $\sigma = 0.05$. (b), $\sigma = 1$. (c), $\sigma = 1.4$.}
 \end{center}
\end{figure}

For $\sigma = 0.05$ on Fig.~\ref{fig:svm_sigma_5}, the samples of each classification group are close together creating clusters, hence
there are no red markers on the upper left area or green markers on the lower right region, there are not even markers inside the
margin area. The transversal length of the margin area is  $l_{m} = 0.29$ units.

On  Fig.~\ref{fig:svm_sigma_100} where  $\sigma = 1$, we can see that the two group of data is spread wider and they share a common
area, which the support vector machine algorithm marks as the margin region. This time, there are some green markers on the lower right
region and some red markers on the upper left region. This time the transversal length of the margin area is $l_{m} = 1.16$ units.

Finally, for $\sigma = 1.4$ the data is mixed together and the support vector machine algorithm\footnote{The value $\sigma = 1.4$ was critical, for slightly bigger values, the algorithm could not converge}  requires more support vectors, see Fig.~\ref{fig:svm_sigma_140}. The transversal length of the margin area is $l_{m} = 2.64$ units.

\subsection{Estimating Structural Risk}
\label{sec:ExperimentEstimatingStructuralRisk}

The two requested plots are presented on Fig.~\ref{fig:RisksVSValueC}, and Fig.~\ref{fig:RiskVSNElements}. The purpose
of these two graphs was to corroborate that as the empirical risk reduces as the value of $C$ tends to infinity, the actual and structural risks
increase, while the actual risk bounds both the empirical and estructural risk.

\begin{figure}[ht!]
 \begin{center}
  	\subfigure[\label{fig:RisksVSValueC}]{\includegraphics[width=0.40\textwidth]{SolutionMaterial/RisksVSValueC2.eps}}
 	\subfigure[\label{fig:RiskVSNElements}]{\includegraphics[width=0.40\textwidth]{SolutionMaterial/RiskVSNElements.eps}}
         \caption{Evolution of empirical and actual risk with respect to (a), value $C$. (b), the number of predictors $N$.}
 \end{center}
\end{figure}

The file \verb|data.m| was used to generate both the training and test data. A key element to obtain the results described earlier on is to
compute several executions and then average them, that is, we conducted $1000$ realizations; additionally, the choice of the interval for the free parameter $C$ is instrumental, the interval of $C$ is $[10^{-2},10^{7}]$.

\subsection{SVM for Regression}
\label{sec:ExperimentSVMRegression}

\begin{figure}[ht!]
 \begin{center}
  	\subfigure[\label{fig:SimulatedProccess}]{\includegraphics[width=0.56\textwidth]{SolutionMaterial/SimulatedProccess.eps}}
 	\subfigure[\label{fig:rmserror}]{\includegraphics[width=0.56\textwidth]{SolutionMaterial/rms_error.eps}}
         \caption{SVM for regression (a), Simulated process: the blue represents the input $x[n]$, the red line is the evolution of the process $y[n]$, and
         the green line is $o[n]$. (b), RMS Error.}
 \end{center}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Conclusion}
\label{sec:conclusion}

\subsection{Construction of a classifier with the model parameters}
\label{sec:ConclusionsConstructionClassifierModelParameters}

The key differences are: the constructed machine is not conducting a regularization

\subsection{Graphical representation of an SVM}
\label{sec:ConclusionsGraphicalRepresentationSVM}

the support vector machine algorithm performs well for  Fig.~\ref{fig:svm_sigma_5}. it does not have problems
at classifying because the data is already clustered according to its label.

For  Fig.~\ref{fig:svm_sigma_100}, the support vector machine include more samples as support vectors and  on Fig.~\ref{fig:svm_sigma_140} even more.

As the variance is bigger, the support vector machine algorithm is pushed harder.

 \subsection{Estimating Structural Risk}
 \label{sec:ConclusionEstimatingStructuralRisk}

We want to know the actual risk of a trained machine because we want to know how susceptible our trained machine is under a new
subset of data. However, the definition given on \cite{Burges1998} requires knowledge of the probability function associated with the
data, which is not available. Hence, we need to estimate this quantity. We do that by generating a new data set and use it as our test set,
and then use Equation \ref{eq:Error} with the test set.

We think the reason behind of why the minimum actual risk does not coincide with the intersection of the other two errors has to do with
complexity. Note that on Figure.~\ref{fig:RisksVSValueC} the minimum of the actual risk is almost the same as the value of the empirical risk
for low values of $C$. This is because the trained machine has a low value $C$ so the machine has the same clearance in both the train and test
set. However, as $C$ increases, the complexity does too, so the trained machined is forced to map better the training data to the training labels,
but at the same time the machine is not going to perform as well with a new set of data as the test set. Then the actual error is going to increase.
The minimum of the actual risk is not going to coincide with the intersection of the other two errors because the manipulation of the value $C$ is
a trade off between the complexity and regularization: The minimum of the actual error is one side of the coin; the machine does not overfit but it
is too simple. The intersection happens in the middle of the logarithmic domain, where $C$ is bigger and you have made your machine more complex
so it can describe the data closer.

 For the second question, the procedure seems consistent, we have more data, but after $N \approx 70$ you don't gain more.

% \subsection{SVM for Regression}
%\label{sec:ConclusionSVMRegression}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%References are important to the reader; therefore, each citation must be complete and correct. If at all possible, references should be commonly available publications.

%\pagebreak

\bibliographystyle{IEEEtran}
\bibliography{References}

%%%%%% APPENDIX %%%%%%%
%q\input{Appendix}
\end{document}


\documentclass[a4paper,man,natbib]{apa6}
\usepackage[english]{babel}
\usepackage[utf8x]{inputenc}
\usepackage{setspace}

%% Images
% \usepackage{graphicx}
% 	\graphicspath{ {./images/} }
% 	\usepackage{caption}

% Minted
	\usepackage[cache=false]{minted}
	\usepackage{xcolor}
		\definecolor{light-gray}{gray}{0.97}
	\usemintedstyle{vs}

% Title
	\title{Intro to Computational Statistics: Homework 5}
	\shorttitle{HW04}
	\author{Author: Brandon Hosley}
	\affiliation{Instructor: Liang Kong, Ph.D}

%\abstract{}

\usepackage{pdfpages}

\begin{document}
\maketitle
\singlespacing

\emph{A researcher designed an experiment to study the effects of three factors (location, nitrogen, and water) to the yields of certain crop. For each one of the two locations (Trtc: c0 \& c1), he planted the crops in 48 plots. Then he applied one of the 12 combinations of treatments, four different rates of nitrogen application (Trta: a0, a1, a2, and a3) and three different level of watering (Trtb: b0, b1, and b2) to four randomly select plots. The results of yields are recorded in a MS Excel file (HW5.xlsx which is located at “Data Set” folder). In the file, the rate of nitrogen application (Trta) is stored in column A; the level of watering (Trtb) is stored in column B, the location (Trtc) is stored in column C, and the yields (Yeild1, Yield2, Yield3, \& Yield4) are stored I the column D, E, F, \& G.
\\ Write a SAS Program to answer the following questions.
}

\subsection*{Q1: Import MS Excel file HW5.xlsx and write it as a SAS data set. }
\inputminted[breaklines,bgcolor=light-gray]{sas}{./Snippets/DAT502-HW5-1.sas}
% \includepdf[pages=-]{./Snippets/SAS Output HW5-1.pdf}

\subsection*{Q2: Fit the data with a statistical model with all the main effects and interaction effects. }
\inputminted[breaklines,bgcolor=light-gray]{sas}{./Snippets/DAT502-HW5-2.sas}
% \includepdf[pages=-]{./Snippets/SAS Output HW5-2.pdf}

\subsection*{Q3: Fit the data with a statistical model with all significant ($\alpha$=0.05) main effects and interaction effects. }
\inputminted[breaklines,bgcolor=light-gray]{sas}{./Snippets/DAT502-HW5-3.sas}
% \includepdf[pages=-]{./Snippets/SAS Output HW5-3.pdf}

\subsection*{Q4: Based on the results of problem 3, using Scheffe multiple-comparison procedure with $\alpha$=0.01 to perform a multiple comparison of the average yields for different rates of nitrogen application. Which rate(s) has (have) the lowest average yields? }
\inputminted[breaklines,bgcolor=light-gray]{sas}{./Snippets/DAT502-HW5-4.sas}
% \includepdf[pages=-]{./Snippets/SAS Output HW5-4.pdf}

The Nitrogen levels corresponding to treatment a0 and a3 had the lower average yield.

\subsection*{Q5: Based on the results of problem 3, using Duncan multiple-range test with $\alpha$=0.1 to perform a multiple comparison of the average yields for different levels of watering. Which level(s) has (have) the highest average yield? }
\inputminted[breaklines,bgcolor=light-gray]{sas}{./Snippets/DAT502-HW5-5.sas}
% \includepdf[pages=-]{./Snippets/SAS Output HW5-5.pdf}

Watering level treatment b1 had the highest average yield.

\subsection*{Q6: Based on the results of problem 3, using Fisher’s LSD test with $\alpha$=0.025 to perform a multiple comparison of the average yields for different locations. Which location has a higher average yields? }
\inputminted[breaklines,bgcolor=light-gray]{sas}{./Snippets/DAT502-HW5-6.sas}
% \includepdf[pages=-]{./Snippets/SAS Output HW5-6.pdf}

Location c1 had the higher average yield.

\subsection*{Q7: Based on the results of problem 3, using Tukey studentized range test with $\alpha$=0.025 to perform a multiple comparison of the average yields for different rates of nitrogen application. Which rate(s) has (have) the highest average yields? }
\inputminted[breaklines,bgcolor=light-gray]{sas}{./Snippets/DAT502-HW5-7.sas}
% \includepdf[pages=-]{./Snippets/SAS Output HW5-7.pdf}

Nitrogen treatments a1 and a2 have the highest average yield and are not significantly different.

\subsection*{Q8: Based on the results of problem 3, using Performs Dunnett’s two-tailed t test with $\alpha$=0.025 to perform a comparison of the average yields of the control rate (a0) with other rates of nitrogen application. Which rates of nitrogen application has a significantly different average yields with the control rate? }
\inputminted[breaklines,bgcolor=light-gray]{sas}{./Snippets/DAT502-HW5-8.sas}
% \includepdf[pages=-]{./Snippets/SAS Output HW5-8.pdf}

All three of the other nitrogen treatment rates have a significant difference at the level of $\alpha$=0.025.

\subsection*{Attachments: First two pages of output are identical for all problems above and are omitted in a number of the attached documents for the sake of brevity.}

\includepdf[pages=-]{./Snippets/SAS Output HW5-2.pdf}
\includepdf[pages=-]{./Snippets/SAS Output HW5-3.pdf}
\includepdf[pages=3-]{./Snippets/SAS Output HW5-4.pdf}
\includepdf[pages=3-]{./Snippets/SAS Output HW5-5.pdf}
\includepdf[pages=3-]{./Snippets/SAS Output HW5-6.pdf}
\includepdf[pages=3-]{./Snippets/SAS Output HW5-7.pdf}
\includepdf[pages=3-]{./Snippets/SAS Output HW5-8.pdf}

\end{document}

2021-Spr Intro to Computational Stats/HW05/bhosl2-HW05.tex


\documentclass{article}
    % General document formatting
    \usepackage[margin=0.7in]{geometry}
    \usepackage[parfill]{parskip}
    \usepackage[utf8]{inputenc}
    \usepackage{amsmath}
    \usepackage{amssymb}
    \usepackage{tikz}
    \usepackage{fancyhdr}
    \usepackage{listings}
    \usepackage{multicol}
    \usepackage{polynom}

\pagestyle{fancy}
\fancyhf{}
\rhead{Edgar Jacob Rivera Rios - A01184125}

\begin{document}
\begin{titlepage}

    \newcommand{\HRule}{\rule{\linewidth}{0.5mm}} % Defines a new command for the horizontal lines, change thickness here

    \center % Center everything on the page

    %----------------------------------------------------------------------------------------
    %	HEADING SECTIONS
    %----------------------------------------------------------------------------------------

    \textsc{\LARGE Tecnológico de Monterrey}\\[1.5cm] % Name of your university/college
    \textsc{\Large Fundamentos de computación}\\[0.5cm] % Major heading such as course name
    %\textsc{\large Minor Heading}\\[0.5cm] % Minor heading such as course title

    %----------------------------------------------------------------------------------------
    %	TITLE SECTION
    %----------------------------------------------------------------------------------------

    \HRule \\[0.4cm]
    { \huge \bfseries Homework 8}\\[0.4cm] % Title of your document
    \HRule \\[1.5cm]

    %----------------------------------------------------------------------------------------
    %	AUTHOR SECTION
    %----------------------------------------------------------------------------------------

    \begin{minipage}{0.4\textwidth}
    \begin{flushleft} \large
    \emph{Student:}\\
    Jacob \textsc{Rivera} % Your name
    \end{flushleft}
    \end{minipage}
    ~
    \begin{minipage}{0.4\textwidth}
    \begin{flushright} \large
    \emph{Professor:} \\
    Dr. Hugo \textsc{Terashima} % Supervisor's Name
    \end{flushright}
    \end{minipage}\\[2cm]

    % If you don't want a supervisor, uncomment the two lines below and remove the section above
    %\Large \emph{Author:}\\
    %John \textsc{Smith}\\[3cm] % Your name

    %----------------------------------------------------------------------------------------
    %	DATE SECTION
    %----------------------------------------------------------------------------------------

    {\large \today}\\[2cm] % Date, change the \today to a set date if you want to be precise

    %----------------------------------------------------------------------------------------
    %	LOGO SECTION
    %----------------------------------------------------------------------------------------

    \includegraphics[width=0.4\textwidth,height=\textheight,keepaspectratio]{logo-tec-negro.png} % Include a department/university logo - this will require the graphicx package

    %----------------------------------------------------------------------------------------

    \vfill % Fill the rest of the page with whitespace

\end{titlepage}

\section{Problems}
Solve the following problems:
\begin{enumerate}
    \item Investigate the algorithm for computing the maximum flow on a graph. Provide and example and apply the algorithm, showing each step.

    \item Given a graph $G = (V, E, W)$ and a MST $T$, suppose that we decrease the weight of one of the edges not in $T$. Design an algorithm and its computational complexity to find the MST in the modified graph.

    The simplest way would be to add the edge to the MST, which will provoke for a cycle to be created. After that you just have to search for the biggest edge in that cycle and then remove it. As such, the complexity would be $O(n)$, because in the worst scenario, the cycle would include all the edges of the original MST

    \begin{figure}[ht]
        \centering
        \includegraphics{Hw8/P1.png}
        \caption{Graph}
        \label{graph}
    \end{figure}
    \item Obtain the MST for the graph in Figure 1 using both the Kruskal and Dijkstra/Prim algorithms.

    \textbf{Kruskal}

    Order the edges

    \begin{multicols}{2}
        \begin{center}
        \begin{tabular}{| c | c |}
            \hline
            edge & weight  \\ \hline
            ab & 2  \\
            ap & 3  \\
            bd & 3  \\
            bq & 4  \\
            de & 10 \\
            dp & 4  \\
            ef & 4  \\
            eq & 7  \\
            fi & 4  \\
            fj & 2  \\
            gq & 2  \\
            gi & 11 \\
            hj & 5  \\
            hi & 7  \\
            hs & 3  \\
            jk & 5  \\
            js & 4  \\
            kp & 8  \\
            pr & 10 \\
            rs & 9 \\ \hline
        \end{tabular}

        \columnbreak
        \begin{tabular}{| c | c |}
            \hline
            edge & weight  \\ \hline
            ab & 2  \\
            fj & 2  \\
            gq & 2  \\
            ap & 3  \\
            bd & 3  \\
            hs & 3  \\
            bq & 4  \\
            dp & 4  \\
            ef & 4  \\
            fi & 4  \\
            js & 4  \\
            hj & 5  \\
            jk & 5  \\
            eq & 7  \\
            hi & 7  \\
            kp & 8  \\
            rs & 9 \\
            de & 10 \\
            pr & 10 \\
            gi & 11 \\ \hline
        \end{tabular}
        \end{center}
    \end{multicols}

    Construct the MST by adding the unconnected graphs, from lowest to highest by cost.

    \begin{tabular}{| c | c | c |}
        \hline
        edge & weight & \\ \hline
        ab & 2 & added \\
        fj & 2 & added \\
        gq & 2 & added  \\
        ap & 3 & added \\
        bd & 3 & added \\
        hs & 3 & added \\
        bq & 4 & added \\
        dp & 4 & not added\\
        ef & 4 & added \\
        fi & 4 & added \\
        js & 4 & added \\
        hj & 5 & not added  \\
        jk & 5 & added \\
        eq & 7 & added \\
        hi & 7 & not added \\
        kp & 8 & not added\\
        rs & 9 & added \\
        de & 10 & not added \\
        pr & 10 & not added \\
        gi & 11 & not added \\ \hline
    \end{tabular}
    \begin{figure}[ht]
        \centering
        \includegraphics{Hw8/R1.png}
        \caption{MST by Kruskal}
        \label{graph2}
    \end{figure}

    \textbf{Prim–Dijkstra}\\
    Starting conditions:\\
    Start with $a$
    \begin{itemize}
        \item Tree = \{$a$\}
        \item Frontier = $\{b, p\}$
        \item Unvisited = $\{d, e, f, h, i, j, k, q, r, s \}$
        \item MST = $\{\}$
    \end{itemize}
    Final conditions:
    \begin{itemize}
        \item Tree = \{$a, b, d, e, f, h, i, j, k, p, q, r, s$\}
        \item Frontier = $\{\}$
        \item Unvisited = $\{\}$
        \item MST = $\{ab, ap, bd, bq, qg, qe, ef, fj, fi, js, sh, jk, sr\}$
    \end{itemize}

    \begin{figure}[ht]
        \centering
        \includegraphics{Hw8/R2.png}
        \caption{MST by Prim–Dijkstra}
        \label{graph3}
    \end{figure}

    \item Design a graph with at least 4 components (biconnected, each with three or more nodes) and run the algorithm seen in class to obtain them. Show the steps.
    \begin{figure}[ht]
        \centering
        \includegraphics{Hw8/G1.png}
        \caption{4 biconnected components graph}
        \label{graph4}
    \end{figure}

    \begin{itemize}
        \item Star with 1
        \item A $\rightarrow$ (1, 1)
        \item B $\rightarrow$ (2, 2)
        \item E $\rightarrow$ (3, 3)
        \item Find back edge to A
        \item E $\rightarrow$ (3, 1), B $\rightarrow$ (2, 1)
        \item C $\rightarrow$ (4, 4)
        \item D $\rightarrow$ (5, 5)
        \item Find back edge to E
        \item C $\rightarrow$ (4, 3), D $\rightarrow$ (5, 3)
        \item F $\rightarrow$ (6, 6)
        \item G $\rightarrow$ (7, 7)
        \item Find back edge to E
        \item F $\rightarrow$ (6, 3), G $\rightarrow$ (7, 3)
        \item H $\rightarrow$ (8, 8)
        \item I $\rightarrow$ (9, 9)
        \item Find back edge to E
        \item H $\rightarrow$ (8, 3), I $\rightarrow$ (9, 3)
    \end{itemize}

    \begin{figure}[ht]
        \centering
        \includegraphics{Hw8/G2.png}
        \caption{5 biconnected components graph colored}
        \label{graph5}
    \end{figure}

    \pagebreak
    \item Design a directed graph and establish an origin and a destination and apply the algorithm Dijkstra/Prim to obtain the shortest path between both nodes. Show the steps.

    \begin{figure}[ht]
        \centering
        \includegraphics{Hw8/G3.png}
        \caption{Graph for shortest path}
        \label{graph6}
    \end{figure}

    Get a shortest path from $A$ to $F$

    \begin{itemize}
        \item Open = $\{(A,0)\}$, Frontier=$\{(B,2)\}$, Closed=$\{(C,\infty),(D,\infty), (E,\infty), (F,\infty)\}$
        \item Open = $\{(A,0), (B,2)\}$, Frontier=$\{(C,7),(D,3)\}$, Closed=$\{(E,\infty),(F,\infty)\}$
        \item Open = $\{(A,0), (B,2), (D,3)\}$, Frontier=$\{(C,5),(F,10)\}$, Closed=$\{(E,\infty)\}$
        \item Open = $\{(A,0), (B,2), (D,3), (C,5)\}$, Frontier=$\{(E,9),(F,10)\}$, Closed=$\{\}$
        \item Open = $\{(A,0), (B,2), (D,3), (C,5), (E,9)\}$, Frontier=$\{(F,10)\}$, Closed=$\{\}$
        \item Open = $\{(A,0), (B,2), (D,3), (C,5), (E,9), (F,10)\}$, Frontier=$\{\}$, Closed=$\{\}$
    \end{itemize}
    Shortest path is $A \rightarrow B \rightarrow D \rightarrow F$, alternate same length route is $A \rightarrow B \rightarrow D \rightarrow C \rightarrow E \rightarrow F$
\end{enumerate}
\end{document}