\documentclass{article}
\usepackage[left=1in,right=1in,top=1in,bottom=1in]{geometry}
\usepackage{amsmath, amsthm, verbatim, enumerate, xcolor, mathpartir}

\input{macros}

\title{IIT CS595: Topics \& Applications in Programming Languages\\
  {\large Homework 2: Memory and Cost Semantics}}
\author{Dustin Van Tate Testa}
\date{Out: Tuesday, Oct. 19\\
  Due: Wednesday, Nov. 3 11:59pm CDT\\
  %\vspace{0.5cm}
  %\updated{Sep. 15}
}

\begin{document}
\maketitle

%\textbf{This assignment contains~{\numwtasks} written tasks and~{\numptasks}
%  programming tasks, for a total of~{\totpoints} points.}

\section*{Logistics}
The submission and collaboration instructions (as well as the preference
for submissions typeset in LaTeX) are the same as for HW0 and HW1.

\textbf{In particular:}
Submit your answers as a single .pdf or .doc file  on Blackboard
under
the correct assignment.

\section{Allocation and Memory Use}

As we discussed in class, cost semantics can be useful for more than just
counting the number of steps a program will take to execute.
%
In this assignment, we'll use them to track memory usage.
%
Consider this new variant of STLC:

\[
\begin{array}{r l l}
  \tau & \bnfdef & \kwunitt \bnfalt
  \kwarr{\tau}{\tau} \bnfalt
  \kwprod{\tau}{\tau}\\
  e & \bnfdef & x \bnfalt
  \kwunit \bnfalt
  \kwfun{x}{\tau}{e} \bnfalt
  \kwapp{e}{e} \bnfalt
  \kwpair{e}{e} \bnfalt
  \kwvpair{e}{e} \bnfalt
  \kwfst{e} \bnfalt
  \kwsnd{e}\\
\end{array}
\]

We now have two forms for pairs: $\kwpair{e_1}{e_2}$ and $\kwvpair{e_1}{e_2}$.
The latter form, $\kwvpair{e_1}{e_2}$, is used for a pair that has fully
evaluated and been allocated in memory (this models how structures like pairs
are executed in real languages; when they are constructed, memory for the
structure is allocated, e.g., using \texttt{malloc}).
%
This is made clearer in the dynamic semantics.
%
Like when we covered imperative languages, the state has a store~$\store$.
%
These stores are actually just lists of all of the pairs that have been
produced/allocated.
%
We'll use~$\stjoin{\store_1}{\store_2}$ to combine stores
and~$|\store|$ to refer to the size of a store (i.e., the length of the list).

The key different rules are (V-3) and (S-6).
%
Note from the (V-\#) rules that~$\kwpair{e_1}{e_2}$ is {\em not} a value,
even if~$e_1$ and~$e_2$ are values.
%
Instead, (S-6) says that when~$v_1$ and~$v_2$ are values,~$\kwpair{v_1}{v_2}$
takes an additional step to allocate the memory for the pair by adding it
to the store and steps to~$\kwvpair{v_1}{v_2}$, which is now a value%
\footnote{If we really wanted to faithfully model how this works in real
  languages, we would return a representation of the location in memory where
  the pair is stored (basically a pointer), and look it up in memory if we
  need to get the components of the pair later. The way we're doing it here
  is a little contrived but avoids a bunch of cumbersome notation I didn't
  want to burden you with.}.
%
For the same reason,~$\kwfst{e}$ and~$\kwsnd{e}$ evaluate~$e$ until
it looks like~$\kwvpair{v_1}{v_2}$: the pair must be allocated before we can
take its components and if we have a pair~$\kwvpair{v_1}{v_2}$, we know it's
been allocated.

Make sure you understand the language and the distinctions made above
before continuing.
%
If you have questions, feel free to ask for clarifications on Discord or
at office hours.

{
  \centering
  \def \MathparLineskip {\lineskip=0.43cm}
  \begin{mathpar}
    \Rule{V-1}
         {\strut}
         {\kwunit\val}
    \and
    \Rule{V-2}
         {\strut}
         {\kwfun{x}{\tau}{e}\val}
    \and
    \Rule{V-3}
         {v_1 \val\\
           v_2 \val}
         {\kwvpair{v_1}{v_2} \val}
    \and
    \Rule{S-1}
         {\cfg{\store}{e_1} \step \cfg{\store'}{e_1'}}
         {\cfg{\store}{\kwapp{e_1}{e_2}}
           \step
           \cfg{\store'}{\kwapp{e_1'}{e_2}}}
    \and
    \Rule{S-2}
         {\cfg{\store}{e_2} \step \cfg{\store'}{e_2'}}
         {\cfg{\store}{\kwapp{(\kwfun{x}{\tau}{e})}{e_2}}
           \step
           \cfg{\store'}{\kwapp{(\kwfun{x}{\tau}{e})}{e_2'}}}
    \and
    \Rule{S-3}
         {v\val}
         {\cfg{\store}{\kwapp{(\kwfun{x}{\tau}{e})}{v}}
           \step
           \cfg{\store}{\sub{v}{x}{e}}}
    \and
    \Rule{S-4}
         {\cfg{\store}{e_1} \step \cfg{\store'}{e_1'}}
         {\cfg{\store}{\kwpair{e_1}{e_2}}
           \step
           \cfg{\store'}{\kwpair{e_1'}{e_2}}}
    \and
    \Rule{S-5}
         {v_1\val\\
           \cfg{\store}{e_2} \step \cfg{\store'}{e_2'}}
         {\cfg{\store}{\kwpair{v_1}{e_2}}
           \step
           \cfg{\store'}{\kwpair{v_1}{e_2'}}}
    \and
    \Rule{S-6}
         {v_1\val\\
           v_2\val}
         {\cfg{\store}{\kwpair{v_1}{v_2}}
           \step
           \cfg{\stjoin{\store}{\stsing{\kwvpair{v_1}{v_2}}}}
               {\kwvpair{v_1}{v_2}}
         }
    \and
    \Rule{S-7}
         {\cfg{\store}{e} \step \cfg{\store'}{e'}}
         {\cfg{\store}{\kwfst{e}} \step \cfg{\store'}{\kwfst{e'}}}
    \and
    \Rule{S-8}
         {\strut}
         {\cfg{\store}{\kwfst{\kwvpair{v_1}{v_2}}}
           \step \cfg{\store}{v_1}}
    \and
    \Rule{S-9}
         {\cfg{\store}{e} \step \cfg{\store'}{e'}}
         {\cfg{\store}{\kwsnd{e}} \step \cfg{\store'}{\kwsnd{e'}}}
    \and
    \Rule{S-10}
         {\strut}
         {\cfg{\store}{\kwsnd{\kwvpair{v_1}{v_2}}}
         \step \cfg{\store}{v_2}}
\end{mathpar}
}

The cost semantics to track memory usage then should correspond not to the
number of steps a program will take to execute, but the number of pairs it
allocates, that is, the length of~$\store$ when it is done.
%
I've filed out the big-step semantics for the language for you.
%
Note that the big-step semantics {\em don't} carry around a $\store$; that's
too low-level a detail for the abstract big-step semantics.

{
  \centering
  \def \MathparLineskip {\lineskip=0.43cm}
  \begin{mathpar}
    \Rule{E-1}
         {v\val}
         {\eval[0]{v}{v}}
    \and
    \Rule{E-2}
         {\eval[n_1]{e_1}{\kwfun{x}{\tau}{e}}\\
           \eval[n_2]{e_2}{v}\\
           \eval[n_3]{\sub{v}{x}{e}}{v'}}
         {\eval[n_1 + n_2 + n_3]{\kwapp{e_1}{e_2}}{v'}}
    \and
    \Rule{E-3}
         {\eval[n_1]{e_1}{v_1}\\
           \eval[n_2]{e_2}{v_2}
         }
         {\eval[n_1 + n_2 + 1]{\kwpair{e_1}{e_2}}{\kwvpair{v_1}{v_2}}}
    \and
    \Rule{E-4}
         {\eval[n]{e}{\kwvpair{v_1}{v_2}}}
         {\eval[n]{\kwfst{e}}{v_1}}
    \and
    \Rule{E-5}
         {\eval[n]{e}{\kwvpair{v_1}{v_2}}}
         {\eval[n]{\kwsnd{e}}{v_2}}
  \end{mathpar}
}

\begin{task}
  Fill in the blanks in the big-step semantics to turn it into a cost semantics
  for tracking memory usage.
  %
  If you're having trouble deciding exactly what operations to ``charge'' for,
  look ahead at the theorems we want to prove about the cost semantics.
  %
  (Conversely, if you're having trouble proving the results later, you
  might want to come back and re-evaluate whether you need to change the
  cost semantics.)
\end{task}

We'll now prove that the cost semantics corresponds to the memory used by
the small-step semantics.
%
That is:
\[
\eval[n]{e}{v} \Leftrightarrow (\cfg{\estore}{e} \step^* \cfg{\store}{v}
\text{~and~} n = |\store|)
\]
where~$|\store|$ is the size of the store and~$\estore$ is the empty store.
%
As you'd expect, $|\estore| = 0$.

We prove this in several pieces.
%
Recall that the forward direction involves proving several annoying lemmas.
%
Here's the first annoying lemma.
%
It says that if~$e \step^* e'$ and $e$ is actually a part of a bigger
expression, that bigger expression also steps appropriately.
%
It has 6 parts for each kind of ``bigger expression.''

\begin{lemma}\label{lem:compose-exps}
  \begin{enumerate}
  \item
    If~$\cfg{\store}{e_1} \step^* \cfg{\store'}{e_1'}$
    then~$\cfg{\store}{\kwapp{e_1}{e_2}} \step^*
    \cfg{\store'}{\kwapp{e_1'}{e_2}}$.
  \item
    If~$\cfg{\store}{e_2} \step^* \cfg{\store'}{e_2'}$
    then~$\cfg{\store}{\kwapp{e_1}{e_2}} \step^*
    \cfg{\store'}{\kwapp{e_1}{e_2'}}$.
  \item
    If~$\cfg{\store}{e_1} \step^* \cfg{\store'}{e_1'}$
    then~$\cfg{\store}{\kwpair{e_1}{e_2}} \step^*
    \cfg{\store'}{\kwpair{e_1'}{e_2}}$.
  \item
    If~$\cfg{\store}{e_2} \step^* \cfg{\store'}{e_2'}$
    then~$\cfg{\store}{\kwpair{e_1}{e_2}} \step^*
    \cfg{\store'}{\kwpair{e_1}{e_2'}}$.
  \item
    If~$\cfg{\store}{e} \step^* \cfg{\store'}{e'}$
    then~$\cfg{\store}{\kwfst{e}} \step^*
    \cfg{\store'}{\kwfst{e}}$.
  \item
    If~$\cfg{\store}{e} \step^* \cfg{\store'}{e'}$
    then~$\cfg{\store}{\kwsnd{e}} \step^*
    \cfg{\store'}{\kwsnd{e}}$.
  \end{enumerate}
\end{lemma}

To do the proof, recall that the~$\step^*$ judgment is also defined
inductively:

{
  \centering
  \begin{mathpar}
    \Rule{M-1}
         {\strut}
         {\cfg{\store}{e} \step^* \cfg{\store}{e}}
    \and
    \Rule{M-2}
         {\cfg{\store}{e} \step \cfg{\store''}{e''}\\
           \cfg{\store''}{e''} \step^* \cfg{\store'}{e'}}
         {\cfg{\store}{e} \step^* \cfg{\store'}{e'}}
  \end{mathpar}
}

The parts are all reasonably similar.
I'll prove one, you'll prove another.

\begin{proof}
  \begin{enumerate}
  \item
    By induction on the derivation of
    $\cfg{\store}{e_1} \step^* \cfg{\store'}{e_1'}$.
    \begin{itemize}
    \item (M-1). Then~$e_1 = e_1'$ and~$\store = \store'$.
      By (M-1),
      $\cfg{\store}{\kwapp{e_1}{e_2}} \step^*
      \cfg{\store}{\kwapp{e_1}{e_2}}$.
    \item (M-2)
      Then~$\cfg{\store}{e_1} \step \cfg{\store''}{e_1''}$
      and $\cfg{\store''}{e_1''} \step^* \cfg{\store'}{e_1'}$.
      By (S-1),
      $\cfg{\store}{\kwapp{e_1}{e_2}} \step \cfg{\store''}{\kwapp{e_1''}{e_2}}$.
      By induction,
      $\cfg{\store''}{\kwapp{e_1''}{e_2}} \step^* \cfg{\store'}{\kwapp{e_1'}{e_2}}$
      so by (M-2),
      $\cfg{\store}{\kwapp{e_1}{e_2}} \step^* \cfg{\store'}{\kwapp{e_1'}{e_2}}$.
    \end{itemize}
  \end{enumerate}
\end{proof}

\begin{task}
  Prove part 5 of Lemma~\ref{lem:compose-exps}.
\end{task}

\begin{proof}
  By induction on the derivation of
    $\cfg{\store}{e} \step^* \cfg{\store'}{e'}$.
    \begin{itemize}
    \item (M-1). Then~$e = e'$ and~$\store = \store'$.
      By (M-1),
      $\cfg{\store}{\kwfst{e}} \step^*
      \cfg{\store}{\kwfst{e}}$.
    \item (M-2)
      Then~$\cfg{\store}{e} \step \cfg{\store''}{e''}$
      and $\cfg{\store''}{e''} \step^* \cfg{\store}{e'}$.\\
      By (S-7),
      $\cfg{\store}{\kwfst{e}} \step \cfg{\store''}{\kwfst{e''}}$.
      By induction,
      $\cfg{\store''}{\kwfst{e''}} \step^* \cfg{\store'}{\kwfst{e'}}$
      so by (M-2),
      $\cfg{\store}{\kwfst{e}} \step^* \cfg{\store'}{\kwfst{e'}}$
    \end{itemize}
\end{proof}

We also need to know that if we have two sequences of steps, we can ``glue''
them together.

\begin{lemma}\label{lem:compose-steps}
  If~$\cfg{\store}{e} \step^* \cfg{\store'}{e'}$
  and~$\cfg{\store'}{e'} \step^* \cfg{\store''}{e''}$
  then~$\cfg{\store}{e} \step^* \cfg{\store''}{e''}$.
\end{lemma}

\begin{task}
  Prove Lemma~\ref{lem:compose-steps}.
  You have two choices of what to induct on. Take an educated guess based on
  the rules you'll have to apply to conclude that
  $\cfg{\store}{e} \step^* \cfg{\store''}{e''}$.
  If the proof isn't working out, go back and try inducting on the other
  assumption.
\end{task}

\begin{proof}
  Your proof here.
  \begin{itemize}
    \item by induction on derivation of the step judgement, either
        \begin{itemize}
            \item $e = e' = e''$ and $\store = \store' = \store''$:\\
                in which case for both step judgements (M-1) would be applied

            \item $e' = e''$ and $\store' = \store''$:\\
                in which case (M-2) is applied to the first step and (M-1) applied after
                % \begin{mathpar}
                %     \inferrule*
                %     {\inferrule* {$ $}{\cfg{\store}{e} \step \cfg{\store'''}{e'''}}\\
                %       \inferrule* {$ $}{\cfg{\store'''}{e'''} \step^* \cfg{\store'}{e'}}}
                %     {\cfg{\store}{e} \step^* \cfg{\store'}{e'}}
                % \end{mathpar}
            \item $e = e'$ and $\store = \store'$:\\
                in which case (M-1) is applied to make the first step and then (M-2) after

            \item all $e$ and all $\store$ values are unique:\\
                in which case (M-2) is applied twice
        \end{itemize}
  \end{itemize}
\end{proof}

Now we can prove the forward direction of the theorem.
%
Note that we don't actually prove
that~$\cfg{\estore}{e} \step^* \cfg{\store}{v}$.
%
We actually need to be able to start with a non-empty store in order to
have a strong enough induction hypothesis.
%
Think about why.
%
We also prove that when~$e$ steps to~$v$ starting with store~$\store$,
the resulting store is strictly an extension of~$\store$: that is,
it's~$\stjoin{\store}{\store'}$, where~$\store'$ contains what we've added
to the store while stepping.
%
This means you don't have to worry about tracking things getting de-allocated
or changed (we don't have garbage collection.)

\begin{thm}\label{thm:forward}
  If $\eval[n]{e}{v}$ then for all~$\store$, it is the case that
  $\cfg{\store}{e} \step^* \cfg{\stjoin{\store}{\store'}}{v}$
  and $|\store'| = n$.
\end{thm}

I'll prove one case as an example.

\begin{proof}
  By induction on the derivation of $\eval[n]{e}{v}$.
  \begin{itemize}

  \item (E-2).
    Then~$e = \kwapp{e_1}{e_2}$
    and~$\eval[n_1]{e_1}{\kwfun{x}{\tau}{e}}$
    and~$\eval[n_2]{e_2}{v}$
    and~$\eval[n_3]{\sub{v}{x}{e}}{v'}$ and~$n = n_1 + n_2 + n_3$.
    By induction,
    \[
    \cfg{\store}{e_1} \step^*
    \cfg{\stjoin{\store}{\store_1}}{\kwfun{x}{\tau}{e}}
    \text{~and~}\cfg{\stjoin{\store}{\store_1}}{e_2} \step^*
    \cfg{\stjoin{\stjoin{\store}{\store_1}}{\store_2}}{v}
    \text{~and~}\cfg{\stjoin{\stjoin{\store}{\store_1}}{\store_2}}
    {\sub{v}{x}{e}} \step^*
    \cfg{\stjoin{\stjoin{\stjoin{\store}{\store_1}}{\store_2}}{\store_3}}
        {v'}
    \]
    and
    \[|\store_1| = n_1\text{~and~}|\store_2| = n_2\text{~and~}|\store_3| = n_3\]
    By Lemma~\ref{lem:compose-exps},
    $\cfg{\store}{e} \step^*
    \cfg{\stjoin{\store}{\store_1}}{\kwapp{(\kwfun{x}{\tau}{e})}{e_2}}$
    and~$\cfg{\stjoin{\store}{\store_1}}{\kwapp{(\kwfun{x}{\tau}{e})}{e_2}}
    \step^*
    \cfg{\stjoin{\stjoin{\store}{\store_1}}{\store_2}}
          {\kwapp{(\kwfun{x}{\tau}{e})}{v}}$.\\
          By Lemma~\ref{lem:compose-steps},
      $\cfg{\store}{e} \step^*
        \cfg{\stjoin{\stjoin{\store}{\store_1}}{\store_2}}
            {(\kwapp{\kwfun{x}{\tau}{e})}{v}}$.\\
      By (S-3) and Lemma~\ref{lem:compose-steps},
      $\cfg{\store}{e} \step^*
      \cfg{\stjoin{\stjoin{\stjoin{\store}{\store_1}}{\store_2}}{\store_3}}
          {v'}$.
          We have
      $|\stjoin{\store_1}{\stjoin{\store_2}{\store_3}}| =
      |\store_1| + |\store_2| + |\store_3| = n$.

\item (E-1). Then $e \val$ and $\eval[0]{e}{v}$ and $e = v$ and $n = 0$ which by lemma 1 translates to $\cfg{\estore}{e} \step^* \cfg{\store}{e}$. Because values cannot step according to the defined small-step semantics rules (exhaustion of rules (V- 1..3)), the $\step^*$ judgement must follow rule (M-1), thus $\estore = \store$ and $|\store| = n = 0$.

\item (E-3).
    Then~$e = \kwpair{e_1}{e_2}$
    and~$\eval[n_1]{e_1}{v_1}$
    and~$\eval[n_2]{e_2}{v_2}$
    and~$\eval[1]{\kwpair{v_1}{v_2}}{\kwvpair{v_1}{v_2}}$
    and~$n = n_1 + n_2 + 1$.
    By induction,
    \[
    \cfg{\store}{e_1}
    \step^* \cfg{\stjoin{\store}{\store_1}}{v_1}
        \text{~and~}
    \cfg{\store}{e_2}
    \step^* \cfg{\stjoin{\stjoin{\store}{\store_1}}{\store_2}}{v_2}
        \text{~and~}
    \cfg{\store}{e_2}
    \step^* \cfg{\stjoin{\stjoin{\stjoin{\store}{\store_1}}{\store_2}}{\store_3}}{v_2}
    \]
    and
    \[|\store_1| = n_1\text{~and~}|\store_2| = n_2\text{~and~}|\store_3| = 1\]
    By Lemma~\ref{lem:compose-exps},
    $
        \cfg{\store}{e}
        \step^* \cfg{\stjoin{\store}{\store_1}}{\kwpair{v_1}{e_1}}
    $and~$
        \cfg{\stjoin{\store}{\store_1}}{\kwpair{v_1}{e_1}}
        \step^* \cfg{\stjoin{\stjoin{\store}{\store_1}}{\store_2}}{\kwpair{v_1}{v_2}}
    $.\\
    By Lemma~\ref{lem:compose-steps},
      $\cfg{\store}{e}
        \step^*
         \cfg{\stjoin{\stjoin{\store}{\store_1}}{\store_2}}{\kwpair{v_1}{v_2}}$.\\
    By (S-6) and Lemma~\ref{lem:compose-steps},
      $\cfg{\store}{e} \step^*
      \cfg{\stjoin{\stjoin{\stjoin{\store}{\store_1}}{\store_2}}{\stsing{\kwvpair{v_1}{v_2}}}}
          {\kwvpair{v_1}{v_2}}$.
          We have
      $|\stjoin{\store_1}{\stjoin{\store_2}{\stsing{\kwvpair{v_1}{v_2}}}}| =
      |\store_1| + |\store_2| + |\stsing{\kwvpair{v_1}{v_2}}| = n$.

\item (E-4).
    Then~$e = \kwfst{e_1}$
    and~$\eval[n_1]{e_1}{\kwvpair{v_1}{v_2}}$
    and~$\eval[0]{e}{v_1}$
    and~$n = n_1 + 0$.
    By induction,
    \[
    \cfg{\store}{e_1} \step^* \cfg{\stjoin{\store}{\store_1}}{\kwvpair{v_1}{v_2}}
    \text{~and~}
    \cfg{\stjoin{\store}{\store_1}}{\kwfst{\kwvpair{v_1}{v_2}}}
        \step^*
        \cfg{\stjoin{\stjoin{\store}{\store_1}}{\store_2}}{v_1}
    \]
    and
    \[|\store_1| = |\store_2| = n_1 \text{~and~} n = n_1 \]
    By Lemma~\ref{lem:compose-exps},
    $
        \cfg{\store}{e} \step^* \cfg{\stjoin{\store}{\store_1}}{\kwfst{\kwvpair{v_1}{v_2}}}
    $~and~$
        \cfg{\stjoin{\store}{\store_1}}{\kwfst{\kwvpair{v_1}{v_2}}}
        \step^*
        \cfg{\stjoin{\stjoin{\store}{\store_1}}{\store_2}}{v_1}
    $.\\
    By Lemma~\ref{lem:compose-steps},
      $\cfg{\store}{e} \step^*
        \cfg{\stjoin{\stjoin{\store}{\store_1}}{\store_2}}{v_1}$.\\
    By (S-8) and Lemma~\ref{lem:compose-steps},
      $\cfg{\store}{e} \step^*
      \cfg{\stjoin{\stjoin{\store}{\store_1}}{\store_2}}{v_1}$.
          We have
      $|\stjoin{\store_1}{\store_2}| =
      |\store_1| = n$.

\item (E-5).
    Then~$e = \kwsnd{e_1}$
    and~$\eval[n_1]{e_1}{\kwvpair{v_1}{v_2}}$
    and~$\eval[0]{e}{v_1}$
    and~$n = n_1 + 0$.
    By induction,
    \[
    \cfg{\store}{e_1} \step^* \cfg{\stjoin{\store}{\store_1}}{\kwvpair{v_1}{v_2}}
    \text{~and~}
    \cfg{\stjoin{\store}{\store_1}}{\kwsnd{\kwvpair{v_1}{v_2}}}
        \step^*
        \cfg{\stjoin{\stjoin{\store}{\store_1}}{\store_2}}{v_2}
    \]
    and
    \[|\store_1| = |\store_2| = n_1 \text{~and~} n = n_1 \]
    By Lemma~\ref{lem:compose-exps},
    $
        \cfg{\store}{e} \step^* \cfg{\stjoin{\store}{\store_1}}{\kwsnd{\kwvpair{v_1}{v_2}}}
    $~and~$
        \cfg{\stjoin{\store}{\store_1}}{\kwsnd{\kwvpair{v_1}{v_2}}}
        \step^*
        \cfg{\stjoin{\stjoin{\store}{\store_1}}{\store_2}}{v_2}
    $.\\
    By Lemma~\ref{lem:compose-steps},
      $\cfg{\store}{e} \step^*
        \cfg{\stjoin{\stjoin{\store}{\store_1}}{\store_2}}{v_2}$.\\
    By (S-9) and Lemma~\ref{lem:compose-steps},
      $\cfg{\store}{e} \step^*
      \cfg{\stjoin{\stjoin{\store}{\store_1}}{\store_2}}{v_2}$.
          We have
      $|\stjoin{\store_1}{\store_2}| =
      |\store_1| = n$.

  \end{itemize}
\end{proof}

\begin{task}
  Prove the remaining cases of Theorem~\ref{thm:forward}.
  There are many similarities between the cases, but some small, important
  differences. Copying and pasting between cases is allowed and encouraged,
  but make sure you know what to change!

  \textbf{Note:} I actually proved the {\em second} case; you still have
  to go back and prove (E-1) as well as (E-3) through (E-5).
\end{task}

The reverse direction is pretty straightforward given a key lemma introduced
in class.
%
The lemma says that if~$\eval[n]{e'}{v}$ and~$e$ steps to~$e'$,
then~$\eval[n+k]{e}{v}$, where~$k$ counts any pairs allocated by that step.

\begin{lemma}\label{lem:closed}
  If~$\eval[n]{e'}{v}$
  and~$\cfg{\store}{e} \step \cfg{\stjoin{\store}{\store'}}{e'}$
  then~$\eval[n+k]{e}{v}$
  and~$|\store'| = k$.
\end{lemma}
  \begin{proof}
    By induction on the derivation of
    $\cfg{\store}{e} \step \cfg{\stjoin{\store}{\store'}}{e'}$.
    \begin{itemize}
    \item (S-1)
      Then~$e = \kwapp{e_1}{e_2}$ and~$e' = \kwapp{e_1'}{e_2}$.
      By inversion on (E-2),
      $\eval[n_1]{e_1'}{\kwfun{x}{\tau}{e}}$
      and~$\eval[n_2]{e_2}{v}$
      and~$\eval[n_3]{\sub{v}{x}{e}}{v'}$
      and~$n = n_1 + n_2 + n_3$.
      By induction,
      $\eval[n_1 + k]{e_1}{\kwfun{x}{\tau}{e}}$
      and~$|\store'| = k$.
      By (E-2),
      $\eval[n+k]{e}{v'}$.

    \item (S-2) Then~$e = \kwapp{\kwfun{x}{\tau}{e_1}}{e_2}$
        and~$e' = \kwapp{\kwfun{x}{\tau}{e_1}}{e_2'}$.
        By inversion on (E-2),
        $\eval[0]{\kwfun{x}{\tau}{e_1}}{\kwfun{x}{\tau}{e_1}}$
        and~$\eval[n_2]{e_2}{v'}$
        and~$\eval[n_3]{\sub{v'}{x}{e_1}}{v}$
        and~$n = 0 + n_2 + n_3$.
        By induction,
        $\eval[n' + k]{e_2}{v'}$
        and~$|\store'| = k$.
        By (E-2),
        $\eval[n+k]{e}{v}$.

    \item (S-3) Then~$e = \kwapp{\kwfun{x}{\tau}{e_1}}{v'}$
        and~$e' = \kwapp{\kwfun{x}{\tau}{e_1}}{e_2'}$
        and~$v' \val$.
        By inversion on (E-2),
        $\eval[0]{\kwfun{x}{\tau}{e_1}}{\kwfun{x}{\tau}{e_1}}$
        and~$\eval[0]{v'}{v'}$
        and~$\eval[n_1]{\sub{v'}{x}{e_1}}{v}$
        and~$n = 0 + 0 + n_1$.
        By induction,
        $\eval[n' + k]{\sub{v'}{x}{e_1}}{v}$
        $|\store'| = k$.
        By (E-2),
        $\eval[n+k]{e}{v}$.

    \item (S-4) Then~$e = \kwpair{e_1}{e_2}$
        and~$e' = \kwpair{e_1'}{e_2}$.
        By inversion on (E-3),
        $\eval[n_1]{e_1}{v_1}$
        and~$\eval[n_2]{e_2}{v_2}$
        and~$\eval[1]{\kwpair{v_1}{v_2}}{\kwvpair{v_1}{v_2}}$
        and~$n = n_1 + n_2 + 1$.
        By induction,
        $\eval[n_1 + k]{e_1}{v_1}$
        $|\store'| = k$.
        By (E-3),
        $\eval[n+k]{e}{v}$
    \item (S-5) Then~$e = \kwpair{v_1}{e_2}$
        and~$e' = \kwpair{v_1}{e_2'}$.
        By inversion on (E-3),
        $\eval[0]{v_1}{v_1}$
        and~$\eval[n_2]{e_2}{v_2}$
        and~$\eval[1]{\kwpair{v_1}{v_2}}{\kwvpair{v_1}{v_2}}$
        and~$n = 0 + n_2 + 1$.
        By induction,
        $\eval[n_2 + k]{e_2}{v_2}$
        $|\store'| = k$.
        By (E-3),
        $\eval[n+k]{e}{v}$
    \item (S-6) Then~$e = \kwpair{v_1}{v_2}$
        and~$e' = \kwpair{v_1}{v_2}$.
        By inversion on (E-3),
        $\eval[0]{v_1}{v_1}$
        and~$\eval[0]{v_2}{v_2}$
        and~$\eval[1]{\kwpair{v_1}{v_2}}{\kwvpair{v_1}{v_2}}$
        and~$n = 0 + 0 + 1$.
        By induction,
        $\eval[n_2 + k]{e}{\kwvpair{v_1}{v_2}}$
        $|\store'| = k = 1$.
        By (E-3),
        $\eval[n+k]{e}{v}$

    \item (S-7) Then~$e = \kwfst{e_1}$ and~$e' = \kwfst{e_1'}$.
        By inversion on (E-4),
        $\eval[n_1]{e_1}{\kwvpair{v_1}{v_2}}$
        and~$\eval[n_1]{\kwfst{e_1}}{v_1}$
        and~$n = n_1$.
        By induction,
        $\eval[n_1 + k]{e}{v_1}$
        $|\store'| = k$.
        By (E-4),
        $\eval[n+k]{e}{v}$

    \item (S-8) Then~$e = \kwfst{\kwvpair{v_1}{v_2}}$ and~$e' = v_1$.
        By inversion on (E-4),
        $\eval[0]{\kwvpair{v_1}{v_2}}{\kwvpair{v_1}{v_2}}$
        $\eval[0]{e'}{v_1}$, so $n = 0$.
        By (S-8), $\store' = \estore$, so $k = 0$.
        By (E-4),
        $\eval[n+k]{e}{v}$

    \item (S-9) Then~$e = \kwsnd{e_1}$ and~$e' = \kwsnd{e_1'}$.
        By inversion on (E-4),
        $\eval[n_1]{e_1}{\kwvpair{v_1}{v_2}}$
        and~$\eval[n_1]{\kwfst{e_1}}{v_2}$
        and~$n = n_1$.
        By induction,
        $\eval[n_1 + k]{e}{v_1}$
        $|\store'| = k$.
        By (E-5),
        $\eval[n+k]{e}{v}$

    \item (S-10) Then~$e = \kwfst{\kwvpair{v_1}{v_2}}$ and~$e' = v_1$.
        By inversion on (E-5),
        $\eval[0]{\kwvpair{v_1}{v_2}}{\kwvpair{v_1}{v_2}}$
        $\eval[0]{e'}{v_2}$, so $n = 0$.
        By (S-10), $\store' = \estore$, so $k = 0$.
        By (E-5),
        $\eval[n+k]{e}{v}$

    \end{itemize}
  \end{proof}

\begin{task}
  Prove the remaining cases of Lemma~\ref{lem:closed}.
  There are many similarities between the cases, but some small, important
  differences. Copying and pasting between cases is allowed and encouraged,
  but make sure you know what to change!
\end{task}

Given that lemma, the theorem is pretty straightforward to prove.
%
I'll just do it so you can be done with the homework.

\begin{thm}
  If $\cfg{\store}{e} \step^* \cfg{\stjoin{\store}{\store'}}{v}$ and~$v \val$,
  then~$\eval[n]{e}{v}$ and~$n = |\store'|$.
\end{thm}
\begin{proof}
  By induction on the derivation of $\cfg{\estore}{e} \step^* \cfg{\store}{v}$.
  If~$e = v,$ then $\store = \estore$ and by (E-1), $\eval[0]{v}{v}$.
  Otherwise,
  $\cfg{\estore}{e} \step \cfg{\stjoin{\estore}{\store_1}}{e'}
  \step^* \cfg{\stjoin{\estore}{\stjoin{\store_1}{\store_2}}}{v}$.
  By induction,~$\eval[n]{e'}{v}$
  and~$n = |\store_2|$.
  By Lemma~\ref{lem:closed}, $\eval[n + k]{e}{v}$
  where~$k = |\store_1|$.
  We have~$n + k = |\stjoin{\store_1}{\store_2}|$.
\end{proof}

\end{document}


\documentclass{llncs}

\usepackage[utf8]{inputenc}
\usepackage[english]{babel}

\usepackage{amsmath,amsfonts,amssymb}
\usepackage{array}
\usepackage{url}

\usepackage{tikz,pgfplots}

\tikzset{
  baseline=(current bounding box.center)
}

\newcommand{\doctype}{Parallel Radix Sort on OpenCL}

\usepackage{hyperref}
\hypersetup{
  breaklinks=true,
  colorlinks=true,
  citecolor=blue,
  linkcolor=blue,
  urlcolor=blue,
  bookmarksnumbered,
  bookmarksopen,
  pdftitle={\doctype},
  pdfauthor={Hennadiy Yatskov, Nico Mürdter},
  pdfsubject={},
  pdfkeywords={},
}

% Final Submit TODO: remove following line which makes margins smaller
\hypersetup{pdfpagescrop={92 62 523 748}}

\usepackage{breakurl}

\title{\doctype}
\author{Hennadiy Yatskov\\ Nico Mürdter}
\institute{
Karlsruhe Institute of Technology, Karlsruhe, Germany\\
\email{hennadiy.yatskov@student.kit.edu\\ nico.muerdter@student.kit.edu}}

\begin{document}

%% sp-process: load data in sql-table 'stats'
% IMPORT-DATA stats ../run.log

%% DEFMACRO REFORMAT(precision=0)
%% SELECT MAX(it) AS iterationcnt FROM stats
\def\iterationcnt{5}

\maketitle

\begin{abstract}
Parallel sorting algorithm can increase the performance of lots of applications tremendously. This draft shows a rather simple implementation of Radix-Sort on OpenCL where it can achieve a high speedup providing a good sorting method for integers of any kind.
\end{abstract}

% Final Submit TODO: remove this prior to FINAL submission
\pagestyle{plain}

\section{Algorithm}
\label{Algorithm}
This algorithm is based on an open source implementation of P. Helluy \cite{ocl-radix-helluy}. Before we come to the detailed explanation of the algorithm, we first need to define several values. $K(j)$, represents the non-sorted input values which consist only of integers. The implementation of P. Helluy is only capable of sorting 32-bit unsigned integers. So in his implementation, each of these integers in $K(j)$ for $j=0 \dots N-1$ is positive and will be called a \textit{key} in the further explanation. To expand the usage of this algorithm, this draft explains the expansion to support also 64-bit unsigned integers, and signed 32 and 64-bit integers. Because of the fact, that Radix-Sort is a non-comparative integer sorting algorithm, this implementation will not support floating point numbers. In order to allow operation with signed integers, an offset is added to the integers. This offset is in case of a signed input list, the absolute value of the minimum possible value of the given integer. In case of a 32-bit signed integer this would be $abs(-2^{31})$. The GPU kernels are still working on the unsigned representation of the given input data type, and the offset incrementation is done on the fly, when the key is first accessed. Also the decrementation at the end of the algorithm is done on the fly to prevent performance losses. So basically from the outside it seems the algorithm is working directly on the signed integer values, but on the inside it is working with a shifted number range to fit into the unsigned number range.

The Radix is represented by $R = 2^r$ where $r$ is the number of bits necessary representing $R$. We suppose that $b$, the total number of bits representing the keys is divisible by $r$ and so the number of passes is denoted by $p = b/r$. This assumption can be satisfied by an appropriate definition of $r$. Each pass $q=0 \dots p-1$ of the algorithm is sorting the list according to the $q^{th}$ digit (in base $R$) $K(j)_q$. A pass can be imagined as a reordering of the list by this $q^{th}$ Radix. So after $p = b/r$ passes the list is sorted totally. It is important to say, that each pass sorts the corresponding elements in a stable manner. So the work done in each pass is not corrupted by previous passes and will not corrupt following passes.

The final algorithm is seperated into three different phases, which are executed consecutively in every pass. These phases are called Histograming, Scanning and Reordering. The explanation is listed below.

%If the number of elements in $K$ can't satisfy this assertion, it will be extended with maximum integer values so that is satisfies it. This extension is done with maximum values, so that the sorting is only influenced in the first phase of the algorithm, which will be explained in the next sections.

\subsection{Histograming}
This first phase of the algorithm is in charge of calculating key \textit{histograms}. A histogram represents the number of occuring Radix in the given list of elements $K$. To do this fully parallel, the porcessing units are seperated in Groups $G$ with Items $I$, where an Item represents a Processor. So based on the GPU Architecture, the total amount of available Processing Units is $GI$. For the explanation we can suppose, that $N=GI$. If this is (most likely) not the case in a real scenario, the data is extended by adding keys which represent the biggest possible value. Each Item is in charge of a part of the list. It computes the Histogram of its own sublist and adds his histogram to the histogram of the Group. This is simply done by just inspecting every element in the list, computing every $q^{th}$ bit sequence with the length $r$ and then incrementing the corresponding entry in the local Histogram, which represents the computed Radix.

%TODO Geschickte Groupwahl etc noch einbringen.

\subsection{Scanning}
The next step in this algorithm is the so called Scanning. In this part the calculated Histograms are collected over a Prefixsum algorithm. P. Helluy \cite{ocl-radix-helluy} and also the here shown improved implementation is using the Blelloch algorithm for parallel scans \cite{blelloch1989scans}. After the Prefixsum, every Processor knows in his own Histogram, how many elements according to the current Radix are smaller. This information given by the Prefixsum, allows a reordering based on the calculated Histograms from the previous section.

\subsection{Reordering}
To finally reorder the elements, the Prefixsum of the Histograms is needed. As described in \cite{ocl-radix-helluy}: \textit{after the prefix sum, the histogram $H(c, g, i)$ contains the position, in the ordered list, of the first key with radix $c$ that is examined by the item $i$ in the group $g$.} So the Histogram can than be used to reorder the elements acording to the Radix of the corresponding pass. To do this, every Processor once again looks at the Radix of the current pass of every element, and then moves the element to the new position according to the Histogram. Afterwards the entry in the Historgam is increased by 1 for the next element with the same Radix. This reordering can also be done fully parallel.

\newpage
\section{Implementation Details}
In this chapter the focus lies on the implementation. The basecode is written for OpenCL 1.2 and the base features like compiling the OpenCL code is inspired by the code base of the GPGPU practicum at the KIT.

The main routine focuses on instantiating buffers and delegating the computational tasks to the GPU processors. The three main parts, shown in section \ref{Algorithm}, are represented by so called OpenCL kernels. For each of these described tasks one kernel is in charge. Because a GPU is a PRAM architecture, all processors are working on a common main memory. So the main difficulty here was to effectively seperate the memory each Processor is working on to avoid data corruption. To achieve this the kernels made use of the standard OpenCL routines to get to know what their operating ranges are on the main memory. This restriction also allows to use the OpenCL \textit{restrict} keyword, which gives the compiler a hint, that the marked buffers' adress spaces do not overlap. Therefore, the compiler is able to increase the overall performance by making use of cache optimizations.

To ensure the usage of different datatypes on the same codebase, this implementation is heavily based on the concept of C++ templates. The whole sorting algorithm can be instantiated with one of 4 supported data types: int32 / int64 / uint32 / uint64. This makes it easy to use this implementation for further projects without the need to accept performance losses if only 32-bit integers are of interest.

These type restrictions can easily be handled in C++ but for OpenCL we needed a more special method. Usually, the codebase of the OpenCL kernels is loaded during runtime and then compiled for the local GPU. Right before the compilation the code is imported with appended prepocessor \#define directives which set the internal data types to either 32 or 64 bit unsigned integer and the \textit{offset} shown in section \ref{Algorithm} to the appropriate value, which depends on whether the input data type is signed or unsigned.

Another notable implementation detail is the usage of local memory and barriers within the OpenCL code. The processors need to be aligned after several operations like copying the histograms to the local memory. To avoid data corruption these actions are controlled by memory barriers. The usage of the local memory of every processor leads once again to an overall significant performance increase as it allows for a considerable boost in access time and can be, broadly speaking, regarded as a cache.
% \subsection{Kernel}

\newpage
\section{Experimental Results}
All experimental benchmarks were made on a NVIDIA GeForce GTX 680 and a Intel i7-3770K CPU. GPU Driver Version 361.91 with OpenCL 1.2. For comparison the data was also sorted with std::sort and a custom single threaded Radix-Sort implementation on the CPU. The test data started at an input amount of $2^{13}$ elements and ended at $2^{25}$ elements. To test different scenarios every input consisted  of 5 different datasets:

\begin{enumerate}
  \item \textbf{Zeros}\\ Dataset containing only zeros
  \item[]
  \item \textbf{Random}\\ Dataset containing random generated integers by a mersenne twister
  \item[]
  \item \textbf{RandomDistributed}\\  Dataset containing random generated integers by a uniform distribution random generator
  \item[]
  \item \textbf{Range}\\ Dataset containing a sorted range of integers beginning by the minimum value of the used datatype
  \item[]
  \item \textbf{InvertedRange}\\ Dataset containing the integers from range in inverted order
\end{enumerate}

Below, there are the charts of benchmarks and speedup of the zero values and the uniform distributed random values. All these benchmarks were made with uint64 integers and were executed on the machine mentioned above. The experimental results are illustraded by these two input groups. As can be seen in figure \ref{fig:zero_bench} the runtime of the GPU algorithm on only zero values is much higher than the runtime of the standard C++ sorting algorithm \texttt{std::sort}. This behavior is explainable by the behaviour of \texttt{std::sort} and Radix-Sort itself. While Radix-Sort is always going through every bucket, the quicksort algorithm of \texttt{std::sort} can be optimized to recognize such inputs, consisting of identical elements, before starting the whole sorting process. Therefore the overhead of copying the input to the GPU and executing the whole algorithm on this pre-sorted dataset is inefficient. As expected the speedup of GPU Radix-Sort is below 1 which can be seen in figure \ref{fig:zero_speedup}. For other inputs like uniform distributed random integers, the runtime of GPU Radix-Sort increases significantly as shown in figure \ref{fig:uniform_bench} and reaches a speedup up to a factor of 25, which can be seen in figure \ref{fig:uniform_speedup}. The growing curve promises an even better performance for higher problem sizes above $2^{25}$ elements. All other inputs described above, behave the same way. For more details about the average execution times of the used algorithm parts figure \ref{fig:components} can be regarded.

\begin{figure}
  \begin{center}
    \includegraphics[width=\columnwidth]{charts/zero_benchmark.png}
  \end{center}
  \caption{Benchmark of only zero values.}\label{fig:zero_bench}
\end{figure}

\begin{figure}
  \begin{center}
    \includegraphics[width=\columnwidth]{charts/zero_speedup.png}
  \end{center}
  \caption{Speedup of only zero values compared to std::sort.}\label{fig:zero_speedup}
\end{figure}

\begin{figure}
  \begin{center}
    \includegraphics[width=\columnwidth]{charts/uniform_benchmark.png}
  \end{center}
  \caption{Benchmark of uniform distributed random values.}\label{fig:uniform_bench}
\end{figure}

\begin{figure}
  \begin{center}
    \includegraphics[width=\columnwidth]{charts/uniform_speedup.png}
  \end{center}
  \caption{Speedup of uniform distributed random values compared to std::sort.}\label{fig:uniform_speedup}
\end{figure}

\begin{figure}
  \begin{center}
    \includegraphics[width=\columnwidth]{charts/uniform_details.png}
  \end{center}
  \caption{Average runtimes of the components of the GPU Radix-Sort algorithm.}\label{fig:components}
\end{figure}

\section{Usage}
Below, there is a short list of command line options to run this implementation. All tests are automatically executed on all dataset configurations on all supported data types.

\begin{itemize}
  \item[] \texttt{--num-elements}\\ Number of elements to be sorted.
  \item[] \texttt{--perf-to-stdout}\\ Print all measured performances to stdout.
  \item[] \texttt{--perf-to-csv}\\ Write all measured performances to automatically generated csv files.
  \item[] \texttt{--perf-csv-to-stdout}\\ Print all measured performances to stdout in a csv format.
  \item[] \texttt{-v or --verbose}\\ Enable verbose prints for debugging.
\end{itemize}

\bibliographystyle{splncs03}
\bibliography{sources}

\end{document}


\section{Summary}
\label{sec:summary}
To summarize, we find, that a CNN can be trained and optimized to correctly classify flower images into one of the eleven species we chose with an accuracy of almost $\SI{80}{\percent}$.
Looking at which species get more often confused with one another, it can be said that most of the more common confusions are reasonable in a sense, that a human classification by hand
probably confuses these subsets of species with one another at times, as well.
Comparing the performance with a simpler approach, like using the kNN algorithm based only on the colors of an image, shows the use of CNNs for this problem to be appropriate
and it yields a considerable performance increase.