\commentout{
\begin{code}
  module HighAssurance where

  import Enigma
  import Classic
\end{code}
}

%#####################################################################
\chapter{High-assurance programming}
\label{cha:high-assur-progr}

Writing correct software is the holy grail of programming. Bugs
inevitably exist, however, even in thoroughly tested projects.  One
fundamental issue is the lack of support in typical programming
languages to let the programmer {\em state} what it means to be
correct, let alone formally establish any notion of correctness.  To
address this shortcoming, Cryptol advocates the high-assurance
programming approach: programmers explicitly state correctness
properties\indProperty along with their code, which are explicitly
checked by the Cryptol toolset.  Properties are not comments or mere
annotations, so there is no concern that they will become obsolete as
your code evolves.  The goal of this chapter is to introduce you to
these tools, and to the notion of high-assurance programming in
Cryptol via examples.

%=====================================================================
\section{Writing properties}
\label{sec:writingproperties}
\sectionWithAnswers{Writing properties}{sec:writingproperties}

Consider the equality:
$$
x^2 - y^2 = (x-y) * (x+y)
$$
Let us write two Cryptol functions that capture both sides of this
equation:\indTimes\indExponentiate\indMinus\indPlus
\begin{code}
  sqDiff1 (x, y) = x^^2  - y^^2
  sqDiff2 (x, y) = (x-y) * (x+y)
\end{code}
We would like to express the property that {\tt sqDiff1} and {\tt
  sqDiff2} are precisely the same functions: Given the same {\tt x}
and {\tt y}, they should return exactly the same answer. We can
express this property in Cryptol using a {\tt properties}
declaration:\indProperty
\begin{code}
  sqDiffsCorrect : ([8], [8]) -> Bit
  property sqDiffsCorrect (x, y) = sqDiff1 (x, y) == sqDiff2 (x, y)
\end{code}
The above declaration reads as follows: {\tt sqDiffsCorrect} is a
property stating that for all {\tt x} and {\tt y}, the expression {\tt
  sqDiff1 (x, y) == sqDiff2(x, y)} evaluates to {\tt
  True}. Furthermore, the type signature restricts the type of the
property to apply to only 8-bit values. As usual, the type-signature
is optional.\indSignature If not given, Cryptol will infer one for
you.

\begin{Exercise}\label{ex:thm:intro}
  Put the above property in a file and load it into Cryptol. Then
  issue the command:
\begin{Verbatim}
  Cryptol> :t sqDiffsCorrect
\end{Verbatim}
What do you see?\indCmdInfo
\end{Exercise}
\begin{Answer}\ansref{ex:thm:intro}
  Cryptol will print the property location, name, and the type. The
  command {\tt :i} stands for {\tt info}. It provides data about the
  properties, type-synonyms, etc. available at the top-level of your
  program.\indCmdInfo
\end{Answer}

\note{It is important to emphasize that the mathematical equality
  above and the Cryptol property are {\em not} stating precisely the
  same property.  Remember that all Cryptol arithmetic is
  modular,\indModular while the mathematical equation is over
  arbitrary numbers, including negative, real, or even complex
  numbers.  The takeaway of this discussion is that we are only using
  this example for illustration purposes: Cryptol properties relate to
  Cryptol programs, and should not be used for expressing mathematical
  theorems (unless, of course, you are stating group theory theorems
  or theorems in an appropriate algebra)!  In particular, {\tt
    sqDiffsCorrect} is a property about the Cryptol functions {\tt
    sqDiff1} and {\tt sqDiff2}, not about the mathematical equation
  that inspired it.}

\begin{Exercise}\label{ex:thm:0}
  Write a property {\tt revRev} stating that {\tt reverse} of a {\tt
    reverse} returns a sequence unchanged.\indReverse
\end{Exercise}
\begin{Answer}\ansref{ex:thm:0}\indReverse
\begin{code}
  property revRev xs = reverse (reverse xs) == xs
\end{code}
\end{Answer}

\begin{Exercise}\label{ex:thm:1}
  Write a property {\tt appAssoc} stating that append is an
  associative operator.\indAppend
\end{Exercise}
\begin{Answer}\ansref{ex:thm:1}\indAppend
\begin{code}
  property appAssoc (xs, ys, zs) = xs # (ys # zs) == (xs # ys) # zs
\end{code}
\end{Answer}

\begin{Exercise}\label{ex:thm:2}
  Write a property {\tt revApp} stating that appending two sequences
  and reversing the result is the same as reversing the sequences and
  appending them in the reverse order, as illustrated in the following
  expression:\indReverse\indAppend
\begin{Verbatim}
   reverse ("HELLO" # "WORLD") == reverse "WORLD" # reverse "HELLO"
\end{Verbatim}
\end{Exercise}
\begin{Answer}\ansref{ex:thm:2}\indReverse\indAppend
\begin{code}
  property revApp (xs, ys) = reverse (xs # ys)
                             == reverse ys # reverse xs
\end{code}
\end{Answer}

\begin{Exercise}\label{ex:thm:3}
  Write a property {\tt lshMul} stating that shifting left by $k$ is
  the same as multiplying by $2^k$.
\end{Exercise}
\begin{Answer}\ansref{ex:thm:3}
\begin{code}
  property lshMul (n, k) = n << k == n * 2^^k
\end{code}
\end{Answer}

\nb{A {\tt property} declaration simply introduces a property about
  your program, which may or may {\em not} actually hold. It is an
  assertion about your program, without any claim of correctness. In
  particular, you can clearly write properties that simply do not
  hold:}
\begin{code}
  property bogus x = x != x
\end{code}
It is important to distinguish between {\em stating} a property and
actually {\em proving} it. So far, our focus is purely on
specification. We will focus on actual proofs in
Section~\ref{sec:establishcorrectness}.

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
\subsection{Property-function correspondence}\indThmFuncCorr
\label{sec:prop-funct-corr}

In Cryptol, properties can be used just like ordinary definitions:
\begin{Verbatim}
  Cryptol> sqDiffsCorrect (3, 5)
  True
  Cryptol> :t sqDiffsCorrect
  sqDiffsCorrect : ([8],[8]) -> Bit
\end{Verbatim}
That is, a property over {\tt$(x, y)$} is the same as a function over
the tuple {\tt (x, y)}. We call this the property-function
correspondence. Property declarations, aside from the slightly
different syntax, are \emph{precisely} the same as Cryptol functions
whose return type is \texttt{Bit}. There is no separate language for
writing or working with properties. We simply use the full Cryptol
language write both the programs and the properties that they satisfy.

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
\subsection{Capturing test vectors}
\label{sec:thmvec}

One nice application of Cryptol properties is in capturing test
vectors:\indZero
\begin{code}
  property inctest = [ f x == y | (x, y) <- testVector ] == ~zero
    where  f x = x + 1
           testVector = [(3, 4), (4, 5), (12, 13), (78, 79)]
\end{code}
Notice that the property {\tt inctest} does not have any parameters
(no {\em forall} section), and thus acts as a simple {\tt Bit} value
that will be true precisely when the given test case succeeds.

\todo[inline]{Figure out how to re-run Cryptol in this chapter to make
  sure the tweak to the above example (removal of dummy single formal
  parameter) works.}

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
\subsection{Polymorphic properties}
\label{sec:polythm}

Just like functions, Cryptol properties can be polymorphic as well. If
you want to write a property for a polymorphic function, for instance,
your properties will naturally be polymorphic too. Here is a simple
trivial example:
\begin{code}
  property multShift x = x * 2 == x << 1
\end{code}
If we ask Cryptol the type of {\tt multShift}, we get:
\begin{Verbatim}
  Cryptol> :t multShift
  multShift : {a} (fin a, a >= 2) => [a] -> Bit
\end{Verbatim}
That is, it is a property about all words of size at least two. The
question is whether this property does indeed hold? In the particular
case of {\tt multShift} that is indeed the case, below are some
examples using the property-function correspondence:\indThmFuncCorr
\begin{Verbatim}
  Cryptol> multShift (5 : [8])
  True
  Cryptol> multShift (5 : [10])
  True
  Cryptol> multShift (5 : [16])
  True
\end{Verbatim}
However, this is {\em not} always the case for all polymorphic Cryptol
properties! The following example demonstrates:
\begin{code}
  property flipNeverIdentity x = x != ~x
\end{code}
The property {\tt flipNeverIdentity} states that
complementing\indComplement the bits of a value will always result in
a different value: a property we might expect to hold
intuitively. Here is the type of {\tt flipNeverIdentity}:
\begin{Verbatim}
  Cryptol> :t flipNeverIdentity
  flip : {a} (fin a) => a -> Bit
\end{Verbatim}
So, the only requirement on {\tt flipNeverIdentity} is that it
receives some finite type.\indFin Let us try some examples:
\begin{Verbatim}
  Cryptol> flipNeverIdentity True
  True
  Cryptol> flipNeverIdentity 3
  True
  Cryptol> flipNeverIdentity [1 2]
  True
\end{Verbatim}
However:
\begin{Verbatim}
  Cryptol> flipNeverIdentity (0 : [0])
  False
\end{Verbatim}
That is, when given a {\tt 0}-bit wide value, the complement will in
fact do nothing and return its argument unchanged! Therefore, the
property {\tt flipNeverIdentity} is not valid, since it holds at
certain monomorphic types, but not at all types.\indMonomorphism

\begin{Exercise}\label{ex:polythm:1}
  Demonstrate another monomorphic type where {\tt flipNeverIdentity}
  does {\em not} hold.
\end{Exercise}
\begin{Answer}\ansref{ex:polythm:1}
  There are many such types, all sharing the property that they do not
  take any space to represent. Here are a couple examples:\indZero
\begin{Verbatim}
  Cryptol> flipNeverIdentity (zero : ([0], [0]))
  False
  Cryptol> flipNeverIdentity (zero : [0][8])
  False
\end{Verbatim}
\end{Answer}

\nb{The moral of this discussion is that the notion of polymorphic
  validity\indThmPolyvalid (i.e., that a given polymorphic property
  will either hold at all of its monomorphic instances or none) does
  not hold in Cryptol. A polymorphic property can be valid at some,
  all, or no instances of it.}

\begin{Exercise}\label{ex:polythm:2}
  The previous exercise might lead you to think that it is the 0-bit
  word type ({\tt [0]}) that is at the root of the polymorphic
  validity issue.  This is not true. Consider the following
  example:\indWidth\indThmPolyvalid
\begin{code}
  property widthPoly x = (w == 15) || (w == 531)
     where w = width x
\end{code}
What is the type of {\tt widthPoly}? At what instances does it hold?
Write a property {\tt evenWidth} that holds only at even-width word
instances.
\end{Exercise}
\begin{Answer}\ansref{ex:polythm:2}
\begin{Verbatim}
  Cryptol> :t widthPoly
  widthPoly : {a, b} (fin a) => [a]b -> Bit
\end{Verbatim}
It is easy to see that {\tt widthPoly} holds at the instances:
\begin{Verbatim}
  {b} [15]b -> Bit
\end{Verbatim}
and
\begin{Verbatim}
  {b} [531]b -> Bit
\end{Verbatim}
but at no other. Based on this, we can write {\tt evenWidth} as
follows:\indWidth
\begin{code}
  property evenWidth x = (width x) ! 0 == False
\end{code}
remembering that the 0'th bit of an even number is always {\tt
  False}. We have:
\begin{Verbatim}
  Cryptol> evenWidth (0:[1])
  False
  Cryptol> evenWidth (0:[2])
  True
  Cryptol> evenWidth (0:[3])
  False
  Cryptol> evenWidth (0:[4])
  True
  Cryptol> evenWidth (0:[5])
  False
\end{Verbatim}
\end{Answer}

%=====================================================================
\section{Establishing correctness}
\label{sec:establishcorrectness}
\sectionWithAnswers{Establishing correctness}{sec:establishcorrectness}

Our focus so far has been using Cryptol to {\em state} properties of
our programs, without actually trying to prove them correct. This
separation of concerns is essential for a pragmatic development
approach.  Properties act as contracts that programmers state along
with their code, which can be separately checked by the
toolset~\cite{erkok-matthews-cryptolEqChecking-09}.  This approach
allows you to state the properties you want, and then work on your
code until the properties are indeed satisfied. Furthermore,
properties stay with your program forever, so they can be checked at a
later time to ensure changes (improvements/additions/optimizations
etc.) did not violate the stated properties.

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
\subsection{Formal proofs}
\label{sec:formal-proofs}

Recall our very first property, {\tt sqDiffsCorrect}, from
Section~\ref{sec:writingproperties}. We will now use Cryptol to
actually prove it automatically.  To prove {\tt sqDiffsCorrect}, use
the command {\tt :prove}:\indCmdProve
\begin{Verbatim}
  Cryptol> :prove sqDiffsCorrect
  Q.E.D.
\end{Verbatim}
Note that the above might take a while to complete, as a formal proof
is being produced behind the scenes.  Once Cryptol formally
establishes the property holds, it prints ``{\tt Q.E.D.}'' to tell the
user the proof is complete.\indQED\indProve

\nb{Cryptol uses off-the-shelf SAT\glosSAT and SMT\glosSMT solvers to
  perform these formal
  proofs~\cite{erkok-matthews-cryptolEqChecking-09}.  By default,
  Cryptol will use the CVC4 SMT solver~\cite{cvc4WWW} under the hood,
  but it can be configured to use other SAT/SMT solvers as well, such
  as SRI's Yices~\cite{YicesWWW}\indYices, or Microsoft Research's
  Z3~\cite{Z3WWW}\footnote{To do this, first install the package(s)
    from the URLs provided in the bibliography. Once a prover has been
    installed you can activate it with, for example, {\tt :set
      prover=z3}.}. Note that the {\tt :prove} command is a
  push-button tool: once the proof starts there is no user
  involvement. Of course, the external tool used may not be able to
  complete all the proofs in a feasible amount of time, naturally.

  \todo[inline]{Is this going to be a SAW-only capability, or Cryptol
    as well?  Ensure that a ticket exists for forward-porting this
    functionality.}

  % While it is out of the scope of this book, let us also mention that
  % Cryptol can also translate properties to Isabelle/HOL\indIsabelleHOL
  % for semi-automated theorem proving purposes~\cite{Isabelle-book},
  % where more complicated properties can be tackled with the human
  % guidance if necessary.
}

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
\subsection{Counterexamples}
\label{sec:counterexamples}

Of course, properties can very well be invalid, due to bugs in code or
the specifications themselves. In these cases, Cryptol will always
print a counterexample value demonstrating why the property does not
hold. Here is an example demonstrating what happens when things go
wrong:
\begin{code}
  failure : [8] -> Bit
  property failure x = x == x+1
\end{code}
We have:
\begin{Verbatim}
  Cryptol> :prove failure
  failure 0 = False
\end{Verbatim}
Cryptol tells us that the property is falsifiable, and then
demonstrates a particular value ({\tt 0} in this case) that it fails
at. These counterexamples are extremely valuable for debugging
purposes.\indCounterExample

If you try to prove an invalid property that encodes a test vector
(Section~\ref{sec:thmvec}), then you will get a mere indication that
you have a contradiction, since there is no universally quantified
variables to instantiate to show you a
counterexample.\indContradiction  If the expression evaluates to {\tt
  True}, then it will be a trivial proof, as expected:
\begin{Verbatim}
  Cryptol> :prove False
  False = False
  Cryptol> :prove True
  Q.E.D.
  Cryptol> :prove 2 == 3
  2==3 = False
  Cryptol> :prove reverse [1, 2] == [1, 2]
  reverse [1, 2] == [1,2] = False
  Cryptol> :prove 1+1 == 0
  Q.E.D.
\end{Verbatim}
The very last example demonstrates modular arithmetic in operation, as
usual.\indModular

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
\subsection{Dealing with polymorphism}
\label{sec:deal-with-polym}

As we mentioned before, Cryptol properties can be polymorphic. As we
explored before, we cannot directly prove polymorphic properties as
they may hold for certain monomorphic instances while not for others.
In this cases, we must tell Cryptol what particular monomorphic
instance we would like it to prove the property at.  Let us
demonstrate this with the {\tt multShift} property from
Section~\ref{sec:polythm}:
\begin{Verbatim}
  Cryptol> :prove multShift
    Not a monomorphic type:
    {a} (a >= 2, fin a) => [a] -> Bit
\end{Verbatim}
Cryptol is telling us that it cannot prove a polymorphic property
directly. We can, however, give a type annotation to monomorphise
it,\indTypeAnnotation and then prove it at a desired instance:
\begin{Verbatim}
  Cryptol> :prove multShift : [16] -> Bit
  Q.E.D.
\end{Verbatim}
In fact, you can use this very same technique to pass any bit-valued
function to the {\tt :prove} command:\indCmdProve\indWhere
\begin{Verbatim}
  Cryptol> :prove dbl where dbl x = (x:[8]) * 2 == x+x
  Q.E.D.
\end{Verbatim}
Of course, a \lamex (Section~\ref{sec:lamex}) would work just as well
too:\indLamExp
\begin{Verbatim}
  Cryptol> :prove \x -> (x:[8]) * 2 == x+x
  Q.E.D.
\end{Verbatim}

\begin{Exercise}\label{ex:prove:1}
  Prove the property {\tt revRev} you wrote in
  Exercise~\ref{sec:writingproperties}-\ref{ex:thm:0}. Try different
  monomorphic instantiations.
\end{Exercise}
\begin{Answer}\ansref{ex:prove:1}
  If we try to prove {\tt revRev} directly, we will get an error from
  Cryptol:
\begin{Verbatim}
  Cryptol> :prove revRev
  Not a valid predicate type:
  {a, b} (fin a, Cmp b) => [a]b -> Bit
\end{Verbatim}
Cryptol is telling us that the property has a polymorphic type, and
hence cannot be proven. We can easily prove instances of it, by either
creating new properties with fixed type signatures\indSignature, or by
monomorphising it via type annotations\indTypeAnnotation. Several
examples are given below:
\begin{Verbatim}
  Cryptol> :prove revRev : [10][8] -> Bit
  Q.E.D.
  Cryptol> :prove revRev : [100][32] -> Bit
  Q.E.D.
  Cryptol> :prove revRev : [0][4] -> Bit
  Q.E.D.
\end{Verbatim}
\end{Answer}

\begin{Exercise}\label{ex:prove:2}
  Prove the property {\tt appAssoc} you wrote in
  Exercise~\ref{sec:writingproperties}-\ref{ex:thm:1}, at several
  different monomorphic instances.
\end{Exercise}

\begin{Exercise}\label{ex:prove:3}
  Prove the property {\tt revApp} you wrote in
  Exercise~\ref{sec:writingproperties}-\ref{ex:thm:2}, at several
  different monomorphic instances.
\end{Exercise}

\begin{Exercise}\label{ex:prove:4}
  Prove the property {\tt lshMul} you wrote in
  Exercise~\ref{sec:writingproperties}-\ref{ex:thm:3}, at several
  different monomorphic instances.
\end{Exercise}

\begin{Exercise}\label{ex:prove:5}
  Use the {\tt :prove} command to prove and demonstrate
  counterexamples for the property {\tt widthPoly} defined in
  Exercise~\ref{sec:writingproperties}-\ref{ex:polythm:2}, using
  appropriate monomorphic instances.
\end{Exercise}
\begin{Answer}\ansref{ex:prove:5}
We have:
\begin{Verbatim}
  Cryptol> :prove widthPoly : [15] -> Bit
  Q.E.D.
  Cryptol> :prove widthPoly : [531] -> Bit
  Q.E.D.
  Cryptol> :prove widthPoly : [8] -> Bit
  widthPoly:[8] -> Bit 0 = False
  Cryptol> :prove widthPoly : [32] -> Bit
  widthPoly:[32] -> Bit 0 = False
\end{Verbatim}
\end{Answer}

%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
\subsection{Conditional proofs}
\label{sec:condproof}

It is often the case that we are interested in a property that only
holds under certain conditions.  For instance, in
Exercise~\ref{sec:arithmetic}-\ref{ex:arith:5:1} we have explored the
relationship between Cryptol's division, multiplication, and modulus
operators, where we asserted the following
property:\indMod\indDiv\indTimes
$$
  x = (x / y) \times y + (x\,\%\,y)
$$
Obviously, this relationship holds only when $y \not= 0$. The idea
behind a conditional Cryptol property\indThmCond is that we would like
to capture these side-conditions formally in our specifications.

We simply use an ordinary {\tt if-then-else} expression in Cryptol to
write conditional properties (at least until we add boolean logic
operators to Cryptol).  If the condition is invalid, we simply return
{\tt True}, indicating that we are not interested in that particular
case. Depending on how natural it is to express the side-condition or
its negation, you can use one of the following two patterns:
% the following looks ODDLY laid out in the source but comes out OK when latex'ed
\begin{Verbatim}[commandchars=\\\{\}]
   if  {\em side-condition-holds}         if   {\em side-condition-fails}
   then{\em property-expression}          then  True
   else True                         else {\em property-expression}
\end{Verbatim}

\begin{Exercise}\label{ex:cond:1}
  Express the relationship between division, multiplication, and
  modulus using a conditional Cryptol property.  Prove the property
  for various monomorphic instances.\indMod\indDiv\indTimes\indThmCond
\end{Exercise}
\begin{Answer}\ansref{ex:cond:1}
\begin{code}
  property divModMul (x,y) = if y == 0
                             then True   // precondition fails => True
                             else x == (x / y) * y + x % y
\end{code}
We have:
\begin{Verbatim}
  Cryptol> :prove divModMul : ([4], [4]) -> Bit
  Q.E.D.
  Cryptol> :prove divModMul : ([8], [8]) -> Bit
  Q.E.D.
\end{Verbatim}
\end{Answer}

%% TODO: Pedagogy here without exceptions
%% \nb{It is tempting to write a function:}
%% \begin{code}
%%  implies (a, b) = if a then b else True
%%\end{code}
%%to simplify writing conditional properties.
%% However, this approach is flawed as it fails to account correctly for
%% exceptions. We shall revisit this topic in detail in
%% Sections~\ref{sec:pitfall:evorder} and~\ref{sec:pitfall:thmexceptions}.

\paragraph*{Recognizing messages} Our work on classic ciphers
(Chapter~\ref{chapter:classic}) and the enigma
(Chapter~\ref{chapter:enigma}) involved working with messages that
contained the letters {\tt 'A'} .. {\tt 'Z'} only. When writing
properties about these ciphers it will be handy to have a recognizer
for such messages, as we explore in the next exercise.

\begin{Exercise}\label{ex:cond:2}
Write a function:
\begin{code}
  validMessage : {n} (fin n) => String n -> Bit
\end{code}
that returns {\tt True} exactly when the input only consists of the
letters {\tt 'A'} through {\tt 'Z'}. \lhint{Use the functions {\tt
    all} defined in Exercise~\ref{sec:zero}-\ref{ex:zero:1},\indAll
  and {\tt elem} defined in
  Exercise~\ref{sec:recandrec}-\ref{ex:recfun:4:1}.\indElem}
\end{Exercise}

\begin{Answer}\ansref{ex:cond:2}
  Using {\tt all}\indAll and {\tt elem}\indElem, it is easy to express
  {\tt validMessage}:
\begin{code}
  validMessage = all (\c -> elem (c, ['A' .. 'Z']))
\end{code}
Note the use of a \lamex to pass the function to {\tt all}. Of course,
we could have defined a separate function for it in a {\tt
  where}-clause\indWhere, but the function is short enough to directly
write it inline.
\end{Answer}

\todo[inline]{Should we revisit directly some past examples or
  solutions in the appendix and show what kinds of properties are
  appropriate to write and which ones can be verified?}

\begin{Exercise}\label{ex:cond:3}
  Recall the pair of functions {\tt caesar} and {\tt dCaesar} from
  Section~\ref{sec:caesar}.\indCaesarscipher Write a property, named
  {\tt caesarCorrect}, stating that {\tt caesar} and {\tt dCaesar} are
  inverses of each other for all {\tt d} (shift amount) and {\tt msg}
  (message)'s. Is your property valid? What extra condition do you
  need to assert on {\tt msg} for your property to hold? Prove the
  property for all messages of length 10.
\end{Exercise}
\begin{Answer}\ansref{ex:cond:3}
  A naive attempt would be to write:
\begin{code}
  property caesarCorrectBOGUS (d,msg) =
           dCaesar(d, caesar(d, msg)) == msg
\end{code}
However, this property is not correct for all {\tt msg}'s, since
Caesar's cipher only works for messages containing the letters {\tt
  'A'} \ldots {\tt 'Z'}, not arbitrary 8-bit values as the above
property suggests. We can see this easily by providing a bad input:
\begin{Verbatim}
  Cryptol> caesar (3, "1")
  invalid sequence index: 240
\end{Verbatim}
(240 is the difference between the ASCII value of {\tt '1'}, 49, and
the letter {\tt 'A'}, 65, interpreted as an 8-bit offset.) We should
use the {\tt validMessage} function of the previous exercise to write
a conditional property instead:
\begin{code}
  property caesarCorrect (d,msg) = if validMessage msg
                                   then dCaesar(d, caesar(d, msg)) == msg
                                   else True
\end{code}
We have:
\begin{Verbatim}
  Cryptol> :prove caesarCorrect : ([8], String(10)) -> Bit
  Q.E.D.
\end{Verbatim}
\end{Answer}

\begin{Exercise}\label{ex:cond:4}
  Write and prove a property for the {\tt modelEnigma} machine
  (Page~\pageref{def:modelEnigma}), relating the {\tt enigma} and {\tt
    dEnigma} functions from Section~\ref{enigma:encdec}.
\end{Exercise}
\begin{Answer}\ansref{ex:cond:4}
\begin{code}
  property modelEnigmaCorrect pt =
       if validMessage pt
       then dEnigma (modelEnigma, enigma (modelEnigma, pt)) == pt
       else True
\end{code}
We have:
\begin{Verbatim}
  Cryptol> :prove modelEnigmaCorrect : String(10) -> Bit
  Q.E.D.
\end{Verbatim}
\end{Answer}
This may take a long time to prove, depending on the speed of your
machine, and the prover you choose.

%=====================================================================
\section{Automated random testing}
\label{sec:quickcheck}
\sectionWithAnswers{Automated random testing}{sec:quickcheck}

Cryptol's {\tt :prove} command\indCmdProve constructs rigorous formal
proofs using push-button tools.\footnote{While some of the solvers
  that Cryptol uses are capable of \emph{emitting} proofs, such
  functionality is not exposes as a Cryptol feature.}  The underlying
technique used by Cryptol (SAT-\glosSAT and SMT-based\glosSMT
equivalence checking) is complete\indProofCompleteness, i.e., it will
always either prove the property or find a
counterexample.\indCounterExample In the worst case, however, the
proof process might take infeasible amounts of resources, potentially
running out of memory or taking longer than the amount of time you are
willing to wait.

What is needed for daily development tasks is a mechanism to gain some
confidence on the correctness of the properties without paying the
price of formally proving them.  This is the goal of Cryptol's {\tt
  :check}\indCmdCheck command, inspired by Haskell's quick-check
library~\cite{quickcheck}.  Instead of trying to formally prove your
property, {\tt :check} tests it at random values to give you quick
feedback.  This approach is very suitable for rapid development.  By
using automated testing you get frequent and quick updates from
Cryptol regarding the status of your properties, as you work through
your code. If you introduce a bug, it is likely (although not
guaranteed) that the {\tt :check} command will alert you right
away. Once you are satisfied with your code, you can use the {\tt
  :prove} command to conduct the formal proofs, potentially leaving
them running overnight.

The syntax of the {\tt :check} command is precisely the same as the
{\tt :prove} command. By default, it will run your property over 100
randomly generated test cases.

\begin{Exercise}\label{ex:quick:0}
  Use the {\tt :check} command to test the property {\tt
    caesarCorrect} you have defined in
  Exercise~\ref{sec:condproof}-\ref{ex:cond:2}, for messages of length
  10.  Use the command {\tt :set tests=1000}\indQCCount to change the
  number of test cases to 1,000.  Observe the test coverage statistics
  reported by Cryptol. How is the total number of cases computed?
\end{Exercise}
\begin{Answer}\ansref{ex:quick:0}
  Here is the interaction with Cryptol (when you actually run this,
  you will see the test cases counting up as they are
  performed):\indQCCount
\begin{Verbatim}
  Cryptol> :check (caesarCorrect : ([8], String 10) -> Bit)
  Using random testing.
  passed 100 tests.
  Coverage: 0.00% (100 of 2^^88 values)
  Cryptol> :set tests=1000
  Cryptol> :check (caesarCorrect : ([8], String 10) -> Bit)
  Using random testing.
  passed 1000 tests.
  Coverage: 0.00% (1000 of 2^^88 values)
\end{Verbatim}
In each case, Cryptol tells us that it checked a minuscule portion of
all possible test cases: A good reminder of what {\tt :check} is
really doing.  The number of test cases is: $2^{8+8\times10} =
2^{88}$. We have 8-bits for the {\tt d} value, and $10*8$ bits total
for the 10 characters in {\tt msg}, giving us a total of 88
bits. Since the input is 88 bits wide, we have $2^{88}$ potential test
cases. Note how the number of test cases increase exponentially with
the size of the message.
\end{Answer}

\begin{Exercise}\label{ex:quick:1}
  If the property is {\em small} in size, {\tt :check} might as well
  prove/disprove it. Try the following commands:
\begin{Verbatim}
  :check True
  :check False
  :check \x -> x==(x:[8])
\end{Verbatim}
\end{Exercise}
\begin{Answer}\ansref{ex:quick:1}
\begin{Verbatim}
  Cryptol> :check True
  Using exhaustive testing.
  passed 1 tests.
  QED
  Cryptol> :check False
  Using exhaustive testing.
  FAILED for the following inputs:
  Cryptol> :check \x -> x == (x:[8])
  Using exhaustive testing.
  passed 256 tests.
  QED
\end{Verbatim}
Note that when Cryptol is able to exhaust all possible inputs, it returns QED, since the property is effectively proven.
\end{Answer}

\begin{Exercise}\label{ex:quick:2}
  Write a bogus property that will be very easy to disprove using {\tt
    :prove}, while {\tt :check} will have a hard time obtaining the
  counterexample.  The moral of this exercise is that you should try
  {\tt :prove} early in your development and not get too comfortable
  with the results of {\tt :check}!\indCmdProve\indCmdCheck
\end{Exercise}
\begin{Answer}\ansref{ex:quick:2}
\begin{code}
  property easyBug x = x != (76123:[64])
\end{code}
The {\tt :prove} command will find the counterexample almost
instantaneously, while {\tt :check} will have a hard time!
\end{Answer}

\paragraph*{Bulk operations} If you use {\tt :check}\indCmdCheck and
{\tt :prove}\indCmdProve commands without any arguments, Cryptol will
check and prove all the properties defined in your program. This is a
simple means of exercising all your properties automatically.

\todo[inline]{Do we want to reintroduce the \texttt{-n} switch, per
  \ticket{276}?}
%% \paragraph*{Automatic invocation} You will notice that Cryptol will
%% automatically {\tt :check} the properties in your file when you
%% load the file from the command line, giving you a quick status
%% report. You can turn this behavior off by passing Cryptol the flag
%% {\tt -n}, like this: {\tt cryptol -n myfile.cry}. You can also
%% interrupt a running {\tt :prove}\indCmdProve or {\tt
%% :check}\indCmdCheck command by pressing {\tt
%% Ctrl-C}.\indCmdAutoCheck

%=====================================================================
\section{Checking satisfiability}
\label{sec:sat}
\sectionWithAnswers{Checking satisfiability}{sec:sat}

Closely related to proving properties is the notion of checking
satisfiability. In satisfiability checking,\indCmdSat we would like to find
arguments to a bit-valued function such that it will evaluate to {\tt True},
i.e., it will be satisfied.\indSat

One way to think about satisfiability checking is {\em intelligently}
searching for a solution. Here is a simple example. Let us assume we
would like to compute the modular square-root of 9 as an 8-bit
value. The obvious solution is {\tt 3}, of course, but we are
wondering if there are other solutions to the equation $x^2 \equiv 9
\imod{2^8}$. To get started, let us first define a function that will
return {\tt True} if its argument is a square-root of 9:
\begin{code}
  isSqrtOf9 : [8] -> Bit
  isSqrtOf9 x = x*x == 9
\end{code}
Any square-root of 9 will make the function {\tt isSqrtOf9} return
{\tt True}, i.e., it will {\em satisfy} it.  Thus, we can use
Cryptol's satisfiability checker to find those values of {\tt x}
automatically:
\begin{Verbatim}
  Cryptol> :sat isSqrtOf9
  isSqrtOf9 3 = True
\end{Verbatim}
Not surprisingly, Cryptol told us that 3 is one such value. We can
search for other solutions by explicitly disallowing 3:
\begin{Verbatim}
  Cryptol> :sat \x -> isSqrtOf9 x && ~(elem (x, [3]))
  \x -> isSqrtOf9 x && ~(elem (x, [3])) 131 = True
\end{Verbatim}
Note the use of the \lamex to\indLamExp indicate the new
constraint. (Of course, we could have defined another function {\tt
  isSqrtOf9ButNot3} for the same effect, but the \lamex is really
handy in this case.) We have used the function {\tt elem} you have
defined in Exercise~\ref{sec:recandrec}-\ref{ex:recfun:4:1}\indElem to
express the constraint {\tt x} must not be 3. In response, Cryptol
told us that {\tt 125} is another solution. Indeed $125 * 125 =
9\imod{2^7}$, as you can verify separately.  We can search for more:
\begin{Verbatim}
  Cryptol> :sat \x -> isSqrtOf9 x && ~(elem (x, [3, 125]))
  \x -> isSqrtOf9 x && ~(elem (x, [3, 131])) 253 = True
\end{Verbatim}
Rather than manually adding solutions we have already seen, we can
search for other solutions by asking the satisfiability checker for
more solutions using the {\tt satNum} setting:
\begin{Verbatim}
  Cryptol> :set satNum = 4
  Cryptol> :sat isSqrtOf9
  isSqrtOf9 3 = True
  isSqrtOf9 131 = True
  isSqrtOf9 125 = True
  isSqrtOf9 253 = True
\end{Verbatim}
By default, {\tt satNum} is set to {\tt 1}, so we only see one
solution. When we change it to {\tt 4}, the satisfiability checker
will try to find {\em up to} 4 solutions. We can also set it to {\tt
  all}, which will try to find as many solutions as possible.
\begin{Verbatim}
  Cryptol> :set satNum = 4
  Cryptol> :sat isSqrtOf9
  isSqrtOf9 3 = True
  isSqrtOf9 131 = True
  isSqrtOf9 125 = True
  isSqrtOf9 253 = True
\end{Verbatim}
So, we can rest assured that there are exactly four 8-bit square roots
of 9; namely 3, 131, 125, and 253. (Note that Cryptol can return the
satisfying solutions in any order depending on the backend-solver and
other configurations. What is guaranteed is that you will get
precisely the same set of solutions at the end.)

The whole point of the satisfiability checker is to be able to quickly
search for particular values that are solutions to potentially
complicated bit-valued functions. In this sense, satisfiability
checking can also be considered as an automated way to invert a
certain class of functions, going back from results to arguments. Of
course, this search is not done blindly, but rather using SAT\glosSAT
and SMT\glosSMT solvers to quickly find the satisfying
values. Cryptol's {\tt :sat} command\indCmdSat hides the complexity,
allowing the user to focus on the specification of the problem.

\todo[inline]{Re-add chapter on verification, or two chapters on
  \texttt{:sat} and \texttt{:prove}.}
% (We will see several larger applications of the satisfiability
% checker in Chapter~\ref{chap:usingsat}.)

\begin{Exercise}\label{ex:sat:0}
  Fermat's last theorem states that there are no integer solutions to
  the equation $a^n + b^n = c^n$ when $a, b, c > 0,$ and $n > 2$. We
  cannot code Fermat's theorem in Cryptol since we do not have
  arbitrary integers, but we can code the modular version of it where
  the exponentiation and addition is done modulo a fixed
  bit-size. Write a function {\tt modFermat} with the following
  signature:
\begin{code}
  type Quad a = ([a], [a], [a], [a])
  modFermat : {s} (fin s, s >= 2) => Quad s -> Bit
\end{code}
such that {\tt modFermat (a, b, c, n)} will return {\tt True} if the
modular version of Fermat's equation is satisfied by the values of
{\tt a}, {\tt b}, {\tt c}, and {\tt n}. Can you explain why you need
the constraints {\tt fin s} and {\tt s >= 2}?
\end{Exercise}
\begin{Answer}\ansref{ex:sat:0}
\begin{code}
  modFermat (a, b, c, n) = (a > 0) && (b > 0) && (c > 0) && (n > 2)
                         && (a^^n + b^^n == c^^n)
\end{code}
The {\tt fin s} predicate comes from the fact that we are doing
arithmetic and comparisons. The predicate {\tt s >= 2} comes from the
fact that we are comparing {\tt n} to {\tt 2}, which needs at least
$2$ bits to represent.
\end{Answer}

\begin{Exercise}\label{ex:sat:1}
  Use the {\tt :sat} command to see if there are any satisfying values
  for the modular version of Fermat's last theorem for various bit
  sizes.  Surprised? What can you conclude from your observations?
\end{Exercise}
\begin{Answer}\ansref{ex:sat:1}
We can try different instantiations as follows:
\begin{Verbatim}
  Cryptol> :sat modFermat : Quad(2) -> Bit
  modFermat : Quad(2) -> Bit (1, 2, 1, 3) = True
  Cryptol> :sat modFermat : Quad(3) -> Bit
  modFermat : Quad(3) -> Bit (4, 4, 4, 4) = True
  Cryptol> :sat modFermat : Quad(4) -> Bit
  modFermat : Quad(4) -> Bit (4, 4, 4, 8) = True
\end{Verbatim}
The modular form of Fermat's last theorem does not hold for any of the
instances up to and including 12-bits wide, when I stopped
experimenting myself. It is unlikely that it will hold for any
particular bit-size, although the above demonstration is not a proof.
(We would need to consult a mathematician for the general result!)
Also note that Cryptol takes longer and longer to find a satisfying
instance as you increase the bit-size.
\end{Answer}

%=====================================================================
%% \section{Safety checking}
%% \label{sec:safetychecking}

%% Safety checking is all about run-time exceptions: Division-by-zero
%% or index-out-of-bounds being the most infamous ones. Unlike many
%% other languages, Cryptol does not suffer from a certain class of
%% safety violations, such as trying to put in the number 1000 in a
%% buffer that is only 8-bits wide, or trying to multiply a string
%% with an integer; but certain run-time exceptions are still
%% possible. The goal of {\em safety checking} is to ensure that these
%% exceptions cannot happen at run-time.\indSafe\indCmdSafe

%% Here is a simple example to demonstrate:
%% \begin{Verbatim}
%% Cryptol> :safe \(x, y) -> x/y:[8]
%% *** 1 safety condition to be checked.
%% *** Checking for violations of: ``"command line", line 1, col 13: divide by zero''
%% *** Violation detected:
%% (\(x, y) -> x/y:[8]) (0, 0)
%%          = "command line", line 1, col 14: divide by zero
%% *** 1 problem found.
%%\end{Verbatim}
%%Cryptol is telling us that the function {\tt $\backslash$(x, y) ->
%% x/y} is {\em not} safe. When given the argument {\tt (0, 0)} it
%% will cause a division by zero exception. If we properly guard for
%% the condition, then the safety check will indeed pass:
%%\begin{Verbatim}
%% Cryptol> :safe \(x, y) -> if y == 0 then 0 else x/y:[8]
%% *** 1 safety condition to be checked.
%% *** Checking for violations of: ``"command line", line 1, col 35: divide by zero''
%% *** Verified safe.
%% *** All safety checks pass, safe to execute
%%\end{Verbatim}
%%By using {\tt :safe}\indCmdSafe on our top-level functions we can
%% ensure that our program is guaranteed not to have any run-time
%% exceptions.  Cryptol's safety-checker will detect the following
%% run-time exceptions:
%%\begin{itemize}
%% \item Division ({\tt /}) and modulus ({\tt \%}) by 0, both modular
%%   and polynomial versions (See
%%   Section~\ref{sec:polynomials})\indDiv\indMod\indDivPoly\indModPoly
%% \item Index-out-of-bounds errors for {\tt @}, {\tt @@}, {\tt !} and
%%   {\tt !!},\indIndex\indIndexs\indRIndex\indRIndexs
%%  \item Calling {\tt lg2} on {\tt 0},\indLg
%%  \item Data dependence on {\tt error} and the {\tt undefined}
%%    values,\indError\indUndefined
%%  \item Failure cases for Cryptol's {\tt ASSERT}
%%    expression.\indAssert
%%\end{itemize}

%%\begin{Exercise}\label{ex:safe:0}
%%Consider the function:
%%\begin{code}
%%  lkup : ([10][8], [4]) -> [8]
%%  lkup (xs, i) = xs @ i
%%\end{code}
%%Is it safe? For what values of {\tt i} does it throw an exception?
%%\end{Exercise}
%%\begin{Answer}\ansref{ex:safe:0}
%%  {\tt lkup} will throw an exception whenever {\tt i} is at least
%%  {\tt 10}. We can use Cryptol to detect this safety violation:
%%\begin{Verbatim}
%%  Cryptol> :safe lkup
%%  *** 1 safety condition to be checked.
%%  *** Checking for violations of: ``..: index of symbolic-value is out of bounds
%%  (valid range is 0 thru 9).''
%%  *** Violation detected:
%%  lkup ([255 255 255 255 255 255 255 255 255 255], 13)
%%           = .. index of 13 is out of bounds
%%  (valid range is 0 thru 9).
%%  *** 1 problem found.
%%\end{Verbatim}
%%In this case Cryptol identified that there will be an exception when
%% {\tt i} is 13.
%%\end{Answer}

%%\begin{Exercise}\label{ex:safe:1}
%%Here is an attempt to make {\tt lkup} safe:
%%\begin{code}
%%  lkup1 : ([10][8], [4]) -> [8]
%%  lkup1 (xs, i) = if i > 10 then 0 else xs @ i
%%\end{code}
%%Is the fix successful? Use the {\tt :safe} command to check your
%% answer.
%%\end{Exercise}
%%\begin{Answer}\ansref{ex:safe:1}
%%No. {\tt lkup1} suffers from the infamous off-by-1 error!
%%\begin{Verbatim}
%%  Cryptol> :safe lkup1
%%  *** 1 safety condition to be checked.
%%  *** Checking for violations of: ``..: index of symbolic-value is out of bounds
%%  (valid range is 0 thru 9).''
%%  *** Violation detected:
%%  lkup1 ([255 255 255 255 255 255 255 255 255 255], 10)
%%           = ..: index of 10 is out of bounds
%%  (valid range is 0 thru 9).
%%  *** 1 problem found.
%%\end{Verbatim}
%%This time Cryptol will pin-point to the only potential error value,
%% where {\tt i} is 10.
%%\end{Answer}
%%
%%\begin{Exercise}\label{ex:safe:2}
%%  Write a version of {\tt lkup} that fixes the safety issue. Use the
%%  {\tt :safe} command to verify that your fix is good.
%%\end{Exercise}
%%\begin{Answer}\ansref{ex:safe:2}
%%\begin{code}
%%  lkup2 : ([10][8], [4]) -> [8]
%%  lkup2 (xs, i) = if i > 9 then 0 else xs @ i
%%\end{code}
%%We have:
%%\begin{Verbatim}
%%  Cryptol> :safe lkup2
%%  *** 1 safety condition to be checked.
%%  *** Checking for violations of: ``..: index of symbolic-value is out of bounds
%%  (valid range is 0 thru 9).''
%%  *** Verified safe.
%%  *** All safety checks pass, safe to execute.
%%\end{Verbatim}
%%\end{Answer}
%%
%%\paragraph*{Over-conservative safety checking}
%% Cryptol's safety checker is very conservative: If it says that a
%% function is {\em safe to execute}, you are guaranteed that the
%% execution can not raise any run-time exception. However, the
%% converse is not always true: In certain cases, Cryptol can signal a
%% potential safety violation which will actually not cause an error
%% when {\em run in the normal interpreter}.
%%
%% Here is an example to illustrate the idea. (The example is somewhat
%% contrived, but similar patterns of coding does arise in Cryptol
%% programming.) We will use Cryptol's {\tt undefined}
%% value\indUndefined, which would throw an exception if executed:
%%\begin{Verbatim}
%%  Cryptol> (1:[8]) + undefined
%%  Entered Cryptol 'undefined' value
%%\end{Verbatim}
%%
%%Cryptol's {\tt undefined}\indUndefined is useful when representing
%% values that should {\em not} be needed during execution, as
%% illustrated below:
%%\begin{code}
%%  choose : [8] -> [8]
%%  choose c = ([undefined] # [1 ..]) @ index
%%    where index = if c == 0 then 1 else c
%%\end{code}
%%Notice that {\tt undefined} will not be referenced: The {\tt index}
%% can never be {\tt 0}. (Why?) If we ask {\tt :safe}, though, it will
%% tell us the following:
%%\begin{Verbatim}
%%  Cryptol> :safe choose
%%  *** 1 safety condition to be checked.
%%  *** Checking for violations of: ``Entered Cryptol 'undefined' value''
%%  *** Violation detected:
%%  choose 0
%%           = 1
%%  *** ALERT: This safety issue is only relevant for certain target platforms
%%  ***        and hence does not cause an exception in the above run.
%%  ***        C and hardware translations might be subject to stricter semantics.
%%  *** 1 problem found.
%%\end{Verbatim}
%%The {\tt :safe} command\indCmdSafe told us that there is a safety
%% violation when {\tt c} is {\tt 0}, then it also showed us that if
%% we run {\tt choose} with {\tt 0} we get the value {\tt 1}, not an
%% exception. At first, this sounds quite counterintuitive. However,
%% the {\tt ALERT} given by Cryptol sheds some light into the
%% potential problem.  When Cryptol is used in the normal interpreter
%% mode, this program will indeed not produce any exceptions. However,
%% Cryptol programs can also be compiled to other targets, such as C
%% or Java, or other hardware platforms. What Cryptol is telling us is
%% that these translations are {\em not} guaranteed to be exception
%% free, and hence it is worth making sure the generated code will
%% behave properly.
%%
%% While it is important to understand what Cryptol is telling us, the
%% user can safely ignore these alerts most of the time, especially
%% when one is focusing only at the Cryptol level. (We will revisit
%% this topic in Section~\ref{sec:pitfall:thmexceptions}.)
%%
%%\begin{Exercise}\label{ex:safe:3}
%%  Rewrite {\tt choose} so that {\tt :safe} will not issue any
%%  alerts.
%%\end{Exercise}
%%\begin{Answer}\ansref{ex:safe:3}
%%  One way would be to replace {\tt undefined}\indUndefined with {\tt
%%  zero}.\indZero Since the result will never be needed, we can
%%  safely replace it with {\tt 0} without changing the semantics.
%%\end{Answer}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "../main/Cryptol"
%%% End:


%%
%% Automatically generated file from DocOnce source
%% (https://github.com/hplgit/doconce/)
%%
%%

%-------------------- begin preamble ----------------------

\documentclass[%
oneside,                 % oneside: electronic viewing, twoside: printing
final,                   % draft: marks overfull hboxes, figures with paths
10pt]{article}

\listfiles               %  print all files needed to compile this document

\usepackage{relsize,makeidx,color,setspace,amsmath,amsfonts,amssymb}
\usepackage[table]{xcolor}
\usepackage{bm,ltablex,microtype}

\usepackage[pdftex]{graphicx}

\usepackage{fancyvrb} % packages needed for verbatim environments

\usepackage[T1]{fontenc}
%\usepackage[latin1]{inputenc}
\usepackage{ucs}
\usepackage[utf8x]{inputenc}

\usepackage{lmodern}         % Latin Modern fonts derived from Computer Modern

% Hyperlinks in PDF:
\definecolor{linkcolor}{rgb}{0,0,0.4}
\usepackage{hyperref}
\hypersetup{
    breaklinks=true,
    colorlinks=true,
    linkcolor=linkcolor,
    urlcolor=linkcolor,
    citecolor=black,
    filecolor=black,
    %filecolor=blue,
    pdfmenubar=true,
    pdftoolbar=true,
    bookmarksdepth=3   % Uncomment (and tweak) for PDF bookmarks with more levels than the TOC
    }
%\hyperbaseurl{}   % hyperlinks are relative to this root

\setcounter{tocdepth}{2}  % levels in table of contents

% Tricks for having figures close to where they are defined:
% 1. define less restrictive rules for where to put figures
\setcounter{topnumber}{2}
\setcounter{bottomnumber}{2}
\setcounter{totalnumber}{4}
\renewcommand{\topfraction}{0.95}
\renewcommand{\bottomfraction}{0.95}
\renewcommand{\textfraction}{0}
\renewcommand{\floatpagefraction}{0.75}
% floatpagefraction must always be less than topfraction!
% 2. ensure all figures are flushed before next section
\usepackage[section]{placeins}
% 3. enable begin{figure}[H] (often leads to ugly pagebreaks)
%\usepackage{float}\restylefloat{figure}

% prevent orhpans and widows
\clubpenalty = 10000
\widowpenalty = 10000

% --- end of standard preamble for documents ---

% insert custom LaTeX commands...

\raggedbottom
\makeindex
\usepackage[totoc]{idxlayout}   % for index in the toc
\usepackage[nottoc]{tocbibind}  % for references/bibliography in the toc

%-------------------- end preamble ----------------------

\begin{document}

% matching end for #ifdef PREAMBLE

\newcommand{\exercisesection}[1]{\subsection*{#1}}

% ------------------- main content ----------------------

% ----------------- title -------------------------

\thispagestyle{empty}

\begin{center}
{\LARGE\bf
\begin{spacing}{1.25}
Project 5, deadline  December 9, 2016
\end{spacing}
}
\end{center}

% ----------------- author(s) -------------------------

\begin{center}
{\bf Astronomy project, $N$-body simulation of an open galactic cluster${}^{}$} \\ [0mm]
\end{center}

\begin{center}
% List of all institutions:
\end{center}

% ----------------- end author(s) -------------------------

% --- begin date ---
\begin{center}
Fall semester 2016
\end{center}
% --- end date ---

\vspace{1cm}

\subsection*{Theoretical background and description of the system}

The goal of this project is to develop a code that can perform
simulations of an open cluster using Newtonian gravity.
This project is inspired by an article by
\href{{http://dx.doi.org/10.1063/1.3462740}}{M. Joyce and co-workers}, see also the references below.  When solving this
project, we recommend downloading this article. It is a good companion
to understand the physics discussed here.
This project requires a good  background in cosmology and
astrophysics. The project can be seen as a continuation of project 3 and we will use again
the Velocity Verlet algorithm.

We will try to build a simple model of an open
cluster, as done by \href{{http://dx.doi.org/10.1063/1.3462740}}{M. Joyce and co-workers}. An
open cluster is a group of up to a few thousand gravitationally bound
stars created from the collapse of a molecular cloud. This collapse
leads to a flurry of star formation. Open clusters are usually found
in the arms of spiral galaxies, or in irregular galaxies. Since stars
in an open cluster have roughly the same age, and are made from the
same material, they are interesting in the study of stellar evolution,
since many of the variable parameters we have when comparing two stars
are kept constant.

Once open clusters are formed they gradually dissipate as members get
ejected from the cluster due to random collisions, this means that
open clusters generally last only a few hundred million years. In
the figure here we see the Hertzsprung-Russell diagrams for two open
clusters.

\begin{figure}[!ht]  %
  \centerline{\includegraphics[width=1.0\linewidth]{open_cluster_hr_diagram_ages.png}}
  \caption{
  Hertzsprung-Russell diagrams for two open clusters, M67 and   NGC 188. We see that most of the stars are on the main sequence. In   the older cluster, NGC 188, we see that the heaviest stars are just   now leaving the main sequence, while the younger cluster, M67.
  }
\end{figure}
%\clearpage % flush figures

We will study a simple model for how an open cluster is made from
the gravitational collapse and interaction among a large number of
stars. We want to study this collapse, and the statistical properties
of the collapsed system.
We will simulate what is called a
\emph{cold collapse}, this means that we start the particles with little
or no initial velocity. You will also find it useful to start with the code you developed in project 3.

\paragraph{Project 5a): Extension of project 3.}
Extend your code from project 3 to an arbitrary number of particles, $N$,
  starting with a uniform (random) distribution within a sphere of a
  given radius $R_0$. Start the particles at rest, with masses
  randomly distributed by a Gaussian distribution around ten solar
  masses with a standard deviation of one solar mass. Use solar masses
  and light years as units of mass and length and make your equations dimensionless.
This function is included below and  can be used to generate
random numbers which follow a Gaussian (or normal) distribution.  Alternatively, you can use
the \emph{random} class in c++ and generate random numbers according to the normal distribution.

How large time steps are required given $R_0 = 20$ ly
(light years), and  $N = 100$? Do we have any units of time that fit
this timescale?
In the limit where $N \rightarrow \infty$, keeping $\rho_0$ constant,
we get a continuous fluid. In this case the system collapses into a
singularity at a finite time $\tau_{\mathrm{crunch}} = \sqrt{\frac{3\pi}{32G\rho_0}}$.

Why do we not observe this singularity in our model? Use
$\tau_{\mathrm{crunch}}$ as the unit of time, and find $G$ in these units ($G$
will become a function of the number of particles $N$, and the average
mass of the particles, $\mu$).

\paragraph{Project 5b): Time to reach equilibrium.}
Run the system for a few $\tau_{\mathrm{crunch}}$. Save the positions
  of the particles at different times to file. Does the system
  reach an equilibrium? How long time does this take?

\paragraph{Project 5c): Kinetic and potential energy.}
Make a function that calculates the kinetic and potential
  energy of the system. Is the energy conserved? Some of the
  particles are ejected from the system, how can we identify these
  particles from the energies we have calculated? How much of the
  energy of the system is taken away by particle ejection? How does
  this change with different values of $N$? Are there still particles
  being ejected after the system reaches equilibrium?

\paragraph{Project 5d): Numerical instabilities.}
We will now introduce a smoothing function to take care of
  the numerical instability that arises when two of the particles come
  very close. There are a lot of ways of inserting such a smoothing, but
  we will just look at a very simple one. We will modify the Newtonian
  force law to make it finite at short ranges

\begin{equation*}
F_{mod} = -\frac{GM_1M_2}{r^2 + \epsilon^2}.
\end{equation*}

The parameter $\epsilon$ is a \emph{small} real constant. What should the value of this parameter be? Try out different values (which one gives you the best energy conservation?). Can we justify this correction to the pure Newtonian force by noting that our particles do not represent actual point particles
but rather mass distributions of some finite extent? Does the addition
of this correction change any of the results from the previous exercise?

\paragraph{Project 5e): Test of the virial theorem.}
Now we will look at the particles that are bound (not ejected).
What is the distribution of potential and kinetic energy?

The virial theorem says that for a bound gravitational system in
equilibrium we have

\begin{equation*}
2\langle K\rangle = -\langle V \rangle,
\end{equation*}
where $\langle K\rangle$
is the average (over time) kinetic energy of the system and $ \langle V
\rangle$ is the average potential energy.

By the ergodic hypothesis we can take an ensemble
average (average over a large system) instead of the time average.

Are your results consistent with the virial theorem?

\paragraph{Project 5f): Density of particles.}
Try to plot the radial density of the particles (the particle density as a function of radius) in the
  equilibrium state. How would you extract such an information from your calculations? (Hint: make a histogram for the radial
particle density) What is the average distance? What is the
  standard deviation? Plot the radial distribution of particles.

Run the code for different number of initial particles, keeping the
total mass constant. The interpretation of one particle as one or a few stars will not be useful any more when you increase the number of particles beyond a few thousand, however the analysis of gravitationally bound systems of many particles have much broader applications than cold collapse of open clusters, so the results are still highly relevant. What is the average distance from the centre of the cluser as a function of $N$?

The radial distribution of particles in this kind of cold collapse can often be fit very well with the simple expression

\begin{equation*}
n(r) = \frac{n_0}{\left(1 +\left(\frac{r}{r_0}\right)^4\right)}.
\end{equation*}
Try to fit your data to this curve, what is the value $n_0$ and $r_0$? Can you find how these values depend on N?

How many particles can you simulate?

Compare your results with those found in  \href{{http://dx.doi.org/10.1063/1.3462740}}{M. Joyce and co-workers}.

If you want, you can also compare your results to the well-known Navarro-Frenk-White profile

\begin{equation*}
\rho(r) = \frac{\rho_0}{\frac{r}{r_0}\left(1 +\frac{r}{r_0}\right)^2}.
\end{equation*}
Does this give a  better fit?

\paragraph{References.}
\begin{enumerate}
\item \href{{http://dx.doi.org/10.1063/1.3462740}}{M. Joyce, B. Marcos, and F. Sylos Labini, Cold uniform spherical collapse revisited, AIP Conf. Proc. 1241, 955 (2010)}. See also \href{{http://arxiv.org/abs/1011.0614}}{arXiv1011.0614 (2011)}.

\item P. J. E. Peebles, \emph{The Large-Scale Structure of the Universe}, Princeton University Press, 1980. See also C. Payne-Gaposchkin,  \emph{Stars and clusters}, (Cambridge, Harvard University Press, 1979).

\item A. Friedman,  On the Curvature of Space,  General Relativity and Gravitation 31,  1991 (1999)":"http://link.springer.com/article/10.1023%2FA%3A1026751225741".
\end{enumerate}

\noindent
\paragraph{Function for Gaussian random numbers.}
\begin{verbatim}
// ran2 for uniform deviates, initialize with negative seed.
double ran2(long *);

// function for gaussian random numbers
double gaussian_deviate(long *);

/*
** The function
**         ran2()
** is a long periode (> 2 x 10^18) random number generator of
** L'Ecuyer and Bays-Durham shuffle and added safeguards.
** Call with idum a negative integer to initialize; thereafter,
** do not alter idum between sucessive deviates in a
** sequence. RNMX should approximate the largest floating point value
** that is less than 1.
** The function returns a uniform deviate between 0.0 and 1.0
** (exclusive of end-point values).
*/

#define IM1 2147483563
#define IM2 2147483399
#define AM (1.0/IM1)
#define IMM1 (IM1-1)
#define IA1 40014
#define IA2 40692
#define IQ1 53668
#define IQ2 52774
#define IR1 12211
#define IR2 3791
#define NTAB 32
#define NDIV (1+IMM1/NTAB)
#define EPS 1.2e-7
#define RNMX (1.0-EPS)

double ran2(long *idum)
{
  int            j;
  long           k;
  static long    idum2 = 123456789;
  static long    iy=0;
  static long    iv[NTAB];
  double         temp;

  if(*idum <= 0) {
    if(-(*idum) < 1) *idum = 1;
    else             *idum = -(*idum);
    idum2 = (*idum);
    for(j = NTAB + 7; j >= 0; j--) {
      k     = (*idum)/IQ1;
      *idum = IA1*(*idum - k*IQ1) - k*IR1;
      if(*idum < 0) *idum +=  IM1;
      if(j < NTAB)  iv[j]  = *idum;
    }
    iy=iv[0];
  }
  k     = (*idum)/IQ1;
  *idum = IA1*(*idum - k*IQ1) - k*IR1;
  if(*idum < 0) *idum += IM1;
  k     = idum2/IQ2;
  idum2 = IA2*(idum2 - k*IQ2) - k*IR2;
  if(idum2 < 0) idum2 += IM2;
  j     = iy/NDIV;
  iy    = iv[j] - idum2;
  iv[j] = *idum;
  if(iy < 1) iy += IMM1;
  if((temp = AM*iy) > RNMX) return RNMX;
  else return temp;
}
#undef IM1
#undef IM2
#undef AM
#undef IMM1
#undef IA1
#undef IA2
#undef IQ1
#undef IQ2
#undef IR1
#undef IR2
#undef NTAB
#undef NDIV
#undef EPS
#undef RNMX

// End: function ran2()

// random numbers with gaussian distribution
double gaussian_deviate(long * idum)
{
  static int iset = 0;
  static double gset;
  double fac, rsq, v1, v2;

  if ( idum < 0) iset =0;
  if (iset == 0) {
    do {
      v1 = 2.*ran2(idum) -1.0;
      v2 = 2.*ran2(idum) -1.0;
      rsq = v1*v1+v2*v2;
    } while (rsq >= 1.0 || rsq == 0.);
    fac = sqrt(-2.*log(rsq)/rsq);
    gset = v1*fac;
    iset = 1;
    return v2*fac;
  } else {
    iset =0;
    return gset;
  }
} // end function for gaussian deviates

\end{verbatim}

\subsection*{Introduction to numerical projects}

Here follows a brief recipe and recommendation on how to write a report for each
project.

\begin{itemize}
  \item Give a short description of the nature of the problem and the eventual  numerical methods you have used.

  \item Describe the algorithm you have used and/or developed. Here you may find it convenient to use pseudocoding. In many cases you can describe the algorithm in the program itself.

  \item Include the source code of your program. Comment your program properly.

  \item If possible, try to find analytic solutions, or known limits in order to test your program when developing the code.

  \item Include your results either in figure form or in a table. Remember to        label your results. All tables and figures should have relevant captions        and labels on the axes.

  \item Try to evaluate the reliabilty and numerical stability/precision of your results. If possible, include a qualitative and/or quantitative discussion of the numerical stability, eventual loss of precision etc.

  \item Try to give an interpretation of you results in your answers to  the problems.

  \item Critique: if possible include your comments and reflections about the  exercise, whether you felt you learnt something, ideas for improvements and  other thoughts you've made when solving the exercise. We wish to keep this course at the interactive level and your comments can help us improve it.

  \item Try to establish a practice where you log your work at the  computerlab. You may find such a logbook very handy at later stages in your work, especially when you don't properly remember  what a previous test version  of your program did. Here you could also record  the time spent on solving the exercise, various algorithms you may have tested or other topics which you feel worthy of mentioning.
\end{itemize}

\noindent
\subsection*{Format for electronic delivery of report and programs}

The preferred format for the report is a PDF file. You can also use DOC or postscript formats or as an ipython notebook file.  As programming language we prefer that you choose between C/C++, Fortran2008 or Python. The following prescription should be followed when preparing the report:

\begin{itemize}
  \item Use Devilry to hand in your projects, log in  at  \href{{http://devilry.ifi.uio.no}}{\nolinkurl{http://devilry.ifi.uio.no}} with your normal UiO username and password and choose either 'fys3150' or 'fys4150'. There you can load up the files within the deadline.

  \item Upload \textbf{only} the report file!  For the source code file(s) you have developed please provide us with your link to your github domain.  The report file should include all of your discussions and a list of the codes you have developed.  Do not include library files which are available at the course homepage, unless you have made specific changes to them.

  \item In your git repository, please include a folder which contains selected results. These can be in the form of output from your code for a selected set of runs and input parametxers.

  \item In this and all later projects, you should include tests (for example unit tests) of your code(s).

  \item Comments  from us on your projects, approval or not, corrections to be made  etc can be found under your Devilry domain and are only visible to you and the teachers of the course.
\end{itemize}

\noindent
Finally,
we encourage you to work two and two together. Optimal working groups consist of
2-3 students. For this specific report you can  hand in a common report.

% ------------------- end of main content ---------------

\end{document}


\lab{SQL 2 (The Sequel)}{SQL 2 (The Sequel)}
\objective{
Since SQL databases contain multiple tables, retrieving information about the data can be complicated.
In this lab we discuss joins, grouping, and other advanced SQL query concepts to facilitate rapid data retrieval.
% SQL queries can often get very complex, and can involve several relations in a database.
% In this lab, we discuss how to join tables together in order to get information stored between them.
% We also discuss several other advanced SQL selection concepts, such as: aggregate functions, ordering, grouping, case expressions, and comparison structures.
}

We will use the following database as an example throughout this lab, found in \texttt{students.db}.

\begin{table}[H]
\begin{subtable}{0.55\textwidth}
    \centering
    \begin{subtable}{.45\textwidth}
        \centering
        \footnotesize
        \begin{tabular}{|l|l|}
            \hline MajorID & MajorName \\ \hline
            1 & Math \\
            2 & Science \\
            3 & Writing \\
            4 & Art \\ \hline
        \end{tabular}
        \caption{MajorInfo}
        \label{table:sql2-student-majorinfo}
    \end{subtable}
    \hfil
    \begin{subtable}{.45\textwidth}
        \centering
        \footnotesize
        \begin{tabular}{|l|l|}
            \hline CourseID & CourseName \\ \hline
            1 & Calculus \\
            2 & English \\
            3 & Pottery \\
            4 & History \\ \hline
        \end{tabular}
        \caption{CourseInfo}
        \label{table:sql2-student-courseinfo}
    \end{subtable}
    \\[1.em] % NOTE: Use 4.01em to align the tables at the top and bottom.
    \begin{subtable}{\textwidth}
        \centering
        \footnotesize
        \begin{tabular}{|l|l|l|}
            \hline StudentID & StudentName & MajorID \\ \hline
            401767594 & Michelle Fernandez & 1 \\
            678665086 & Gilbert Chapman & NULL \\
            553725811 & Roberta Cook & 2 \\
            886308195 & Rene Cross & 3 \\
            103066521 & Cameron Kim & 4 \\
            821568627 & Mercedes Hall & NULL \\
            206208438 & Kristopher Tran & 2 \\
            341324754 & Cassandra Holland & 1 \\
            262019426 & Alfonso Phelps & NULL \\
            622665098 & Sammy Burke & 2 \\ \hline
        \end{tabular}
        \caption{StudentInfo}
        \label{table:sql2-student-info}
    \end{subtable}
\end{subtable}
\hfil
\begin{subtable}{.35\textwidth}
    \centering
    \footnotesize
    \begin{tabular}{|l|l|l|}
        \hline StudentID & CourseID & Grade \\ \hline
        401767594 & 4 & C \\
        401767594 & 3 & B-- \\
        678665086 & 4 & A+ \\
        678665086 & 3 & A+ \\
        553725811 & 2 & C \\
        678665086 & 1 & B \\
        886308195 & 1 & A \\
        103066521 & 2 & C \\
        103066521 & 3 & C-- \\
        821568627 & 4 & D \\
        821568627 & 2 & A+ \\
        821568627 & 1 & B \\
        206208438 & 2 & A \\
        206208438 & 1 & C+ \\
        341324754 & 2 & D-- \\
        341324754 & 1 & A-- \\
        103066521 & 4 & A \\
        262019426 & 2 & B \\
        262019426 & 3 & C \\
        622665098 & 1 & A \\
        622665098 & 2 & A-- \\ \hline
    \end{tabular}
    \caption{StudentGrades}
    \label{table:sql2-student-grades}
\end{subtable}
\caption{Student database.}
\end{table}

\section*{Joining Tables} % ===================================================

A \emph{join} combines rows from different tables in a database based on common attributes.
In other words, a join operation creates a new, temporary table containing data from 2 or more existing tables.
Join commands in SQLite have the following general syntax.
\begin{lstlisting}
SELECT <alias.column, ...>
    FROM <table> AS <alias> JOIN <table> AS <alias>, ...
    ON <alias.column> == <alias.column>, ...
    WHERE <condition>;
\end{lstlisting}

The \lsql{ON} clause tells the query how to join tables together.
Typically if there are $N$ tables being joined together, there should be $N-1$ conditions in the \lsql{ON} clause.

\subsection*{Inner Joins} % ---------------------------------------------------

An \emph{inner join} creates a temporary table with the rows that have exact matches on the attribute(s) specified in the \lsql{ON} clause.
Inner joins \textbf{intersect} two or more tables, as in Figure \ref{fig:sql2-inner-join}.

\begin{figure}[H] % Inner and outer joins.
\captionsetup[subfigure]{justification=centering}
\centering
\begin{subfigure}{.4\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/inner_join.pdf}
    \caption{An inner join of A, B, and C.}
    \label{fig:sql2-inner-join}
\end{subfigure}
\hfil
\begin{subfigure}{.4\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/left_outer.pdf}
    \caption{A left outer join of A with B and C.}
    \label{fig:sql2-left-join}
\end{subfigure}
%
% \begin{subfigure}{.3\textwidth}
%     \centering
%     \includegraphics[width=\textwidth]{figures/full_outer.pdf}
%     \caption{A full outer join of A, B, and C.}
%     \label{fig:sql2-full-join}
% \end{subfigure}
\caption{}
\label{fig:joins}
\end{figure}

For example, Table \ref{table:sql2-student-info} (\li{StudentInfo}) and Table \ref{table:sql2-student-majorinfo} (\li{MajorInfo}) both have a \li{MajorID} column, so the tables can be joined by pairing rows that have the same \li{MajorID}.
Such a join temporarily creates the following table.

\begin{table}[H]
\centering
\footnotesize
\begin{tabular}{|l|l|l|l|l|}
    \hline StudentID & StudentName & MajorID & MajorID & MajorName \\ \hline
    401767594 & Michelle Fernandez & 1 & 1 & Math \\
    % 678665086 & Gilbert Chapman & NULL \\
    553725811 & Roberta Cook & 2 & 2 & Science \\
    886308195 & Rene Cross & 3 & 3 & Writing \\
    103066521 & Cameron Kim & 4 & 4 & Art \\
    % 821568627 & Mercedes Hall & NULL \\
    206208438 & Kristopher Tran & 2 & 2 & Science \\
    341324754 & Cassandra Holland & 1 & 1 & Math \\
    % 262019426 & Alfonso Phelps & NULL \\
    622665098 & Sammy Burke & 2 & 2 & Science \\ \hline
\end{tabular}
\caption{An inner join of \li{StudentInfo} and \li{MajorInfo} on \li{MajorID}.}
\label{table:sql2-inner-join-temp}
\end{table}

Notice that this table is missing the rows where \li{MajorID} was \lsql{NULL} in the \li{StudentInfo} table.
This is because there was no match for \lsql{NULL} in the \li{MajorID} column of the \li{MajorInfo} table, so the inner join throws those rows away.

Because joins deal with multiple tables at once, it is important to assign table aliases with the \lsql{AS} command.
Join statements can also be supplemented with \lsql{WHERE} clauses like regular queries.

\begin{lstlisting}
>>> import sqlite3 as sql
>>> conn = sql.connect("students.db")
>>> cur = conn.cursor()

>>> cur.execute("SELECT * "
...             "FROM StudentInfo AS SI INNER JOIN MajorInfo AS MI "
...             "ON SI.MajorID == MI.MajorID;").fetchall()
<<[(401767594, 'Michelle Fernandez', 1, 1, 'Math'),
 (553725811, 'Roberta Cook', 2, 2, 'Science'),
 (886308195, 'Rene Cross', 3, 3, 'Writing'),
 (103066521, 'Cameron Kim', 4, 4, 'Art'),
 (206208438, 'Kristopher Tran', 2, 2, 'Science'),
 (341324754, 'Cassandra Holland', 1, 1, 'Math'),
 (622665098, 'Sammy Burke', 2, 2, 'Science')]>>

# Select the names and ID numbers of the math majors.
>>> cur.execute("SELECT SI.StudentName, SI.StudentID "
...             "FROM StudentInfo AS SI INNER JOIN MajorInfo AS MI "
...             "ON SI.MajorID == MI.MajorID "
...             "WHERE MI.MajorName == 'Math';").fetchall()
<<[('Cassandra Holland', 341324754), ('Michelle Fernandez', 401767594)]>>
\end{lstlisting}

\begin{problem} % Simple inner join.
Write a function that accepts the name of a database file.
Assuming the database to be in the format of Tables \ref{table:sql2-student-majorinfo}--\ref{table:sql2-student-grades}, query the database for the list of the names of students who have a B grade in any course (not a B-- or a B+).
Be sure to return a list of strings, not a list of tuples of strings.
\end{problem}

\subsection*{Outer Joins} % ---------------------------------------------------

A \emph{left outer join}, sometimes called a \emph{left join}, creates a temporary table with \textbf{all} of the rows from the first (left-most) table, and all the ``matched'' rows on the given attribute(s) from the other relations.
Rows from the left table that don't match up with the columns from the other tables are supplemented with \lsql{NULL} values to fill extra columns.
Compare the following table and code to Table \ref{table:sql2-inner-join-temp}.

\begin{table}[H]
\centering
\footnotesize
\begin{tabular}{|l|l|l|l|l|}
    \hline StudentID & StudentName & MajorID & MajorID & MajorName \\ \hline
    401767594 & Michelle Fernandez & 1 & 1 & Math \\
    678665086 & Gilbert Chapman & NULL & NULL & NULL \\
    553725811 & Roberta Cook & 2 & 2 & Science \\
    886308195 & Rene Cross & 3 & 3 & Writing \\
    103066521 & Cameron Kim & 4 & 4 & Art \\
    821568627 & Mercedes Hall & NULL & NULL & NULL \\
    206208438 & Kristopher Tran & 2 & 2 & Science \\
    341324754 & Cassandra Holland & 1 & 1 & Math \\
    262019426 & Alfonso Phelps & NULL & NULL & NULL \\
    622665098 & Sammy Burke & 2 & 2 & Science \\ \hline
\end{tabular}
\caption{A left outer join of \li{StudentInfo} and \li{MajorInfo} on \li{MajorID}.}
\label{table:sql2-left-outer-join}
\end{table}

\begin{lstlisting}
>>> cur.execute("SELECT * "
...             "FROM StudentInfo AS SI LEFT OUTER JOIN MajorInfo AS MI "
...             "ON SI.MajorID == MI.MajorID;").fetchall()
<<[(401767594, 'Michelle Fernandez', 1, 1, 'Math'),
 (678665086, 'Gilbert Chapman', None, None, None),
 (553725811, 'Roberta Cook', 2, 2, 'Science'),
 (886308195, 'Rene Cross', 3, 3, 'Writing'),
 (103066521, 'Cameron Kim', 4, 4, 'Art'),
 (821568627, 'Mercedes Hall', None, None, None),
 (206208438, 'Kristopher Tran', 2, 2, 'Science'),
 (341324754, 'Cassandra Holland', 1, 1, 'Math'),
 (262019426, 'Alfonso Phelps', None, None, None),
 (622665098, 'Sammy Burke', 2, 2, 'Science')]>>
\end{lstlisting}

Some flavors of SQL also support the \lsql{RIGHT OUTER JOIN} command, but \li{sqlite3} does not recognize the command since \lsql{T1 RIGHT OUTER JOIN T2} is equivalent to \lsql{T2 LEFT OUTER JOIN T1}.

\subsection*{Joining Multiple Tables} % ---------------------------------------

Complicated queries often join several different relations.
If the same kind of join is being used, the relations and conditional statements can be put in list form.
For example, the following code selects courses that Kristopher Tran has taken, and the grades that he got in those courses, by joining three tables together.
Note that 2 conditions are required in the \lsql{ON} clause in this case.

\begin{lstlisting}
>>> cur.execute("SELECT CI.CourseName, SG.Grade "
...             "FROM StudentInfo AS SI "           # Join 3 tables.
...                 "INNER JOIN CourseInfo AS CI, StudentGrades SG "
...             "ON SI.StudentID==SG.StudentID AND CI.CourseID==SG.CourseID "
...             "WHERE SI.StudentName == 'Kristopher Tran';").fetchall()
<<[('Calculus', 'C+'), ('English', 'A')]>>
\end{lstlisting}

To use different kinds of joins in a single query, append one join statement after another.
The join closest to the beginning of the statement is executed first, creating a temporary table, and the next join attempts to operate on that table.
The following example performs an additional join on Table \ref{table:sql2-left-outer-join} to find the name and major of every student who got a C in a class.

\begin{lstlisting}
# Do an inner join on the results of the left outer join.
>>> cur.execute("SELECT SI.StudentName, MI.MajorName "
...             "FROM StudentInfo AS SI LEFT OUTER JOIN MajorInfo AS MI "
...             "ON SI.MajorID == MI.MajorID "
...             "INNER JOIN StudentGrades AS SG "
...             "ON SI.StudentID == SG.StudentID "
...             "WHERE SG.Grade == 'C';").fetchall()
<<[('Michelle Fernandez', 'Math'),
 ('Roberta Cook', 'Science'),
 ('Cameron Kim', 'Art'),
 ('Alfonso Phelps', None)]>>
\end{lstlisting}

In this last example, note carefully that Alfonso Phelps would have been excluded from the result set if an inner join was performed first instead of an outer join (since he lacks a major).

\begin{problem} % Combining joins.
Write a function that accepts the name of a database file.
Query the database for all tuples of the form \li{(Name, MajorName, Grade)} where \li{Name} is a student's name and \li{Grade} is their grade in Calculus.
Only include results for students that are actually taking Calculus, but be careful not to exclude students who haven't declared a major.
\end{problem}

\section*{Grouping Data} % ===================================================

% \begin{table}
% \begin{tabular}{c|l}
% Function & Description \\
% \hline
% MIN() & Retrieve the smallest numeric value of a column \\
% MAX()& Retrieve the largest numeric value of a column \\
% SUM() & Sum the numeric values of a column \\
% AVG() & Retrieve the average numeric value of the column \\
% COUNT() & Retrieve the total number of matching records in a column \\
% \end{tabular}
% \caption{SQL aggregation functions.}
% \label{table:aggregations}
% \end{table}

Many data sets can be naturally sorted into groups.
The \lsql{GROUP BY} command gathers rows from a table and groups them by a certain attribute.
The groups are then combined by one of the \emph{aggregate functions} \lsql{AVG()}, \lsql{MIN()}, \lsql{MAX()}, \lsql{SUM()}, or \lsql{COUNT()}.
Each of these functions accepts the name of the column to be operated on.
Note that the first four of these require the column to hold numerical data.
Since our database has no such data, we will delay examples of using the functions other than \lsql{COUNT()} until later.

The following code groups the rows in Table \ref{table:sql2-student-grades} by \li{studentID} and counts the number of entries in each group.
 % (the number of courses that student is enrolled in).

\begin{lstlisting}
>>> cur.execute("SELECT StudentID, COUNT(*) "   # * means "all of the rows".
...             "FROM StudentGrades "
...             "GROUP BY StudentID;").fetchall()
<<[(103066521, 3),
 (206208438, 2),
 (262019426, 2),
 (341324754, 2),
 (401767594, 2),
 (553725811, 1),
 (622665098, 2),
 (678665086, 3),
 (821568627, 3),
 (886308195, 1)]>>
\end{lstlisting}

\lsql{GROUP BY} can also be used in conjunction with joins.
The join creates a temporary table like Tables \ref{table:sql2-inner-join-temp} or \ref{table:sql2-left-outer-join}, the results of which can then be grouped.

\begin{lstlisting}
>>> cur.execute("SELECT SI.StudentName, COUNT(*) "
...             "FROM StudentGrades AS SG INNER JOIN StudentInfo AS SI "
...             "ON SG.StudentID == SI.StudentID "
...             "GROUP BY SG.StudentID;").fetchall()
<<[('Cameron Kim', 3),
 ('Kristopher Tran', 2),
 ('Alfonso Phelps', 2),
 ('Cassandra Holland', 2),
 ('Michelle Fernandez', 2),
 ('Roberta Cook', 1),
 ('Sammy Burke', 2),
 ('Gilbert Chapman', 3),
 ('Mercedes Hall', 3),
 ('Rene Cross', 1)]>>
\end{lstlisting}

Just like the \lsql{WHERE} clause chooses rows in a relation, the \lsql{HAVING} clause chooses groups from the result of a \lsql{GROUP BY} based on some criteria related to the groupings.
For this particular command, it is often useful (but not always necessary) to create an alias for the columns of the result set with the \lsql{AS} operator.
For instance, the result set of the previous example can be filtered down to only contain students who are taking 3 courses.

\begin{lstlisting}
>>> cur.execute("SELECT SI.StudentName, COUNT(*) as num_courses "   # Alias.
...             "FROM StudentGrades AS SG INNER JOIN StudentInfo AS SI "
...             "ON SG.StudentID == SI.StudentID "
...             "GROUP BY SG.StudentID "
...             "HAVING num_courses == 3;").fetchall()   # Refer to alias later.
<<[('Cameron Kim', 3), ('Gilbert Chapman', 3), ('Mercedes Hall', 3)]>>

# Alternatively, get just the student names.
>>> cur.execute("SELECT SI.StudentName "                            # No alias.
...             "FROM StudentGrades AS SG INNER JOIN StudentInfo AS SI "
...             "ON SG.StudentID == SI.StudentID "
...             "GROUP BY SG.StudentID "
...             "HAVING COUNT(*) == 3;").fetchall()
<<[('Cameron Kim',), ('Gilbert Chapman',), ('Mercedes Hall',)]>>
\end{lstlisting}

\begin{problem} % HAVING
Write a function that accepts a database file.
Query the database for the list of the names of courses that have at least 5 student enrolled in them.
Be sure to return a list of strings.
\end{problem}

\section*{Other Miscellaneous Commands} % =====================================

\subsection*{Ordering Result Sets} % ------------------------------------------

The \lsql{ORDER BY} command sorts a result set by one or more attributes.
Sorting can be done in ascending or descending order with \lsql{ASC} or \lsql{DESC}, respectively.
This is always the very last statement in a query.

\begin{lstlisting}
>>> cur.execute("SELECT SI.StudentName, COUNT(*) AS num_courses "   # Alias.
...             "FROM StudentGrades AS SG INNER JOIN StudentInfo AS SI "
...             "ON SG.StudentID == SI.StudentID "
...             "GROUP BY SG.StudentID "
...             "ORDER BY num_courses DESC, SI.StudentName ASC;").fetchall()
<<[('Cameron Kim', 3),>>                # The results are now ordered by the
<< ('Gilbert Chapman', 3),>>            # number of courses each student is in,
<< ('Mercedes Hall', 3),>>              # then alphabetically by student name.
<< ('Alfonso Phelps', 2),
 ('Cassandra Holland', 2),
 ('Kristopher Tran', 2),
 ('Michelle Fernandez', 2),
 ('Sammy Burke', 2),
 ('Rene Cross', 1),
 ('Roberta Cook', 1)]>>
\end{lstlisting}

\begin{problem} % COUNT(), ORDER BY.
Write a function that accepts a database file.
Query the given database for tuples of the form \li{(MajorName, N)} where \li{N} is the number of students in the specified major.
Sort the results in descending order by the count \li{N}, and then in alphabetic order by MajorName.
Include \textbf{Null} majors.
\end{problem}

\subsection*{Searching Text with Wildcards} % ---------------------------------

The \lsql{LIKE} operator within a \lsql{WHERE} clause matches patterns in a \lsql{TEXT} column.
The special characters \lsql{\%} and \lsql{_} and called \emph{wildcards} that match any number of characters or a single character, respectively.
For instance, \li{\%Z\_} matches any string of characters ending in a Z then another character, and \li{\%i\%} matches any string containing the letter i.

\begin{lstlisting}
>>> results = cur.execute("SELECT StudentName FROM StudentInfo "
...                       "WHERE StudentName LIKE '%i%';").fetchall()
>>> [r[0] for r in results]
<<['Michelle Fernandez', 'Gilbert Chapman', 'Cameron Kim', 'Kristopher Tran']>>
\end{lstlisting}

\begin{problem} % LIKE
Write a function that accepts a database file.
Query the database for tuples of the form \li{(StudentName, MajorName)} where the last name of the specified student begins with the letter C.
Include \textbf{Null} majors.
\end{problem}

\subsection*{Case Expressions} % ----------------------------------------------

A case expression maps the values in a column using boolean logic.
There are two forms of a case expression: simple and searched.
A \emph{simple case expression} matches and replaces specified attributes.

\begin{lstlisting}
# Replace the values MajorID with new custom values.
>>> cur.execute("SELECT StudentName, CASE MajorID "
...                 "WHEN 1 THEN 'Mathematics' "
...                 "WHEN 2 THEN 'Soft Science' "
...                 "WHEN 3 THEN 'Writing and Editing' "
...                 "WHEN 4 THEN 'Fine Arts' "
...                 "ELSE 'Undeclared' END "
...             "FROM StudentInfo "
...             "ORDER BY StudentName ASC;").fetchall()
<<[('Alfonso Phelps', 'Undeclared'),
 ('Cameron Kim', 'Fine Arts'),
 ('Cassandra Holland', 'Mathematics'),
 ('Gilbert Chapman', 'Undeclared'),
 ('Kristopher Tran', 'Soft Science'),
 ('Mercedes Hall', 'Undeclared'),
 ('Michelle Fernandez', 'Mathematics'),
 ('Rene Cross', 'Writing and Editing'),
 ('Roberta Cook', 'Soft Science'),
 ('Sammy Burke', 'Soft Science')]>>
\end{lstlisting}

A \emph{searched case expression} involves using a boolean expression at each step, instead of listing all of the possible values for an attribute.

\begin{lstlisting}
# Change NULL values in MajorID to 'Undeclared' and non-NULL to 'Declared'.
>>> cur.execute("SELECT StudentName, CASE "
...                 "WHEN MajorID IS NULL THEN 'Undeclared' "
...                 "ELSE 'Declared' END "
...             "FROM StudentInfo "
...             "ORDER BY StudentName ASC;").fetchall()
<<[('Alfonso Phelps', 'Undeclared'),
 ('Cameron Kim', 'Declared'),
 ('Cassandra Holland', 'Declared'),
 ('Gilbert Chapman', 'Undeclared'),
 ('Kristopher Tran', 'Declared'),
 ('Mercedes Hall', 'Undeclared'),
 ('Michelle Fernandez', 'Declared'),
 ('Rene Cross', 'Declared'),
 ('Roberta Cook', 'Declared'),
 ('Sammy Burke', 'Declared')]>>
\end{lstlisting}

\section*{Chaining Queries} % =================================================

The result set of any SQL query is really just another table with data from the original database.
Separate queries can be made from result sets by enclosing the entire query in parentheses.
For these sorts of operations, it is very important to carefully label the columns resulting from a subquery.

\begin{lstlisting}
# Count how many declared and undeclared majors there are
# The subquery changes NULL values in MajorID to 'Undeclared' and
#    non-NULL to 'Declared'.
>>> cur.execute("SELECT majorstatus, COUNT(*) AS majorcount "
...             "FROM ( "                                   # Begin subquery.
...                 "SELECT StudentName, CASE "
...                 "WHEN MajorID IS NULL THEN 'Undeclared' "
...                 "ELSE 'Declared' END AS majorstatus "
...                 "FROM StudentInfo) "                    # End subquery.
...             "GROUP BY majorstatus "
...             "ORDER BY majorcount DESC;").fetchall()
[('Declared', 7), ('Undeclared', 3)]
\end{lstlisting}

Subqueries can also be joined with other tables, as in the following example.
Note also that a subquery can be used to create numerical data out of non-numerical data, which can then be passed into any of the aggregate functions.
\begin{lstlisting}
# Find the proportion of classes each student has an A+, A, or A- in.
# The inner query creates a column 'gradeisa' which is 1 if the student's grade
#    is A+, A, or A-, and 0 otherwise.
>>> cur.execute("SELECT SI.StudentName, AVG(SG.gradeisa) "
...             "FROM ("
...                 "SELECT StudentID, CASE Grade "
...                     "WHEN 'A+' THEN 1 "
...                     "WHEN 'A' THEN 1 "
...                     "WHEN 'A-' THEN 1 "
...                     "ELSE 0 END AS gradeisa "
...                 "FROM StudentGrades) AS SG "
...             "INNER JOIN StudentInfo AS SI "
...             "ON SG.StudentID == SI.StudentID "
...             "GROUP BY SG.StudentID;").fetchall()
[('Cameron Kim', 0.3333333333333333),
 ('Kristopher Tran', 0.5),
 ('Alfonso Phelps', 0.0),
 ('Cassandra Holland', 0.5),
 ('Michelle Fernandez', 0.0),
 ('Roberta Cook', 0.0),
 ('Sammy Burke', 1.0),
 ('Gilbert Chapman', 0.6666666666666666),
 ('Mercedes Hall', 0.3333333333333333),
 ('Rene Cross', 1.0)]
\end{lstlisting}

\begin{problem} % Average GPA.
Write a function that accepts the name of a database file.
Query the database for tuples of the form \li{(StudentName, N, GPA)} where \li{N} is the number of courses that the specified student is enrolled in and \li{GPA} is their grade point average based on the following point system:

\begin{center}
\begin{tabular}{lclclcl}
A+, A   = 4.0 & & B   = 3.0 & & C   = 2.0 & & D   = 1.0 \\
    A-- = 3.7 & & B-- = 2.7 & & C-- = 1.7 & & D-- = 0.7 \\
    B+  = 3.4 & & C+  = 2.4 & & D+  = 1.4 & &
\end{tabular}
\end{center}
Order the results from greatest GPA to least.
\end{problem}

\begin{comment}

\newpage

\section*{Additional Material}

\subsection*{Database Normalization}

When creating a database, once must carefully consider its structure.
When a database is organized in such a way as to avoid redunancy, it is \emph{normalized}.

Consider a company whose database has a table that stores customer contact information and a table that contains sales data.
This company wants to track who buys what products.
To do so, they store all the contact information of a particular buyer with every item they purchased.
Now two tables store the customer contact information.
If they need to update a customer's phone number, they have to update two tables.
While that may not be bad for small databases, larger databases would be near impossible to update correctly.
The idea of normalizing this database would be to store all customer contact information in one place.
All other tables that might need a customer's name, phone number, or address would reference that contact information table.
When an update needs to be performed, only the contact information table would be updated.
Any table that now references this information would automatically be up to date.

To properly normalize a database, we must consider the types of relations discussed in the below subsections.

\subsection*{One to One}

A one-to-one relation is between one record and one other record.
An example of this relationship is a U.S. citizen and his or her Social Security number.
There is only one social security member per U.S. citizen and visa versa.
This is the simplest relation to model and is often done using one table

\subsection*{One to Many}

A one-to-many relation occurs when one record can link to many records, and those records only link back to the original record.
Consider the example of a mother and her children.
A mother can have multiple children, but a child can only have one birth mother.
This relationship must be modeled with at least two tables.

\subsection*{Many to Many}

A many-to-many relation occurs when a record can link to multiple other records and vice versa.
For example, a textbook can be written by multiple authors.
On the other hand, each author can write multiple textbooks.
This relationship requires at least three tables.

Continuing with the example, one table contains author data.
Each author is given a unique ID, or \emph{primary key}.
Another table contains textbook data.
Similarly, each textbook is given an ID in the form of a primary key.
A third table, called the \emph{junction table}, serves as a map between the primary keys of the other two tables.

\subsection*{More Joins} % ----------------------------------------------------

A \emph{cross join} is a Cartesian product on a set of relations.
Cross joins \textbf{union} two or more tables by combining rows in every way possible and filling in empty entries with \lsql{NULL} values.
A cross join should only be used on small tables or with a strict condition, since the result set will likely be very large.
% This join is equivalent to selecting both tables, and the sql command \lsql{FULL OUTER JOIN} which is not currently supported by Python's sqlite package.

% TODO: this is a crappy example because it has a WHERE...
\begin{lstlisting}
# Join together multiple tables with multiple conditionals.
>>> cur.execute("SELECT * "
...             "FROM MajorInfo AS MI "
...                 "CROSS JOIN CourseInfo CI, StudentInfo AS SI "
...              "WHERE MI.MajorID == CI.CourseID "
...                 "AND SI.StudentName == 'Mercedes Hall';").fetchall()
[(1, 'Math', 1, 'Calculus', 821568627, 'Mercedes Hall', 'NULL'),
 (2, 'Science', 2, 'English', 821568627, 'Mercedes Hall', 'NULL'),
 (3, 'Writing', 3, 'Pottery', 821568627, 'Mercedes Hall', 'NULL'),
 (4, 'Art', 4, 'History', 821568627, 'Mercedes Hall', 'NULL')]

# Cross join is equivalent to selecting everything from all tables.
>>> cur.execute("SELECT * FROM MajorInfo MI, CourseInfo CI WHERE MI.MajorID \
... = CI.CourseID;")
>>> cur.fetchall()
[(1, 'Math', 1, 'Calculus'),
 (2, 'Science', 2, 'English'),
 (3, 'Writing', 3, 'Pottery'),
 (4, 'Art', 4, 'History')]
\end{lstlisting}

% TODO: SELF JOIN?

% A left/right outer join selects all rows from just one relation, while a \emph{cross join} (also known as a \emph{full outer join}) selects all rows from all relations.

\end{comment}