%\documentclass[a4paper,10pt]{article}
\documentclass{article}
\usepackage{epsfig,relsize,moreverb,A4}
%\addtolength{\textwidth}{1cm}

\newcommand{\beq}{\begin{equation}}
\newcommand{\eeq}{\end{equation}}
\newcommand{\beqa}{\begin{eqnarray}}
\newcommand{\eeqa}{\end{eqnarray}}
\newcommand{\beqan}{\begin{eqnarray*}}
\newcommand{\eeqan}{\end{eqnarray*}}
%
\newcommand{\ben}{\begin{enumerate}}
\newcommand{\een}{\end{enumerate}}
\newcommand{\bit}{\begin{itemize}}
\newcommand{\eit}{\end{itemize}}

\newcommand{\bfx}{\mathbfx{x}}

\newcommand{\defbox}[1]{
  \begin{center}
    \fbox{\parbox{0.9\textwidth}{
      \begin{center}
        \begin{minipage}{0.8\textwidth}
          #1
        \end{minipage}
      \end{center} }
    }
  \end{center}
}

\newcommand{\labelP}[1]{\label{##1}}
\newcommand{\ep}{\thinspace . }
\newcommand{\refeq}[1]{(\ref{#1})}
\newcommand{\ec}{, }

\newcommand{\mathbfx}[1]{{\mbox{\boldmath $#1$}}}
\newcommand{\mathbfxg}[1]{{\mbox{\boldmath $#1$}}}

\newcommand{\n}{\mathbfx{n}}

\newcommand{\emp}[1]{{\smaller\texttt{#1}}}

\newcommand{\algc}[1]{{\sc\footnotesize #1}}
% newalg is not used
\newcommand{\newalg}{
\usepackage{float}
% declare algorithm float:
\floatstyle{boxed}
\newfloat{algorithm}{thp}{loa}
\floatname{algorithm}{Algorithm}
\newcommand{\algor}[2]{\begin{algorithm}\caption{##1}
\begin{tabbing}
\hspace*{0.5cm}\= \hspace{0.5cm} \= \hspace{0.5cm} \= \hspace{0.5cm} \= \hspace{0.5cm} \= \hspace{0.5cm}
\= \kill ##2 \end{tabbing}\end{algorithm}}
}
\newcommand{\oldalg}{
%\newtheorem{algorithm}[theorem]{Algorithm}{\bfseries}{\rm}
%\newtheorem{algorithm}[chapter]{Algorithm}{\bfseries}{\rm}
\newtheorem{algorithm}{Algorithm}{\bfseries}{\rm}
\newcommand{\algor}[2]{\begin{algorithm} \defbox{ \emph{##1}
\begin{tabbing}
\hspace*{0.5cm}\= \hspace{0.5cm} \= \hspace{0.5cm} \= \hspace{0.5cm} \= \hspace{0.5cm}
\= \kill ##2 \end{tabbing} }\end{algorithm}}
}
\oldalg

% end of hpl-dillemikk
% begin of vetler-magic
\newcommand{\snippet}[1]{\footnotesize \texttt{#1} \normalsize}
% \emp looks nicer...?

\begin{document}

\author{Hans Petter Langtangen\thanks{Dept.~of Scientific Computing,
Simula Research Laboratory, and Dept.~of Informatics, University of Oslo.
\texttt{hpl@ifi.uio.no}.}}
\title{The 2D Wave Equation}
\date{\today}
\maketitle
\tableofcontents

\section{Introduction}
Simulation of physical phenomena often involves
numerical solution of partial differential equations (PDEs).
This is particularly the case in mechanics, geophysics, astrophysics,
and many engineering disciplines,
as well as in parts of physics, geology, biology, and finance.
When discussing high-performance computing aspects of numerical
solution of PDEs, it is convenient to work with a model problem with
as simple mathematics and numerics as possible, yet with physical
and large-scale simulation relevance.
The 2D or 3D standard, linear wave equation constitute such a problem.
The wave equation is one of the simplest PDEs in physics from
a mathematical point of view, and it can be solved by the simplest
numerical methods for PDEs, namely explicit finite difference schemes.
Fortunately, the resulting simulation code can (especially with
a proper choice of boundary conditions) contain important constructions
for exemplifying basic high-performance computing aspects, e.g.,
\begin{itemize}
\item large, nested do-loops
\item if-tests and function calls inside/outside loops
\item 5- or 7-point stensils and cache problems
\end{itemize}
This note describes in detail the description of an appropriate model
problem to be used as a main example in a high-performance computing course.

\begin{quote}
{\bf NOTE: You do not need to read or understand the mathematics in
Section~\ref{math} to carry out the exercises. Hence, you
can jump directly to Section~\ref{F77impl}, which describes a sample
code in Fortran 77, and then you can start with the exercises
in Section~\ref{exer}. If you are an experienced programmer, you will
understand the basics of the exercises and the code without
understanding the mathematical details behind the code.}
\end{quote}

\section{Problem description}
\label{math}

\subsection{The mathematical model}
Wave propagation is a physical process of fundamental importance; just think
of water waves, sound waves for oral communication, and
electromagnetic waves for transmission of light as well as radio and TV
programs. The simplest mathematical model for waves consists of a
partial differential equation, to so-called \emph{wave equation},
\beq
{\partial^2u\over\partial t^2} = \nabla\cdot [ \lambda \nabla{u} ]
\label{Wave3D:eq1} \ep \eeq
This equation for the unknown function $u$
can describe, for example, water surface waves, electromagnetic
waves, earthquake waves, waves on a drum, or sound waves in air.
When it comes to water and earthquake waves, equation \refeq{Wave3D:eq1}
is actually an approximation to the real-world physics, valid under
certain assumptions; the water waves need to be long compared to the depth
of water. This is satisfied for destructive flood waves generated by
natural hazards such as avalanches, slides, earthquakes.

Understanding the right-hand side of equation \refeq{Wave3D:eq1}
requires familiarity with vector calculus. We can write out the term
$\nabla\cdot [\lambda \nabla{u} ]$ explicitly for a two-dimensional
application of the wave equation, i.e., when $u$ depends on two
spatial coordinates, $x$ and $y$:
\beq
{\partial^2u\over\partial t^2} =
{\partial\over\partial x}\left( \lambda(x,y) {\partial u\over\partial x}\right) +
{\partial\over\partial y}\left( \lambda(x,y) {\partial u\over\partial y}\right)
\label{Wave2D:eq1} \ep \eeq
The parameter $\sqrt{\lambda}$
represents the velocity of the waves and depends in general
on the properties of the medium in which wave the propagation takes place.
In the case of water waves, $\lambda (x,y) = gH(x,y)$, where $g$ is the
acceleration of gravity and $H(x,y)$ is the still-water depth.
The function $u(x,y,t)$ is the elevation of the surface, and $u=0$ corresponds
to still water. If $H$ is constant, $\lambda$ is constant and can be moved
outside the spatial derivative. This results in a simpler equation,
\beq {\partial^2u\over\partial t^2} = \lambda\left(
{\partial^2u\over\partial x^2} + {\partial^2u\over\partial y^2}\right)\ep
\label{Wave2D:cH}
\eeq
The right-hand side is the Laplacian of $u$, often denoted by $\nabla^2 u$.
Hence, an alternative form of \refeq{Wave2D:cH} is
\[ {\partial^2u\over\partial t^2} = \lambda\nabla^2 u\ep\]
Some readers might recall this wave equation from other contexts.

Equation \refeq{Wave2D:eq1} is defined in a \emph{domain} $\Omega$, i.e., a
two-dimensional area in which we want to simulate the wave propagation.
This can be a harbor or just a part of a big ocean basin.
The boundary of the domain is often denoted by $\partial\Omega$.

There are an infinite number of solutions to \ref{Wave2D:eq1} unless we
specify additional conditions. These conditions consists of
\emph{boundary conditions}
 and \emph{initial conditions}.
For the water wave application,
\beq
{\partial u\over\partial n}\equiv \nabla u\cdot\mathbfx{n} = 0
\label{Wave2D:bc}
\eeq
is an appropriate boundary condition
that we will make use of.
The vector $\mathbfx{n}$ is an outward unit normal to the boundary $\partial\Omega$.
The condition \refeq{Wave2D:bc} is to be applied at every point
on $\partial\Omega$.
If the boundary is the line $x=0$, $\mathbfx{n}$ is a unit vector along
the $x$ axis, and $\partial u /\partial n$ simply becomes $\partial u/
\partial x$.
Physically, equation \refeq{Wave2D:bc} tells that waves are perfectly reflected
from the boundary, which is relevant if the boundary is, e.g.,
a coastline with steep hills.
A simpler boundary condition is
\beq
u = 0 \ep
\eeq
Initial conditions specify the shape of $u$ at initial time $t=0$ throughout
the domain $\Omega$:
\beq
u(x,y,0)=I(x,y)\ep
\eeq
The physical interpretation of this equation is that
the shape of the water surface is described by the function
$I(x,y)$. In addition to this condition, we need to specify
the time derivative of $u$ at $t=0$ as well. This time derivative is, from a
physical point of view, the velocity of the water surface at $t=0$.
For simplicity, we assume that the surface is at rest (zero velocity):
\beq
{\partial\over\partial t} u(x,y,0) =0\ep
\eeq
The assumption of zero velocity can easily be relaxed.

The partial differential equations with the boundary and initial conditions
constitute the complete mathematical model for simulating
water waves\footnote{The term ``water waves'' refers in this document
to water waves where the typical wave length is much greater than the
water depth. If this assumption is violated, the solution to the
wave equation will be qualitatively wrong.}.
In particular, we need to specify the depth function $H(x,y)$ and
the initial surface shape $I(x,y)$ before the simulation can take place.

\subsection{The numerical method}
\label{sec2num}
In the old days, clever mathematicians tried to find solutions to
equations like \refeq{Wave2D:eq1} by paper and pencil.
This is an extremely challenging task and in general impossible for a
depth function $H(x,y)$ taken from a sea map.
Solution in terms of a computer is, however, quite simple.
A fairly short program is capable of calculating $u(x,y,t)$ with
sufficient accuracy in seconds or minutes for any choice of $H(x,y)$ and
initial shape $I(x,y)$.

The numerical solution method to partial differential equations to be
used here is based on \emph{finite differences}.
Finite difference methods are quite easy to explain and program if
the domain $\Omega$ has a rectangular shape, which we will
assume in the rest of the document. We write
\[ \Omega = [x_{\min},x_{\max}]\times [y_{\min},y_{\max}] \ep \]
The finite difference method is based on some fundamental ideas:
\begin{enumerate}
\item The domain $\Omega$ is represented by a \emph{grid}, which is
a rectangular set of points $(x_i,y_j)$:
\beq
x_i = x_{\min} + (i-1)\Delta x,\quad y_j = y_{\min} + (j-1)\Delta y,
\eeq
for $i=1,\ldots, n_x$, and $j=1,\ldots,n_y$. The points form a rectangular
array with equal spacings $\Delta x$ and $\Delta y$ between the points,
see figure~\ref{gridfig} (in this figure, $\Delta x = \Delta y$ but
in general we can have $\Delta x \neq \Delta y$).
The value of $n_x$ and $n_y$ in figure~\ref{gridfig} is 9 and 6,
respectively. The size the grid spacings is determined by
\[ \Delta x = {x_{\max} - x_{\min}\over n_x -1},
\quad \Delta x = {y_{\max} - y_{\min}\over n_y -1}\ep \]

\item Also in time a grid is introduced:
\[ t_0=0, \ t_1=\Delta t,\ t_2=2\Delta t,\ t_3=3\Delta t,\ldots\]
or expressed in terms of an index $\ell$: $t_{\ell}=\ell\Delta t$,
$\ell = 0,1,\ldots$

\item We compute numerical approximations to $u(x_i,y_j,t_{\ell})$,
i.e., $u$ at the space-time grid points. The approximations are denoted
by $u_{i,j}^\ell$.

\item The governing partial differential equation \refeq{Wave2D:eq1}
is supposed to be satisified at all space-time points $(x_i,y_j,t_\ell)$,
$i=1,\ldots,n_x$, $j=1,\ldots,n_y$, $\ell =0,1,\ldots$.
This is an approximation as equation \refeq{Wave2D:eq1} in the
mathematical model is to be fulfilled at all the inifinte number of points
in $\Omega$ and for all $t>0$.

\item Derivatives in the partial differential equation is replaced by
following finite difference approximations:
\beqa
{\partial^2\over\partial t^2} u(x_i,y_j,t_\ell)
&\approx& { u_{i,j}^{\ell -1} -2u_{i,j}^\ell
+ u_{i,j}^{\ell +1}\over\Delta t},\\
{\partial\over\partial x}\left( \lambda (x_i,y_j)
{\partial\over\partial x} u(x_i,y_j,t_\ell)\right) &\approx &
{1\over\Delta x}\left(
\lambda_{i+{1\over2},j}{{u}_{i+1,j}^\ell-{u}_{i,j}^\ell\over\Delta x}
-
\lambda_{i-{1\over2},j}{{u}_{i,j}^\ell-{u}_{i-1,j}^\ell\over\Delta x}
\right) ,\\
{\partial\over\partial y}\left( \lambda (x_i,y_j)
{\partial\over\partial y} u(x_i,y_j,t_\ell)\right) &\approx &
{1\over\Delta y}\left(
\lambda_{i,j+{1\over2}}{{u}_{i,j+1}^\ell-{u}_{i,j}^\ell\over\Delta y}
-
\lambda_{i,j-{1\over2}}{{u}_{i,j}^\ell-{u}_{i,j-1}^\ell\over\Delta y}
\right) \ep
\eeqa
When $\lambda$ is constant, the formulas simplify to (hopefully) well-known
formulas for the second derivative:
\[ {\partial^2\over\partial x^2} u(x_i,y_j,t_\ell) \approx
{u_{i-1,j}^\ell - 2u_{i,j}^\ell + u_{i+1,j}^\ell\over\Delta x^2}\ep
\]
\item At each time level we assume that $u$ from previous time levels
are already computed, i.e., they are known values.
\end{enumerate}

\begin{figure}
\centerline{\protect{\psfig{figure=grid.xfig.eps,width=10cm}}}
  \caption{Example on a two-dimensional grid suitable for finite difference
           solution of partial differential equations. \label{gridfig}}
\end{figure}
Inserting the finite difference  approximations into the
wave equation results in what we refer to as the finite difference
equations or simply the discrete equations.
We make the important observation that there is only one
term involving time level $\ell + 1$ in the discrete equations.
Since we assume that everything on
the previous time levels $\ell$ and $\ell -1$ are known, there is actually
only one unknown value, $u_{i,j}^{\ell +1}$. We can solve the discrete
equations with respect to this value and get something like
\[ u_{i,j}^{\ell +1} = \hbox{known terms from previous time levels} \]
This means, algorithmically, that we can just run through all the grid
points $i=1,\ldots,n_x$, $j=1,\ldots,n_y$, and compute new approximations
to $u$ from a simple formula. It is common to classify such methods as
\emph{explicit} schemes.
For other partial differential equations and other finite difference
discretizations, new $u$ values at time level
$\ell +1$ will in general
be coupled with each other in a linear system of algebraic
equations (matrix system). This is referred to as an \emph{implicit}
scheme.

A notation, which is convenient when translating a computational
algorithm to program statements in a code, is to set
\[
u^+_{i,j}\equiv u^{\ell+1}_{i,j},\quad
u_{i,j}\equiv u^{\ell}_{i,j},\quad u^-_{i,j}\equiv
u^{\ell-1}_{i,j}\ep\]
We can now write the discrete equations compactly:
\beq
u_{i,j}^+ = 2u_{i,j} - u_{i,j} + [\triangle u]_{i,j}
\label{Wave2D:fd1}
\eeq
where
\beqa
[\triangle  u]_{i,j} &\equiv&
  \left({\Delta t\over\Delta x}\right)^2
  (\lambda_{i+{1\over2},j}(u_{i+1,j}-u_{i,j}) -
     \lambda_{i-{1\over2},j}(u_{i,j}-u_{i-1,j})) + \nonumber\\ &&
  \left({\Delta t\over\Delta y}\right)^2
  (\lambda_{i,j+{1\over2}}(u_{i,j+1}-u_{i,j}) -
     \lambda_{i,j-{1\over2}}(u_{i,j}-u_{i,j-1})) \ep
\label{Wave3D:eq3}
\eeqa
Assuming that $u_{i,j}$ and $u_{i,j}^-$ are known for all $(i,j)$ values
in the grid, equation \refeq{Wave2D:fd1} gives the new $u$ value at
time level $\ell +1$ at the grid point with index $i$ and $j$.

The ``geometry'' of \refeq{Wave2D:fd1} is interesting to observe: When
computing a new $u$ value at a spatial point with index $i$ and $j$,
we make use of previous $u$ values at this point and the four neighboring
points (west, east, north, and south). This is illustrated in
figure~\ref{grid2fig}.
\begin{figure}
\centerline{\protect{\psfig{figure=grid2.xfig.eps,width=10cm}}}
  \caption{A new $u$ value time level $\ell +1$ depends on
values at previous time levels $\ell$ and $\ell -1$ at the same point and
values at time level $\ell$ at the nieghboring points to the
north, south, east, and west.}
  \label{grid2fig}
\end{figure}
This observation, together with the fact that the order in which we
visit the spatial indices $i$ and $j$ in \refeq{Wave2D:fd1} is
irrelevant, forms the basis for constructing an algorithm for wave
simulation on parallel computers.

There is one immediate basic problem with \refeq{Wave2D:fd1}: If we apply
it to the first time level, $\ell =1$, it will involve $u^-_{i,j}$, i.e.,
$u$ values at time level $-1$ ($t=-\Delta t$), which are unknown to us.
The reason why we run into this problem is that we have not applied the
initial condition $\partial u/\partial t=0$ at $t=0$. A typical finite
difference approximation to this condition reads
\[ {\partial\over\partial t} u(x_i,y_j,0) \approx
{u_{i,j}^1 - u_{i,j}^{-1}\over 2\Delta t} = 0,
\]
which implies
\[ u_{i,j}^{-1} = u_{i,j}^1 \ep\]
Hence, we can just substitute $u^-_{i,j}$ by $u^+_{i,j}$ in
\refeq{Wave2D:fd1} at the first time level.
Alternatively, we can use \refeq{Wave2D:fd1} as it stands, even for the
first time level, if we just define $u$ at time level -1 to be
\[ u^{-1}_{i,j} = u_{i,j} +  {1\over 2}[\triangle  u]_{i,j} \ep \]
Another problem with \refeq{Wave2D:fd1} is that we have not incorporated
the boundary values. For simplicity, we assume that $u=0$ on the boundaries.
We can then just use the zero value when \refeq{Wave2D:fd1} involves
$u$ at a boundary.
The complete computational recipe is summarized in
algorithm \ref{Wave3D:algor1}.
\clearpage

\algor{Explicit scheme for the two-dimensional
wave equation with $u=0$ on the boundary.\label{Wave3D:algor1}}{
define $u^+_{i,j}$ , $u_{i,j}$ and $u^-_{i,j}$ to represent
$u^{\ell+1}_{i,j}$, $u^{\ell}_{i,j}$ and $u^{\ell-1}_{i,j}$, resp.\\
define $[\triangle  u]_{i,j}$ as in \refeq{Wave3D:eq3}\\
define $(i,j)\in\bar{\cal I}$ to be $i=1,\ldots,n_x$, $j=1,\ldots,n_y$\\
define $(i,j)\in{\cal I}$ to be $i=2,\ldots,n_x-1$, $j=2,\ldots,n_y-1$\\
set $u_{i,j}=0,\quad (i,j)\in\bar{\cal I}$\\
\algc{set the initial conditions:}\\
$u_{i,j} = I(x_i,y_j), \quad (i,j)\in{\cal I}$\\
\algc{define the value of the artificial quantity} $u^{-}_{i,j}$:\\
$ u^-_{i,j} = u_{i,j} +  {1\over 2}[\triangle  u]_{i,j},\quad
(i,j)\in{\cal I}$\\
$t=0$\\
while time $t \leq t_{\rm {stop}}$\\
\> $t \leftarrow t + \Delta t$\\
\> \algc{update all inner points:}\\
\> $u_{i,j}^+ = 2u_{i,j} - u^-_{i,j} + [\triangle  u]_{i,j},\quad
(i,j)\in{\cal I}$\\
\> \algc{initialize for next step:}\\
\> $u^-_{i,j}=u_{i,j},\quad u_{i,j}=u^+_{i,j},\quad (i,j)\in{\cal I}$
}

Notice that we do not explicitly set $u_{i,j}=0$ at the boundary.
Instead, we set $u_{i,j}=0$ initially and never touch the boundary values.

We remark that if $\lambda$ is constant and
$\Delta x = \Delta y = h$,
the numerical scheme simplifies greatly, and
the Laplace term $\nabla\cdot [ \lambda\nabla{u} ] = \lambda\nabla^2{u}$
takes the well-known discrete form \index{five-point Laplace operator}
\beq
[\triangle  u]_{i,j} = \lambda\left({\Delta t\over h}\right)^2\left(
-{u}_{i-1,j} - {u}_{i,j-1} - {u}_{i+1,j} - {u}_{i,j+1} +
4{u}_{i,j}\right)
\ep
\label{Heat1Dv:eq9}\eeq
This formula for approximating $\lambda\nabla^2u$ can be graphically
exposed as in figure \ref{Wave2D:5ptstar}.
\begin{figure}
\centerline{\protect{\psfig{figure=5pt-star.xfig.eps,width=5cm}}}
\caption{\label{Wave2D:5ptstar}
Illustration of the finite difference stencil for
approximating the Laplace operator $\nabla^2 u$ in a regular grid.}
\end{figure}
The circles denote the
points in the grid that are used in the approximation, and the
numbers reflect the weight of the point in the finite difference formula.
One often refers to such a graphical representation as a
\index{finite difference!five-point stencil}
{\em finite difference stencil} or a
\index{computational molecule}{\em computational molecule}.
In the more general case when $\lambda$ is not constant, the
same stencil arises, except that the weights are different.

Algorithm \ref{Wave3D:algor1} assumes that $u=0$ on the boundary.
We shall now demonstrate how to implement a more complicated boundary
condition, $\partial u/\partial n =0$, in the scheme.
The boundary condition is discretized
by a centered difference at the boundary. For example,
at the line $i=1$ we then
require
\[ {u_{2,j}^\ell - u_{0,j}^\ell\over\Delta x} =0\quad\Rightarrow
\quad u_{0,j}^\ell = u_{2,j}^\ell \ep \]
Notice that this involves
a fictitious  value $u_{0,j}^\ell$ outside the grid. Using the discrete
finite difference equation \refeq{Wave2D:fd1} at
the same boundary point, with  $u_{0,j}^\ell = u_{2,j}^\ell$ from the
boundary condition, enables elimination of the fictitious value.
The $[\triangle  u]_{i,j}$ operator is then modified to
\beqa && [\triangle  u]_{1,j: i-1\rightarrow i+1} \equiv
\left({\Delta t\over\Delta x}\right)^2
  (\lambda_{1+{1\over2},j}(u_{2,j}-u_{1,j}) -
     \lambda_{1-{1\over2},j}(u_{1,j}-u_{2,j})) + \nonumber\\  &&
  \left({\Delta t\over\Delta y}\right)^2
  (\lambda_{1,j+{1\over2}}(u_{1,j+1}-u_{1,j}) -
     \lambda_{1,j-{1\over2}}(u_{1,j}-u_{1,j-1})) ,\ j\neq 1,n_y\ep
\label{Wave3D:eq4}
\eeqa
At the boundary $i=n_x$ we would then
apply the modification $[\triangle  u]_{n_x,j: i+1\rightarrow i-1}$.
Similarly, for $j=1$ and $j=n_y$ we replace the original
$[\triangle  u]_{i,j}$ operator by
$[\triangle  u]_{i,1: j-1\rightarrow j+1}$ and
$[\triangle  u]_{i,n_y: j+1\rightarrow j-1}$, respectively.
The corner points of the grid require modification of both indices, for
example, the $i=j=1$ point leads to
$[\triangle  u]_{1,1: i-1\rightarrow i+1,j-1\rightarrow j+1}$.

Algorithm~\ref{Wave3D:algor2} precisely explains
the updating of internal and boundary points in terms of a function
$\mbox{WAVE}(u^+,u,u^-,a,b,c)$. As a special case, the call
$\mbox{WAVE}(u^+,u,u^-,1,1,1)$ reproduces the original finite difference
scheme with modifications due to homogeneous Neumann conditions
$\partial u /\partial n = 0$.

\algor{Basic finite difference updating formula for the two-dimensional wave equation
with ${\partial u\over\partial n}=0$ on the boundary.\label{Wave3D:algor2}}{
define $[\triangle  u]_{i,j}$ as in \refeq{Wave3D:eq3}\\
define $[\triangle  u]_{1,j: i-1\rightarrow i+1}$,
$[\triangle  u]_{n_x,j: i+1\rightarrow i-1}$,
$[\triangle  u]_{i,1: j-1\rightarrow j+1}$, \\
\> and
$[\triangle  u]_{i,n_y: j+1\rightarrow j-1}$ according to \refeq{Wave3D:eq4}\\
define $(i,j)\in{\cal I}$ to be $i=2,\ldots,n_x-1$, $j=2,\ldots,n_y-1$\\
define $u^+_{i,j}$ , $u_{i,j}$ and $u^-_{i,j}$ to represent
$u^{\ell+1}_{i,j}$, $u^{\ell}_{i,j}$ and $u^{\ell-1}_{i,j}$, resp.\\
function $\mbox{WAVE}(u^+,u,u^-,a,b,c)$:\\
\>\algc{update all inner points:}\\
\>  $u_{i,j}^+ = 2au_{i,j} - bu^-_{i,j} + c[\triangle  u]_{i,j},\quad
(i,j)\in{\cal I}$\\
\> \algc{update boundary points:}\\
\> $i=1$;
 $u_{i,j}^+ = 2au_{i,j} - bu^-_{i,j} + c[\triangle  u]_{i,j: i-1\rightarrow i+1},\ j=2,\ldots,n_y-1$\\
\> $i=n_x$;
 $u_{i,j}^+ = 2au_{i,j} - bu^-_{i,j} + c[\triangle  u]_{i,j: i+1\rightarrow i-1},\ j=2,\ldots,n_y-1$\\
\> $j=1$;
 $u_{i,j}^+ = 2au_{i,j} - bu^-_{i,j} + c[\triangle  u]_{i,j: j-1\rightarrow j+1},\ i=2,\ldots,n_x-1$\\
\> $j=n_y$;
 $u_{i,j}^+ = 2au_{i,j} - bu^-_{i,j} + c[\triangle  u]_{i,j: j-1\rightarrow j+1},\ i=2,\ldots,n_x-1$\\
\> \algc{update corner points on the boundary:}\\
\> $i=1$, $j=1$;
 $u_{i,j}^+ = 2au_{i,j} - bu^-_{i,j} + c[\triangle  u]_{i,j: i-1\rightarrow i+1, j-1\rightarrow j+1}$\\
\> $i=n_x$, $j=1$;
 $u_{i,j}^+ = 2au_{i,j} - bu^-_{i,j} + c[\triangle  u]_{i,j: i+1\rightarrow i-1, j-1\rightarrow j+1}$\\
\> $i=1$, $j=n_y$;
 $u_{i,j}^+ = 2au_{i,j} - bu^-_{i,j} + c[\triangle  u]_{i,j: i-1\rightarrow i+1, j+1\rightarrow j-1}$\\
\> $i=n_x$, $j=n_y$;
 $u_{i,j}^+ = 2au_{i,j} - bu^-_{i,j} + c[\triangle  u]_{i,j: i+1\rightarrow i-1, j+1\rightarrow j-1}$
}

Using the WAVE function from Algorithm~\ref{Wave3D:algor3}, we can
devise a compact description of all the computational tasks for the
discrete two-dimensional wave equation with homogenous Neumann conditions. The
steps are listed in Algorithm~\ref{Wave3D:algor3}.
\clearpage

\algor{Complete scheme for the two-dimensional wave equation with ${\partial u\over\partial n}=0$ on the boundary.\label{Wave3D:algor3}}{
define quantities in Algorithm~\ref{Wave3D:algor2}\\
define $(i,j)\in\bar{\cal I}$ to be $i=1,\ldots,n_x$, $j=1,\ldots,n_y$\\
set $u_{i,j}=0,\quad (i,j)\in\bar{\cal I}$\\
\algc{set the initial conditions:}\\
$u_{i,j} = I(x_i,y_j), \quad (i,j)\in{\cal I}$\\
\algc{define the value of the artificial quantity} $u^{-}_{i,j}$:\\
$\mbox{WAVE}(u^-,u,u^-,0.5,0,0.5)$\\
$t=0$\\
while time $t \leq t_{\rm {stop}}$\\
\> $t \leftarrow t + \Delta t$\\
\> \algc{update all points:}\\
\> $\mbox{WAVE}(u^+,u,u^-,1,1,1)$\\
\> \algc{initialize for next step:}\\
\> $u^-_{i,j}=u_{i,j},\quad u_{i,j}=u^+_{i,j},\quad (i,j)\in{\cal I}$
}

At this point a newcomer to partial differential equations and finite
difference schmes might be overwhelmed and confused by all the
indices and book-keeping in these algorithms.
Surely, the amount of details is significant, and hands-on work with
the formulas, either with pencil and paper or in a computer code, is
necessary to become familiar with the ideas and the notation.
We should, however, point out that the underlying ideas are quite simple;
the whole business here is mostly a matter of becoming familiar with
the index notation. A very good exercise for repeating the material and
working towards a better understanding is to devise an algorithm for
the case where $u=0$ at the whole boundary, except the line
$x=x_{\max}$. Readers who are experienced with finite difference schemes
will do this in a minute, while newcomers need to carefully go through
the previous pages and find the right elements that are needed in
the algorithm for this slightly perturbed problem.

\subsection{Implementation of the algorithm}
\label{sec2impl}
The basic operations in algorithm~\ref{Wave3D:algor3} consist of
running through all grid points,
\[ i=1,\ldots,n_x,\quad j=1,\ldots,n_y\]
and computing some formula at each grid point. The formula typically involves
$u^+_{i,j}$, $u_{i,j}$, and $u^-_{i,j}$ values.
If we store the latter quantities in three two-dimensional arrays,
where $i$ and $j$ become indices in the array,
the basic operation
consists of two nested loops and manipulation on the arrays inside
the loop. This type of operation can be programmed in almost any
computer language, since two-dimensional arrays are data structures
found in most languages.

\section{Implementation in Fortran 77}
\label{F77impl}
A rough implementation of the numerical problem described in the
previous section has been realized in Fortran 77.
It is not necessary to be familiar with the numerics in detail in
order to play around with the code.
We remark that the program applies a rough first-order approximation
to the initial condition $\partial u/\partial t=0$, the other
numerical expressions are as outlined in the section \ref{sec2impl}.

The program consists of two files, \emp{main.f} and \emp{F77WAVE.fcp}.
The \emp{main.f} file contains
\bit
\item a main program, where data structures are declared and some
constants set, before the main routine \emp{timeloop} is called,
\item a function \emp{h} implementing an expression for the
$\lambda (x,y)$ function (here just 1.0 is returned),
\item a function \emp{bell} for defining the initial
condition (a Gaussian bell),
\item a function \emp{setIC} for initializing arrays based on
the problem's initial condition,
\item a function \emp{timeloop}, which runs the finite difference
scheme for a given number of time steps,
\item a function \emp{dump} for dumping data to file.
\eit
The \emp{timeloop} function calls another function \emp{F77WAVE}
for implementing the WAVE function defined in section \ref{sec2impl}.
The WAVE function is just the core finite difference scheme for
the wave equation, i.e., the ``heart'' of this simulation code.
The \emp{F77WAVE} function is written in Fotran 77, but we have used
a C preprocessor macro to define an inline function \emp{LaplaceU},
corresponding to $[\triangle u]_{i,j}$ in section \ref{sec2num}:

{
\renewcommand{\baselinestretch}{0.85}\footnotesize
\begin{verbatimtab}[8]
#define LaplaceU(u,lambda,i,j,im1,ip1,jm1,jp1) \
  (dt*dt)/(dx*dx)*\
  ( 0.5*(lambda(ip1,j  )+lambda(i  ,j  ))*(u(ip1,j  )-u(i  ,j  )) \
   -0.5*(lambda(i  ,j  )+lambda(im1,j  ))*(u(i  ,j  )-u(im1,j  )))\
 +(dt*dt)/(dy*dy)*\
  ( 0.5*(lambda(i  ,jp1)+lambda(i  ,j  ))*(u(i  ,jp1)-u(i  ,j  )) \
   -0.5*(lambda(i  ,j  )+lambda(i  ,jm1))*(u(i  ,j  )-u(i  ,jm1)))
\end{verbatimtab}
}

\noindent The equivalent mathematical expression is
\beqan
[\triangle  u]_{i,j} &\equiv&
  \left({\Delta t\over\Delta x}\right)^2
  (\lambda_{i+{1\over2},j}(u_{i+1,j}-u_{i,j}) -
     \lambda_{i-{1\over2},j}(u_{i,j}-u_{i-1,j})) + \\ &&
  \left({\Delta t\over\Delta y}\right)^2
  (\lambda_{i,j+{1\over2}}(u_{i,j+1}-u_{i,j}) -
     \lambda_{i,j-{1\over2}}(u_{i,j}-u_{i,j-1})) \ep
\eeqan
The symbols \emp{ip1} (``\emp{i} plus 1''), \emp{im1} (``\emp{i} minus 1''),
and so on corresponds to $i+1$, $i-1$ etc.

A special script, \emp{fcpp.py}, translates Fortran 77 code with C macros
into plain Fortran 77. The resulting file is named \emp{F77WAVE.f}.
With the \emp{LaplaceU} macro the code becomes much more readable, and it
is easier to play around with modifications of the implementation of
the finite difference scheme.

The script \emp{make.sh} runs \emp{fcpp.py} and compiles the application
with profiling turned on (plain Unix \emp{gprof}).

When working with high-performance computing issues related to this
wave simulation code, we shall modify the \emp{F77WAVE.fcp} file only.
The core of this file is the WAVE function:

{
\renewcommand{\baselinestretch}{0.85}\footnotesize
\begin{verbatimtab}[8]
      SUBROUTINE F77WAVE(up, u, um, lambda, a, b, c, nx, ny,
     >                   dt, dx, dy)
      IMPLICIT LOGICAL (A-Z)
      INTEGER nx, ny
      REAL*8 up(nx,ny), u(nx,ny), um(nx,ny), lambda(nx,ny)
      REAL*8 a, b, c
      REAL*8 dt, dx, dy
      INTEGER i,j

      DO 20 j = 2, ny-1
         DO 10 i = 2, nx-1
            up(i,j) = a*2*u(i,j) - b*um(i,j) +
     >          c*LaplaceU(u,lambda,i,j,i-1,i+1,j-1,j+1)
 10      CONTINUE
 20   CONTINUE

C     Boundary points:

      i=1
      DO 30 j = 2, ny-1
         up(i,j) = a*2*u(i,j) - b*um(i,j) +
     >      c*LaplaceU(u,lambda,i,j,i+1,i+1,j-1,j+1)
 30   CONTINUE

      i=nx
      DO 40 j = 2, ny-1
         up(i,j) = a*2*u(i,j) - b*um(i,j) +
     >      c*LaplaceU(u,lambda,i,j,i-1,i-1,j-1,j+1)
 40   CONTINUE

      j=1
      DO 50 i = 2, nx-1
         up(i,j) = a*2*u(i,j) - b*um(i,j) +
     >      c*LaplaceU(u,lambda,i,j,i-1,i+1,j+1,j+1)
 50   CONTINUE

      j=ny
      DO 60 i = 2, nx-1
         up(i,j) = a*2*u(i,j) - b*um(i,j) +
     >      c*LaplaceU(u,lambda,i,j,i-1,i+1,j-1,j-1)
 60   CONTINUE

C     Corners:
      i=1
      j=1
      up(i,j) = a*2*u(i,j) - b*um(i,j) +
     >   c*LaplaceU(u,lambda,i,j,i+1,i+1,j+1,j+1)

      i=nx
      j=1
      up(i,j) = a*2*u(i,j) - b*um(i,j) +
     >   c*LaplaceU(u,lambda,i,j,i-1,i-1,j+1,j+1)

      i=1
      j=ny
      up(i,j) = a*2*u(i,j) - b*um(i,j) +
     >   c*LaplaceU(u,lambda,i,j,i+1,i+1,j-1,j-1)

      i=nx
      j=ny
      up(i,j) = a*2*u(i,j) - b*um(i,j) +
     >   c*LaplaceU(u,lambda,i,j,i-1,i-1,j-1,j-1)
      RETURN
      END
\end{verbatimtab}
}

\noindent In the implementation we have paid attention to constructing loops
with the first index as the fastest index. This ensures that arrays
are accessed in the way they are stored in Fortran.
We have also treated the boundary points in separate loops.
In some of the first exercises you are asked to rewrite the code in
less efficient ways and measure the efficiency.

\section{Exercises}
\label{exer}

\subsection*{CPU-Time Measurements}
In most of the exercises below, the purpose is to perform some actions and
measure the impact on the CPU time.
The CPU time can be measure in different ways. The simplest is to
use the program (or built-in shell command) \emp{time}:

{
\begin{quote}
\renewcommand{\baselinestretch}{0.85}\footnotesize
\begin{verbatimtab}[8]
time app
\end{verbatimtab}
\end{quote}
}

\noindent
The output from \emp{time} when running the program \emp{app} may look
as follows:

{
\begin{quote}
\renewcommand{\baselinestretch}{0.85}\footnotesize
\begin{verbatimtab}[8]
real    0m4.314s
user    0m3.950s
sys     0m0.020s
\end{verbatimtab}
\end{quote}
}

\noindent
Here, \emp{real} is elapsed time (the time you have waited for the program
to finish),
\emp{user} is user time (e.g.,
arithmetics in the code), and
\emp{sys} is system time (e.g., I/O in the code).
The CPU time is the sum of the user time and
the system time.

The preferred way to measure CPU time in these exercises is to run
\emp{prof} or \emp{gprof}. With \emp{gprof}, you will find the
CPU time at the last line of the \emp{cumulative seconds} column
in the table of the CPU-time consumption of the various functions.
This command picks out the relevant number:

{
\begin{quote}
\renewcommand{\baselinestretch}{0.85}\footnotesize
\begin{verbatimtab}[8]
gprof app | perl -ne 'if (/MAIN__$/)
                      { $cpu=(split /\s+/)[2]; print "CPU=$cpu\n"; }'
\end{verbatimtab}
\end{quote}
}

\noindent
This command is available as the script \emp{cpu.sh}.
To display the table of CPU-time consumption of the various functions,
you can type

{
\begin{quote}
\renewcommand{\baselinestretch}{0.85}\footnotesize
\begin{verbatimtab}[8]
gprof app | head -12
\end{verbatimtab}
\end{quote}
}

\noindent
which displays the first 12 lines of the \emp{gprof} output.

If you want to have a look at the solution produced by the
Fortran 77 program, you can run the script \emp{Verify/test1.sh}.
The plotting requires that you have the X11 plotting program
\emp{plotmtv} installed.

\subsection*{Exercises}
\ben
\item
\emph{Determine the appropriate problem size.}
CPU-time measurements are not reliable unless they last for some
(say at least 10) seconds.
In the present code example, the grid size and the number of
time steps govern the CPU time. We have fixed the number of time steps
at 40, so the parameter to tune is the number of grid points in each
spatial direction. This parameter is set in the top of the \emp{main.f.orig}
file:

{
\begin{quote}
\renewcommand{\baselinestretch}{0.85}\footnotesize
\begin{verbatimtab}[8]
      PARAMETER (n=501)
\end{verbatimtab}
\end{quote}
}

\noindent
Compile the code by

{
\begin{quote}
\renewcommand{\baselinestretch}{0.85}\footnotesize
\begin{verbatimtab}[8]
make.sh
\end{verbatimtab}
\end{quote}
}

\noindent
and run it:

{
\begin{quote}
\renewcommand{\baselinestretch}{0.85}\footnotesize
\begin{verbatimtab}[8]
time app
\end{verbatimtab}
\end{quote}
}

\noindent
Adjust the value of \emp{n} in the \emp{PARAMETER} statement until you
have a CPU time between 10 and 30 seconds.

\item
\emph{Computations with and without I/O.}
Measure the CPU time of the original implementation of the
wave simulation code. Then ``uncomment'' the calls to \emp{dump}, i.e.,
write the solution to file at every time step.
Compile and measure the CPU time of a run.
In the rest of the exercises, we shall \emph{not} write results to
file so make sure you insert the comment again before proceeding.

\item
\emph{Function calls inside loops.}
The $\lambda (x,y)$ coefficient in the wave equation is in our implementation
stored in an array \emp{lambda}. Alternatively, we could call a function,
here \emp{h(x,y)}. Modify the \emp{LaplaceU} macro to call \emp{h} instead
of using \emp{lambda}. First try to replace \emp{lambda} by \emp{h(0,0)}
(take a copy of the original \emp{F77WAVE.fcp} file!):

{
\renewcommand{\baselinestretch}{0.85}\footnotesize
\begin{verbatimtab}[8]
#define LaplaceU(u,lambda,i,j,im1,ip1,jm1,jp1) \
  (dt*dt)/(dx*dx)*\
  ( 0.5*(h(0,0) + h(0,0))*(u(ip1,j  )-u(i  ,j  )) \
   -0.5*(h(0,0) + h(0,0))*(u(i  ,j  )-u(im1,j  )))\
 +(dt*dt)/(dy*dy)*\
  ( 0.5*(h(0,0) + h(0,0))*(u(i  ,jp1)-u(i  ,j  )) \
   -0.5*(h(0,0) + h(0,0))*(u(i  ,j  )-u(i  ,jm1)))
\end{verbatimtab}
}

\noindent In addition you need to declare \emp{h} in the \emp{F77WAVE} function
as \emp{REAL*8}.
Compare timings with the original code.

Smart compilers will notice that \emp{h(0,0)} is a constant inside
the loop and evaluate \emp{h(0,0)} just once outside the loop.
Replace \emp{lambda(i,j)} by \emp{h((i-1)*delta,(j-1)*delta)},
where \emp{delta} is the cell size, which must be initialized before
the loops in the \emp{F77WAVE} function:

{
\renewcommand{\baselinestretch}{0.85}\footnotesize
\begin{verbatimtab}[8]
REAL*8 delta
delta = 10.0/(n-1)
\end{verbatimtab}
}

\noindent This time the \emp{h} needs to be called inside the loop, unless the
compiler is so smart that it detects that the function declaration
of \emp{h} is simple (just 1 is returned) and inlines that function.
The timings will uncover how smart your compiler is.

\item
\emph{If-tests inside loops.}
Instead of splitting the finite difference scheme into a loop over
the internal grid points, \emp{i,j=2,...,n-1}, we can merge all
loops and insert appropriate if-tests. This is a typical coding habit
of novice numerical programmers:

{
\renewcommand{\baselinestretch}{0.85}\footnotesize
\begin{verbatimtab}[8]
      DO 20 j = 1, ny
         DO 10 i = 1, nx
	    if (i .ge. 2 .and. i .le. nx-1 .and. j .ge. 2
     >          .and. j .le. ny-1) then
            up(i,j) = a*2*u(i,j) - b*um(i,j) +
     >          c*LaplaceU(u,lambda,i,j,i-1,i+1,j-1,j+1)
            else if ...
C           treat boundary points...
 10      CONTINUE
 20   CONTINUE
\end{verbatimtab}
}

\noindent Implement just one such if-test, i.e., the one given above with
\emp{else if} replaced by \emp{end if}.
Is your compiler sufficiently smart to replace the loop by new
limits? Or do you get an increase in the CPU time?
\een

\section{Projects}
\ben
\item Extend the code to simulation of acoustic waves in a
heterogenous three-dimensional box.
The governing PDE is the same,
\beq
{\partial^2 u\over\partial t^2} = \nabla\cdot (\lambda\nabla u) .
\eeq
The boundary conditions are also the same: $\partial u/\partial n = 0$.
\item Repeat the tests from the exercises with the two-dimensional
version of the program. Comment upon differences in the results.
\item Reimplement the program in C or C++. Repeat the efficiency tests.
\een

\end{document}

\emph{Remarks.}
When one faces
more complicated partial differential equations and
numerical methods, array manipulations in nested loops are still the
core of most algorithms, but the programs now become much larger.
Simulators for industrial problems might contain some hundred thousand
lines of code. Organizing such programs in terms of array structures
only make strong demands to the programmers in order to achieve a code
that can easily be maintained and extended in the future.
To simplify programming tasks, one can create new data structures that
better reflect the mathematical abstractions that are involved in
the numerical solution process. Experience has shown that this gives
more modular code that is easier to reuse, maintain, and extend.
We shall demonstrate how one can implement algorithm~\ref{Wave3D:algor3}
using such techniques. Although the pay-off in this simple mathematical
problem is minor, the techniques are particularly advantageous
when the equation, the domain, and the discretization
become more complicated. Even in this simple example, working with
data structures that are closer to the mathematics than plain arrays,
will introduce a way of thinking that we believe is of general
importance when developing numerical codes.

A common strategy when building programs that are larger than a couple
of pages, is to break the problem into small pieces, and then break
each piece into even smaller pieces.
Traditionally, the ``pieces'' in numerical codes have been
functions (also referred to as subroutines or procedures).
The question was then to find the right set of functions and shuffle
data structures in and out of the functions to get the problem solved.
In this document we shall present an implementation strategy where
the ``pieces'' are \emph{classes}.  In a sense, classes are
program modules consisting of both data structures and
operations on these data structures.
From the classes one makes objects, which are nothing but variables
like reals and integers, except that the objects may contain large
data structures and built-in operations on the data structures.
Objects are hence special variables, defined by the programmer and tailored
to the problem.
The complete program declares a few objects that are manipulated in
a few functions. Most of the work is done inside the objects.
Experience has shown that well-designed objects can easily be reused
in other programs solving different types of problems.


\documentclass[12pt]{article}

\include{preamble}

\title{Math 341 / 650 Spring 2019 \\ Final Examination}
\author{Professor Adam Kapelner}

\date{Thursday, May 16, 2019}

\begin{document}
\maketitle

\noindent Full Name \line(1,0){410}

\thispagestyle{empty}

\section*{Code of Academic Integrity}

\footnotesize
Since the college is an academic community, its fundamental purpose is the pursuit of knowledge. Essential to the success of this educational mission is a commitment to the principles of academic integrity. Every member of the college community is responsible for upholding the highest standards of honesty at all times. Students, as members of the community, are also responsible for adhering to the principles and spirit of the following Code of Academic Integrity.

Activities that have the effect or intention of interfering with education, pursuit of knowledge, or fair evaluation of a student's performance are prohibited. Examples of such activities include but are not limited to the following definitions:

\paragraph{Cheating} Using or attempting to use unauthorized assistance, material, or study aids in examinations or other academic work or preventing, or attempting to prevent, another from using authorized assistance, material, or study aids. Example: using an unauthorized cheat sheet in a quiz or exam, altering a graded exam and resubmitting it for a better grade, etc.
\\

\noindent I acknowledge and agree to uphold this Code of Academic Integrity. \\

\begin{center}
\line(1,0){250} ~~~ \line(1,0){100}\\
~~~~~~~~~~~~~~~~~~~~~signature~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ date
\end{center}

\normalsize

\section*{Instructions}

This exam is 120 minutes and closed-book. You are allowed \textbf{three} pages (front and back) of a \qu{cheat sheet.} You may use a graphing calculator of your choice. Please read the questions carefully. If the question reads \qu{compute,} this means the solution will be a number otherwise you can leave the answer in \textit{any} widely accepted mathematical notation which could be resolved to an exact or approximate number with the use of a computer. I advise you to skip problems marked \qu{[Extra Credit]} until you have finished the other questions on the exam, then loop back and plug in all the holes. I also advise you to use pencil. The exam is 100 points total plus extra credit. Partial credit will be granted for incomplete answers on most of the questions. \fbox{Box} in your final answers. Good luck!

\pagebreak

\input{R_equations_table}

\problem Consider a call center that experiences calls from a large population of which there is very small probability each customer will call in each day. Let $\Xoneton$ denote the number of calls in a period of $n$ days. However, there are two types of customers: A and B (since there are two main products the company offers). And thus each day is either calls from company A or B:

\beqn
\Xoneton~|~\lambda_1, \lambda_2, \rho ~\iid~ \begin{cases} \frac{e^{-\lambda_1}\lambda_1^x}{x!} ~~w.p.~~\rho \\
\frac{e^{-\lambda_2}\lambda_2^x}{x!} ~~w.p.~~(1-\rho)
\end{cases}
\eeqn

\benum
\subquestionwithpoints{6} Provide uninformative priors for the three parameters. \spc{3}

We now introduce the new parameters $I_1, \ldots, I_n$ where each is defined as:

\beqn
I_i := \begin{cases}
1 ~~\text{if the calls from day $i$ come from customer type A} \\
0 ~~\text{if the calls from day $i$ come from customer type B} \\
\end{cases}
\eeqn

This is an idea known as \qu{data augmentation}. The likelihood now  becomes:

\beqn
&& \cprob{X_1, \ldots, X_n}{\lambda_1, \lambda_2, \rho, I_1, \ldots, I_n} = \prod_{i=1}^n \tothepow{\frac{e^{-\lambda_1}\lambda_1^{x_i}}{x_i!}}{I_i} \tothepow{\frac{e^{-\lambda_2}\lambda_2^{x_i}}{x_i!}}{1 - I_i} \rho^{I_i} (1 - \rho)^{1 - I_i}
\eeqn

\subquestionwithpoints{5} Denote $\sum I_i := \sum_{i=1}^n I_i$. By using the likelihood above and your priors from part (a), Show all your work to prove that the kernel of $\cprob{\lambda_1, \lambda_2, \rho, I_1, \ldots, I_n}{X_1, \ldots, X_n}$, the posterior  is

\vspace{-0.5cm}
\beqn
e^{-(\sum I_i) \lambda_1 } \lambda_1^{\sum I_i x_i} e^{-(n - \sum I_i) \lambda_2 } \lambda_2^{\sum (1 - I_i) x_i} \rho^{\sum I_i} (1 - \rho)^{n - \sum I_i}
\eeqn\spc{4}

\subquestionwithpoints{2} Is this a kernel from a known distribution? Yes / no. \spc{-0.5}

\subquestionwithpoints{4} Name two strategies that can be used to creates samples $\bracks{\lambda_1, \lambda_2, \rho, I_1, \ldots, I_n}$ from the posterior $\cprob{\lambda_1, \lambda_2, \rho, I_1, \ldots, I_n}{X_1, \ldots, X_n}$. \spc{1}

\subquestionwithpoints{4} Which strategy would be better and why? \spc{2}

\subquestionwithpoints{4} Find the conditional distribution $\cprob{\lambda_1}{X_1, \ldots, X_n, \lambda_2, \rho, I_1, \ldots, I_n}$. If the PMF / PDF are from a known, brand name random variable, name the distribution and provide its parameter(s). Show all work. \spc{2}

\subquestionwithpoints{4} Find the conditional distribution $\cprob{\lambda_2}{X_1, \ldots, X_n, \lambda_1, \rho, I_1, \ldots, I_n}$. If the PMF / PDF are from a known, brand name random variable, name the distribution and provide its parameter(s). Show all work.\spc{2}

\subquestionwithpoints{6} Find the conditional distribution $\cprob{\rho}{X_1, \ldots, X_n, \lambda_1, \lambda_2, I_1, \ldots, I_n}$. If the PMF / PDF is from a known, brand name random variable, name the distribution and provide its parameter(s). Show all work.\spc{4}

\subquestionwithpoints{5} [Extra Credit] Find the conditional distribution for $I_1$ given the data and the other parameters. If the PMF / PDF is from a known, brand name random variable, name the distribution and provide its parameter(s). Show all work.\spc{6}

Below is a bar plot of our data for $n = 300$ days:

\begin{figure}[h]
\centering
\includegraphics[width=4.9in]{data_barplot}
\end{figure}
\FloatBarrier

We now build a Gibbs Sampler for all parameters $\lambda_1, \lambda_2, \rho, I_1, \ldots, I_n$ based on your answers from (a)-(h) and initially sample $10,000$ times. Below is the first 50 samples from the chains for $\lambda_1, \lambda_2, \rho$:

\begin{figure}[h]
\centering
\includegraphics[width=7.5in]{burn_in}
\end{figure}
\FloatBarrier

\subquestionwithpoints{3} How many of the initial samples should be thrown away? \spc{7}

After removing the initial part of the chain according to your answer in the previous question, below is an autocorrelation plot of the chains for $\lambda_1, \lambda_2, \rho$ with maximum lag 10:

\begin{figure}[h]
\centering
\includegraphics[width=7.5in]{acf}
\end{figure}
\FloatBarrier

\subquestionwithpoints{3} How many iterations do you think should be skipped between samples in the burned-in chains if you want $\iid$ samples from the posterior? \spc{1}

\subquestionwithpoints{4} How many $\iid$ samples from the posterior do you have now? \spc{2}

The next page shows four different histograms: one for each of the burned and thinned chains for $\lambda_1, \lambda_2, \rho$ and $I_{31}$.

\begin{figure}[h]
\centering
\includegraphics[width=7in]{lambda1s}
\includegraphics[width=7in]{lambda2s}
\includegraphics[width=7in]{rhos}
\includegraphics[width=5in]{I31}
\end{figure}
\FloatBarrier

\subquestionwithpoints{6} Provide an approximate $CR_{\lambda_1, 90\%}$. \spc{4}

\subquestionwithpoints{5} Provide an approximate Bayesian point estimate for $\lambda_2$. Indicate which point estimate you are providing by specifying the subscript to $\thetahat$. \spc{4}

\subquestionwithpoints{8} Test $\rho \neq 0.3$ using $\delta = 0.05$. List the hypotheses, pick an $\alpha$ level and provide an approximate Bayesian $p$-value and give a one sentence interpretation of the test result. \spc{7}

\subquestionwithpoints{4} In your estimation, what type of customer dominates day 31's phone calls? \spc{1}

\subquestionwithpoints{8} Explain how you would provide \emph{one} prediction for the next day's number of phone calls. Clearly write out all your steps. Make sure the company can run your code. The functions you use from Table~\ref{tab:eqs} must have explicit numeric values.  Below are 10 random samples from the burned and thinned chains for parameters $\lambda_1, \lambda_2, \rho$:

\begin{verbatim}
          [,1]  [,2]  [,3]  [,4]  [,5]  [,6]  [,7]  [,8]  [,9] [,10]
lambda1s 0.731 0.573 0.861 0.770 0.797 0.683 0.597 0.649 0.670 0.784
lambda2s 7.312 7.203 7.259 7.077 7.328 7.489 7.006 6.709 6.950 7.036
rhos     0.356 0.271 0.291 0.388 0.301 0.314 0.296 0.254 0.251 0.260
\end{verbatim}

Your answer must use some of the above numbers somehow.\spc{9}

\eenum

\problem This question is about building a model to understand the accuracy of this beverage-filling machine

\begin{figure}[htp]
\centering
\includegraphics[width=2.4in]{milk_filling.jpg}
\end{figure}

\noindent which fills 12oz plastic bottles. We decide to do an experiment and select $n = 20$ bottles at random and measure the amount of liquid in each bottle. Here are the volumes (in oz):

\begin{verbatim}
       10.76 11.02 11.62 10.20 12.03 12.18 12.06 11.42 11.93 10.68
       11.81 11.27 11.68 11.31 11.29 12.37 12.00 10.74 12.04 10.69
\end{verbatim}

\noindent and sample statistics: $\xbar = 11.6$oz and $s = 0.53$oz. We will consider these measurements realizations from $\mathcal{F} : \Xoneton~|~\theta, \sigsq \iid \normnot{\theta}{\sigsq}$ where both $\theta$ and $\sigsq$ are unknown.
\benum

\subquestionwithpoints{3} If the posterior of interest is $\cprob{\theta, \sigsq}{\Xoneton}$, provide a non-informative conjugate prior for both $\theta$ and $\sigsq$ below. It does not need to be proper.\spc{1}

\subquestionwithpoints{5} Using the prior from (a) and function(s) from Table~\ref{tab:eqs}, provide an exact $CR_{\theta, 95\%}$ .\spc{2}

\subquestionwithpoints{5} Using the prior from (a), find $\sigsqhat_{MMSE}$.\spc{3}

\subquestionwithpoints{6} Using the prior from (a) and function(s) from Table~\ref{tab:eqs}, find the probability that the next milk bottle will have less than 12oz of milk.\spc{3}

\subquestionwithpoints{5} Compute the following integral as a function of $a$, $b$ and fundamental constants. To get full credit you must show and justify all steps. \\~\\
$\displaystyle\int_\reals e^{ax - bx^2} dx = $\spc{5}
\eenum

\end{document}


\lab{Computer Caches}{Caches}
\objective{Understand the role caches play in computational performance.}

The real computational bottleneck of computers lies in the memory system, not the processor.
In the early days of computing, memory speed and CPU speed were evenly matched.
The CPU accessed data directly from RAM without creating bottlenecks.
Now, the CPU spends a large amount of time waiting for data to process.
RAM simply isn't fast enough to keep up.  With CPUs crunching numbers at faster and faster rates, higher performance memory needed to be placed between a relatively slow RAM and the CPU to cache data.
This topic can very quickly become extremely technical.
The aim of the exercises in this lab are to provide a general knowledge of caching and its impact on performance.

\section*{Caching}
% TODO: add pictures of a library room with elements of this story.
A small story may help explain why we care about caching.
Imagine that you are working on an academic thesis.
Your thesis covers a very complicated topic and you need to reference a great number of papers and other resources.
It's late at night and your are alone.
You select a smallish desk that you will use for writing.
You decide to start by looking up a long list of books and their call numbers.  Bringing back as many books as you can carry, you commence writing.
Not surprisingly, you quickly discover that your desk simply is not big enough to hold all the books that you brought back.
You spot an unused library cart standing by a nearby wall.
Bringing the cart over to your desk you use it to store books that are not currently being used.
The cart also is useful for queuing books that will you will soon use.
However, searching the cart for a particular book soon becomes tiresome.
Not only do you have to find the book, but you have to find the correct page number.
You see a medium-sized table not far from your desk.
You decide to move the table adjacent to your desk so you can place open books on the desk.
As you work through the night and early hours of the morning, you establish a very efficient system.
Every hour or so, you collect books from off the shelves using the library cart, returning unused books back to the shelves.
Books that you need soon are placed on the large table, and books needed immediately are placed directly on your desk.
You notice that your most used books stay on your desk, while useful, frequently referenced books stay on open on the large table.
As the first rays of sunlight shine through the east library window, you place the last finishing touches on your thesis.
What you have mimicked to some degree is the caching system of modern computers.

Caches were introduced to keep often-used memory accessible very quickly.  High speed memory is very expensive; there are economical constraints on how much cache is available in a CPU.
Another constraint is efficiency.  The faster the memory cache, the smaller it must be.
A final constraint is size.  The fastest levels of cache need to physically close to the processing units of a CPU.
Today, there are several levels of cache located on the same chip as the CPU.

Caches are effective in two main scenarios: temporal locality and spatial locality.
Temporal locality describes the situation where we reuse specific parts of memory over and over again.
In our story, these would be books that we referred to many times.
These often-used memory locations are cached to make them more quickly accessible.
Spatial locality describes the situation where we access memory locations that are close together.
In our story, these books are physically located close to each other on the shelves.
We can cache a particular memory address and all the addresses around it as well.
Today's caches can also try to predict what data will be needed next and pre-fetch it from slower levels of memory.
If we use algorithms that can take advantage of these two data usage patterns, caching will provide a tremendous speed boost.
However, if we are not intelligent in our choice of algorithm, we will be forcing the CPU to sit idle as it waits for data to be fetched from main memory.
The optimal strategy for an efficient algorithm is to fetch sequential blocks of data and reuse them as much as possible before requesting more data from main memory.
For the last few decades, the bottleneck in performance has been memory speed, not processor speed.

\section*{Cache lines}
The cache made up of \emph{cache lines}.
Cache lines contain blocks of bytes that have been fetched from main memory.
Accessing other bytes within the same cache line has to close to no overhead.
When a requested byte is already in cache, it is called a \emph{cache hit}.
We can measure the cache line size by stepping through an array with different step sizes.
This is done by measuring runtimes as we try to step off the end of a cache line.
We have done this in Figure \ref{fig:linesize}.  We are using 32 bit integer arrays in all of our tests.
When using stepping values of 2 through 16, we are fetching the same number of cache lines from the array.
In essence, we are fetching the entire array into cache.
With step sizes larger than 16, we are not fetching the entire array into cache.
We would expect, however, that runtimes would decrease as we choose larger step sizes.
This indicates that we are accessing values within the same cache line.
Since a 32-bit integer is 4 bytes, we arrive at a cache line size of 64 bytes ($4 \mbox{~bytes} * 16 \mbox{~integers} = 64 \mbox{~bytes}$).
\begin{figure}[h]
\centering
\includegraphics[width=\textwidth]{cache_line.pdf}
\caption{Determining cache line size by looping through a 128MB array with different step sizes.}
\label{fig:linesize}
\end{figure}

\begin{problem}
Find the cache line size for your machine.
To compile you the included C program, you will need a C compiler.
We recommend gcc with the flags \texttt{-O3 -mtune=native}.
The C program expects two arguments: first, the candidate cache line size, and second, the number of loops to run during testing.
The program will print out how many seconds on average one loop took to execute in seconds.
Plot the results of your timings.
\label{prob:cacheline}
\end{problem}

\section*{Cache Size}
% TODO: insert graphics that illustrate the memory hierarchy
Various levels of caches were inserted between main memory and the CPU.
As cache sizes grow, their access times slow down.
Level 1 (L1) cache was wired directly into the processor.  Level 2 (L2) cache is also wired directly into processor.
Many multi-core processors these days have a third (L3) cache in addition to L1 and L2 caches.
L1 cache is smallest cache at around 32 kilobytes with the fastest with access times of around 1 nanosecond.
In our story, this was represented by the writing desk.
L2 cache comes in at about 3 nanoseconds.  L3, the largest, but slowest of the caches, clocks in at about 10 to 20 nanoseconds.
The library table represented L2 and L3 caches.
Main memory finally takes about 100 nanoseconds to access.  While 100 nanoseconds is really quick in our perception of time, it is a really long time in the processor's measurement of time.
The library cart represented main memory.
A \emph{clock cycle} is the fundamental unit of time for a processor.
Accessing data from L1 cache takes about 4 clock cycles on modern processors.  Fetching data from L2 cache takes about 10 clock cycles and from L3 about 75 clock cycles.
Fetching data from RAM takes several hundred clock cycles.
Each cache stores a subset of the cache above it.  L2 cache stores has a copy of all of the data in L1, L3 has a copy of all of the data in L2 and so on.
To request data from mass storage like a hard drive or optical disc takes millions of clock cycles.  This is a relative eternity in processor time.
If a clock cycle is equated to one second, fetching data from the hard drive would take about 1 year.
This is why caching is important.  Operating systems generally use part of RAM as a cache for the hard drive.

We can measure the drops in performance that occur when we overflow the cache (Figure \ref{fig:cachesizes}).
We performed measurements on an Intel i7-2640M processor.  The cache line size is 64 bytes (the size of 16 32-bit integers).
\begin{figure}[h]
\centering
\includegraphics[width=\textwidth]{cache_size.pdf}
\caption{The increases in runtime occur at 32KB, 256KB and 4MB which correspond to the sizes of L1, L2, and L3 caches.}
\label{fig:cachesizes}
\end{figure}

\begin{problem}
Measure the size of your L1 and L2 caches using the cache line value obtained in
Problem \ref{prob:cacheline}.
To compile you the included C program, you will need a C compiler.
We recommend gcc with the flags \texttt{-O3 -mtune=native}.
The C program expects three arguments: first, the candidate array size, second, the cache line size, and third, the number of loops to run during testing.
The program will print out how many seconds on average one loop took to execute in seconds.
Plot the results of your measurements.
\end{problem}

\section*{Solutions to CPU Starving}
\emph{CPU starvation} refers to the condition of have to wait a relatively long time data to compute.
The way to solve the CPU starvation problem is to be aware of how your data is laid out in memory.
NumPy has support for two main types of memory layouts for arrays.
% TODO: add illustration of row major vs, column major format for arrays.
\begin{description}
\item[Row-major:] Arrays are stored by rows in continuous memory.
Languages such as C and Python use row-major indexing.  NumPy arrays by default use this indexing convention.
Slicing down columns is slow.  In NumPy, row-major arrays are identified as \emph{C contiguous} (\li{order=`C'}).
\item[Column-major:] Arrays are stored by columns in contiguous memory.
Languages like FORTRAN, MATLAB, and R use column-major indexing.  Slicing across rows is slow.
In NumPy, column-major arrays are identified as \emph{FORTRAN contiguous} (\li{order=`F'}).
\end{description}
Paying attention to how your arrays are indexed will be beneficial to the performance of your algorithms.
Some algorithms work better with one type of indexing

While NumPy is very fast for most purposes, it has a tendency to create many temporary arrays as it evaluates results.
The code below will result in the creation of many different arrays, each the size of \li{x}!
Calculating $y$ takes about one second on the author's machine.
\begin{lstlisting}
x = np.linspace(-1, 1, 10*1024*1024)
y = 1.25*x**3 - .33*x**2 + .75*x + 1.2
\end{lstlisting}
We can reduce the runtime by using a little bit of algebra.
Factoring the expression results results in a much easier expression for the computer to evaluate.
Calculating the equivalent expression takes $.146$ seconds!
\begin{lstlisting}
y = ((1.25*x - .33)*x + .75)*x + 1.2
\end{lstlisting}
A tremendous speed up for not a whole lot of work.

Numexpr, a library for python, is designed to utilize the CPU cache as well as avoid unnecessary temporary arrays.
It evaluates the following expression allocating only a single array and operating on $x$ in blocks that fit into the CPU cache.
It calculates the entire expression for that particular block, then saves the result.
One the authors machine, numexpr executed the unfactored expression in $.0387$ seconds and the factored expression in $.0349$ seconds.
\begin{lstlisting}
import numexpr as ne
yne = ne.evaluate(`1.25*x**3 - .33*x**2 + .75*x + 1.2')
\end{lstlisting}
Using numexpr instead of NumPy can speed up simple vector computations significantly.
These large gains in performance are primarily for large arrays.
Numexpr will operate on these large arrays in blocks that fit into the CPU cache.
Numexpr is also designed to easily take advantage of multiple cores by default, a feature that NumPy lacks.
The above evaluations using numexpr we done using 4 threads.
Please refer to the numexpr Users Guide (\url{https://code.google.com/p/numexpr/wiki/UsersGuide}) for information on what operations are supported by numexpr.

Many scientific libraries have been optimized to utilize the CPU cache.
Libraries like LAPACK and BLAS have been tuned by many experts to efficiently use memory.
Other packages, like ATLAS (a alternative BLAS implementation), can automatically tune their algorithms to optimize performance on a particular platform.


\section{Online Resources}
\begin{itemize}
	\item \href{https://www.khanacademy.org/math/multivariable-calculus}{Khan Academy - Multivariable Calculus}
	\item \href{http://tutorial.math.lamar.edu/Classes/CalcIII/CalcIII.aspx}{Paul's Online Notes - Calc III}
	\item \href{https://www.youtube.com/playlist?list=PLF83D74BA4DE75897}{PatrickJMT - Calculus / Third Semester / Multivariable Calculus}
	\item \href{https://ocw.mit.edu/courses/mathematics/18-02sc-multivariable-calculus-fall-2010/}{MIT OpenCourseWare - Multivariable Calculus}
	\item \href{https://www.whitman.edu/mathematics/multivariable/multivariable.pdf}{Stewart  - Single and Multivariable Calculus}
\end{itemize}


% === CSC207 Summer 2018 ===
% __author__ = 'Eric Koehli'
% === Problem Set 1 ===

% **************************************************************************
%                              BEGIN PREAMBLE
% **************************************************************************

\documentclass[12pt, a4paper]{amsart}

\usepackage{amssymb, amscd, amsmath, amsthm, url, microtype, enumitem}
\usepackage[margin=2.5cm]{geometry}
\usepackage{csc}
\usepackage{setspace} % A package for spacing (double space, etc.)
% \linespread{1.3} % halfspaced
\usepackage{enumitem} % package for roman (i), (ii), etc.

\pagestyle{plain} % Avoids repeating the authors on every page.
\onehalfspacing   % part of setspace package
% \vspace{5mm} % adds a blank line.

% Either use these or the geometry package:
% \addtolength{\oddsidemargin}{-0.5in}
% \addtolength{\evensidemargin}{-0.5in}
% \addtolength{\textwidth}{1in}
% \addtolength{\topmargin}{-0.3in}
% \addtolength{\textheight}{0.3in}

\newtheorem {Theorem}                    {Theorem}
\newtheorem {Proposition}[Theorem]       {Proposition}
\newtheorem {Lemma}      [Theorem]       {Lemma}
\newtheorem {Corollary}  [Theorem]       {Corollary}
\newtheorem*{Claim}       {Claim}
\newtheorem*{Question}   {Question}
\newtheorem*{Exercise}   {Exercise}

\theoremstyle{definition}
\newtheorem  {Definition} [Theorem]{Definition}

\theoremstyle{remark}
\newtheorem*{Remark}	{Remark}
\newtheorem{Example}[Theorem]	{Example}

\numberwithin{Theorem}{section}

\newcommand{\mute}[1] {}
\newcommand{\mc}[1]{\mathcal{#1}}
\newcommand{\mf}[1]{\mathfrak{#1}}
\newcommand{\cal}[1]{\mathcal{#1}}

% \def \N{\mathbb{N}}
% \def \Z{\mathbb{Z}}
% \def \R{\mathbb{R}}
% \def \C{\mathbb{C}}
% \def \Q{\mathbb{Q}}

\def \vphi{\varphi}
\def \eps{\varepsilon}
\def \ssm{\smallsetminus}

% ******************************************************************************
%                               END PREAMBLE
% ******************************************************************************
% Document starts here
\begin{document}

% Document metadata
\title{CSC207H1 Summer 2018: Problem Set 1}
\author{By: Eric Koehli}
\date{May 26, 2018}
\maketitle
\newpage
% End

\section{Problem 1}

Prove that
$\forall n \in \N$, $n \geqslant 1 \implies 9 \mid 4^n  + 15n - 1$.

\begin{proof}

Let $n \in \N$ and define $P(n) : 9 \mid 4^n  + 15n - 1$.
We will prove $P(n)$ is true for all natural numbers $n \geqslant 1$
by induction.

\base

Let $n = 1$. Then $4^1 + 15(1) - 1 = 18$. Since $9 \mid 18$,
we have shown $P(1)$ is true.

\istep

Let $k \in \N$ and assume $P(k)$ is true. With the definition of
divisibility, we are assuming that
$\exists c_k \in \Z$, $4^k + 15k - 1 = 9c_k$. We want to show that
$P(k + 1)$ is also true. That is,
$P(k + 1)$ : $\exists c_{k+1} \in \Z$,
$4^{k+1} + 15(k + 1) - 1 = 9c_{k+1}$.

Let $c_{k + 1} = \frac{1}{3}(4^k - 1) + 18 + 9c_k$. From the LHS, we have

\begin{align*}
    4^{k+1} + 15(k + 1) - 1 &= 4 \cdot 4^k + 15k + 15 - 1 \\
    &= 3 \cdot 4^k + 4^k + 15k + 15 - 1 \\
    &= 3 \cdot 4^k - 3 + 18 + (4^k + 15k - 1) \\
    &= 3(4^k - 1) + 18 + 9c_k \tag{By I.H.} \\
    &= 9 \cdot \big[ \frac{1}{3}(4^k - 1) + 2 + c_k \big ] \\
    &= 9c_{k+1}
\end{align*}

The final step is not justified until we can show that
$3 \mid (4^k - 1)$ (i.e. is a multiple of 3). We will prove this by
induction (albeit less formally than the current proof).

\base

Let $k = 0$. Then $4^0 - 1 = 0$ and $3 \mid 0$. Thus $P(0)$ is true.

\istep

Assume $P(k) : 4^k - 1 = 3p$ for some $p \in \Z$. We will show
$P(k + 1) : 4^{k+1} - 1 = 3q$ for $q \in \Z$. Let $q = 4^k + p$.
From the LHS, we have

\begin{align*}
    4^{k+1} - 1 &= 3\cdot 4^k + (4^k - 1) \\
    &= 3 \cdot 4^k + 3p \tag{By the I.H.} \\
    &= 3(4^k + p) \\
    &= 3q
\end{align*}

This shows that $4^k - 1$ is a multiple of three, which now
justifies our final step in the original proof.
Thus $P(k + 1)$ follows from $P(k)$ and this completes the induction step.
Having shown steps $1$ and $2$, we can conclude by the
Principle of Mathematical Induction that $P(n)$ is true for all
natural numbers $n \geqslant 1$.

\end{proof}

\newpage

\section{Problem 2}

\begin{enumerate}
    \item [(a)] Prove that every natural number $n \geqslant 1$ has
    a binary representation.

    \item [(b)] Prove that the binary representation is unique. That is,
    $\forall n \in \N$, $n$ is a binary representation $\implies n$
    is unique.
\end{enumerate}

\begin{proof}
    Proof for $(a)$. Since binary numbers are written as powers of $2$,
    we will show that any number $n$ (base 10) can be written as a sum
    of powers of $2$. We will use strong induction to show that this is true.
    Let $n \in \N$ (base $10$) and define
    $P(n)$ : $n$ has a binary representation as the equation:
    $$n = \sum_{i = 0}^{r}b_i 2^i$$
    where $r \in \Z^+$, and $b_i = 0, 1$ for $i = 0, 1, \dots, r$.

    \base

    We can actually show $0$ and $1$ are true (even though we are only
    asked for $n \geqslant 1$).
    Let $n = 0$, $r = 0$, $b_0 = 0$. Then
    $$0 = \sum_{i = 0}^{0}b_i 2^i = b_0 2^0 = 0 \cdot 1 = 0$$
    To show $P(1)$, let $n = 1$, $r = 0$, $b_0 = 1$. Then
    $$1 = \sum_{i = 0}^{0}b_i 2^i = b_0 2^0 = 1 \cdot 1 = 1$$
    Thus $P(0)$ and $P(1)$ are true.

    \istep

    Let $k \in \N$ (base $10$) and assume $P(0), P(1), P(2), \dots, P(k)$
    are all true. That is,
    $$P(j): j = \sum_{i = 0}^{r}b_i 2^i$$
    for $0 \leqslant j \leqslant k$. We now want to show that $P(k + 1)$
    is true. We'll split the proof into two cases depending on
    whether $k + 1$ is even or odd.

    \textbf{Case 1: k + 1 is even}

    In this case, $\frac{k + 1}{2}$ is an integer and
    $0 \leqslant \frac{k + 1}{2} \leqslant k$. Then we can use the
    induction hypothesis:

    \begin{align*}
        \frac{k + 1}{2} &= \sum_{i = 0}^{r}b_i 2^i \\
        k + 1 &= \sum_{i = 0}^{r}b_i 2^{i + 1} \tag{multiply both sides by 2}
    \end{align*}

    \textbf{Case 2: k + 1 is odd}

    In this case $\frac{k}{2}$ is an integer and
    $0 \leqslant \frac{k}{2} \leqslant k$, so the induction hypothesis
    applies and we get

    \begin{align*}
        \frac{k}{2} &= \sum_{i = 0}^{r}b_i 2^i \\
        k &= \sum_{i = 0}^{r}b_i 2^{i + 1} \\
        k + 1 &= \sum_{i = 0}^{r}b_i 2^{i + 1} + 2^0
    \end{align*}
    (since $2^0 = 1$). Thus $P(k + 1)$ follows from $P(k)$ and this
    completes the induction step. Having showing steps $1$ and $2$,
    we can conclude by the Principle of Strong Induction that
    $P(n)$ is true for all natural numbers $n$ (i.e. for every natural
    number, there \emph{exists} has a binary representation).

\end{proof}

\begin{proof}
    Proof of $(b)$, the uniqueness of binary numbers. We will prove
    that the binary representation of $n$ is unique by contradiction.

    Suppose that $n$ is not unique. Then there must exist some
    $m \in \N$ (base $10$) such that $m \neq n$, but
    $m$ and $n$ have the same binary representation:

    \begin{align*}
        \sum_{i = 0}^{r}b_i 2^i = \sum_{i = 0}^{s}c_i 2^i
    \end{align*}
    for some positive arbitrary integers $r$ and $s$ and
    "bits" $b_i, c_i \in \{ 0, 1 \}$. We may now assume
    that $r > s$ (or $r < s$). Then we can show that

    \begin{align*}
        2^r &> 2^{s + 1} - 1 \tag{by geometric series} \\
        &= 1 + 2 + \dots + 2^{s - 1} + 2^s \tag{geometric series expanded} \\
        &= \sum_{i = 0}^{s}2^i \\
        &\geqslant \sum_{i = 0}^{s}c_i 2^i
    \end{align*}

    The last step follows since some $c_i$ \emph{can} equal $0$.
    Therefore, this shows that

    \begin{align*}
        \sum_{i = 0}^{r}b_i 2^i > \sum_{i = 0}^{s}c_i 2^i
    \end{align*}
    which contradicts our assumption that $n$ is not unique.
    Then it must follow that the binary representation
    of $n$ is indeed unique.
\end{proof}

\newpage

% \section*{Problem 2}

% Simple version: Intuition only. Do not mark.
% \begin{enumerate}
%     \item [(a)] Prove that every natural number $n \geqslant 1$ has
%     a binary representation.

%     \item [(b)] Prove that the binary representation is unique. That is,
%     $\forall n \in \N$, $n$ is a binary representation $\implies n$
%     is unique.
% \end{enumerate}

% \begin{proof}
%     Proof for $(a)$. Let $n \in \N$ and define $P(n)$ :
%     $n$ has a binary representation.

%     \base

%     Let $n = 1$. Then $1_{10} = 1 \times 2^0 = 1_2$. Thus $P(1)$ is true.

%     \istep

%     Let $k \in \N$ and assume $P(1), P(2), \dots, P(k)$ are true. We want
%     to show that $P(k + 1)$ is true. That is,
%     $P(k + 1) : k + 1$ has a binary representation.

%     Since we know $k$ has a binary representation, $P(k)$, and we also
%     know that $1_{10}$ has a binary representation, $P(1)$, then we can
%     take the sum,
%     $$P(k + 1) = P(k) + P(1)$$
%     with the understanding that the addition operation holds for
%     binary numbers.

%     Thus $P(k + 1)$ follows from $P(k)$ and this completes the
%     induction step. Having shown steps $1$ and $2$, we can
%     conclude by the Principle of Mathematical Induction that
%     $P(n)$ is true for all natural numbers $n \geqslant 1$.
%     Note that we did not need to use strong induction here.
%     However, if the base case was $0$, then strong
%     induction would be needed (to prove it in this way).

% \end{proof}

% \begin{proof}
%     Proof of $(b)$. We will prove that $n$ is unique by contradiction.

%     Suppose that $n$ is not unique. Then there must exist some
%     $m_{10} \in \N$ (base $10$) such that $m_{10} \neq n_{10}$ and
%     $m_{10}, n_{10}$ have the same binary representation.

%     Since $n$ is not unique, then there exists ``bits"
%     $b_0, b_1, \dots, b_p \in \{ 0, 1 \}$ for some $p \in \N$
%     such that $b_0 b_1 \dots b_p = m_{10}$ and $b_0 b_1 \dots b_p = n_{10}$.
%     But we know that $n_{10} \neq m_{10}$. Therefore $b_0, b_1, \dots, b_p$
%     cannot represent two separate numbers in base $10$.
%     Thus our original assumption is false. Then it follows that $n$ is unique.

% \end{proof}

% \newpage

\section{Problem 3}

Prove
$\forall n \in \N$, $n > 1 \implies 2^{n/2}
\leqslant g(n) \leqslant 2^{n}$.

\begin{proof}
    Let $n \in \N$ and define $P(n) : 2^{n/2} \leqslant g(n) \leqslant 2^{n}$.
    We will prove that $P(n)$ is true for all natural numbers $n > 1$
    by complete induction.

    \base

    Let $n = 2$. Then

    \begin{align*}
        2^{2/2} = 2^1 = 2 &\leqslant g(2) \\
        &= g(2 - 2) + g(2 - 1) = g(0) + g(1) = 4 \\
        &\leqslant 2^2 = 4
    \end{align*}
    Thus $P(2)$ is true.

    \istep

    Let $k \in \N$ and assume $P(2), P(3), \dots, P(k)$ is true.
    For reference, we are assuming that the following holds:

    \begin{enumerate}
        \item $2^{k/2} \leqslant g(k) \leqslant 2^{k}$
        \item $2^{(k - 1)/2} \leqslant g(k - 1) \leqslant 2^{k-1}$
    \end{enumerate}

    We want
    to show $P(k + 1) : 2^{(k+1)/2} \leqslant g(k + 1) \leqslant 2^{k+1}$.
    That is, we want to show the following two conditions hold:

    \begin{enumerate}[label=\roman*), font=\itshape]
        \item $2^{(k + 1)/2} \leqslant g(k + 1)$
        \item $g(k + 1) \leqslant 2^{k + 1}$
    \end{enumerate}

    We begin by showing \emph{(i)} as follows

    \begin{align*}
        g(k + 1) &= g(k) + g(k - 1) \\
        &\geqslant 2^{k/2} + 2^{(k-1)/2} \tag{by (1) and (2)} \\
        &= 2^{k/2} + 2^{k/2} \cdot 2^{-1/2} \\
        &= 2^{k/2} \cdot (1 + \frac{1}{\sqrt{2}}) \\
        &\geqslant 2^{k/2} \cdot 2^{1/2} \tag{by calculator} \\
        &= 2^{(k+1)/2}
    \end{align*}
    This shows that the first condition holds.

    \newpage
    For \emph{(ii)} we have

    \begin{align*}
        g(k + 1) &= g(k) + g(k - 1) \\
        &\leqslant 2^k + 2^{k-1} \tag{by (1) and (2)} \\
        &\leqslant 2^k + 2^k \\
        &= 2^{k + 1}
    \end{align*}

    By showing \emph{(i)} and \emph{(ii)} holds, we can conclude
    that $P(k + 1)$ follows from $P(2), P(3), \dots, P(k)$, and this
    completes the induction step. Having shown steps $1$ and $2$
    we can conclude by complete induction
    that $P(n)$ is true for all natural numbers $n > 1$.

\end{proof}

\newpage

\section{Problem 4}

Let $n \in \N$ and let $T(n)$ denote the time complexity of $L(n)$
for inputs $n$. The base case is when $n < 7$ and the function
$L(n)$ runs in constant time. Thus $T(n) = c$ in this case.

If $n \geqslant 7$, then $L(n)$ makes a recursive call. To
analyze the runtime we need to consider the recursive and
non-recursive parts separately.

For the non-recursive part, only a constant number of steps occur
(the \texttt{if} check, the addition, and the \texttt{return}), so
we can say that the non-recursive part takes $d$ steps.

The recursive call is $L(n/7)$, which has a worst-case runtime
of $T(n/7)$. Thus when $n \geqslant 7$, we get the recurrence
relation $T(n) = T(n/7) + d$. Therefore the full recursive definition
of $T$ is:

\[
    T_1(n) =
    \begin{cases}
        c & \text{if $n < 7$} \\
        T(n/7) + d & \text{otherwise}
   \end{cases}
\]

Now we want to find the closed form of $T$. We first make the
substitution $m = \log_7(n)$. Then $n = 7^m$.

\begin{align*}
    &T(7^0) = T(1) = c + 0d \tag{m = 0} \\
    &T(7^1) = T(7) = T(1) + d = c + 1d \tag{m = 1} \\
    &T(7^2) = T(49) = T(7) + d = c + 2d \tag{m = 2} \\
    &\vdots \\
    &T(7^m) = \dots = c + md
\end{align*}
Then when we convert back to $n$, we get the closed form
shown below

\[
    T_2(n) = c + d \cdot \log_7(n)
\]

\newpage

Proof of Correctness: Prove $\forall n \in \N$,
$n \geqslant 7 \implies T_2(n) = c + d \cdot \log_7(n)$.

\begin{proof}
    Let $n \in \N$ and define $P(n) : T_1(n)$ is equivalent to the
    closed form $T_2(n)$.
    We want to prove that $T_2(n)$ is the closed form of $T_1(n)$.

    \base

    Let $n = 7$. Then $T_1(7) = T_1(7/7) = T_1(1) = c + d$. Also,
    $T_2(7) = c + d \cdot \log_7(7) = c + d$. Thus $P(7)$ is true.

    \istep

    Let $m, k \in \N$ and assume $P(m)$ is true for $7 \leqslant m \leqslant k$.
    That is, $P(m): T_1(m)$ is equivalent to $T_2(m) = c + d \cdot \log_7(m)$.
    We want to show $P(k + 1): T_1(k + 1)$ is equivalent to
    $T_2(k + 1) = c + d \cdot \log_7(k + 1)$. From our time complexity
    function, we have

    \[
    T_1(k + 1) =
    \begin{cases}
        c & \text{if $k + 1 < 7$} \\
        T(\frac{k + 1}{7}) + d & \text{otherwise}
   \end{cases}
    \]
    For reference we note that $(a)$:
    $7 \leqslant \frac{k + 1}{7} \leqslant k$. Then we have

    \begin{align*}
        T_1(k + 1) &= T_1(\frac{k + 1}{7}) + d \\
        &= c + d \cdot \log_7(\frac{k+1}{7}) + d \tag{by I.H. and (a)} \\
        &= c + d \cdot \big [ \log_7(k + 1) - \log_7(7) \big ] + d \\
        &= c + d \cdot \big [ \log_7(k + 1) - 1 \big ] + d \\
        &= c + d \cdot \log_7(k+1) - d + d \\
        &= c + d \cdot \log_7(k+1) \\
        &= T_2(k + 1)
    \end{align*}

    Thus $P(k + 1)$ follows from $P(m)$ by complete induction.
    Since the basis and induction steps have been shown, we can
    conclude that $P(n)$ is true for all natural numbers
    $n \geqslant 7$. Thus
    we have shown that the closed form $T_2$ is equivalent to
    the time complexity function $T_1$. This completes the
    proof of correctness.

\end{proof}

\newpage

\section{Problem 5}

Define the set $F$ recursively as follows:

\begin{enumerate}
    \item [(a)] $7 \in F$
    \item [(b)] if $u, v \in F$, then $u + v \in F$
    \item [(c)] Nothing else is in $F$
\end{enumerate}

Prove by structural induction that $\forall w \in F$, $w$ mod $7 = 0$
(i.e. $7 \mid w$).

\begin{proof}
We will prove by structural induction that $\forall w \in F$, $7 \mid w$.
Given any element in $F$, let the property $P$ be the claim that
$7$ divides the element. That is, define $P(w): 7 \mid w$.

\base

We want to show that each element in the base of $F$ satisfies $P$.
The only element in the base of $F$ is $7$, and we know that $7 \mid 7$.
Thus, $P(7)$ is true.

\istep

We now want to show that for each rule in the recursion for $F$,
if the rule is applied to an element in $F$ that satisfies
the property $P$, then the element defined by the rule also
satisfies the property $P$.

The recursive rule for $F$ consists of one step denoted by (b) above.

Suppose $u, v \in F$ are any two elements such that
$7 \mid u$ and $7 \mid v$. (This is our induction hypothesis).
That is, expanding the definition of divisibility, we have
$u = 7k_1$ and $v = 7k_2$ for $k_1, k_2 \in \Z$.
When rule (b) is applied to $u$ and $v$, the result is $u + v$,
which we know is also in $F$ (by the recursive definition of $F$).
So we must now show that $u + v$ is divisible by $7$.
That is, $u + v = 7k_3$ for $k_3 \in \Z$. Let $k_3 = k_1 + k_2$.
Then by substitution, we have

\begin{align*}
    u + v &= 7k_1 + 7k_2 \\
    &= 7(k_1 + k_2) \\
    &= 7k_3
\end{align*}

This shows that $7 \mid u + v$.
Thus when the recursive rule is applied to any element divisible
by $7$ in $F$, the result is another element (say $w = u + v$),
that is also divisible by $7$. Therefore all elements in $F$
are divisible by $7$ (i.e. $P(w)$ is true for all $w \in F$).

\end{proof}

\end{document}