\documentclass[12pt]{report}
\usepackage[letterpaper, margin=0.75in]{geometry}
\usepackage{graphicx}
\usepackage{multirow}
\usepackage{booktabs}
\usepackage{float}
\usepackage{varwidth}
\usepackage{amsmath}
\usepackage{lscape}
\usepackage[makeroom]{cancel}
\usepackage{subcaption,hyperref}

\title{\textbf{MATH 6644 -- Iterative Methods \\~ \\ Final Project}}
\author{Komahan Boopathy}

\begin{document}
\maketitle
%\tableofcontents
%\clearpage
\chapter{An Application of Domain Decomposition Method Using Fortran Coarrays}

\section{Introduction and Background}

This work demonstrates domain decomposition method for solving partial
differential equations (PDEs) using Coarray Fortran parallel
programming features. We will begin with a brief review of
domain-decomposition method and coarray fortran.

\subsubsection{Need for Domain Decomposition}

The domain decomposition method works by splitting the main problem
into subproblems on subdomains and iterating to coordinate the
solution between adjacent subdomains.The problems on the subdomains
are independent (except for enforcement of compatibility conditions
across interface) , which makes domain decomposition methods suitable
for parallel computing. For example, a problem of solving a large
linear system $A\vec{x}=b$ can be decomposed into different processors as
shown in Figure~\ref{fig:decomposition}.  Each subdomain with a
smaller problem size can be solved using a suitable method of
choice. Once the subproblems are solved, information is exchanged
across neighboring processors.  %Domain decomposition can be broadly
%classified into overlapping and non-overlapping methods.

\begin{figure}[H]
  \centering
  \includegraphics[width=0.5\textwidth]{decomposition.pdf}
  \caption{Decomposition of the problem domain into subdomains on each
    processor.}
  \label{fig:decomposition}
\end{figure}

\subsubsection{Fortran Coarrays}

Fortran programming language had been in existence since 1960s. It is
the only language that boasts a standard's committee whose target
audience is scientific programmers. Most of today's scientific code is
based on Fortran. At the turn of the century there has been a lot of
emphasis on parallel programming for tackling larger problem sizes
arising in modern scientific simulations such as CFD, FEA
etc. Motivated by this, \emph{Coarray fortran} was developed by Numrich
and Reid, in 1990s and has been in existence as an extension to the
language. Coarray fortran works under the principle that the
scientific programmers do not have to worry about parallelizing by
making external library calls such as MPI or others. Instead, minimal
syntactic adjustments were introduced to communicate the
parallelization intent to the compiler which handles the parallelism
internally (by making MPI calls or others). This greatly simplifies
the parallel code and eliminates any non-optimal calls made by the
scientific programmer. The Coarray Fortran was inducted as language
standard since 2008 and has seen increased interest.  Fortran coarrays
fall under Partitioned Global Address Space (PGAS) parallel
programming model. During execution, a Fortran program containing
coarrays is interpreted as if it were replicated a fixed number of
times and all copies were executed asynchronously. Each copy has its
own set of data objects and is called an \emph{image}. The array
syntax of Fortran is extended with additional trailing subscripts in
square brackets (\texttt{[]}) to give a clear and straightforward
representation of access to data on other images.  References without
square brackets are to local data objects, so code that can run
independently is uncluttered. Any occurrence of square brackets in
dicates communication between images, which might be slow.  Fortran
standard allows for the compiler to chose any means to parallelize the
code e.g., MPI library.

\subsubsection{Scope of this Work}

In this work, a simple demonstration of the domain decomposition method
is performed on a one-dimensional partial differential equation. This
is a first attempt at using many of the core concepts of domain
decomposition and parallel programming at work. Therefore, realistic
scientific studies that need a lot of computational resources
(e.g. Navier-Stokes equations) are infeasible at this point. Future
work entails the extension of the code and libraries to work with many
solution algorithms discussed in the class such as classical
iterations, GMRES, MINRES.

\section{Model Problem}

The following differential equation is considered as the model
problem:

$$ -u^{\prime\prime}(x) = f(x) = 2x -\frac{1}{2}$$
$$u(0) = 1, u(1) = 0 $$
$$x \in [0,1]$$

The differential equation is discretized using central difference
scheme with $n$ interior mesh points.  The discretized ODE is $$-
u_{i+1} +2 u_i - u_{i-1} = h^2f(x_i),$$ where $h = \frac{b-a}{n+1}$,
$x_i=ih$, and $i = 0, \ldots, n+1$. This leads to a tridiagonal banded
linear system as follows.

$$ \begin{bmatrix}
  2 & -1  &  0 & 0 & 0\\
  -1 & 2 & -1  &  0 & 0 \\
  0 & \ddots & \ddots & \ddots & 0 \\
  0 & 0 & -1 & 2 & -1 \\
  0 & 0 &  0 & -1 & 2 \\
\end{bmatrix}
\begin{bmatrix}
  u(x_1) \\
  u(x_2) \\
  \vdots\\
  u(x_{n-1}) \\
  u(x_{n}) \\
\end{bmatrix}
=
\begin{bmatrix}
  1 +  h^2f(x_1) \\
  h^2f(x_2) \\
  \vdots\\
  h^2f(x_{n-1}) \\
  0 + h^2f(x_{n}) \\
\end{bmatrix}$$

In order to demonstrate the effectiveness of domain decomposition
method in solving large problems a sequence of mesh points are
chosen. The resulting linear system $Ax=b$ is solved using a Conjugate
Gradient Algorithm on each
sub-domain. The problem is decomposed as
illustrated in Figure~\ref{fig:decomposition} and subproblems were
solved using Conjugate Gradient method. Usually, the splitting of domain for each processor comes
directly from the geometry of the problem and meshing. However, as a
first step the matrix is directly decomposed and optimizations are not
considered. The exchange of information across domain are handled
through Fortran's collective subroutines that were introduced in 2015.

\section{Computer Implementation}

Here details of the implementation of DDM using CG as the
subdomain solver is discussed.

\subsection{Collective Functions}

In an iterative algorithm, it is essential to compute the inner
product and multiply matrices with vectors. These are provided by
Fortran as intrinsic function to the user: \texttt{dot\_product} and
\texttt{matmul}. For this project, it became necessary to extend the
functionalities to vectors and matrices that are distributed among
different processors.

\subsubsection{Vector Norm}

$L-2$ norm of a vector $x$ is defined as $\sqrt{(x,x)}$. When the
vector $x$ is broken down into $x=[x^{1},x{2},\ldots,x^{p}]$, a
collective summation of local inner products on each processor is
necessary. This can be seen implemented in the following function to
compute the L2-norm.

\begin{verbatim}

  !===================================================================!
  ! Function to compute the norm of a distributed vector
  !===================================================================!

  function co_norm2(x) result(norm)

    real(8), intent(in) :: x(:)
    real(8) :: xdot, norm

    ! find dot product, sum over processors, take sqrt and return
    xdot = dot_product(x,x)
    call co_sum (xdot)
    norm = sqrt(xdot)

  end function co_norm2

\end{verbatim}

\subsubsection{Matrix-Vector Product}

As far as the decomposition of the Jacobian matrix goes, it can be
done in one of the three popular ways:
\begin{enumerate}
\item columnwise decomposition
\item rowwise decomposition
\item checkerboard decomposition
\end{enumerate}
The schematic of matrix decomposition is illustrated in Figure~\ref{fig:matrix-decomposition}.
\begin{figure}[H]
  \centering
  \includegraphics[width=\textwidth]{matrix-decomposition-methods.pdf}
  \caption{Popular matrix decomposition strategies where each color
    refers to a processor.}
  \label{fig:matrix-decomposition}
\end{figure}

In this work, columnwise decomposition is used as it aligns with the
memory layout of fortran programs which is column major order. The
downside of columnwise decomposition is that it requires a temporary
vector of global size (labeled as \texttt{work} in the following
code).  However, one can optimize this a little bit by trading off
communication for storage requirements. In principle, it is possible
to implement a matrix-free jacobian vector products where the matrix
need not be formed explicitly. However, jacobian-free implementations
are more complex to implement.~\footnote{A prototype jacobian-free
  matrix vector product based code was implemented but did not perform
  correctly.}

\begin{verbatim}

  !===================================================================!
  ! Function that computes the matrix vector product in a distributed
  ! fashion for columnwise decomposition of matrix
  ! ===================================================================!

  function co_matmul(A, x) result(b)

    ! Arguments
    real(8), intent(in) :: A(:,:)
    real(8), intent(in) :: x(:)
    real(8) :: b(size(x))

    ! Local variables
    integer :: nimages
    integer :: me, local_size
    integer :: stat
    character(10) :: msg

    ! Create a local vector of global sizse (optimize this!)
    real(8), allocatable :: work(:)
    allocate(work, mold=A(:,1))

    ! Determine partition
    nimages = num_images()
    me = this_image()
    local_size = size(x)

    ! Multiply, sum and distrbute
    work = matmul(A,x)
    call co_sum(work, stat=stat, errmsg=msg)
    b = work((me-1)*local_size+1:me*local_size)

    deallocate(work)

  end function co_matmul

\end{verbatim}

\section{DDM-CG Algorithm}

The CG algorithm outlined in literature~\ref{Kelly} is implemented. It
was observed that the only changes that were required to the serial
algorithm to work for subdomains was:
\begin{enumerate}
\item replacing \texttt{norm2(...)} with \texttt{co\_norm2(...)} calls
\item replacing \texttt{matmul(...)} with \texttt{co\_matmul(...)} calls
\end{enumerate}
Once these calls were replaced, the original CG algorithm is
executed on all processors. Each processor contained information only
about their subdomain until exchange of information happened through
specific calls.

\section{Results}

Although basic arithmetic operations such as addition, subtraction,
multiplication and division workload are divided among processors,
there is a communication overhead when information needs to be
exchanged between processors. There is no well known rule for the
optimal selection of number of processors for a given chunk of
data. Scaling studies give a rough idea of this information.  The
following section presents the scaling studies with increasing number
of processors for a fixed problem size. Once the scaling studies are
used to establish the number of processors that give the best results,
it will then be applied to solve the linear system.  These studies
were performed on a cluster with 2.50GHz Intel Xeon CPU E5-2680-v3
compute nodes.

\subsection{Parallel Scalability Studies}

%As noted in the introduction, the problem domain is decomposed into
%different processors.

The key operations in any iterative solution algorithm are:
\begin{enumerate}
\item norm computations
\item matrix-vector products
\end{enumerate}
In both these operations \emph{inner products} are essential. In the
following the parallel scalability of these two operations are studied
for different number of processors.

\paragraph{Norm Computations}

The efficiency of calculations are studied for finding the norm after
the problem domain has been split into different processors. For this
test, a random vector with 1 billion entries was decomposed between
processors ranging from 1 to 24. Figure~\ref{fig:norm} shows the
speedup and runtimes observed during the test.  The ideal scaling
deteriorates after 8 processors and stays almost linear upto 14
processors beyond which communication overhead takes over the
performance. The $1/p^2$ stagnation can be observed from the runtime
plot which is inline with the theoretical predictions.

\begin{figure}[H]
  \centering
  \begin{minipage}{0.45\linewidth}
    \includegraphics[width=\textwidth]{norm-speedup.pdf}
    \label{Speedup}
  \end{minipage}
  \begin{minipage}{0.45\linewidth}
    \includegraphics[width=\textwidth]{norm-runtime.pdf}
    \label{Runtime}
  \end{minipage}
  \caption{Parallel scalability of $L_2$-\emph{norm} computations of
    a distributed vector of total size 1 billion. }
  \label{fig:norm}
\end{figure}

\paragraph{Matrix-Vector Computations}

The sub-matrix on each processor along the column is multiplied with a
vector during the iterative solution process. Figure~\ref{fig:matmul}
shows the performance of distributed matrix-vector product
computations.  Ideal scaling is observed upto 8 processors. Beyond 16
processors we do not observe any speedup. Note that sparsity of the
matrix for the model problem is not accounted for in this work,
therefore even zeros are multiplied. This can be alleviated with the
implementation of blockwise sparse storage of matrix such as BCSR
format.

\begin{figure}[H]
  \centering
  \begin{minipage}{0.45\linewidth}
    \includegraphics[width=\textwidth]{matmul-speedup.pdf}
    \label{Speedup}
  \end{minipage}
  \begin{minipage}{0.45\linewidth}
    \includegraphics[width=\textwidth]{matmul-runtime.pdf}
    \label{Runtime}
  \end{minipage}
  \caption{Parallel scalability of matrix-vector computations of
    a distributed vector of total size 100,000. }
  \label{fig:matmul}
\end{figure}

\subsection{Performance of DDM-CG}

The numerical tests were done to assess the performance of the DDM-CG
in terms of the iteration numbers and associated computational times.
Two problem sizes were used for these tests: 10000 and 25000~\footnote{Larger problem sizes could not be tried due to memory allocation
errors limited by the operating system. However, with the
implementation of jacobian-free methods, this problem can be alleviated.
}, with
varying number of processors.  The convergence criterion is that
The performances of DDM-CG method for both test cases is summarized in
Table~\ref{tab:summary-10000} and ~\ref{tab:summary-25000}.

\begin{table}[H]
  \caption{Comparison of performance of DDM-CG method for problem size of 10000.}
  \centering
  \begin{tabular}{c|c|c|c}
    \toprule
    nprocs   & Num. CG Iters.   & Time per iteration (sec) & Total time (sec) \\
    \midrule
    1 & 10000 & 5.5E-002 & 15,750 \\
    2 & 10000 & 3.1E-002 & 9,290 \\
    4 & 10000 & 2.7E-002 & 6,930 \\
    8 & 10000 & 2.6E-002 & 5,479 \\
    \bottomrule
  \end{tabular}
  \label{tab:summary-10000}
\end{table}

\begin{table}[H]
  \caption{Comparison of perform-ace of DDM-CG method for problem size of 25000.}
  \centering
  \begin{tabular}{c|c|c|c}
    \toprule
    nprocs   & Num. CG Iters.   & Time per iteration (sec) & Total time (sec) \\
    \midrule
    1 & 25000 & 2.40 & 7,28,202 \\
    2 & 25000 & 1.36 & 4,11,586 \\
    4 & 25000 & 0.68 & 2,12,985 \\
    8 & 25000 & 0.32 & 1,05,152 \\
    12 & 25000 & 0.225 & 70, 605 \\
    16 & 25000 & 0.162 &50, 771 \\
    20 & 25000 & 0.128 & 39,685  \\
    24 & 25000 & 0.107 & 33,772  \\
    \bottomrule
  \end{tabular}
  \label{tab:summary-25000}
\end{table}

The speedup and runtime plots were generated for each DDM-CG iteration
based on the problem of size 25000 and is shown in Figure~\ref{fig:cg}.
\begin{figure}[H]
  \centering
  \begin{minipage}{0.45\linewidth}
    \includegraphics[width=\textwidth]{cg-speedup.pdf}
    \label{Speedup}
  \end{minipage}
  \begin{minipage}{0.45\linewidth}
    \includegraphics[width=\textwidth]{cg-runtime.pdf}
    \label{Runtime}
  \end{minipage}
  \caption{Parallel scalability of each conjugate gradient iteration
    for a problem size of 25000. }
  \label{fig:cg}
\end{figure}

Overall the following observations can be made from the results:
\begin{itemize}
\item The DDM-CG algorithm obeys theoretical number of iterations to converge.
\item The time for each iteration and therefore the overall time are
  reduced with the use of more processors. There are no major
  performance bottlenecks observed with CG iterations.
\end{itemize}

\section{Software}
The computer code written by me as a part of this project and
coursework is made publicly available at \newline
\href{https://github.com/komahanb/coarray-fortran}{https://github.com/komahanb/coarray-fortran}
and
\href{https://github.com/komahanb/}{https://github.com/komahanb/iterative-algebra}.
\\
It is intended that the functionalities presented in this project
will be extended and improved in future for realistic problems arising
in science and engineering.

\begin{thebibliography}{9}

\bibitem{Kelley} Kelley, C.T. \emph{Iterative Methods for
  Optimization. In: Frontiers in Applied Mathematics}, Vol. 18, SIAM,
  Philadelphia, 1990.  http://dx.doi.org/10.1137/1.9781611970920
\end{thebibliography}

\end{document}


\documentclass[12pt,letterpaper,DIV=11]{scrartcl}
\usepackage[T1]{fontenc}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{booktabs}
\usepackage{hyperref}
\usepackage{mathtools}
\usepackage{microtype}
\usepackage{minted}
\usepackage{csquotes}
\usepackage{multirow}
\usepackage{siunitx}
\usepackage{tikz}
\usepackage{xparse}

\usetikzlibrary{matrix}

\theoremstyle{plain}
\newtheorem{theorem}{Theorem}[section]
%\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{corollary}{Corollary}

\theoremstyle{definition}
%\newtheorem{definition}{Definition}[section]
\newtheorem{example}{Example}[section]

\theoremstyle{remark}
%\newtheorem*{fact}{Fact}
\newtheorem{claim}{Claim}
\newtheorem*{comment}{Comment}

\NewDocumentCommand{\abs}{m}{\left\lvert#1\right\rvert}
\NewDocumentCommand{\norm}{m}{\left\lVert#1\right\rVert}

\DeclareDocumentCommand{\tran}{}{^{\mkern-1.5mu\mathsf{T}}}

\begin{document}
\noindent{}Textbooks:
\begin{itemize}
  \item John Loustau, Elements of Numerical Analysis with \emph{Mathematica}
  \item Eugene Isaacson, Analysis of Numerical Methods
  \item Lloyd Trefethen, Numerical Linear Algebra
\end{itemize}

\begin{displayquote}
  \emph{Numerical analysis is the study of algorithms for the problems of continuous mathematics.}
  \hfill --- Lloyd Trefethen and David Bau
\end{displayquote}
Numerical methods is a branch of mathematics concerned with approximations.
It was founded by John von Neumann and his colleagues in the Manhattan Project during World War II to simulate physical processes governed by differential equations too complex for analytic solutions.

\section{Iterative Solutions of Non-Linear Equations}
We start our survey of numerical methods with a simple introduction to root finding.
This dates back to 1600--1800 BCE (see example Babylonian tablet).

Let $V$ and $W$ be vector spaces.
Our task is to determine roots of the equation $f(x) = 0$, where $f : V \to W$.
What values of $x$ will satisfy this mapping?
Note that:
\begin{itemize}
  \item we are working with non-linear equations; that is, there is no restriction for $f$ to be linear,
  \item we require $\dim(V) = \dim(W) = k$ (when $k = 1$, we have a scalar function of a single variable) as the output becomes the input when we iterate.
\end{itemize}
Suppose we want to find the root of $f(x) = \cos(x)$.
We can reframe the problem to instead find the intersection of $g(x) = x - \cos(x)$ and $y = x$.
This intersection is selected such that the output of $f$ is equal to the input, and thus is called a \textbf{fixed point}.

We will be looking for fixed points of $g$, not roots of $f$.
For now, we can think that $g \equiv x - f(x)$.
If $f(\alpha) = 0$, then $g(\alpha) = \alpha$.

An iterative method gives a series of approximations given an initial approximation called $x_0$.
These iterative methods will be written as
\begin{displaymath}
  x_{i + 1} = g(x_i) \begin{cases}
    \begin{aligned}
      x_1 &= g(x_0) \\
      x_2 &= g(x_1) \\
          &\vdotswithin{=} \\
    \end{aligned}
  \end{cases}
\end{displaymath}

\begin{example}
  Suppose $g(x) = \frac{1}{2} \left( \frac{2}{x} + x \right)$.
  Let $x_0 = 1$.
  Then \begin{displaymath}
    \begin{aligned}
      x_1 &= g(x_0)  \\
          &= \frac{1}{2} \left( \frac{2}{1} + 1 \right) \\
          &= 1.5
    \end{aligned}
    \qquad\qquad
    \begin{aligned}
      x_2 &= g(x_1) \\
          &= \frac{1}{2} \left( \frac{4}{3} + \frac{3}{2} \right) \\
          &= 1.41\overline{6},
    \end{aligned}
  \end{displaymath} and so on.
  We see that $\lim\limits_{i \to \infty} x_i = \sqrt{2}$, and sure enough,
  $g(\sqrt{2}) = \frac{1}{2} \left( \frac{2}{\sqrt{2}} + \sqrt{2} \right) = \sqrt{2}$.
\end{example}
Unfortunately, this method only works when:
\begin{theorem}[Banach fixed-point theorem]\label{thm:banachfpt}
  Given $g$ and $x_0$, if \begin{enumerate}
    \item $g$ takes a closed and bounded set into itself,
    \item $g$ is contracting, i.e. $\norm{g(x) - g(y)} \leq M \norm{x - y}$ with $M < 1$ for all $x, y \in S$ (M is called the Lipschitz constant),
  \end{enumerate}
  then we will converge to a fixed point.
\end{theorem}
This scheme is called a Picard iteration or functional iteration method.
It was originally used to approximate solutions of ordinary differential equations.

Choosing a good $x_0$ is an unresolved problem, but we can usually figure out a good estimate from the problem formulation.

\begin{theorem}[Functional iteration for a single equation (scalar valued)]\label{thm:iterscalar}
  Let $g(x)$ satisfy the Lipschitz condition $\abs{g(x) - g(y)} \leq \lambda \abs{x - y}$ for all $x, y \in I$,
  where $I = [x_0 - \rho, x_0 + \rho]$ and $0 \leq \lambda < 1$ ($\lambda$ is the Lipschitz constant).
  Furthermore, let the original estimate $x_0$ be such that $\abs{x_0 - g(x_0)} \leq (1 - \lambda) \rho$.
  Then the following results hold: \begin{enumerate}
    \item $x_i \in I$ for all $i$,
    \item $\lim\limits_{i \to \infty} x_i = \alpha$ (existence of a fixed point),
    \item $\alpha$ is the only fixed point in $I$ (uniqueness).
  \end{enumerate}

  \begin{proof}\leavevmode
    \begin{enumerate}
      \item We prove $x_i \in I$ by induction.
        \begin{enumerate}
          \item Let $x_0 \in I$.
            Since $g(x_0) = x_1$, we have \begin{align*}
              \abs{x_0 - g(x_0)} &= \abs{x_0 - x_1} \\
                                 &\leq (1 - \lambda) \rho && \text{by our second condition} \\
                                 &\leq \rho && \text{as $0 \leq \lambda < 1$, thus $1 - \lambda < 1$.}
            \end{align*}
          \item Suppose $\abs{x_i - x_0} \leq \rho$ for all $i \leq m$; that is, $x_i \in I$.
            By the Lipschitz condition, we have that for all $i \leq m$, \begin{align*}
              \abs{g(x_i) - g(x_{i - 1})} &\leq \lambda \abs{x_i - x_{i - 1}} \\
                                          &\leq \lambda \abs{g(x_{i - 1}) - g(x_{i - 2})} \\
                                          &\leq \lambda^2 \abs{x_{i - 1} - x_{i - 2}} && \text{again by the Lipschitz condition} \\
                                          &\vdotswithin{\leq} \\
                                          &\leq \lambda^m \abs{x_i - x_0} \\
                                          &\leq \lambda^m (1 - \lambda) \rho.
            \end{align*}
            To show $x_{m + 1} \in I$, consider \begin{align*}
              \abs{x_{m + 1} - x_0} &= \abs{(x_{m + 1} - x_m) + (x_m - x_{m - 1}) + \cdots + (x_1 - x_0)} \\
                                    &\leq \abs{x_{m + 1} - x_m} + \abs{x_m - x_{m - 1}} + \cdots + \abs{x_1 - x_0} \\
                                    &\leq \lambda^m (1 - \lambda) \rho + \lambda^m (1 - \lambda) \rho + \cdots + \lambda^m (1 - \lambda) \rho \\
                                    &= (\lambda^m + \lambda^{m - 1} + \cdots + 1) (1 - \lambda) \rho \\
                                    &= \left( \frac{1 - \lambda^{m + 1}}{1 - \lambda} \right) (1 - \lambda) \rho \\
                                    &= \left( 1 - \lambda^{m + 1} \right) \rho \\
                                    &\leq \rho
            \end{align*} as $0 \leq \lambda < 1$, thus $1 - \lambda < 1$.
        \end{enumerate}
        Thus $x_{m + 1} \in I$, completing the proof by induction.

      \item To show the existence of a limit for the sequence $\{ x_i \}$, we first show that the sequence is Cauchy.
        Given $\epsilon > 0$, we find $N(\epsilon)$ such that for $m, m + p > N$, we have $\abs{x_m -  x_{m + p}} < \epsilon$.
        First, examine \begin{align*}
          \abs{x_m - x_{m + p}} &= \abs{(x_m - x_{m + 1}) + (x_{m + 1} - x_{m + 2}) + \cdots + (x_{m + p - 1} - x_{m + p})}.
          \intertext{By the triangle inequality, we have}
          \abs{x_m - x_{m + p}} &\leq \abs{x_m - x_{m + 1}} + \abs{x_{m + 1} - x_{m + 2}} + \cdots + \abs{x_{m + p - 1} - x_{m + p}} \\
                                &\leq (\lambda^m + \lambda^{m + 1} + \cdots + \lambda^{m + p - 1}) (1 - \lambda) \rho \\
                                &= (1 - \lambda^p) \lambda^m \rho \\
                                &\leq \lambda^m \rho.
        \end{align*}
        So if we want $\abs{x_m - x_{m +p}} < \epsilon$, take $N$ such that $\lambda^N \rho < \epsilon$.
        For instance, let $N = \left\lceil \frac{\log \frac{\epsilon}{p}}{\log \lambda} \right\rceil$.
        Thus we are guaranteed that $\abs{x_m - x_{m + p}} < \epsilon$ for $m, m + p > N$.
        Hence the sequence is Cauchy, and $\lim\limits_{i \to \infty} x_i = \alpha$ since $I$ is a complete metric space.
        %[for homework, show that if $\mathbb{R}$ is complete and $I$ is closed, then $I$ is complete]
        Furthermore, we have that \begin{align*}
          g(\lim_{i \to \infty} x_i) &= \lim_{i \to \infty} g(x_i) && \text{as $g$ is continuous} \\
                                     &= g(\alpha) \\
                                     &= \alpha,
        \end{align*} showing the existence of a fixed point.

        \begin{comment}
          How does $x_i$ approach our fixed point $\alpha$?
          Look at how far our initial guess is from $\alpha$, and the number of iterations we make: \begin{align*}
            \abs{x_m - \alpha} &= \abs{g(x_{m - 1}) - g(\alpha)} \\
                               &\leq \lambda \abs{x_{m - 1} - \alpha} && \text{by the Lipschitz condition} \\
                               &= \lambda \abs{g(x_{m - 2}) - g(\alpha)} \\
                               &\leq \lambda^2 \abs{x_{m - 2} - \alpha} \\
                               &\vdotswithin{\leq} && \text{using the Lipschitz inequality}\\
                               &\leq \lambda^m \abs{x_0 - \alpha} \\
                               &\leq \lambda^m  \rho,
          \end{align*} so each iterate is contained within a smaller and smaller set surrounding $\alpha$.
        \end{comment}

      \item To show uniqueness, suppose $\alpha$ and $\beta$ are both fixed points in $I$.
        Then we have that \begin{align*}
          \abs{\alpha - \beta} &= \abs{g(\alpha) - g(\beta)} \\
                               &\leq \lambda \abs{\alpha - \beta}
        \end{align*} by the Lipschitz condition.
        But since $\lambda < 1$, it must be the case that $\abs{\alpha - \beta} = 0$.
        \qedhere
    \end{enumerate}
  \end{proof}
\end{theorem}

What does this mean for differentiable functions?
Differentiable functions are closely related to Lipschitz, but they are not necessarily Lipschitz.
For example, there is no Lipschitz condition number that will satisfy the inequality for $x^2$, where $x \in \mathbb{R}$.
\begin{corollary}\label{cor:iterscalar}
  If $\abs{g'(x)} \leq \lambda < 1$ for $\abs{x - x_0} \leq \rho$, and $\abs{x_0 - g(x_0)} \leq (1 - \lambda) \rho$, then results (1), (2), and (3) of Theorem~\ref{thm:iterscalar} hold.
  \begin{proof}
    The Mean Value Theorem gives us a way to go between secant lines (our Lipschitz condition) and tangent lines (our differentiable function).
    It states that if $f$ is differentiable on the interval $[a, b]$, then there exists $\xi$ such that $a < \xi < b$ and $f'(\xi) = \frac{f(b) - f(a)}{b - a}$.

    Let $x, y \in I$, and assume $x < y$ without loss of generality.
    By the Mean Value Theorem, we have \begin{align*}
      g(x) - g(y) &= g'(\xi)(x - y) \\
      \abs{g(x) - g(y)} &\leq \lambda \abs{x - y} && \text{since $\abs{g'(\xi)} \leq \lambda$}
    \end{align*} for $x < \xi < y$, so our Lipschitz condition number is at most $\lambda < 1$ for all $x, y \in I$.
    All the results of Theorem~\ref{thm:iterscalar} follow.
  \end{proof}
\end{corollary}

\subsection{Visualizing Iterative Methods}
To understand our convergence theorem, consider $\frac{\Delta y}{\Delta x} = \lambda$ and $\frac{\Delta y}{\Delta x} = - \lambda$ on the interval $[x_0 - \rho, x_0 + \rho]$, both passing through the point $(x_0, g(x_0))$ to form a \enquote{bowtie}.
Notice that at $x = x_0 + \rho$, $g(x) = g(x_0) + \lambda \rho$.
Consider $y = x$, and let $\epsilon = \abs{x_0 - g(x_0)}$.
Then at $x = x_0 + \rho$, we have that the distance between $y = x$ and $\frac{\Delta y}{\Delta x} = \lambda$ is $\rho - \epsilon - \lambda \rho$.
[diagram omitted] % TODO?

If $\lambda > 1$, then our bowtie becomes too tall, and thus $y = x$ does not intersect with the top part of the triangle.
If $\lambda < 1$, but $\rho - \epsilon - \lambda \rho < 0 \iff \epsilon > (1 - \lambda) \rho$, then $y = x$ does not intersect with the top part of the triangle as well.
  Thus $\lambda < 1$ and $\abs{x_0 - g(x_0)} < (1 - \lambda) \rho$ guarantees us a fixed point in $I$.
  There is a second mirrored case discussing the bottom part of the triangle omitted for brevity.

  Why do we want this?
  We need $y = x$ to intersect with the bowtie because that is where $g$ lives, from our Lipschitz condition.
  The premise of fixed points is that our function intersects with $y = x$, the same as how our function intersects with $y = 0$ when root-finding.
  Otherwise, we are not guaranteed that a fixed point exists in this interval, as $g$ could \enquote{escape} in between $y = x$ and the top portion of the bowtie.

  \begin{theorem}[Convergence Theorem 2]\label{thm:convergence2}
    If \begin{enumerate}
      \item[$(\star)$] $g(x)$ has a fixed point $\alpha$ in the interval $I = [\alpha - \rho, \alpha + \rho]$ and
      \item[$(\star\star)$] $g$ satisfies $\abs{g(x) - g(\alpha)} \leq \lambda \abs{x - \alpha}$ for all $x \in I$ with $\lambda < 1$,
      \end{enumerate} then for all choice of $x_0 \in I$, \begin{enumerate}
      \item $x_i \in I$ for all $i$,
      \item $\lim\limits_{i \to \infty} x_i = \alpha$,
      \item the fixed point $\alpha$ is unique in $I$.
    \end{enumerate}
    \begin{proof}\leavevmode
      \begin{enumerate}
        \item We prove $x_i \in I$ by induction. \begin{enumerate}
          \item $x_0 \in I$ by our hypothesis.
          \item $x_i \in I$ for $i \leq m - 1$.
            Since $g(\alpha) = \alpha$, we have \begin{align*}
              \abs{\alpha - x_m} &= \abs{g(\alpha) - g(x_{m - 1})} \\
                                 &\leq \lambda \abs{\alpha - x_{m - 1}} && \text{by hypothesis $(\star\star)$} \\
                                 &\leq \lambda \rho \\
                                 &< \rho && \text{as $\lambda < 1$},
            \end{align*} so $x_m \in I$.
            Furthermore, if we look at these as errors, the $m$\textsuperscript{th} error is bounded above by the $(m - 1)$\textsuperscript{th} error (as $\lambda < 1$).
        \end{enumerate}

      \item We have that \begin{align*}
          \abs{\alpha - x_m} &\leq \lambda \abs{\alpha - x_{m - 1}} \\
                             &\vdotswithin{\leq} && \text{applying $(\star\star)$ recursively} \\
                             &\leq \lambda^m \abs{\alpha - x_0},
          \end{align*} so \begin{align*}
          \lim_{i \to \infty} \abs{\alpha - x_i} &\leq \lim_{i \to \infty} \lambda^i \abs{\alpha - x_0} \\
                                                 &= 0 && \text{since $\lambda^i \to 0$ as $i \to \infty$},
        \end{align*} thus it must be the case that $\lim\limits_{i \to \infty} x_i = \alpha$.

      \item Uniqueness follows as in the proof for Theorem~\ref{thm:iterscalar}.
        \qedhere
    \end{enumerate}
  \end{proof}
\end{theorem}
Now we prove a corollary similar to Corollary~\ref{cor:iterscalar} for Theorem~\ref{thm:iterscalar}.
This will highlight the role of $g'$ in the utility of our iterative scheme.
\begin{corollary}\label{cor:upperboundg'}
  If we replace premise $(\star\star)$ with $\abs{g'(x)} \leq \lambda < 1$ for $x \in I$, then results (1), (2), and (3) of Theorem~\ref{thm:convergence2} hold.
  \begin{proof}
    As in the proof of Corollary~\ref{cor:iterscalar}, we will use the Mean Value Theorem.
    By the Mean Value Theorem, there exists $\xi_{i - 1}$ between $\alpha$ and $x_{i - 1}$ such that \begin{align*}
      g'(\xi_{i - 1}) &= \frac{g(\alpha) - g(x_{i - 1})}{m - x_{i - 1}} \\
                      &= \frac{\alpha - x_i}{\alpha - x_{i - 1}} \\
      \abs{g'(\xi_{i - 1})} \abs{\alpha - x_{i - 1}} &= \abs{\alpha - x_i} \\
      \lambda \abs{\alpha - x_{i - 1}} &\geq  && \text{by our hypothesis.}
    \end{align*}
    Thus our Lipschitz condition is satisfied, and Theorem~\ref{thm:convergence2} will follow.
  \end{proof}
\end{corollary}

\begin{table}
  \centering
  \begin{tabular}{l l l}
    \toprule
  & Theorem~\ref{thm:iterscalar} & Theorem~\ref{thm:convergence2} \\
  \midrule
    \multirow{2}{*}{Conditions} & $\abs{g(x) - g(y)} \leq \lambda \abs{x - y}$ & $\abs{g(x) - g(\alpha)} \leq \lambda \abs{x - \alpha}$ \\
                                & $\abs{x_0 - g(x_0)} \leq (1 - \lambda) \rho$ & $x_0 \in I$ \\
                                \midrule
    Result & $\exists! \alpha$ & $\lim\limits_{i \to \infty} \lambda_i = \alpha$ for all $x_0 \in I$ \\
    \bottomrule
  \end{tabular}
  \caption{Differences between Theorem~\ref{thm:iterscalar} and Theorem~\ref{thm:convergence2}.}
\end{table}

$g'$ is important because it stands in as our Lipschitz condition number.
When the condition number is smaller, we assume that our iterates approach our fixed point quicker.

If $x_i$ converges to $\alpha$, then $\xi_i$ converges to $\alpha$ as well as it is is between $x_i$ and $\alpha$.
From this convergence and our previous Mean Value Theorem iterated, \begin{align*}
  \abs{\alpha - x_{k + m}} &= \abs{g'(\xi_{k + m - 1})} \abs{\alpha - x_{k + m - 1}} \\
                           &\vdotswithin{=} \\
                           &= \abs{g'(\xi_{k + m - 1})} \dots \abs{g'(\xi_k)} \abs{\alpha - x_k} \\
                           &\approx \abs{g'(\alpha)}^m \abs{\alpha - x_k}
\end{align*} for large enough $x_i$ so that $\xi_k$ is close to $\alpha$.
Thus our iterates get closer by approximately a factor of $g'(\alpha)$ each iteration.

We call $\hat{\lambda} \equiv \abs{g'(\alpha)}$ the (asymptotic) convergence factor.
When $g'(\alpha) \neq 0$, $g$ is a first order method.

\subsection{Error Propagation}
When computing $g(x)$, we will be limited by machine precision.
Given a values $x$, we represent our approximation of $g(x)$ by $G(x) = g(x) + \delta(x)$, where $\delta(x)$ is the error committed when we evaluate.

\begin{example}
  Let $g(x) = \frac{x}{3}$.
  Then $g(1) = \frac{1}{3}$, but if $G(1) = 0.33$, then $\delta(1) = 0.00\bar{3}$.
\end{example}
In practice, we can obtain a bound $\delta$ on $\delta(x)$.
For our iteration schemes, we write $X_{i + 1} = g(X_i) + \delta_i$, where $i = 0, 1, 2, \dots$,
where $X_i$ are the calculated iterates and $\delta_i$ are the errors added each iteration.
Because we cannot store infinite digits (that is, $\delta_i$ is always present), we cannot expect $X_i$ to converge all the time.

We still expect $X_i$ to approximate a fixed point.
The accuracy of the approximation is determined by $\delta$, the maximum value of the error term.

\begin{example}
  Let $g(x) = \alpha + \lambda (x - \alpha)$.
  We can easily verify that our fixed point is $g(\alpha) = \alpha$.
  Observe that $y = g(x) - \delta$ intersects $y = x$ at $x = \alpha - \frac{\delta}{1 - \lambda}$ and $y = g(x) + \delta$ intersects $y = x$ at $x = \alpha + \frac{\delta}{1 - \lambda}$.
  Thus our uncertainty in $\alpha$ lies in the range $\pm \frac{\delta}{1 - \lambda}$.

  If $\lambda$ is close to 1, then the range of uncertainty becomes large!
  Our estimates of $g(x)$ might leave $I$.
  We say that the problem isn't \enquote{properly posed} if $\lambda$ is close to 1.
\end{example}

\begin{theorem}[Error propagation theorem]
  Let $g$ satisfy the conditions of Theorem~\ref{thm:convergence2}
  Let $X_0$ be any point \begin{enumerate}
    \item[$(\star)$] in the interval $I = [\alpha - \rho_0, \alpha + \rho_0]$,
    \item[$(\star\star)$] where $\rho_0$ is given by $0 < \rho_0 \leq \rho - \frac{\delta}{1 - \lambda}$ (taking care of the \enquote{properly posed} problem).
  \end{enumerate}
  Then the iterates $X_i$ with errors bounded by $\delta$ lie in the interval $\abs{\alpha - X_i} < \rho$ and $\abs{\alpha - X_i} \leq \frac{\delta}{1 - \lambda} + \lambda^i \left( \rho_0 - \frac{\delta}{1 - \lambda} \right)$.

  \begin{proof}
    We prove $X_i \in I$ by induction.
    \begin{enumerate}
      \item First show $X_0 \in I$.
        By rearranging premise $(\star\star)$, we get \begin{align*}
          \abs{\alpha - X_0} &\leq \rho_0 && \text{by condition $(\star)$} \\
                             &\leq \rho_0 + \frac{\delta}{1 - \lambda} && \text{as $\frac{\delta}{1 - \lambda}$ is positive} \\
                             &\leq \rho - \frac{\delta}{1 - \lambda} + \frac{\delta}{1 - \lambda} && \text{by condition $(\star\star)$} \\
                             &= \rho.
        \end{align*}

      \item Now assume $X_i \in I$ for $i \leq m - 1$.
        \begin{align*}
          \abs{\alpha - X_m} &= \abs{g(\alpha) - \left[ g(X_{m - 1}) + \delta_{m - 1} \right]} \\
                             &\leq \abs{g(\alpha) - g(X_{m - 1})} + \abs{-\delta_{m - 1}} && \text{by the triangle inequality} \\
                             &\leq \lambda \abs{\alpha - X_{m - 1}} + \delta \\
                             &\leq \lambda^2 \abs{\alpha - X_{m - 2}} + \lambda \delta  + \delta && \text{by our Lipschitz condition} \\
                             &\vdotswithin{\leq} \\
                             &\leq \lambda^m \abs{\alpha - X_0} + \lambda^{m - 1} \delta + \cdots + \lambda \delta + \delta \\
                             &= \lambda^m \rho_0 + \left( \frac{1 - \lambda^m}{1 - \lambda} \right) \delta \\
                             &= \frac{\delta}{1 - \lambda} - \lambda^m \left( \rho_0 - \frac{\delta}{1 - \lambda} \right) && \text{giving us the second result}.
        \end{align*}
        This separates the computational error from the $m$-dependent terms.
    \end{enumerate}
    We still must prove the first result.
    Continuing from before distributing \begin{align*}
      \abs{\alpha - X_i} &\leq \lambda^i \rho_0 + \left( \frac{1 - \lambda^i}{1 - \lambda} \right) \delta \\
                         &= \lambda^i \rho_0 + \frac{\delta}{1 - \lambda} - \lambda^i \frac{\delta}{1 - \lambda} \\
                         &\leq \rho_0 + \frac{\delta}{1 - \lambda} \\
                         &\leq \rho && \text{by premise $(\star\star)$.}
    \end{align*}
    Therefore, $X_i \in I$, and our process is well-defined.
  \end{proof}
\end{theorem}

\subsection{Understanding convergence factor}
How does the convergence factor relate to the number of iterations we need?
From Theorem~\ref{thm:convergence2} we have $\abs{\alpha - X_{m + k}} \approx \hat{\lambda}^k \abs{\alpha - X_m}$.
To increase accuracy by $10^{-m}$, we need $\frac{\abs{\alpha - X_{m + k}}}{\abs{\alpha - X_m}} = 10^{-m} \approx \hat{\lambda}^k$, so $k = \frac{m}{\log \frac{1}{\hat{\lambda}}}$.
Taking into account the resolution from machine precision, we only need to iterate up to \begin{displaymath}
  \min\left( \frac{m}{\log \frac{1}{\hat{\lambda}}}, \underbrace{\frac{\log \frac{\delta}{(1 - \hat{\lambda}) \rho_0}}{\log \hat{\lambda}}}_\text{resolution bound} \right)
\end{displaymath} times.
However, we can take fewer iterates by breaking this condition from our first order methods: $\hat{\lambda} = \abs{g'(\alpha)} \neq 0$.

\section{Second and higher order iteration methods}
To get an intuition of why this helps, suppose $g'(\alpha) = 0$.
If the slope at our fixed point is zero, we expect the slope around the fixed point to be close to zero, i.e.\ $0 < \abs{g'(x)} \ll 1$ (much less than 1) around $\alpha$.

Furthermore, we will assume that $g'(\alpha) = 0$ and $g''(x)$ exists and is bounded in $I \equiv [\alpha - \rho, \alpha + \rho]$.
Then for all $x \in I$, we can use Taylor expansion to get \begin{align*}
  g(x) &= g(\alpha) + g'(\alpha)(x - \alpha) + g''(\xi) \frac{{(x - \alpha)}^2}{2} \\
       &= g(\alpha) + g''(\xi) \frac{{(x - \alpha)}^2}{2} && \text{by our first assumption} \\
       &= \alpha + g''(\xi) \frac{{(x - \alpha)}^2}{2}
\end{align*} where $\xi$ is between $x$ and $\alpha$ by Taylor's theorem due to our second assumption.

From this result, for any $i > 0$, \begin{align*}
  \underbrace{\abs{x_i - \alpha}}_\text{$i$\textsuperscript{th} error} &= \abs{g(x_{i - 1}) - g(\alpha)} \\
                                                                       &= \abs{\alpha - g''(\xi_{i - 1}) \frac{{(x_{i - 1} - \alpha)}^2}{2} - \alpha} \\
                                                                       &= \frac{1}{2} \underbrace{\left\vert g''(\xi_{i - 1}) \right\vert}_\text{bounded} {\underbrace{\left\vert x_{i - 1} - \alpha \right\vert}_\text{previous error}}^2
\end{align*} which is a significant improvement.

Assuming that $g''(\alpha) \neq 0$, we call this a second order method.

\subsection{Convergence Factor and Error Propagation}
Let the bound on $g''$ be denoted as $\abs{g''(x)} \leq 2M$ for all $x \in I$.
Then \begin{align*}
  \underbrace{\abs{x_i - \alpha}}_\text{$i$\textsuperscript{th} error} &\leq \frac{1}{2} (2M) \abs{x_{i - 1} - \alpha}^2 \\
                                                                       &= M \abs{x_{i - 1} - \alpha}^2 \\
                                                                       &\leq M \cdot M^2 \abs{x_{i - 2} - \alpha}^4 \\
                                                                       &\leq M \cdot M^2 \cdot M^4 \abs{x_{i - 3} - \alpha}^8 \\
                                                                       &\vdotswithin{\leq} && \text{using Taylor expansion} \\
                                                                       &\leq M^{2^i - 1} \abs{x_0 - \alpha}^{2^i} \\
                                                                       &\leq {(M \abs{x_0 - \alpha})}^{2^i - 1} \underbrace{\abs{x_0 - \alpha}}_\text{initial error},
\end{align*}
so we converge if $M \abs{x_0 - \alpha} = \hat{\lambda} < 1$.

Suppose we wanted to reduce error by at least $10^{-m}$.
Then $\abs{X_{i + k} - \alpha} \leq 10^{-m} \abs{X_i - \alpha}$, and thus $k \approx \frac{1}{\log 2} \log \frac{m}{\log \left( \hat{\lambda} \frac{1}{\rho} \right)}$.

\subsection{Comparison to the first order scheme}
To get equal reduction of error between our first order scheme conversion factor $\hat{\lambda}_1$ and our second order scheme conversion factor $\hat{\lambda}_2$, $\hat{\lambda}_1^{k_1} = \hat{\lambda}_2^{2^{k_2}}$.
In Corollary~\ref{cor:upperboundg'} we thought of $\hat{\lambda}_1$ as $g'$, and now we think of $\hat{\lambda}_2$ as $g'' \rho$.
Assuming $\hat{\lambda}_1 = \hat{\lambda}_2$, it is easy to see that $k_1 = 2^{k_2}$, which means we need to do $2^{k_2}$ iterations to get the same reduction of error using the first order scheme.
For example, if we need 5 more iterations to get the necessary increase in precision using the second order scheme, this translates to needing 32 more iterations with the first order scheme to see the same improvement.
This basically means that the second order scheme is exponentially better than the first.

\begin{example}
  Find the root of $f(x) = x^2 - 2$.
  \begin{displaymath}
    \begin{array}{c c}
      \toprule
      \text{First order scheme} & \text{Second order scheme} \\
      \midrule
      \begin{aligned}
        g'_1(x) &= x - \frac{1}{2} (x^2 - 2) \\
        g'_1(x) &= 1 - x \\
        g'_1(\sqrt{2}) &= 1 - \sqrt{2}
      \end{aligned} &
      \begin{aligned}
        g_2(x) &= \frac{x}{2} + \frac{1}{x} \\
        g'_2(x) &= \frac{1}{2} - \frac{1}{x^2} \\
        g'_2(\sqrt{2}) &= \frac{1}{2} - \frac{1}{2} = 0
      \end{aligned} \\
      \bottomrule
    \end{array}
  \end{displaymath}
  Let $x_0 = 1$.
  Then we get the following values for $g_1$ and $g_2$, assuming rounding to 5 digits:
  \begin{table}[h]
    \centering
    \begin{tabular}{l S S}
      \toprule
      Iteration & {$g_1$} & {$g_2$} \\
      \midrule
      $x_0$ & 1. & 1. \\
      $x_1$ & 1.5 & 1.5 \\
      $x_2$ & 1.375 & 1.4167 \\
      $x_3$ & 1.42969 & 1.41422 \\
      $x_4$ & 1.40768 & 1.41421 \\
      $x_5$ & 1.4169 & 1.41421 \\
      $x_6$ & 1.4131 & 1.41421 \\
      $x_7$ & 1.41467 & 1.41421 \\
      $x_8$ & 1.41402 & 1.41421 \\
      $x_9$ & 1.41429 & 1.41421 \\
      $x_{10}$ & 1.41418 & 1.41421 \\
      \bottomrule
    \end{tabular}
  \end{table}

  Notice that $g_2$ is already accurate to 2 decimal places by the second iteration, and accurate to 5 decimal places by the fourth iteration, while $g_1$ requires many more iterations to reach that accuracy.
  It turns out that for every iteration of a second order scheme, the number of correct digits doubles, assuming $M < 1$ (when $M > 1$ it is asymptotically true).

  \begin{proof}
    Assume that $\abs{x_m - \alpha} \leq 10^{-k_m}$ for all $m \geq m_0$; i.e.\ past this point, our iterates are correct to $k_m$ decimals.
    Then we have that \begin{align*}
      \abs{x_m - \alpha} &\leq M {\left\vert x_{m - 1} - \alpha \right\vert}^2 \\
      10^{-k_m} &\leq M \cdot 10^{-2k_{m - 1}}
    \end{align*} or $k_m \geq {2 k_{m - 1}}^{-\log M}$,
    so for $M < 1$ we have $-\log M > 0$.
    Thus the number of correct decimals \emph{more than doubles} each iteration (which is only asymptotically true when $M > 1$).
  \end{proof}

  In practice, we want to check the convergence criterion; that is, $g'(x) < 1$ in $I \equiv [\sqrt{2} - \rho, \sqrt{2} + \rho]$.
\end{example}

\subsection{Higher order schemes}
Imagine what would happen in an even better case scenario.
Following the same pattern, assume $g'(\alpha) = g''(\alpha) = \cdots = g^{(n - 1)}(\alpha) = 0$ and $g^{(n)}(\alpha) \neq 0$, with $\abs{g^{(n)}(x)} \leq n!\, M$
in $\abs{x - \alpha} \leq \rho$.
Now consider $g(x) = \alpha + g'(\alpha)(x - \alpha) + \cdots + g^{(n)}(\xi) \frac{{(x - \alpha)}^n}{n!}$, where $\xi$ is between $x$ and $\alpha$ and all intermediate terms are 0 by our first assumption.
Then as from before, we can express our $i$\textsuperscript{th} error as \begin{align*}
  \abs{x_{i + 1} - \alpha} &= \frac{\abs{g^{(n)}(\xi_i)}}{n!} \abs{x_i - \alpha}^n \\
                           &\leq M \abs{x_i - \alpha}^n
\end{align*} by our second assumption, giving us our convergence factor.
This is called an $n$\textsuperscript{th} order procedure.
It multiplies the number of correct decimals by $n$ each iteration (at least asymptotically).

\subsubsection{Error propagation}
Since $g'(\alpha) = 0$, then the upper bound $\lambda$ is close to 0, so the fudge factor $\pm \frac{\delta}{1 - \delta}$ becomes $\pm \delta$.
Thus error propagation isn't really a concern with higher order procedures.

\section{Explicit iteration procedures}
Going back to the original task of finding roots of any scalar equation $f(x) = 0$ in some interval $a \leq x \leq b$,
let $\phi(x)$ be any function such that $0 < \abs{\phi(x)} < \infty$.
Now our placeholder function $g \equiv x - f(x)$ will be $g(x) \equiv x - \phi(x) f(x)$.
Is this suitable for root finding (i.e.\ does a fixed point in $g$ yield a root in $f$)?
Assuming $g(\alpha) = \alpha$, we get $\alpha = \alpha - \phi(\alpha) f(\alpha)$; that is, $\phi(\alpha) f(\alpha) = 0$.
Since $0 < \abs{\phi(x)}$, the only factor that can approach 0 is $f(\alpha)$, so $\alpha$ is a root of $f$.
There are many different choice of $\phi(x)$ that will lead to different iterative procedures.

An alternative method of going from fixed point to root is composition.
Let $g(x) \equiv x - F(f(x))$, where $F$ is a function such that $F(y) = 0$ if and only if $y = 0$.
Composition methods often describe higher order schemes.

\subsection{Chord Method}
The simplest choice for $\phi(x)$ is to let $\phi(x) \equiv m \neq 0$, where $m \in \mathbb{R}$.
If $f$ is differentiable, by Corollary~\ref{cor:iterscalar}, we get convergence when $\abs{g'(x)} < 1$.
Looking at $g'$, we see that \begin{align*}
  g(x) &= x - m f(x) \\
  g'(x) &= 1 - m f'(x),
\end{align*}
so for convergence, \begin{equation}
  \abs{g'(x)} = \abs{1 - m f'(x)} < 1
\end{equation} or
\begin{equation}\label{eqn:chordconvergence2}
  0 < m f'(x) < 2.
\end{equation}
From these inequalities, we see that $m$ must have the same parity as $f'$, and if $f'(x) = 0$, then the convergence condition cannot be satisfied.
This means that it does not work for higher order schemes, and we don't want $f$ to be constant in our interval when using this method.

\subsubsection{Visualizing the chord method}
By $x_{i + 1} = g(x_i)$, we have $x_{i + 1} = x_i - m f(x_i)$.
Given the point $p_i = (x_i, f(x_i))$, we use the point-slope formula to find the line starting at $p_i$ with slope $s$ is $y - f(x_i) = s (x - x_i)$.
Suppose the line has an $x$-intercept of $x_{i + 1}$.
Then it also goes through $(x_{i + 1}, 0)$ and we see that \begin{align*}
  - f(x_i) &= s (x_{i + 1} - x_i) \\
           &= s (-m f(x_i)) && \text{rearranging $x_{i + 1} = x_i - m f(x_i)$ from above} \\
  1 &= sm;
\end{align*} i.e.\ the slope of the line is $\frac{1}{m}$.
Thus the next iteration is the $x$-intercept of the line that passes through the previous iteration's point with a slope of $\frac{1}{m}$.
Eventually this will yield a root.

Note that from Equation~\ref{eqn:chordconvergence2}, we have $\frac{1}{2} f'(x) < \frac{1}{m} < \infty$, which means our slope is greater than half of the tangent line and is not vertical.

\subsubsection{Implementation in Mathematica}
\begin{minted}[mathescape]{mathematica}
f[x_] := x^2 - 2; (* the method we want to find the root of *)
g[x_] := x - 0.5 * f[x]; (* if it doesn't converge, make $m$ (0.5) smaller *)

a = 10; (* Initial guess $x_0$ *)
thresh = 10^(-3); (* residual kick out threshold *)
maxiter = 10; (* maximum number of iterations *)
For[iter = 0, iter < maxiter && Abs[f[a]] > thresh, a = g[a]; Print[a]; iter++]
\end{minted}

\subsection{Newton's Method}
Suppose we are able to adapt the slope of our chord each iteration, so that $g'(x_i) = 1 - \phi(x_i) f'(x_i) = 0$.
Then we have a second order method.
The only choice we have for $\phi$ is $\phi(x) = \frac{1}{f'(x)}$, so $g(x) = x - \frac{\phi(x)}{f'(x)}$.
Observe that with this, we must have $f'(x) \neq 0$ for all $x \in I$.

\subsubsection{Visualization}
Our visualization follows right from the Chord Method, except the chord now has a slope of $\frac{1}{f'(x)}$, tangent to each point.
The lines adjust to look for the root rather than being always parallel.

\subsection{Method of False Position}
So far we've only seen root-solving applications, but this analysis is usually used for solving differential equations.
Sometimes it's not straightforward how to evaluate $f'$, which happens with ordinary differential equations when $f$ is not explicitly given.
However, we can approximate derivatives using divided differences.

Using a secant approximation of $m = \frac{1}{f'(x_i)}$ instead of tangent lines,
we get the procedure \begin{equation}\label{eqn:falseposition}
  x_{i + 1} = x_i - f(x_i) \frac{x_i - x_{i - 1}}{f(x_i) - f(x_{i - 1})}
\end{equation} for $i \in \mathbb{N}$.
Note that we need two successive iterations to calculate $x_{i + 1}$, unlike in Newton's Method, where we only need one.
This means we cannot write our procedure as $x_{i + 1} = g(x_i)$ anymore and therefore will need to express it as some $h(x_i, x_{i - 1})$.
To examine this, let $\alpha$ be a root of $f(x) = 0$.
Then from Equation~\ref{eqn:falseposition} we have \begin{equation}
  \begin{split}\label{eqn:divdiff}
    \alpha - x_{i + 1} &= \left( \alpha - x_i \right) + f(x_i) \frac{x_i - x_{i - 1}}{f(x_i) - f(x_{i - 1})} \\
                       &= \left( \alpha - x_i \right) \frac{f[x_{i - 1}, x_i] - f[x_i, \alpha]}{f[x_{i - 1, x_i}]} \\
                       &= (\alpha - x_i) (\alpha - x_{i - 1}) \left( - \frac{f[x_{i - 1}, x_i, \alpha]}{f[x_{i - 1}, x_i]} \right)
  \end{split}
\end{equation}
where $f[a, b] \equiv \frac{f(b) - f(a)}{b - a}$ is the first divided difference and $f[x_{i - 1}, x_i, \alpha] \equiv \frac{f[x_{i - 1}, x_i] - f[x_i, \alpha]}{x_{i - 1} - \alpha}$ is the second divided difference.

If $f$ has a continuous second derivative in an interval including the points $x_i$, $x_{i - 1}$, and $\alpha$, then by Taylor's Theorem there exists $\xi_i$ and $\eta_i$  such that $f[x_{i - 1}, x_i] = f'(\xi_i)$ and $f[x_{i - 1}, x_i, \alpha] = \frac{1}{2} f''(\eta_i)$.
So by Equation~\ref{eqn:divdiff} we have \begin{displaymath}
  \alpha - x_{i + 1} = \frac{f''(\eta_i)}{2 f'(\xi_i)} (\alpha - x_i) (\alpha - x_{i - 1})
\end{displaymath} for $i \in \mathbb{N}$,
assuming $x_i \in I \equiv [\alpha - \rho, \alpha + \rho]$ for all $i$ and
$\abs{\frac{f''(\eta)}{2 f'(\xi)}} \leq M < \frac{1}{\rho}$ for $\eta, \xi \in I$.

Let $e_i \equiv M \abs{\alpha - x_i}$, then we have \begin{align*}
  \abs{\alpha - x_{i + 1}} &= \abs{\frac{f''(\eta_i)}{2 f'(\xi_i)}} \abs{\alpha - x_i} \abs{\alpha - x_{i - 1}} \\
  \frac{e_{i + 1}}{M} &\leq M \cdot \frac{e_i}{M} \cdot \frac{e_{i - 1}}{M} \\
  e_{i + 1} &\leq e_i e_{i - 1}.
\end{align*}
This says that the subsequent error is bounded by the product of the two previous errors.

Define $\delta \equiv \max(e_0, e_1)$, then we can continue and get the inequalities \begin{align*}
  e_0 &\leq \delta \\
  e_1 &\leq \delta \\
  e_2 &\leq \delta^2 \\
  e_3 &\leq \delta^3 \\
  e_4 &\leq \delta^5 \\
      &\vdotswithin{\leq} \\
  e_i &\leq \delta^{m_i}
\end{align*} where $m_0 = m_1 = 1$ and $m_{i + 1} = m_i + m_{i - 1}$; i.e. $m_i$ is the $i$\textsuperscript{th} element of the Fibonacci sequence.
We call this a golden ratio order procedure as our errors go to zero at an order of roughly 1.618.

\section{Systems of Linear Equations}
Now we look at systems of linear equations: many simple equations that have to be satisfied concurrently.
It turns out that many mathematical problems boil down to just solving a system of linear equations.

\begin{example}
  Michael buys two bags of chips and three boxes of pretzels for \$7.
  He then buys another bag of chips and two more boxes of pretzels for \$4.
  Find the cost of each bag of chips and each box of pretzels.

  Let $c$ be the price of one bag of chips and $p$ the price of one box of pretzels.
  Recall that scalar multiplication and addition are linear operations, and two equations have to be satisfied at the same time, hence this is a system of two linear equations: \begin{align*}
    2c + 3p &= 7 \\
    c + 2p &= 4,
  \end{align*} which can be solved by reframing in terms of matrices and vectors such that $A \vec{x} = \vec{b}$, where $A$ is the coefficient matrix, $\vec{x}$ is a vector of unknowns (or solutions), and $\vec{b}$ is a vector of constants:
  \begin{displaymath}
    \begin{bmatrix}
      2 & 3 \\
      1 & 2
    \end{bmatrix}
    \begin{bmatrix}
      c \\
      p
    \end{bmatrix} =
    \begin{bmatrix}
      7 \\
      4
    \end{bmatrix}.
  \end{displaymath}
  This equation could be solved by hand using Gaussian Elimination.
\end{example}

\subsection{LU Factorization}
Gaussian Elimination takes nonsingular (invertible) linear systems and gives an upper triangular one by applying simple linear transformations.

An upper triangular matrix $U$ has $u_{ij} = 0$ when $i > j$ (zeros beneath the main diagonal), while a lower triangular matrix $L$ has $l_{ij} = 0$ when $i < j$ (zeros above the main diagonal).
Solving the system of linear equations above becomes quite simple once $A$ has been factored into its $L$ and $U$ components.

Let $A \in \mathbb{C}^{m \times m}$ be a square matrix.
To go from $A$ to $U$ we need to introduce zeros below the diagonal.
This \enquote{elimination} is done column by column, from left to right, by multiplying lower triangular matrices (linear transformations) on the left of $A$.
The process essentially applies linear transformations to the system of linear equations in order to make a different system that is easier to solve: \begin{align*}
  L_{m - 1} \cdots L_2 L_1 A &= U && \text{where $L_i$ has zeros in column $i$} \\
  A &= L_1^{-1} L_2^{-1} \cdots L_{m - 1}^{-1} U \\
  A &= LU && \text{where $L = L_1^{-1} \cdots L_{m - 1}^{-1}$}.
\end{align*}
Having the actual factorization of $A$ isn't necessary, but can be helpful in some other aspects.
We will see $L$ is lower triangular due to the nature of these linear transformations.

\begin{figure}[h]
  \centering
  \begin{tikzpicture}[
    auto,
    Matrix/.style={matrix of nodes,align=center,left delimiter={[},right delimiter={]},inner xsep=1pt},
    Highlight/.style={fill,rounded corners=3pt,fill opacity=0.05},
    Line/.style={->,line width=5pt,draw opacity=0.3},
    ]

    \matrix[Matrix,label=$A$] at (0,0) (A) {
      X & X & X & X \\
      X & X & X & X \\
      X & X & X & X \\
      X & X & X & X \\
    };
    \matrix[Matrix,label=$\textcolor{blue}{L_1} A$] at (4,0) (LA) {
      X & X & X & X \\
      0 & X & X & X \\
      0 & X & X & X \\
      0 & X & X & X \\
    };
    \matrix[Matrix,label=$\textcolor{red}{L_2} \textcolor{blue}{L_1} A$] at (8,0) (LLA) {
      X & X & X & X \\
      0 & X & X & X \\
      0 & 0 & X & X \\
      0 & 0 & X & X \\
    };
    \matrix[Matrix,label=$\textcolor{green}{L_3} \textcolor{red}{L_2} \textcolor{blue}{L_1} A$] at (12,0) (LLLA) {
      X & X & X & X \\
      0 & X & X & X \\
      0 & 0 & X & X \\
      0 & 0 & 0 & X \\
    };

    \draw[Highlight,blue]  (  LA-2-2.north west) rectangle (  LA-4-4.south east);
    \draw[Highlight,blue]  ( LLA-2-2.north west) rectangle ( LLA-2-4.south east);
    \draw[Highlight,red]   ( LLA-3-3.north west) rectangle ( LLA-4-4.south east);
    \draw[Highlight,blue]  (LLLA-2-2.north west) rectangle (LLLA-2-4.south east);
    \draw[Highlight,red]   (LLLA-3-3.north west) rectangle (LLLA-3-4.south east);
    \draw[Highlight,green] (LLLA-4-4.north west) rectangle (LLLA-4-4.south east);

    \draw[Line,blue]    (A) -- (LA) node   [midway,above] {\textcolor{blue}  {$L_1$}};
    \draw[Line,red]    (LA) -- (LLA) node  [midway,above] {\textcolor{red}   {$L_2$}};
    \draw[Line,green] (LLA) -- (LLLA) node [midway,above] {\textcolor{green} {$L_3$}};
  \end{tikzpicture}
  \caption{The linear transformations performed when factoring a $4 \times 4$ matrix $A$ into its upper triangular form.
    Non-zero values are represented by $X$, with highlighted regions showing which values change as linear transformations are applied.
  }
\end{figure}

In general, the $k$\textsuperscript{th} linear transformation introduces zeros below the diagonal in column $k$.
It does this by subtracting multiples of row $k$ from rows under row $k$.
Since columns $1, \dots, k$ already have zeros in the proper location, the transformation does not introduce nonzeros into those elements.

\begin{example}
  Now that we understand shape, let \begin{displaymath}
    A = \begin{bmatrix}
      2 & 1 & 1 & 0 \\
      4 & 3 & 3 & 1 \\
      8 & 7 & 9 & 5 \\
      6 & 7 & 9 & 8
    \end{bmatrix}.
  \end{displaymath}
  The first step is to introduce zeros into the first column: \begin{displaymath}
    L_1 A = \underbrace{\begin{bmatrix*}[r]
        1 & 0 & 0 & 0 \\
        -2 & 1 & 0 & 0 \\
        -4 & 0 & 1 & 0 \\
        -3 & 0 & 0 & 1
    \end{bmatrix*}}_{L_1}
    \underbrace{\begin{bmatrix}
        2 & 1 & 1 & 0 \\
        4 & 3 & 3 & 1 \\
        8 & 7 & 9 & 5 \\
        6 & 7 & 9 & 8
    \end{bmatrix}}_A =
    \begin{bmatrix}
      2 & 1 & 1 & 0 \\
      0 & 1 & 1 & 1 \\
      0 & 3 & 5 & 5 \\
      0 & 4 & 6 & 8
    \end{bmatrix},
  \end{displaymath}
  then do the same with the second column: \begin{displaymath}
    L_2 L_1 A = \underbrace{\begin{bmatrix*}[r]
        1 & 0 & 0 & 0 \\
        0 & 1 & 0 & 0 \\
        0 & -3 & 1 & 0 \\
        0 & -4 & 0 & 1
    \end{bmatrix*}}_{L_2}
    \underbrace{\begin{bmatrix}
        2 & 1 & 1 & 0 \\
        0 & 1 & 1 & 1 \\
        0 & 3 & 5 & 5 \\
        0 & 4 & 6 & 8
    \end{bmatrix}}_{L_1 A} =
    \begin{bmatrix}
      2 & 1 & 1 & 0 \\
      0 & 1 & 1 & 1 \\
      0 & 0 & 2 & 2 \\
      0 & 0 & 2 & 4
    \end{bmatrix},
  \end{displaymath}
  and the third column: \begin{displaymath}
    L_3 L_2 L_1 A =
    \underbrace{\begin{bmatrix*}[r]
        1 & 0 & 0 & 0 \\
        0 & 1 & 0 & 0 \\
        0 & 0 & 1 & 0 \\
        0 & 0 & -1 & 1
    \end{bmatrix*}}_{L_3}
    \underbrace{\begin{bmatrix}
        2 & 1 & 1 & 0 \\
        0 & 1 & 1 & 1 \\
        0 & 0 & 2 & 2 \\
        0 & 0 & 2 & 4
    \end{bmatrix}}_{L_2 L_1 A} =
    \begin{bmatrix}
      2 & 1 & 1 & 0 \\
      0 & 1 & 1 & 1 \\
      0 & 0 & 2 & 2 \\
      0 & 0 & 0 & 2
    \end{bmatrix}.
  \end{displaymath}
\end{example}
It is trivial to compute $L = L_1^{-1} L_2^{-1} L_3^{-1}$, as the inverse of a matrix in this form is simply the non-diagonal elements negated, so: \begin{displaymath}
  L = \begin{bmatrix}
    1 & 0 & 0 & 0 \\
    2 & 1 & 0 & 0 \\
    4 & 3 & 1 & 0 \\
    3 & 4 & 1 & 1
  \end{bmatrix}.
\end{displaymath}

\subsubsection{General Formula}
Suppose $x_k$ denotes the $k$\textsuperscript{th} column of the matrix.
Choose $L_k$ so that there are zeros everywhere below the diagonal element $x_{k k}$.
To do this, subtract $l_{jk}$ multiplied by row $k$ from row $j$, where $l_{jk} = \frac{x_{jk}}{x_{kk}}$ (the element to get rid of divided by the diagonal element) for all $j > k$.
We can verify that \begin{align*}
  L_k \cdot x_{k} &= -l_{k + 1, k} \cdot x_{kk} + i1 \cdot x_{k + 1, k} \\
                  &= - \frac{x_{k + 1, k}}{x_{kk}} \cdot x_{kk} + x_{k + 1,k} && \text{by the definition of $l$ above} \\
                  &= - x_{k + 1, k} + x_{k + 1,k} \\
                  &= 0,
\end{align*} which is what we expected.

\begin{claim}
  Two nice properties allow for a quick lower triangular matrix factorization:
  $L_k$ can be inverted by negating the subdiagonal entries ($l_{jk}^{-1} = -l_{jk}$ for $j > k$), and
  $L$ can be formed by inserting $L_k$ into $I$ below the diagonal (creating a matrix with $l_{jk}$ in the subdiagonal and 1 on main diagonal).

  \begin{proof}
    Define $l_k = \begin{bmatrix}
      0 & \cdots & 0 & l_{k + 1, k} & \cdots & l_{mk}
    \end{bmatrix}\tran$, which is the $k$\textsuperscript{th} transformation without the 1 in the diagonal element.
    This gives us the nice representation $L_k = I - l_k \cdot e_k\tran$, where $e_k$ is the standard basis vector.
    Note that $l_k$ is a $m \times 1$ matrix, and $e_k\tran$ is a $1 \times m$ matrix, so the dimensions of the outer product $l_k \cdot e_k\tran$ are $m \times m$.
    By the same reasoning, the dot product $e_k\tran \cdot l_k$ is a scalar: \begin{align*}
      e_k \cdot l_k &= 0 \cdot 0 + \cdots + 1 \cdot 0 + 0 \cdot l_{k + 1, k} + \cdots + 0 \cdot l_{mk} \\
                    &= 0,
    \end{align*} and therefore
    \begin{align*}
      L_k * L_k^{-1} &= (I - l_k e_k\tran) (I + l_k e_k\tran) \\
                     &= I + l_k e_k\tran - l_k e_k\tran - l_k e_k\tran l_k e_k\tran && \text{by the distributive property} \\
                     &= I - l_k e_k\tran l_k e_k\tran \\
                     &= I - l_k (e_k\tran l_k) e_k\tran && \text{by associativity of matrix multiplication} \\
                     &= I && \text{as the dot product is 0},
    \end{align*} showing that $L_k^{-1} = I + l_k e_k\tran$ is indeed the inverse of $L_k$.

    For the second part, consider the product \begin{align*}
      L_k^{-1} L_{k + 1}^{-1} &= (I + l_k e_k\tran) (I + l_{k + 1} e_{k + 1}\tran) \\
                              &= I + l_k e_k\tran + l_{k + 1} e_{k + 1}\tran + l_k e_k\tran l_{k + 1} e_{k + 1}\tran && \text{by the distributive property} \\
                              &= I + l_k e_k\tran + l_{k + 1} e_{k + 1}\tran + l_k (e_k\tran l_{k + 1}) e_{k + 1}\tran && \text{by associativity} \\
                              &= I + l_k e_k\tran + l_{k + 1} e_{k + 1}\tran && \text{as the dot product is 0}.
    \end{align*}
    In other words, $L_k^{-1} L_{k + 1}^{-1}$ is the identity matrix, plus the $k$\textsuperscript{th} and $(k + 1)$\textsuperscript{th} columns beneath their respective diagonal elements.
    This is a \enquote{unit} lower triangular matrix.
    It follows that the multipliers are put into the subdiagonal elements of the lower triangular matrix $L$ by the multiplier
    \begin{displaymath}
      L = L_1^{-1} L_2^{-1} \cdots L_{m - 1}^{-1} = \begin{bmatrix}
        1       &        &        &             &   \\
        l_{21}  & 1      &        &             &   \\
        l_{31}  & l_{32} & \ddots &             &   \\
        \vdots  & \vdots & \ddots & 1           &   \\
        l_{m1}  & l_{m2} & \cdots & l_{m,m - 1} & 1
      \end{bmatrix}.
      \qedhere
    \end{displaymath}
  \end{proof}

  Due to these two properties, $L_k$ is never formed and multiplied, and $l_{jk} = \frac{x_{jk}}{x_{kk}}$ for $j > k$.
  Note that the diagonal entries $x_{kk}$ change every time a linear transformation is applied.
\end{claim}

\subsubsection{LU Algorithm}
A more robust implementation of this is available in Julia as \mintinline{julia}{LinearAlgebra.lu(A)::LU}.
\begin{minted}{julia}
function lu(A)
    U = copy(A)
    L = one(A) # identity matrix of same dims
    m = size(A,1)
    for k in 1:(m - 1) # no entries below diagonal in the last column
        for j in (k + 1):m
            L[j, k] = U[j, k] / U[k, k]
            U[j, k:m] -= L[j, k] * U[k, k:m]
        end
    end
    L, U
end
\end{minted}

\end{document}


\documentclass{article}

\title{\sc\LARGE CSCA67 Tutorial, Week 4\\
{\Large Oct. 5th-Oct. 9rd, 2015}}
\date{}
\author{\sc Compiled by {\em G. Singh Cadieux}\\[1ex]
\sc Adapted from\\
A. Bretscher, \href{http://www.utsc.utoronto.ca/~bretscher/a67/lectures/w4.pdf}{\em CSCA67 Week 4 Lecture Notes},\\
\href{http://www.intmath.com/counting-probability/6-probability-event.php}{\em Interactive Mathematics: Probability of an Event},\\
\href{http://www.intmath.com/counting-probability/singapore-toto.php}{\em Interactive Mathematics: Singapore TOTO} \&\\
\href{http://www.intmath.com/counting-probability/poker.php}{\em Interactive Mathematics: Probability and Poker}}

\usepackage{fullpage}
\usepackage{amsmath,amssymb}
\usepackage{color}
%\usepackage{multicol}
\usepackage{tikz}
\usepackage{hyperref}

\begin{document}
\maketitle

\section{\sc Review of last week's lecture}
\subsection*{\em More probability}

\subsection{\em Probability function}
A \textsc{probability function} is a function on a \textit{discrete} (containing a finite or countably infinite number of elements) sample space $\mathcal{S}$ that has domain $\mathcal{S}$ and satisfies
\begin{enumerate}
\item $0\leq P(A)\leq 1\;\text{for every event }A\subseteq\mathcal{S}$\\[1ex]
(That is, the probability of any event $A$ in \textsl{S} is between 0 and 1.)
\item $\sum\limits_{s\in \mathcal{S}}P(s)=P(\mathcal{S})=1,\;P(\emptyset)=0$\\[1ex]
(That is, the sum of the probabilities of every outcome in $\mathcal{S}$ is 1.\\
The probability of the entire sample space $\mathcal{S}$ - i.e., of any outcome in $\mathcal{S}$ occurring - is also 1.\\
The probability of the empty set - i.e., of no outcome in $\mathcal{S}$ occurring - is 0.)
\item $P(A_1\cup A_2\cup\ldots)=P(A_1)+P(A_2)+\ldots$ for \textit{disjoint} events $A_1,\,A_2,\ldots\subseteq \mathcal{S}$\\[1ex]
(That is, the probability of multiple, disjoint events $A_1,\,A_2,\ldots$ in $\mathcal{S}$ occurring is the sum of their individual probabilities.)
\end{enumerate}
\begin{description}
\item[\sc Disjoint:]\hfill\\
Two or more events $A_1,\,A_2,\ldots$ are disjoint if $A_1\cap A_2\cap\ldots=\emptyset$.\\[1ex]
(That is, if $i\neq j$, then $A_i$ and $A_j$ do not contain any of the same outcomes.)\\[1ex]
Represented as a Venn diagram:\\
\begin{tikzpicture}
\draw (0,0) circle[radius=1.3];
\draw (0.3,0.6) circle[radius=0.6];
\draw (-.5,-0.3) circle[radius=0.5];
\node at (0.3,.6) {$A_i$};
\node at (-.5,-.3) {$A_j$};
\node at (0.4,-.4) {\large $\mathcal{S}$};
\end{tikzpicture}\\[1ex]
\textbf{eg.} for the experiment of throwing a standard die, events \{1, 3\}, \{4\}, and \{5, 6\} are disjoint
\end{description}

\subsection{\em Theorem: Complement rule}
If two events $E$ and $F$ are \textit{complementary}, then
\begin{equation*}
P(E)=1-P(F)\Leftrightarrow P(F)=1-P(E)
\end{equation*}
\begin{description}
\item[\sc Complement:] The complement of any event $A$ is the event ``not A", i.e., the event that $A$ does not occur.\\
The event $A$ and its complement $\bar{A}$ are \textit{mutually exclusive} and \textit{exhaustive}; that is, $A$ and $\bar{A}$ cannot occur simultaneously (mutually exclusive), and all outcomes in $\mathcal{S}$ are represented in $A$ together with $\bar{A}$ (exhaustive).\\[1ex]
Represented as a Venn diagram:\\
\begin{tikzpicture}[scale=1.1]
\draw[fill=red!30] (0,0) circle[radius=1];
\draw[fill=blue!30] (0.2,0.3) circle[radius=0.6];;
\node at (0.2,.3) {$A$};
\node at (-.5,-.3) {$\bar{A}$};
\node[pin=below right:{\large $\mathcal{S}$}] at (.5,-.7) {};
\end{tikzpicture}\\
\textbf{eg.} for the experiment of throwing a standard die, the complement of \{1, 2\} is \{3, 4, 5, 6\}
\end{description}

\section{\sc Probability problems}
\textsc{N.B. There are} several possible, equally valid ways to arrive at these answers, including different methods that may have been demonstrated in lecture.

\subsection{Singapore TOTO}
In the Singapore lottery game of TOTO, 6 (distinct) winning numbers plus one ``additional" winning number are drawn at random from the numbers 1 to 45. In the ordinary game, players choose 6 numbers/ticket in the hope of matching some or all of the winning numbers and becoming instant millionaires.

\subsubsection*{Prizes}
\begin{tabular}{|c|l|l|}\hline
Group&Prize Amount&Winning Numbers Matched\\\hline\hline
1&33\% of prize pool&6\\\hline
2&13\% of prize pool&5 + additional number\\\hline
3&13\% of prize pool&5\\\hline
4&13\% of prize pool&4 + additional number\\\hline
5&\$30 per winning combination&4\\\hline
6&\$20 per winning combination&3 + additional number\\\hline
\end{tabular}

\subsubsection*{Q: {\em What are the odds of being in Group 1?}}
Let us choose \textsl{S} = \{all combinations of 6 numbers from 1-45\}, so that our sample space represents all the possible combinations of 6 numbers from 1-45. Since the numbers are drawn at random, each outcome in the sample space is equally likely, so we can use the classical definition of probability:
\begin{equation*}
P(\{\text{Group 1}\})=\dfrac{|\{\text{Group 1}\}|}{|\textsl{S}|}
\end{equation*}
In this case, there is only 1 combination of 6 numbers we can choose that will cause us to be in Group 1: the 6 winning numbers. So
\begin{equation*}
P(\{\text{Group 1}\})=\dfrac{1\text{ winning combination}}{\binom{45}{6}\text{ possible combinations}}=\dfrac{1}{8\,145\,060}
\end{equation*}

\subsubsection*{Q: {\em What are the odds of being in Group 2?}}
The probability of being in Group 2 is the probability that we have chosen 5 winning numbers and 1 non-winning number, \textit{and} that the non-winning number is the additional number. We again use the classical definition of probability, and count the combinations of 6 numbers that contain 5 winning numbers and 1 non-winning number.
\begin{align*}
P(\{\text{Group 2}\})& =P(\{\text{5 winning numbers}\}\cap\{\text{1 additional number}\})\\
& =P(\{\text{5 winning numbers}\})\times P(\{\text{1 additional number}\})\\
& =\dfrac{|\{\text{5 winning numbers}\}|}{|\textsl{S}|}\times P(\{\text{1 additional number}\})
\end{align*}
There are $\binom{6}{5}$ ways in which we can choose 5 of the 6 winning numbers, and $\binom{39}{1}$ ways in which we can choose our sixth non-winning number from the 39 non-winning numbers. So by the product rule, there are $\binom{6}{5}\times\binom{39}{1}$ combinations of 5 winning numbers and 1 non-winning number.\\[1em]
The event of choosing the correct additional number from all possible additional numbers lies in a different sample space, which we will call \textsl{S'}. If \textsl{S'} = \{all possible additional numbers\}, then $|\textsl{S'}|=39$, since there are 39 possible additional numbers once the 6 winning numbers have been drawn.\\
Again, all outcomes in \textsl{S'} are equally likely, and there is only 1 way in which we can choose the correct additional number: our sixth, non-winning number must match the additional number. Thus,
\begin{equation*}
P(\{\text{Group 2}\})=\dfrac{\binom{6}{5}\times\binom{39}{1}}{\binom{45}{6}}\times \dfrac{1}{39}=\dfrac{6}{8\,145\,060}=\dfrac{1}{1\,355\,510}
\end{equation*}
Notice that the probability of being in Group 2 is 6 times the probability of being in Group 1.

\subsubsection*{Q: {\em What are the odds of being in Group 3?}}
The probability of being in Group 3 is the probability that we have chosen 5 winning numbers and 1 non-winning number, and that the non-winning number is \textit{not} the additional number.\\[1ex]
We calculated above that the probability of the former is $\dfrac{\binom{6}{5}\times\binom{39}{1}}{\binom{45}{6}}$.\\[1ex]
We also calculated that the probability of correctly choosing the additional number is $\frac{1}{39}$. By the complement rule, the probability of the complement, \textit{not} correctly choosing the additional number, is $1-\frac{1}{39}=\frac{38}{39}$. Thus,
\begin{align*}
P(\{\text{Group 3}\})& =P(\{\text{5 winning numbers}\}\cap\{\text{no additional number}\})\\
& =P(\{\text{5 winning numbers}\})\times P(\{\text{no additional number}\})\\
& =\dfrac{\binom{6}{5}\times\binom{39}{1}}{\binom{45}{6}}\times\dfrac{38}{39}\\
& =\dfrac{228}{8\,145\,060}=\dfrac{1}{35\,723.95}
\end{align*}

\subsubsection*{Q: {\em What are the odds of being in Group 4?}}
The probability of being in Group 4 is the probability that we have chosen 4 winning numbers and 2 non-winning numbers, \textit{and} that the non-winning number is the additional number.\\
There are $\binom{6}{4}$ ways in which we can choose 4 of the 6 winning numbers, and $\binom{39}{2}$ ways in which we can choose our 2 non-winning numbers from the 39 non-winning numbers. So by the product rule, there are $\binom{6}{4}\times\binom{39}{2}$ combinations of 4 winning numbers and 2 non-winning numbers.\\[1ex]
With regards to the additional number, there are 2 ways in which we can choose the correct additional number: either our first non-winning number matches the additional number, or our second non-winning number does. Thus,
\begin{align*}
P(\{\text{Group 4}\})& =P(\{\text{4 winning numbers}\}\cap\{\text{1 additional number}\})\\
& =P(\{\text{4 winning numbers}\})\times P(\{\text{1 additional number}\})\\
& =\dfrac{\binom{6}{4}\times\binom{39}{2}}{\binom{45}{6}}\times\dfrac{2}{39}\\
& =\dfrac{570}{8\,145\,060}=\dfrac{1}{14\,290}
\end{align*}

\subsubsection*{Q: {\em What are the odds of getting {\em any} prize?}}
The odds of getting \textit{any} prize are equivalent to the odds of belonging to \textit{any} group, that is, $P(\{\text{Group 1}\}\cup\{\text{Group 2}\}\cup\ldots\cup\{\text{Group 6}\})$. Since you cannot win more than one prize/belong to more than one group, these events are disjoint. Thus,
\begin{equation*}
P(\{\text{Group 1}\}\cup\{\text{Group 2}\}\cup\ldots\cup\{\text{Group 6}\})=P(\{\text{Group 1}\})+P(\{\text{Group 2}\})+\ldots+P(\{\text{Group 6}\})
\end{equation*}
If we were to calculate each of these probabilities, their sum would be $3.1197\times 10^{-3}$.

\subsubsection*{System entries}
In most lottery games, you can buy a ``system". Your chances of winning increase, but of course, you pay more as well.\\[1ex]
For example, buying System 7 means you choose 7 numbers instead of the usual 6.\\
There are $\binom{7}{6}$ combinations of 6 numbers from 7, and so, System 7 gives you 7 times the chances of winning with an ordinary 6-number ticket.\\[1ex]
Suppose that you choose 1, 3, 5, 7, 9, 11, 13 as your 7 numbers. Then you will win if the winning numbers are any of these 7 combinations:
\begin{itemize}
\item 1, 3, 5, 7, 9, 11
\item 1, 3, 5, 7, 9, 13
\item 1, 3, 5, 7, 11, 13
\item 1, 3, 5, 9, 11, 13
\item 1, 3, 7, 9, 11, 13
\item 1, 5, 7, 9, 11, 13
\item 3, 5, 7, 9, 11, 13
\end{itemize}
In Singapore TOTO, you can buy up to System 12.

\subsubsection*{Q: {\em By how much do your chances of winning increase if you buy a System 12, relative to the chances of winning with an ordinary ticket?}}
Since there are $\binom{12}{6}=924$ combinations of 6 numbers from 12, there are 924 ways in which you can win, which is 924 times the single way in which you can win with only 6 numbers.

\subsection{Poker}
In the standard card game of poker, each player gets a hand of 5 cards and places a bet, hoping his hand is better than the other players' hands. The lower the probability of a hand occurring, the more valuable it is.\\[1ex]
The game is played with a pack containing 52 cards in 4 suits, consisting of:
\begin{itemize}
\item 13 hearts: \textcolor{red}{$\heartsuit$ 2 3 4 5 6 7 8 9 10 J Q K A}
\item 13 diamonds: \textcolor{red}{$\diamondsuit$ 2 3 4 5 6 7 8 9 10 J Q K A}
\item 13 clubs: $\clubsuit$ 2 3 4 5 6 7 8 9 10 J Q K A
\item 13 spades: $\spadesuit$ 2 3 4 5 6 7 8 9 10 J Q K A
\end{itemize}

\subsubsection*{Ranking, frequency and probability of poker hands}
\begin{tabular}{|l|l|l|p{3in}|}\hline
\textsc{Hand}&\textsc{No. of Ways}&\textsc{Probability}&\textsc{Description}\\\hline\hline
Royal Flush&4&0.000002&Ten, J, Q, K, A of one suit.\\\hline
Straight Flush&36&0.000015&5 cards of one suit, in order.\newline
Example: 5, 6, 7, 8, 9, all spades.\\\hline
Four of a Kind&624&0.000240&4 cards of the same denominator.\newline Example: 4 kings and any other card.\\\hline
Full House&3744&0.001441&3 cards of one denominator and 2 cards of another.\newline Example: 3 aces and 2 kings.\\\hline
Flush&5108&0.001965&All 5 cards are from the same suit (excludes royal and straight flushes).\newline Example: 2, 4, 5, 9, J, all hearts.\\\hline
Straight&10\,200&0.003925&5 cards in order (excludes royal straight flushes).\newline Example: 3, 4, 5, 6, 7.\\\hline
Three of a Kind&54\,912&0.021129&3 cards of the same denominator.\newline Example: 3 aces, one J and one Q.\\\hline
Two Pairs&123\,552&0.047539&2 cards of one denominator and 2 cards of another.\newline Example: 3, 3, Q, Q, 5.\\\hline
One Pair&1\,098\,240&0.422569&2 cards of the same denominator.\newline Example: 10, 10, 4, 6, K.\\\hline
Nothing&1\,302\,540&0.501177&Example: 3, 6, 8, 9, K, collectively of at least two different suits.\\\hline
\end{tabular}

\subsubsection*{Q: {\em How do we know that the probability of being dealt a royal flush is 0.000002, as given above?}}
Let us choose \textsl{S} = \{all possible combinations of 5 cards\}. Since there are 52 cards in a deck, there are $\binom{52}{5}=2\,598\,960$ combinations of 5 cards.\\[1ex]
Only 4 of these combinations are royal flushes: (\textcolor{red}{$\heartsuit$ 10, J, Q, K, A}), (\textcolor{red}{$\diamondsuit$ 10, J, Q, K, A}), ($\clubsuit$ 10, J, Q, K, A), and ($\spadesuit$ 10, J, Q, K, A).\\[1ex]
Since all combinations of 5 cards are equally likely, we can use the classical definition of probability to determine that
\begin{equation*}
P(\{\text{royal flush}\})=\dfrac{|\{\text{royal flush}\}|}{|\textsl{S}|}=\dfrac{4}{2\,598\,960}\approx 0.000002
\end{equation*}

\subsubsection*{Q: {\em How do we know that the probability of being dealt a straight flush is 0.0.000015, as given above?}}
Within the same sample space as above, we need to determine the number of combinations of 5 cards that are straight flushes. For each suit, there are 10 straights: one that starts with a 2 (2, 3, 4, 5, 6), one that starts with a 3 (3, 4, 5, 6, 7), and so on, up to one that starts with a 10 (10, J, Q, K, A). Since there are 4 suits, there are $10\times 4=40$ possible straight flushes altogether. Thus,
\begin{equation*}
P(\{\text{straight flush}\})=\dfrac{|\{\text{straight flush}\}|}{|\textsl{S}|}=\dfrac{40}{2\,598\,960}\approx 0.000015
\end{equation*}
Note, however, that we should not strictly consider the royal flush, i.e. the straight flush starting with a 10, as a straight flush. If we exclude the royal flush, there are 9 straights for each suit, and $9\times 4=36$ in total. Then
\begin{equation*}
P(\{\text{straight flush}\})=\dfrac{|\{\text{straight flush}\}|}{|\textsl{S}|}=\dfrac{36}{2\,598\,960}\approx 0.000014
\end{equation*}

\subsubsection*{Q: {\em How do we know that the probability of being dealt a full house is 0.001441, as given above?}}
Within the same sample space as above, we need to determine the number of combinations of 5 cards that are full houses. We know that there are 13 denominators (2, 3, \ldots, A) and 4 cards of each denominator.\\[1ex]
If we select cards to form a full house, we must select 3 cards of 1 denominator and 2 cards of another. There are 13 ways to choose the denominator of the first 3 cards, and 12 ways to choose a different denominator for the next 2. Then, having chosen the denominator of each group, there are $\binom{4}{3}$ ways to choose 3 cards from 4 of the same denominator, and $\binom{4}{2}$ ways to choose 2 cards from 4 of the same denominator.\\[1ex]
Using the product rule, we determine that the number of ways in which we can choose a 5-card combination that is a full house is $13\times\binom{4}{3}\times 12\times\binom{4}{2}$, and so,
\begin{equation*}
P(\{\text{full house}\})=\dfrac{|\{\text{full house}\}|}{|\textsl{S}|}=\dfrac{13\times\binom{4}{3}\times 12\times\binom{4}{2}}{2\,598\,960}=\dfrac{3744}{2\,598\,960}\approx 0.001441
\end{equation*}

\section{\sc Additional practice problems}

{\bf Q: In Singapore TOTO, what are the odds of being in prize group 5?}\\[1em]
{\bf Q: In Singapore TOTO, what are the odds of being in prize group 6?}\\[1em]
On Monday, June 11th, 2001, the winning numbers for Singapore TOTO were 1, 10, 19, 23, 29, 45 with additional number 34. The next consecutive draw, on Thursday, June 14th, 2001, had the winning numbers 1, 10, 14, 19, 23, 34 with additional number 33.\\[1ex]
Thus, the five numbers 1, 10, 19, 23, 34 were repeated from the Monday draw. This is very rare and quite amazing.\\[1ex]
{\bf Q: What is the probability of this occurring?}\\[1em]
{\bf Q: For each hand of poker, as listed in (2.2), what is the probability of being dealt said hand?}

\end{document}


% This is a model template for the solutions in computational science. You can find a very useful documentation for LaTeX in Finnish at ftp://ftp.funet.fi/pub/TeX/CTAN/info/lshort/finnish/ or in English at ftp://ftp.funet.fi/pub/TeX/CTAN/info/lshort/english/. The section List of mathematical symbols in Chapter 3 is especially useful for the typesetting of mathematical formulas.

% Compile the document to PDF by command 'pdflatex model.tex' in the terminal. The command must be run twice for the references in the text to be correct.

\documentclass[a4paper,11pt]{article}
\usepackage[utf8]{inputenc}
% This includes letters such as � and �
\usepackage[T1]{fontenc}
% Use here 'Finnish' for Finnish hyphenation. You may have to compile the code twice after the change.
\usepackage[english]{babel}
\usepackage{graphicx}
% Some math stuff
\usepackage{amsmath,amsfonts,amssymb,amsbsy,commath,booktabs}
% This is just to include the urls
\usepackage{hyperref}
\usepackage[margin=2cm]{geometry}

\setlength{\parindent}{0mm}
\setlength{\parskip}{1.0\baselineskip}

\usepackage{listings}
\usepackage{color}

\definecolor{dkgreen}{rgb}{0,0.6,0}
\definecolor{gray}{rgb}{0.5,0.5,0.5}
\definecolor{mauve}{rgb}{0.58,0,0.82}

\lstset{frame=tb,
	language=Python,
	aboveskip=3mm,
	belowskip=3mm,
	showstringspaces=false,
	columns=flexible,
	basicstyle={\small\ttfamily},
	numbers=none,
	numberstyle=\tiny\color{gray},
	keywordstyle=\color{blue},
	commentstyle=\color{dkgreen},
	stringstyle=\color{mauve},
	breaklines=true,
	breakatwhitespace=true,
	tabsize=4
}

\begin{document}

\title{Becs-114.1100 Computational Science -- exercise round 2} % Replace the exercise round number
\author{Kunal Ghosh, 546247} % Replace with your name and student number
\maketitle
\section{Solution to Question 2 (a)}\label{prob2a}

\begin{center}
	\begin{table}[h!]
		\begin{tabular}{|c|c|c|c|}
			\hline
			& \textbf{$x_n$} & \textbf{$f(x_n)$} & \\
			\hline
			& 0.57735 & -5.38490017946 & \\
			& -5774691.30483 & -1.92568975495e+20 & \\
			& -3849794.20322 & -5.70574742208e+19 & \\
			& -2566529.46881 & -1.69059182876e+19 & \\
			& -1711019.64587 & -5.00916097411e+18 & \\
			& -1140679.76392 & -1.48419584418e+18 & \\
			& -760453.175945 & -4.39761731609e+17 & \\
			& -506968.783963 & -1.30299772329e+17 & \\
			& -337979.189309 & -3.86073399492e+16 & \\
			& -225319.45954 & -1.14392118368e+16 & \\
			& -150212.973028 & -3.38939609977e+15 & \\
			& -100141.98202 & -1.00426551103e+15 & \\
			& -66761.3213489 & -2.97560151409e+14 & \\
			& -44507.5475693 & -8.8165970783e+13 & \\
			& -29671.6983845 & -2.61232505991e+13 & \\
			& -19781.1322638 & -7.74022239752e+12 & \\
			& -13187.4215204 & -2.29339922743e+12 & \\
			& -8791.6143638 & -6.79525696041e+11 & \\
			& -5861.07626779 & -2.01340946325e+11 & \\
			& -3907.38421639 & -59656576256.1 & \\
			& -2604.92286769 & -17676022306.3 & \\
			& -1736.61533019 & -5237339750.94 & \\
			& -1157.74368087 & -1551804243.31 & \\
			& -771.829311282 & -459793765.406 & \\
			& -514.553159306 & -136235133.874 & \\
			& -343.035865117 & -40365928.7734 & \\
			& -228.691210394 & -11960251.0783 & \\
			& -152.461746781 & -3543762.45335 & \\
			& -101.642550399 & -1049993.69253 & \\
			& -67.7637253191 & -311103.009159 & \\
			& -45.1787335034 & -92174.945674 & \\
			& -30.1232585281 & -27309.0439797 & \\
			& -20.0877147577 & -8090.63230212 & \\
			& -13.3987478053 & -2397.030783 & \\
			& -8.93981373361 & -710.532509955 & \\
			& -5.96389606269 & -211.160294544 & \\
			& -3.96624263936 & -63.427040294 & \\
			& -2.59316212981 & -19.8445302435 & \\
			& -1.558162687 & -7.22485525043 & \\
			& -0.40836950412 & -4.65973250245 & \\
			& -9.73337273448 & -917.392196014 & \\
			& -6.49417233487 & -272.392833164 & \\
			& -4.32410615269 & -81.5275724202 & \\
			& -2.84430721072 & -25.1663756491 & \\
			& -1.76282441275 & -8.71524043281 & \\
			& -0.715653053283 & -4.65087530829 & \\
			& 7.95362390822 & 490.193684532 & \\
			& 5.35698948537 & 143.37433958 & \\
			& 3.67205642057 & 40.841946317 & \\
			& 2.63682496403 & 10.6966126523 & \\
			\hline
			\end{tabular}
			\caption{ Table logging the Newton's approximation of $x_n$, $f(x_n)$ at each iteration. }
			\label{table:1}
		\end{table}
	\end{center}

\begin{figure}[h]
	\centering
	\includegraphics[scale=0.45]{solution2a_fig.pdf}
	\caption{The graph is the function $x^3 - x - 5 = 0$ and the point is the value of $x_n$ Newton's method converged to after 50 iterations.}
	\label{fig:solution2a_fig}
\end{figure}

It can be observed from the table of $x_n$ and $f(x_n)$ that initially the root found by newton's method goes to really large values very far away from the actual root. This is because we hit the "Flat Spot" at that point where the slope $f'(x)$ is almost zero which means $\frac{1}{f'(x)}$ becomes really large and when we subtract that from $x_n$ we get really large approximation which might throw us far away from the actual root.

The corresponding python code can be found at \ref{code:problem2a}
\clearpage

\section{Solution to Question 2 (b)}\label{prob2b}

In the solution to this problem I arbitrarily chose bounds of -5 and +5 for the bisection method. Then everytime, the $x_n$ from newton method went out of bounds, I used bisection method and updated the bounds of the bisection method, as can be seen in the table below. The number of steps taken to converge to the root reduced drastically !

\begin{center}
	\begin{table}[h!]
		\begin{tabular}{|c|c|c|c|c|c|}
			\hline
			& \textbf{Method} & \textbf{New $x$} & \textbf{$F(x)$} & \textbf{Comments} & \\
			\hline
			& Newton & -5774691.30482718 & -1.925689754950669e+20 & - & \\
			\hline
			& Bisection & 0.0 & -5.0 & new bisec min = 0.0  max =5 & \\
			\hline
			& Newton & -5.0 & -125.0 & - & \\
			\hline
			& Bisection & 2.5 & 8.125 & new bisec min = 0.0 max = 2.5 & \\
			\hline
			& Newton & 2.0422535211267605 & 1.475576330428514 & - &\\
			\hline
			& Newton & 1.914080721014927 & 0.09854639952939426 &- & \\
			\hline
			& Newton & 1.904217317447332 & 0.0005576843842707291 & - &\\
			\hline
			& Newton & 1.904160860978299 & 1.820794359730371e-08 & - &\\
			\hline
			& Newton & 1.9041608591349206 & 0.0 & - &\\
			\hline
			& Newton & 1.9041608591349206 & 0.0 & - &\\
			\hline
		\end{tabular}
		\caption{ Table logging the approximation technique used, $x_n$, $f(x_n)$ and few other values at each iteration.}
		\label{table:2}
	\end{table}
\end{center}

\begin{figure}[h]
\centering
\includegraphics[scale=0.45]{solution2b_fig.pdf}
\caption{Plot of the function $x^3 - x - 5 = 0$ and our approximated root $x_n$ plotted by the point.}
\label{fig:solution2b_fig}
\end{figure}

The corresponding python code can be found at \ref{code:problem2b}
\clearpage

\section{Solution to Question 3}\label{prob3}

This problem is essentially the same as the second question. The only difference is that the new functions must handle compelx numbers instead of real values.

Note : Early stopping of iteration is really important and I noticed it in this problem since my first draft of the code did not have the stopping criterion and the plot for 1000 points took a really really long time ( maybe around 7 hours, I ran it overnight ).

\begin{figure}[h]
	\centering
	\includegraphics[scale=0.45]{Basin_1000.png}
	\caption{Basin of attraction with Real values on X axis and Imaginary values on the Y scale for $z^3 - 1 = 0$  Green is $(-1,0j)$ , Red is $(-5, -0.866j)$ and Blue is $(-5, 0.866j)$}
	\label{fig:basin1000}
\end{figure}

\begin{figure}[h]
	\centering
	\includegraphics[scale=0.45]{figure_q3_new.pdf}
	\caption{Basin of attraction plotted for 100 values each, on X and Y axes. Real values on X axis and Imaginary values on the Y scale for $z^3 - 1 = 0$  Green is $(-1,0j)$ , Red is $(-5, -0.866j)$ and Blue is $(-5, 0.866j)$}
	\label{fig:basin100}
\end{figure}

The corresponding python code can be found at \ref{code:problem3}

\clearpage
%\ldots

\section{Appendix A}\label{code:problem2a}
Python source code for \ref{prob2a}.
{\footnotesize
\begin{lstlisting}
from __future__ import division
import doctest
import pylab as pl
import numpy as np
# use newton's method to calculate roots
# equation = x^3 - x - 5
# x_initial = 0.57735
# max_iter = 50
# print the results and explain them

## TODO : Write tests
# x^3 - x - 5 , re-written to get rid of powers
def f(x):
	'''
	>>> f(1)
	-5
	>>> f(2)
	1
	>>> f(3)
	19
	'''
	return (x * (x+1) * (x-1)) - 5

# 3 x^2 - 1 , re-written to get rid of powers
def f_dash(x):
	'''
	>>> f_dash(1)
	2
	>>> f_dash(2)
	11
	'''
	return (3 * (x + 1) * (x - 1)) + 2

def get_next_appox(x_n_minus_1):
	x = x_n_minus_1
	return x - (f(x) / f_dash(x))

if __name__ == '__main__':
	doctest.testmod()
	x_n_minus_1 = 0.57735 # x_initial
	# iterate for a maximum of 50 iterations
	max_iter = 50
	output = []
	for _ in xrange(max_iter):
		x_n = get_next_appox(x_n_minus_1)
		output.append((x_n_minus_1, f(x_n_minus_1)))
		x_n_minus_1 = x_n

	for val in output:
		print val

	# plot the curve
	pl.figure(1)
	x = np.linspace(-5,5,1000)
	y = [f(_) for _ in x]

	pl.grid(True)
	pl.plot(x,y)
	pl.scatter(output[-1][0],0)

	pl.savefig("solution2a_fig.pdf")
	pl.show()
\end{lstlisting}
}

\clearpage
\section{Appendix B}\label{code:problem2b}
Python source code for \ref{prob2b}.
{\footnotesize
\begin{lstlisting}
from __future__ import division
import doctest
import pylab as pl
import numpy as np
# a)	use newton's method to calculate roots
# 		equation = x^3 - x - 5
#		x_initial = 0.57735
#		max_iter = 50
#		print the results and explain them

# x^3 - x - 5 , re-written to get rid of powers
def f(x):
	'''
	>>> f(1)
	-5
	>>> f(2)
	1
	>>> f(3)
	19
	'''
	return (x * (x+1) * (x-1)) - 5

# 3 x^2 - 1 , re-written to get rid of powers
def f_dash(x):
	'''
	>>> f_dash(1)
	2
	>>> f_dash(2)
	11
	'''
	return (3 * (x + 1) * (x - 1)) + 2

def is_within_bound(val, lower, upper):
	returnVal = False
	# Checks for strict bounds
	if val >= lower and val <= upper:
		returnVal = True
	return returnVal

def get_next_approx_newton(x_n_minus_1):
	x = x_n_minus_1
	return x - (f(x) / f_dash(x))

def get_next_approx_bisection(func, b_min, b_max):
	# returns the new approx root of func
	# via bisection method, new lower and upper bounds
	mid = 0.5 * (b_min + b_max)
	new_b_min = b_min
	new_b_max = b_max
	if (func(new_b_min) * func(mid) < 0):
		new_b_max = mid
	elif (func(mid) * func(new_b_max) < 0):
		new_b_min = mid
	return mid, new_b_min, new_b_max

if __name__ == '__main__':
	doctest.testmod()
	x_n_minus_1 = 0.57735 # x_initial
	# iterate for a maximum of 50 iterations
	max_iter = 50
	# Bounds for bisection method.
	# Assuming [-5, 5] arbitrarily and it also satisfies
	# f(-5) * f(5) < 0
	bisection_min, bisection_max = -5, 5

	# run till new root approximation and old root approximation
	# are not same upto the given precision
	tolerance = 0.0000000001

	# first approximation from newton's method to setup the while loop
	x_n = get_next_approx_newton(x_n_minus_1)
	print ["method", "New x", "f(x)"]
	print ["newton", x_n, f(x_n)]

	while abs(x_n - x_n_minus_1) > tolerance:
		x_n_minus_1 = x_n
		if is_within_bound(x_n, bisection_min, bisection_max):
			# if the root is within bounds then use newton's method
			x_n = get_next_approx_newton(x_n_minus_1)
			print ["newton", x_n, f(x_n)]
		else:
			# else use bisection method
			x_n, bisection_min, bisection_max = get_next_approx_bisection(f, bisection_min, bisection_max)
			print ["bisection", x_n, f(x_n), "new bisec min = ", bisection_min, "new bisec max = ", bisection_max]

	pl.figure(1)
	# plot the curve
	x = np.linspace(-5,5,1000)
	y = [f(_) for _ in x]
	pl.grid(True)
	pl.plot(x,y)
	pl.scatter(output[-1][0],0)
	pl.savefig("solution2b_fig.pdf")
	pl.show()
\end{lstlisting}
}

\clearpage
\section{Appendix C}\label{code:problem3}
Python source code for \ref{prob3}.
{\footnotesize
\begin{lstlisting}
from __future__ import division
import doctest
import pylab as pl
import numpy as np
# a)	use newton's method to calculate roots
# 		equation = x^3 - x - 5
#		x_initial = 0.57735
#		max_iter = 50
#		print the results and explain them

## TODO : Write tests
# x^3 - 1 , re-written to get rid of powers
def f(x_r,x_i):
	'''
	>>> f(1,1)
	(-3.0, 2.0)
	>>> f(2,1)
	(1.0, 11.0)
	>>> f(1,0)
	(0.0, 0.0)
	'''
	z = complex(x_r, x_i)
	retVal = (z * z * z) - 1
	return retVal.real, retVal.imag

# 3 x^2 , re-written to get rid of powers
def f_dash(x_r, x_i):
	'''
	>>> f_dash(1,1)
	(0.0, 6.0)
	>>> f_dash(2,1)
	(9.0, 12.0)
	'''
	z = complex(x_r, x_i)
	retVal = 3 * z * z
	return retVal.real, retVal.imag

def get_next_appox(x_r, x_i):
	'''
	x_r : previous real value of x
	x_i : previous imag value of x
	'''
	xr = x_r
	xi = x_i
	x = complex(xr, xi)

	fx_tuple = f(xr, xi)
	fx = complex(fx_tuple[0], fx_tuple[1])

	fdashx_tuple = f_dash(xr, xi)
	fdashx = complex(fdashx_tuple[0], fdashx_tuple[1])

	retVal = x - (fx / fdashx)
	return retVal.real, retVal.imag

def get_magnitude(z):
	xr, xi = z
	return xr * xr + xi * xi

def run_newton_method(xr_init, xi_init, epsilon=10 ** (-12), max_iter=100):
	'''
	xr_init = initial real value of x
	xi_init = initial imag value of x
	'''
	xr_old = xr_init
	xi_old = xi_init
	xr_new, xi_new = get_next_appox(xr_init, xi_init)

	while(max_iter != 0):
		max_iter -= 1
		xr_old, xi_old = xr_new, xi_new
		xr_new, xi_new = get_next_appox(xr_old, xi_old)
		if abs(xr_old - xr_new) < epsilon and abs(xi_old - xi_new) < epsilon:
			break
		if f(xr_new, xi_new) < epsilon:
			break
	return xr_new, xi_new

def whichRoot(xr, xi):
	retVal = -1
	if xr < -0.4 and xr > -0.6 and xi  > -0.9 and xi < -0.7:
		retVal = "r."
	elif (xr + -1 < .01 and xi < .01):
		retVal = "g."
	elif (xr > -0.6 and xr < -0.4) and (xi > 0.8 and xi < 0.9):
		retVal = "b."
	return retVal

if __name__ == '__main__':
	doctest.testmod()

	x_real_nums = np.linspace(-1, 1, 1000)
	x_imag_nums = np.linspace(-1, 1, 1000)

	outputs = []

	for xr in x_real_nums:
		outputs.append([])
		for xi in x_imag_nums:
			fx_r, fx_i = run_newton_method(xr, xi)
			outputs[-1].append(whichRoot(fx_r,fx_i))
	pl.figure(1)
	pl.ion()
	pl.show()
	pl.pcolor(np.array(outputs, np.float))
	pl.draw()
	pl.savefig("figure_q3_test.pdf")

\end{lstlisting}
}

\end{document}


\section{Termination Criteria}
As is customary for most genetic programming approaches, certain termination criteria are necessary to ensure a finite amount of time before a solution is produced. For this assignment we include a maximum number of generations to be run as well as a problem-specific success predicate as the termination criteria.

The success predicate entails an individual predicting the cases, deaths and recoveries for all dates in the training set within a 5\% range. The maximum number of generations allowed was chosen as 256. Notably, in the tests that were run the success predicate was never triggered.