\section{Formalising requirements}
\subsection{}

\begin{frame}
\frametitleTC{A time-domain viewpoint}
\framesubtitleTC{}
\myPause
 \begin{itemize}[<+-| alert@+>]
 \item Requirements are expressed in the time domain very naturally.
 \item Suppose to apply a step variation to the set point $w(k)$, and consider four possible\\
       responses of the controlled variable $y(k)$ --- points are joined with segments\\
       to improve readability:
       \begin{center}
        \vspace{2mm}\includegraphics[width=0.60\columnwidth]{./Unit-04/img/StepRespForRequirements.pdf}
       \end{center}
 \end{itemize}
\end{frame}

\begin{frame}
\frametitleTC{A time-domain viewpoint}
\framesubtitleTC{}
\myPause
 \begin{center}
  \includegraphics[width=0.40\columnwidth]{./Unit-04/img/StepRespForRequirements.pdf}
 \end{center}\myPause
 \begin{itemize}[<+-| alert@+>]
 \item The ``control quality'' (for the moment, in the most intuitive sense)\\
       is different:
       \begin{itemize}[<+-| alert@+>]
       \item \textcolor{blue!80!black}{blue} -- good;
       \item \textcolor{red!95!black}{red} -- does not reach the new desired value exactly;
       \item \textcolor{green!80!black}{green} -- too slow;
       \item \textcolor{magenta!95!black}{magenta} -- oscillatory.
       \end{itemize}
 \end{itemize}
\end{frame}

\begin{frame}
\frametitleTC{A time-domain viewpoint}
\framesubtitleTC{}
\myPause
 \begin{itemize}[<+-| alert@+>]
 \item Indeed, many requirements are straightforwardly expressed by stating e.g. how the response
       of the controlled variable to a certain variation of the set point has to look.
 \item For example, one may require that
       \begin{itemize}[<+-| alert@+>]
       \item[] when $w(k)$ undergoes a step variation
       \item[] $y(k)$ has to enter a $\mp$5\% band around it in maximum 5 steps
       \item[] and then never exit that band,
       \item[] also never reaching a value more than 10\% above the new set point,
       \item[] and with a one-step variation not greater than half of the total one,
       \end{itemize}
 \item[] or express analogous desires.
 \end{itemize}
\end{frame}

\begin{frame}
\frametitleTC{A time-domain viewpoint}
\framesubtitleTC{}
\myPause
 \begin{itemize}[<+-| alert@+>]
 \item A quite articulated example to illustrate the idea is shown below: the response has to lie
       entirely within the evidenced area.
       \begin{center}
        \includegraphics[width=0.60\columnwidth]{./Unit-04/img/StepRespWithConstraints-SP.pdf}
       \end{center}
 \item Blue responses are acceptable, red ones are not.
 \end{itemize}
\end{frame}

\begin{frame}
\frametitleTC{A time-domain viewpoint}
\framesubtitleTC{}
\myPause
 \begin{itemize}[<+-| alert@+>]
 \item The key idea we are soon introducing is that prescribing a certain aspect for the response
       of $y(k)$ to $w(k)$, in fact means prescribing characteristics of the transfer function
       from the former to the latter (as already anticipated).
 \item The same idea can be applied to the responses of $y(k)$ to disturbances, if the issue is to
       reject them.
 \item \vspace{3mm}This is not the only way to set requirements, but here we stick to this one.
 \item In the following we shall see how the idea above leads to synthesising\\
       a suitable controller --- and the PID structure will come into play\\
       \TC{with consciousness of the reasons for its wide applicability}.
 \item \vfill But prior to this, also to understand what a reasonable ``desired $y/w$\\
       transfer function'' can look like, we need to analyse the control loop\\
       formally.
 \end{itemize}
\end{frame}


\documentclass[12pt]{article}
\usepackage{graphics}
\begin{document}

\section*{NYU Physics 1---In-class Exam 3}

\vfill

\paragraph{Name:} ~

\paragraph{email:} ~

\paragraph{recitation:} ~

\vfill

This exam consists of two problems.  Write only in this booklet.  Be
sure to show your work.

\vfill ~

\clearpage

\section*{Problem 1}

\noindent~\hfill\includegraphics{../mp/rollinghill.eps}\hfill~

A thin hoop of mass $M$ and radius $R$ (with moment of inertia
$I=M\,R^2$) rolls without slipping along the hilly landscape shown.
If it was released from rest at point A, what is its speed $v$ at
point B?  Give your answer in terms of $h$, $g$, $M$, and $R$.  Recall
that the kinetic energy $K$ of a translating, rotating object is
$K=(1/2)\,M\,v^2+(1/2)\,I\,\omega^2$.

\clearpage

\section*{Problem 2}

A figure skater spins in place on frictionless ice at angular speed
$\omega_i$ with her hands outstretched.  She has a total moment of
inertia $I_i$.  As the skater draws her hands into her body, her
moment of inertia decreases to $I_f=I_i/2$.  Does her kinetic energy
$K$ increase, decrease, or stay the same?  If it increases, where does
the energy come from?  If it decreases, where does the energy go to?
\emph{Explain all your answers concisely but clearly.}

\clearpage

[This page intentionally left blank for calculations or other work.]

\end{document}


% Instructions: you don't need to change anything in the macros, but
% feel free to define new commands as you wish. Starting from the main
% body, change the specs (e.g., your
% name). use \begin{solution} \end{solution} environment to write your
% solutions. Don't forget to list your collaborators.

\documentclass[12pt,addpoints,answers]{exam}
%============Macros==================%
\usepackage{amsmath,amsfonts,amssymb,amsthm}
\usepackage[margin=1in]{geometry}
%--------------Cosmetic----------------%
\usepackage{mathtools}
\usepackage{hyperref}
\usepackage{fullpage}
\usepackage{microtype}
\usepackage{xspace}
\usepackage[svgnames]{xcolor}
\usepackage[sc]{mathpazo}
\usepackage{enumitem}
\setlist[enumerate]{itemsep=1pt,topsep=2pt}
\setlist[itemize]{itemsep=1pt,topsep=2pt}
%----------Header--------------------%
\def\course{CSCE629 Analysis of Algorithms}
\def\term{Texas A\&M U, Fall 2019}
\def\prof{Lecturer: Fang Song}
\newcommand{\handout}[5]{
   \renewcommand{\thepage}{\arabic{page}}
   \begin{center}
   \framebox{
      \vbox{
    \hbox to 5.78in { \hfill \large{\course} \hfill }
       \vspace{2mm}
       \hbox to 5.78in { {\Large \hfill \textbf{#5}  \hfill} }
       \vspace{2mm}
       \hbox to 5.78in { \term \hfill \emph{#2}}
       \hbox to 5.78in { {#3 \hfill \emph{#4}}}
      }
   }
   \end{center}
   \vspace*{4mm}
}
\newcommand{\hw}[4]{\handout{#1}{#2}{#3}{#4}{Homework #1}}

%-----defs and commands-----%
%\def\mG{[\textbf{G}]\xspace}
\def\veps{\varepsilon}
\newcommand{\bit}{\{0,1\}}
\newcommand{\negl}{\text{negl}}
\newcommand{\corr}[1]{{\color{blue}{#1}}}
\newcommand{\alg}[1]{\textsf{#1}}
\newcommand{\class}[1]{\text{#1}}
\newcommand{\NP}{\class{NP}}
%=======Main document==============%
\begin{document}

%----Specs: change accordingly-----%
\newif\ifstudent % comment out false
 \studenttrue
% \studentfalse

\def\hwnum{10}
\def\issuedate{11/08/19}
\def\duedate{10am, 11/15/19} %
\def\yourname{your name} % put your name here
%------------------------------%

\ifstudent
\hw{\hwnum}{\issuedate}{Student: \yourname}{Due: \duedate}%
\else
\hw{\hwnum}{\issuedate}{\prof}{Due: \duedate}%
\fi

\noindent \textbf{Instructions.}

\begin{itemize}
\item Typeset your submission by \LaTeX, and submit in PDF
  format. Your solutions will be graded on \emph{correctness} and
  \emph{clarity}. You should only submit work that you believe to be
  correct, and you will get significantly more partial credit if you
  clearly identify the gap(s) in your solution. You may opt for the
  ``I’ll take 15\%'' option.
\item You may collaborate with others on this problem set.  However,
  you must \textbf{{write up your own solutions}} and \textbf{{list
      your collaborators and any external sources}} for each
  problem. Be ready to explain your solutions orally to a course staff
  if asked.
\item For problems that require you to provide an algorithm, you must
  give a precise description of the algorithm, together with a proof
  of correctness and an analysis of its running time. You may use
  algorithms from class as subroutines. You may also use any facts
  that we proved in class or from the book.
\item {If you describe a Greedy algorithm, you will get no
    credit without a formal proof of correctness, even if your
    algorithm is correct.}
\end{itemize}

\noindent This assignment contains \numquestions\ questions,
\numpages\ pages for the total of \numpoints \ points and
\numbonuspoints\ bonus points. A random subset of the problems will be
graded. \medskip

% \paragraph{Exercises. Do not turn in.}
\newpage

\begin{questions}

  \question (Integer linear programming) An \emph{integer
    linear-programming} problem is a linear-programming problem with
  the additional constraint that the variables $x$ must take on
  \emph{integral} values.

  \begin{parts}
    \part[7] Show that \emph{weak duality} (CLRS Lemma 29.8) holds for an
    integer linear program.

    \part[8] Show that \emph{strong duality} (CLRS Theorem 29.10) does
    not always hold for an integer linear program.

    \part[10] Given a primal linear program in standard form, let us
    define $P$ to be the optimal objective value for the primal linear
    program, $D$ to be the optimal objective value for its dual, $IP$
    to be the optimal objective value for the integer version of the
    primal (that is, the primal with the added constraint that the
    variables take on integer values), and $ID$ to be the optimal
    objective value for the integer version of the dual. Assuming that
    both the primal integer program and the dual integer program are
    feasible and bounded, show that
    \[ IP\le P = D \le ID \, .\]
  \end{parts}

  \newpage
  \question ($\NP$ under regular operations) Given any sets
  $X\subseteq \bit^*$ and $Y \subseteq \bit^*$, we can define new sets
  under regular operations. For example, the \emph{union} is
  $X \cup Y = \{x: x\in X \text{ or }x\in Y\}$; and the
  \emph{concatenation} is
  $X \circ Y = \{xy: x\in X \text{ and } y \in Y \}$. We consider
  applying these operations on problems in NP.
  \begin{parts}
  \part[7] Prove or disprove that $\NP$ is closed under union. Namely
    for any $X\in \NP$ and $Y\in \NP$, does $X\cup Y \in \NP$ always
    hold?

    \part[8] Prove or disprove that $\NP$ is closed under
    concatenation.
  \part (Exercise. Do not turn in.) Prove or disprove that $\NP$ is
    closed under complement.
  \end{parts}

  \newpage

  \question[15] (2-SAT) Consider 2-SAT, in which the input is a formula
  with at most 2 literals per clause. Show that 2-SAT is in P. (Hint:
  you may want to consider an algorithm for testing if a graph is
  bipartite).
\end{questions}
\end{document}


\documentclass[12pt]{article}

\include{preamble}

\title{Math 341 / 650 Spring 2017 \\ Midterm Examination Two}
\author{Professor Adam Kapelner}

\date{Thursday, April 6, 2017}

\begin{document}
\maketitle

\noindent Full Name \line(1,0){410}

\thispagestyle{empty}

\section*{Code of Academic Integrity}

\footnotesize
Since the college is an academic community, its fundamental purpose is the pursuit of knowledge. Essential to the success of this educational mission is a commitment to the principles of academic integrity. Every member of the college community is responsible for upholding the highest standards of honesty at all times. Students, as members of the community, are also responsible for adhering to the principles and spirit of the following Code of Academic Integrity.

Activities that have the effect or intention of interfering with education, pursuit of knowledge, or fair evaluation of a student's performance are prohibited. Examples of such activities include but are not limited to the following definitions:

\paragraph{Cheating} Using or attempting to use unauthorized assistance, material, or study aids in examinations or other academic work or preventing, or attempting to prevent, another from using authorized assistance, material, or study aids. Example: using an \emph{unauthorized} cheat sheet in a quiz or exam, altering a graded exam and resubmitting it for a better grade, etc.
\\

\noindent I acknowledge and agree to uphold this Code of Academic Integrity. \\

\begin{center}
\line(1,0){250} ~~~ \line(1,0){100}\\
~~~~~~~~~~~~~~~~~~~~~signature~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ date
\end{center}

\normalsize

\section*{Instructions}

This exam is seventy five minutes and closed-book. You are allowed \textbf{one} page (front and back) of a \qu{cheat sheet.} You may use a graphing calculator of your choice. Please read the questions carefully. If the question reads \qu{compute,} this means the solution will be a number otherwise you can leave the answer in \textit{any} widely accepted mathematical notation which could be resolved to an exact or approximate number with the use of a computer. I advise you to skip problems marked \qu{[Extra Credit]} until you have finished the other questions on the exam, then loop back and plug in all the holes. I also advise you to use pencil. The exam is 100 points total plus extra credit. Partial credit will be granted for incomplete answers on most of the questions. \fbox{Box} in your final answers. Good luck!

\pagebreak

\begin{table}[htp]
\centering
\small
\begin{tabular}{l | llll}
Distribution                  & Quantile  & PMF / PDF  &CDF       & Sampling  \\
of r.v. &  Function & function         & function &  Function \\ \hline
beta & \texttt{qbeta}($p$, $\alpha$, $\beta$)
& \texttt{d-}($x$, $\alpha$, $\beta$)
& \texttt{p-}($x$, $\alpha$, $\beta$)
& \texttt{r-}($\alpha$, $\beta$) \\
betabinomial & \texttt{qbetabinom}($p$, $n$, $\alpha$, $\beta$)
& \texttt{d-}($x$, $n$, $\alpha$, $\beta$)
& \texttt{p-}($x$, $n$, $\alpha$, $\beta$)
& \texttt{r-}($n$, $\alpha$, $\beta$) \\

betanegativebinomial & \texttt{qbeta\_nbinom}($p$, $r$, $\alpha$, $\beta$)
& \texttt{d-}($x$, $r$, $\alpha$, $\beta$)
& \texttt{p-}($x$, $r$, $\alpha$, $\beta$)
& \texttt{r-}($r$, $\alpha$, $\beta$) \\

binomial & \texttt{qbinom}($p$, $n$, $\theta$)
& \texttt{d-}($x$, $n$, $\theta$)
& \texttt{p-}($x$, $n$, $\theta$)
& \texttt{r-}($n$, $\theta$) \\

exponential & \texttt{qexp}($p$, $\theta$)
& \texttt{d-}($x$, $\theta$)
& \texttt{p-}($x$, $\theta$)
& \texttt{r-}($\theta$) \\

gamma & \texttt{qgamma}($p$, $\alpha$, $\beta$)
& \texttt{d-}($x$, $\alpha$, $\beta$)
& \texttt{p-}($x$, $\alpha$, $\beta$)
& \texttt{r-}($\alpha$, $\beta$) \\

geometric & \texttt{qgeom}($p$, $\theta$)
& \texttt{d-}($x$, $\theta$)
& \texttt{p-}($x$, $\theta$)
& \texttt{r-}($\theta$) \\

inversegamma & \texttt{qinvgamma}($p$, $\alpha$, $\beta$)
& \texttt{d-}($x$, $\alpha$, $\beta$)
& \texttt{p-}($x$, $\alpha$, $\beta$)
& \texttt{r-}($\alpha$, $\beta$) \\

negative-binomial & \texttt{qnbinom}($p$, $r$, $\theta$)
& \texttt{d-}($x$, $r$, $\theta$)
& \texttt{p-}($x$, $r$, $\theta$)
& \texttt{r-}($r$, $\theta$) \\

normal (univariate) & \texttt{qnorm}($p$, $\theta$, $\sigma$)
& \texttt{d-}($x$, $\theta$, $\sigma$)
& \texttt{p-}($x$, $\theta$, $\sigma$)
& \texttt{r-}($\theta$, $\sigma$) \\

normal (multivariate) &
& \multicolumn{2}{l}{\texttt{dmvnorm}($\x$, $\muvec$, $\bSigma$)}
& \texttt{r-}($\muvec$, $\bSigma$) \\

poisson & \texttt{qpois}($p$, $\theta$)
& \texttt{d-}($x$, $\theta$)
& \texttt{p-}($x$, $\theta$)
& \texttt{r-}($\theta$) \\

T (standard) & \texttt{qt}($p$, $\nu$)
& \texttt{d-}($x$, $\nu$)
& \texttt{p-}($x$, $\nu$)
& \texttt{r-}($\nu$) \\

T (nonstandard) & \texttt{qt.scaled}($p$, $\nu$, $\mu$, $\sigma$)
& \texttt{d-}($x$, $\nu$, $\mu$, $\sigma$)
& \texttt{p-}($x$, $\nu$, $\mu$, $\sigma$)
& \texttt{r-}($\nu$, $\mu$, $\sigma$) \\

uniform & \texttt{qunif}($p$, $a$, $b$)
& \texttt{d-}($x$, $a$, $b$)
& \texttt{p-}($x$, $a$, $b$)
& \texttt{r-}($a$, $b$) \\
\end{tabular}
\caption{Functions from $\texttt{R}$ (in alphabetical order) that can be used on this exam. The hyphen in colums 3, 4 and 5 is shorthand notation for the full text of the r.v. which can be found in column 2.
}
\label{tab:eqs}
\end{table}

\problem The table below displays an excerpt of historical data of the S\&P500 index for American equities going back to 1975, a total of $n = 42$ years.

\begin{table}[htp]
\centering
\begin{tabular}{c|cccc}
Year & Beginning Price & Ending Price & Gain or Loss & Percent Gain or Loss \\ \hline
1975 & 68.56           & 90.19        & 21.63        & 31.55\%              \\
1976 & 90.19           & 107.46       & 19.15        & 19.15\%              \\
\vdots \\
2013 & 1426.19         & 1848.36      & 422.17       & 29.60\%              \\
2014 & 1848.36         & 2058.9       & 210.54       & 11.39\%              \\
2015 & 2058.9          & 2043.94      & -14.96       & -0.73\%              \\
2016 & 2043.94         & 2238.83      & 194.89       & 9.54\%
\end{tabular}
\end{table}

For the yearly percent gains (or losses) we calculate the sample average and sample standard deviation:

\beqn
\xbar &=& 9.90 \\
s &=& 15.86
\eeqn

We also assume that the yearly percent gains (or losses) is modeled via

\beqn
\Xoneton ~|~\theta,\sigsq \iid \normnot{\theta}{\sigsq}
\eeqn
\pagebreak

\benum

\subquestionwithpoints{6} Is the model assumption a realistic assumption? To get full credit, you must address two distinct assumptions.\spc{7}

\subquestionwithpoints{6} Regardless of your answer in (a), assume the normal likelihood model for the duration of the problem. We now assume that $\sigsq$ is known and it is equal to $s^2$. Provide an uninformative prior for $\theta$ below and give some sort of short justification for why you wish to use this prior. \spc{5}

\subquestionwithpoints{6} Find the posterior distribution of $\theta$. \spc{2}

\subquestionwithpoints{6} Provide an expression for a 99\% credible region for $\theta$.\spc{2}

\subquestionwithpoints{6} Provide an expression for the probability that the mean return is greater than 11\%.\spc{2}

\subquestionwithpoints{6} Provide an expression for the probability that \textit{next year's} return will be greater than 11\%.\spc{3}

\subquestionwithpoints{6} We now pretend that $\theta$ is known and equal to $\xbar$. Provide an uninformative prior for $\sigsq$ below and give some sort of short justification for why you wish to use this prior. \spc{5}

\subquestionwithpoints{6} Find the posterior distribution of $\sigsq$. \spc{2}

\subquestionwithpoints{6} Provide an expression for a 99\% credible region for $\sigsq$.\spc{542}

\subquestionwithpoints{6} Test that the standard error return is less than 11\% by finding an expression for the p value and indicating the comparison you would do. (Note standard error is the square root of variance). \spc{7}

\subquestionwithpoints{6} Provide an expression for the probability that next year's return will be less than 11\%. If there is no closed form expression or no expression in Table~\ref{tab:eqs}, please provide an expression that can be evaluated numerically \spc{11}

\eenum

\problem This question continues our discussion of \qu{batting averages} in baseball.

\noindent Every hitter's \emph{sample} batting average (BA) is defined as:

\beqn
BA := \frac{\text{sample \# of hits}}{\text{sample \# of at bats}}
\eeqn

\begin{figure}[htp]
\centering
\includegraphics[width=2.2in]{baseball.jpg}
\end{figure}
\noindent In class we discussed a means to infer a hitter's true batting average $\theta$ by using empirical Bayes with the binomial likelihood model where the prior we considered was $\theta \sim \betanot{\alpha = 78.7}{\beta = 224.8}$.

\benum

\subquestionwithpoints{8} Given that a new batter starts his season with $n=6$ at bats and 3 hits, find an expression for the probability he has exactly 100 hits during his next 387 at bats. \spc{6}

\eenum

\problem This question has some theoretical questions. The parts are independent from each other unless otherwise stated.

\benum

\subquestionwithpoints{4} If $\Xoneton~|~\theta \iid \text{Geometric}(\theta)$ and $\theta \sim \betanot{\alpha}{\beta}$, what is $\expe{\theta~|~\Xoneton}$? \spc{6}

\subquestionwithpoints{4} If $\Xoneton~|~\theta \iid \text{Poisson}(\theta)$ and $\theta \sim \text{Gamma}\parens{\alpha, ~\beta}$, what is the kernel of $\cprob{\theta}{\Xoneton}$? \spc{6}

\subquestionwithpoints{2} Under the model in (b), what is the Jeffrey's prior? \spc{2}

\subquestionwithpoints{6} Under the model in (b), find $\cprob{X^*}{\Xoneton}$ where $X^*$ is one future realization from the same poisson likelihood model. For the parameters of this distribution, only use notation given in (b). \spc{5}

\subquestionwithpoints{4} If $\Xoneton ~|~\theta,\sigsq \iid \normnot{\theta}{\sigsq}$ with $\sigsq$ known and the prior is $\theta ~|~\sigsq \sim \normnot{\mu_0}{\tau^2}$, what is $\cprob{\theta}{\Xoneton, \sigsq}$? \spc{3}

\subquestionwithpoints{6} For the model in (e) reparameterize the prior using $\rho$, the quantity that indicates shrinkage to the prior expectation. \spc{7}

\subquestionwithpoints{6} [Extra Credit] If $\Xoneton ~|~\theta,\sigsq \iid \normnot{\theta}{\sigsq}$ with $\sigsq$ known and the prior is a \textit{mixture} of $K$ Gaussians e.g. $\theta_1, \theta_2, \ldots, \theta_K \inddist \normnot{\mu_k}{\tau^2_k}$, demonstrate that the posterior is normal and find the posterior parameters. Assume the mixture is uniform i.e., each of the $k$ mixture components has the same probabilitiy i.e. $1/K$.\spc{5}

\eenum

\end{document}

\problem This question continues our discussion about extrasensory perception. A famous series of experiments called the Ganzfeld experiments work as follows. A human subject called the \qu{receiver} sits in a room in darkness with their eyes covered and with headphones to filter out any noise. An example subject is pictured below:

\begin{figure}[htp]
\centering
\includegraphics[width=2in]{Ganzfeld.jpg}
\end{figure}

\noindent Another human subject called the transmitter sits in another room. In the beginning of the experiment, the transmitter is shown an image. The transmitter than tries to \qu{telephathically transmit} the image to the receiver. At the conclusion of the experiment, the receiver is shown four images --- three decoys and the true image the transmitter was given --- and is asked to choose one. If the receiver chooses the true image, this is termed a \qu{hit}.

\benum

\subquestionwithpoints{4} Under the null hypothesis of \qu{no ESP}, the receiver chooses one image randomly from four images where one image make the \qu{hit}. Create a r.v. $X$ for the \qu{hit} under the null. Indicate the type of r.v. and the value of $\theta$. \spc{2}

\subquestionwithpoints{5} A psychic claims that his ability to identify the target image is 50\%. Create a prior on both the null $\theta$ and the psychic's claimed $\theta$. Use the principle of indifference.\spc{1}

\subquestionwithpoints{7} We run three independent experimental trials of which the psychic gets two of the three correct. Calculate $\prob{X}$.   \spc{5}

\subquestionwithpoints{6} Find the probability that the psychic's assessment of his abilities is correct given this data. \spc{5}

\subquestionwithpoints{7} Calculate the probability the next experiment for this psychic (i.e. the fourth, unobserved experiment) will be a \qu{hit}. \spc{5}

\subquestionwithpoints{6} We now put a uniform prior on $\theta$ across all values in the support. Find the probability that this psychic has \qu{better than normal} abilities given the data. You are free to leave your answer in notation from Table~\ref{tab:eqs}. \spc{2}

\subquestionwithpoints{5} Instead of a uniform prior, you want to factor in what you've seen previously. You figure your prior experience is worth 18 trials and you haven't seen any evidence of ESP in those trials. Using the conjugate prior we discussed in class, what is the prior for $\theta$ now? \spc{1}

\subquestionwithpoints{7} Illustrate this prior the best you can. No need for perfect scale. Make sure to label axes appropriately. \spc{10}

\subquestionwithpoints{4} Compute the prior mean. \spc{2}

\subquestionwithpoints{5} Given this prior and the data, calculate the shrinkage proportion towards $\expe{\theta}$ if you were to use $\thetahatmmse$ for your point estimation strategy. \spc{1}

\subquestionwithpoints{5} If there was a lot of data, what would be the influence of the prior on the point estimate of $\theta$? Try to use only one word. \spc{0}

\subquestionwithpoints{6} Given this prior and the data, fill in the following boxes with only the following symbols: $<, ~\leq, ~>, ~\geq, ~=$ to indicate the numerical relationships.\vspace{-0.4cm}

\beqn
\thetahatmap~~\framebox(20,20){}~~ \thetahatmmse, \quad \thetahatmae~~\framebox(20,20){}~~ \thetahatmmse, \quad \thetahatmap~~\framebox(20,20){}~~ \thetahatmae
\eeqn

\subquestionwithpoints{10} In 2010, Lance Storm, Patrizio Tressoldi, and Lorenzo Di Risio analyzed 29 ganzfeld studies from 1997 to 2008 amassing data on 1,498 trials with different receiver and transmitter subjects. Of the 1,498 trials, 483 were hits. Test the existence of ESP in the Ganzfeld experiments using the Frequentist (i.e. the \textit{non-Bayesian}) two-sided test. You must (1) write your hypotheses clearly, (2) choose your own significance level and (3) state clearly the conclusion of the test.\spc{7}

\subquestionwithpoints{4} Using the Storm et al. (2010) Ganzfeld data and a uniform prior, create a 95\% credible region for $\theta$. You are free to leave your answer in notation from Table~\ref{tab:eqs}. \spc{2}

\subquestionwithpoints{4} Describe in one sentence how you would use the answer from the previous question to test the existence of ESP in the Storm et al. (2010) Ganzfeld data using one of the Bayesian two-sided hypothesis test strategies discussed in class.\spc{2}

\subquestionwithpoints{10} Assess the evidence for the ESP phenomenon in the Ganzfeld experiments using Bayes Factors. You will not be able to solve for $B$ numerically but you need to get as far as you can. 5 points extra credit if you can solve numerically and interpret your result. You can use $\natlog{n!} \approx n\natlog{n} - n + 1$. \spc{8}

\eenum

\problem This is a theoretical question.

\benum
\subquestionwithpoints{5} Consider the case of assessing the evidence of $\theta_a$ versus $\theta_0$. Show that $B$ is the odds ratio of the posterior odds to the prior odds.

\eenum

\end{document}
%-------------------------------------------------------

\problem This question is ironically about theory of testing.

\begin{figure}[htp]
\centering
\includegraphics[width=4in]{examination.jpg}
\end{figure}

\noindent Every hitter's \emph{sample} batting average (BA) is defined as:

\beqn
BA := \frac{\text{sample \# of hits}}{\text{sample \# of at bats}}
\eeqn

In this problem we care about estimating a hitter's \emph{true} batting average which we call $\theta$. Each player has a different $\theta$ but we focus in this problem on one specific player. In order to estimate the player's true batting average, we use the sample batting average as defined above.

\benum
\subquestionwithpoints{2} For the remainder of the problem, we assume that each at bat (for any player) are \emph{conditionally} $\iid$ based on the players' true batting average, $\theta$. So if a player has $n$ at bats, then each successful hit in each at bat can be modeled via

\beqn
X_1~|~\theta, ~X_2~|~\theta, \ldots, ~X_n~|~\theta \iid \bernoulli{\theta}.
\eeqn

Under this model above, if the player had $n=4$ at bats, would the $\prob{X_3,X_2,X_4,X_1}$ be equal to the $\prob{X_1,X_2,X_3,X_4}$? Yes / no. \spc{1}

\subquestionwithpoints{3} If the player had $n=4$ at bats and $\sum_{i=0}^n x_i=0$ hits, compute $\thetahatmle$.\spc{2}

\subquestionwithpoints{4} Compute a frequentist confidence interval for $\theta$ given the data in (b).\spc{2}

\subquestionwithpoints{3} Describe in English the main problem with the interval in (c).\spc{2}

\subquestionwithpoints{2} Set the following prior: $\theta \sim \stduniform$. Is this an informative prior for the true batting average? Yes/no \spc{1}

\subquestionwithpoints{5} Given the prior in (e) and the data in (b), find the posterior distribution of this player's true batting average.\spc{3}

\subquestionwithpoints{3} Based on your posterior distribution in (f), give your best estimate to the value of $\theta$ which minimizes squared error loss.\spc{3}

\subquestionwithpoints{2} Based on your posterior distribution in (f), describe using an integral or \texttt{R}-language expression your best estimate to the value of $\theta$ which minimizes absolute error loss but do not compute.\spc{4}

\subquestionwithpoints{4} Based on your posterior distribution in (f), give your best estimate to the value of $\theta$ using the posterior mode.\spc{2}

\subquestionwithpoints{5} Find an integral expression for the probability this hitter bats above a 300 batting average (which means the true batting average is 0.3 or greater). Do not compute. \spc{2}

\subquestionwithpoints{5} Assuming you have access to \texttt{R} and its function \texttt{qbeta}, give the 95\% credible region for $\theta$. The three arguments for \texttt{qbeta} are (1) quantile (2) alpha and (3) beta. Then, provide an interpretation for this interval. \spc{2}

\subquestionwithpoints{3} What would the Jeffrey's prior be in our model situation described in (a)? \spc{1}

\subquestionwithpoints{2} Would the posterior under the Haldane prior be proper given the data in (b)? Yes / No. \spc{0.5}

\subquestionwithpoints{5} The batting average is \textit{only} measured as the batting average and never logged or transformed. Would there be any value in using the Jeffrey's prior instead of the prior in (e)? Discuss. \spc{5}

\subquestionwithpoints{5} Looking at the entire dataset for 6,061 batters who had 100 or more at bats, I fit a beta function to the sample batting averages and estimated $\alpha = 42.3$ and $\beta = 127.7$ (which we called \qu{empirical Bayes} estimates in class). Consider building a prior from this estimate as

\beqn
\theta \sim \betanot{42.3}{127.7}.
\eeqn

Would a prior based on these hyperparameter estimates be \qu{objective}? Yes / No. Why? \spc{3}

\subquestionwithpoints{2} Is the prior from (o) considered a \qu{conjugate prior}? Yes / No.\spc{0.5}

\subquestionwithpoints{3} Using the prior from (o), find the $\thetahatmmse$ without considering the data whatsoever. Round to 3 digits. \spc{3}

\subquestionwithpoints{4} Using the prior from (o) and the data from (b), find the posterior $\thetahatmmse$. Round to 3 digits. \spc{2}

\subquestionwithpoints{5} The posterior estimate from (q) is different from the frequentist estimate in (b) due to shrinkage. What is the proportion of shrinkage for the posterior estimate in (q)? We denoted this as $\rho$ in class. Round to 3 digits. \spc{3}

\subquestionwithpoints{4} [Extra Credit] Using the Bayesian CLT, compute a 95\% credible region for $\theta$ for the data in (b) and the prior in (o). Round to 3 digits. \spc{3}

\subquestionwithpoints{3} Based on the data in (b) and the prior in (o), what is the probability this batter gets a hit on his next at bat? \spc{3}

\subquestionwithpoints{5} Based on the data in (b) and the prior in (o), write an exact expression for the batter getting 14 or more hits on the next 20 at bats. You can leave your answer in terms of the beta function. Do not compute explicitly. \spc{5}

\subquestionwithpoints{6} Based on the data in (b) and the prior in (o), find the kernel of the distribution for the number of hits this batter gets in the next $m$ at bats. Partial credit is given. \spc{3}

\subquestionwithpoints{2} Based on the data in (b) and the prior in (o), the joint posterior predictive distribution for $n=4$ looks like as follows.

%library(VGAM)
%barplot(dbetabinom.ab(0 : 4, 4, shape1 = 42.3 + 0, shape2 = 127.7 + 4), names = 0 : 4, xlab = "X* (# hits)", ylab = "prob(X*|X)")
\begin{figure}[htp]
\centering
\includegraphics[width=3.0in]{post_pred.pdf}
\end{figure}

Does the data from (b) look abnormal for this model? Yes / no. \spc{0.5}

\subquestionwithpoints{7} Test the following hypotheses by finding an integral or \texttt{R}-language expression for the Bayesian $p$-val for the data in (b) and prior in (o):

\beqn
&& H_0: \theta \geq \theta_0 \\
&& H_a: \theta < \theta_0
\eeqn

where $\theta_0 = \expe{\theta}$. That is, we're testing if this batter is truly \qu{below-average} as compared to the 6,061 career major league baseball players from the official dataset.\spc{8}

\subquestionwithpoints{7} Write an integral or \texttt{R}-language expression for $K$, the Bayes Factor in favor of $H_a$.\spc{9}

\subquestionwithpoints{3} For the model in (a), specify $\mathcal{F}$ (the likelihood model) of a hit at a single at bat. \spc{3}

\subquestionwithpoints{3} [Extra credit] For the the data in (b) and prior in (o), compute $\expesub{X}{\cexpesub{\theta}{\theta}{X}}$ using the Law of Iterated Expectation. \spc{2}

\subquestionwithpoints{3} [Extra credit] For the data in (b), what is the frequentist predictive distribution?

\eenum

\end{document}


\documentclass{report}

\usepackage{634format}
\usepackage{amsmath}

\usepackage{listings}
\usepackage{textcomp}

\usepackage{hyperref}

\usepackage{holtex}
\usepackage{holtexbasic}

\usepackage{amsmath}
\input{commands}

\begin{document}
\lstset{language = ML}

\title{Project 4}
\author{\textbf{Jinhao Wei}}
\date{Feb 9, 2019}

\maketitle{}

\begin{abstract}
This project is mainly a practice on programming in HOL. In this project, we attempted higher order logics in HOL. We provided solutions for several problems, including problems on:
\begin{enumerate}
\item HOL expressions that are equivalent to ACL semantics or standard notations.
\item HOL functions that operates on HOL terms.
\end{enumerate}

 For each of the problems, this report contains:
\begin{itemize}
\item Problem Statements
\item Relevant Code
\item Execution Transcripts of Test Cases
\end{itemize}

\end{abstract}

\begin{acknowledgments}
This project followed Professor Shiu-Kai Chin's sample documents' format, and used multiple \emph{.sty} files, such as \emph{634format.sty}, \emph{holtex.sty}, \emph{holtexbasic.sty}, which were provided in Professor Shiu-Kai Chin's sample report. What's more, this report inherits \emph{P2_jwei15.tex}, a file that I had created while working on Project2, so I can save some time on formatting.
\end{acknowledgments}

\tableofcontents{}
\chapter{Executive Summary}
\label{cha:Executive-Summary}

\textbf{All requirements for this project are satisfied} Specifically,

\begin{description}
\item[Report Contents] \ \\
This report consists of:

\begin{enumerate}
\item[] Chapter~\ref{cha:Executive-Summary}: Executive Summary
\item[] Chapter~\ref{cha:exercise-6.2.1}: Exercise 6.2.1
    \begin{enumerate}
      \item[] Section~\ref{sec:6.2.1-problem-statement}: Problem Statement
      \item[] Section~\ref{sec:6.2.1-relevant-code}: Relevant Code
      \item[] Section~\ref{sec:6.2.1-test-cases}: Execution Transcripts of Test Cases
    \end{enumerate}

\item[] Chapter~\ref{cha:exercise-7.3.1}: Exercise 7.3.1
\begin{enumerate}
      \item[] Section~\ref{sec:7.3.1-problem-statement}: Problem Statement
      \item[] Section~\ref{sec:7.3.1-relevant-code}: Relevant Code
      \item[] Section~\ref{sec:7.3.1-test-cases}: Execution Transcripts of Test Cases
    \end{enumerate}
\item[] Chapter~\ref{cha:exercise-7.3.2}: Exercise 7.3.2
\begin{enumerate}
      \item[] Section~\ref{sec:7.3.2-problem-statement}: Problem Statement
      \item[] Section~\ref{sec:7.3.2-relevant-code}: Relevant Code
      \item[] Section~\ref{sec:7.3.2-test-cases}: Execuion Transcripts of Test Cases
    \end{enumerate}

\item[] Chapter~\ref{cha:exercise-7.3.3}: Exercise 7.3.3
\begin{enumerate}
      \item[] Section~\ref{sec:7.3.3-problem-statement}: Problem Statement
      \item[] Section~\ref{sec:7.3.3-relevant-code}: Relevant Code
      \item[] Section~\ref{sec:7.3.3-test-cases}: Execution Transcripts of Test Cases
    \end{enumerate}

\item[] Appendix~\ref{appendix:A}: Source Code For Exercise 6.2.1
\item[] Appendix~\ref{appendix:B}: Source Code For Exercise 7.3.1
\item[] Appendix~\ref{appendix:C}: Source Code For Exercise 7.3.2
\item[] Appendix~\ref{appendix:D}: Source Code For Exercise 7.3.3
\end{enumerate}

\item [Reproducibility in ML and \LaTeX]\ \\
All ML and \LaTeX{} source files compile well on the environment provided by this course.
\end{description}

\chapter{Exercise 6.2.1}
\label{cha:exercise-6.2.1}

\section{Problem Statement}
\label{sec:6.2.1-problem-statement}

Our basic task in this problem is to write HOL codes that are equivalent to ACL semantics or standard notations. We will also print and observe the result of each piece of code.

There are no extra test cases required in this exercise problem.

\section{Relevant Code}
\label{sec:6.2.1-relevant-code}

We will define each of our functions as below:

\lstset{frameround = fttt}
\begin{lstlisting}[frame= trBL]
(*  6.2.1.1  *)
``P x ==> Q y``;

(*  6.2.1.2  *)
``P (x:num) ==> Q (y:bool)``;
``P x ==> Q y``;

(*  6.2.1.3  *)
``!x y.P x ==> Q y``;

(*  6.2.1.4  *)
``?(x:num).R(x:'a)``;

(*  6.2.1.5  *)
``~(!x.(P x) \/ (Q x)) = ?x.(~(P x)) /\ (~(Q x))``;

(*  6.2.1.6  *)
``!x.(P x ==> M x)``;

(*  6.2.1.7  *)
``?x.P x /\ Funny x``;
\end{lstlisting}

\section{Execution Transcripts of Test Cases}
\label{sec:6.2.1-test-cases}

Exercise 6.2.1 did not provide extra test cases, so we simply send each piece of the relevant codes to the HOL, and we will see results as below:

\subsection{Question 6.2.1.1}

In this question, we will write HOL codes that are equivalent to semantic ${P(x) \supset Q(y)}$.
\lstset{frameround = fttt}
\begin{lstlisting}[frame= trBL]
(*  6.2.1.1  *)
``P x ==> Q y``;
\end{lstlisting}

The code above will give us the execution transcript as below:
\setcounter{sessioncount}{0}
\begin{session}
\begin{scriptsize}
\begin{verbatim}

 > # <<HOL message: inventing new type variable names: 'a, 'b>>
val it =
   ``(P :'a -> bool) (x :'a) ==> (Q :'b -> bool) (y :'b)``:
   term

\end{verbatim}
\end{scriptsize}
\end{session}

\subsection{Question 6.2.1.2}

In this question, we will constrain the data types in the previous question. Specifically, we will constrain $x$ to \emph{num} type and $y$ to \emph{bool} type. Then we will compare the running result of this "restrained" statement and last exercise's "default" statement.
\lstset{frameround = fttt}
\begin{lstlisting}[frame= trBL]
(*  6.2.1.2  *)
``P (x:num) ==> Q (y:bool)``;
``P x ==> Q y``;
\end{lstlisting}

From the transcript below we could see that in the first evaluation, $x$ is of type \emph{num} and $y$ is of type \emph{bool}. Function $P$ is a mapping from type \emph{num} to \emph{bool} and $Q$ is a mapping from type \emph{bool} to type \emph{bool}. In the second evaluation, $x$ is of type $'a$ and $y$ is of type $'b$. Function $P$ is a mapping from type $'a$ to type \emph{bool} and $Q$ is a mapping from type $'b$ to type \emph{bool}.
\setcounter{sessioncount}{1}
\begin{session}
\begin{scriptsize}
\begin{verbatim}

> # val it =
   ``(P :num -> bool) (x :num) ==> (Q :bool -> bool) (y :bool)``:
   term
> <<HOL message: inventing new type variable names: 'a, 'b>>
val it =
   ``(P :'a -> bool) (x :'a) ==> (Q :'b -> bool) (y :'b)``:
   term
\end{verbatim}
\end{scriptsize}
\end{session}

\subsection{Question 6.2.1.3}
In this question, we will attempt to write codes that are equivalent to a "forall" statement $\forall x, y. P (x) \supset Q (y)$.

\lstset{frameround = fttt}
\begin{lstlisting}[frame= trBL]
(*  6.2.1.3  *)
``!x y.P x ==> Q y``;
\end{lstlisting}

If we send the above region to HOL window, we will see transcript as below:
\setcounter{sessioncount}{2}
\begin{session}
\begin{scriptsize}
\begin{verbatim}

> > # <<HOL message: inventing new type variable names: 'a, 'b>>
val it =
   ``!(x :'a) (y :'b). (P :'a -> bool) x ==> (Q :'b -> bool) y``:
   term

\end{verbatim}
\end{scriptsize}
\end{session}

\subsection{Question 6.2.1.4}

In this question, we will attempt to constrain $x$ to type \emph{num} but call it as a data of type $'a$.

\lstset{frameround = fttt}
\begin{lstlisting}[frame= trBL]
(*  6.2.1.4  *)
``?(x:num).R(x:'a)``;
\end{lstlisting}

If we send the above region to HOL, we will see transcript as below(line numbers and character numbers may change, due to different HOL windows). This is because we had previously constrained $x$ to type \emph{num} but later re-constrained it to type \emph{'a}. Therefore the type inference fails.

\setcounter{sessioncount}{3}
\begin{session}
\begin{scriptsize}
\begin{verbatim}

> #
Type inference failure: the term

(x :num)

on line 12, characters 13-17

can not be constrained to be of type

:'a

unification failure message: ???
Exception-
   HOL_ERR
     {message =
      "on line 12, characters 13-17:\n\nType inference failure: the term\n\n(x :num)\n\non line 12, characters 13-17\n
\ncan not be constrained to be of type\n\n:'a\n\nunification failure message: ???\n",
      origin_function = "type-analysis", origin_structure = "Preterm"} raised
\end{verbatim}
\end{scriptsize}
\end{session}

\subsection{Question 6.2.1.5}

In this question, we will attempt to implement a HOL equivalent statement. We will use HOL to show that "not all $x$ satisfies either $P(x)$ or $Q(x)$" is equivalent to "there exists some $x$, that does not satisfy either $P(x)$ nor $Q(x)$".
\lstset{frameround = fttt}
\begin{lstlisting}[frame= trBL]
(*  6.2.1.5  *)
``~(!x.(P x) \/ (Q x)) = ?x.(~(P x)) /\ (~(Q x))``;
\end{lstlisting}

The line above will give us the result like:
\setcounter{sessioncount}{4}
\begin{session}
\begin{scriptsize}
\begin{verbatim}

> # <<HOL message: inventing new type variable names: 'a>>
val it =
   ``~(!(x :'a). (P :'a -> bool) x \/ (Q :'a -> bool) x) <=>
  ?(x :'a). ~P x /\ ~Q x``:
   term

\end{verbatim}
\end{scriptsize}
\end{session}

\subsection{Question 6.2.1.6}
In this question, we will attempt to implement a HOL statement that is equivalent to "all people are mortal". We can interprete the statement this way: for all creature, if "it is a human", then "it is mortal". So we have the code below:

\lstset{frameround = fttt}
\begin{lstlisting}[frame= trBL]
(*  6.2.1.6  *)
``!x.(P x ==> M x)``;
\end{lstlisting}

and the execution result is like:
\setcounter{sessioncount}{5}
\begin{session}
\begin{scriptsize}
\begin{verbatim}

> # # <<HOL message: inventing new type variable names: 'a>>
val it =
   ``!(x :'a). (P :'a -> bool) x ==> (M :'a -> bool) x``:
   term

\end{verbatim}
\end{scriptsize}
\end{session}

\subsection{Question 6.2.1.7}
In this question, we will use HOL to represent a statement "some people are funny". I thought this is the equivalent meaning as "there exists some $x$, someone who is a people and someone who is funny".

\lstset{frameround = fttt}
\begin{lstlisting}[frame= trBL]
(*  6.2.1.7  *)
``?x.P x /\ Funny x``;
\end{lstlisting}
By putting the above region to HOL, we shall see the following result:
\setcounter{sessioncount}{0}
\begin{session}
\begin{scriptsize}
\begin{verbatim}

> <<HOL message: inventing new type variable names: 'a>>
val it =
   ``?(x :'a). (P :'a -> bool) x /\ (Funny :'a -> bool) x``:
   term
\end{verbatim}
\end{scriptsize}
\end{session}

\chapter{Exercise 7.3.1}
\label{cha:exercise-7.3.1}

\section{Problem Statement}
\label{sec:7.3.1-problem-statement}
In this problem, we will implement a HOL function \emph{andImp2Imp term}, which takes in terms of form $p \land q \supset r$ and returns $p \supset q \supset r$.

\section{Relevant Code}
\label{sec:7.3.1-relevant-code}

\lstset{frameround = ftttt, framexrightmargin = 0em}
\begin{lstlisting}[frame = trBL]
(*Takes in ``p/\q==>r`` and returns ``p==>q==>r``*)
fun andImp2Imp term =
    let
        val (tmp1, r) = dest_imp term
        val (p, q) = dest_conj tmp1
    in
        mk_imp(p, mk_imp(q,r))
end;

andImp2Imp ``p/\q==>r``;

\end{lstlisting}

\section{Execution Transcripts of Test Cases}
\label{sec:7.3.1-test-cases}
If we send the above region to HOL,  we will see result as below:
\setcounter{sessioncount}{0}
\begin{session}
\begin{scriptsize}
\begin{verbatim}

> # # # # # # # val andImp2Imp = fn: term -> term
> > > val it =
   ``(p :bool) ==> (q :bool) ==> (r :bool)``:
   term
>

\end{verbatim}
\end{scriptsize}
\end{session}

\chapter{Exercise 7.3.2}
\label{cha:exercise-7.3.2}

\section{Problem Statement}
\label{sec:7.3.2-problem-statement}
In this problem, we will create a function named \emph{impImpAnd term}, which takes in terms of form $p \supset q \supset r$ and returns $p \land q \supset r$. Apart of that, we will also show that this function is just a reverse of the previous function defined in Exercise 7.3.1.

\section{Relevant Code}
\label{sec:7.3.2-relevant-code}

\lstset{frameround = ftttt, framexrightmargin = 0em}
\begin{lstlisting}[frame = trBL]
(*Takes in ``p==>q==>r`` and returns ``p/\q==>r``*)

fun impImpAnd term =
    let
        val (p, tmp) = dest_imp term
        val (q, r) = dest_imp tmp
    in
        mk_imp(mk_conj(p,q),r)
end;

(*Takes in ``p/\q==>r`` and returns ``p==>q==>r``*)
fun andImp2Imp term =
    let
        val (tmp1, r) = dest_imp term
        val (p, q) = dest_conj tmp1
    in
        mk_imp(p, mk_imp(q,r))
end;

andImp2Imp ``p/\q==>r``;

impImpAnd(andImp2Imp ``(p/\q)==>r``);
andImp2Imp(impImpAnd ``p==>q==>r``);

\end{lstlisting}

\section{Execution Transcripts of Test Cases}
\label{sec:7.3.2-test-cases}

From the above code, we will see transcripts as below. The first \emph{val it} is the result of test case $p \supset q \supset r$. The second and the third \emph{val it} shows that \emph{impImpAnd} and \emph{andImp2Imp} are each others' reverse.

\setcounter{sessioncount}{0}
\begin{session}
\begin{scriptsize}
\begin{verbatim}

> val impImpAnd = fn: term -> term
val andImp2Imp = fn: term -> term
val it =
   ``(p :bool) ==> (q :bool) ==> (r :bool)``:
   term
val it =
   ``(p :bool) /\ (q :bool) ==> (r :bool)``:
   term
val it =
   ``(p :bool) ==> (q :bool) ==> (r :bool)``:
   term
val it = (): unit
>
*** Emacs/HOL command completed ***

\end{verbatim}
\end{scriptsize}
\end{session}

\chapter{Exercise 7.3.3}
\label{cha:exercise-7.3.3}

\section{Problem Statement}
\label{sec:7.3.3-problem-statement}
In this problem, we will implement a HOL function \emph{andImp2Imp term}, which takes in terms of form $\neg \exists x. P(x)$ and returns $\forall x. \neg P(x)$.

\section{Relevant Code}
\label{sec:7.3.3-relevant-code}

\lstset{frameround = ftttt, framexrightmargin = 0em}
\begin{lstlisting}[frame = trBL]
(*takes in ~?x.P x and returns !x.~(P x)*)

fun notExists term =
    let
        val (x, y)= dest_exists(dest_neg(term))
    in
        mk_forall(x, mk_neg y)
end;

notExists ``~?z.Q z``;

\end{lstlisting}

\section{Execution Transcripts of Test Cases}
\label{sec:7.3.3-test-cases}
If we send the above region to HOL,  we will see result as below:
\setcounter{sessioncount}{0}
\begin{session}
\begin{scriptsize}
\begin{verbatim}

> # # # # # # # val notExists = fn: term -> term
> > <<HOL message: inventing new type variable names: 'a>>
val it =
   ``!(z :'a). ~(Q :'a -> bool) z``:
   term
>

\end{verbatim}
\end{scriptsize}
\end{session}

\appendix{}
\chapter{Source Code for Exercise 6.2.1}
\label{appendix:A}
The following codes are from \emph{ex-6-2-1.sml}
\lstinputlisting{../ML/ex-6-2-1.sml}
\chapter{Source Code for Exercise 7.3.1}
\label{appendix:B}
The following code is from \emph{ex-7-3-1.sml}
\lstinputlisting{../ML/ex-7-3-1.sml}
\chapter{Source Code for Exercise 7.3.2}
\label{appendix:C}
The following code is from \emph{ex-7-3-2.sml}
\lstinputlisting{../ML/ex-7-3-2.sml}
\chapter{Source Code for Exercise 7.3.3}
\label{appendix:D}
The following code is from \emph{ex-7-3-3.sml}
\lstinputlisting{../ML/ex-7-3-3.sml}
\end{document}


\documentclass[11pt,a4wide]{article}
\usepackage{verbatim}
\usepackage{listings}
\usepackage{graphicx}
\usepackage{a4wide}
\usepackage{color}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage[dvips]{epsfig}
\usepackage[T1]{fontenc}
\usepackage{cite} % [2,3,4] --> [2--4]
\usepackage{shadow}
\usepackage{hyperref}

\setcounter{tocdepth}{2}

\lstset{language=c++}
\lstset{alsolanguage=[90]Fortran}
\lstset{basicstyle=\small}
\lstset{backgroundcolor=\color{white}}
\lstset{frame=single}
\lstset{stringstyle=\ttfamily}
\lstset{keywordstyle=\color{red}\bfseries}
\lstset{commentstyle=\itshape\color{blue}}
\lstset{showspaces=false}
\lstset{showstringspaces=false}
\lstset{showtabs=false}
\lstset{breaklines}

\newcommand{\der} [2]{\frac{\mathrm{d} #1}{\mathrm{d} #2}}   % Derivative.
\newcommand{\eq} [1]{Eq.\ (\ref{eq:#1})}
\newcommand{\Eq} [1]{Eq.\ (\ref{eq:#1})}

\begin{document}
\section*{Introduction to numerical projects}

Here follows a brief recipe and recommendation on how to write a report for each
project.
\begin{itemize}
\item Give a short description of the nature of the problem and the eventual
numerical methods you have used.
\item Describe the algorithm you have used and/or developed. Here you may find it convenient
to use pseudocoding. In many cases you can describe the algorithm
in the program itself.

\item Include the source code of your program. Comment your program properly.
\item If possible, try to find analytic solutions, or known limits
in order to test your program when developing the code.
\item Include your results either in figure form or in a table. Remember to
       label your results. All tables and figures should have relevant captions
       and labels on the axes.
\item Try to evaluate the reliabilty and numerical stability/precision
of your results. If possible, include a qualitative and/or quantitative
discussion of the numerical stability, eventual loss of precision etc.

\item Try to give an interpretation of you results in your answers to
the problems.
\item Critique: if possible include your comments and reflections about the
exercise, whether you felt you learnt something, ideas for improvements and
other thoughts you've made when solving the exercise.
We wish to keep this course at the interactive level and your comments can help
us improve it.
\item Try to establish a practice where you log your work at the
computerlab. You may find such a logbook very handy at later stages
in your work, especially when you don't properly remember
what a previous test version
of your program did. Here you could also record
the time spent on solving the exercise, various algorithms you may have tested
or other topics which you feel worthy of mentioning.
\end{itemize}

\section*{Format for electronic delivery of report and programs}
%
The preferred format for the report is a PDF file. You can also
use DOC or postscript formats or as an ipython notebook file.
As programming language we prefer that you choose between C/C++, Fortran2008 or Python.
The following prescription should be followed when preparing the report:
\begin{itemize}
\item Use Devilry to hand in your projects, log in  at
\url{ http://devilry.ifi.uio.no} with your normal UiO username and password.
There you can load up the files within the deadline.
\item Upload {\bf only} the report file!  For the source code file(s) you have developed please provide us with your link to your github domain.
The report file should include all of your discussions and a list of the codes you have developed.
Do not include library files which are available at the course homepage, unless you have
made specific changes to them.
\item In your git repository, please include a folder which contains selected results. These can be in the form of output from your code
for a selected set of runs and input parameters.
\item Comments  from us on your projects, approval or not, corrections to be made
etc can be found under
your Devilry domain and are only visible to you and the teachers of the course.

\end{itemize}

Finally,
we encourage you to work two and two together. Optimal working groups consist of
2-3 students. You can then hand in a common report.

\section*{Project 1, Challenge version, deadline Monday 19 September 12pm (noon)}

This project has many of the same aims as the normal version but it
contains also additional challenges.  In particular, those of you who
are familiar with numerical algorithms behind Gaussian elmination and C++ programming could
attempt at studying this alternative project.
The difficulty of this project is a minor step up from
the standard project, but the extra difficulty lies almost exclusively
in the mathematics and the handling of differential equations and
their boundary conditions. The programming involved is no
more difficult than in the standard project 1.

The programming aim of this project is to get familiar with various vector and matrix operations,
from dynamic memory allocation to the usage of programs in the library
package of the course.
For Fortran users memory handling and most matrix and vector operations
are included in the ANSI standard of Fortran 90/95. Array handling in Python is also rather trivial. For C++ user however,
there are several possible options. Two are listed here.
\begin{enumerate}
\item For this exercise we recommend that you make your own functions for dynamic memory allocation of a
vector and a matrix. You don't need to write a class for this operations.
Use then the
library package lib.cpp with its header file
lib.hpp for obtaining LU-decomposed matrices, solve linear equations
etc.
\item A very good and often recommended library for C++ handling of arrays is the library Armadillo, to be found at \url{arma.sourceforge.net}.  We will discuss the usage of this library during the lab sessions and lectures. Armadillo has also an interface to Lapack functions for solving systems of linear equations.
\end{enumerate}

Your program, whether it is written in C++, Python
or Fortran2008, should include
dynamic memory handling of matrices and vectors.

The material needed for this project is covered by chapter 6 of the lecture notes, in particular section 6.4 and subsequent sections.

Many important differential equations in the Sciences can be written as linear second-order differential equations
\begin{align}
\der{^2 y}{x^2} + k^2(x) y = f(x),
\end{align}
where $f$ is normally called the inhomogeneous term and $k^2$ is a real function.

A classical equation from electromagnetism is Poisson’s equation. The electrostatic poten-
tial $\Psi$ is generated by a localized charge distribution $\rho(r)$. In three dimensions it reads
\begin{align}
\nabla^2 \Psi = -4\pi \rho(r).
\end{align}
With a spherically symmetric $\Psi$ and $\rho(r)$ the equations simplifies to a one-dimensional equation in $r$, namely
\begin{align}
\frac{1}{r^2}\der{}{r} \left(r^2 \der{\Psi}{r} \right) = -4\pi \rho(r),
\end{align}
which can be rewritten via a substitution $\Psi(r) = \psi(r)/r$ as
\begin{align}
\der{^2\psi}{r^2} = -4\pi r \rho(r).
\end{align}
The inhomogeneous term $f$ or source term is given by the charge distribution $\rho$ multiplied by $r$ and the constant $−4\pi$.

We will rewrite this equation by letting $\psi \rightarrow u$ and $r\rightarrow x$. The general one-dimensional Poisson equation reads then
\begin{align}
-u''(x) = f(x).
\end{align}

\begin{itemize}
  \item[(a)] In this project we will solve the one-dimensional Poissson equation with Dirichlet bound-
ary conditions by rewriting it as a set of linear equations.

To be more explicit we will solve the equation
\begin{align}
-u''(x) = f(x), \ \ \ x\in(0,1), \ \ \ u(0)=u(1)=0,
\end{align}
and we define the discretized approximation to $u$ as $v_i$ with grid points $x_i = ih$ in the interval from $x_0 = 0$ to $x_n = 1$. The step length or spacing is defined as $h = 1/n$. We have then the boundary conditions $v_0 = v_{n+1} = 0$. Now, where we would ordinarily approximate the second derivative with a three point formula, we will use a more sophisticated method. We will instead use the five point method
\begin{align}
-v_i'' \approx -\frac{-v_{i-2}+16v_{i-1}-30v_i+16v_{i+1}-v_{i+2}}{12h^2} = f_i \ \ \ \text{ for } i=0,1,2,\dots,n, \label{eq:1}
\end{align}
where $f_i=f(x_i)$. Please note that the local error of this five point method scales as $\mathcal{O}(h^4)$, whereas the corresponding local error of the three point method scales as $\mathcal{O}(h^2)$. We thus expect the five point method to result in significantly less error at the same number of points, $n$.

Show that you can rewrite this equation as a linear set of equations of the form
\begin{align}
A{\bf v} = {\bf \tilde{b}},
\end{align}
where $\tilde{b}_i = -12h^2 f_i$ and $A$ is the $n\times n$ pentadiagonal matrix
  \begin{align}
A = \begin{bmatrix}  &        &        &         &       &      &       &       &       &       \\
                            & \ddots &        &    -1   &       &      &       &       &       &       \\
                            &        &  -30   &    16   &   -1  &      &       &       &       &       \\
                            &    -1  &   16   &   -30   &   16  &  -1  &       &       &       &       \\
                            &        &   -1   &    16   &  -30  &  16  &   -1  &       &       &       \\
                            &        &        &    -1   &   16  & -30  &   16  &   -1  &       &       \\
                            &        &        &         &   -1  &  16  &  -30  &   16  &  -1   &       \\
                            &        &        &         &       &  -1  &   16  &  -30  &       &       \\
                            &        &        &         &       &      &   -1  &       & \ddots&       \\
                            &        &        &         &       &      &       &       &       &
\end{bmatrix}
  \end{align}
Recall that the vector ${\bf v}$ is the discretized solution vector. Note: Do \emph{not} worry about the boundary conditions just yet, i.e. just consider the internal points $i=2,3,\dots,n-3,n-2$.

In our case we will assume that the source term is $f(x) =100\exp{(−10x)}$, and keep the same interval and boundary conditions. Then the above differential equation has a closed-form solution given by
\[
u(x) = 1 -\left(1−\exp{(-10)}\right)x - \exp{(-10x)}.
\]
Convince yourself that this is correct by inserting the solution in the Poisson equation, or solve
the equation explicitly by integrating $u''(x)$ twice and use the
boundary conditions to determine the resulting two integration
constants. We will compare our numerical solution with this result in
the next exercise.
\end{itemize}
We will now consider in some detail what happens near the boundaries, i.e. at $x=0$ and $x=1$. Since we know the solution at the points $x_0=0$ and $x_n=1$, we may demand that our equation set adhere to this by inserting the equation
\begin{align}
v_0 = 0.
\end{align}
This corresponds to setting the first row of the matrix $A$ equal to $\begin{bmatrix}1 & 0 & 0 & \dots & 0 \end{bmatrix}$, while also setting $\tilde{b}_0=0$.

Next, we need to figure out what to do about the $i=1$ equation from \eq{1}, which states
\begin{align}
-v_{-1}+16v_{-0}-30v_1+16v_{2}-v_{3} = -12h^2f_0. \label{eq:2}
\end{align}
The reason why this is a problem is the presence of the $v_{-1}$ term. According to our discretization from earlier, we only have $v_i$ defined for points $i=0,1,2,\dots,n$. A common way of dealing with this extra term is to consider the derivative of $v$ at the boundary. Consider the central difference approximation to the first derivative,
\begin{align}
v'(x) \approx \frac{v(x+h)-v(x-h)}{h}. \label{eq:3}
\end{align}
\begin{itemize}
  \item[(b)] Use \eq{3} at $x=0$, in order to derive an expression for $v_{-1}$ in terms of $v_1$, $h$. Use the extra boundary condition $u'(0)=9+e^{-10}$. Insert the expression for $v_{-1}$ into \eq{2} and insert the result into the second row of the matrix $A$. Remember to also modify the right hand side vector, ${\bf \tilde{b}}$, in the appropriate way. Note: Keep careful track of the signs!
  \item[(c)] Perform the same steps at the right boundary, at $x=1$. In other words, modify the last row of the matrix $A$ and the last element of the right hand side vector, $\tilde{b}_n$, in the appropriate way. Also, set up the $i=n-1$ version of \eq{1}, and use the derivative of $u$ at $x=1$ in order to eliminate the $v_{n+1}$ element. Use the extra boundary condition $u'(1)=-1+11e^{-10}$.

\item[(d)] Having now set up the matrix fully, we are ready to derive the algorithm for solving matrix-vector equation. First, we can rewrite our matrix $A$ in terms of one-dimensional vectors $a, b, c, d$, and $e$ of length $n+1$. Our linear equation reads
\begin{align}
A{\bf v}=\begin{bmatrix}
                       c_0 & d_0  & e_0  & & & &\\
                       b_1 & c_1 & d_1  & e_1  & & & &\\
                       a_2 &  b_2 & c_2 & d_2  & e_2  & & &\\
                       & a_3  & b_3  & c_3 & d_3 & e_3 & &\\
                       & \dots    &  \dots   &  \dots   &  \dots &\dots  &\dots  & \dots\\
                       &     &     & a_{n-2}  & b_{n-2} &  c_{n-2} & d_{n-2} & e_{n-2} \\
                       &     &     &     & a_{n-1}  & b_{n-1} &  c_{n-1} & d_{n-1} \\
                       &     &     &     &     &  a_n & b_n &  c_n \end{bmatrix}
\begin{bmatrix}v_0 \\ v_1 \\ v_2 \\ \\ \vdots \\ \\ \\ v_n\end{bmatrix} =
\begin{bmatrix}\tilde{b}_0 \\ \tilde{b}_1 \\ \tilde{b}_3 \\ \\ \vdots \\ \\ \\ \tilde{b}_n \end{bmatrix}.
\end{align}
Now, perform Gaussian elimination on the augmented matrix $\begin{bmatrix}A & {\bf \tilde{b}}\end{bmatrix}$ in order to arrive at an efficient algorithm for solving the pentadiagonal matrix system.

Hint: Start from the top and use the $c_i$ element in row $i$ to eliminate $b_{i+1}$ and $a_{i+2}$ in the subsequent two rows. Remember to account for what happens to the elements of the right hand side vector. After doing this all the way down to the $n$th row, you should have eliminated all the $a$s and all the $b$s. You may need to handle the last $b$ element, $b_n$, outside of the loop

Next, start from the $n$th row and work your way up, solving for $v_i$ as you go. On row $i$, you will need the solution from the previous two rows, $v_{i+1}$ and $v_{i+2}$.
\item[(e)] Set $n=10$ and solve the matrix-vector equation you set up in exercises (a) through (c), using your newly developed algorithm.
  Test your implementation by comparing the solution to one obtained by a solver known to be correct (e.g. the built-in solve function in Armadillo, the \textbackslash \ function in Matlab, or the solve function in Numpy, etc.) If the two solutions differ by more than say $~10^{-14}$, your solver is not implemented correctly \emph{or} the matrix $A$ is not set up correctly.

  \item[(f)] Use your solver to solve systems with $n$ ranging from $n=10$ to $n=10^6$ (use at least $6$ different $n$ in this range), and compute the maximum value of the absolute error in your approximation for each $n$. Use the known exact solution, $u(x) = 1-(1-e^{-10})x-e^{-10x}$.

  Plot the error as a function of $n$ in a log-log plot. Comment on what you see. What is the slope of the curve in your log-log plot, and what does it tell you?

  Note: Which $n$ values you choose is up to you. The simplest possible way is to use $n=10,10^2,10^3,10^4,10^5,10^6$, but the plot will be easier to interpret if you choose a more sophisticated scheme and more points. Using $N$ linearily spaced points between $n=10$ and $n=10^6$ might give a better result, but you may note that the data is very dense for low values of $n$ and very sparse for the large values. The best alternative is to use $N$ logarithmically spaced points from $n=10$ to $n=10^6$. Using $N=50$ should be sufficient.
\item[(g)] Calculate how the absolute error, $\max_{x\in(0,1)}|u(x)-v(x)|$ scales as a function of the step length, $h$. Remember that the five point approximation we use for the second derivative has an error scaling on the order of $\mathcal{O}(h^4)$. Is the error you find in line with what you would expect, given this? If not, can you offer an explanation as to why we dont see this behaviour?
\end{itemize}

\end{document}


%Comment out lines 3-4 if you want slides, line 2 if you want notes
%\documentclass[ignorenonframetext, red]{beamer}
\documentclass[11pt,a4paper]{article}
\usepackage{beamerarticle}
\mode<article>
{
	\usepackage[colorlinks,
	pdfauthor="Len Thomas",
	pdftitle="MT4113 Lecture notes"]{hyperref}
    \usepackage[a4paper,margin=1cm,footskip=.5cm]{geometry}
}
\mode<presentation>
{
%    \usetheme{Singapore}
  \pgfdeclareimage[height=1cm]{StA-logo}{logo}
  \logo{\pgfuseimage{StA-logo}}
    \usetheme{EastLansing}
%    \usetheme{JuanLesPins}
%    	\usecolortheme{sidebartab}
	\setbeamertemplate{navigation symbols}{\insertsectionnavigationsymbol}
}

\usepackage[english]{babel}
\usepackage[latin1]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{epsf,graphics,graphicx,fancyhdr,color,amsmath,url,enumerate,alltt}
\usepackage{ifthen}

%Commented out proportional font -- use a monospaced font instead
% -- better for displaying the tables with binary numbers in them, etc, in this lecture
%\usepackage[sfdefault]{cabin}
\usepackage[scaled=0.85]{beramono}

\newcommand{\bc}{\begin{center}}
\newcommand{\ec}{\end{center}}
\newcommand{\bn}{\begin{enumerate}}
\newcommand{\en}{\end{enumerate}}
\newcommand{\be}{\begin{eqnarray}}
\newcommand{\ee}{\end{eqnarray}}
\newcommand{\bes}{\begin{eqnarray*}}
\newcommand{\ees}{\end{eqnarray*}}

\title[MT4113]{MT 4113, Computing in Statistics}

\subtitle{Lecture 4 - An introduction to computer arithmetic}

\author[Lecture 4]{Len Thomas}

%\institute{School of Mathematics and Statistics, University of St Andrews}

\date[1/10/2018] {Oct 1 2018}
\begin{document}

\begin{frame}
  \titlepage
\end{frame}

\mode<article>{
\maketitle
}

\mode<presentation> {
\begin{frame}
  \frametitle{Outline}
	\tableofcontents
\end{frame}
}

%-----------------------------------------------------------------

\section{An Introduction to Computer Arithmetic}

\subsection{Introduction}

\begin{frame}
	\frametitle{About this lecture}

This is a rough guide to how computers store and manipulate numbers\footnote{A much more
in-depth treatment is in, e.g., Chapter 2 of Gentle, J.E. Computational Statistics, available from the library as an electronic book.}
\end{frame}

\begin{frame}[fragile]
	\frametitle{Motivating example using \texttt{R}}

What does this code do?

\begin{verbatim}
x <- 0.1
x <- x + 0.05
x

if (x == 0.15) {
  cat("Len is a great guy!\n")
} else {
  cat("Len is a loser!\n")
}
\end{verbatim}

\end{frame}

\begin{frame}
	\frametitle{Numbers on computers}

	\begin{itemize}
		\item Computers really can't do arithmetic.  They recognize two states: on (1) and off (0).  Numbers on computers are really a collection of binary digits (`bits') -- they only have meaning by human convention.

		\item Numbers are represented on computers in two ways:
		\begin{itemize}
      \item fixed point (integers)
	    \item floating point (reals)
		\end{itemize}
	\end{itemize}

\end{frame}

\begin{frame}
	\frametitle{Definitions: Data size}

\begin{itemize}
    \item bit: short for \underline{bi}nary digi\underline{t}
      \\smallest unit of storage on a machine -- holds either 0 or 1
    \item byte: short for \underline{by}nary \underline{te}rm \footnote{Computer scientists can't spell!}
      \\8 bits -- enough to hold a single character of text (see later)
    \item word: natural data size of a computer.  Depends on the CPU -- for most desktop systems currently it is 64 bits (or 32 bits for older machines).
    \item kilobyte (K): $2^{10}=1,024$ bytes
    \item megabyte (MB): $2^{20}=1,048,576$ bytes$ = 1,024$ K
    \item gigabyte (GB): $2^{30}=1,073,741,824$ bytes$\footnote{Hard disk drive manufacturers sometimes prefer to use $1,000,000,000$ bytes as it makes their drives look bigger.} = 1,024$ MB
\end{itemize}

\end{frame}

\subsection{Fixed point}[fragile]

\begin{frame}
	\frametitle{Fixed point representations}

	\begin{itemize}
	    \item Example representation: signed integer\footnote{there are other more efficient representations, e.g., 1s complement and 2s complement}
	    \begin{itemize}
	        \item 1st bit is the sign (0=+ve, 1=-ve)
	        \item Remaining bits are the absolute value in base 2
	    \end{itemize}
	    \item <2-> E.g., on a 4-bit system:
	    \bc
	    \begin{tabular}{ll}
	        0000 is 0 & 1000 is -0 \\
	        0001 is 1 & 1001 is -1 \\
	        0010 is 2 & 1010 is -2 \\
	        0011 is 3 & 1011 is -3 \\
	        $\vdots$  & $\vdots$\\
	        0111 is 7 & 1111 is -7
	    \end{tabular}
	    \ec
	\end{itemize}
\end{frame}

\begin{frame}[fragile]
	\frametitle{Fixed point arithmetic - addition}

	\begin{itemize}
	    \item E.g. addition:
	    \bc
	    \begin{tabular}{rrl}
	        3  & 0011 &\\
	        +2 & 0010 &\\
	        \cline{1-2}
	        5  & 0101 & add
	    \end{tabular}
	    \ec
	\end{itemize}

\end{frame}

\begin{frame}[fragile]
	\frametitle{Fixed point arithmetic - multiplication}

	\begin{itemize}
	    \item Multiply by a power of 2 - just shift the bits left and fill in rightmost bits with zeros
	    \bc
	    \begin{tabular}{rrl}
	        3  & 0011 &\\
	        x2 & 0010 &\\
	        \cline{1-2}
	        6  & 0110 & x2 $\Rightarrow$ shift left one
	    \end{tabular}
	    \ec
	    \item <2-> If not a power of 2, use the `shift and add' algorithm: express as a sum of powers of 2.\\
			E.g., $2\times3 = (2\times2) + (2\times1)$
	    \item <3-> \bc
	    \begin{tabular}{rrl}
	        2  & 0010 &\\
	        x3 & 0011 &\\
	        \cline{1-2}
					4 & 0100 & x2 $\Rightarrow$ shift left one\\
					2 & 0010 & x1 $\Rightarrow$ shift left none\\
	        \cline{1-2}
	        6  & 0110 & add
	    \end{tabular}
	    \ec
	\end{itemize}

\end{frame}

\begin{frame} [fragile]
	\frametitle{Fixed point arithmetic - subtraction and division}

	\begin{itemize}
			\item <1-> Dividing by a power of two is easy
			\begin{itemize}
				\item <2->just shift right (and drop any fractional part)
			\end{itemize}
	    \item <3-> Otherwise, dividing and subtracting are not very intuitive (or fast) in the signed integer representation, so we do not cover them explicitly here
			\item <3-> They are better (easier) in other representations -- e.g., 1s or 2s complement
	\end{itemize}

\end{frame}

\begin{frame}[fragile]
	\frametitle{Fixed point overflow}

    \bc
    \begin{tabular}{rr}
        3  & 0011\\
        +5 & 0101\\
        \cline{1-2}
        0  & 0000
    \end{tabular}
    \ec
\begin{itemize}
    \item <2->Machine needs a protocol to deal with and warn user of fixed point overflow
\end{itemize}
\end{frame}

\begin{frame}[fragile]
	\frametitle{Fixed point numbers in R}

\begin{itemize}
    \item <1->In R, the fixed point data type is called \texttt{integer}.
		\item <2->Integers aren't actually used much in R, except in loops and indexing elements in arrays.
		\item <3->Integers in R are stored in 32 bits.
\begin{semiverbatim}\mode<presentation>{\tiny}{}
> as.integer(2 ^ 31 - 1)
[1] 2147483647
> as.integer(2 ^ 31 - 1) + as.integer(1)
[1] NA
Warning message:
In as.integer(2 ^ 31 - 1) + as.integer(1) : NAs produced by integer overflow\end{semiverbatim}
\end{itemize}
\end{frame}

\begin{frame}[fragile]
	\frametitle{Summary: Fixed point representations}

	 \begin{itemize}
	    \item Advantages:
	    \begin{itemize}
	        \item Results are exact
	        \item Fast (compared with floating point)
	    \end{itemize}
	    \item <2-> Disadvantages:
	    \begin{itemize}
	        \item Limited applicability: integers and related (e.g., financial applications, dates\footnote{Dates can use unsigned integers representations})
	        \item Limited range
	    \end{itemize}
	\end{itemize}

\end{frame}

\subsection{Floating point}

\begin{frame}
	\frametitle{Floating point representations}

	\begin{itemize}
	    \item Example $S .F \times B^E$ e.g., $+.1 \times 10^{+1}$
	    \begin{itemize}
	        \item S = sign
	        \item F = fraction -- unsigned integer.  Typically `normalized' so that 1st digit is non-zero.
	        \item B = base
	        \item E = exponent -- signed integer.
	    \end{itemize}
	    \item On a computer
	    \begin{itemize}
	        \item B is 2
	        \item S is 1 bit (0=+, 1=-)
	        \item F and E are allocated a fixed number of bits each
	    \end{itemize}
	\end{itemize}
\end{frame}
\begin{frame}[fragile]
	\mode<presentation>{\frametitle{Floating point representations}}
	\begin{itemize}
	    \item For example, consider an 8-bit floating point representation
	    with
	    \begin{itemize}
	        \item 1 bit for the sign
	        \item 3 for the exponent (a signed integer)
	        \item 4 for the fraction (an unsigned integer)
	    \end{itemize}
	\end{itemize}
	    \bc
	    {\tiny
	    \begin{tabular}{rcrcrcrrr}
	        & & \multicolumn{1}{c}{decimal} &   & \multicolumn{1}{c}{binary} & &
	            \multicolumn{1}{c}{sign} & \multicolumn{1}{c}{exponent} &
	            \multicolumn{1}{c}{fraction}\\
	        1 & = & $+.1 \times 10^{+1}$ & = & $+.1_{\mbox{two}} \times 2^{+1}$ & = &
	            0 & 001 & 1000\\
	        3 & = & $+.3 \times 10^{+1}$ & = & $+.11_{\mbox{two}} \times 2^{+2}$ & = &
	            0 & 010 & 1100\\
	    $-1/8$ & = & $-.125 \times 10^{+0}$ & = & $-.1_{\mbox{two}} \times 2^{-2}$ & = &
	            1 & 110 & 1000\\
	    \end{tabular}
	    }
	    \ec
\end{frame}

\begin{frame}[fragile]
	\frametitle{Limitations}

	\begin{itemize}
    \item Given that the exponent and fraction are stored in a fixed number of bits, there are limitations both to the size and accuracy with which numbers can be stored
    \item <2-> Size: in our example
    \begin{itemize}
        \item Largest possible number is
				\item <3-> $+.1111_{\mbox{two}} \times 2^{3} = 7.5$
        \item <4-> Smallest possible positive number\footnote
        {if we allow the leading digits of the fraction to be zero (`de-normalized')}
        is
				\item <5-> $+.0001_{\mbox{two}} \times 2^{-3} = 1/128 = 0.0078125$
    \end{itemize}
	\end{itemize}
\end{frame}
\begin{frame}
	\mode<presentation>{\frametitle{Limitations}}
	\begin{itemize}
    \item Accuracy:
    \bc
    \begin{tabular}{rrrrr}
        & \multicolumn{1}{c}{binary} &
            \multicolumn{1}{c}{sign} & \multicolumn{1}{c}{exponent} &
            \multicolumn{1}{c}{fraction}\\
        1 & $.1000_{\mbox{two}} \times 2^{+1}$ &
            0 & 001 & 1000\\
        +1/128 & $+.0001_{\mbox{two}} \times 2^{-3}$ &
            0 & 111 & 0001\\
        \hline
        1 & $.1000_{\mbox{two}} \times 2^{+1}$ &
            0 & 001 & 1000
    \end{tabular}
    \ec
    \begin{itemize}
        \item to see why, re-express 1/128 as
		    \bc
		    \begin{tabular}{rrrrr}
		        +1/128 & $+.00000001_{\mbox{two}} \times 2^{+1}$ &
		            0 & 001 & 00000001\\
		    \end{tabular}
		    \ec
    \end{itemize}
	\end{itemize}
\end{frame}
\begin{frame}[fragile]
	\mode<presentation>{\frametitle{Limitations}}
	\begin{itemize}
    \item Let $\epsilon$ be the smallest positive number, such that $\epsilon + 1.0 \neq 1.0$
    \item In our example
    \begin{itemize}
			\item <2-> $\epsilon = 1/8$:
    \bc
%    \ifthenelse{\talk = 1}{\small}{}
    \begin{tabular}{rrrrr}
        & \multicolumn{1}{c}{binary} &
            \multicolumn{1}{c}{sign} & \multicolumn{1}{c}{exponent} &
            \multicolumn{1}{c}{fraction}\\
        1 & $.1000_{\mbox{two}} \times 2^{+1}$ &
            0 & 001 & 1000\\
        +1/8 & $+.1000_{\mbox{two}} \times 2^{-2}$ &
            0 & 110 & 1000\\
        \hline
        1 1/8 & $.1001_{\mbox{two}} \times 2^{+1}$ &
            0 & 001 & 1001
    \end{tabular}
    \ec
        \item <2-> to see why, re-express 1/8 as
		    \bc
		    \begin{tabular}{rrrrr}
		        +1/8 & $+.0001_{\mbox{two}} \times 2^{+1}$ &
		            0 & 001 & 0001\\
		    \end{tabular}
		    \ec
    \end{itemize}
    \item <3->In general, $\epsilon$, the  `machine epsilon' (or closely related `machine unit'), governs the relative accuracy of calculations (see later for an example).   $\epsilon = 2^{(1-d)}$ where $d$ is the number of bits in the fraction.
	\end{itemize}
\end{frame}

\begin{frame}[fragile]
	\frametitle{Some consequences of lack of accuracy}

		For clarity, the examples here use a base 10 representation with 4 decimal places in the fraction.

	\begin{itemize}
	    \item <2-> Things rarely add up exactly, due to rounding errors.
	\bes
	    1/3+1/3+1/3 &=& 1 \\
	    .3333 +.3333 + .3333 &=& .9999
	\ees
	    \item <3->Order of summation matters (associative rule doesn't work).  E.g.:
	\bes
	        1.000 + .0001 + \{\mbox{9,998 more .0001s}\} + .0001 = 1.000\\
	        .0001 + \{\mbox{9,998 more .0001s}\} + .0001 + 1.000 = 2.000
	\ees
	    \item <4->When adding $n$ values to $x$, the maximum error $<\frac{n  \epsilon x}{2}$.
	\end{itemize}
\end{frame}
\begin{frame}[fragile]
	\mode<presentation>{\frametitle{Some consequences of lack of accuracy}}
	\begin{itemize}

	    \item Mixed signs can cause problems -- `catastrophic cancellation': subtraction of two nearly equal numbers eliminates the most significant digits, and any small rounding errors that were previously hidden end up becoming important.
	    \begin{itemize}
	        \item <2->Poor example: $.1000 \times 10^0 - .9999 \times 10^{-1} = .1000 \times 10^{-3}$ instead of $.1000 \times 10^{-4}$ -- so the answer is wrong in what is now the most significant digit.
	        \item <3->Better example: calculating the sample variance using the usual $s^2=\frac{\sum{\left(x^2\right)}}{n} - \left(\frac{\sum{x}}{n}\right)^2$ -- can get -ve estimates!  If you want to see, try it for the sample $\{356, 357, 358, 359, 360\}$ keeping 4 decimal places in the fraction.

	    \end{itemize}
	\end{itemize}

\end{frame}

\begin{frame}[fragile]
	\frametitle{Floating point numbers in R}

	\begin{itemize}
    \item <1->In R, the floating point data type is called \texttt{numeric}.
		\item <2->It is the default for numbers in R.
\begin{semiverbatim}\mode<presentation>{\footnotesize}
> x <- 3
> class(x)
[1] "numeric"
\end{semiverbatim}
    \item <3->\texttt{numeric} values are represented using an IEEE standard for double precision.  64 bits in total, 1 bit for the sign, 11 bits for the exponent, 53 for the the fraction (uses a trick to gain 1 extra bit).
	\end{itemize}
\end{frame}
\begin{frame}[fragile]
	\mode<presentation>{\frametitle{Floating point numbers in R}}
	\begin{itemize}
    \item So,
    \begin{itemize}
        \item <2-> the largest possible number is\footnote{it is slightly higher, due to some tricks}
         $+.1111\ldots_{\mbox{two}} \times 2^{+1023} = 8.98\times 10^{+307}$
\begin{semiverbatim}
> 1 * 2 ^ 1023
[1] 8.988466e+307
> 2 * 2 ^ 1023
[1] Inf\end{semiverbatim}
        \item <3-> accuracy: machine epsilon is\footnote{again, it's slightly lower, due to some tricks} $\epsilon = 2^{(1-53)} = 2.20\times 10^{-16}$
\begin{semiverbatim}
> 1 + 2 ^ (1 - 53) == 1
[1] FALSE
> 1 + (0.5 * 2 ^ (1 - 53)) == 1
[1] TRUE\end{semiverbatim}
    \end{itemize}
	\end{itemize}
\end{frame}
\begin{frame}[fragile]
	\mode<presentation>{\frametitle{Floating point numbers in R}}
  \begin{itemize}
    \item <1->The standard dictates how some arithmetic operations should be done, and defines special values +Inf and -Inf (for overflows), and NaN (not a number).
\begin{semiverbatim}\mode<presentation>{\footnotesize}
> 1 / 0
[1] Inf
> 0 / 0
[1] NaN
> sqrt(-1)
[1] NaN
Warning message:
In sqrt(-1) : NaNs produced
> Inf / Inf
[1] NaN
\end{semiverbatim}
    \item <2->Underflows are set to 0, and no warning is generated.
\begin{semiverbatim}\mode<presentation>{\footnotesize}
> exp(-1000)
[1] 0
\end{semiverbatim}

\end{itemize}

\end{frame}

\begin{frame}[fragile]
	\frametitle{Living with floating point limitations}

	\begin{itemize}
	  \item Re-work expressions and algorithms to
    \begin{itemize}
      \item<2-> Avoid generating very large numbers that might overflow.  Underflow is generally better than overflow.  E.g. Rewrite $\frac{e^x}{1+e^x}$ as $\frac{1}{1+e^{-x}}$
	\begin{semiverbatim}\mode<presentation>{\footnotesize}
	> x <- 1000
	> exp(x) / (1 + exp(x))
	[1] NaN
	> 1 / (1 + exp(-x))
	[1] 1\end{semiverbatim}
	   \item <3->Minimize large differences in scale in intermediate calculations, so as to preserve accuracy.  For example, instead of taking the product of a large number of likelihoods, take the sum of the log likelihoods and then exponentiate.
	 \end{itemize}
  \end{itemize}
\end{frame}
\begin{frame}[fragile]
	\mode<presentation>{\frametitle{Living with floating point limitations}}
  \begin{itemize}
     \item Normalize\footnote{Note: different meaning here from normalizing the fractional part of a floating point number} -- scale values so they are centred near 0, and perhaps divide by some suitably large value so they don't overflow.  Allows the maximum range of values.
	    \item <2->Change exact tests to `good enough` tests.  From\\
	\texttt{if(x == y) then...}\\
	to\\
	\texttt{if(((x - e) <= y) \& (x + e >= y)) then ...}\\
	or, better (scale independent)\\
	\texttt{if(abs(x - y) <= abs(y) * e) then ...}

  \end{itemize}
\end{frame}
\begin{frame}[fragile]
	\mode<presentation>{\frametitle{Living with floating point limitations}}
  \begin{itemize}

	    \item Be careful about the order of operations - e.g., consider sorting numbers before adding (there are even smarter tricks here).

	    \item <2-> If you're desperate, use double-precision (at least for intermediate calculations) -- but it uses twice the memory and can use 3-4 times the time. (Although this does depend on the size of the floats relative to computer word size, so best to try it and see.)\footnote{Doesn't apply to R, which always uses double precision.  If you want even more accuracy, you can use the \texttt{Rmpfr} package.}
	\end{itemize}

\end{frame}
\begin{frame}[fragile]
	\mode<presentation>{\frametitle{Living with floating point limitations}}

	Some comments:
	\begin{itemize}
	    \item Note that many of these show a trade-off between accuracy and speed.
	    \item <2->So, first ask yourself if it's going to be a problem, before spending lots of time fixing a problem that isn't important!
	\end{itemize}

\end{frame}

\begin{frame}[fragile]
	\frametitle{Summary: floating point representations}

	\begin{itemize}
	    \item Advantages:
	    \begin{itemize}
	        \item Wide applicability (all numbers on the real line)
	        \item Wider range than fixed point
	    \end{itemize}
	    \item <2->Disadvantages:
	    \begin{itemize}
	        \item Results are not exact -- need to be careful
	        \item Slow (compared with fixed point)
	    \end{itemize}
	\end{itemize}
\end{frame}

\begin{frame}[fragile]
	\frametitle{Motivating example \texttt{R}edux}

What should we do instead of this?

\begin{verbatim}
if (x == 0.15) {
  cat("Len is a great guy!\n")
} else {
  cat("Len is a loser!\n")
}
\end{verbatim}
\end{frame}

\mode<presentation>
%This frame not shown in the handout!
\begin{frame}[fragile]
  \frametitle{Motivating example \texttt{R}edux}

Good enough test, such as:

\begin{verbatim}
e <- 1E-15
if (abs(x - 0.15) < 0.15 * e) {
  cat("Len is a great guy!\n")
} else {
  cat("Len is a loser!\n")
}
\end{verbatim}
\end{frame}

\mode*
%Back to showing things in the handout
\subsection{Character storage}

\begin{frame}[label=current]
	\frametitle{Character storage\footnote{OK - I know characters aren't numbers but I thought I'd include this stuff anyway}}

	\begin{itemize}
	    \item Characters are stored as binary numbers on the computer.  The correspondence between the stored numbers and the displayed symbols is determined by a character set.
	\end{itemize}
\end{frame}
\begin{frame}[label=current]
	\mode<presentation>{\frametitle{Character storage}}
	\begin{itemize}
	    \item Example character sets:
	    \begin{itemize}
	        \item ASCII (American Standard Code for Information Exchange).  A 7-bit system, so there are 128 unique characters.  0-31 are non-printing control characters (e.g. 27 is ESCape), the other 96 are alphanumeric characters and symbols (e.g. 63 is ?, 77 is M).
	        \item Extended ASCII. A set of 8-bit systems - the extra 128 characters are used for European accents, etc.  There are many regional variants.  Used in old versions of windows, and (I think) in R.
	        \item UNICODE.  A 16-bit system, so there are 65,536 characters.  Designed to include extended-character languages like Chinese and Japanese - but not really sufficient for that.  Used in MS Windows (2000 and later).
	    \end{itemize}
	\end{itemize}
\end{frame}
\begin{frame}[label=current]
	\mode<presentation>{\frametitle{Character storage}}
	\begin{itemize}
	    \item Practical note: there can be hassles switching between operating systems or locales within operating systems. E.g., Unix (LF) to Windows (CR+LF).
	\end{itemize}
\end{frame}

\subsection*{Practice questions}

\begin{frame}[fragile,label=current]
	\mode<presentation>{\frametitle{Practice questions}}
	\mode<presentation>{\footnotesize}
\bn
    \item Write out -15-3=-18 in binary using an 8-bit signed integer system
%  -15 10001111
%-  +3 00000011
%= -18 10010010
    \item What is the largest positive number that can be represented in this system?
%01111111 = 127
    \item Why is it very fast to multiply numbers represented as integers by 2?  What manipulation is required to the bits making up the number to achieve this operation?
%Just need to shift the bits to the left (apart from the sign bit and the most significant digit -- if the latter is already 1 then you get an overflow), and put a 0 in the least significant digit bit
    \item What is the machine epsilon on a 32-bit floating point system where 1 bit is for the sign, 16 bits for the exponent and 16 bits\footnote{using the same trick as in the IEEE standard to get one extra bit for free; note that this is not a very sensible allocation as it gives far too much of the space to the exponent} for the fraction?
% epsilon = 2^(1-d)=2^(-15)
    \item In this system, what would be the result of computing $2* 2^{-17} + 1$?
%2x2^-17 = 1x2^-16 -- below the machine epsilon so the result would be 1
    \item What do you get if you compute $512 * 0.25$ in the above floating point system, and then convert it into the above signed integer system?\footnote{A calculation like this once cost the EU space program about \$500 million: \href{https://www.ima.umn.edu/~arnold/disasters/ariane.html}{https://www.ima.umn.edu/$\sim$arnold/disasters/ariane.html}}
%512*0.25 is easily evaluated exactly in the floating point system, giving 128.  This is larger than the largest integer, however, so you'd get an overflow
 \en
\end{frame}

\end{document}