\documentclass[letterpaper,9pt,twoside,printwatermark=false]{pinp}

%% Some pieces required from the pandoc template
\providecommand{\tightlist}{%
  \setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}

% Use the lineno option to display guide line numbers if required.
% Note that the use of elements such as single-column equations
% may affect the guide line number alignment.

\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}

% The geometry package layout settings need to be set here...
\geometry{layoutsize={0.95588\paperwidth,0.98864\paperheight},%
          layouthoffset=0.02206\paperwidth,%
		  layoutvoffset=0.00568\paperheight}

\definecolor{pinpblue}{HTML}{185FAF}  % imagecolorpicker on blue for new R logo
\definecolor{pnasbluetext}{RGB}{101,0,0} %

\title{Assignment 5 - Inference for a Population Mean. Due October 14, 11:59pm
2018}

\author[a]{EPIB607 - Inferential Statistics}

  \affil[a]{Fall 2018, McGill University}

\setcounter{secnumdepth}{5}

% Please give the surname of the lead author for the running footer
\leadauthor{Bhatnagar and Hanley}

% Keywords are not mandatory, but authors are strongly encouraged to provide them. If provided, please include two to five keywords, separated by the pipe symbol, e.g:
 \keywords{  t-test |  One sample mean |  Bootstrap  }

\begin{abstract}
In this assignment you will practice conducting inference for a one
sample mean using either the z procedure, t procedure, or the bootstrap.
Answers should be given in full sentences (DO NOT just provide the
number). All figures should have appropriately labeled axes, titles and
captions (if necessary). All graphs and calculations are to be completed
in an R Markdown document using the provided template. You are free to
choose any function from any package to complete the assignment. Concise
answers will be rewarded. Be brief and to the point. Please submit both
the compiled HTML report and the source file (.Rmd) to myCourses by
October 14, 2018, 11:59pm. Both HTML and .Rmd files should be saved as
`IDnumber\_LastName\_FirstName\_EPIB607\_A5'.
\end{abstract}

\dates{This version was compiled on \today}
\doi{\url{https://sahirbhatnagar.com/EPIB607/}}

\pinpfootercontents{Assignment 5 due October 14, 2018 by 11:59pm}

\begin{document}

% Optional adjustment to line up main text (after abstract) of first page with line numbers, when using both lineno and twocolumn options.
% You should only change this length when you've finalised the article contents.
\verticaladjustment{-2pt}

\maketitle
\thispagestyle{firststyle}
\ifthenelse{\boolean{shortarticle}}{\ifthenelse{\boolean{singlecolumn}}{\abscontentformatted}{\abscontent}}{}

% If your first paragraph (i.e. with the \dropcap) contains a list environment (quote, quotation, theorem, definition, enumerate, itemize...), the line after the list may have some extra indentation. If this is the case, add \parshape=0 to the end of the list environment.

\section*{Template}\label{template}
\addcontentsline{toc}{section}{Template}

The \texttt{.Rmd} template for Assignment 5 is available
\href{https://github.com/sahirbhatnagar/EPIB607/raw/master/assignments/a5/a5_template.Rmd}{here}

\section{Food intake and weight gain}\label{food-intake-and-weight-gain}

If we increase our food intake, we generally gain weight. Nutrition
scientists can calculate the amount of weight gain that would be
associated with a given increase in calories. In one study, 16 nonobese
adults, aged 25 to 36 years, were fed 1000 calories per day in excess of
the calories needed to maintain a stable body weight. The subjects
maintained this diet for 8 weeks, so they consumed a total of 56,000
extra calories. According to theory, 3500 extra calories will translate
into a weight gain of 1 pound. Therfore we expect each of these subjects
to gain 56,000/3500=16 pounds (lb). Here are the weights (given in the
\texttt{weightgain.csv} file) before and after the 8-week period
expressed in kilograms (kg):

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{weight <-}\StringTok{ }\KeywordTok{read.csv}\NormalTok{(}\StringTok{"weightgain.csv"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{ShadedResult}
\begin{verbatim}
#     subject before after
#  1        1   55.7  61.7
#  2        2   54.9  58.8
#  3        3   59.6  66.0
#  4        4   62.3  66.2
#  5        5   74.2  79.0
#  6        6   75.6  82.3
#  7        7   70.7  74.3
#  8        8   53.3  59.3
#  9        9   73.3  79.1
#  10      10   63.4  66.0
#  11      11   68.1  73.4
#  12      12   73.7  76.9
#  13      13   91.7  93.1
#  14      14   55.9  63.0
#  15      15   61.7  68.2
#  16      16   57.8  60.3
\end{verbatim}
\end{ShadedResult}

\begin{enumerate}
\def\labelenumi{\alph{enumi}.}
\tightlist
\item
  Calculate a 95\% confidence interval for the mean weight change and
  give a sentence explaining the meaning of the 95\%. State your
  assumptions.
\item
  Calculate a 95\% bootstrap confidence interval for the mean weight
  change and compare it to the one obtained in part (a). Comment on the
  bootstrap sampling distribution and compare it to the assumptions you
  made in part (a).
\item
  Convert the units of the mean weight gain and 95\% confidence interval
  to pounds. Note that 1 kilogram is equal to 2.2 pounds.
\item
  Test the null hypothesis that the mean weight gain is 16 lbs. State
  your assumptions and justify your choice of test. Be sure to specify
  the null and alternative hypotheses. What do you conclude?
\end{enumerate}

\newpage

\section{Attitudes toward school}\label{attitudes-toward-school}

The Survey of Study Habits and Attitudes (SSHA) is a psychological test
that measures the motivation, attitude toward school, and study habits
of students. Scores range from 0 to 200. The mean score for U.S. college
students is about 115, and the standard deviation is about 30. A teacher
who suspects that older students have better attitudes toward school
gives the SSHA to 25 students who are at least 30 years of age. Their
mean score is \(\bar{y}\) = 132.2 with a sample standard deviation
\(s = 28\).

\begin{enumerate}
\def\labelenumi{\alph{enumi}.}
\tightlist
\item
  The teacher asks you to carry out a formal statistical test for her
  hypothesis. Perform a test, provide a 95\% confidence interval and
  state your conclusion clearly.
\item
  What assumptions did you use in part (a). Which of these assumptions
  is most important to the validity of your conclusion in part (a).
\end{enumerate}

\section{Does a full moon affect
behavior?}\label{does-a-full-moon-affect-behavior}

Many people believe that the moon influences the actions of some
individuals. A study of dementia patients in nursing homes recorded
various types of disruptive behaviors every day for 12 weeks. Days were
classified as moon days if they were in a 3-day period centered at the
day of the full moon. For each patient, the average number of disruptive
behaviors was computed for moon days and for all other days. The
hypothesis is that moon days will lead to more disruptive behavior. We
look at a data set consisting of observations on 15 dementia patients in
nursing homes (available in the \texttt{fullmoon.csv} file):

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{fullmoon <-}\StringTok{ }\KeywordTok{read.csv}\NormalTok{(}\StringTok{"fullmoon.csv"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

\begin{ShadedResult}
\begin{verbatim}
#     patient moon_days other_days
#  1        1      3.33       0.27
#  2        2      3.67       0.59
#  3        3      2.67       0.32
#  4        4      3.33       0.19
#  5        5      3.33       1.26
#  6        6      3.67       0.11
#  7        7      4.67       0.30
#  8        8      2.67       0.40
#  9        9      6.00       1.59
#  10      10      4.33       0.60
#  11      11      3.33       0.65
#  12      12      0.67       0.69
#  13      13      1.33       1.26
#  14      14      0.33       0.23
#  15      15      2.00       0.38
\end{verbatim}
\end{ShadedResult}

\begin{enumerate}
\def\labelenumi{\alph{enumi}.}
\tightlist
\item
  Calculate a 95\% confidence interval for the mean difference in
  disruptive behaviors. State the assumptions you used to calculate this
  interval.
\item
  Calculate a 95\% bootstrap confidence interval for the mean difference
  in disruptive behaviors and compare to the one obtained in part (a).
  Comment on the bootstrap sampling distribution and compare it to the
  assumptions you made in part (a).
\item
  Test the hypothesis that moon days will lead to more disruptive
  behavior. State your assumptions and provide a brief conclusion based
  on your analysis.
\item
  Find the minimum value of the mean difference in disruptive behaviors
  (\(\bar{y}\)) needed to reject the null hypothesis.
\item
  What is the probability of detecting an increase of 1.0 aggressive
  behavior per day during moon days? \emph{Hint: calculate the
  probability of the event calculated in part (d) using a normal
  distribution with \(\mu=1\) and \(\sigma =\) the standard error of the
  mean}
\end{enumerate}

\newpage

\section{How deep is the ocean?}\label{how-deep-is-the-ocean}

This question is based on the
\href{https://github.com/sahirbhatnagar/EPIB607/blob/master/exercises/water/students/260194225_water_exercise_epib607.pdf}{in-class
Exercise} on sampling distributions and builds on
\href{https://github.com/sahirbhatnagar/EPIB607/raw/master/assignments/a4/a4_clt_ci.pdf}{Question
4 from Assignment 4}. For your sample of \(n=20\) of depths of the ocean

\begin{enumerate}
\def\labelenumi{\alph{enumi}.}
\tightlist
\item
  Calculate a 95\% Confidence interval using the \(t\) procedure
\item
  Plot the qnorm, bootstrap, and \(t\) procedure confidence intervals on
  the same plot and comment on the how the \(t\) interval compares to
  the other 2 intervals. You may use the \texttt{compare\_CI} function
  provided below to produce the plot.
\end{enumerate}

\begin{Shaded}
\begin{Highlighting}[]
\NormalTok{compare_CI <-}\StringTok{ }\ControlFlowTok{function}\NormalTok{(ybar, QNORM, BOOT, TPROCEDURE,}
                       \DataTypeTok{col =} \KeywordTok{c}\NormalTok{(}\StringTok{"#E41A1C"}\NormalTok{,}\StringTok{"#377EB8"}\NormalTok{,}\StringTok{"#4DAF4A"}\NormalTok{)) \{}

\NormalTok{  dt <-}\StringTok{ }\KeywordTok{data.frame}\NormalTok{(}\DataTypeTok{type =} \KeywordTok{c}\NormalTok{(}\StringTok{"qnorm"}\NormalTok{, }\StringTok{"bootstrap"}\NormalTok{, }\StringTok{"t"}\NormalTok{),}
                   \DataTypeTok{ybar =} \KeywordTok{rep}\NormalTok{(ybar, }\DecValTok{3}\NormalTok{),}
                   \DataTypeTok{low =} \KeywordTok{c}\NormalTok{(QNORM[}\DecValTok{1}\NormalTok{], BOOT[}\DecValTok{1}\NormalTok{], TPROCEDURE[}\DecValTok{1}\NormalTok{]),}
                   \DataTypeTok{up =} \KeywordTok{c}\NormalTok{(QNORM[}\DecValTok{2}\NormalTok{], BOOT[}\DecValTok{2}\NormalTok{], TPROCEDURE[}\DecValTok{2}\NormalTok{])}
\NormalTok{  )}

  \KeywordTok{plot}\NormalTok{(dt}\OperatorTok{$}\NormalTok{ybar, }\DecValTok{1}\OperatorTok{:}\KeywordTok{nrow}\NormalTok{(dt), }\DataTypeTok{pch =} \DecValTok{20}\NormalTok{, }\DataTypeTok{ylim =} \KeywordTok{c}\NormalTok{(}\DecValTok{0}\NormalTok{, }\DecValTok{5}\NormalTok{), }
       \DataTypeTok{xlim =} \KeywordTok{range}\NormalTok{(}\KeywordTok{pretty}\NormalTok{(}\KeywordTok{c}\NormalTok{(dt}\OperatorTok{$}\NormalTok{low, dt}\OperatorTok{$}\NormalTok{up))),}
       \DataTypeTok{xlab =} \StringTok{"Depth of ocean (m)"}\NormalTok{, }\DataTypeTok{ylab =} \StringTok{"Confidence Interval Type"}\NormalTok{,}
       \DataTypeTok{las =} \DecValTok{1}\NormalTok{, }\DataTypeTok{cex.axis =} \FloatTok{0.8}\NormalTok{, }\DataTypeTok{cex =} \DecValTok{3}\NormalTok{)}

  \KeywordTok{abline}\NormalTok{(}\DataTypeTok{v =} \DecValTok{37}\NormalTok{, }\DataTypeTok{lty =} \DecValTok{2}\NormalTok{, }\DataTypeTok{col =} \StringTok{"black"}\NormalTok{, }\DataTypeTok{lwd =} \DecValTok{2}\NormalTok{)}
  \KeywordTok{segments}\NormalTok{(}\DataTypeTok{x0 =}\NormalTok{ dt}\OperatorTok{$}\NormalTok{low, }\DataTypeTok{x1 =}\NormalTok{ dt}\OperatorTok{$}\NormalTok{up,}
           \DataTypeTok{y0 =} \DecValTok{1}\OperatorTok{:}\KeywordTok{nrow}\NormalTok{(dt), }\DataTypeTok{lend =} \DecValTok{1}\NormalTok{,}
           \DataTypeTok{col =}\NormalTok{ col, }\DataTypeTok{lwd =} \DecValTok{4}\NormalTok{)}

  \KeywordTok{legend}\NormalTok{(}\StringTok{"topleft"}\NormalTok{,}
         \DataTypeTok{legend =} \KeywordTok{c}\NormalTok{(}\KeywordTok{eval}\NormalTok{(}\KeywordTok{substitute}\NormalTok{( }\KeywordTok{expression}\NormalTok{(}\KeywordTok{paste}\NormalTok{(mu,}\StringTok{" = "}\NormalTok{,}\DecValTok{37}\NormalTok{)))),}
                    \KeywordTok{sprintf}\NormalTok{(}\StringTok{"qnorm CI: [%.f, %.f]"}\NormalTok{,QNORM[}\DecValTok{1}\NormalTok{], QNORM[}\DecValTok{2}\NormalTok{]),}
                    \KeywordTok{sprintf}\NormalTok{(}\StringTok{"bootstrap CI: [%.f, %.f]"}\NormalTok{,BOOT[}\DecValTok{1}\NormalTok{], BOOT[}\DecValTok{2}\NormalTok{]),}
                    \KeywordTok{sprintf}\NormalTok{(}\StringTok{"t CI: [%.f, %.f]"}\NormalTok{,TPROCEDURE[}\DecValTok{1}\NormalTok{], TPROCEDURE[}\DecValTok{2}\NormalTok{])),}
         \DataTypeTok{lty =} \KeywordTok{c}\NormalTok{(}\DecValTok{1}\NormalTok{,}\DecValTok{1}\NormalTok{,}\DecValTok{1}\NormalTok{,}\DecValTok{1}\NormalTok{),}
         \DataTypeTok{col =} \KeywordTok{c}\NormalTok{(}\StringTok{"black"}\NormalTok{,col), }\DataTypeTok{lwd =} \DecValTok{4}\NormalTok{)}
\NormalTok{\}}

\CommentTok{# example of how to use the function:}
\KeywordTok{compare_CI}\NormalTok{(}\DataTypeTok{ybar =} \DecValTok{36}\NormalTok{, }\DataTypeTok{QNORM =} \KeywordTok{c}\NormalTok{(}\DecValTok{25}\NormalTok{,}\DecValTok{40}\NormalTok{), }\DataTypeTok{BOOT =} \KeywordTok{c}\NormalTok{(}\DecValTok{31}\NormalTok{, }\DecValTok{38}\NormalTok{), }\DataTypeTok{TPROCEDURE =} \KeywordTok{c}\NormalTok{(}\DecValTok{28}\NormalTok{, }\DecValTok{40}\NormalTok{))}
\end{Highlighting}
\end{Shaded}

%\showmatmethods

\bibliography{pinp}
\bibliographystyle{jss}

\end{document}


\documentclass[../main]{subfiles}
\begin{document}

\chapter{Private key authentication schemes}

\section{Difference between secrecy and authentication}

\begin{figure}[h]
    \centering
    \includegraphics[width=0.5\textwidth]{images/difference_secrecy_authentication}
    \caption{Difference between secrecy and authentication.}
\end{figure}
\noindent
In \textbf{secrecy schemes}, the channel is secure and the focus of the adversary is reading what goes through the channel.
In \textbf{authentication schemes}, the channel is not secure and the focus of the adversary is not to read the content of the channel, but to write on it.
That is why the receiver wants to be sure that the message sent along the channel is actually coming from the expected sender.
For that we need a \textbf{Message Authentication Code (MAC)}.\\
\noindent
Authentication and secrecy are different: solving the problem of secrecy does not solve the problem of authentication.
We do not want adversaries to be able to \textit{forge} messages of their choice $m$ without knowing a key $k$.

\section{Message Authentication Codes}
The same way ciphers guarantee secrecy, \textbf{Message Authentication Codes (MACs)} guarantee authentication.
\begin{definition}
    A MAC is a triple $\Pi = (Gen, Mac, Vrfy)$ of PPT algorithms such that:
    \begin{itemize}
        \item $Gen$ takes as input a string in the form $1^n$ (the security parameter) and outputs a key $k$ which can be such that $|k|\ge n$.
        \item $Mac$ takes as input a key $k$ and a message $m$ and outputs a \textbf{tag} $t$.
        \item $Vrfy$ takes as input a key $k$, a message $m$ and a tag $t$, and outputs a boolean $b$ (i.e., $b \in{} \{0,1\}$).
    \end{itemize}
    A MAC $\Pi = (Gen, Mac, Vrfy)$ is \textit{correct} when $Vrfy(k, m, Mac(k, m)) = 1$.
\end{definition}

\noindent
In order to be safe and sound, the assumptions are very pessimistic:
\begin{itemize}
    \item the adversary $A$ has access to an oracle $Mac_k(\cdot)$ which can be used to generate all possible tags;
    \item having access to a previously generated pair ($m$,$t$) is not interesting;
    \item $A$ has to be PPT;
    \item the new experiment is called $MacForge_{A,\Pi}(n)$.
\end{itemize}

\noindent
Let us present the new experiment:\\
$MacForge_{A,\Pi}(n):$\\
$k \leftarrow{} Gen(1^n)$\\
$(m,t) \leftarrow{} A(1^n,Mac_k(\cdot)) \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \text{// Invoked once.}$\\
$\mathbb{Q} \leftarrow \{m \; | \; A \text{ queries } Mac_k(\cdot) \text{ on } m\} \;\;\; \text{// Set of queries already performed by MacForge.}$\\
$\textbf{Result:} (m \notin{\mathbb{Q}} \wedge{} Vrfy(k,m,t) = 1) \;\; \text{// $m$ must be new and $Vrfy$ must succeed.}$

\begin{definition}
    A MAC $\Pi$ is secure if and only if for every PPT adversary $A$ there exists a function $\varepsilon{} \in{} \mathcal{NGL}$ such that:
    $$Pr(MacForge_{\Pi,A}(n) = 1) = \varepsilon(n)$$
\end{definition}
%----------------------------------------------------------------------------------
\subsection{Constructing a secure MAC}
\begin{definition}[FP induced MAC]
    Given an FP $F$, the MAC $\Pi{}^F = (Gen, Mac, Vrfy)$ is defined as follows:
    \begin{itemize}
        \item The algorithm $Gen$ for input $1^n$ outputs every string long $n$ with same probability, i.e. $\frac{1}{2^n}$.
        \item $Mac(k,m) = F_k(m)$.
        \item $Vrfy(k,m,t) = (F_k(m)\stackrel{?}{=}t)$ (it is $1$ (true) if and only if $F_k(m)$ equals $t$).
    \end{itemize}
\end{definition}

\begin{theorem}
    If $F$ is pseudorandom, then the MAC $\Pi{}^F$ is secure.
\end{theorem}

\paragraph{Proof}
Let's start by introducing an auxiliary MAC $\Tilde{\Pi}$ as follows:
\begin{itemize}
    \item $\Tilde{Gen}$ on input $1^n$ does not generate a key but a random function $f: \{0,1\}^n \rightarrow{} \{0,1\}^n$;
    \item $\Tilde{Mac}$ and $\Tilde{Vrfy}$ works pretty much like the corresponding algorithms in $\Pi{}^F$
\end{itemize}
First of all, observe that $\Tilde{\Pi}$ is secure: since $f$ is a function and the adversary is required to output a valid pair $(m,t)$ ($m \notin{} \mathbb{Q}$),
we are basically asking the adversary to guess a random string, namely $f(m)$. This probability is $\frac{1}{2^n}$.
As a consequence:
$$Pr(MacForge_{\Tilde{\Pi},A}(n) = 1) = \frac{1}{2^n}$$
Now, let's prove that the behaviour of any adversary $A$ against $\Tilde{\Pi}$ and $\Pi^F$ cannot be too different, i.e.,
$$|Pr(MacForge_{\Tilde{\Pi},A}(n) = 1) - Pr(MacForge_{\Pi^F,A}(n) = 1)| \le{} \varepsilon(n)$$
with $\varepsilon \in{} \mathcal{NGL}$.\\
We can construct a distinguisher $D_A$ by exploiting the adversary $A$ as follows:
\begin{figure}[H]
    \centering
    \includegraphics[width=0.8\textwidth]{images/security_of_mac_given_pseudorandom_F}
    \caption{How a distinguisher $D_A$ can be generated out of an attacker $A$.}
\end{figure}

\noindent
By this construction of $D_A$ we notice the following:
$$Pr(MacForge_{\Tilde{\Pi},A}(n) = 1) = Pr(D_A^{f(\cdot)}(1^n) = 1)$$
$$Pr(MacForge_{\Pi^F,A}(n) = 1) = Pr(D_A^{F_k(\cdot)}(1^n) = 1)$$
Since $F$ is pseudorandom, then we can prove what we wanted, that is:
$$|Pr(D_A^{f(\cdot)}(1^n) = 1) - Pr(D_A^{F_k(\cdot)}(1^n) = 1)| \le{} \varepsilon(n)$$
$$\Updownarrow$$
$$|Pr(MacForge_{\Tilde{\Pi},A}(n) = 1) - Pr(MacForge_{\Pi^F,A}(n) = 1)| \le{} \varepsilon(n)$$
%----------------------------------------------------------------------------------
\subsubsection{Handling Variable-Length messages}
$Mac^{*}(k, m)$:\\
$m_1 \mid\mid{} \ldots{} \mid\mid{} m_d \; \leftarrow{} \; m$\\
$\ell{} \; \leftarrow{} \; \mid{} m \mid{}$\\
\textbf{// such that $\mid{} m_i \mid{} = \frac{\ell{}}{4}$}\\
$r \; \leftarrow{} \; \{0, 1\}^{\frac{n}{4}}$\\
$\textbf{for} \; i \; \leftarrow{} \; 1 \; \textbf{to} \; d \; \textbf{do}$\\
$\quad{}\quad{} t_i \; \leftarrow{} Mac(k, \; r\mid\mid{}\ell\mid\mid{}i\mid\mid{}m_1)$\\
\textbf{Result}: $(r, t_1, \ldots{},t_d)$
\newline
\newline
$Vrfy^{*}(k, m, (r, t_1, \ldots{}, t_d))$:\\
$m_1 \mid\mid{} \ldots{} \mid\mid{} m_d \; \leftarrow{} \; m$\\
$\ell{} \; \leftarrow{} \; \mid{} m \mid{}$\\
\textbf{// such that $\mid{} m_i \mid{} = \frac{\ell{}}{4}$}\\
$\textbf{for} \; i \; \leftarrow{} \; 1 \; \textbf{to} \; d \; \textbf{do}$\\
$\quad{}\quad{} \textbf{if} \; Vrfy(k, r\mid\mid{}\ell\mid\mid{}i\mid\mid{}m_1) = 0 \; \textbf{then}$\\
$\quad{}\quad{}\quad{} \textbf{Result}: 0$\\
\textbf{Result}: 1

\begin{theorem}
    If $\Pi{}$ is secure, then $\Pi{}^*$ is also secure.
\end{theorem}
%----------------------------------------------------------------------------------
\subsubsection{CBC-MAC construction}
$Mac^{CBC}(k, m)$:\\
$\ell{} \; \leftarrow{} \; \ell{}(\mid{} k \mid)$\\
$m_1 \mid\mid{} \ldots{} \mid\mid{} m_{\ell} \; \leftarrow{} \; m$\\
\textbf{//such that $\mid{} m_i \mid{} = \; \mid{}k\mid{}$}\\
$t_0 \; \leftarrow{} \; 0^n$
$\ell{} \; \leftarrow{} \; \mid{} m \mid{}$\\
$\textbf{for} \; i \; \leftarrow{} \; 1 \; \textbf{to} \; \ell{} \; \textbf{do}$\\
$\quad{}\quad{} t_i \; \leftarrow{} \; F_k(t_{i-1} \oplus{} m_i)$\\
\textbf{Result}: $t_{\ell}$
\newline
\newline
$Vrfy^{CBC}(k, m, t)$:\\
\textbf{if} $\ell{}(\mid k \mid) \cdot{} \mid{} k \mid{} \neq{} \mid{} m \mid{}$ \textbf{then}\\
$\quad{}\quad{} \textbf{Result}: 0$\\
$\textbf{Result}: t \stackrel{?}{=} MAC^{CBC}(k, m)$\\

\begin{theorem}
    If $\ell{}$ is a polynomial and $F$ is $FP$, then $\Pi{}^{CBC}$ is a secure MAC.
\end{theorem}
%----------------------------------------------------------------------------------
\section{Hash functions}
\begin{definition}
    A hash function is a pair of PPT algorithms $(Gen, H)$ such that:
    \begin{itemize}
        \item Gen is an algorithm that takes as input a security parameter $1^n$ and return a key $s$ (from which $n$ can be efficiently calculated).
        \item There exists a polynomial $\ell{}$ such that $H(s, x)$ returns a string of length $\ell{}(n)$ (where $n$ is implicit parameter in $s$)
    \end{itemize}
\end{definition}
%----------------------------------------------------------------------------------
\subsection{Collision-Resistant hash functions}
$HashCol_{\mathcal{A}, \Pi{}}(n):$\\
$s \; \leftarrow{} \; Gen(1^n)$\\
$(x, y) \; \leftarrow{} \; A(s)$\\
\textbf{Result}: $(x \neq{} y) \wedge{} (H(x) = H(y))$

\begin{definition}
    A hash function $\Pi{} = (Gen, H)$ is collision-resistant if and only if for every adversary PPT $A$ there exists a negligible function $\varepsilon{}$ such that $$Pr(HashColl_{\mathcal{A}, \Pi{}}(n)=1) \leq{} \varepsilon{}(n)$$
\end{definition}
%----------------------------------------------------------------------------------
\subsubsection{Birthday attacks}
\begin{theorem}[Birthday theorem]
    Given $q$ uniformly chosen random values in a finite set of cardinality $N$, the probability that two of them are identical is $\frac{q^2}{N}$.
\end{theorem}
\paragraph{Proof}
If we take $q=\Theta(2^{\frac{\ell}{2}})$, then we get $\Theta(\frac{(2^{\frac{\ell}{2}})^2}{2^\ell}) = \Theta(\frac{2^\ell}{2^\ell}) = \Theta(1)$.
That is, we get a constant probability of success, independent of $q$ and $\ell$.
%----------------------------------------------------------------------------------
\subsubsection{The Merkle-Damgård transform}
$H^{MD}(s, x):$\\
$B \; \leftarrow{} \; \lceil{} \frac{\mid{}x\mid{}}{n} \rceil{}$\\
$x_1 \mid\mid{} x_B \; \leftarrow{} \; x$\\
\textbf{//such that $\mid{} x_i \mid{} = n$}\\
$x_{B+1} \; \leftarrow{} \; \mid{} x \mid{}$\\
$z_0 \; \leftarrow{} \; 0^n$\\
$\textbf{for} \; i \; \leftarrow{} \; 1 \; \textbf{to} \; B+1 \; \textbf{do}$\\
$\quad{}\quad{} z_i \; \leftarrow{} \; H(s, z_i - 1, \mid\mid{} x_i)$
\textbf{Result}: $z_{B+1}$

\begin{theorem}
    If $(Gen, H)$ is collision-resistant, then so is $(Gen, H^{MD})$.
\end{theorem}
%----------------------------------------------------------------------------------
\subsubsection{Hash-and-Mac}
\begin{theorem}
    If $\Pi{}$ is a secure MAC and $(Gen', H)$ is collision-resistant, then $\Pi{}^H$ is secure.
\end{theorem}

\paragraph{Proof}
Let's describe what $\Pi^H$ is: $\Pi^H = (Gen^H, Mac^H, Vrfy^H)$, with $Gen^H$ returning a pair of keys according to $H$ (i.e., $Gen'$) and according to $\Pi$ (i.e., $Gen$); $Mac^H$ returns $Mac_k(H^s(m))$.\\
Let's take $Coll$ (which stands for \textit{collision}) as a probabilistic event that inclues all the executions in which there is $m \in{} \mathbb{Q}$ such that $H^s(m) = H^s(m^*)$ where $m^*$ is the message the adversary produces at the end of its execution.
\begin{center}
    $Pr(MacForge_{A,\Pi^H}(n)=1) =$\\
    $Pr(MacForge_{A,\Pi^H}(n)=1 \wedge{} Coll)\; +$
    $Pr(MacForge_{A,\Pi^H}(n)=1 \wedge{} \neg{}Coll) \leq{}$
    $Pr(Coll) + Pr(MacForge_{A,\Pi^H}(n)=1 \wedge{} \neg{}Coll)$
\end{center}
We want to prove that both $Pr(Coll)$ and $Pr(MacForge_{A,\Pi^H}(n)=1 \wedge{} \neg{}Coll)$ are negligible:
\begin{enumerate}
    \item proving $Pr(Coll)$ requires a reduction building an adversary for $H$ out of $A$,
    \item while proving $Pr(MacForge_{A,\Pi^H}(n)=1 \wedge{} \neg{}Coll)$ requires a reduction towards $\Pi$.
\end{enumerate}
Let's prove the first reduction:
\begin{itemize}
    \item we want to build, out of $A$, an adversary $C$ for $H$ such that $Pr(HashColl_{C,H}(n)=1)=Pr(Coll)$
    \item TO COMPLETE...
\end{itemize}
\end{document}


\section{Guard}

Guards make sure the wrong people don't get into the wrong areas or take the wrong things.\\
\\
Strength: 8\\
Vitality: 7\\
Dexterity: 4\\
Agility: 6\\
Intellect: 5\\
Perception: 7\\
Empathy: 5\\
\\
Combat: 3\\
Adventuring: 2\\
Social: 2\\
\\
HP: 3 x 1d6 + 21 = 32\\
Stamina: 3 x 1d6 + 24 = 35\\
Mana: 1 x 1d6 + 5 = 9\\
\\
Actions:\\
Arming Sword (2 AP, 1ps): +11 + 1d4, 1d6 + 4 piercing\\
Arming Sword (3 AP, 1ps): +11 + 1d4, 1d12 + 4 cutting\\
Brutal Attack II\\
Shield (2 AP): 1d6 Blunt\\
Shield (Block): 2d4 Cutting, Blunt or Piercing\\
\\
RV: 18\\
Dodge: 11\\
Armor (Leather Gambeson)\\
Blunt: 5\\
Cutting: 4\\
Piercing: 2\\
\\
Heat: 1\\
Cold: 1\\


\documentclass{amsart}
%\documentclass[a4paper,10pt]{scrartcl}

\usepackage[utf8x]{inputenc}
\usepackage[british]{babel}
%\usepackage[a4paper, inner=0.5cm, outer=0.5cm, top=1cm,
%bottom=1.5cm, bindingoffset=1cm]{geometry}
\usepackage{amsmath}
\usepackage{amssymb, latexsym}
\usepackage{longtable}
\usepackage[table]{xcolor}
\usepackage{textcomp}
\usepackage{stmaryrd}
\usepackage{graphicx}
\usepackage{enumitem}
\usepackage{yfonts}
\usepackage{algpseudocode}
\usepackage{algorithm}
\usepackage{hyperref}
\usepackage{MnSymbol}

\setlist[enumerate]{label*=\arabic*.}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{example}{Example}[section]
\newtheorem{definition}{Definition}[section]
\newtheorem{proposition}{Proposition}[section]
\newtheorem{notation}{Notation}[section]

\renewcommand{\algorithmicrequire}{\textbf{Input:}}
\renewcommand{\algorithmicensure}{\textbf{Output:}}

\title{Risk Based Testing}
\author{Henriette Harmse}
\date{\today}

\pdfinfo{%
  /Title    (Risk Based Testing)
  /Author   (Henriette Harmse)
  /Creator  ()
  /Producer ()
  /Subject  (testing)
  /Keywords ()
}

\begin{document}
  \maketitle

  A question that is asked regularly in testing circles is: ``When should you stop testing?''. Proponents of code coverage tools may suggest once some percentage of coverage is achieved you can stop testing. However, what do you do when the budget is severely constrained? An even more difficult situation to address is when a project starts out with a given budget, but during its lifetime the budget gets significantly reduced (i.e. due to economic downturn). This forces us to be able to provide the highest level of quality for the least amount of money. In this post I will explain how \textbf{risk based testing} can help to answer this question.

\section{Motivation}
A mistake that is often made is that the testing effort is distributed equally across the system – both
critical and non-critical parts of the system are tested equally. This results in critical parts of the
system not being tested sufficiently and non-critical parts being tested to the point of diminishing
returns.

A further mistaken mindset of developers is that there is no such thing as a useless test. This may entice developers to add tests for the sake of adding tests. In actual fact every test (unit-, integration- or systems test) has to earn its place in the codebase. If there is no good motivation for a test, the test must be deleted from the codebase. Why is that? Because every test  adds to the volume of code that developers have to master and maintain to be productive members of the team. Tests add to the volume of code that has to be maintained. As such tests adds to the overall cost of maintenance of a system. \textbf{The most cost effective code to maintain is the code that has never been written}.

Risk based testing is a testing approach that helps developers and testers to prioritize the testing effort by assigning a relative risk to each software component.

\section{Calculating the Relative Risk of a Software Component}
The relative risk of a software component is based on the product of the relative risks assigned to the criteria deemed relevant for the project. Criteria for determining risk will be discussed in the next section. For each criteria a value is assigned between $1..X$, where $1$ indicates that the relative risk of a criterion is insignificant and $X$ indicates that the relative risk of the criterion is critical for the given software component.

Assuming we have software components ($S$) $1..m$, criteria ($C$) $1..n$ where we assign relative risks, $A_{i,1} .. A_{i,n}$ for each criterion of a software component $S_i$, the relative risk $R_i$ for each software component is given by:

\[R_1 =  A_{1,1} \times ... \times A_{1,n} \]
\[...\]
\[R_m =  A_{m,1} \times ... \times A_{m,n} \]

Ordering $R_1 ... R_m$ in descending order of their values will cause the software components with the highest relative risk to be at the top of the list. Note that the calculated relative risk for a specific software component by itself is meaningless. A calculated risk of an component is only meaningful if it can be compared to the calculated risk of another component calculated in the same way (that is, using the same criteria and value for $X$). That is why we refer to the \textbf{relative} risk of a software component.

What value should you choose for $X$? In general you should choose $X$ to be equal or greater than the number of criteria you will be using in calculating the risk. If many of the calculated relative risks $R_i$ have the same value, it means you have to increase the value of $X$.

\section{What Criteria should be Used?}
In this section we give examples of criteria that can be used in the calculating the risk of each software component. However, these criteria merely serve as examples. You are free to choose whatever criteria are deemed of importance for your project.

Some general criteria that you may want to consider using are:
\begin{description}
 \item[Frequency of use] How frequently is this software component used? The more often a software component is used, the higher is its potential for having an adverse effect when it fails.
 \item[Cost] What will be the cost of this software component failing? The cost can be monatary, but it does not have to be. It can be cost in terms of reputation, loss of clients, or whatever factor that can hurt the business. Here it may be a good idea to get the input of the project sponsor or project owner.
 \item[Complexity] What is the complexity of this software component? If this software component forms part of a green fields project, this criteria can express the complexity of this software component in comparison with other newly developed components. In the case where enhancements are made to an existing system, code analysis tools can be used to determine complexity of software components that need to be changed. For example, when an enhancement requires that 2 classes need to be changed substantially,  the class with a higher cyclomatic complexity (the number of linear independent paths through the code) represents a higher risk.
 \item[Frequency of change] How often will the software component be changed? If a software component realizes the core value proposition of a business, it is likely that this component will constantly need to be enhanced as the business is trying to stay competitive. If there is little chance that a software component is going to be changed after it has been completed, the tests for that software component may be of limited value in the long term.
 \item[Number of bugs] For a system that is already in production, the number of bugs that are logged (that can be related back to specific software components) can help give an indication as to where testing effort should be focused.
\end{description}

\section{An Example}
As an example, let us assume we are working on a legacy online share trading system, naturally with zero automated tests. As with most legacy systems, a large number of users depend on the system. A steady stream of user requests for enhancements and bugs logged require the system to be frequently updated. These changes often
introduce new bugs.

We further assume the system consists of the following modules:

\begin{itemize}
	\item A user management, authentication and authorization module through which users and their permissions are managed. It is very seldom that any changes are required to this module.
	\item A share price notification engine of which the main purpose is to notify users in realtime of changes in share prices. The correct functioning of this engine is critical to the business since it enables users to buy/sell shares as and when needed. Any failure of the engine could mean that users cannot buy/sell shares as needed which could result in massive losses. However, since the initial problems with the engine have been resolved, bugs are seldomly logged for this module.
	\item The account management system keeps track of the funds of users as they trade, calculates daily interest earned or charged (a positive balance earns interest and for a negative balance interest is charged) and generate monthly statements of transactions.
	\item The mark-to-market process runs each day after the stock exchange has closed to calculate the gains/losses for each account. There are frequent changes to this process as quants are forever fine tuning the way commission is calculated on trades. Most of the bugs logged are related to this module.
\end{itemize}

To improve the situation, management wants the development team to add automated tests. How should we approach testing the system?

  \rowcolors{1}{white}{lightgray}
\begin{table}
	\footnotesize
	%     \scriptsize
	\begin{center}
		\caption{Relative Calculated Risk for the Online Share Trading System}
		\label{tab_Example1}
		\smallskip
		\begin{tabular}{|>{\footnotesize}p{3cm}|>{\footnotesize}p{0.75cm}|>{\footnotesize}p{0.75cm}|>{\footnotesize}p{1.75cm}|>{\footnotesize}p{1.25cm}|>{\footnotesize}p{0.75cm}|>{\footnotesize}p{0.75cm}|}
			\hline
			\textbf{Module} & \textbf{Usage} & \textbf{Cost} & \textbf{Complexity} & \textbf{Change} & \textbf{Bugs} & \textbf{Risk} \\
			\hline
			User management & 1 & 2 & 1 & 1 & 1 & 2\\
			Share price notification & 5 & 5 & 5 & 1 & 1 & 125 \\
			Account management & 4 & 5 & 3 & 3 & 2 & 360\\
			Mark-to-market & 3 & 5 & 5 & 5 & 5 & 1875\\
			\hline
		\end{tabular}
	\end{center}
	\normalsize
\end{table}

If we apply risk based testing as in seen in Table~\ref{tab_Example1}, it is clear that we need to focus our testing efforts on the mark-to-market process. Let us further assume the mark-to-market process consists of the following steps for each account:
\begin{itemize}
	\item determine prices at which shares have been bought,
	\item determine current prices of shares held,
	\item calculate the profit/loss, and
	\item update the account balance.
\end{itemize}

  \rowcolors{1}{white}{lightgray}
\begin{table}
	\footnotesize
	%     \scriptsize
	\begin{center}
		\caption{Relative Calculated Risk for the Mark-to-Market Process}
		\label{tab_Example2}
		\smallskip
		\begin{tabular}{|>{\footnotesize}p{3.5cm}|>{\footnotesize}p{0.75cm}|>{\footnotesize}p{0.75cm}|>{\footnotesize}p{1.75cm}|>{\footnotesize}p{1.25cm}|>{\footnotesize}p{0.75cm}|>{\footnotesize}p{0.75cm}|}
			\hline
			\textbf{Methods/Functions} & \textbf{Usage} & \textbf{Cost} & \textbf{Complexity} & \textbf{Change} & \textbf{Bugs} & \textbf{Risk} \\
			\hline
			determineBuyPrices & 2 & 5 & 2 & 1 & 3 & 60\\
			determineCurrentPrices & 4 & 1 & 5 & 1 & 1 & 20 \\
			calculateProfitLoss & 2 & 5 & 5 & 5 & 4 & 1000\\
			postAccountUpdate & 2 & 5 & 1 & 1 & 1 & 10\\
			\hline
		\end{tabular}
	\end{center}
	\normalsize
\end{table}

A detailed plan of how to approach testing of the mark-to-market process can be drawn up (see Table~\ref{tab_Example2}), but instead of listing modules, it lists the software components that the mark-to-market process consists of. These can for example be classes, methods, functions, scripts etc. Table~\ref{tab_Example2} indicates that we need to particularly focus our attention on the \texttt{calculateProfitLoss} software component.

\section{Advantages/Disadvantages of Risk Based Testing}
The advantages of risk based testing are:
\begin{itemize}
 \item The highest risk items can be developed and tested first which reduces the overall risk on
the project.
 \item If testing has to be watered down, a guideline exists for deciding what to test and what not
to test.
 \item At any given time an indication can be given of the risk of the project by considering the
highest risk software components that has not been tested.
\item It is a valuable tool for communicating risk to management, project sponsors and project owners.
\end{itemize}

The potential disadvantages of risk based testing are:
\begin{itemize}
	\item With a risk based testing approach many parts of the system will go untested. However, this will be an intentional decision rather than an accidental one.
	\item The relative risk value, that are assigned for each criterion of a software component, is likely to be subjective. That means that another person may assign a different relative risk value for a criterion of a software component. However, it is just as subjective as, for example, story points in agile methodology. It similarly makes sense to assign relative risk values as part of a team discussion.
	\item Some teams feel risk based testing adds to their documentation load. That is, they now have to draw up a complete risk profile before they can start testing. Do I always draw up risk profiles for all my projects? Honestly? No. What I will do is to have a discussion with the team regarding what areas of the system we need to test to death and which areas we can skimp on. If there is some disagreement, then I may write it out. Writing it out is very useful when the testing approach needs to be discussed with management, the project sponsor or the project owner.
\end{itemize}

\section{Conclusion}
The main aim of risk base testing is to ensure that the testing effort is focused where it will bring the most business value.

\end{document}

blog/series/testing/Risk Based Testing/Risk Based Testing.tex


% Created 2017-10-13 Fri 09:04
% Intended LaTeX compiler: pdflatex
\documentclass[presentation]{beamer}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{graphicx}
\usepackage{grffile}
\usepackage{longtable}
\usepackage{wrapfig}
\usepackage{rotating}
\usepackage[normalem]{ulem}
\usepackage{amsmath}
\usepackage{textcomp}
\usepackage{amssymb}
\usepackage{capt-of}
\usepackage{hyperref}
\usepackage{minted}
\usepackage{helvet}
\usepackage{xcolor}
\definecolor{lgray}{rgb}{0.90,0.90,0.90}
\usetheme{Boadilla}
\usecolortheme{seahorse}
\author{Dale J. Barr}
\date{University of Glasgow}
\title{Research Cycle 04: dplyr one-table verbs}
\hypersetup{
 pdfauthor={Dale J. Barr},
 pdftitle={Research Cycle 04: dplyr one-table verbs},
 pdfkeywords={},
 pdfsubject={},
 pdfcreator={Emacs 24.5.1 (Org mode 8.3.4)},
 pdflang={English}}
\begin{document}

\maketitle

\section*{Main}
\label{sec:orgb59e705}

\begin{frame}[label={sec:org9f5ac3d}]{Tidy data}
\framesubtitle{Wickham (2014)}
\begin{Large}

\begin{quote}
``Happy families are all alike; every unhappy family is unhappy in its own way.''
\end{quote}

\end{Large}
\begin{flushright}-Tolstoy\end{flushright}

\begin{itemize}
\item Tidy datasets conform to a standardized way of linking
\alert{data~structure} to \alert{data semantics} (meaning)
\end{itemize}
\end{frame}

\begin{frame}[label={sec:org92ee0de}]{Tidy data}
\framesubtitle{(see also Codd, 1990; ``3rd normal form'')}

A dataset is a collection of \structure{values} observed on
\structure{variables} across different \structure{observation units}.

\begin{definition}[Tidy Data]
\begin{enumerate}
\item Each variable forms a column.
\item Each observation forms a row.
\item Each type of observational unit forms a table.
\end{enumerate}
\end{definition}

\begin{scriptsize}

\begin{center}
\begin{tabular}{rrlrl}
\hline
SubjectID & ItemID & Cond & RT & Choice\\
\hline
1 & 1 & E & 637 & A\\
1 & 2 & C & 998 & B\\
1 & 3 & E & 773 & B\\
1 & 4 & C & 890 & B\\
2 & 1 & C & 590 & A\\
2 & 2 & E & 911 & B\\
2 & 3 & C & 708 & B\\
2 & 4 & E & 621 & A\\
\hline
\end{tabular}
\end{center}

\end{scriptsize}
\end{frame}

\begin{frame}[label={sec:org4790edd}]{One of infinitely many messy versions}
\begin{tiny}
\begin{center}
\begin{tabular}{rllllrrrrllll}
\hline
SubjectID & Cond1 & Cond2 & Cond3 & Cond4 & RT1 & RT2 & RT3 & RT4 & Ch1 & Ch2 & Ch3 & Ch4\\
\hline
1 & E & C & E & C & 637 & 998 & 773 & 890 & A & B & B & B\\
2 & C & E & C & E & 590 & 911 & 708 & 621 & A & B & B & A\\
\hline
\end{tabular}
\end{center}
\end{tiny}

\begin{itemize}
\item wide format
\item one column for each item for each variable, no easy mapping from
structure to semantics
\item column names different for same variable (e.g., RT1..RT4)
\item different strategies for different obs units (e.g., calc subject means at each level of Cond)
\begin{itemize}
\item must be done by hand, and thus, \alert{error prone}
\end{itemize}
\end{itemize}
\end{frame}

\begin{frame}[label={sec:orga617e56}]{Tidy-ish representation of multilevel data}
\begin{tiny}
\begin{center}
\begin{tabular}{rllrlrlrl}
\hline
SubjectID & ListID & Gender & ItemID & Freq & TrialID & Cond & RT & Choice\\
\hline
1 & X & F & 1 & L & 1 & E & 637 & A\\
1 & X & F & 2 & H & 2 & C & 998 & B\\
1 & X & F & 3 & L & 3 & E & 773 & B\\
1 & X & F & 4 & H & 4 & C & 890 & B\\
2 & Y & M & 1 & L & 5 & C & 590 & A\\
2 & Y & M & 2 & H & 6 & E & 911 & B\\
2 & Y & M & 3 & L & 7 & C & 708 & B\\
2 & Y & M & 4 & H & 8 & E & 621 & A\\
\hline
\end{tabular}
\end{center}
\end{tiny}

\begin{itemize}
\item it obeys principles 1 \& 2 (obs=rows, vars=cols), but violates 3
\item PROBLEM: redundant information in the table, difficult to change
values for certain variables, or add new variables at the subject
level, \alert{error prone}
\end{itemize}
\end{frame}

\begin{frame}[label={sec:orgdf48c7e}]{Tidy representation of multilevel data}
\begin{columns}
\begin{column}{.5\columnwidth}
\begin{block}{Subject}
\begin{scriptsize}

\begin{center}
\begin{tabular}{rll}
SubjectID & ListID & Gender\\
\hline
1 & X & F\\
2 & Y & M\\
\end{tabular}
\end{center}

\end{scriptsize}
\end{block}
\end{column}
\end{columns}

\begin{columns}
\begin{column}{.2\columnwidth}
\begin{block}{Item}
\begin{scriptsize}

\begin{center}
\begin{tabular}{rl}
ItemID & Freq\\
\hline
1 & H\\
2 & L\\
3 & H\\
4 & L\\
\end{tabular}
\end{center}

\end{scriptsize}
\end{block}
\end{column}

\begin{column}{.7\columnwidth}
\begin{block}{Trial}
\begin{scriptsize}

\begin{center}
\begin{tabular}{rrrlrl}
SubjectID & ItemID & TrialID & Cond & RT & Choice\\
\hline
1 & 1 & 1 & E & 637 & A\\
1 & 2 & 2 & C & 998 & B\\
1 & 3 & 3 & E & 773 & B\\
1 & 4 & 4 & C & 890 & B\\
2 & 1 & 5 & C & 590 & A\\
2 & 2 & 6 & E & 911 & B\\
2 & 3 & 7 & C & 708 & B\\
2 & 4 & 8 & E & 621 & A\\
\end{tabular}
\end{center}

\end{scriptsize}
\end{block}
\end{column}
\end{columns}
\end{frame}

\begin{frame}[fragile,label={sec:orgc14ae03}]{Tidy tools}
 \framesubtitle{Wickham (submitted)}

\begin{columns}
\begin{column}{.5\columnwidth}
\begin{block}{Tidy tools}
\begin{center}
\begin{tabular}{lll}
tidy data & \(\rightarrow\) & tidy data\\
\emph{input} &  & \emph{output}\\
\end{tabular}
\end{center}
\end{block}
\end{column}
\end{columns}

\begin{description}[transforming]
  \item[transform] create/modify variables, rearranging columns
  \item[filter] include/exclude observations (rows)
  \item[aggregate] collapse subsets of observations into single values
  \item[order] sort observations
\end{description}

Not all tools in base \texttt{R} are tidy. Wickham's package \texttt{dplyr} adds tidy
versions, plus additional functionality. Also, optimized for speed!
\end{frame}

\begin{frame}[fragile,label={sec:org1d34f4b}]{\texttt{dplyr} and the Wickham Six}
 According to R developer Hadley Wickham (@hadleywickham), 90\% of data
analysis can be reduced to the operations described by six English
verbs.

\begin{center}
\begin{tabular}{ll}
\texttt{select()} & Include or exclude certain variables (columns)\\
\texttt{filter()} & Include or exclude certain observations (rows)\\
\texttt{mutate()} & Create new variables (columns)\\
\texttt{arrange()} & Change the order of observations (rows)\\
\texttt{group\_by()} & Organize the observations into groups\\
\texttt{summarise()} & Derive aggregate variables for groups of observations\\
\end{tabular}
\end{center}

These functions reside in the add-on package \texttt{dplyr}.  See the data
wrangling cheat sheet!
\end{frame}

\begin{frame}[label={sec:org2695c8c}]{Boolean expressions}
\begin{small}

\begin{center}
\begin{tabular}{lll}
Operator & Name & is TRUE if and only if\\
\hline
A < B & less than & A is less than B\\
A <= B & less than or equal & A is less than or equal to B\\
A > B & greater than & A is greater than B\\
A >= B & greater than or equal & A is greater than or equal to B\\
A == B & equivalence & A exactly equals B\\
A != B & not equal & A does not exactly equal B\\
A \%in\% B & in & A is an element of vector B\\
\end{tabular}
\end{center}

\end{small}
\end{frame}
\end{document}


\documentclass[a4paper, 12pt]{article}

%% Language and font encodings
\usepackage[english]{babel}
\usepackage[utf8x]{inputenc}
\usepackage[T1]{fontenc}

%% Sets page size and margins
\usepackage[a4paper,top=3cm,bottom=2cm,left=3cm,right=3cm,marginparwidth=1.75cm]{geometry}

%% Useful packages
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage[colorinlistoftodos]{todonotes}
\usepackage[colorlinks=true, allcolors=blue]{hyperref}

\title{708 Assignment 1}
\date{}
\author{Karan Dasgupta}
%===============================================================================
\begin{document}
\maketitle

%\begin{abstract}
%Your abstract.
%\end{abstract}

\section*{Question 3}
Showing that Hamiltonians conserve energy, that is, $\frac{dH}{dt} = 0 $ along $x(t)$:\\
Using $H=H(q(t),p(t))$, without any explicit time dependence, that is, $\frac{\partial H}{\partial t} = 0 $

$$ \frac{dH}{dt} = \frac{\partial H}{\partial q_i} \frac{\partial q_i}{\partial t} + \frac{\partial H}{\partial p_i} \frac{\partial p_i}{\partial t}
$$
Substituting $$ \frac{\partial q_i}{\partial t} = \frac{\partial H}{\partial p_i}  \hspace{1cm} \& \hspace{1cm} \frac{\partial p_i}{\partial t} = - \frac{\partial H}{\partial q_i} $$
Thus:
$$ \frac{dH}{dt} = \frac{\partial H}{\partial q_i}\frac{\partial H}{\partial p_i} - \frac{\partial H}{\partial p_i} \frac{\partial H}{\partial q_i} = 0
$$
as we required. \\

This is a significant result for a few reasons. Firstly, the entropy of this system is shown to be an extensive variable, which according to the fundamental postulate of thermodynamics suggests that one can characterise the state of a thermodynamic system by a set of extensive variables, which are all observables.

This also shows that even though the accessible phase space volume can change, we expect the Hamiltonian of the system to be conserved. This may also help us to identify this accessible phase space, since if we choose a phase space that doesn't satisfy this condition, we can determine that this is NOT part of the accessible phase space of the system.

In relation to ergodicity, which states that the variable, in this case energy, averaged over periods of time, exhibits similar behaviour to that over the accessible phase space volume, we show that if the Hamiltonian is conserved, that the internal energy doesn't change as $t \rightarrow \infty$.
\newpage
\section*{Question 5}

\subsection*{Random Walks}

There are a number of different kinds of random walks. One of the simplest is called the Pearson Random Walk, after Karl Pearson, where we have steps of a fixed length, in a random direction. Similarly, we can also have a random walk on a lattice - where we move between the nearest-neighbours on a fixed lattice.  Some others to consider are:
\begin{itemize}
\item Levy flight: where the steps are sized according to the power law distribution, and in random directions. Here, we may find that the effect of the largest step can dominate over the effect of the smaller steps.
\item Shrinking Steps: here, the random walk steps are decreasing in size, the random walker is getting tired. The length of step n is given by $\lambda ^n$, where $\lambda$ is a "shrinking constant" and is $<1$.
\item Growing Steps: here, the random walk steps increase in size, similar to the above case for shrinking steps. An example of this is in turbulent diffusion.
\end{itemize}
More generally, we can look at random walks in continuous space where steps are discrete in time, some discrete space where steps happen in continuous time, or as in the case of diffusion, where both space and time are continuous.\\

Now, we can begin to examine the role that the spatial dimension plays in random walks. If we have a series of steps that trace some path of a random walker, we can define the "Exploration Sphere" as the range of points the random walker can be expected to visit in time $t$, in some arbitrary dimension $d$. We have seen that if the number of steps is proportionate to the length of time that particles move for, then the mean-square displacement is proportional to $\sqrt{t}$. The density of visited sites, $\rho$, is given by the ratio of sites visited to size of the exploration sphere:
$$ \rho \sim \frac{t}{\sqrt{t}^d} \sim t ^{\left( 1-\frac{d}{2} \right) } $$
We arrive at this result by recognising that the numerator, $t$ is simply the number of sites visited, and $\sqrt{t}^d$ is a representation of the volume (omitting some constants). \\

Examining $\rho$ for various $d$:
$$ d<2: \rho \rightarrow 0 $$
$$ d=2: \rho \rightarrow constant$$
$$ d>2: \rho \rightarrow \infty$$

This gives us a few insights about whether we return to the starting point or not. The first is that, for $d>2$, we are guaranteed to return the starting point, since the density of visited sites tends to infinity, termed "recurrent". Secondly, for $d<0$, $\rho$ tends to $0$, so we can't be certain that we will return to the starting point, termed "transient". Lastly, for the case $d=2$, that is, a path in the 2D plane, this technically falls within the recurrent regime, however we find that the mean time to return is $\infty$.
\end{document}


\section{Lists} % (fold)
\label{sec:lists}

\begin{frame}\frametitle{Lists}

    \begin{itemize}
        \item Group variables together
        \item Specific order
        \item Access items using square brackets: [ ]
    \end{itemize}

    However, do not confuse a list with the
    mathematical notion of a vector.

\end{frame}

\begin{frame}\frametitle{Accessing elements}\framesubtitle{The basics}

    \begin{itemize}
        \item First item: [0]
        \item Last item: [-1]
    \end{itemize}

    \codeblock{code/list_access.py}

    Note: can mix element types!

\end{frame}

\begin{frame}\frametitle{Slicing and adding}\framesubtitle{More on lists}

    \begin{itemize}
        \item Lists can be sliced: [2:5]
        \item Lists can be multiplied
        \item Lists can be added
    \end{itemize}

    % Look into this: confusing!
    \codeblock{code/list_add_slice.py}

\end{frame}

\begin{frame}\frametitle{Multiplication}\framesubtitle{Say what?}

    We can even multiply a list by an integer

    \codeblock{code/list_mult.py}

\end{frame}

\begin{frame}\frametitle{Lists are mutable}

    Lists are mutable, this means that individual elements can be changed.

    \codeblock{code/list_mutable.py}
    % list is unchanged after changing x

\end{frame}

\begin{frame}\frametitle{Copying a list}
    How to copy a list?

    % Look into this, confusing
    \codeblock{code/list_copy.py}.

\end{frame}

\begin{frame}\frametitle{What just happened?}
    Variables in Python really are tags:

    \begin{figure}
        \centering
        \includegraphics{img/list_tag.png}
    \end{figure}

    So \texttt{b = a} means: \texttt{b} is same tag as \texttt{a}.

    \vfill
    \tiny{Image from \url{http://henry.precheur.org/python/copy_list.html}}

\end{frame}

\begin{frame}\frametitle{Copying a list}

    Instead: we want:
    \begin{figure}
        \centering
        \includegraphics{img/list_tag2.png}
        \caption{\texttt{b = list(a)} or \texttt{b = a[:]}}
    \end{figure}

    \vfill
    \tiny{Image from \url{http://henry.precheur.org/python/copy_list.html}}

\end{frame}

\begin{frame}\frametitle{Copying a list}

    \codeblock{code/list_copy_id.py}

\end{frame}

\begin{frame}\frametitle{Functions modify lists}
    \framesubtitle{}

    Consider the following function:

    \codeblock{code/list_function.py}

    What is printed?

    \vfill \pause

    \texttt{[1, 2, 3]}, \texttt{[0, 2, 3]}

    \emph{Why does the list change, but variables do not?}

\end{frame}

\begin{frame}\frametitle{Why does the list change, but variables do not?}
    We have not changed the tag, only the contents of the list.
    The variable \texttt{l}, that is attached to the list, becomes local.
    The elements however, do not!

    \pause

    What happens in this case?
    \codeblock{code/list_function2.py}
\end{frame}

\begin{frame}\frametitle{More control over lists}

    \begin{itemize}
        \item \texttt{len(xs)}
        \item \texttt{xs.append(x)}
        \item \texttt{xs.count(x)}
        \item \texttt{xs.insert(i, x)}
        \item \texttt{xs.sort()} and \texttt{sorted(xs)}: what's the difference?
        \item \texttt{xs.remove(x)}
        \item \texttt{xs.pop()} or \texttt{xs.pop(i)}
        \item \texttt{x in xs}
    \end{itemize}

    \pause
    All these can be found in the Python documentation, google: `python list'

    Or using \texttt{dir(xs)} / \texttt{dir([])}

\end{frame}

\begin{frame}\frametitle{Looping over elements}

    It is very easy to loop over elements of a list using \texttt{for}, we have
    seen this before using \texttt{range}.

    \codeblock{code/list_loop.py}

\end{frame}

\begin{frame}\frametitle{Looping over elements}
\framesubtitle{and indices}
    Using \texttt{enumerate}, we can loop over both element and index at the same time.

    \codeblock{code/list_loop2.py}

\end{frame}

\begin{frame}\frametitle{Map}
    \framesubtitle{}

    We can apply a function to all elements of a list using \texttt{map}

    \codeblock{code/list_map.py}

\end{frame}

\begin{frame}\frametitle{Filter}
    \framesubtitle{}

    We can also filter elements of a list using \texttt{filter}

    \codeblock{code/list_filter.py}

\end{frame}

\begin{frame}\frametitle{List comprehensions}
    \framesubtitle{The powers of Python}

    A very powerful and concise way to create lists is using \emph{list comprehensions}

    \codeblock{code/list_compr_intr.py}

    This is often more readable than using \texttt{map} or \texttt{filter}

\end{frame}

\begin{frame}\frametitle{List comprehensions}\framesubtitle{Down the rabbit hole}

    \codeblock{code/list_compr.py}

    \pause

    Note how we can have a lists as elements of a list!

\end{frame}

\begin{frame}\frametitle{Implementing map using list comprehensions}

    Let's implement \texttt{map} using list comprehensions
    \pause

    \codeblock{code/list_mymap.py}

    Implement filter by yourself in one of the exercises.

\end{frame}

% section lists (end)


\documentclass[a4paper,12pt]{article}
\usepackage[left=2.5cm,right=2.5cm,top=2.5cm,bottom=2.5cm]{geometry}
\usepackage{color}
\usepackage[usenames,dvipsnames]{xcolor}
\usepackage{amsmath,amssymb,amsthm,algorithm,algorithmic,graphicx,yhmath,url,enumitem,lscape,mathtools}
\usepackage{wrapfig,subfigure}

\newcounter{problem}
\newenvironment{problem}{\refstepcounter{problem} \noindent {\bf Problem \arabic{problem}}}{\newpage}
\newenvironment{solution}{\vspace{0.3cm} \par \noindent {\bf Solution}}{}
\newenvironment{verification}{\vspace{0.3cm} \par \noindent {\bf Verification}}{}
\newenvironment{hint}{\vspace{0.3cm} \par {\bf Hint:}}{}

\newcounter{remark}
\newenvironment{remark}{\refstepcounter{remark} \vspace{0.3cm} \par \noindent {\bf Remark \arabic{remark}}}{\vspace{0.3cm}}
\newcounter{lesson}
\newenvironment{lesson}{\refstepcounter{lesson} \vspace{0.3cm} \par \noindent {\bf Lesson \arabic{lesson}}}{\vspace{0.3cm}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\N}{\mathbb{N}}
\newcommand{\Rn}{\mathbb{R}^n}
\newcommand{\Rnn}{\mathbb{R}^{n \times n}}
\newcommand{\bes}{\begin{equation*}}
\newcommand{\ees}{\end{equation*}}
\newcommand{\be}{\begin{equation}}
\newcommand{\ee}{\end{equation}}
\newcommand{\eps}{\epsilon}
\newcommand{\fl}{\text{fl}}

\begin{document}

\title{5DV005, Fall 2018, Lab session 1}
\author{Carl Christian Kjelgaard Mikkelsen}

\maketitle
\tableofcontents

\section{The time and the place}
Our first lab session will take place on
\begin{center}
Wednesday, November 14th, 2018, (kl. 13.00-16.00), Room MA416-426.
\end{center}

\section{Compilation of \LaTeX~documents}

Documents written in \LaTeX have the extention {\tt .tex}. They are translated into .pdf files using a multi-pass compiler such at {\tt pdflatex}. The simplest case requires two pass compilation, i.e.,
\begin{verbatim}
>pdflatex lab2
>pdflatex lab2
\end{verbatim}
If the document reads a .bib file to generate a list of references, then the compilation sequence is
\begin{verbatim}
>pdflatex lab2
>bibtex lab2
>pdflatex lab2
>pdflatex lab2
\end{verbatim}
\newpage

\section{The problems}

\begin{problem}
  \begin{enumerate}
  \item Copy the function {\tt lab2/scripts/l2p1.m} into {\tt /lab2/work/my\_sum} and identify the differences between this function and {\tt my\_simple\_sum}. In particular:
    \begin{enumerate}
    \item How is the correct value of the unit roundoff identified?
    \item What is the effect of the reshape command?
    \item How is user stupidity handled?
    \end{enumerate}
  \item Extend the function {\tt my\_sum} to include the a priori error bound given by \cite{mikkelsen2018}, Theorem 4.3. The appropriate call sequence is
\begin{verbatim}
[y, aeb, reb]=my_sum(a)
\end{verbatim}
    where
    \begin{itemize}
    \item {\tt y} is the computed sum
    \item {\tt aeb} is the a priori error bound
    \item {\tt reb} is the running error bound
    \end{itemize}
    \begin{enumerate}
    \item Remember to update the documentation.
    \item Remember to include comments related to the calculation of the error bounds.
    \item Remember to update the revision history.
    \item Remember to update the list of programmers.
    \end{enumerate}
  \item Copy {\tt scripts/l2p1\_mwe1.m} into {\tt work/my\_sum\_mwe1.m} and complete the script so that it computes the two sums
    \be
    a = \sum_{j=1}^n \frac{1}{j^4}, \quad b = \sum_{j=1}^n \frac{(-1)^j}{j}, \quad n = 2^{20},
    \ee
    using both single and double precision as well as the error, an a priori error bound and a running error bound. You can obtain a \textit{very} accurate value of the true sums using Kahan's method {\tt kahan\_sum} and double precision arithmetic.
  \item Verify that the errors are bounded by the a priori error bounds.
  \item Verify that the errors are bounded by the running error bounds.
  \item Which technique gives the best error bounds?
  \item Why is it reasonable that the running error bounds are smaller than the a priori error bounds? {\bf Hint:} Which technique draws on the largest body of relevant information?
  \end{enumerate}
\end{problem}

\begin{problem}
  \begin{enumerate}
  \item Copy {\tt /lab2/scripts/l2p2.m} into {\tt /lab2/work/my\_recursive\_sum.m} and complete the function according to its specification.
  \item Extend {\tt my\_recursive\_sum} to return a running error bound.
  \item Develop a minimal working example {\tt my\_recursive\_sum\_mwe1} which computes the same sums as the previous problem using recursion threshold $m=2^{10}$.
  \item Verify that that the absolute value of each error is less than corresponding running error bound.
  \item What are the consequences of reducing the recursion threshold $m$ from $m=2^{10}$ to $m=1$?
    \begin{enumerate}
    \item What happens to the error?
    \item What happens to the run-time? Use {\tt tic}, {\tt toc} to measure the run-time.
    \end{enumerate}
  \end{enumerate}
\end{problem}

\begin{problem}
  You will find the command {\tt close all} very useful for this problem.
  \begin{enumerate}
  \item Copy {\tt lab2/scripts/l2p3.m} in the function {\tt lab2/work/my\_horner.m} and complete the function according to the specification.
  \item Develop a minimal working example {\tt my\_horner\_mwe1} which plots the two polynomials
\begin{verbatim}
p = @(x)(x-2)^3
q = @(x)x.^3 - 6 x.^2 + 12 x - 8
\end{verbatim}
    for $x \in 2 + [-2^{-15},2^{-15}]$ using 101 equidistant points. Let MATLAB evaluate $p$ as stated, but use {\tt my\_horner} to evaluate $q$. Be careful and specify the coefficients in the right order!
  \item Prove mathematically that $p(x) = q(x)$ for all $x \in \R$, i.e., the two polynomials are in fact identical, although the computed values are quite different.
    \item Extend {\tt my\_horner} to include the calculation of the a priori error bound given by \cite{mikkelsen2018}, Theorem 4.10. The appropriate call sequence is
\begin{verbatim}
[y, aeb] = my_horner(a,x)
\end{verbatim}
  \item Extend {\tt my\_horner} further to include the calculation of a running error bound. The appropriate call sequence is
\begin{verbatim}
[y, aeb, reb] = my_horner(a,x)
\end{verbatim}
  \item Extend the minimal working example {\tt my\_horner\_mwe1} to create a second plot which plots the absolute value of $q$, the a priori error bound and the running error bound all in a single figure.
  \item Use the running error bound to identify the region where the sign of the computed values of $q$ can be trusted. A command such as {\tt find(array1>array2)} will prove useful.
  \end{enumerate}
\end{problem}

\begin{problem} In this problem you will apply the bisection algorithm to compute a firing solution for a target located at a distance $d$ from the gun.
\begin{enumerate}
\item Read through the documentation of the function {\tt bisection} and execute the minimal working example to verify that the code makes sense.
\end{enumerate}
In order to automatically hit a target at coordinates $(d,0)$ we have to set up a function which can be passed to the bisection algorithm along with pertinant information. The following steps takes you through this process. Rename the script {\tt lab2/script/l2p4.m} to {\tt lab2/work/l2s4.m} and completed it as suggested.
\begin{enumerate}[resume]
\item Use the minimal working example of {\tt range\_rkx.m} to initialize a gun and define a range function using those values, i.e.
\begin{verbatim}
range=@(theta)range_rkx(param,v0,theta,method,dt,maxit)
\end{verbatim}
\item Similarly, define a function which measures the distance from the point of impact to the target, i.e
\begin{verbatim}
res=@(theta)range(theta)-d
\end{verbatim}
\end{enumerate}
Here {\tt res} is an abbreviation of the word ``residual'', i.e. ``that which remains''. Hitting the target precisely is equivalent to solving {\tt res = 0}. In practice, we can settle for a residual which has a small absolute value, because the shell destroys anything with a certain radius $\rho$. This so-called ``kill-radius'' depends among other things on the amount and type of explosive and the type of target.

In order to use the bisection method, we need to find a bracket, i.e. a pair of angles $\theta_1$ and $\theta_2$ such that the two residuals have different signs. In terms of physics, this corresponds to firing a pair of shells such that one passes over the target and the other falls short of the target. Suitable brackets can be found if a ballistic table has already been computed.
\begin{enumerate}[resume]
\item Compute a suitable ballistic table using the function {\tt compute\_range}.
\item Explain to your friends, why the command {\tt find(table(2,:)>d)} can be used to rapidly determine a bracket!
\item Compute the elevation necessary to hit a xenomorph at $d = 12345$ using the bisection algorithm, see Figure \ref{xenomorph}. Xenomorphs are very sturdy, so we want a residual which is less than 1 meter. Moreover, we want both the high and the low angle. Use the function {\tt range\_rkx} to generate the trajectories and plot them in the same figure. You should end up with a figure which looks like Figure \ref{trajectories}.

\begin{figure}
\centering
\includegraphics[width=14cm]{xenomorph} \caption{A typical xenomorph worker/warrior} \label{xenomorph}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=14cm]{trajectories.pdf} \caption{A plot of the high and the low trajectories for hitting a target at {\tt d = 12345}.} \label{trajectories}
\end{figure}

\item A UAV (Unmanned Aerial Vehicle, i.e. a drone) has spotted a group of xenomorphs hiding behind a large hill. Your friend has analyzed the telemetry and gives you the new range. Compute the high and the low trajectory. Explain why the high trajectory is the probably the only viable option.
\end{enumerate}
\end{problem}

\section{Concluding remarks}

\begin{enumerate}

\item Programs without proper description of the call sequence, input/output specification are useless to the user and the next programmer. Programs without meaningful comments are impossible to maintain. Good programmers are not afraid of writing their name on their products. Without contact information it is impossible to report bugs or ask for clarification.
  \begin{center}
    {\bf Programs which do not follow these principles will not be accepted.}
  \end{center}

\item Recursion is an elegant solution to many problems, but be wary of the overhead incurred by the function calls. Recursion is appropriate when the overhead is insignificant compared with the actual computation. This is not case if the computation consists of a single addition. Your recursive summation function uses a threshold {\tt m} which sacrifies some accuracy for the sake of speed. It is possible to do tree-wise summation without recursion, but the code is a tad more complicated as demonstrated by {\tt tree\_sum}.

\item There is a lesson to be learned from question about the UAV. You should never blindly rely on the numbers generated by a computer. Always ask yourself what the numbers mean in the real world. In this case the low trajectory is likely to intersect the hill and it is useless for the practical purpose of destroying the xenomorphs. The hill is not part of the simulation, but it is part of the real world. The low trajectory is a valid firing solution, but only within the confines of the simulation.

\end{enumerate}

\bibliographystyle{plain}
  \bibliography{../../../lecture-notes/refs}

\end{document}


\documentclass[letterpaper,12pt]{article}

%\usepackage[margin=1.5in]{geometry}
\usepackage{fullpage}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{bbm}
\usepackage{amsthm}
\usepackage[authoryear]{natbib}

\usepackage{mathrsfs}
\usepackage{mathtools}
\usepackage{parskip}
\usepackage{graphicx}
\usepackage{enumitem}
\usepackage{listings}
\usepackage{geometry}
\usepackage{physics}
\usepackage{hyperref}
\lstset{
  tabsize=2
}

\renewenvironment{abstract}
 {
  \begin{center}
  \vspace{3em} \bfseries \abstractname\vspace{0em}\vspace{0pt}
  \end{center}
  \list{}{%
    \setlength{\leftmargin}{20mm}% <---------- CHANGE HERE
    \setlength{\rightmargin}{\leftmargin}%
  }%
  \item\relax}
 {\endlist}

\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator*{\Var}{Var}
\newcommand{\R}{\mathbb{R}}
\newcommand{\E}{\mathbb{E}}
\newcommand{\1}{\mathbbm{1}}
\newcommand*{\QED}{\hfill\ensuremath{\square}}%

\newcommand{\elf}{M_{\mathrm{ELF}}}
\newcommand{\shelf}{M_{\mathrm{SHELF}}}

\newcommand\numberthis{\addtocounter{equation}{1}\tag{\theequation}}
\newtheorem{theorem}{Theorem}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{definition}[theorem]{Definition}

\begin{document}

\title{Efficient and Truthful Forecasting Competitions}
\author{Rafael Frongillo, Robert Gomez, Anish Thilagar, Bo Waggoner}
\date{\today}
\maketitle

\begin{abstract}
   \citet{witkowski2018incentive} studied competitions among forecasters who predict a set of common events.
   When such a competition picks a single winner, incentives can be distorted, so that paper introduced the ELF mechanism which incentivizes truthful forecasts.
   In this paper, we give a tighter analysis of the ELF mechanism, then give a mechanism that improves upon it exponentially, in the sense of also being truthful but requiring only logarithmically many events in order to select the best forecaster with high probability.
   These results have potential applications to machine-learning competitions such as Kaggle, where winners are currently selected in non-truthful ways.
\end{abstract}

\section{Introduction}

\section{Model and Definitions}
Our model will consist of $n$ forecasters s.t. $n \ge 2$ where $i \in [1,n]$ denotes the $i$th forecaster, $m$ independent events where $k \in [1,m]$ denotes the $k$th event. Each event $k \in [1,m]$ will be associated with a random variable $X_k$. $X_k = 1$ will denote that event $k$ occurred, and, likewise, $X_k = 0$ will denote event $k$ did not occur. $\forall k \in [m] \exists \theta_k$ s.t. $\theta_k$ denotes the true, unknown probability of event $k$ occurring;  $\theta_k = P(X_k = 1) = 1 - P(X_k = 0)$. We will denote each outcome drawn as $x_i \sim X_i$. For each forecaster $i$, we denote their true belief that event $k$ will occur as $\hat{\theta}_{i,k} \in (0,1)$. We then denote $y_{i,k} \in [0,1]$ as the actual report forecaster $i$ makes for event $k$. Note that $y_{i,k}$ need not equal $\hat{\theta}_{i,k}$.

After every event $k \in [1,m]$ has occurred, and thus each forecaster has reported $y_{i,k} \forall i f\forall k$, a mechanism will select exactly one forecaster from the set of $[n]$ as the 'best forecaster'.

\begin{definition}
  \label{mechanism}
  A \textbf{selection mechanism} $M(y, x)$ is a map from the forecasts $y = (y_1, ..., y_n)$ and outcomes of the events $x = (x_1, ..., x_k)$ to an index $w \in [1, n]$.
\end{definition}
We say that forecaster $w$ wins the competition if $M(y, x) = w$. We call a forecaster selection mechanism truthful iff, for all forecasters, their probability of winning is maximized when they reports their beliefs as their forecasts.

\begin{definition}
  \label{truthfullness}
  $M$ is \textbf{truthful} iff $\forall i \in [n]$, $\forall \hat{\theta}_i$, $\forall y^{\prime}_i \neq \hat{\theta}_i$, and $\forall y_j s.t. j \neq i$,
\[ P(M( (y_1, ..., \hat \theta_i, .., y_n), x ) = i) \geq  P(M( (y_1, ..., \hat y_i, .., y_n), x ) = i)\]
$M$ is \textbf{strictly truthful} if the strict inequality inequality holds.
\end{definition}

Additionally, we would like our mechanism to choose the forecaster with the best predictions. We define the accuracy of a single forecaster $i$ as 1 minus the average squared loss of their forecasts compared to the true probabilities.

\begin{definition}
  \label{accuracy}
  A forecaster's \textbf{accuracy} is $a_i = 1 - \frac{1}{m}\sum_{k=1}^m (y_{i, k} - \theta_k)^2$.
\end{definition}
An ideal mechanism would choose forecasters with larger accuracies with higher probabilities. Let $i^* = \argmax_i a_i$ be the highest accuracy forecaster, and let $a^* = a_{i^*}$.

\begin{definition}
  \label{accuracy_gap}
  The \textbf{accuracy gap} $\epsilon = a^* - \max_{i \neq i^*} a_i$ is the difference in accuracy of the first and second best forecasters.
\end{definition}

For a given mechanism $M$ and $n$ forecasters with accuracy gap $\epsilon$, we let $m_\delta$ be the minimum number of events such that $M$ necessarily chooses $i^*$ with probability at least $1 - \delta$. Specifically, for any $m > m_\delta(M, n, \epsilon)$, $P(M(y, x) \neq i^*) \leq \delta$. $m_\delta$ gives us a measure of how many events are needed for a mechanism to pick the best forecaster. We will use it as a metric to compare mechanisms, where we say $M$ is more efficient than $M'$ if $m_\delta(M, n, \epsilon) < m_\delta(M', n, \epsilon)$, since $M$ needs less events to consistently pick the best forecaster.

Similar to work in \cite{witkowski2018incentive}, we will focus on using the proper scoring rule, the quadratic scoring rule in our forecaster selection mechanism.

\begin{definition}
  \label{scoring_rule}
  A \textbf{scoring rule}, $R$ is proper if $\forall p,y \in {0,1}, E_{x~p}[R(p,x)] \ge E_{x~p}[R(y,x)]$
\end{definition}

\begin{definition}
  \label{quadratic_scoring_rule}
  The \textbf{quadratic scoring rule}, is $R_q = 1 - (y - x)^2$, where $x$ denotes the observed outcome, and $y$ denotes the report.
\end{definition}

\subsection{Useful facts}
Bo: TODO name this subsection better.

The following facts, easy to derive and used by \citet{witkowski2018incentive}, will be useful to us as well.

\begin{lemma} \label{lemma:score-accuracy}
  The expected score, from the ground truth point of view, of forecaster $i$ on event $k$ is $1 - \theta_k(1-\theta_k) - (y_{i,k} - \theta_k)^2$.
\end{lemma}
\begin{proof}
By definition, the expected score is
\begin{align*}
  &\theta_k \left[1 - (1 - y_{i,k})^2\right] + (1 - \theta_k) \left[1 - y_{i,k}^2\right]  \\
  &= \theta_k \left[2y_{i,k} - y_{i,k}^2\right] + (1-\theta_k)\left[1 - y_{i,k}^2\right]  \\
  &= 1 - \theta_k + 2 \theta_k y_{i,k} - y_{i,k}^2  \\
  &= 1 - \theta_k + \theta_k^2 - (y_{i,k} - \theta_k)^2 .
\end{align*}
\end{proof}

Averaging over events, we get that $i$'s score on average over the $m$ events, in expectation, is exactly her accuracy minus some penalty term that only depends on how extreme the ground truth events are.
\begin{corollary} \label{cor:avg-score-accuracy}
  The expectation of forecaster $i$'s average score on all events, from the ground truth point of view, is $a_i - \frac{1}{m}\sum_{k=1}^m \theta_k (1 - \theta_k)$, where $a_i$ is $i$'s accuracy.
\end{corollary}

We also immediately obtain:
\begin{corollary} \label{cor:diff-score-accuracy}
  The expected difference in average scores of forecaster $i$ and $j$, from a ground truth point of view, is $a_i - a_j$.
\end{corollary}

\section{Simple average mechanism}
A straightforward selection mechanism often used in practice is what we call the \emph{simple average mechanism} $M_s$. For this mechanism, we assign each forecaster a score $f_{i, k} = R_q(y_{i, k}, x_k)$ for each event. Then, we assign their final score by summing these over all events, $F_i = \sum_{j=1}^m f_{i, j}$. Finally, we choose the forecaster with the highest cumulative score $M_s(y, x) = \argmax_i F_i$.

%%%% Bo: this is good, but I thought we don't need this detail.
%The idea of this mechanism is that each forecaster's final score in this mechanism is $F_i = \sum_{i=1}^m 1 - (y_{i, j} - x_i)^2$ is an analog for their cumulative accuracy, $m a_i = \sum_{k=1}^m 1 - (y_{i, k} - \theta_k)^2$. Therefore, the forecaster with the highest score will correlate to the forecaster with the highest accuracy. However, we will show that this is not necessarily the case.

\subsection{Truthfulness}

%show simple counterexample(s?) \\
%- single event favors extremes \\
%- multiple events can be dominated by single untruthful event when rest are isomorphic \\

As observed by \citet{witkowski2018incentive} and discussed in depth by \citet{lichtendahl2007probability}, this mechanism is generally not truthful. We can consider a simple example with three forecasters and one event, whose true probability is $0.5$.
Alice predicts $0.5$, but Bob predicts $1$ and Charlie predicts $0$.
Observe that Alice \emph{cannot win}, despite being the best forecaster by far: either the event occurs (Bob wins) or it doesn't (Charlie wins).
She can only win be predicting either $0$ or $1$, raising her probability of winning from $0$ to $0.25$, assuming a random tiebreaking rule.

\subsection{Efficiency}
Although the simple average mechanism is not truthful, it is worth studying its efficiency because, intuitively, it seems unlikely to do better using a mechanism that relies on the quadratic scores of the participants.
So as a benchmark for truthful quadratic-score-based mechanisms, let us consider the efficiency of $M_s$ if forecasters were to report their true beliefs $\hat \theta_i$.

For a single event, the most extreme forecasters will always get selected, as in the example above.
However, for multiple events, the averaging starts to favor more accurate forecasters.
In fact, one can show that with $n$ forecasters and an accuracy gap of $\epsilon$, only

  \[ m = \frac{2}{\epsilon^2}\ln\left(\frac{n}{\delta}\right) \]
events suffice to select the best forecaster with probability $1-\delta$.

Sketch (similar to the \citet{witkowski2018incentive} analysis of ELF): Let $i$ be the best forecaster.
She beats some competitor $j$ if she has a higher average score in the competition.
The expected difference in average scores is the difference in accuracies (Corollary \ref{cor:diff-score-accuracy}), which is $a_i - a_j \geq \epsilon$.
On any given event, the difference in scores is an independent random variable bounded in $[-1,1]$.
A standard Hoeffding bound \cite{Bo:todo-cite-or-reference}  shows that the average of $m = \frac{2}{\epsilon^2}\ln\tfrac{n}{\delta}$ such variables is at least $\epsilon$ below its mean with probability $1 - \tfrac{\delta}{n}$.
Complete the argument by taking a union bound over all $n-1 \leq n$ opponents $j$.

This shows that it is possible to select the best forecaster using a number of events only logarithmic in $n$, the number of forecasters -- if truthfulness is not required.
We next consider the number of events used by the ELF mechanism.

\section{ELF}
The ELF mechanism $\elf$ \citep{witkowski2018incentive}, improves upon $M_s$, by incentivizing forecasters to be truthful. For each event $k$, we hold a lottery to choose a winner. Forecaster $i$ is chosen with probability
\[ f_{i, k} = \frac{1}{n} + \frac{1}{n} \left(R_q(y_{i, k}, x_k) - \frac{\sum_{j\neq i} R_q(y_{j, k}, x_k)}{n-1} \right)\]
This score is the additively normalized quadratic score among all forecasters. A forecaster increasing their own quadratic score increases the chances they win an event, while decreasing the chances others win that event. Let $w_k$ be the winner of event $k$. Then we choose the forecaster $\elf(y, x) = \argmax_i \sum_{k=1}^m \1(w_k = i)$ as the winner of the competition. Essentially, we give the winner of each event lottery a point, and then pick the forecaster who got the most points as the overall winner.

In Theorem 6 of \cite{witkowski2018incentive}, it is proven that $\elf$ is strictly truthful. In Theorem 8 of the same paper, they also upper bound $m_\delta$ by showing
\[ m_\delta(\elf, n, \epsilon) \leq \frac{2(n-1)^2}{\epsilon^2} \ln \left(\frac{4(n-1)}{\delta}\right)\]
For a fixed forecaster accuracy gap and desired mechanism accuracy, the number of events ELF needs to consistently choose the right forecaster is proportional to $\frac{1}{\epsilon^2} n^2 \ln n$. However, this upper bound is not tight.

Let $F_{i, k}$ be the indicator function for forecaster $i$ winning the lottery for event $k$. Then, $F_i = \sum_k F_{i, k}$ is the random variable that is the number of events forecaster $i$ wins. The proof of Theorem 8 \cite{witkowski2018incentive} utilized the Hoeffding bound applied to $F_i$, which assumes they have high variance. However, this is not the case, and utilizing that fact gives an improvement in the lower bound.
\begin{theorem}[Bernstein's Inequality]
  \label{bernstein}
  Given random variables $X_i$ for $1 \leq i \leq n$ such that $0 \leq X_i \leq 1$ almost surely, let $S = \sum_i X_i$. Then,
  \[ P\left(|S - \E[S]| < t\right) < 2 \: e^{\frac{-t^2 }{2\left(\Var(S) + \frac{t}{3}\right)} } \]
\end{theorem}
\begin{theorem}
  \label{elf_bound}
    $m_\delta(\elf, n, \epsilon) \leq \frac{5(n-1)}{\epsilon^2}\ln\left(\frac{4(n-1)}{\delta}\right)$.
    % For an accuracy gap $\epsilon$, ELF correctly chooses the most accurate forecaster with probability at least $1 - 4 (n-1) e^{-\frac{m \epsilon}{6(n - 1)}}$.
\end{theorem}
\emph{Proof:} Note that for any single event $k$, every forecaster wins with probability at best $\frac{2}{n}$, and at worst $\frac{1}{n} - \frac{1}{n(n-1)}$ (achieved when $y_i = 1$, $y_{j\neq i} = 0$, and $x_i = 1$). Since $F_{i, k}$ is just a binary random variable, it will have highest variance when it's expectation is as close to $\frac{1}{2}$ as possible. For nontrivially small $n$, this will mean its expectation is $\frac{2}{n}$ (or equivalently $P(F_{i, k} = 1) = \frac{2}{n}$), so its variance will be at most $\frac{2(n-2)}{n^2}$. So we have $\Var(F_{i, k}) \leq \frac{2(n-2)}{n^2} < \frac{2}{n}$.

Since $F_i = \sum_k F_{i, k}$, we have $\Var(F_i) = \sum_k \Var(F_{i, k}) < \sum_k \frac{2}{n} = \frac{2m}{n}$. As shown in \citet{witkowski2018incentive}, for $j\neq i$ we have
\[ \E[F_i] - \E[F_j] \geq \frac{m \epsilon}{n - 1}\]
Therefore, if $F_j \geq F_i$, (forecaster $j$ beats forecaster $i$), then either forecaster $j$ overperformed their expectation by at least $\frac{m \epsilon}{2(n - 1)}$, or forecaster $i$ underperformed their expectation by $\frac{m \epsilon}{n - 1}$. Specifically, either $\E[F_i] - F_i \geq \frac{m \epsilon}{2(n - 1)}$ or $F_j - \E[F_j] \geq \frac{m \epsilon}{2(n - 1)}$

Applying Theorem \ref{bernstein}, we have
\begin{align*}
    P\left(\left| F_i - \E[F_i]\right| < \frac{m \epsilon}{2(n - 1)} \right)
    &< 2 \: e^{\frac{-\left(\frac{m \epsilon}{2(n - 1)}\right)^2 }{2\left(\frac{2m}{n} + \frac{m \epsilon}{6(n - 1)}\right)}} \\
    &< 2 \: e^{\frac{-\left(\frac{m \epsilon}{2(n - 1)}\right)^2 }{2\left(\frac{m}{n-1}(2 + \frac{\epsilon}{6}) \right)}} \\
    &= 2 \: e^{\frac{-\frac{m \epsilon^2}{2(n - 1)} }{(2 + \frac{\epsilon}{6})}} \\
    &< 2 \: e^{-\frac{m \epsilon^2}{5(n - 1)} }  \numberthis \label{elf_bernstein}
\end{align*}
And similarly for $F_j$. Putting both of those cases together, we have by the union bound
\begin{align*}
  P(F_j \geq F_i) &= P\left(\left(F_j - \E[F_j] < \frac{m \epsilon^2}{2(n - 1)}\right) \cup \left(\E[F_i] - F_i < \frac{m \epsilon^2}{2(n - 1)} \right) \right) \\
  &\leq P\left(F_j - \E[F_j] < \frac{m \epsilon^2}{2(n - 1)}\right) + P\left(\E[F_i] - F_i < \frac{m \epsilon^2}{2(n - 1)} \right) \\
  &\leq 4 \: e^{-\frac{m \epsilon^2}{5(n - 1)} }
\end{align*}
Using the union bound over all $j$, we have
\begin{align*}
  P\left(M_l(y_1, ..., y_n, x) = i\right)
  &= 1 - \sum_{j\neq i} P\left(M_l(y_1, ..., y_n, x) = j\right) \\
  &\geq 1 - \sum_{j\neq i} P\left(F_j \geq F_i\right) \\
  &\geq 1 - 4 (n-1) e^{-\frac{m \epsilon^2}{5(n - 1)}}
\end{align*}

Assigning this probability to $1 - \delta$, we see that for a fixed $n, \epsilon$, ELF will choose the correct forecaster with probability at least $1 - \delta$ if $m$ is large enough for the above inequality to hold. Solving it for $m$, we get the minimum such satisfying value
\[m_\delta \leq \frac{5(n-1)}{\epsilon^2}\ln\left(\frac{4(n-1)}{\delta}\right) \]
\hfill\QED

\begin{theorem}
  \label{elf_lower_bound}
    For $\delta < \frac{1}{2}$, $m_\delta(\elf, n, \epsilon) > n \log n$
\end{theorem}
\emph{Proof:} Pick any $\delta < \frac{1}{2}$, $0 < c \leq 1$ and consider the case where $m = c n \log n$ and $\theta_k = 1$ for every event. Let $y_{1, k} = 1$ and $y_{i, k} = 0$ for every $i > 1$. Therefore, forecaster 1 always forecasts correctly and has a quadratic score $R_q(1, 1) = 1$ for each event, while everyone else is wrong and has a quadratic score of $R_q(0, 1) = 0$ for each event. Then, $a_1 = 1$, and $a_i = 0$ for $i > 1$, so we have the worst possible accuracy gap $\epsilon = 1$.

For every event, forecaster 1's score will be
\begin{align*}
  f_1 &= \frac{1}{n}\left(1 + R_q(y_1, 1) - \frac{1}{n-1} \sum_{j \neq i} R_q(y_j, x) \right) \\
      &= \frac{1}{n}\left(1 + 1 - \frac{1}{n-1} \sum_{j \neq i} 0 \right)\\
      &= \frac{2}{n}
\end{align*}
Since all the other forecasters are symmetric, their $f_i$ must all be the same, and since they sum to 1, we must have $f_i = \frac{n-2}{n(n-1)} < \frac{1}{n}$.

We can reframe the way winners are chosen for each event. Instead of holding a lottery, we first flip a coin that gives forecaster 1 a win with probability $\frac{2}{n}$. If it does not give them the win, we run a normal lottery for the remaining forecasters, where they each have probability $\frac{1}{n-1}$ of winning. This gives them an overall probability of $\frac{n-2}{n(n-1)}$ to win each event, so it is the same the original lottery.

Since all the events are uniform, the expected number of lotteries forecaster 1 will win is $m f_1 = \frac{2m}{n} = 2 c \log n$. Specifically, their wins will follow the Bernoulli distribution $B(m, \frac{2}{n})$. With some probability $C_1 = P(B(m, \frac{2}{n}) < 2 c \log n) \approx \frac{1}{2}$, forecaster 1 will win less events than expected.

The distribution of wins for the remaining forecasters will follow a uniform multinomial distribution. Specifically, we can model it with Balls and Bins. Let $M$ be the maximum number of wins of any of the remaining forecasters.
\begin{lemma}
    $P(M > 2 c \log n) = 1 - o(1)$ for any $0 < c \leq 1$.
\end{lemma}
\emph{Proof:}
From Theorem 1 of \citet{raab1998balls}, we know that for $m = c n \log n$, $P(M > k_\alpha) = 1 - o(1)$, where $k_\alpha = (d_c - 1 + \alpha) \log n$, and $0 < 1 < \alpha$, such that $d_c$ is the solution of $1 + x(\log c - \log x + 1) - c= 0$ such that $x > c$.

If $c = 1$, this reduces to $d_1 (1 - \log d_1) = 0$, so $d_1 = e$. Since this inequality holds for any $\alpha$, we let $\alpha = 3 - e$. Plugging in both values gives us $k_\alpha = 2 \log n$, so $P(M > 2 \log n) = 1 - o(1)$.

We can then rewrite the equation that determines $d_c$ as $1 + \frac{1}{x} = \frac{c}{x} - \log \frac{c}{x} $. $\frac{d}{dx}$ of the left hand side is $-\frac{1}{x^2} < 0$ while the right hand side is $\frac{x-c}{x^2} > 0$, since we are only considering the solution where $x > c$. For any $c < 1$, $x = ce$ will not satisfy the equation, since the right hand side remains constant while the left is increasing from our $d_1$ solution. In order for the left side to decrease and the right side to increase, $x$ must increase. Therefore, $d_c > ce$ for $0 < c < 1$.

Letting $\alpha = \max (c (2-e) + 1, 0)$, we have
\[
k_\alpha = (d_c - 1 + \alpha) \log n > (ce - 1 + c (2-e) + 1) \log n = 2c \log n \:.
\]
We can again apply Theorem 1 of \citet{raab1998balls} to get
\[
P(M > 2 c \log n) > P(M > k_\alpha) = 1 - o(1)
\]
for any $c < 1$. Therefore, for any $c \leq 1$, $P(M > 2 c \log n) = 1 - o(1)$. \hfill \QED

Using the lemma, for any $C_2 < 1$ and sufficiently large $n$, $P(M > 2 c \log n) > C_2$. We choose some $C_2 > 2\delta$. Now, the probability that forecaster 1 is not chosen is at least
\[
  P\left(B\left(m, \frac{2}{n}\right) < 2c \log n\right) \times P(M > 2 c \log n) > \frac{1}{2} \times 2\delta = \delta
\]
Therefore, it is not true that for any $n$, $\epsilon$, $P(\elf(y, x) \neq i^*) \leq \delta$ when $m = c n \log n$. Therefore, $m_\delta(\elf, n, \epsilon) \neq c n \log n$ for any $c < 1$.

Therefore, $m_\delta(\elf, n, \epsilon) > n \log n$.
\hfill \QED

\subsection{Other point-per-round mechanisms}
We can extend Theorem \ref{elf_lower_bound} to apply to any normal, anonymous mechanism that assigns a single point per event to 1 forecaster, and then picks the forecaster with the highest score. We can treat such a forecaster selection mechanism as a budget-balanced wagering mechanism for each event, where each forecaster has a shared, fixed wager of $\frac{1}{n}$. Since we want the mechanism to by symmetric, we can assume it satisfies anonymity as described in [wagering mechanism paper].

(TODO: justify normality).

Then, by Lemma 4 in [wagering mechanism paper], we have that forecaster $i$'s payout for an event will be
\[
\frac{1}{n} + f(y_i, x) - \frac{1}{n-1} \sum_{j \neq i} f(y_j, x)
\]
where $f$ is a proper scoring rule whose values are contained in an interval of size $\frac{1}{n}$.

\bibliographystyle{plainnat}
\bibliography{references}

\end{document}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End: