\documentclass{scrartcl}
\usepackage[a4paper,left=1in,right=1in,top=1.2in,bottom=1in]{geometry}
\usepackage{siunitx}
\usepackage{graphicx}
\usepackage{gensymb}
\setkomafont{disposition}{\normalfont\bfseries}

%title
\title{Exercise 06:\\Tuning curves and receptive fields}
\subtitle{Theoretical Neuroscience I}
\author{Maria del Cerro \and Johannes G\"atjen \and Lorena Morton}

%use these for structure/overview
\newcommand\Question{%
  \textbf{Question:}%
}
\newcommand\Answer{%
  \textbf{Answer:}%
}

\begin{document}
\maketitle

We are provided with two Matlab functions for ``mysterious neurons'' MN1 and MN2, which take an image as an input parameter and return the firing rate of the neuron. We investigate their properties by giving them different images of size $50\times 50$ pixels and observing their responses.

\section{Tuning curve}

First we investigate if the neurons are selective for bars of specific orientations. We use twelve different orientations and repeatedly stimulate the neurons with different inputs to obtain an average response for a specific orientation and the standard deviation of the neuronal responses. This is repeated for both `on bars' and `off bars'. The results are plotted in Figure \ref{tuning}. The neuronal responses are virtually identical between the `on bar' and the `off bar' stimuli. MN1 has a maximum sensitivity to bars with an orientation of 30\degree and MN2 is most sensitive to bars with an orientation of 60\degree. Moreover, MN1 has a much higher standard deviation of the firing rate at its peak sensitivity than MN2 does. This means, that MN1 can have both high and low responses when the bar orientation is `correct', whereas MN2 always has a strong response to bars with the preferred orientation.

\begin{figure}
\centering
\includegraphics[trim = {1.3cm 0.5cm 1.8cm 0.9cm}, width=\textwidth, clip]{../pics/tuning_on_off}
\caption{The neuron's average firing rate $\mu$ (blue) plus and minus the standard deviation $\sigma$ (dashed red) plotted as a function of the orientation of bars at random positions in the stimulus image. Left column: `On bar'. Right column: `Off bar'. Top row: MN1. Bottom row: MN2. MN1 is most sensitive to bars with an orientation of 30\degree and the standard deviation increases strongly with the neuron's responsiveness. MN2 is most sensitive to bars with an orientation of 60\degree and the standard deviation also increases with a larger response, but to a much lesser extent.}
\label{tuning}
\end{figure}

\section{Response-weighted average stimulus}
We repeatedly stimulate the neurons with random images and compute the response-weighted average stimulus. We do this for `on spot', `off spot' and `white noise' stimuli. The results are shown in Figure \ref{avestim}.

The average stimuli for MN1 show a clear stripe pattern at the angle indicated by the tuning curve. With the `on spot' stimulus we see light stripes on a dark background, with the `off spot' we see dark stripes on a light background and with the `white noise' we see both light and dark stripes on a background of medium intensity. The average stimuli show, that the neuron is not only sensitive to orientations of bars, but also to the position, which explains the high variance in the average response: When a bar is in the correct orientation, but at the wrong position, the neuron's firing rate is decreased instead of increased. We can conclude that MN1 is a simple neuron.

The average stimuli for MN2 only show the extent of the receptive field and that it is more sensitive to stimuli at the center of the receptive field. This indicates, that the neuron has no preference for the positions of bar stimuli, although we have shown with the tuning curve, that it is sensitive to the orientation. This also explains the comparatively low standard deviation of the neuron's firing rate. Because the position is irrelevant to the neuron \textit{all} bars with the correct orientation will elicit a high response. In conclusion, MN2 is a complex neuron.

The `on spot' and `off spot' stimuli require fewer trials than `white noise' to obtain a weighted average stimulus, with which you can see the structure of the preferred stimulus. However, there is no distinction between areas of the image that the neuron does not care about and areas that are sensitive to the same stimulus intensity as the majority of the example stimulus. Only by looking at the preferred stimulus obtained from both stimulus types can we conclude, that the receptive field of the neuron does not extend all the way to the edge of the input images, and that it is more sensitive to stimuli in the center of the receptive field. By using the `white noise' stimulus on the other hand, we see both the regions that are sensitive to light and dark stimuli, and can simultaneously differentiate them from regions the neuron is indifferent to. However, we do not see a clear difference areas outside of the receptive field and areas of the receptive field that are ignored.

\begin{figure}
\centering
\frame{\includegraphics[width=0.3\textwidth]{../pics/ave_on1}} \hspace{3pt}
\frame{\includegraphics[width=0.3\textwidth]{../pics/ave_off1}} \hspace{3pt}
\frame{\includegraphics[width=0.3\textwidth]{../pics/ave_wn1}}\\
\vspace{6pt}
\frame{\includegraphics[width=0.3\textwidth]{../pics/ave_on2}} \hspace{3pt}
\frame{\includegraphics[width=0.3\textwidth]{../pics/ave_off2}} \hspace{3pt}
\frame{\includegraphics[width=0.3\textwidth]{../pics/ave_wn2}}
\caption{The response-weighted average stimulus images for the mysterious neurons with different stimulus types. Top row: MN1. Bottom row: MN2. Columns left to right: `On spot', `off spot', `white noise'.}
\label{avestim}
\end{figure}

\section{Response-weighted average covariance}

Since MN2 is a complex neuron, calculating the response-weighted average stimulus does not inform us about the neuron's preferred stimuli. We can however calculate the response-weighted average covariance and calculate the principal eigenvectors of the covariance matrix (see Figure \ref{eigen}). This is essentially doing a principal component analysis. The principal eigenvectors of the covariance matrix are those, whose multiples added to the average stimulus elicit the greatest variability in neuronal response. In other words, they are the type of stimulus the neuron ``cares about''.

\begin{figure}
\centering
\begin{tabular}{c c c}
\frame{\includegraphics[width=0.3\textwidth]{../pics/eigenvec1_wn}}&
\frame{\includegraphics[width=0.3\textwidth]{../pics/eigenvec2_wn}} & \begin{tabular}[b]{c c}
  \frame{\includegraphics[width=0.145\textwidth]{../pics/eigenvec1_on}}&
  \frame{\includegraphics[width=0.145\textwidth]{../pics/eigenvec2_on}} \\
  \frame{\includegraphics[width=0.145\textwidth]{../pics/eigenvec1_off}} &
  \frame{\includegraphics[width=0.145\textwidth]{../pics/eigenvec2_off}}
 \end{tabular}
\end{tabular}
\caption{The eigenvectors of the response-weighted average covariance for MN2 with different stimulus types. Left and center: Eigenvectors of the first and second eigenvalues of the average covariance matrix for `white noise' stimulus. Right: First two eigenvectors for `on spot' and `off spot' (first eigenvector left, `on spot' top). The method works for `white noise' stimuli, but fails for the `spot' stimuli. From the eigenvectors obtained with the `white noise' stimuli we can determine the preferred orientation and spatial frequency for the neuron.}
\label{eigen}
\end{figure}

%include picture
%\begin{figure}
%\centering
%\includegraphics[trim = {1.3cm 0 2cm 0.9cm}, width=\textwidth, clip]{../pics/picname}
%\caption{caption text}
%\label{label}
%\end{figure}
\end{document}


\documentclass[letterpaper,12pt]{article}

%\usepackage[margin=1.5in]{geometry}
\usepackage{fullpage}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{bbm}
\usepackage{amsthm}
\usepackage[authoryear]{natbib}

\usepackage{mathrsfs}
\usepackage{mathtools}
\usepackage{parskip}
\usepackage{graphicx}
\usepackage{enumitem}
\usepackage{listings}
\usepackage{geometry}
\usepackage{physics}
\usepackage{hyperref}
\lstset{
  tabsize=2
}

\renewenvironment{abstract}
 {
  \begin{center}
  \vspace{3em} \bfseries \abstractname\vspace{0em}\vspace{0pt}
  \end{center}
  \list{}{%
    \setlength{\leftmargin}{20mm}% <---------- CHANGE HERE
    \setlength{\rightmargin}{\leftmargin}%
  }%
  \item\relax}
 {\endlist}

\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator*{\Var}{Var}
\newcommand{\R}{\mathbb{R}}
\newcommand{\E}{\mathbb{E}}
\newcommand{\1}{\mathbbm{1}}
\newcommand*{\QED}{\hfill\ensuremath{\square}}%

\newcommand{\elf}{M_{\mathrm{ELF}}}
\newcommand{\shelf}{M_{\mathrm{SHELF}}}

\newcommand\numberthis{\addtocounter{equation}{1}\tag{\theequation}}
\newtheorem{theorem}{Theorem}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{definition}[theorem]{Definition}

\begin{document}

\title{Efficient and Truthful Forecasting Competitions}
\author{Rafael Frongillo, Robert Gomez, Anish Thilagar, Bo Waggoner}
\date{\today}
\maketitle

\begin{abstract}
   \citet{witkowski2018incentive} studied competitions among forecasters who predict a set of common events.
   When such a competition picks a single winner, incentives can be distorted, so that paper introduced the ELF mechanism which incentivizes truthful forecasts.
   In this paper, we give a tighter analysis of the ELF mechanism, then give a mechanism that improves upon it exponentially, in the sense of also being truthful but requiring only logarithmically many events in order to select the best forecaster with high probability.
   These results have potential applications to machine-learning competitions such as Kaggle, where winners are currently selected in non-truthful ways.
\end{abstract}

\section{Introduction}
- SHELF\\
- GNOME\\
- ???

keywords: event, forecast, voting

EFV

\section{Model and Definitions}
Our model will consist of $n$ forecasters s.t. $n \ge 2$ where $i \in [1,n]$ denotes the $i$th forecaster, $m$ independent events where $k \in [1,m]$ denotes the $k$th event. Each event $k \in [1,m]$ will be associated with a random variable $X_k$. $X_k = 1$ will denote that event $k$ occurred, and, likewise, $X_k = 0$ will denote event $k$ did not occur. $\forall k \in [m] \exists \theta_k$ s.t. $\theta_k$ denotes the true, unknown probability of event $k$ occurring;  $\theta_k = P(X_k = 1) = 1 - P(X_k = 0)$. We will denote each outcome drawn as $x_i \sim X_i$. For each forecaster $i$, we denote their true belief that event $k$ will occur as $\hat{\theta}_{i,k} \in (0,1)$. We then denote $y_{i,k} \in [0,1]$ as the actual report forecaster $i$ makes for event $k$. Note that $y_{i,k}$ need not equal $\hat{\theta}_{i,k}$.

After every event $k \in [1,m]$ has occurred, and thus each forecaster has reported $y_{i,k} \forall i f\forall k$, a mechanism will select exactly one forecaster from the set of $[n]$ as the 'best forecaster'.

\begin{definition}
  \label{mechanism}
  A \textbf{selection mechanism} $M(y, x)$ is a map from the forecasts $y = (y_1, ..., y_n)$ and outcomes of the events $x = (x_1, ..., x_k)$ to an index $w \in [1, n]$.
\end{definition}
We say that forecaster $w$ wins the competition if $M(y, x) = w$. We call a forecaster selection mechanism truthful iff, for all forecasters, their probability of winning is maximized when they reports their beliefs as their forecasts.

\begin{definition}
  \label{truthfullness}
  $M$ is \textbf{truthful} iff $\forall i \in [n]$, $\forall \hat{\theta}_i$, $\forall y^{\prime}_i \neq \hat{\theta}_i$, and $\forall y_j s.t. j \neq i$,
\[ P(M( (y_1, ..., \hat \theta_i, .., y_n), x ) = i) \geq  P(M( (y_1, ..., \hat y_i, .., y_n), x ) = i)\]
$M$ is \textbf{strictly truthful} if the strict inequality inequality holds.
\end{definition}

Additionally, we would like our mechanism to choose the forecaster with the best predictions. We define the accuracy of a single forecaster $i$ as 1 minus the average squared loss of their forecasts compared to the true probabilities.

\begin{definition}
  \label{accuracy}
  A forecaster's \textbf{accuracy} is $a_i = 1 - \frac{1}{m}\sum_{k=1}^m (y_{i, k} - \theta_k)^2$.
\end{definition}
An ideal mechanism would choose forecasters with larger accuracies with higher probabilities. Let $i^* = \argmax_i a_i$ be the highest accuracy forecaster, and let $a^* = a_{i^*}$.

\begin{definition}
  \label{accuracy_gap}
  The \textbf{accuracy gap} $\epsilon = a^* - \max_{i \neq i^*} a_i$ is the difference in accuracy of the first and second best forecasters.
\end{definition}

For a given mechanism $M$ and $n$ forecasters with accuracy gap $\epsilon$, we let $m_\delta$ be the minimum number of events such that $M$ necessarily chooses $i^*$ with probability at least $1 - \delta$. Specifically, for any $m > m_\delta(M, n, \epsilon)$, $P(M(y, x) \neq i^*) \leq \delta$. $m_\delta$ gives us a measure of how many events are needed for a mechanism to pick the best forecaster. We will use it as a metric to compare mechanisms, where we say $M$ is more efficient than $M'$ if $m_\delta(M, n, \epsilon) < m_\delta(M', n, \epsilon)$, since $M$ needs less events to consistently pick the best forecaster.

Similar to work in \cite{witkowski2018incentive}, we will focus on using the proper scoring rule, the quadratic scoring rule in our forecaster selection mechanism.

\begin{definition}
  \label{scoring_rule}
  A \textbf{scoring rule}, $R$ is proper if $\forall p,y \in {0,1}, E_{x~p}[R(p,x)] \ge E_{x~p}[R(y,x)]$
\end{definition}

\begin{definition}
  \label{quadratic_scoring_rule}
  The \textbf{quadratic scoring rule}, is $R_q = 1 - (y - x)^2$, where $x$ denotes the observed outcome, and $y$ denotes the report.
\end{definition}

\subsection{Useful facts}
Bo: TODO name this subsection better.

The following facts, easy to derive and used by \citet{witkowski2018incentive}, will be useful to us as well.

\begin{lemma} \label{lemma:score-accuracy}
  The expected score, from the ground truth point of view, of forecaster $i$ on event $k$ is $1 - \theta_k(1-\theta_k) - (y_{i,k} - \theta_k)^2$.
\end{lemma}
\begin{proof}
By definition, the expected score is
\begin{align*}
  &\theta_k \left[1 - (1 - y_{i,k})^2\right] + (1 - \theta_k) \left[1 - y_{i,k}^2\right]  \\
  &= \theta_k \left[2y_{i,k} - y_{i,k}^2\right] + (1-\theta_k)\left[1 - y_{i,k}^2\right]  \\
  &= 1 - \theta_k + 2 \theta_k y_{i,k} - y_{i,k}^2  \\
  &= 1 - \theta_k + \theta_k^2 - (y_{i,k} - \theta_k)^2 .
\end{align*}
\end{proof}

Averaging over events, we get that $i$'s score on average over the $m$ events, in expectation, is exactly her accuracy minus some penalty term that only depends on how extreme the ground truth events are.
\begin{corollary} \label{cor:avg-score-accuracy}
  The expectation of forecaster $i$'s average score on all events, from the ground truth point of view, is $a_i - \frac{1}{m}\sum_{k=1}^m \theta_k (1 - \theta_k)$, where $a_i$ is $i$'s accuracy.
\end{corollary}

We also immediately obtain:
\begin{corollary} \label{cor:diff-score-accuracy}
  The expected difference in average scores of forecaster $i$ and $j$, from a ground truth point of view, is $a_i - a_j$.
\end{corollary}

\section{Simple average mechanism}
A straightforward selection mechanism often used in practice is what we call the \emph{simple average mechanism} $M_s$. For this mechanism, we assign each forecaster a score $f_{i, k} = R_q(y_{i, k}, x_k)$ for each event. Then, we assign their final score by summing these over all events, $F_i = \sum_{j=1}^m f_{i, j}$. Finally, we choose the forecaster with the highest cumulative score $M_s(y, x) = \argmax_i F_i$.

%%%% Bo: this is good, but I thought we don't need this detail.
%The idea of this mechanism is that each forecaster's final score in this mechanism is $F_i = \sum_{i=1}^m 1 - (y_{i, j} - x_i)^2$ is an analog for their cumulative accuracy, $m a_i = \sum_{k=1}^m 1 - (y_{i, k} - \theta_k)^2$. Therefore, the forecaster with the highest score will correlate to the forecaster with the highest accuracy. However, we will show that this is not necessarily the case.

\subsection{Truthfulness}

%show simple counterexample(s?) \\
%- single event favors extremes \\
%- multiple events can be dominated by single untruthful event when rest are isomorphic \\

As observed by \citet{witkowski2018incentive} and discussed in depth by \citet{lichtendahl2007probability}, this mechanism is generally not truthful. We can consider a simple example with three forecasters and one event, whose true probability is $0.5$.
Alice predicts $0.5$, but Bob predicts $1$ and Charlie predicts $0$.
Observe that Alice \emph{cannot win}, despite being the best forecaster by far: either the event occurs (Bob wins) or it doesn't (Charlie wins).
She can only win be predicting either $0$ or $1$, raising her probability of winning from $0$ to $0.25$, assuming a random tiebreaking rule.

\subsection{Efficiency}
Although the simple average mechanism is not truthful, it is worth studying its efficiency because, intuitively, it seems unlikely to do better using a mechanism that relies on the quadratic scores of the participants.
So as a benchmark for truthful quadratic-score-based mechanisms, let us consider the efficiency of $M_s$ if forecasters were to report their true beliefs $\hat \theta_i$.

For a single event, the most extreme forecasters will always get selected, as in the example above.
However, for multiple events, the averaging starts to favor more accurate forecasters.
In fact, one can show that with $n$ forecasters and an accuracy gap of $\epsilon$, only

  \[ m = \frac{2}{\epsilon^2}\ln\left(\frac{n}{\delta}\right) \]
events suffice to select the best forecaster with probability $1-\delta$.

Sketch (similar to the \citet{witkowski2018incentive} analysis of ELF): Let $i$ be the best forecaster.
She beats some competitor $j$ if she has a higher average score in the competition.
The expected difference in average scores is the difference in accuracies (Corollary \ref{cor:diff-score-accuracy}), which is $a_i - a_j \geq \epsilon$.
On any given event, the difference in scores is an independent random variable bounded in $[-1,1]$.
A standard Hoeffding bound \cite{Bo:todo-cite-or-reference}  shows that the average of $m = \frac{2}{\epsilon^2}\ln\tfrac{n}{\delta}$ such variables is at least $\epsilon$ below its mean with probability $1 - \tfrac{\delta}{n}$.
Complete the argument by taking a union bound over all $n-1 \leq n$ opponents $j$.

This shows that it is possible to select the best forecaster using a number of events only logarithmic in $n$, the number of forecasters -- if truthfulness is not required.
We next consider the number of events used by the ELF mechanism.

\section{ELF}
The ELF mechanism $\elf$ \citep{witkowski2018incentive}, improves upon $M_s$, by incentivizing forecasters to be truthful. For each event $k$, we hold a lottery to choose a winner. Forecaster $i$ is chosen with probability
\[ f_{i, k} = \frac{1}{n} + \frac{1}{n} \left(R_q(y_{i, k}, x_k) - \frac{\sum_{j\neq i} R_q(y_{j, k}, x_k)}{n-1} \right)\]
This score is the additively normalized quadratic score among all forecasters. A forecaster increasing their own quadratic score increases the chances they win an event, while decreasing the chances others win that event. Let $w_k$ be the winner of event $k$. Then we choose the forecaster $\elf(y, x) = \argmax_i \sum_{k=1}^m \1(w_k = i)$ as the winner of the competition. Essentially, we give the winner of each event lottery a point, and then pick the forecaster who got the most points as the overall winner.

In Theorem 6 of \cite{witkowski2018incentive}, it is proven that $\elf$ is strictly truthful. In Theorem 8 of the same paper, they also upper bound $m_\delta$ by showing
\[ m_\delta(\elf, n, \epsilon) \leq \frac{2(n-1)^2}{\epsilon^2} \ln \left(\frac{4(n-1)}{\delta}\right)\]
For a fixed forecaster accuracy gap and desired mechanism accuracy, the number of events ELF needs to consistently choose the right forecaster is proportional to $\frac{1}{\epsilon^2} n^2 \ln n$. However, this upper bound is not tight.

Let $F_{i, k}$ be the indicator function for forecaster $i$ winning the lottery for event $k$. Then, $F_i = \sum_k F_{i, k}$ is the random variable that is the number of events forecaster $i$ wins. The proof of Theorem 8 \cite{witkowski2018incentive} utilized the Hoeffding bound applied to $F_i$, which assumes they have high variance. However, this is not the case, and utilizing that fact gives an improvement in the lower bound.
\begin{theorem}[Bernstein's Inequality]
  \label{bernstein}
  Given random variables $X_i$ for $1 \leq i \leq n$ such that $0 \leq X_i \leq 1$ almost surely, let $S = \sum_i X_i$. Then,
  \[ P\left(|S - \E[S]| < t\right) < 2 \: e^{\frac{-t^2 }{2\left(\Var(S) + \frac{t}{3}\right)} } \]
\end{theorem}
\begin{theorem}
  \label{elf_bound}
    $m_\delta(\elf, n, \epsilon) \leq \frac{5(n-1)}{\epsilon^2}\ln\left(\frac{4(n-1)}{\delta}\right)$
    % For an accuracy gap $\epsilon$, ELF correctly chooses the most accurate forecaster with probability at least $1 - 4 (n-1) e^{-\frac{m \epsilon}{6(n - 1)}}$.
\end{theorem}
\emph{Proof:} Note that for any single event $k$, every forecaster wins with probability at best $\frac{2}{n}$, and at worst $\frac{1}{n} - \frac{1}{n(n-1)}$ (achieved when $y_i = 1$, $y_{j\neq i} = 0$, and $x_i = 1$). Since $F_{i, k}$ is just a binary random variable, it will have highest variance when it's expectation is as close to $\frac{1}{2}$ as possible. For nontrivially small $n$, this will mean its expectation is $\frac{2}{n}$ (or equivalently $P(F_{i, k} = 1) = \frac{2}{n}$), so its variance will be at most $\frac{2(n-2)}{n^2}$. So we have $\Var(F_{i, k}) \leq \frac{2(n-2)}{n^2} < \frac{2}{n}$.

Since $F_i = \sum_k F_{i, k}$, we have $\Var(F_i) = \sum_k \Var(F_{i, k}) < \sum_k \frac{2}{n} = \frac{2m}{n}$. As shown in \citet{witkowski2018incentive}, for $j\neq i$ we have
\[ \E[F_i] - \E[F_j] \geq \frac{m \epsilon}{n - 1}\]
Therefore, if $F_j \geq F_i$, (forecaster $j$ beats forecaster $i$), then either forecaster $j$ overperformed their expectation by at least $\frac{m \epsilon}{2(n - 1)}$, or forecaster $i$ underperformed their expectation by $\frac{m \epsilon}{n - 1}$. Specifically, either $\E[F_i] - F_i \geq \frac{m \epsilon}{2(n - 1)}$ or $F_j - \E[F_j] \geq \frac{m \epsilon}{2(n - 1)}$

Applying Theorem \ref{bernstein}, we have
\begin{align*}
    P\left(\left| F_i - \E[F_i]\right| < \frac{m \epsilon}{2(n - 1)} \right)
    &< 2 \: e^{\frac{-\left(\frac{m \epsilon}{2(n - 1)}\right)^2 }{2\left(\frac{2m}{n} + \frac{m \epsilon}{6(n - 1)}\right)}} \\
    &< 2 \: e^{\frac{-\left(\frac{m \epsilon}{2(n - 1)}\right)^2 }{2\left(\frac{m}{n-1}(2 + \frac{\epsilon}{6}) \right)}} \\
    &= 2 \: e^{\frac{-\frac{m \epsilon^2}{2(n - 1)} }{(2 + \frac{\epsilon}{6})}} \\
    &< 2 \: e^{-\frac{m \epsilon^2}{5(n - 1)} }  \numberthis \label{elf_bernstein}
\end{align*}
And similarly for $F_j$. Putting both of those cases together, we have by the union bound
\begin{align*}
  P(F_j \geq F_i) &= P\left(\left(F_j - \E[F_j] < \frac{m \epsilon^2}{2(n - 1)}\right) \cup \left(\E[F_i] - F_i < \frac{m \epsilon^2}{2(n - 1)} \right) \right) \\
  &\leq P\left(F_j - \E[F_j] < \frac{m \epsilon^2}{2(n - 1)}\right) + P\left(\E[F_i] - F_i < \frac{m \epsilon^2}{2(n - 1)} \right) \\
  &\leq 4 \: e^{-\frac{m \epsilon^2}{5(n - 1)} }
\end{align*}
Using the union bound over all $j$, we have
\begin{align*}
  P\left(M_l(y_1, ..., y_n, x) = i\right)
  &= 1 - \sum_{j\neq i} P\left(M_l(y_1, ..., y_n, x) = j\right) \\
  &\geq 1 - \sum_{j\neq i} P\left(F_j \geq F_i\right) \\
  &\geq 1 - 4 (n-1) e^{-\frac{m \epsilon^2}{5(n - 1)}}
\end{align*}

Assigning this probability to $1 - \delta$, we see that for a fixed $n, \epsilon$, ELF will choose the correct forecaster with probability at least $1 - \delta$ if $m$ is large enough for the above inequality to hold. Solving it for $m$, we get the minimum such satisfying value
\[m_\delta \leq \frac{5(n-1)}{\epsilon^2}\ln\left(\frac{4(n-1)}{\delta}\right) \]
\hfill\QED

\begin{theorem}
  \label{elf_lower_bound}
    For $\delta < \frac{1}{2}$, $m_\delta(\elf, n, \epsilon) > n \log n$
\end{theorem}
\emph{Proof:} Pick any $\delta < \frac{1}{2}$, $0 < c \leq 1$ and consider the case where $m = c n \log n$ and $\theta_k = 1$ for every event. Let $y_{1, k} = 1$ and $y_{i, k} = 0$ for every $i > 1$. Therefore, forecaster 1 always forecasts correctly and has a quadratic score $R_q(1, 1) = 1$ for each event, while everyone else is wrong and has a quadratic score of $R_q(0, 1) = 0$ for each event. Then, $a_1 = 1$, and $a_i = 0$ for $i > 1$, so we have the worst possible accuracy gap $\epsilon = 1$.

For every event, forecaster 1's score will be
\begin{align*}
  f_1 &= \frac{1}{n}\left(1 + R_q(y_1, 1) - \frac{1}{n-1} \sum_{j \neq i} R_q(y_j, x) \right) \\
      &= \frac{1}{n}\left(1 + 1 - \frac{1}{n-1} \sum_{j \neq i} 0 \right)\\
      &= \frac{2}{n}
\end{align*}
Since all the other forecasters are symmetric, their $f_i$ must all be the same, and since they sum to 1, we must have $f_i = \frac{n-2}{n(n-1)} < \frac{1}{n}$.

We can reframe the way winners are chosen for each event. Instead of holding a lottery, we first flip a coin that gives forecaster 1 a win with probability $\frac{2}{n}$. If it does not give them the win, we run a normal lottery for the remaining forecasters, where they each have probability $\frac{1}{n-1}$ of winning. This gives them an overall probability of $\frac{n-2}{n(n-1)}$ to win each event, so it is the same the original lottery.

Since all the events are uniform, the expected number of lotteries forecaster 1 will win is $m f_1 = \frac{2m}{n} = 2 c \log n$. Specifically, their wins will follow the Bernoulli distribution $B(m, \frac{2}{n})$. With some probability $C_1 = P(B(m, \frac{2}{n}) < 2 c \log n) \approx \frac{1}{2}$, forecaster 1 will win less events than expected.

The distribution of wins for the remaining forecasters will follow a uniform multinomial distribution. Specifically, we can model it with Balls and Bins. Let $M$ be the maximum number of wins of any of the remaining forecasters.
\begin{lemma}
    $P(M > 2 c \log n) = 1 - o(1)$ for any $0 < c \leq 1$.
\end{lemma}
\emph{Proof:}
From Theorem 1 of \citet{raab1998balls}, we know that for $m = c n \log n$, $P(M > k_\alpha) = 1 - o(1)$, where $k_\alpha = (d_c - 1 + \alpha) \log n$, and $0 < 1 < \alpha$, such that $d_c$ is the solution of $1 + x(\log c - \log x + 1) - c= 0$ such that $x > c$.

If $c = 1$, this reduces to $d_1 (1 - \log d_1) = 0$, so $d_1 = e$. Since this inequality holds for any $\alpha$, we let $\alpha = 3 - e$. Plugging in both values gives us $k_\alpha = 2 \log n$, so $P(M > 2 \log n) = 1 - o(1)$.

We can then rewrite the equation that determines $d_c$ as $1 + \frac{1}{x} = \frac{c}{x} - \log \frac{c}{x} $. $\frac{d}{dx}$ of the left hand side is $-\frac{1}{x^2} < 0$ while the right hand side is $\frac{x-c}{x^2} > 0$, since we are only considering the solution where $x > c$. For any $c < 1$, $x = ce$ will not satisfy the equation, since the right hand side remains constant while the left is increasing from our $d_1$ solution. In order for the left side to decrease and the right side to increase, $x$ must increase. Therefore, $d_c > ce$ for $0 < c < 1$.

Letting $\alpha = \max (c (2-e) + 1, 0)$, we have
\[
k_\alpha = (d_c - 1 + \alpha) \log n > (ce - 1 + c (2-e) + 1) \log n = 2c \log n \:.
\]
We can again apply Theorem 1 of \citet{raab1998balls} to get
\[
P(M > 2 c \log n) > P(M > k_\alpha) = 1 - o(1)
\]
for any $c < 1$. Therefore, for any $c \leq 1$, $P(M > 2 c \log n) = 1 - o(1)$. \hfill \QED

Using the lemma, for any $C_2 < 1$ and sufficiently large $n$, $P(M > 2 c \log n) > C_2$. We choose some $C_2 > 2\delta$. Now, the probability that forecaster 1 is not chosen is at least
\[
  P\left(B\left(m, \frac{2}{n}\right) < 2c \log n\right) \times P(M > 2 c \log n) > \frac{1}{2} \times 2\delta = \delta
\]
Therefore, it is not true that for any $n$, $\epsilon$, $P(\elf(y, x) \neq i^*) \leq \delta$ when $m = c n \log n$. Therefore, $m_\delta(\elf, n, \epsilon) \neq c n \log n$ for any $c < 1$.

Therefore, $m_\delta(\elf, n, \epsilon) > n \log n$.
\hfill \QED

\section{SHELF}

For this mechanism, we let the binary random variable $X_{i, k}$ be 1 with probability $R_q(y_{i, k}, x_i)$, and 0 otherwise. Then, for each event $k$ we assign forecaster $i$ the score $f_{i, k} = X_{i, k}$ obtained by sampling this variable. Finally, we assign their final score by summing these over all events, $F_i = \sum_{j=1}^m f_{i, j}$, then chose the forecaster with the highest score $\shelf(y, x) = \argmax_i F_i$, breaking ties uniformly at random.

\subsection{Truthfulness}
Note: The following analysis is inspired by and follows the work done in \citet{witkowski2018incentive}.

(This is wrong because we can't just look at their probability of winning a point. We need to also factor in the tiebreaker weighting some wins differently)

For a single event with true probability $\theta$, every forecaster will either win 1 or 0 events. The probability that their final score is 1 will just be their expected score for the single event. Therefore, to maximize their final score, each forecaster must maximize their expected score for the single event. Their score is just $\E_{x \sim \theta}(R_q(y_i, x)) = \E_{x \sim \theta}(1 - (y_i - x)^2)$. This is maximized when $y_i = \theta$, so the mechanism is truthful. We can also see this directly by noting that $R_q$ is a strictly proper scoring rule, so its expectation is maximized when $y_i = \theta$.

\begin{theorem}
  \label{shelf_truthful}
    $\shelf$ is truthful for $m \geq 1$ events.
\end{theorem}
\emph{Proof:} For each forecaster $j \in [1, n]$ fix their report.

Suppose that for some forecaster $i$, $i$'s report on some subset of events, where $k \in [1, m]$ denotes the size of this subset, does not report their true belief for any of the $k$ events.\\
Let $k^\prime$ be one of those events.

We will now show that $i$ will maximize their probability of being chosen as the best forecaster in the SHELF mechanism iff they alter their $k$ reports to report their true belief on each of the $k$ events. A guiding intuition here would be as follows: The higher the expected score a forecaster has on a given event, the higher the probability that SHELF awards them a point on this specific event, and the only way to maximize one's probability of being selected as the best forecaster is to increase the number of points they are awarded. Therefore, for a single independent event, increasing one's expected score will either increase the probability of being selected as the best forecaster or leave it unchanged, and it does so without altering results of any other event. In this way, we will show that being truthful for a singular event will always increase one's probability of being selected as the best forecaster, because, using a quadratic scoring rule, a forecaster's expected score is maximized when reporting their true belief. We can then repeat this process for each untruthful report to maximize the probability of being selected as the best forecaster.\\\\
Formally, we will analyze three cases relating to the context provided by the $m - 1$ events other than $k^\prime$.

Let $V(i)$ denote the number of points forecaster $i$ has accrued.\\
Let $f_{i,k}$ denote the probability forecaster $i$ scores a point on event $k$.\\

\begin{enumerate}
    \item Some forecaster $j$ scores at least two more points than $i$, OR $i$ scores at least two more points than any other forecaster.\\
    $$\exists j \in [1,m] \text{ s.t. } j \ne i \text{ and } V(j) - V(i) \geq 2$$
    $$OR$$
    $$\forall j \in [1,m] \text{ s.t. } j \ne i; V(i) - V(j) \geq 2$$
    For this case, $i$'s probability of being selected as the best forecaster is either exactly $1$ or $0$. It is also unaffected by being truthful or not in $k^\prime$, because gaining, or failing to gain, $1$ point in this context is not enough to alter the outcome as only the forecaster with the most number of points is chosen as the best forecaster with ties broken uniformly.

    \item Some forecaster $j$ scores exactly one more point than $i$, AND no other forecasters score strictly more points than $i$.\\
    $$\exists j \in [1,m] \text{ s.t. } j \neq i \text{ and } V(j) - V(i) = 1$$
    $$AND$$
    $$\forall l \in [1,m] \text{ s.t. } l \neq i \text{ and } l \neq j; V(i) - V(l) \geq 0$$
    For this case, $i$ must seek to maximize $f_{i,k^\prime}$ to have any non-zero probability of being selected as the best forecaster under the SHELF mechanism.
    In the SHELF mechanism, the maximum payout a forecaster can receive for a single event is $1$, and that point is awarded to them with probability exactly equal to their quadratic score. Thus, we know a forecaster's expected value for a singular event is exactly their quadratic score. Since the quadratic scoring rule is proper, we know that the expectation of a forecaster's quadratic score is maximized when the forecaster is reporting their true beliefs. As a result, we can conclude that the only way for $i$ to maximize their probability of receiving a point in $k^\prime$ is to report their true beliefs. By reporting their true beliefs, which maximizes the probability that they score another point, $i$ also maximizes the probability of being selected as the best forecaster which, in this context, occurs when only $i$ scores a point and the tie between $j$ and $i$ is broken uniformly via the rules of SHELF.

    \item No forecasters score more points than $i$, but there is at least one forecaster $j$ that scores the same number of points as $i$ or scores exactly one less point than $y_i$.
    $$\exists j \in [1,m] \text{ s.t. } j \ne i \text{ and } V(i) - V(j) \in \{0,1\}$$
    In any given configuration, a forecaster's probability of being selected as the best forecaster is maximized when the minimal number of other forecaster have as many points as them. In the SHELF mechanism, any forecaster's probability of scoring a point on a given event is independent of all others. Therefore, there is no action that can be taken to minimize the probability of any competitor's probability of being selected as the best forecaster except strictly increasing one's own point total. In this final case, the only way for $i$ to increase their probability of being chosen as the best forecaster would be to increase the probability of scoring a point in $k^\prime$. The same logic as case 2 holds here that $i$ would then prefer to be truthful and maximize their expected score in $k^\prime$ to increase their probability of being selected as the best forecaster.
\end{enumerate}

Since any configuration of the $m-1$ events falls into one of these three cases, we have shown that any forecaster is weakly incentivized to report truthfully.
To show that any forecaster is strongly incentivized to report truthfully, we simply show that $i$ believes cases 2 or 3 will happen with positive probability. Since $y_{j,k} = p_{j,k} \in (0,1) \forall j \forall k$, we know that every forecaster has some uncertainty on each event. As a result, we can conclude that $f_{j,k} > 0 \forall j \forall k$. We can think of all possible outcomes as the set of all possible binary strings of length $n$, and since $f_{j,k} > 0 \forall j \forall k$ there exists at least one sequence of outcomes strings picked from our set that constitutes cases 2 or 3. As a result, there is a positive probability that cases 2 or 3 occur, and thus forecasters are strongly incentivized to report truthfully.

\subsection{Accuracy}

% From Proposition 2 of \citet{witkowski2018incentive}, we have that $E_{x \sim \theta}(R_q(y, x)) = R_q(y, \theta) + \theta^2 - \theta$. Additionally, for a single event, a forecaster's accuracy $a_i$ is by definition just $R_q(y, \theta)$. So, $E_{x \sim \theta}(R_q(y, x)) = a_i + \theta^2 - \theta$. For a single event with probability $\theta$, forecaster $i$'s expected score is $\E_{x \sim \theta} f_i = E_{x \sim \theta}(R_q(y_i, x)) = a_i + \theta^2 - \theta$.

% \subsection{Truthfulness for a single event}
% (section 4.3, proposition 5).  For forecaster $i$ to be selected, they either need to have $f_i = 1$, and then be chosen uniformly at random from all others who scored 1, or everyone must score 0 and they will be chosen uniformly at random. In the first case, the expected number of other forecasters who also score 1 is $\sum_{i' \neq i} \E_{x \sim \theta} f_{i'} = \sum_{i' \neq i} (a_{i'} + \theta^2 - \theta) = (n-1) (\theta^2 - \theta) \sum_{i' \neq i} a_{i'}$. Then, the probability that forecaster $i$ is chosen is
% \[
%   P(f_i = 1) \frac{1}{1 + (n-1) (\theta^2 - \theta) \sum_{i' \neq i} a_{i'}} = \frac{a_i +\theta^2 - \theta}{1 + (n-1) (\theta^2 - \theta) \sum_{i' \neq i} a_{i'}}
% \]
% In the other case, the probability $i$ wins is just
% \[
%   \frac{1}{n} \prod_{i} P(f_i = 0) = \frac{\prod_i (1 - a_i -\theta^2 + \theta)}{n}
% \]

% There's probably a better way to do this... Actually coming back to this I'm not even sure that the first step is allowed (using expected number of winners directly instead of summing over cases)

% \subsection{Truthfulness for multiple events}
\begin{theorem}
  \label{shelf_bound}
    $m_\delta(\shelf, n, \epsilon) \leq \frac{2}{\epsilon^2} \ln \frac{4(n-1)}{\delta}$
\end{theorem}
\emph{Proof:} This proof is a slight modification of the proof of Theorem 8 in \cite{witkowski2018incentive}.

We first bound the difference in the expected number of wins of $i^*$ and $j$, for any $j \neq i^*$.
\begin{align*}
  \E \left( \sum_{k=1}^m f_{i^*, k}\right) - \E \left( \sum_{k=1}^m f_{j, k}\right)
  &= \left( \sum_{k=1}^m \E f_{i^*, k}\right) - \left( \sum_{k=1}^m \E f_{j, k}\right)\\
  &= \sum_{k=1}^m \left(\E (f_{i^*, k}) - \E (f_{j, k})\right)\\
  &= \sum_{k=1}^m \left((a_{i^*} + \theta_k^2 - \theta_k) - (a_j + \theta_k^2 - \theta_k)\right)\\
  &= \sum_{k=1}^m \left(a_{i^*} - a_j\right)\\
  &\geq \sum_{k=1}^m \epsilon\\
  &=m \epsilon
\end{align*}
If $F_j \geq F_{i^*}$, then either $\E(F_{i^*}) - F_{i^*} \geq \frac{m \epsilon}{2}$ or $F_j - \E(F_j) \geq \frac{m \epsilon}{2}$. By Hoeffding's, we can upper bound the probability of each of these events by $2 e^{-\frac{2 (\frac{m \epsilon}{2})^2}{m}} = 2 e^{\frac{- m \epsilon^2}{2}}$. Using the union bound, the probability that either of those events occurs is twice this, or $4 e^{\frac{- m \epsilon^2}{2}}$.

Using the union bound again over all $j \neq i$, we have
\begin{align*}
  P(\shelf(y, x) = i) &= P(F_{i^*} > F_j \quad \forall j \neq i^*) \\
                      &\geq 1 - \sum_{j\neq i^*} P(F_{i^*} \leq F_j) \\
                      &= 1 - 4 (n-1) e^{\frac{- m \epsilon^2}{2}}
\end{align*}

Letting this quantity be $1 - \delta$ and solving for $m$, we see that this mechanism correctly chooses forecaster $i$ with probability $1 - \delta$ when $m$ is large enough to satisfy the inequality. Therefore
\[ m_\delta(\shelf, n, \epsilon) \leq \frac{2}{\epsilon^2} \ln \frac{4(n-1)}{\delta} \]
\hfill \QED

Comparing this to the result of Theorem \ref{elf_bound}, we see that $\shelf$ is much more efficient than $\elf$. We can also directly apply the proof of Theorem \ref{shelf_bound} to $M_s$, since the mean and variance of $F_i$ is the same in both cases. Therefore, $m_\delta(\shelf, n, \epsilon) = m_\delta(M_s, n, \epsilon)$.

\newpage
\section{Raf notes}

Let's study the one-round version of ELF + SHELF + noise.
And actually, we won't really need to use SHELF per se.
Let's actually define
$M_{f,\sigma}(y,x) = \max_{i\in[n]} f(y,x)_i + X_i$ where the $X_i \sim N(0,\sigma^2)$ are indpendent, and $f$ is any function with $f(y,x) \in [0,m]^n$ where if you change only one entry of $y$ by $c$, then $f$ changes by at most $c^2$.
(For example, $f$ could be the sum of scores.)
Let $M_\alpha$ be the mechanism that runs $M_{f,\sigma}$ with probability $\alpha$ and $\elf$ with probability $1-\alpha$.

In what follows, when I omit $x$, I'm implicitly taking the randomness over $x$ into account (from $\theta$).
Assume $y$ is truthful for forecaster $i$ and $y'$ is any vector that differs only in $i$'s forecast on a particular round.
And actually let's focus on the case $m=1$, so there is only one round anyway.
From a direct calculation (see also a Prop.\ in \cite{witkowski2018incentive}), we have, from forecaster $i$'s point of view,
\begin{align}
  \Pr[\elf(y)=i] - \Pr[\elf(y')=i] = \tfrac 1 n (y_i - y'_i)^2~,
\end{align}
since forecaster $i$ believes $y_i = \theta_1$.

By differential privacy (I have not checked this carefully!), and the fact that $\exp(c) \leq 1+2c$ for $c\in[0,1]$, we have, for each $x$,
\begin{align}
  \Pr[M_{f,\sigma}(y',x)=i]
  & \leq \exp\left( \frac{(y_i-y'_i)^2}{2\sigma^2} \right) \Pr[M_{f,\sigma}(y,x)=i]
  \\
  & \leq \left(1 + \frac{(y_i-y'_i)^2}{\sigma^2}\right) \Pr[M_{f,\sigma}(y,x)=i]
  \\
  & \leq \Pr[M_{f,\sigma}(y,x)=i] + \frac{(y_i-y'_i)^2}{\sigma^2}~.
\end{align}

Combining, the probability of $i$ winning for $y$ and $y'$ can be compared as follows.
\begin{align}
  &\Pr[M_\alpha(y)=i]-\Pr[M_\alpha(y')=i]\\
  &=\alpha\left(\Pr[M_{f,\sigma}(y)=i] - \Pr[M_{f,\sigma}(y')=i]\right)\\
  &\qquad + (1-\alpha)\left(\Pr[\elf(y)=i] - \Pr[\elf(y')=i]\right)~.
  \\
  &\geq -\alpha \frac{(y_i-y'_i)^2}{\sigma^2} + (1-\alpha) \frac{(y_i-y'_i)^2}{n}
  \\
  &\geq \left((1-\alpha) \frac{1}{n} -\alpha \frac{1}{\sigma^2}\right) (y_i-y'_i)^2~.
\end{align}
Thus, setting $\sigma^2 = n \tfrac \alpha {1-\alpha}$ gives truthfulness, e.g.\ $\sigma^2=n$ for $\alpha=\tfrac 1 2$.
Which is maybe not what we were hoping for, but it's something.

\bibliographystyle{plainnat}
\bibliography{references}

%\begin{thebibliography}{99}
%\bibitem{elf} Witkowski, Freeman  \emph{Incentive-Compatible Forecasting Competitions} \url{https://las.inf.ethz.ch/files/witkowski-etal-aaai2018.pdf}.
%
%\bibitem{bnb} Raab, Steger  \emph{“Balls into Bins” — A Simple and Tight Analysis} \url{http://www.dblab.ntua.gr/~gtsat/collection/scheduling/Balls%20into%20Bins%20A%20Simple%20and%20Tight%20Anal.pdf}.
%\end{thebibliography}

\end{document}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End:


\documentclass[12pt]{article}
\usepackage[english]{babel}
\usepackage{graphicx}
\usepackage{float}
\usepackage{mathtools}
\usepackage{gensymb}
\usepackage{fancyhdr}
\pagestyle{fancy}

\fancyhead[R]{}
\fancyhead[L]{\rightmark}

%\renewcommand{\headrulewidth}{0pt}

\title{CMSC 6950 Final Project}
\smallskip
\author{
        \textbf{Team 2:}\\
        A. Ghayour-Khiavi, S. Gallant,\\
        T. Dasyam, Y. Ramezani \\
        \and
}
\smallskip
\date{\textit{June 14, 2018}}

\DeclareMathSizes{12}{18}{12}{12}

% % % % % % % % % % % % % % % % % % % % %

\begin{document}
\maketitle

\begin{abstract}
Growing degree days (GDD) are values used in phenology measuring total heat accumulation in a region, in order to predict rates of development in  plants or animals. Using the historical climate data available from the Government of Canada, this project aims to create a coherent workflow leading to the calculation and analysis of GDD for a number of Canadian cities.
\end{abstract}

\smallskip

\tableofcontents

% % % % % % % % % % % % % % % % % % % %
\pagebreak

\section{Motivations}\label{motivations}

Plants require a number of things in order to grow: sunlight,
water, CO\textsubscript{2}, and nutrients. Though sunlight does provide the energy
needed for plants to undergo photosynthesis, their development and maturation
is also largely dependant on daily air temperatures.

\bigskip
\par
Growing degree days (GDD) are a way to monitor the total heat
accumulated by the plant through the ambient temperature, over a
period of time. Many species will only hit development milestones
after a certain accumulated amount of heat, so GDD can be used to
keep track of the heat that has been available in a region over time.

\bigskip
\par
GDD proves an important tool for biologists, horiculturists, and agriculturers,
allowing them to predict things like crop harvest dates,
nutrient timing, or even dates when insects will emerge.

\pagebreak

\section{Calculating GDD}\label{core tasks}
%\subsection{Calculating GDD}\label{calculating gdd}

Growing degrees are defined as the number of degrees the average daily
temperature is, above a set base value (Tbase). This base value is
usually defaulted to 10 \degree C, though each species may have a more
accurate value, and signifies the temperature below which no growth
will occur. If the daily average happens to be below Tbase, the GDD
value for that day is set to zero. \\

\bigskip
\par

The value of daily GDD is calculated with the formula:
$$
\textit{GDD\textsubscript{daily}} =  \text{max}( \tfrac{\text{Tmax}+\text{Tmin}}{2}-\text{Tbase}, 0)
$$

\bigskip
\bigskip

For accumulated GDD, the daily values are summed:
$$
\sum_{i=1}^n ( \tfrac{\text{Tmax}+\text{Tmin}}{2}-\text{Tbase})
$$

\bigskip

\par
Climate data is available for a multitude of Canadian weather stations, provided
by the Government of Canada, where temperatures and precipitation values are reported.
Using this data, GDD values can be calculated for
specific regions and time periods, allowing for analysis of climate trends and patterns.

%\subsection{title}

% % % % % % % % % % % % % % % % % % %
\pagebreak

\section{Results}\label{data analysis}
\subsection{Task 1.2}

\begin{figure}[!htbp]
\centering
\includegraphics[width=0.95\textwidth]{./docs/maxmins_all.jpg}
\caption{\scriptsize Annual cycle of min/max daily temperatures for Montreal, Toronto, and Victoria}
\label{minmax_all}
\end{figure}

\pagebreak
\subsection{Task 1.4}

\begin{figure}[!htbp]
\centering
\includegraphics[width=0.8\textwidth]{./docs/MontrealGDD.png}
\caption{\scriptsize Accumulated GDD for Montreal}
\label{accuGDD_1}
\end{figure}

\begin{figure}[!htbp]
\centering
\includegraphics[width=0.8\textwidth]{./docs/TorontoGDD.png}
\caption{\scriptsize Accumulated GDD for Toronto}
\label{accuGDD_2}
\end{figure}

\begin{figure}[!htbp]
\centering
\includegraphics[width=0.8\textwidth]{./docs/VictoriaGDD.png}
\caption{\scriptsize Accumulated GDD for Victoria}
\label{accuGDD_3}
\end{figure}

\pagebreak

\subsection{Task 2.1}
\begin{figure}[!htbp]
\centering
\includegraphics[width=1.0\textwidth]{./docs/task1.png}
\caption{\scriptsize GDD for Montreal with 5-95\% and 25-75\% percentiles}
\label{GDDwCI}
\end{figure}

\pagebreak
\subsection{Task 2.4}
\begin{figure}[!htbp]
\centering
\includegraphics[width=0.95\textwidth]{./docs/bokeh_plot.png}
\caption{\scriptsize Bokeh plot of accumulative GDD for Montreal in 2017 (Static snapshot)}
\label{bokeh}
\end{figure}

\pagebreak
\subsection{Task 2.6}
\begin{figure}[!htbp]
\centering
\includegraphics[width=0.95\textwidth]{./docs/LinearReg_Ottawa_2014_2017.png}
\caption{\scriptsize Statistical analysis of GDD year-over year for Ottawa}
\label{GDDregression}
\end{figure}

\pagebreak

\subsection{Optional Task 1.1}
\begin{figure}[!htbp]
\centering
\includegraphics[width=1.25\textwidth]{./docs/aprildaily.png}
\caption{\scriptsize Maximum and minimum temperatures through the month of April in Montreal}
\label{Aprilmaxmin}
\end{figure}

\pagebreak

\subsection{Optional Task 1.2}
\begin{figure}[!htbp]
\centering
\includegraphics[width=1.0\textwidth]{./docs/histogram2014.png}
\caption{\scriptsize Histograms of temperature trends in Montreal for 2014}
\label{hist2014}
\end{figure}

\begin{figure}[!htbp]
\centering
\includegraphics[width=1.0\textwidth]{./docs/histogram2015.png}
\caption{\scriptsize Histograms of temperature trends in Montreal for 2015}
\label{hist2015}
\end{figure}

\begin{figure}[!htbp]
\centering
\includegraphics[width=1.0\textwidth]{./docs/histogram2016.png}
\caption{\scriptsize Histograms of temperature trends in Montreal for 2016}
\label{hist2016}
\end{figure}

\begin{figure}[!htbp]
\centering
\includegraphics[width=1.0\textwidth]{./docs/histogram2017.png}
\caption{\scriptsize Histograms of temperature trends in Montreal for 2017}
\label{hist2017}
\end{figure}

% % % % % % % % % % % % % % % % %

\clearpage

\bibliographystyle{abbrv}
\bibliography{main}

\begin{thebibliography}{2}
\bibitem{GovCan}
Historical Climate Data | Environment and Climate Change Canada
\\\texttt{http://climate.weather.gc.ca}
\bibitem{GDD MSU}
Understanding growing degree days | Michigan State University
\\\texttt{http://msue.anr.msu.edu/news/understanding\_growing\_degree\_days}
\bibitem{farmwest}
Growing Degree Days | Farmwest
\\\texttt{http://www.farmwest.com/node/936}
\end{thebibliography}

\end{document}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{amssymb}
\usepackage{booktabs}
\usepackage{csquotes}
\usepackage[margin=1in]{geometry}
\usepackage{url}

\renewcommand{\pod}[1]{\allowbreak\mathchoice%
  {\if@display \mkern18mu\else \mkern8mu\fi (#1)}
  {\if@display \mkern18mu\else \mkern8mu\fi (#1)}
  {\mkern4mu (#1)}
  {\mkern4mu (#1)}
}

\title{On very large sets of numbers that are neither algebraic nor reducible to irrational algebraic}
\date{1851}
\author{Liouville, J.\thanks{%
    Translated by: John Demessemaeker, Bernardo Meurer}}

\begin{document}
\maketitle
\newpage

\paragraph{1.}
A long time ago, I gave a presentation about this subject at the Science
Academy. I also had two Notes inserted in volume XVIII of Comptes rendus
(pages 883 and 910; sessions of the 13th and 20th of May 1844). I believe these
two Notes need to be reproduced and completed here. The first Note is formulated
as the following:
\begin{displayquote}
	To give examples of continued fractions of which we can prove that their
	value is not at the root of any algebraic equation
	$$f(x)=ax^n + bx^{n-1} + \cdots + gx + h = 0$$
	$a, b, \ldots, g$ being integers, it is sufficient to recollect that
	$\frac{p_0}{q_0}$ and $\frac{p}{q}$, being two successive reductions of the
	continued fraction that expresses the development of an immeasurable $x$ of
	this equation, the incomplete quotient $\mu$, that comes after the reduction
	$\frac{p}{q}$, serves to form the next reduction, ends up (that results from
	a Lagrange formula, referencing the Memoires of Berlin, 1768) being, for
	very big $q$ values, constantly inferior to
	$$\pm\frac{df(p,q)}{qf(p,q)dp}$$
	essentially a positive expression where we assume
	$$f(p,q)=q^n f(\frac{p}{q}) = ap^n + bp^{n-1} q + \cdots + hq^n$$
	Given that the abstraction is made of signs, we have now, with more certainty,
	$$\mu < \frac{df(p,q)}{qdp}$$
	since $f(p, q)$ is an equal integer at least to the unity if we admit (what
	is allowed) that the equation
	$$f(x) = 0$$
	has been stripped of any measurable factor; $f(p,q) = 0$ would give, indeed,
	$$f(\frac{p}{q}) = 0$$
	Now represented by $f'(x)$ the differential of $f(x)$, the inequality above
	will become
	$$\mu < q^{n-2}f'(\frac{p}{q})$$
	However, $f'(\frac{p}{q})$ is a finite quantity going towards the limit
	$f'(x)$, like $\frac{p}{q}$ to the limit $x$. Designating with $A$ a certain
	fixed number,which is superior to this limit, we will be sure to have
	$$\mu < Aq^{n-2}$$

	Thereby, the incomplete quotients of a continued fraction representing the
	root $x$ of an algebraic equation of the degree $n$, in rational coefficients,
	are subject to never pass the product of a certain constant number by the
	force $(n-2)$\textsuperscript{nd} of the denominator of the previous
	reduction.

	It is sufficient to give the incomplete quotients $\mu$ a mode of formation
	that makes them grow beyond a predetermined term, to obtain continued
	fractions of which the value will not be able to satisfy any algebraic
	equation itself; it will happen, for example, that if, starting with any first
	incomplete quotient, we form every following $\mu$ with the help of the
	previous reduced $\frac{p}{q}$, according to the law $\mu = q^q$, or according
	to the law $\mu = q^m$, $m$  being the index of the $\mu$ row.

	Moreover, the previous method, which offered the first, is not the only nor
	is it the simplest that we can use. Let’s add that there are also analogue
	theorems for ordinary series. In particular, we cite the series
	$$\frac{1}{l} + \frac{1}{l^{1 \cdot 2}} + \frac{1}{l^{1\cdot 2\cdot 3}}
		+ \cdots + \frac{1}{l^{1\cdot 2\cdot 3\cdots m}} + \cdots$$
	$l$ being a whole number.
\end{displayquote}

\paragraph{2.}
The second Note contains a new and more simple demonstration of the theorem to
which I was led by the Lagrange formula. The real power of our method, as we
will see, is independent of this formula.
\begin{displayquote}
	If $x, x_1, x_2, \ldots, x_{n-1}$ are the $n$ roots (the first real, the others real or
	imaginary) of the algebraic equation
	$$f(x) = ax^n + bx^{n-1} + \cdots + gx + h = 0$$
	which we can assume is irreducible, and where $a, b, \ldots, g, h$ are
	integers that are either positive, negative or zero, as we wish. Let’s
	designate by $\frac{p_0}{q_0}$, $\frac{p}{q}$ two consecutive reductions of
	the continued fraction in which $x$ develops; and by $z$ the complete quotient
	that comes after, so we have
	$$\frac{p}{q} - x = \pm\frac{1}{q(qz + q_0)}$$
	Finally, stating
	$$f(p,q) = q^n f(\frac{p}{q}) = ap^n + bp^{n-1}q + \cdots + hq^n$$

	By the decomposition of $f(\frac{p}{q})$ in factors, with the help of the
	roots $x, x_1, \ldots, x_{n-1}$, we find
	$$\frac{p}{q} - x = \pm\frac{1}{q(qz+q_0)}
		= \frac{f(p,q)}{q^n\cdot a\left(\frac{p}{q}
			- x\right)\cdots\left(\frac{p}{q}-x_{n-1}\right)}$$
	However, in order to converge the reduction $\frac{p}{q}$ towards $x$, the
	quantity
	$$a\left(\frac{p}{q} - x_1\right)\cdots\left(\frac{p}{q} - x_{n-1}\right)$$
	also converges towards a finite limit,
	$$a(x-x_1)\cdots(x-x_{n-1})$$
	there is thus a certain maximum $A$ below which the limit will always remain.
	On the other hand, $f(p,q)$ is a whole number, at least equal to the unity,
	abstraction made of the sign. We have thus
	$$\frac{1}{q(qz+q_0)} > \frac{1}{Aq^n}$$
	of which
	$$z < Aq^{n-2} - \frac{q_0}{q} < Aq^{n-2}$$
	inequality subsists, even more so, when we substitute the complete $z$
	quotient of the integer part that it contains, namely the incomplete quotient
	$\mu$. The theorem we had in mind is hereby proved simply, without having to
	fall back on the Lagrange formula that we used earlier. We can, incidentally,
	apply a similar method to diverse development genres of which the irrational
	quantities are susceptible, and obtain that way interesting results.
\end{displayquote}

\paragraph{3.}
Let’s add some developments to what preceded. Still regarding a real root $x$ of
the equation, irreducible and with whole coefficients,
$$f(x)=ax^n + bx^{n-1} + \cdots + gx + h = 0$$
which, if $n>1$, will also have these other roots $x_1, \ldots, x_{n-1}$,
essentially irrational or imaginary and different to $x$. But let’s stop using,
to get closer and closer to $x$, reductions of continued fractions, and let’s
use any fraction $\frac{p}{q}$. In doing so, as here above,
$$f(p,q) = ap^n + bp^{n-1} q + \cdots + hq^n$$
we will be sure, if $n>1$, that the absolute value of the integer $f(p,q)$ is
at least equal to the unity, and we can again use the equation
$$\frac{p}{q} - x = \frac{f(p, q)}{q^n\cdot a\left(\frac{p}{q} - x_1\right)
		\cdots\left(\frac{p}{q}- x_{n-1}\right)}$$
this consequence, by designating by $A$ a certain fixed number, we must have
(abstraction made of the sign) for all fractions $\frac{p}{q}$ which we now use,
$$\frac{p}{q} - x > \frac{1}{Aq^n}$$
But the case of $n=1$ has to be, in turn, examined closely. This case could not
be presented right away; because if we assume the continued fraction, in which
we developed $x$, made up of an infinite number of terms, we had irrational $x$
and $n > 1$. But here, still assuming infinite numbers of successive fractions
$\frac{p}{q}$ of which $x$ is the limit, we need to consider the case $n = 1$
as possible.

To deal with this case, whether
$$f(x) = ax + b = 0$$
or
$$\frac{p}{q} - x = \frac{ap + bq}{aq}$$
Should it be possible that the numerator $ap + bq$ were zero, we would not be
able to draw any conclusion. But if we have ensured it by any way that we never
have
$$ap + bq = 0$$
namely
$$x = \frac{p}{q}$$
we could then argue that we have
$$\frac{p}{q} - x > \frac{1}{aq}$$
or even
$$\frac{p}{q} - x > \frac{1}{Aq}$$
writing $A$ instead of $a$. The general formula
$$\frac{p}{q} - x > \frac{1}{Aq^n}$$
will thus subsist even in the case of $n=1$.

That being said, granting that the quantity $x$ as such, forming infinite
numbers of fractions $\frac{p}{q}$ going towards $x$, but of which none are
exactly equal to $x$, we end up recognizing that the inequality
$$\frac{p}{q} - x > \frac{1}{Aq^n}$$
does not always occur. It has to be concluded that $x$ cannot be the root of
an equation of the degree $n$. Let’s also add that $x$ will not be root of any
equation of inferior degree $i$; because with
$$\frac{p}{q} - x > \frac{1}{Aq^n}$$
we will have, a fortiori,
$$\frac{p}{q} - x > \frac{1}{Aq^i}$$
for every exponent $i < n$. Therefore, having determined that the inequality
$$\frac{p}{q} - x > \frac{1}{Aq}$$
is in default, we will conclude that the value x is not rational. If the higher
inequality
$$\frac{p}{q} - x > \frac{1}{Aq^2}$$
is inadmissible, we will conclude that $x$ is not rational, nor even the root
of a second-degree equation; and so on. Finally, should it happen that the
inequality
$$\frac{p}{q} - x > \frac{1}{Aq^n}$$
generally takes place in default, any finite number we select for $n$, we will
be able to claim that $x$ is not even an algebraic irrational.

\paragraph{4.}
If, for example,
$$x = \frac{1}{l} + \frac{1}{l^{1 \cdot 2}} + \frac{1}{l^{1\cdot 2\cdot 3}}
	+ \cdots + \frac{1}{l^{1\cdot 2\cdot 3\cdots m}} + \cdots$$
$l$ being a whole number. Contributing to $\frac{p}{q}$ the rough value, but
essentially too small of $x$, that give the $m$’s the first terms of the series,
we will have
$$q = l^{1\cdot 2\cdots m}$$
and
$$x - \frac{p}{q} = \frac{1}{l^{1\cdot 2\cdots m(m + 1)}} + \cdots
	< \frac{2}{q^{m+1}}$$
If the exponent $m$ grows indefinitely, this last quantity decreases faster
than any fraction having a constant numerator and a proportionate denominator of
a given power of $q$, so that the inequality
$$\frac{p}{q} - x > \frac{1}{Aq^n}$$
always ends up being in default. From this, I conclude that $x$ is not rational,
nor even expressible by algebraic irrationals.

We will easily get to a similar conclusion for the much more general series
$$x = \frac{k_1}{l} + \frac{k_2}{l^{1\cdot 2}} + \frac{k_3}{l^{1\cdot 2\cdot 3}}
	+ \cdots + \frac{k_m}{l^{1\cdot 2\cdots m}} + \cdots$$
$k_1, k_2, k_3, \ldots, k_m, \ldots$ designating whole numbers, positive or
negative, of which the absolute value does not exceed a certain maximum $k$.
So if we take $l = 10$, and $k_1, k_2, \ldots$, freely, from $0$ until $9$,
we will form undefined decimal fractions of which the value will never be able
to expressed algebraically. I believe I remember that there is such a theorem,
in a letter from Goldbach to Euler; but I do not know whether its proof has
been provided.

Now let's say
$$x = \frac{1}{l} + \frac{1}{l^4} + \frac{1}{l^9} + \cdots + \frac{1}{l^{m^2}}
	+ \cdots$$
Still assuming for the rough value of $x$ the sum of the first $m$ terms of the
series, we will have
$$q = l^{m^2}$$
and
$$x - \frac{p}{q} = \frac{1}{l{(m + 1)}^2} + \cdots
	< \frac{2}{l^{2m+1}\cdot q}$$
This last quantity, the denominator containing the product of $q$ by $l^{2m+1}$
that grows with $m$, will end up decreasing faster than $\frac{1}{Aq}$. But the
only thing we can conclude from that is that $x$ is not rational.

As a new example, if
$$x = \frac{1}{l} + \frac{1}{l_1} + \frac{1}{l_2} + \cdots
	+ \frac{1}{l_{m-1}} + \cdots$$
$l$ being an integer, and every term has as a denominator the
$(n + 1)$\textsuperscript{th} power of the previous denominator, so that
$$l_m = l_{m-1}^{n+1}$$
It is obvious that $l_{m-1}$ will be a power of $l$, consequently, we will have
$$q = l_{m-1}$$
and
$$x - \frac{p}{q}=\frac{1}{l_m}+\cdots < \frac{2}{q^{n+1}}$$
$x$ will thus not be the root of any algebraic equation of a degree equal or
inferior to $n$.

We would also have other examples, considering the series
$$x = \frac{1}{l} + \frac{1}{u_1} + \frac{1}{u_1 l_2} + \cdots
	+ \frac{1}{u_1 l_2 \cdots l_{m-1}} + \cdots$$
where $l, l_1, l_2, \ldots, l_{m-1}, \ldots$ designate increasingly bigger whole
numbers. This circumstance where $l_m$ grows beyond any limit means that $x$
could no longer be rational. And if $l_m$ grows fast enough with the $m$ index,
we will be sure that $x$ is not even an algebraic irrational.

\paragraph{5.}
Finally, let us observe that, if we assume $a, b, \ldots, g, h$ as imaginary
and complex integers, then move towards a imaginary root $x$ of the equation
$$ax^n + bx^{n-1} + \cdots + gx + h = 0$$
with the help of fractions $\frac{p}{q}$ of which the two terms would also be
complex integers, we would again find the equation
$$\frac{p}{q} - x = \frac{f(p,q)}{q^n\cdot a\left(\frac{p}{q}
		- x_1\right)\cdots\left(\frac{p}{q}-x_{n-1}\right)}$$
and substituting the imaginary’s modules, of which we would easily deduce
$$\mod\left(\frac{p}{q}- x\right) > \frac{1}{A\pmod{q}^n}$$
which allows to extend the results of the imaginaries that we just developed for
real quantities. This way, we recognize, for example, that, whichever be the
complex integer $l$, the sum of the series
$$\frac{1}{l} + \frac{1}{l^{1 \cdot 2}} + \frac{1}{l^{1\cdot 2\cdot 3}}
	+ \cdots + \frac{1}{l^{1\cdot 2\cdot 3\cdots m}} + \cdots$$
is never algebraically expressible.
\end{document}