%\documentclass[authoryear,5p]{elsarticle}
\documentclass[authoryear,review,12pt]{elsarticle}
\bibliographystyle{elsarticle-harv}
\usepackage{graphicx}
\usepackage{amsmath,amsfonts}
\usepackage{lineno}
\linenumbers
\usepackage{subfigure}
\textwidth 6.75in
\oddsidemargin -0.15in
\evensidemargin -0.15in
\textheight 9in
\topmargin -0.5in
\newcommand{\ud}{\mathrm{d}}
\newcommand{\E}{\mathrm{E}}
\newcommand{\C}{\mathrm{Cov}}
\newcommand{\V}{\mathrm{Var}}
\newcommand{\cb}[1]{{\it \color{darkgreen} (#1)}}

%% Redefines the elsarticle footer
\makeatletter
\def\ps@pprintTitle{
\let\@oddhead\@empty
\let\@evenhead\@empty
\def\@oddfoot{\it \hfill\today}
\let\@evenfoot\@oddfoot}
\makeatother

\usepackage{endfloat} % Move floats to the end of the document
\renewcommand{\listoffigures}{} % but supress these lists
\renewcommand{\listoftables}{} % supress these lists
% compile without figures?
\newif\ifhavefigures
\havefigurestrue
\newif\ifnohavefigures % allows alternate condition
%\nohavefigurestrue     % turn on the alternate condition uncommenting this

\journal{\tiny Proceedings of the Royal Society B}
\begin{document}
\begin{frontmatter}
  \title{Early Warning Signals and the Prosecutor's Fallacy}
  \author[cpb]{Carl Boettiger\corref{cor1}}
  \ead{cboettig@ucdavis.edu}
  \author[esp]{Alan Hastings}
  %\author[info]{}
  %\author[davis]{}
  \cortext[cor1]{Corresponding author.}
  \address[cpb]{Center for Population Biology, 1 Shields Avenue, University of California, Davis, CA, 95616 United States.}
  \address[esp]{Department of Environmental Science and Policy, University of California, Davis}

 % \address[info]{ \\
  %              }

\begin{abstract}

Early warning signals have been proposed to forecast the possibility
of a critical transition, such as the eutrophication of a lake, the
collapse of a coral reef, or the end of a glacial period.  Because such
transitions often unfold on temporal and spatial scales that can be
difficult to approach by experimental manipulation, research has often
relied on historical observations as a source of natural experiments.
Here we examine a critical difference between selecting systems for
study based on the fact that we have observed a critical transition
and those systems for which we wish to forecast the approach of a
transition. This difference arises by conditionally selecting systems
known to experience a transition of some sort and failing to account
for the bias this introduces -- a statistical error often known as
the Prosecutor's Fallacy.  By analysing simulated systems that have
experienced transitions purely by chance, we reveal an elevated rate
of false positives in common warning signal statistics. We further
demonstrate a model-based approach that is less subject to this
bias than these more commonly used summary statistics.  We note that
experimental studies with replicates avoid this pitfall entirely.

\end{abstract}

  \begin{keyword}
early warning signals \sep tipping point \sep alternative stable states \sep likelihood methods
   \end{keyword}
 \end{frontmatter}

\section{Introduction}

\begin{quotation}
\noindent \emph{Mathematics \dots while assisting the trier of fact in the search of truth, must not cast a spell over him.}
-- California Supreme court, 1968.
\end{quotation}

\noindent In the case of \emph{People v. Collins} 1968, California Supreme
Court considered the evidence of an expert witness described by the
court as ``an instructor of mathematics at a state college'', which
concluded that the probability that a randomly selected individual
would match the description given by the victim would be less than 1 in
12 million~\citep{PeopleCollins1968}.  The prosecution had produced an
individual matching the prosecutor's detailed description, and convinced
by the mathematics, the lower courts had found him
guilty.

The prosecution has only observed that the probability of seeing the
evidence ($E$) they produced given a random innocent individual ($I$),
$P(E|I)$ is very small.  From this one cannot conclude that the individual
is indeed guilty, that is, that the probability the individual is innocent
given the evidence $P(I|E)$ is also very small. In a city with millions
of people, there might be several individuals who match the description
of the evidence.  Mathematically $P(E|I)$ need not equal $P(I|E)$,
instead, these expressions are related by Bayes theorem,

\begin{equation}
  P(E|I) = P(I|E) \frac{P(E)}{P(I)},
\end{equation}

$P(E) \ll 1$ and $P(I)\approx 1$ , so $P(E|I) \approx P(I|E) P(E)$, and consequently we cannot conclude that $P(I|E) \ll 1$ from $P(E|I) \ll 1$.
Realizing this mistake, the California Supreme Court reversed the decision, and
the case became a widely recognized example of the Prosecutor's
Fallacy~\citep{Thompson1987}.  Here we explore how a similar misconception
can arise from the use of historical data to evaluate methods for
detecting early warning signals of critical transitions.

%\subsection*{Catastrophic Transitions}

Catastrophic transitions or tipping points, where a complex system
shifts suddenly from one state to another, have been implicated in
a wide array of ecological and global climate systems such as lake
ecosystems~\citep{Carpenter2011}, coral reefs~\citep{Mumby2007},
savannah~\citep{Kefi2007}, fisheries~\citep{Berkes2006}, and tropical
forests~\citep{Hirota2011}.  Recent research has begun to identify
statistical patterns commonly associated with these sudden catastrophic
transitions which could be used as an \emph{early warning sign} to
identify an approaching tipping point, which might provide managers time
to react to and avert an undesirable state shift~\citep{Scheffer2009, Lenton2011}.
An array of statistical patterns associated with tipping
point phenomena has been suggested for the detection of early warning
signals associated with such sudden transitions.  Two of the most commonly
used are a pattern of increasing variance~\citep{Carpenter2006} and a
pattern of increasing autocorrelation~\citep{vanNes2007}, which have
been tested in both experimental manipulation~\citep{Drake2010,
Carpenter2011, Veraart2011, Dai2012} and historical
observations~\citep{Livina2007,Dakos2008,Lenton2012,Ditlevsen2010,Guttal2008,Thompson2010}.

\subsection*{Testing patterns on historical data}

Historical examples of sudden transitions taken from the paleo-climate
record provide an important way to test and evaluate potential
leading indicator methods, and have been widely used for this purpose
\citep{Livina2007,Dakos2008,Lenton2012,Ditlevsen2010,Guttal2008,Thompson2010}.
Similarly, it has been suggested that data gathered from ecological
systems such as lakes that were monitored before they experienced sudden
eutrophication, or grasslands subjected to overgrazing, could contain
data that could help reveal when similar systems are approaching a
tipping point~\citep{Carpenter2011}.

However, testing methods for early warning signals against historical
examples of transitions is susceptible to statistical mistakes that arise
from selecting data conditional on that data having already exhibited
a sudden transition.  A central tenant of early warning theory is that
the system in question is slowly approaching a tipping point that lies
some unknown distance away.  If nothing is done to remedy the situation,
this slow change will inevitably carry the system beyond the tipping
point, which introduces a sudden, rapid transition into an undesirable
state~\citep{Scheffer2009}. This process can be described mathematically
as a \emph{bifurcation}, in which a slowly changing parameter reaches
a critical value that causes the system stability to change.

Not all sudden transitions are caused by some ``guilty'' process slowly
driving the system over a tipping point -- the kind of process
that early warning signals are designed to detect.  Some systems may
experience such transitions purely by chance, leaving a stable state on
an extremely unlikely excursion that happens to stray to far from the
stable attractor~\citep[\emph{e.g.} ][consider this possibility in
transitions that arise from analyzing historical climate record]{Ditlevsen2010, Lenton2011}.
Like the evidence presented before the California Supreme
Court in 1968, the chance of observing such an ``innocent'' transition
a priori may be very small, but when selected from a historical record
of many possible transitions, this possibility can no longer be ignored.

Figure 1 shows a schematic illustrating critical transitions under
each of these scenarios.  In the left panel, the system experiences a
bifurcation  and should contain an early warning signal.  In the right
panel, a similar-looking trajectory emerges from a simulation of a stable
system which should not contain a warning signal.  While the simulation of
the bifurcation scenario shown on the left produces a similar transition
every time, the transition shown on the right is somewhat less likely,
occurring in only 1\% of simulations.

\begin{figure}
  \begin{center}
      \ifhavefigures
        \includegraphics[width=\textwidth]{Figure1}
      \fi
      \ifnohavefigures
        \hspace{\textwidth}
      \fi
      \label{fig:bifurcation}
  \end{center} \caption{\textbf{The Prosecutor's Fallacy}.
\textbf{(a)} Plot of the model functions shown in Eq~\eqref{master}
with parameters $a=180$, $K=500$, $e=.5$, and $h=200$.  When the death
rate is higher than the birth rate, the system dynamics drive the state
(population size) to smaller values.  When birth rate is higher, the
system moves right, as indicated by the arrows.\textbf{(b)} The potential
energy is given by the negative integral of $b(n)-d(n)$, shown in the
lower plot.   The potential function gives an intuitive picture of
the stability of a system by imagining the curve as a surface on which
a ball is free to bounce across -- wells correspond to stable points
and peaks to unstable points. While most trajectories remain near the
stable well, some transition out merely by chance.  An example of such
a trajectory is shown in the top panel, in which time increases along
the vertical axis. Though initially oscillating around the stable state,
a chance excursion carries it beyond the Allee threshold (vertical dotted line).
Such chance trajectories can produce the statistical patterns as observed
in true critical transitions seen in panel \textbf{(c)}: Early warning
signals  are aimed at detecting systems which are slowly moving towards
a tipping point or bifurcation, illustrated in the successive curves
(deteriorating and critical). Top panel: An example trajectory from a
simulation under this process shows the state of the system as the
potential moves towards the bifurcation point. The original position of the Allee threshold
is shown by the vertical dotted line (though it moves slightly as the parameter changes).
} \label{fig:1}
\end{figure}

% Consider an equation

\section{Methods and Results}
To investigate if early warning signals are vulnerable to this fallacy,
we simulate a system that is not driven towards a bifurcation such as
in Fig~ref{fig:1}(b).  This simulation approach allows us to determine whether
examining historical events is a valid way to test the utility of these
indicators.  We simulated 20,000 replicates of a stochastic individual-based
birth-death process with an Allee threshold~\citep{Courchamp2008}, which
arises from positive fitness effects at low densities.  Above the Allee
threshold the population returns to a positive equilibrium size, whereas
below the threshold the population decreases to zero. The model can be represented as a continuous time birth-death process where births and deaths are Poisson events which depend on the current density with rates given by

\begin{align}
    b(n) &= \frac{ K n^2}{n^2 + h^2}, \label{master} \\
    d(n) &= e n + a,
\end{align}

%The simulation uses the Gillespie algorithm to provide an exact implementation\footnote{ of the birth-death process,

%\begin{align}
%  \frac{dP(n,t)}{dt} &= b_{n-1} P(n-1,t) + d_{n+1}P(n+1,t) - (b_n+d_n) P(n,t)  \label{master}, \\
%    b_n &= \frac{K n^2}{n^2 + h^2}, \\
%    d_n &= e n + a,
%\end{align}

a model with a linear death rate and density-dependent birth rate
that drives the Allee effect at low densities and limits growth at high
densities.  In this model $n$ indicates the discrete number of individuals
in the population, $K$ indicates a carrying capacity as set by a limiting
resource, $e$ a per-capita death rate (the $e$ scaling term in the birth
equation allows the carrying capacity $K$ to correspond to a positive
equilibrium point), $a$ an additional mortality imposed on the population
such as harvest, $h$ is a parameter controlling at what population size
the addition of more individuals switches from conferring a positive
benefit on growth from Allee interactions $n < h$ to a negative impact
on growth due to increased competition, $n > h$.  The key feature of
this model is the alternate stable states introduced by this effect;
other functional forms for Eq.~\eqref{master} could serve equally
well for these simulations~\citep[see \emph{e.g.}][]{Scheffer2001}.
Though this system can be forced through a bifurcation by increasing
the death rate, in these simulations all parameters are held constant
and no bifurcation occurs.  Consequently we do not anticipate an early
warning signal of an approaching bifurcation.

The simulation starts from the positive equilibrium population size.
Though the chance of a transition across the Allee threshold in any
given time step is small, given enough time this system will eventually
experience such a rare event driving the population extinct.  We ran
each replicate over 50,000 time units, sampling the system every 50
time units.  In this time window 266 of the 1,000 replicates experience
population collapse.  To keep the examples of comparable sample size,
we focus on a section of the data 500 time points prior to the system
approaching the transition.

To test whether selecting systems that have experienced
spontaneous transitions could bias the analysis towards false
positive detection of early warning signals, (the Prosecutor's
Fallacy) we selected replicates conditional on having collapsed
in the simulations.  We then selected a window around each system
that ended just before the collapse, while the population values
were still above the Allee threshold.  For each replicate, we
calculated the most common early warning indicators, variance and
autocorrelation~\citep[\emph{e.g.}][]{Carpenter2006,Dakos2008,Scheffer2009},
around a moving window equal to half the length of that time series.

To test for the presence of a warning signal in these indicators we
computed values of Kendall's $\tau$ for both indicators for each of
the 266 replicates.  Kendall's $\tau$ is a non-parametric measure
of rank correlation frequently used to identify an increasing trend
($\tau > 0 )$ in early warning signals~\citep{Dakos2008, Dakos2011},
defined as $\tau \def \tfrac{(\text{number of concordant pairs}) -
(\text{number of discordant pairs})}{\frac{1}{2} n (n-1) }$ in $n$
observations.  \footnote{A pair of observations $(x_i,y_i)$ and $(x_j,
y_j)$ are concordant if $x_i > x_j$ and $y_i > y_j$ or $x_i < x_j$ and
$y_i < y_j$ and discordant otherwise; equalities excepted.}  $\tau$ takes
values in $(-1, 1)$.  The distribution of $\tau$ values observed across
these replicates is shown in Figure~\ref{fig:indicator}.  We compare
the distribution of $\tau$ from all the simulations to the distribution
conditioned on experiencing a chance transition to the alternative
stable state.  To avoid an effect of sample size the time series are
all chosen to be the same length.

To demonstrate the effect we observe is not unique to models with Allee
effects, we provide an example of the effect arising in a discrete-time
model with two non-zero stable states adapted from~\citep{May1977},
\begin{equation}
X_{t+1} = X_t \exp\left( r \left(1 - \frac{ X_t }{ K } \right) - \frac{ a * X_t ^ {Q - 1} }{X_t ^ Q + H ^ Q} \right). \label{May1977}
\end{equation}
which combines a logistic growth model with a saturating predator response (See~\citet{May1977} for detailed discussion), shown in Figure~\ref{fig:may}.  Code to replicate the analysis can be found at \\
\verb|https://github.com/cboettig/earlywarning/tree/prosecutor/|.

\begin{figure}
  \begin{center}
    \ifhavefigures
    \includegraphics[width=6in]{Figure2}
    \fi
     \ifnohavefigures
     \hspace{6in}
    \fi
 \end{center}
  \caption{The distribution of the correlation statistic $\tau$ for two
  early warning indicators (variance, autocorrelation) on replicates
  conditionally selected for having collapsed by chance in simulations
  is shown in grey bars.  Solid lines indicate the estimated density of
  the statistic from a random sample of the simulations (not conditional
  on observing a transition). Positive values of $\tau$ correspond to
  a pattern of an indicator increasing with time; typically taken as
  evidence that a system is approaching a critical transition.  In these
  simulations, the pattern arises instead from the Prosecutor's fallacy
  of conditional selection.}
  \label{fig:indicator}
\end{figure}

\begin{figure}
  \begin{center}
    \ifhavefigures
    \includegraphics[width=6in]{Figure3}
    \fi
    \ifnohavefigures
     \hspace{6in}
    \fi
  \end{center}
  \caption{The identical analysis from Figure~\ref{fig:indicator} is shown for the model in Eq~\eqref{May1977} using parameters $r = 0.75$, $K=10$, $a=1.7$, $Q=3$, and $H=1$. A similar statistical bias, particularly towards positive values of $tau$ occurs in this model as well.}
  \label{fig:may}
\end{figure}

For each of these replicates we also take a model-based
approach, estimating parameters for an approximate linear
model of the system approaching a saddle node bifurcation, as
described by~\citet{Boettiger2012b},

\begin{equation}
\ud X = \sqrt{ r_t } (\phi(r_t) - X_t)\ud t + \sigma\sqrt{\phi(r_t) } \ud B_t \label{LSN}
\end{equation}

In this model, the parameter $m$
describes the approach towards the saddle-node bifurcation.  Estimates
$m < 0 $ are expected in systems approaching a bifurcation, while for
stable systems $m$ should be approximately zero. None of the estimates
across the 266 simulations differed from zero in our study, hence the
model-based estimation shows no evidence of bias on data that has been
selected conditional on collapse.

\section{Discussion}

The attempts to detect early warning signs for critical transitions
are based on the concept of a deteriorating environment as embodied
in a changing parameter~\cite{Scheffer2009}, which is a different kind
of transition than one which is driven instead by stochasticity in an
environment which is otherwise constant and exhibiting no directional
change. When trying to use historical data to understand critical
transitions we often do not know which category, changing environment
or simply chance, an observed large change falls into.

We have shown here that systems which undergo rare sudden transitions
due to chance look statistically different from their counterparts that
do not, even though they are driven by the same stochastic process.
In particular, such conditionally selected examples are more likely to
show signs associated with an early warning of an approaching tipping
point, such as increasing variance or increasing autocorrelation,
as measured by Kendall's $\tau$.  This increases the risk of false
positives -- cases in which a warning signal being tested appears to
have successfully detected an underlying change in the system leading
to a tipping point, when in fact the example comes instead from a stable
system with no underlying change in parameters. Figure 2 shows that many
of the chance crashes show values of $\tau$ that are significantly larger
than those observed in the otherwise identical replicates that did not
experience a chance transition, thus ``detecting'' an underlying change
in the system dynamics that is not in fact present.

\subsection{Chance transitions are false positives for early warning signals}

It seems tempting to argue that this bias towards positive detection
in historical examples is not problematic -- each of these systems did
indeed collapse, so the increased probability of exhibiting warning
signals could be taken as a successful detection.  Unfortunately this is
not the case. At the moment the forecast is made, these systems are not
likely to transition, since they experience a strong pull towards the
original stable state.  A closer look at the patterns involved shows why
common indicators such as autocorrelation and variance can be misleading.

As the system gets farther from its stable point, it it more likely to
draw a random step that returns it towards the stable point. Despite
this, there is always some probability that it will move further still,
so systems that do cross the tipping point must do so rather quickly by
a string of events.  This pattern, clearly visible before the crashes
in each of the examples in Figure 1, produces a string of observations
that appear more highly autocorrelated (if we are sampling the system
frequently enough to catch the excursion at all) than we observe in the
rest of the fluctuations around the equilibrium.  Yet this autocorrelation
comes from a chance trajectory moving quickly \emph{away} from the
stable state, not from the critical slowing down pattern in the return
times to the stable state which precede a saddle-node bifurcation and
motivate the early warning signal.

This longer than expected excursion results in a higher than expected
variance in that window as well. Both variance and autocorrelation are
calculated using a moving window over the time-series, which allows
the method to pick out a pattern of change as the window moves along
the sequence. If this chance excursion that precedes the crash happens
to fill a significant part of the moving window, the resulting pattern
will tend to show an increase in autocorrelation or variance.  If the
chance excursion is relatively rapid compared to the frequency at which
the system is observed (spacing of the data) or the width of the moving
window, the excursion may not significantly alter the general pattern.
In this way, some of the events in which a crash is observed will
appear to present these statistical patterns of increased variance
or autocorrelation without being harbingers of approaching critical
transitions.

\subsection{ The truncation of observations }

If we had a complete knowledge of the system dynamics, then we could
eliminate the bias we observe here since the bias arises from the
transient branch of the trajectory that crosses the threshold, and
if the system were truncated at the minimum of the potential then the
effect we emphasize here would not appear.  But, it is not possible to
truncate the system in any practical application. The precise location of
the minimum of the potential which is the location of the deterministic
equilibrium is unknown. Moreover, under the hypothesis that the system is
approaching a critical transition, the location of the minimum potential
moves so it cannot easily be estimated by previous observations, (see
Figure 1c where the equilibrium point moves in the direction of the
transition). Thus it is neither practical nor desirable to suggest that
historical time series can be used by following a simple truncation rule
that avoids the branch of a trajectory crossing the threshold to another
basin of attraction.  Exactly where a particular study will choose
to truncate such a trajectory will necessarily be arbitrary without an
underlying model of the process.  Frequently this is done by removing
the very steep, monotonic branch of the trajectory expected once the
system crosses the unstable threshold.  Such an approach corresponds
with our choice of termination and produces the bias we discuss here.

The examples of Figure 1, though only single replicates, may be useful
in illustrating these issues.  Figure 1c, top panel shows a sample
trajectory of a system with a parameter shift, while 1b shows a trajectory
without a shift.  Both trajectories become more highly autocorrelated
and higher variance near the end of the time series (time increases
on the y axis in Figure 1).  The part of the time series following
the critical transition shows a fast and monotonic trajectory to the
unstable trajectory, and would usually be excluded by an analysis for
warning signals in advance of the transition.  No such clear pattern
exists prior to the transition in Figure 1b.   An alternative proposal
to terminate the trajectory in panel B earlier would also risk decreasing
the signal seen in panel c, and would be inconsistent with the application
of warning signals in the forecasting context, where there would be no
such truncation.

\subsection{Comparing to the model-based method}

In our numerical experiment, the model-based estimate of early warning
signals appears more robust than the summary statistics, producing
the same estimates on both the conditionally selected replicates as on
a random sample of the replicates.  This is a consequence of the more
rigid specifications that come with a model-based approach -- the pattern
expected is less general than any increase in variance or autocorrelation,
but instead must be one that matches its approximation of the saddle-node
bifurcation. This observation highlights the difference between the
pattern driving the false positive trends in increasing variance and
increasing autocorrelation and the pattern anticipated in the saddle-node
model. This should not however be taken as evidence that the model-based
approach is immune to the bias of the Prosecutor's Fallacy.

\subsection{Importance of experimental approaches} The problem we
highlight ultimately stems from the difficulty of having only a single
realization with which to examine a complex problem.  The only way
to deal with this problem embodied is through replication, as can
be done in an experimental system in laboratory manipulations such
as~\citet{Drake2010, Veraart2011, Dai2012} and at the scale of whole
lake ecosystems in~\citet{Carpenter2011}.  Experimental procedures avoid
the hazard of the Prosecutor's fallacy by generating a complete sample
of replicates, rather than selecting a subset of cases from some larger
historical sample.

 \section{Acknowledgments}
This research was supported by funding from NSF Grant EF 0742674 to AH
and a Computational Sciences Graduate Fellowship from the Department
of Energy grant DE-FG02-97ER25308 and NERSC Supercomputing grant
DE-AC02-05CH11231 to CB. The authors thank M. Baskett, T.A. Perkins
and N. Ross for helpful comments on earlier drafts of the manuscript,
and also P. Ditlevsen and an anonymous reviewer for their comments.

\begin{thebibliography}{25}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\expandafter\ifx\csname url\endcsname\relax
  \def\url#1{\texttt{#1}}\fi
\expandafter\ifx\csname urlprefix\endcsname\relax\def\urlprefix{URL }\fi

\bibitem[{Berkes et~al.(2006)Berkes, Hughes, Steneck, Wilson, Bellwood, Crona,
  Folke, Gunderson, Leslie, Norberg, Nystr\"{o}m, Olsson, Osterblom, Scheffer,
  and Worm}]{Berkes2006}
Berkes, F., Hughes, T.~P., Steneck, R.~S., Wilson, J.~A., Bellwood, D.~R.,
  Crona, B., Folke, C., Gunderson, L.~H., Leslie, H.~M., Norberg, J.,
  Nystr\"{o}m, M., Olsson, P., Osterblom, H., Scheffer, M., Worm, B., Mar.
  2006. {Ecology. Globalization, roving bandits, and marine resources.} Science
  (New York, N.Y.) 311~(5767), 1557--8.

\bibitem[{Boettiger and Hastings(2012)}]{Boettiger2012b}
Boettiger, C., Hastings, A., May 2012. {Quantifying limits to detection of
  early warning for critical transitions}. Journal of The Royal Society
  Interface 9~(75), 2527 -- 2539.

\bibitem[{Carpenter(2011)}]{Carpenter2011}
Carpenter, J., Feb. 2011. {May the Best Analyst Win}. Science 331~(6018),
  698--699.

\bibitem[{Carpenter and Brock(2006)}]{Carpenter2006}
Carpenter, S.~R., Brock, W.~A., 2006. {Rising variance: a leading indicator of
  ecological transition.} Ecology letters 9~(3), 311--8.

\bibitem[{Courchamp et~al.(2008)Courchamp, Berec, and
  Gascoigne}]{Courchamp2008}
Courchamp, F., Berec, L., Gascoigne, J., 2008. {Allee Effects in Ecology and
  Conservation}. Oxford University Press, USA.

\bibitem[{Dai et~al.(2012)Dai, Vorselen, Korolev, and Gore}]{Dai2012}
Dai, L., Vorselen, D., Korolev, K.~S., Gore, J., May 2012. {Generic Indicators
  for Loss of Resilience Before a Tipping Point Leading to Population
  Collapse}. Science 336~(6085), 1175--1177.

\bibitem[{Dakos et~al.(2011)Dakos, K\'{e}fi, Rietkerk, Nes, and
  Scheffer}]{Dakos2011}
Dakos, V., K\'{e}fi, S., Rietkerk, M., Nes, E. H.~V., Scheffer, M., 2011.
  {Slowing Down in Spatially Patterned Ecosystems at the Brink of Collapse}.
  The American Naturalist.

\bibitem[{Dakos et~al.(2008)Dakos, Scheffer, van Nes, Brovkin, Petoukhov, and
  Held}]{Dakos2008}
Dakos, V., Scheffer, M., van Nes, E.~H., Brovkin, V., Petoukhov, V., Held, H.,
  Sep. 2008. {Slowing down as an early warning signal for abrupt climate
  change.} Proceedings of the National Academy of Sciences 105~(38), 14308--12.

\bibitem[{Ditlevsen and Johnsen(2010)}]{Ditlevsen2010}
Ditlevsen, P.~D., Johnsen, S.~J., Oct. 2010. {Tipping points: Early warning and
  wishful thinking}. Geophysical Research Letters 37~(19), 2--5.

\bibitem[{Drake and Griffen(2010)}]{Drake2010}
Drake, J.~M., Griffen, B.~D., Sep. 2010. {Early warning signals of extinction
  in deteriorating environments}. Nature 467~(7314), 456--459.

\bibitem[{Guttal and Jayaprakash(2008)}]{Guttal2008}
Guttal, V., Jayaprakash, C., Dec. 2008. {Spatial variance and spatial skewness:
  leading indicators of regime shifts in spatial ecological systems}.
  Theoretical Ecology 2~(1), 3--12.

\bibitem[{Hirota et~al.(2011)Hirota, Holmgren, {Van Nes}, and
  Scheffer}]{Hirota2011}
Hirota, M., Holmgren, M., {Van Nes}, E.~H., Scheffer, M., Oct. 2011. {Global
  Resilience of Tropical Forest and Savanna to Critical Transitions}. Science
  334~(6053), 232--235.

\bibitem[{K\'{e}fi et~al.(2007)K\'{e}fi, Rietkerk, Alados, Pueyo, Papanastasis,
  Elaich, and de~Ruiter}]{Kefi2007}
K\'{e}fi, S., Rietkerk, M., Alados, C.~L., Pueyo, Y., Papanastasis, V.~P.,
  Elaich, A., de~Ruiter, P.~C., Sep. 2007. {Spatial vegetation patterns and
  imminent desertification in Mediterranean arid ecosystems.} Nature
  449~(7159), 213--7.

\bibitem[{Lenton(2011)}]{Lenton2011}
Lenton, T.~M., Jun. 2011. {Early warning of climate tipping points}. Nature
  Climate Change 1~(4), 201--209.

\bibitem[{Lenton et~al.(2012)Lenton, Livina, and Dakos}]{Lenton2012}
Lenton, T.~M., Livina, V., Dakos, V., 2012. {Early warning of climate tipping
  points from critical slowing down: comparing methods to improve robustness}.
  Philosophical Transactions of The Royal Society A (in press).

\bibitem[{Livina and Lenton(2007)}]{Livina2007}
Livina, V.~N., Lenton, T.~M., Feb. 2007. {A modified method for detecting
  incipient bifurcations in a dynamical system}. Geophysical Research Letters
  34~(3), 1--5.

\bibitem[{May(1977)}]{May1977}
May, R.~M., Oct. 1977. {Thresholds and breakpoints in ecosystems with a
  multiplicity of stable states}. Nature 269~(5628), 471--477.

\bibitem[{Mumby et~al.(2007)Mumby, Hastings, and Edwards}]{Mumby2007}
Mumby, P.~J., Hastings, A., Edwards, H.~J., Nov. 2007. {Thresholds and the
  resilience of Caribbean coral reefs.} Nature 450~(7166), 98--101.

\bibitem[{Scheffer et~al.(2009)Scheffer, Bascompte, Brock, Brovkin, Carpenter,
  Dakos, Held, van Nes, Rietkerk, and Sugihara}]{Scheffer2009}
Scheffer, M., Bascompte, J., Brock, W.~A., Brovkin, V., Carpenter, S.~R.,
  Dakos, V., Held, H., van Nes, E.~H., Rietkerk, M., Sugihara, G., 2009.
  {Early-warning signals for critical transitions.} Nature 461~(7260), 53--9.

\bibitem[{Scheffer et~al.(2001)Scheffer, Carpenter, Foley, Folke, and
  Walker}]{Scheffer2001}
Scheffer, M., Carpenter, S.~R., Foley, J.~A., Folke, C., Walker, B., Oct. 2001.
  {Catastrophic shifts in ecosystems.} Nature 413~(6856), 591--6.

\bibitem[{{Supreme Court}(1968)}]{PeopleCollins1968}
{Supreme Court}, C., 1968. {People v. Collins}.

\bibitem[{Thompson and Sieber(2010)}]{Thompson2010}
Thompson, J. M.~T., Sieber, J., Dec. 2010. {Climate tipping as a noisy
  bifurcation: a predictive technique}. IMA Journal of Applied Mathematics
  76~(1), 27--46.

\bibitem[{Thompson and Schumann(1987)}]{Thompson1987}
Thompson, W.~C., Schumann, E.~L., 1987. {Interpretation of statistical evidence
  in criminal trials: The prosecutor's fallacy and the defense attorney's
  fallacy.} Law and Human Behavior 11~(3), 167--187.

\bibitem[{van Nes and Scheffer(2007)}]{vanNes2007}
van Nes, E.~H., Scheffer, M., Jun. 2007. {Slow recovery from perturbations as a
  generic indicator of a nearby catastrophic shift.} The American naturalist
  169~(6), 738--47.

\bibitem[{Veraart et~al.(2011)Veraart, Faassen, Dakos, van Nes, L\"{u}rling,
  and Scheffer}]{Veraart2011}
Veraart, A.~J., Faassen, E.~J., Dakos, V., van Nes, E.~H., L\"{u}rling, M.,
  Scheffer, M., Dec. 2011. {Recovery rates reflect distance to a tipping point
  in a living system}. Nature, 2--5.

\end{thebibliography}
 \end{document}


\documentclass{article}
\usepackage{amsthm}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{graphicx}
\usepackage{wrapfig}
\usepackage{tabu}
\usepackage{float}
\usepackage{hyperref}
\graphicspath {{Scribe_Images/}}

\makeatletter
\def\lecture{\@ifnextchar[{\@lectureWith}{\@lectureWithout}}
\def\@lectureWith[#1]{\medbreak\refstepcounter{section}%
  \renewcommand{\leftmark}{Lecture \thesection}
  \noindent{\addcontentsline{toc}{section}{Lecture \thesection: #1\@addpunct{.}}%
  \sectionfont Lecture \thesection. #1\@addpunct{.}}\medbreak}
\def\@lectureWithout{\medbreak\refstepcounter{section}%
  \renewcommand{\leftmark}{Lecture \thesection}
  \noindent{\addcontentsline{toc}{section}{Lecture \thesection.}%
  \sectionfont Lecture \thesection.}\medbreak}
\makeatother
\newcommand{\pardif}[2]{\frac{\delta#1}{\delta#2}}
\newcommand{\secpardif}[2]{\frac{\delta^{2}#1}{\delta#2^{2}}}
\newcommand{\thermdif}[4]{\frac{\delta#1}{\delta#2}\vert_{#3,#4}}
\newcommand{\secthermdif}[4]{\frac{\delta^{2}#1}{\delta#2^{2}}\vert_{#3,#4}}
\newcommand{\dif}[2]{\frac{d#1}{d#2}}
\newcommand{\bltz}{k_{B}}
\newcommand{\sumser}[2]{\sum\limits_{#1}^{#2}}
\newcommand{\sigi}{\sigma_{i}}
\newcommand{\sigj}{\sigma_{j}}

\title{Second Order Phase Transitions}
\author{Paolo Bedaque}
\begin{document}
\maketitle

As expected from from our previous discussion of First Order Phase Transitions, \textbf{Second Order Phase Transitions} (or "Continous Transitions") are transitions where the first derivative of the free energy is now continuous and the second derivative is discontinuous.  The archetypal example of this system is our old friend: The Two-Dimensional Ising Model.

\section{Phase Transitions of the 2-D Ising Model}

In previous lectures we expanded the two dimensional Ising model in both low temperature and high temperature situations in order to determine the spin correlation ($<\sigma_{0}\sigma_{0+R}>$).  At low temperatures we found that

$$<\sigma_{0},\sigma_{0+R}>\approx1+...., <\sigma_{i}>\neq0$$

Implying a high correlation between spins at arbitrary distances apart.  At high temperatures, we found that the correlator had a decayed exponential:

$$<\sigma_{0},\sigma_{0+R}>\approx e^{-R\ln{\frac{1}{\beta}}}, <\sigma_{i}>\neq0$$

Implying an exponentially decaying correlation as the distance between two spin sites increases.  The physical consequence of these two correlations is that a 2-D Ising magnetic system will, at high temperatures, consist of random spin orientations until the temperature lowers enough for the spins to align with the external magnetic field.  This can be summed up in a plot of the average spin vs the system's temperature:

\begin{figure}[H]
	\centering
	\includegraphics[width=0.5\linewidth]{Scribe_Images/spin_symmetry.png}
\end{figure}

The temperature at which the system spontaneously magnetizes is called the \textbf{critical temperature}.  Now, looking at this graph it is plain that at low temperatures the spins in the system tend to align either up or down.  This implies that the average spin of the system be non-zero, $<\sigma_{i}>\neq0$.  But this introduces an apparent paradox: how can the average spin be non-zero if the system is symmetrical?  Put another way, for every spin configuration there is another symmetrical configuration which is equally probable (e.g. the probability of $<\uparrow\uparrow\uparrow>$ is the same as $<\downarrow\downarrow\downarrow>$, the probability of $<\uparrow\downarrow\uparrow>$ is the same as $<\downarrow\uparrow\downarrow>$, etc...)  Shown mathematically:

$$\mathcal{H}_{2D\ Ising}=-J\sum_{links}\sigma_{i}\sigma_{j}-h\sum_{i}\sigma_{i}$$

Therefore the partition function is

$$Z=\sum e^{\beta J\sum_{links}\sigi\sigj+\beta h\sum_{i}\sigi}$$

Using the partition function we can easily show the average sum of the system's spins:

$$\frac{1}{\beta}\pardif{\ln{Z}}{h}=\frac{1}{\beta Z}\sum_{\sigma}(\beta\sum_{i}\sigi)e^{\beta J\sum\sigi\sigj+\beta h\sum_{i}\sigi}=<\sum_{i}\sigi>$$

From here, we want to calculate the average spin at a single lattice site.  This piece of information tells us whether, on average, the system is magnetized ($<\sigi>\neq0$) or unmagnetized ($<\sigi>=0$).  The average of 'up' ($<\sigma>$) spins is expressed as

$$<\sigi>=\frac{1}{Z}\sum_{\sigma}e^{-\beta\mathcal{H}[\sigma]}\sigi$$

To find the average of 'down' ($<-\sigma>$) spins, we can perform a change of variable by setting $\sigma^{'}=-\sigma$.  Plugging this into our expression, we find that this average is equal to the previous 'up' spin average:

\begin{equation*}
\begin{split}
	<\sigi> & =\frac{1}{Z}\sum_{\sigma}e^{-\beta\mathcal{H}[\sigma]}\sigi \\
			   & =\frac{1}{Z}\sum_{\sigma}e^{-\beta\mathcal{H}[-\sigma]}\sigi\ \ (1)\\
			   & =-\frac{1}{Z}\sum_{\sigma^{'}}e^{-\beta\mathcal{H}[\sigma^{'}]}\sigi^{'}\ (2)
\end{split}
\end{equation*}

This is true because $\mathcal{-\sigma}=\mathcal{\sigma}$.  i.e. The hamiltonian has $\mathbb{Z}_{2}$ symmetry.  The consequence of this symmetry is the fact that (1) and (2) above are equal, which implies that $<\sigi>=0$.  So, since the average spin of any single site is 0, clearly we should have see no magnetization at low temperatures!  To resolve this paradox, we must consider the physical system of such a ferromagnet.  If you take a finite amount of material and cool it down, one direction will dominate the spins.  But by chance spins can flip and one enough change at once the entire system flips its spin state:

\begin{wrapfigure}{l}{0.25\textwidth}
	\centering
	\includegraphics[width=1\linewidth]{Scribe_Images/spin_state_time.png}
\end{wrapfigure}

For a large system, a large number of spins need to flip at once in order to affect the state of the entire system.  If we inspect the limit where time approaches infinity, the state will flip and the average spin state will be zero.  However, the time it takes for the system to change from its initial spin-aligned state to its corresponding spin symmetrical state (also known as the system's \textbf{correlation time}) may be much longer than we could possibly observe:

\begin{wrapfigure}{r}{0.25\textwidth}
	\centering
	\includegraphics[width=1\linewidth]{Scribe_Images/Correlation_Time.png}
\end{wrapfigure}

For some systems, the correlation time would be longer than our sun's life span or the heat death of the universe, so for all intents and purposes these systems remain in the magnetized (spin-aligned) state.

In order to resolve this situation in our statistical mechanics formalism, we need to employ a technical trick.

$$m=\frac{1}{V}<\sum_{i}\sigi>=\lim\limits_{h\rightarrow0}\lim\limits_{V\rightarrow0}<\sigi>_{h}$$

Here we are taking a hypothetical system and applying an external field to it to align the individual spins within the system.  We then allow the system's volume to approach infinity, which allows the spin correlation time to become infinite.  This now allows us to take the external field to zero, removing the external magnetic field while keeping the spontaneous magnetization of the system frozen.

In summary, because $\mathcal{H}[-\sigma]=\mathcal{H}[\sigma]:\mathbb{Z}_{2}\ Symmetry$, the system at low temperatures no longer

%add section on spontaneous symmetry breaking

\section{Mean Field Approximation}

Using our new knowledge of the behavior of the average system spin we can approximate highly complex models using a simplified model which relies on the average spin (or average analogous interaction).  This is the core idea of behind the mean field approximation.  However, before we delve deep into this new method, let's start with a warm up problem to illustrate the why this method is so powerful.

First, let's look at a trivial system which we have seen before:  A system with an external magnetic field and no spin coupling.  For such a system we have the hamiltonian:

$$\mathcal{H}_{0}=-h\sum_{i}\sigi$$

The partition function for this system is therefore

$$Z_{0}=\sum_{\sigi}e^{-\beta\mathcal{H}_{0}}=\sum_{\sigma_{1}=\pm}e^{\beta h\sigma_{1}}\sum_{\sigma_{2}=\pm}e^{\beta h\sigma_{2}}...\sum_{\sigma_{v}=\pm}e^{\beta h\sigma_{v}}=(2\cosh{\beta h})^{v}=2^{v}\cosh{\beta h}^{v}$$

Now that we have found Z we can find the magnetization, m:

$$m\frac{1}{\beta}\pardif{\ln{Z_{0}}}{h}=\tanh{\beta h}$$

\begin{figure}[h]
	\centering
	\includegraphics[width=.75\linewidth]{Scribe_Images/mag_curve.png}
\end{figure}

Now that we know how this the magnetization of this simple system behaves, let's try to fit the Ising model to this system.  As before,

$$H=-J\sum_{links}\sigi\sigj-h\sum_{i}\sigi=\sum_{sites}\sigi[-\frac{J}{2}\sum_{Nearest\ Neighbors}\sigj-h]$$

In this final form, we see that we have separated the hamiltonian into two parts: the simple sum of the individual spin sites multiplied by the sum over the nearest neighbor interactions in the presence of the external field.  We can call our second term the \textit{effective magnetic field}, or $h_{eff}$, as it represents the external field taking into account the nearest neighbor interactions.

Simplifying using this new definition, we find that the we can express our Ising model system in the same form as our trivial warm up system:

$$H=-h_{eff}\sum_{sites}\sigi$$

We can now apply the same techniques as the warm up to approximate the properties of our system.

$$H\cong-h_{eff}\sum_{sites}\sigi=\sum_{sites}[-\frac{J}{2}(4)<\sigj>-h]$$

The factor of 4 is introduced because every site within our two-dimensional lattice has 4 nearest neighbors which are contributing to spin-spin interactions.  This approximation for the hamiltonian is known as an \textit{uncontrolled approximation} because it becomes more accurate as the number of dimensions in the system increases.

Solving for the effective magnetic field on our system yields:

$$h_{eff}=h+2J<\sigj>=h+2J[\tanh{\beta h_{eff}}]$$

This expression is known as the \textit{mean field equation} for the effective magnetic field.

From our previous warm up model, we already know the expression for the system's magnetization:

$$\boxed{m=\tanh{\beta h_{eff}}}$$

So now we can graphically display the system's behavior as we vary the temperature and effective field.  For the case where the external field is 0:

\begin{figure}[h]
	\centering
	\includegraphics[width=0.75\linewidth]{Scribe_Images/tcrit_curve.png}
\end{figure}

We see that at high temperatures, the mean field equation has only one solution: $h_{eff}=0$.  This is expected, as at high temperatures we should see no magnetization at all.  At low temperatures there are three intersections: one positive, one negative, and one at the origin.  The positive and negative intersections correspond to the states where the magnetization is positive and negative, respectively.  The origin intersection state is unstable.  While it is theoretically possible for the system to be in the zero magnetization state at low temperatures, any slight perturbation of the system will find it rapidly transitioning to either the positive or negative magnetization state.

Using the information we now have about the system, we can find an expression for the critical temperature.  The critical temperature will be the temperature for which the slope of mean field equation most closely approximates the slope of the linear $h_{eff}$ expression.  Therefore

$$\frac{d}{dh_{eff}}[2J\tanh{\beta_{c}h_{eff}}=1$$

In order to determine the functions behavior when $h_{eff}$ is close to but not equal to 0, we take the Taylor expansion around $h_{eff}=0$ to yield $2J\beta_{c}=1$, which is equivalent to

$$\frac{2J}{\bltz T_{c}}=1$$

Giving our final expression of $\bltz T_{c}=2J$, or

$$\boxed{T_{c}=\frac{2J}{\bltz}}$$

\end{document}


\subsection*{Attendance}

Part of your grade will be based on attendance. Since the course is on zoom, I have a full audit of the second each of you enters and exits the \qu{classroom}. Your score will be computed automatically as the proportion of seconds you attended class divided by the total number of class seconds.


%% -*- coding:utf-8 -*-
\documentclass[14pt,a4paper]{article}

\usepackage[utf8]{inputenc}
\usepackage{minted}
\usepackage{longtable}
\usepackage{hyperref}
\hypersetup{
  pdftex,
  allcolors=blue,
  bookmarksnumbered=true,
  bookmarksopen=true,
  bookmarksopenlevel=1,
  colorlinks=true,
  pdfstartview=Fit,
  pdfpagemode=UseOutlines,
  pdfpagelayout=TwoPageRight,
  pdftitle={Inside STL sort algorithm},
  pdfsubject={STL sort},
  pdfauthor={Ivan Murashkо},
  pdfkeywords={STL, C++, quick sort}
}

\title{Inside STL sort algorithm}
\author{Ivan Murashko}
\date{}
\begin{document}

\maketitle

\section*{Introduction}
One of the trickiest thing in C++ is an error catch. Most part of errors
are located on the application level and you can be quite sure that
the library, especially the well known STL, is error free. But what
should you do if your application code is trivial i.e. it seems to be
error free but you get a SIGSEGV inside a system library. Most
probably you just fell into a situation that is described in the
article.

Note: All examples in the article use gcc 7.4.0 from the docker image
gcc:7:
\begin{minted}{shell}
docker run gcc:7 gcc --version
gcc (GCC) 7.4.0
\end{minted}

The source code for examples can be found on the article git
repository \cite{github:cppsort_ivanmurashko} in the folder
\textbf{cppsort/src}.

\section{Example}
Lets look at the following code:
\inputminted{c++}{./src/sort.cpp}
What output does it produce? One can expect the following one:
\begin{minted}{shell}
$ docker run -v "$PWD":/tmp/test -w /tmp/test gcc:7 ./src/sort
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
$
\end{minted}
i.e. the original array has to be displayed. In reality the following
output will be produced by gcc 7.4.0
\begin{minted}{shell}
$ docker run -v "$PWD":/tmp/test -w /tmp/test gcc:7 ./src/sort
0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
$
\end{minted}
As one can see the array is broken. How it can happen if every
function in the code snapshot seems to be correct? The answer is
below.

\section{Inside STL sort}

The STL library states the following requirements for \textbf{std::sort}
\cite{ISO:2013:IIP}(pp. 897-898): the compare function must follow so
called strict weak ordering requirement. Especially it
says that
\begin{minted}{c++}
comp(x, x) == false;
\end{minted}
The requirement is violated
by the $\le$ operand used in our example.

Why the requirement is so important? The answer is in the algorithm
that is used for sorting. STL uses an optimized version of Quicksort algorithm
\cite{wiki:quicksort}.

The code that produces the problem can be found via the following command
\begin{minted}{shell}
docker run gcc:7 tail -n +1893 \
/usr/local/include/c++/7.4.0/bits/stl_algo.h | head -n 21
\end{minted}
and looks as follow
\begin{minted}{c++}
  /// This is a helper function...
  template<typename _RandomAccessIterator, typename _Compare>
    _RandomAccessIterator
    __unguarded_partition(_RandomAccessIterator __first,
                          _RandomAccessIterator __last,
                          _RandomAccessIterator __pivot, _Compare __comp)
    {
      while (true)
        {
          while (__comp(__first, __pivot))
            ++__first;
          --__last;
          while (__comp(__pivot, __last))
            --__last;
          if (!(__first < __last))
            return __first;
          std::iter_swap(__first, __last);
          ++__first;
        }
    }
\end{minted}
As soon as our example has all elements equal to 1 one can assume
that \textbf{\_\_pivot} also keeps $1$. In the function we start to
compare all elements with the pivot. The used comparator will always
produce \textbf{true} as soon
as $1 \le 1$. The size of the array is 17
\begin{minted}{shell}
(gdb) p data.size()
$1 = 17
\end{minted}
and when we pass through whole array and reach its end we
will find $0$ on the last position (\textbf{\_\_last}), i.e. 18th
element of the array:
\begin{minted}{shell}
(gdb) p *(&data[0] + 17)
$2 = 0
\end{minted}
The found element, that is located outside the array, will be
included into the search as soon as it also satisfies the required
property: $0 \le 1$. The next element will be greater:
\begin{minted}{shell}
(gdb) p *(&data[0] + 18)
$3 = 61777
\end{minted}
The element is 19th element of the array and is located 1 position
after the original \textbf{\_\_last} iterator. Thus the \textbf{\_\_first}
iterator will be 1 position behind the original \textbf{\_\_last} one.
The value will be returned into the following helper
function
\begin{minted}{c++}
  /// This is a helper function...
  template<typename _RandomAccessIterator, typename _Compare>
    inline _RandomAccessIterator
    __unguarded_partition_pivot(_RandomAccessIterator __first,
				_RandomAccessIterator __last, _Compare __comp)
    {
      _RandomAccessIterator __mid = __first + (__last - __first) / 2;
      std::__move_median_to_first(__first, __first + 1, __mid, __last - 1,
				  __comp);
      return std::__unguarded_partition(__first + 1, __last, __first, __comp);
    }
\end{minted}
that will return the result as \textbf{\_\_cut} in
\begin{minted}{c++}
  /// This is a helper function for the sort routine.
  template<typename _RandomAccessIterator, typename _Size, typename _Compare>
    void
    __introsort_loop(_RandomAccessIterator __first,
		     _RandomAccessIterator __last,
		     _Size __depth_limit, _Compare __comp)
    {
      while (__last - __first > int(_S_threshold))
	{
	  if (__depth_limit == 0)
	    {
	      std::__partial_sort(__first, __last, __last, __comp);
	      return;
	    }
	  --__depth_limit;
	  _RandomAccessIterator __cut =
	    std::__unguarded_partition_pivot(__first, __last, __comp);
	  std::__introsort_loop(__cut, __last, __depth_limit, __comp);
	  __last = __cut;
	}
    }
\end{minted}
The \textbf{\_\_last} iterator will be replaced with \textbf{\_\_cut}
that is \textbf{\_\_first + 18} or \textbf{\_\_last + 1}. The range
stays stable until \textbf{\_\_depth\_limit} becomes 0.

Thus \textbf{std::\_\_partial\_sort} will get the range for sort that is
greater (by 1) then original one.  The element ($0$) outside
array will be placed into beginning for the vector as
\textbf{std::\_\_partial\_sort}'s application result.

\section*{Conclusion}
As you can see, the comparator that violates strict weak ordering
requirements will lead to array boundary condition violations. In our
example we were ``lucky'' and just got incorrect data in the result. The
SIGSEGV is more probable obvious result for such error.
\footnote{
In real life the broken data is more probable but often
are not recognized as an error because the result is hidden compare to
SIGSEGV one that is visible immediately.
}
The error can be easily
fixed by using correct comparator, for instance the following one
\begin{minted}{c++}
  auto comp = [](int i1, int i2) { return i1 < i2; };
\end{minted}

The example shown in the article demonstrates the importance of
requirements specified as C++ standard. Everyone who does not follow
the requirements during an application development is in the risk of
serious trouble with the application's undefined behaviour.

\bibliographystyle{gost780s}
\bibliography{cppsort}

\end{document}


\documentclass[12pt]{article}
\newcommand{\m}{\mathrm{m}}
\newcommand{\s}{\mathrm{s}}
\renewcommand{\d}{\mathrm{d}}
\newcommand{\mo}{\mathrm{mo}}
\newcounter{problem}
\begin{document}
\thispagestyle{empty}

\section*{NYU Physics 1---Problem set 3}

Due Tuesday 2009 October 6 at the beginning of lecture.

\paragraph{Problem~\theproblem:}\refstepcounter{problem}%
Chabay \& Sherwood problem 3.P.33.

\paragraph{Problem~\theproblem:}\refstepcounter{problem}%
Imagine a particle of mass $m$ moving back and forth in the $x$
direction according to the equation
\begin{equation}
x(t)= A\,\cos\left(\omega\,t\right) \quad ,
\end{equation}
where $A$ and $\omega$ are constants.  \textsl{(a)}~What are the SI
units for $A$ and $\omega$?  \textsl{(b)}~Derive an exact expression
for the $x$ component $v_x(t)$ of the velocity as a function of time.
\textsl{(c)}~Derive an exact espression for the $x$ component $F_x(t)$
of the net force acting on the particle as a function of time.
\textsl{(d)}~What is the ratio $F_x/x$ as a function of time?

\paragraph{Problem~\theproblem:}\refstepcounter{problem}%
The example problem on pages 85 and 86 of Chabay \& Sherwood gives
masses for the Earth and Sun and an initial position and velocity for
the Earth in its orbit.  Create a spreadsheet or computer program that
integrates the same trajectory as that integrated in the example
problem, but use $1\,\d$ ($8.64\times 10^4\,\s$) time steps instead of
$1\,\mo$ time steps.

I recommend that your spreadsheet or program keep track of the time
$t$, the $x$ component of position $x$, the $y$ component of position
$y$, the magnitude of the position $r=\sqrt{x^2+y^2}$, the $x$
component of momentum $p_x$, the $y$ component of momentum $p_y$, the
magnitude of the force $F$, the $x$ component of the force $F_x$, and
the $y$ component of the force $F_y$.  You can test that you have your
formulae correct by changing the timestep from $1\,\d$ up to $1\,\mo$
and seeing that you reproduce all of the numbers in the textbook
example problem (though check the errata referred to on the web site
for the course).

Integrate your orbit for $100\,d$ and hand in a plot of $x$ vs $y$ for
those 100 days.  Define the time on the first line of your spreadsheet
$t=0$.  On what day does the orbit cross the $y$ axis?  That is, on
what day does the $x$ position become negative?

\paragraph{Extra Problem (will not be graded for credit):}
Imagine you plan a interstellar voyage in which your space ship
accelerates at $g=10~\m\,\s^{-2}$ for two years.  What is wrong with
your plan?  Now imagine you increase your momentum at a constant rate
of momentum increase (that is, constant force) for two years, with
your force set so that in the first few days (when you are not moving
fast), you are accelerating at $g$.  What speed do you end up going at
the end of the two-year period, relative to the frame at which you
were at rest when you started?  \textit{Note: This problem is very
  similar to the the problem done in lecture.}

Recall that the change in momentum is the force times the time, and
recall that momentum is $\gamma\,m\,v$.  You will have to assume that
your spacecraft has some mass $m$, but you will find that the mass
cancels out in your final answer.

\end{document}


\documentclass[preprint]{aastex}
%\documentclass[iop]{emulateapj}
%\usepackage{booktabs,caption,fixltx2e}
\usepackage{natbib}
\bibliographystyle{aa}

\usepackage{graphicx,color,rotating}
\usepackage{footnote,lineno}
\usepackage{ulem}
\usepackage{xspace}
%\linenumbers
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\usepackage{txfonts}
\usepackage{graphicx,amssymb,amsmath,amsfonts,times,hyperref}
%\usepackage{rotating}

%\linenumbers

\def \aap  {A\&A}

\newcommand{\fermipy}{\texttt{Fermipy}\xspace}
\newcommand{\FIXME}[1]{{\color{red}{#1}}}

%\usepackage{epsfig,epstopdf}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
\begin{document}
%
<<<<<<< Updated upstream
\title{Notes on the implementation of likelihood weighting in the ScienceTools.}
=======
\title{Implementation of Likelihood Weighting in the fermitools}
>>>>>>> Stashed changes

\author{
E.~Charles\altaffilmark{1},
J.~Ballet\altaffilmark{2},
}
\altaffiltext{1}{W. W. Hansen Experimental Physics Laboratory, Kavli Institute for Particle Astrophysics and Cosmology, Department of Physics and SLAC National Accelerator Laboratory, Stanford University, Stanford, CA 94305, USA}
\altaffiltext{2}{Laboratoire AIM, CEA-IRFU/CNRS/Universit\'e Paris Diderot, Service d'Astrophysique, CEA Saclay, F-91191 Gif sur Yvette, France}

\date{\today}

\begin{abstract}
<<<<<<< Updated upstream
  This is a collection of notes and equations about the likelihood weighting implementation.
=======
  After several years of operations, the data taken by {\it Fermi} Large Area Telescope ({\it Fermi}-LAT) is extensive
  enough that systematic uncertainties at the few percent level from mis-modeling either the instrument response functions
  or the $\gamma$-ray emission for foreground and background sources dominate over statistical uncertainties
  over large parts of the sky for a significant part of the {\it Fermi}-LATs energy range.   The {\it Fermi}-LAT
  collaboration has developed a method for de-weighting contributions to the likelihood function
  to account for systematic uncertainties in parameter estimation for models of source content in
  a region of the sky.
  Here we present this method, along with implementation details and a worked example.
>>>>>>> Stashed changes
\end{abstract}

%\pacs{}
\maketitle

%%%%%%%%%%%%
\section{Introduction}
%%%%%%%%%%%%

<<<<<<< Updated upstream
This implementation and equations are based on Jean Ballet's work.\footnote{Extensive notes can be found at: \url{https://confluence.slac.stanford.edu/x/ZphdCw}}
=======
When studying a $\gamma$-ray source with data from the {\it Fermi} Large Area Telescope,
it is useful to consider four different types of uncertainty.

\begin{enumerate}
\item{Statistical uncertainties of the $\gamma$-ray counts from the signal signal source.
    In general these will scale as square root of the number of $\gamma$-ray counts coming
    from the source.}
\item{Statistical uncertainties of the $\gamma$-ray counts from background sources.
    In general we can model these as scaling as the square root number of $\gamma$-ray counts
    coming from the region of the source  within instrument resolution.}
\item{Systematic uncertainties of modeling the signal source.  These can come from either
    uncertainties in modeling the emission of the signal sources, or uncertainties in
    characterizing the instrument response.}
\item{Systematic uncertainties of modeling background sources.  These can come from either
    uncertainties in modeling the emission of the background sources, or uncertainties in
    characterizing the instrument response.}
\end{enumerate}

Characterizing the first two types of uncertainty is done by the maximum likelihood fitting
procedures used in analyzing $\gamma$-ray data.  Although characterizing the systematic
uncertainties is very much an open-ended problem,  we have found that a relatively simple
model in which they scale as a fixed fraction of the
$\gamma$-ray counts (rather than as the square root of the $\gamma$-ray counts) is adequate
for many purposes.

In this note we describe a method of de-weighting contributions to the likelihood function that
accounts for systematic uncertainties that is implemented in the {\tt fermitools}.
We include technical details of the implementation.

\section{Example analysis}

To illustrate how we use the weighted likelihood we have performed
an analysis of the source 4FGL J0047.9+3947 using 10 years of {\it Fermi}-LAT
data.  Specifically we used the {\tt fermipy} analysis package
to analyze a $10^\circ \times 10^\circ$ region of interest (ROI)
containing 4FGL J0047.9+3947 both with and without the likelihood weighting.
The analysis configuration parameters are listed in Tab.~\ref{tab:analysis_params}.

\begin{table}[!ht]
\begin{centering}
\begin{tabular}{cc|c}
Parameter & \multicolumn{2}{c}{Value} \\ \hline\hline
Data set & \multicolumn{2}{c}{10 Years} \\
ROI Size & \multicolumn{2}{c}{$10^\circ \times 10^\circ$} \\
ROI Center $(l,b)$ & \multicolumn{2}{c}{($121.1743^\circ$, $-21.5734^\circ$)} \\
Pixel size & \multicolumn{2}{c}{$0.05^\circ \times 0.05^\circ$} \\
Coordinates system  & \multicolumn{2}{c}{Galactic} \\
Projection & \multicolumn{2}{c}{Aitoff} \\
Energy range & \multicolumn{2}{c}{100 MeV to 100 GeV} \\
Energy binning & \multicolumn{2}{c}{8 bins / decade} \\
Event Class & \multicolumn{2}{c}{P8R3\_SOURCE} \\
Instrument Response Functions & \multicolumn{2}{c}{P8R3\_SOURCE\_V2} \\
Energy Dispersion & \multicolumn{2}{c}{Point sources only} \\
Galactic Diffuse Model & \multicolumn{2}{c}{{\it gll\_iem\_v07.fits}} \\
Isotropic Model Prefix & \multicolumn{2}{c}{{\it iso\_P8R3\_SOURCE\_V2\_}} \\
Catalog Sources  & \multicolumn{2}{c}{4FGL} \\ \hline
Event Types & Front-converting & Back-converting \\
Zenith Angle Cut & $90^\circ$ & $80^\circ$ \\
Isotropic Model Suffix & {\it FRONT\_v1.txt} & {\it BACK\_v1.txt} \\
\end{tabular}
\caption{Parameters used for example analysis.  The analysis was done as a two
  component binned likelihood; front-converting and back-converting
  events were treating separately, and the log-likelihood contributions from the two
  components were summed.}
\label{tab:analysis_params}
\end{centering}
\end{table}
>>>>>>> Stashed changes

\section{The effective background: \texorpdfstring{$B$}{B}}

The first step in computing the weights for the spatial and energy bins
is to derive the ``effective background'' for every pixel and energy bin.
This is essentially the $\gamma$-ray counts for the background sources that
contribute to the uncertainty in estimating parameters of a point source at a
particular spatial and energy bin.

We derive the effective background starting from some representation
<<<<<<< Updated upstream
of the counts in the region of interest (ROI).  This can be either
binned data, or a model of the ROI.  Following notation we used
=======
of the counts in the ROI.  This can be either
binned $\gamma$-ray counts, or a model of the ROI.  Following notation we used
>>>>>>> Stashed changes
elsewhere\footnote{Specifically, in the write up of the binned
  likelihood implementation.}.  In either case we will call this quantity $M_{ik}$, where the $i$
index runs over the pixels in the model, and the $k$ index runs over
the energy bins.  The energy bin edges are at $E_k^-, E_k^+$
(typically $E_k^+ = E_{k+1}^-$).  The geometric energy bin centers are
$E_k = \sqrt{E_k^+ E_k^-}$.  The energy bin widths are $\delta E_k = E_k^+
- E_k^-$, and the pixel sizes are $\delta \Omega_i$.

<<<<<<< Updated upstream
In words, we want to define the effective background $B_{ik}$ by
=======
Model counts maps at two different energies of our example analysis are shown in
Fig.~\ref{fig:mcube}.

\begin{figure}[h]
\begin{centering}
\includegraphics[width=0.49\columnwidth]{figures/mcube_E00_00.png}
\includegraphics[width=0.49\columnwidth]{figures/mcube_E01_04.png}
\vspace{-0.10in}
\caption{\label{fig:mcube}Model counts maps ($M_{i}$, in counts per $0.05^\circ\times0.05^\circ$ pixel)
  for an example ROI for two energy bins and two different events types. Left, front-converting events
  from 100~MeV to 133~MeV; right, back-converting events from 316~MeV to 421~MeV.}
\end{centering}
\end{figure}

In words, we define the effective background $B_{ik}$ by
>>>>>>> Stashed changes
convolving $M_{ik}$ with the point-spread function (PSF) at each
energy ($P_k$) and then sum the result over all energies greater than or
equal to a particular energy.

\begin{equation}\label{eq:bkg_eff_defintion}
B_{ik} = \sum_{j \ge k} \frac{M_{ij} \bigotimes P_{j}}{P_{j}^{\rm max}},
\end{equation}

\noindent where $P_{j}^{\rm max}$ is the maximum value of the PSF at energy $j$.
Another way of expressing this is that it is the PSF-weighted integral over
the background map.

The convolution routines in the ScienceTools work on objects of type
{\tt ProjMap} (actually, the sub-classes {\tt HealpixProjMap} and {\tt
  WcsMap2}), which are differential quantities (i.e., they are
intensities, defined at specific energy and directions, rather than
begin integrated across a range of energies and over a pixel.  In
practical terms, this just means that we have to convert the model
counts from either {\tt CountsMap} or {\tt CountsMapHealpix} to a {\tt
  WcsMap2} or {\tt HealpixProjMap}.  We do this just by dividing the
bin contents by the energy bin widths and pixel sizes.

\begin{equation}\label{eq:intensity}
I_{ik} = \frac{M_{ik}}{\delta \Omega_i \delta E_k}.
\end{equation}

\noindent What we actually get back from the PSF convolution routine
is the normalized convolution:

\begin{equation}\label{eq:convolved_intensity}
\tilde{I}_{ik} = I_{ik} \bigotimes P_{k}.
\end{equation}

To get the effective background, we have to convert that quantity back
to counts and sum of all the energy bins greater than or equal to a
particular energy.

\begin{equation}\label{eq:bkg_eff_computation}
B_{ik} = \sum_{j}^{j \ge k} \frac{\tilde{I}_{ij} \delta E_j}{P_{j}^{\rm max}}.
\end{equation}

\noindent Although the factors of $\delta E_k$ in Eq.~\ref{eq:intensity}
and $\delta E_j$ in Eq.~\ref{eq:bkg_eff_computation} cancel, we have
explicitly kept them in the notation, as they appear in the code because
of the way it is structured.

The quantity $B_{ik}$ has units of counts, and is essentially a counts
map.  We store it as such.  It can be produced by the standalone
application {\tt gteffbkg} or with the {\tt pyLikelihood} interface.
The resulting file will look almost identical to a binned counts map
file, including the {\tt EBOUNDS} and {\tt GTI} HDUs, and the DSS
keywords are copied from the input file.  The only difference will be
the addition of keywords to the primary header of the output file:

\begin{enumerate}
\item{{\tt MAPTYPE} is ``BKG\_EFF''.}
\item{{\tt INPUTMAP} is set to the name of the input binned counts map
file.}
\end{enumerate}

<<<<<<< Updated upstream
=======
Effective background maps at two different energies of our example analysis are shown in
Fig.~\ref{fig:beff}.

It is very important to compute the effective background using an ROI
model that has been padded to account for spill-over from the PSF.  To do this
we first analyzed the ROI without likelihood weights, then used the
script {\tt fermipy-make-wmaps} to correctly make the effective
background maps (and all subsequent steps described below) for an ROI that
was padded by $5^\circ$ on all sides.

\begin{figure}[h]
\begin{centering}
\includegraphics[width=0.49\columnwidth]{figures/beff_E00_00.png}
\includegraphics[width=0.49\columnwidth]{figures/beff_E01_04.png}
\vspace{-0.10in}
\caption{\label{fig:beff}Effective background maps ($B_{i}$), in counts per
  $0.05^\circ\times0.05^\circ$ pixel) for an example ROI
  for two energy bins and two different events types.  Left, front-converting events
  from 100~MeV to 133~MeV; right, back-converting events from 316~MeV to 421~MeV.}
\end{centering}
\end{figure}

>>>>>>> Stashed changes
\section{The weighted sum over components: \texorpdfstring{$\alpha$}{alpha}}

In order to properly deal with likelihood deweighting for multiple
data components in one model (e.g., if we are treating
front-converting and back-converting seperately) we need to
compute the weighted sum over the components.  This quantity depends on
the fractional systematic uncertainty (see \S~\ref{sec:intro})
we are considering ($\epsilon$), on the individual $B_{ikm}$ for each component, and on the
minimum $B_{ikm}$ for the various components for each pixel and energy
bin: $\hat{B}_{ik}$.

We define weight as:

\begin{equation}
\alpha_{ik} = \frac{1 + \epsilon^2 \hat{B}_{ik}}{1 + \epsilon^2 \hat{B}_{ik} \sum_{m} (\frac{\hat{B}_{ik}}{B_{ikm}})^2 }.
\end{equation}

\noindent In the case that we are using a single component, then all
of the $\alpha_{ik} \equiv 1$.

This quantity is dimensionless, and has the same binning as the input
counts map.  It can be produced by the standalone application {\tt gtalphabkg}
or with the {\tt pyLikelihood} interface.  This will store almost
exactly the same information as a {\tt CountsMap} or {\tt
  CountsMapHealpix}. Since we are using several components, we remove
the DSS keywords.  However we do add some keywords specifying the
provenance of the map:

\begin{enumerate}
\item{{\tt MAPTYPE} is ``ALPHA\_BKG'';}
\item{{\tt EPSILON}: has the value of $\epsilon$ used in the computation;}
\item{{\tt BKGMAPXX}: lists the input effective background maps.}
\end{enumerate}

<<<<<<< Updated upstream
=======
Maps of the parameter $\alpha_{ik}$ for our example analysis are shown in Fig.~\ref{fig:alpha}.

\begin{figure}[h]
\begin{centering}
\includegraphics[width=0.49\columnwidth]{figures/alpha_E00_00.png}
\includegraphics[width=0.49\columnwidth]{figures/alpha_E01_04.png}
\vspace{-0.10in}
\caption{\label{fig:alpha}Maps of $\alpha_{i}$ for our example ROI
  for two energy bins, obtained by combining front-converting and
  back-converting events and setting the systematic error level
  to $\epsilon = 0.03$.  Left, 100~MeV to 133~MeV; right 316~MeV to 421~MeV.}
\end{centering}
\end{figure}

\section{Choosing the value of the fractional systematic uncertainty}

The {$\it Fermi$}-LAT collaboration arrived at a value of $\epsilon = 0.03$ by
picking a value so that the standard deviation of the uncertainty-scaled residuals
of the energy flux for individual energy bins for a large number of spectral fits
was close to one.  Put another way, we picked $\epsilon = 0.03$ because that value
re-scaled the errors of SED measurements to give a $\chi^2$ per degree of freedom
close to one.
>>>>>>> Stashed changes

\section{The likelihood weights: \texorpdfstring{$w$}{w}}

Given the effective background maps and the $\alpha_{ik}$, the
likelihood weights for a particular component are defined as:

\begin{equation}
w_{ikm} = \frac{\alpha_{ik}}{1 + \epsilon^2 B_{ikm}}.
\end{equation}

This quantity is dimensionless and has the same binning as the input
counts map.  It can be produced by the standalone application {\tt gtwtsmap}
or with the {\tt pyLikelihood} interface.  However, for
historical reasons, the {\tt BinnedLikelihood} object expects the
weights to be given as an object of the class {\tt ProjMap}.  In the
{\tt fermitools} {\tt ProjMap} objects usually represent intensity maps,
but there is nothing enforcing this.  In practical terms the only real
difference for a weights file is that it is defined at specific
energies rather than over energy ranges.  This means that we replace
the {\tt EBOUNDS} HDU with an {\tt ENERGIES} HDU.

Aside from that, the $w_{ik}$ files have almost exactly the
same information as a {\tt CountsMap} or {\tt CountsMapHealpix},
including the DSS keywords copied from the file with the $B_ik$; the
only differences being the additional keywords:

\begin{enumerate}
\item{{\tt MAPTYPE} is ``WEIGHT\_MAP''.}
\item{{\tt EPSILON}: has the value of $\epsilon$ used in the computation;}
\item{{\tt BKGMAP}: has the name of the file with the input effective background map.}
\item{{\tt ALPHAMAP}: has the name of the file containing the input $\alpha$ map (if used).}
\end{enumerate}

\section{Using the likelihood weights.}

The idea is that using the likelihood weights is almost transparent.
If a likelihood weights map is specified when the {\tt BinnedLikelihood}
object is being created, those weights will be used; otherwise no
weights will be used.

Depending on the interface used, the likelihood weights can be
specified in a number of ways.

\begin{enumerate}
\item{By passing an object of type {\tt ProjMap} (actually either a
    {\tt WcsMap2} or a {\tt HealpixProjMap}) into the constructor of
    {\tt BinnedLikelihood}.  The map will be re-sampled, taking the
    coordinate information from the bin and pixel centers of the {\tt CountsMapBase}
    and requesting the {\tt ProjMap} value for those.}
\item{By passing a file name into the constructor of {\tt
      pyLikelihood.BinnedAnalysis}.  This points to any file that
    contains a valid {\tt ProjMap}, and the file will be
    resampled as above.}
\item{By specifying a file name for the hidden {\tt wmap} parameter of
    {\tt gtlike} or {\tt gtscrmaps}.  This points to any file that
    contains a valid {\tt ProjMap}, and the file will be
    resampled as above.}
\end{enumerate}

<<<<<<< Updated upstream
=======

\section{Effect of using the likelihood weights.}

Since the likelihood weights are constructed to be less than one, they
have the effect of reducing the likelihood, and also reducing changes in
the likelihood.  Since uncertainty estimates and upper limits are defined
in terms of changes in the likelihood, using the likelihood weights
will result in larger uncertainties and higher upper limits.

The effect on fits to the normalization of 4FGL J0047.9+3947
for the lowest energy bin in our example analysis is shown in Fig.~\ref{fig:ll}.

\begin{figure}[h]
\begin{centering}
\includegraphics[width=\columnwidth]{figures/ll_compare.pdf}
\vspace{-0.10in}
\caption{\label{fig:ll}Change in profile log likelihood ($\Delta\log\mathcal{L}$ with
  respect to the best-fit value) for 4FGL J0047.9+3947 as a function of the
  normalization in the energy range 100~MeV to 133~MeV.}
\end{centering}
\end{figure}

The effect on fitting the spectral energy density (SED) of a point
source in our example analysis is shown in Fig.~\ref{fig:sed}.  Clearly
the effect of the likelihood weighting is largest for the lower energy bins,
where the effective background is larger and the
weights are smaller.

\begin{figure}[h]
\begin{centering}
\includegraphics[width=0.49\columnwidth]{figures/sed_nowts.png}
\includegraphics[width=0.49\columnwidth]{figures/sed_wts.png}
\vspace{-0.10in}
\caption{\label{fig:sed}SED of 4FGL J0047.9+3947, including the $\Delta\ln\mathcal{L}$ shown
  with the under-laid color scale. Results
  for the un-weighted (weighted) likelihood are shown on the left (right).}
\end{centering}
\end{figure}]

\section{Acknowledgments}

This implementation and equations are based on
Jean Ballet's work.\footnote{Extensive notes can be found at: \url{https://confluence.slac.stanford.edu/x/ZphdCw}}

>>>>>>> Stashed changes
\end{document}

% LocalWords:  ScienceTools ROI convolving PSF ProjMap HealpixProjMap WcsMap2
% LocalWords:  CountsMap CountsMapHealpix gteffbkg pyLikelihood EBOUNDS GTI DSS
% LocalWords:  HDUs MAPTYPE BKG INPUTMAP gtalphabkg BKGMAPXX gtwtsmap HDU wmap
% LocalWords:  BinnedLikelihood BKGMAP ALPHAMAP CountsMapBase BinnedAnalysis Eq
<<<<<<< Updated upstream
% LocalWords:  gtlike gtscrmaps
=======
% LocalWords:  gtlike gtscrmaps fermitools Laboratoire CEA IRFU Astrophysique
% LocalWords:  Saclay Gif sur mis LATs de fermipy MeV GeV P8R3 SOURCE J0047
% LocalWords:  V2 gll iem v07 4FGL v1 txt iso SED un wmaps
>>>>>>> Stashed changes


%
% STAT 100: Chance and Data Analysis - A Course Overview
% Section: Data Collection
%
% Author: Jeffrey Leung
%

\section{Data Collection}
	\label{sec:data-collection}
\subsection{Methods}
	\label{subsec:data-collection:methods}
\begin{easylist}

	& \emph{Census:} Survey which collects information from all individuals of a population
		&& Process important?
		&& Not always viable because it may be:
			&&& Expensive
			&&& Time-consuming
			&&& Impossible (e.g. taking a census of everyone in the world)
		&& E.g. Taking a census of all Canadian citizens

	& \emph{Sample survey:} Survey which collects information from a select group of individuals smaller than and representative of the population
		&& Relatively greater data quality due to smaller samples
		&& E.g. Taking a sample survey of which political party for which people plan to vote

	& \emph{Observational study:} Data collection method in which data is collected through observation without interference
		&& Concludes an correlation or lack thereof between two variables
			&&& Lurking variable(s) (see ref) may connect the two variables and explain the association %TODO reference
		%TODO
		&& E.g. Students were given the option to attend an optional tutorial session. Those who attended the tutorial session were more likely to receive a higher mark on their exam. %TODO

	& \emph{(Randomized) Experiment:} Data collection method in which individuals are chosen randomly to... %TODO
		&& Not all factors can be studied through randomized experiments
		&& E.g. Some students are chosen at random to be given extra tutorial sessions. They receive...
		%TODO

		%TODO references to the subsections below

\end{easylist}
\subsection{Sampling}
	\label{subsec:data-collection:sampling}
\begin{easylist}

	& \emph{Random sample:} %TODO

	& \emph{Simple random sample:} Set of individuals selected from the population such that each individual has an equal chance of being selected
		&& Process: To select $y$ individuals from a group of $x$, number each individual
		&& %TODO

\end{easylist}
\subsection{Sampling Errors}
	\label{subsec:data-collection:sampling-errors}
\begin{easylist}

	& Non-sample errors:

		&& \emph{Response error:} Inaccurate/untruthful response which skews data
			&&& E.g. An experiment which asks a question about a morally shameful activity may have a response error due a tendency to answer with the morally best choice, so as to give a better impression
		&& \emph{Non-response error:} Lack of response which skews data
			&&& E.g. An experiment which offers an optional response to a survey, such as through email, may have a non-response error due to some people not responding because of not checking email, laziness, apathy
			&&& No standard way to determine a good response rate
				&&&& Theoretically, 75\% and above is acceptable
			&&& Analyze the response rate if possible, and the reason for a low response rate
		&& \emph{Question wording:} Syntax/keywords/details of a question in a survey which skews data

	& Voluntary response error:
		&& To explain, calculate the (low) response rate and offer a possible explanation

\end{easylist}
\subsection{Design of an Experiment}
	\label{subsec:data-collection:design-of-an-experiment}
\begin{easylist}

	& \emph{Explanatory variable:} Factor which is hypothesized to affect another factor
		&& \emph{Treatment:} Unique combination of one of each type of explanatory variable
			&&& $Number\ of\ treatments = num\ of\ variable_{1} \times num\ of\ variable_{2} \times \cdots$
			&&& Table of all possible treatments should be written
			&&& Sample is assigned equally and randomly into the number of treatments
	& \emph{Response variable:} Resultant factor which is hypothesized to be affected by another factor

	& Experiment diagram + block diagram %TODO

	& E.g. An experiment was conducted to find out if the length and/or repetitions of a TV commercial affect the desire to buy a product. 20 subjects were chosen for the experiment. \smallskip \\

	\emph{Explanatory variables:} \\
	\indent Length of the TV commercial (1 minute / 5 minutes / 10 minutes) \\
	\indent Number of repetitions of the TV commercial (1 time / 3 times / 5 times) \\
	\emph{Response variable:} \\
	Desire to buy the product in the TV commercial (Scale from 1-5 where 1 = Do not want to buy) \\
	\emph{Treatments:} \\
	$3 \times 3 = 9$ treatments (see table~\ref{tab:experiment-length-repetitions-commercial-treatments}) \\

	\Deactivate
	\begin{table}[!htb]
		\centering
		\caption{Treatments of a Experiment on Length and Repetitions of a Commercial}
		\label{tab:experiment-length-repetitions-commercial-treatments}
		\begin{tabular}{ r r }
			Length (mins): & Repetitions: \\
			\hline
			 1 & 1 \\
			   & 3 \\
			   & 5 \\
			 5 & 1 \\
			   & 3 \\
			   & 5 \\
			10 & 1 \\
			   & 3 \\
			   & 5
		\end{tabular}
	\end{table}
	\Activate

\end{easylist}
\clearpage