\chapter{$b$-jet Triggers}\label{appendix:bjet_trigger}

\section{Parsing Trigger Chains}\label{section:trigger_chain_names}

Trigger algorithms (known as ``chains'' as they are a series of criteria and algorithm decisions) are complex series of logic.
Their naming conventions are not readily clear to the uninitiated, so the following is a very brief summary of how to parse the grammar of $b$-jet triggers (for more detail c.f.~\cite{Feickert:HbbWorkshop2017}). An example chain name in COMA~\cite{TWiki:ConditionsMetadata,COMA:ChainReport}, \texttt{\textcolor{black}{HLT}\_\textcolor{red}{j55}\_\textcolor{green}{gsc75}\_\textcolor{blue}{bmv2c1040}\_\textcolor{purple}{split}\_\textcolor{red}{3j55}\_\textcolor{green}{gsc75}\_\textcolor{orange}{boffperf}\_\textcolor{purple}{split}}, is decomposed below.
\begin{itemize}
 \item \texttt{\textcolor{black}{HLT}}: HLT trigger chain prefix to distinguish from L1 items.
 \item \texttt{\textcolor{red}{nj55}}: Requires at least $n$ jets with $p_{T} > 55~\GeV$.
 \item \texttt{\textcolor{purple}{split}}: \texttt{superROI} configuration being used for 2-step track reconstruction and primary vertex finding.
 \item \texttt{\textcolor{green}{gsc75}}: Apply Global Sequential Corrections (GSC) to jets passing first $p_{T}$ threshold and require jets with GSC $p_{T} > 75~\GeV$.
       \begin{itemize}
        \item GSC requires track reconstruction to be run, but as $b$-tagging requires tracking there is no extra cost in $b$-jet trigger chains.
       \end{itemize}
 \item \texttt{\textcolor{orange}{boffperf}}: The $b$-tagging algorithm (\texttt{MV2c10}) is run over the jets but no selection is applied on the output.
 \item \texttt{\textcolor{blue}{bmv2c1040}}: Requires passing the \texttt{MV2c10} $b$-tagging algorithm at the $40\%$ selection efficiency working point.
\end{itemize}

The result is that the trigger chain name should be parsed to read ``at least 4 jets with GSC $p_{T} > 75~\GeV$ and track reconstruction and primary vertex finding done using Super-RoI configuration to have had \texttt{MV2c10} $b$-tagging algorithm run over them, and at least 1 of them to have passed the \texttt{MV2c10} $b$-tagging algorithm at the 40\% selection efficiency working point.''

\section{Super-RoIs}\label{section:super-RoI}

The purpose of the $b$-jet trigger to provide effective $b$-jet identification and light-jet and $c$-jet rejection in the \gls{HLT} to maximize the number of interesting physics events containing heavy flavor physics.
This still needs to be done quickly, but given that $b$-tagging is being performed on each jet candidate that passes other selection criteria tracking must also be performed.
As a result, the $b$-jet triggers are the leading consumers of CPU out of all triggers in the ATLAS trigger menu.
In Run 2 to address this issue a new CPU and time saving measure was introduced~\cite{Hetherly:2313140}.
Instead of running the track matching algorithms in each \gls{RoI} that existed for the event, even if there was substantial overlap between the RoIs, all RoIs --- regardless of if they have topologically overlap --- are merged into a single ``Super-RoI,'' as seen in \Cref{fig:all_ROI_vs_superROI}.
Once the Super-RoI has been created, fast tracking is run in the Super-RoI to find a primary vertex.
Once a primary vertex has been found, precision tracking is then run in each of the original RoIs, but with the tracks constrained by the Super-RoI primary vertex.

\begin{figure}[htbp]
 \centering
 \includegraphics[width=\textwidth]{appendix/all_ROI_vs_superROI.pdf}
 \caption[Cartoon of the all RoI vs. the Super-RoI configuration for tracking run by the $b$-jet trigger.]{%
  Cartoon of the all RoI (no-split) configuration (left) vs. the Super-RoI (split) configuration (right) configuration for tracking run by the $b$-jet trigger.
  The gird represents $\eta$-$\phi$ space.
  The Super-RoI may not be topologically connected but counts as a single object~\cite{Hetherly:2313140}.}
 \label{fig:all_ROI_vs_superROI}
\end{figure}

\section{$b$-jet Trigger Efficiency in High Pile-up}\label{section:BJetTrig_efficiency}

As the LHC moves into Run 3 the energy and luminosity will increase, causing there to be higher pile-up, as discussed in \Cref{section:LHC_collider}.
To characterize the performance of the 2017 $b$-jet trigger chains in these future environments I performed a study of the $b$-jet trigger efficiency in high pile-up environments using high purity di-lepton $t\bar{t}$ events in the 2017 data-set to provide a sample enriched with $b$-jets.
The results, seen in \Cref{fig:BJetTrigg_efficiency_vs_mu} and \Cref{fig:BJetTrigg_pass_fraction_vs_mu}, show that the $b$-jet triggers compared to the offline $b$-tagging algorithm \texttt{MV2c10} at the $70\%$ efficiency operating point have a high and flat efficiency for pile-up, $\braket{\mu}$, out to beyond $\braket{\mu}=60$.
Likewise, the number of jets that pass the online $b$-tagging requirement remains proportionally low given the different efficiency operating points.
Beyond $\braket{\mu}=60$ there starts to be some degradation for the lower online efficiency operating points, though the statistical uncertainty also begins to increase.
The plots to not extend out to $\braket{\mu}=70$ as there was insufficient number of events to have reasonable statistical uncertainties given the dataset used.
Similar results were obtained when the study was repeated using 2018 triggers and data~\cite{Sekula:2631805}.

\begin{figure}[htbp]
 \centering
 \includegraphics[width=\textwidth]{appendix/BJetTrigg_efficiency_vs_mu.eps}
 \caption[The $b$-jet trigger efficiency with respect to the offline $b$-tagging algorithm (\texttt{MV2c10}) at the 70\% efficiency operating point for various online efficiency operating points vs. the mean number of interactions per crossing.]{%
  The $b$-jet trigger efficiency with respect to the offline $b$-tagging algorithm (\texttt{MV2c10}) at the 70\% efficiency operating point for various online efficiency operating points vs. the mean number of interactions per crossing.
  The relative $b$-jet trigger efficiency is measured in high purity di-lepton $t\bar{t}$ events collected in the 2017 data-set using dedicated single-lepton$+$jets triggers, which are unbiased with respect to the online $b$-tagging.
  The online operating points were defined to have roughly the quoted efficiency for $b$-jets in an unbiased sample of Monte Carlo simulated $t\bar{t}$ events.
  The uncertainty bars shown only represent statistical uncertainties~\cite{Feickert:2294576}.}
 \label{fig:BJetTrigg_efficiency_vs_mu}
\end{figure}

\begin{figure}[htbp]
 \centering
 \includegraphics[width=\textwidth]{appendix/BJetTrigg_pass_fraction_vs_mu.eps}
 \caption[The fraction of trigger jets with Global-Sequential-Calibration-corrected $E_T > 55~\GeV$ that pass the online $b$-tagging algorithm (\texttt{MV2c10}) at various online efficiency operating points vs. the mean number of interactions per crossing.]{%
  The fraction of trigger jets with Global Sequential Calibration (GSC)-corrected $E_T > 55~\GeV$ that pass the online $b$-tagging algorithm (\texttt{MV2c10}) at various online efficiency operating points vs. the mean number of interactions per crossing.
  The pass fraction is measured in a subset of the 2017 data-set with events containing at least one jet with GSC-corrected $E_T > 55~\GeV$.
  The trigger used is unbiased with respect to the online $b$-tagging.
  The online operating points were defined to have roughly the quoted efficiency for $b$-jets in an unbiased sample of Monte Carlo simulated $t\bar{t}$ events.
  The uncertainty bars shown only represent statistical uncertainties~\cite{Feickert:2294576}.}
 \label{fig:BJetTrigg_pass_fraction_vs_mu}
\end{figure}


\chapter{Cluster analysis}
\label{clustan}

Cluster ``analysis'' describes a range of algorithms for investigating structure in data, the main interest is in finding groups of objects who are more alike.  A large number of books are dedicated to this one subject, for example \cite{Kaufman+Rousseeuw:1989,Everitt+etal:2001,Gordon:1999}, the former book supporting some rather interesting code that has been ported to \R (the S-PLUS version is described in  \cite{Struyf+etal:1997}).   It may be worth noting that some multivariate authors do not cover it at all \cite{Flury:1997} preferring a formulation based upon mixture models, and [page 316] \cite{Venables+Ripley:2002} indicate a preference for using visualisation methods for finding such structure in data.  Whilst cluster analysis is most often directed towards finding subsets of individuals that are more alike than other subsets, it is worth noting that variable clustering is also possible.   The heatmap in figure \ref{heatmap} has a dendrogram for both individuals and variables.

\singlespacing
\begin{verbatim}
x  <- as.matrix(mtcars)
rc <- rainbow(nrow(x), start=0, end=.3)
cc <- rainbow(ncol(x), start=0, end=.3)
hv <- heatmap(x, col = cm.colors(256), scale="column",
     RowSideColors = rc, ColSideColors = cc, margin=c(5,10),
     xlab = "specification variables", ylab= "Car Models",
     main = "Heatmap of mtcars data")
\end{verbatim}
\onehalfspacing

\begin{figure}
\begin{center}
\includegraphics[width = 0.7\textwidth]{images/heatmap}
\caption{Heatmap of \texttt{mtcars} data; variables scaled in columns}
\label{heatmap}
\end{center}
\end{figure}

Modern computing facilities have widened the possibilities for visualisation considerable (both in terms of linked displays as well as the ability to numerically optimise various projection criteria).   However, when investigating the results of any scaling or projection methods there may still be interesting structures within the data.   The interest in cluster analysis lies in finding groups within the data.   When the objects within a group are very similar this can be described as internal cohesion (or homogeneity), when they is a large dissimilarity between groups this can be referred to as external separation (or isolation).   Where we have internal cohesion and external separation, one usually refers to the situation as ``clustering''. If the objects have been more-or-less
aribtrarily divided into groups we could refer to this as ``dissection''.
Clustering and dissection may both the useful in different
contexts (e.g. taxonomy and market research).   One attempt to illustrate this concept is given in figure \ref{partclust}.

%par(mfrow = c(1,2))

%x <- rbind(mvrnorm(20, c(0,-1), sigma), mvrnorm(20, c(5,4), sigma), mvrnorm(20, c(-5,4), sigma))
%plot(x, pch = 16, main = "Clustering", xlab = expression(x[1]), ylab = expression(x[2]) )
%lines(c(0,-6), c(4,-2))
%lines(c(0,0), c(4,7))
%lines(c(0,6), c(4,-1.4))

%x <- mvrnorm(60, c(0,4), sigma)
%plot(x, pch = 16, main = "Partitioning", xlab = expression(x[1]), ylab = expression(x[2]) )
%lines(c(0,-6), c(4,-2))
%lines(c(0,0), c(4,7))
%lines(c(0,6), c(4,-1.4))

\begin{figure}
\begin{center}
\includegraphics[width = 0.7\textwidth]{images/partclust}
\caption{Artificial data suggesting a difference between ``clustering'' and ``dissecting''}
\label{partclust}
\end{center}
\end{figure}

There are a wide range of algorithms that have been developed to investigate clustering within data.   These can be considered in a number of ways:

\begin{itemize}
\item Hierarchical Methods
  \begin{itemize}
  \item Agglomerative clustering (\verb+hclust()+, \verb+agnes()+)
  \item Divisive clustering (\verb+diana()+, \verb+mona()+)
  \end{itemize}
\item Partitioning methods (\verb+kmeans()+, \verb+pam()+, \verb+clara()+)
\end{itemize}

Hierarchical clustering provides a set of clusterings, either for $k = n, \ldots, 2$ (agglomerative) or $k = 2, \ldots, l$ (divisive).   The clustering is represented in a dendrogram and can be cut at arbitrary heights to give a fixed number of clusters.   It perhaps has a logical interpretation in numerical taxonomy.  Partitioning methods usually work by assuming a pre-determined number of clusters $k$ (although it is obviously possible to explore solutions over a range of values of $k$).   As [page 379] \cite{Seber:1984} points out, the number ofways $S_{k,n}$ of partitioning $n$ objects into $k$ groups, given by:
\begin{displaymath}
S_{k,n} = \frac{1}{k!} \sum_{j=1}^{k} \left( \begin{array}{c}k \\ j \end{array} \right) (-1)^{k-j} j^{n} \approx_{n \to \infty} \frac{k^{n}}{k!}
\end{displaymath}
is a second type Stirling number, and where $k$ is not specified we have $\sum_{k=1}^{K} S_{k,n}$ partitions.

For $n= 50$ and $k=2$ this is in the order of $6 \times 10^{29}$

\section{Introduction to agglomerative hierarchical cluster analysis}

Hierarchical cluster analysis finds representations that obey the ultrametric inequality.

\begin{displaymath}
d(x_{ij}, x_{ik}) \leq max(d(x_{ij}, x_{il}) d(x_{il}, x_{ik}) )
\end{displaymath}

We will work through three conventional examples of cluster analysis manually before considering an \R implementation in more detail.   For the sake of the demonstations, consider the following distance matrix:

\begin{tabular}{r|ccccc}
 & a & b & c & d & e\\
\hline
a & 0  &   &   &   &\\
b & 2  & 0 &   &   &\\
c & 6  & 5 & 0 &   & \\
d & 10 & 9 & 4 & 0 &  \\
e & 9  & 8 & 5 & 3 & 0\\
\end{tabular}

Initially, each individual is put in its own cluster.   Subsequently, at each stage individuals are joined to other individuals to form clusters, and the distance measure between a units can be readjusted in some way.   Then similar clusters are joined until all individuals have been joined.

\subsection{Nearest neighbour / Single Linkage}

This can be obtained in \R using the \verb+method = "single"+ instruction in the call to \verb+hclust()+, where it suggests that this method finds ``friends of friends'' to join each cluster in a similar way to that used in minimum spanning trees.

The decision to merge groups is based on the distance of the nearest member of the group to the nearest other object.   Clearly, with a distance of 2, inviduals $a$ and $b$ are the most similar.

\begin{minipage}[c]{0.5\textwidth}
\begin{tabular}{r|ccccc}
 & $a$ & $b$ & $c$ & $d$ & $e$\\
\hline
$a$ & 0  &   &   &   &\\
$b$ & 2  & 0 &   &   &\\
$c$ & 6  & 5 & 0 &   & \\
$d$ & 10 & 9 & 4 & 0 &  \\
$e$ & 9  & 8 & 5 & 3 & 0\\
\end{tabular}
\end{minipage}
\begin{minipage}[c]{0.5\textwidth}

We therefore merge these into a cluster at level 2:

\begin{tabular}{ll}
Distance & Groups\\
\hline
0 & a b c d e\\
2 & (ab) c d e
\end{tabular}
\end{minipage}

and we now need to re-write our distance matrix, whereby:

\begin{eqnarray*}
d_{(ab)c)} = min(d_{ac},d_{bc}) = d_{bc} = 5\\
d_{(ab)d)} = min(d_{ad},d_{bd}) = d_{bd} = 9\\
d_{(ab)e)} = min(d_{ae},d_{be}) = d_{be} = 8
\end{eqnarray*}

This gives us a new distance matrix

\begin{minipage}[c]{0.5\textwidth}
\begin{tabular}{r|ccccc}
 & $(ab)$ & $c$ & $d$ & $e$\\
\hline
$(ab)$ & 0 &   &   &  \\
$c$    & 5 & 0 &   &  \\
$d$    & 9 & 4 & 0 &  \\
$e$    & 8 & 5 & 3 & 0\\
\end{tabular}
\end{minipage}
\begin{minipage}[c]{0.4\textwidth}

Now we find that the two nearest objects are $d$ and $e$, these can be merged into a cluster at height 3:

\begin{tabular}{ll}
Distance & Groups\\
\hline
0 & $a$ $b$ $c$ $d$ $e$\\
2 & $(ab)$ $c$ $d$ $e$\\
3 & $(ab)$ $c$ $(de)$
\end{tabular}
\end{minipage}

We now need to find the minimum distance from $d$ and $e$ to the other objects and reform the distance matrix:
\begin{minipage}[c]{0.5\textwidth}
\begin{tabular}{r|ccc}
 & $(ab)$ & $c$ & $(de)$\\
\hline
$(ab)$ & 0 &   &    \\
$c$    & 5 & 0 &   \\
$(de)$ & 8 & 4 & 0  \\
\end{tabular}
\end{minipage}
\begin{minipage}[c]{0.5\textwidth}

Clearly, the next merger is between $(de)$ and $c$, at a height of 4.

\begin{tabular}{ll}
Distance & Groups\\
\hline
0 & $a$ $b$ $c$ $d$ $e$\\
2 & $(ab)$ $c$ $d$ $e$\\
3 & $(ab)$ $c$ $(de)$\\
4 & $(ab)$ $(cde)$
\end{tabular}
\end{minipage}

And it is also clear that the next step will involve merging at a height of 5.

\begin{minipage}[c]{0.5\textwidth}
\begin{tabular}{ll}
Distance & Groups\\
\hline
0 & $a$ $b$ $c$ $d$ $e$\\
2 & $(ab)$ $c$ $d$ $e$\\
3 & $(ab)$ $c$ $(de)$\\
4 & $(ab)$ $(cde)$\\
5 & $(abcde)$
\end{tabular}
\end{minipage}
\begin{minipage}[c]{0.5\textwidth}
\end{minipage}

The corresponding dendrogram is illustrated in figure \ref{simpleclust}.

\subsection{Furthest neighbour / Complete linkage}

Obtained in \R using the \verb+method = "complete"+ instruction in the call to \verb+hclust()+, where it suggests that this method finds similar clusters.   Complete linkage methods tend to find similar clusters.

Groups are merged when the furthest member of the group is close enough to the new object.

\begin{minipage}[c]{0.5\textwidth}
\begin{tabular}{r|ccccc}
 & a & b & c & d & e\\
\hline
a & 0  &   &   &   &\\
b & 2  & 0 &   &   &\\
c & 6  & 5 & 0 &   & \\
d & 10 & 9 & 4 & 0 &  \\
e & 9  & 8 & 5 & 3 & 0\\
\end{tabular}
\end{minipage}
\begin{minipage}[c]{0.5\textwidth}
Start assembling details on the distances; we start as before
\begin{tabular}{ll}
Distance & Groups\\
\hline
0 & a b c d e\\
2 & (ab) c d e
\end{tabular}
\end{minipage}

However, the reduced distance matrix will be different:

\begin{eqnarray*}
d_{(ab)c)} = max(d_{ac},d_{bc}) = d_{ac} = 6\\
d_{(ab)d)} = max(d_{ad},d_{bd}) = d_{ad} = 10\\
d_{(ab)e)} = max(d_{ae},d_{be}) = d_{ae} = 9
\end{eqnarray*}

\begin{minipage}[c]{0.5\textwidth}
\begin{tabular}{r|ccccc}
 & (ab) & c & d & e\\
\hline
(ab) & 0  &   &   &  \\
c    & 6  & 0 &   &  \\
d    & 10 & 4 & 0 &  \\
e    & 9  & 5 & 3 & 0\\
\end{tabular}
\end{minipage}
\begin{minipage}[c]{0.5\textwidth}
Although the next step will be identical (we merge $d$ and $e$ at a height of 3)

\begin{tabular}{ll}
Distance & Groups\\
\hline
0 & a b c d e\\
2 & (ab) c d e\\
3 & (ab) c (de)
\end{tabular}
\end{minipage}

We now need to find the minimum distance from $d$ and $e$ to the other objects and reform the distance matrix:

\begin{minipage}[c]{0.5\textwidth}
\begin{tabular}{r|ccc}
 & (ab) & c & (de)\\
\hline
(ab) & 0 &   &    \\
c    & 6 & 0 &   \\
(de) & 10 & 5 & 0  \\
\end{tabular}
\end{minipage}
\begin{minipage}[c]{0.5\textwidth}
Although we are still going to merge $(de)$ and $c$ note that the height is different being 5.

\begin{tabular}{ll}
Distance & Groups\\
\hline
0 & a b c d e\\
2 & (ab) c d e\\
3 & (ab) c (de)\\
5 & (ab) (cde)
\end{tabular}
\end{minipage}

\begin{minipage}[c]{0.5\textwidth}
\begin{tabular}{r|cc}
 & (ab) & (cde)\\
\hline
(ab)  & 0   &     \\
(cde) & 10 &  0  \\
\end{tabular}
\end{minipage}
\begin{minipage}[c]{0.5\textwidth}
So our final merge will take place at height 10.

\begin{tabular}{ll}
Distance & Groups\\
\hline
0 & a b c d e\\
2 & (ab) c d e\\
3 & (ab) c (de)\\
5 & (ab) (cde)\\
10 & (abcde)
\end{tabular}
\end{minipage}

In this simple demonstration, the dendrogram, illustrated in the centre of figure \ref{simpleclust} obtained is similar in shape, but all the merges are at different height.

\subsection{Group average link}

This requires \texttt{agnes()} in package \texttt{cluster}, called with the \verb+method="average"+ instruction.

This time we merge two groups is the average distance between them is small enough.

\begin{minipage}[c]{0.5\textwidth}
\begin{tabular}{r|ccccc}
 & a & b & c & d & e\\
\hline
a & 0  &   &   &   &\\
b & 2  & 0 &   &   &\\
c & 6  & 5 & 0 &   & \\
d & 10 & 9 & 4 & 0 &  \\
e & 9  & 8 & 5 & 3 & 0\\
\end{tabular}
\end{minipage}
\begin{minipage}[c]{0.5\textwidth}
Start assembling details on the distances; again, we start as before:
\begin{tabular}{ll}
Distance & Groups\\
\hline
0 & a b c d e\\
2 & (ab) c d e
\end{tabular}
\end{minipage}

But the reduced distance matrix will be different again:

\begin{eqnarray*}
d_{(ab)c)} = (d_{ac} + d_{bc})/2  = 5.5\\
d_{(ab)d)} = (d_{ad} + d_{bd})/2  = 9.5\\
d_{(ab)e)} = (d_{ae}+ d_{be})/2  = 8.5
\end{eqnarray*}

\begin{minipage}[c]{0.5\textwidth}
\begin{tabular}{r|ccccc}
 & (ab) & c & d & e\\
\hline
(ab) & 0  &   &   &  \\
c    & 5.5  & 0 &   &  \\
d    & 9.5 & 4 & 0 &  \\
e    & 8.5  & 5 & 3 & 0\\
\end{tabular}
\end{minipage}
\begin{minipage}[c]{0.5\textwidth}
Yet again, the next merge step will be identical (we merge $d$ and $e$, only they are merged at height 3)

\begin{tabular}{ll}
Distance & Groups\\
\hline
0 & a b c d e\\
2 & (ab) c d e\\
3 & (ab) c (de)
\end{tabular}
\end{minipage}

We now need to find the minimum distance from d and e to the other objects and reform the distance matrix:

\begin{minipage}[c]{0.5\textwidth}
\begin{tabular}{r|ccc}
 & (ab) & c & (de)\\
\hline
(ab) & 0 &   &    \\
c    & 5.5 & 0 &   \\
(de) & 9 & 4.5 & 0  \\
\end{tabular}
\end{minipage}
\begin{minipage}[c]{0.5\textwidth}
Again, we still going to merge $(de)$ and $c$ note that the height is different ($4.5$)

\begin{tabular}{ll}
Distance & Groups\\
\hline
0 & a b c d e\\
2 & (ab) c d e\\
3 & (ab) c (de)\\
5 & (ab) (cde)
\end{tabular}
\end{minipage}

\begin{minipage}[c]{0.5\textwidth}
\begin{tabular}{r|cc}
 & (ab) & (cde)\\
\hline
(ab)  & 0   &     \\
(cde) & 7.8 &  0  \\
\end{tabular}
\end{minipage}
\begin{minipage}[c]{0.5\textwidth}
So our final merge will take place at height $7.8$.

\begin{tabular}{ll}
Distance & Groups\\
\hline
0 & a b c d e\\
2 & (ab) c d e\\
3 & (ab) c (de)\\
4.5 & (ab) (cde)\\
7.8 & (abcde)
\end{tabular}
\end{minipage}

In this simple demonstration, the dendrogram obtained is similar in shape, but all the merges are at different height.   This dendrogram is depicted on the right of figure \ref{simpleclust}.   Whilst the role of the dendrogram seems obvious, it should be acknowledged that there are some algorithmic details needed to determine how to present these.   We will consider other clustering methods shortly, but it should be noted that the centroid and median approaches can lead to reversals in the dendrogram.

\begin{figure}
\begin{center}
\includegraphics[width = 0.7\textwidth]{images/democlust}
\caption{Dendrograms from three basic cluster methods}
\label{simpleclust}
\end{center}
\end{figure}

%\begin{itemize}
%\item Ward's minimum variance method aims at finding compact, spherical clusters.
%\item Complete linkage method finds similar clusters.
%\item The single linkage method (which is closely related to the minimal spanning tree) adopts ``friends of friends'' clustering strategy.
%\item Other methods are somewhere between single and complete link methods.
%\end{itemize}

\subsection{Alternative methods for hierarchical cluster analysis}
\label{othermethods}

Given that cluster ``analysis'' is essentially an algorithmically guided exploratory data analysis it is perhaps no surprise, firstly that there have been many other methods proposed and secondly that there have been attempts to generalise the algorithm.   \cite{Lance+Williams:1966,Lance+Williams:1967} proposed a general recurrence formula which gives the distance between a newly amalgated group $C_{k} \bigcup C_{l}$ and some other group $C_{m}$:

\begin{equation}
d_{C_{k} \bigcup C_{l}, C_{m}} = \alpha_{l} d(C_{k}, C_{l}) + \alpha_{m} d(C_{k}, C_{m}) + \beta d(C_{k}, C_{l}) + \gamma \lvert d(C_{k}, C_{m}) -  d(C_{l}, C_{m})\rvert
\end{equation}
where $d_{C_{k} \bigcup C_{l}, C_{m}}$ is the distance between a cluster $C_{k}$ and the merging of two groups  $C_{l}$ and $C_{m}$.   The parameters are constrained such that $\alpha_{l} + \alpha_{m} + \beta = 1$, $\alpha_{l} = \alpha_{m}$, $\beta < 1$ and $\gamma = 0$.   Using these schema, a number of established agglomerative strategies can be expressed as follows:
\begin{tabular}{lrrrr}
Method & \R call &  $\alpha_{k}$ & $\beta$ & $\gamma$ \\
\hline
Single link (nearest neighbour) & \verb+method = "single"+ & $\frac{1}{2}$ & 0 & $-\frac{1}{2}$ \\
Complete link (furthest neighbour) &  \verb+method = "complete"+ & $\frac{1}{2}$ & 0 & $\frac{1}{2}$ \\
Group average link &  \verb+method = "average"+ & $N_{l}|(N_{l} + N_{m})$ & 0 & 0\\
Weighted average link &  \verb+method = "mcquitty"+ & $\frac{1}{2}$ & 0 & 0 \\
Centroid &   \verb+method = "centroid"+ & $N_{l}|(N_{l} + N_{m})$ &  $- N_{l} N_{m}|(N_{l} + N_{m})^2$ & 0\\
%Sum of squares &  \verb+method = ""+ &  $(N_{i} + N_{K})|(N_{i} + N_{j} + N_{k})$ & $-(N_{K} + N_{k})|(N_{i} + N_{j} + N_{K})$
Incremental sum of squares &  \verb+method = "ward"+ & $\frac{N_{k} + N_{m}}{N_{k} + N_{l} + N_{m}}$ &
$\frac{N_{k} + N_{l}}{N_{k} + N_{l} + N_{m}}$ & 0 \\
Median &  \verb+method = "median"+ & $\frac{1}{2}$ & $-\frac{1}{4}$ & 0\\
%Flexible & $\frac{1}{2}(1-\beta)$ & $\beta(<1)$ & 0\\
\end{tabular}
where $N_{k}, N_{l}\ \mbox{and}\ N_{m}$ are the cluster sizes when $C_{k}$ is joined to the other two clusters considered.

This formula facilitates the use of a number of clustering approaches.   \cite{Ward:1963} proposed a method in which clustering proceeds by selecting those merges which minimise the error sum of squares.   If we consider the cluster specific error sum of squares:
\begin{displaymath}
ESS_{k} = \sum_{i+1}^{n_{k}} \sum_{j=1}^{p} (x_{ki,j} - \bar{\boldsymbol{x}}_{k,j})^{2}
\end{displaymath}
where $ \bar{\boldsymbol{x}}_{k,j}$ is the mean of cluster $k$ with respect to variable $j$ and $x_{ki,j}$ is the value of $j$ for each object $i$ in cluster $k$.   The total error sum of squares is therefore given by $\sum_{k=1}^{K} ESS_{k}$ for all clusters $k$.   This method tends to give spherical clusters, whether that is a reasonable solution to find or not.

 Centroid clustering involves merging clusters with the most similar mean vectors; there is a subtle variation on a theme in which the centroid calculation is weighted.   Whilst the calculations underlying these methods tend to use Euclidean distance (to facilitate interpretation in terms of the raw data) that is not compulsory.   There is therefore some quite unsubtle interaction between choice of distance measure and choice of clustering algorithm which provides vast room for ambiguity in conducting cluster analysis.

\subsection{Problems with hierarchical cluster analysis}

A number of problems have been recognised with hierarchical cluster analysis.   Single, complete and average linkage as well as Ward's method have the potential for reversals in the dendrogram; single and complete linkage impose a monotonicity requirement.   One particular problem with single link clustering is ``chaining'', we could illustrate this as follows:
\singlespacing
\begin{verbatim}
x <- rbind(mvrnorm(30, c(0,0), sigma),
  mvrnorm(30, c(8,8), sigma),
  cbind(seq(0,8, by = 0.4), seq(0,8, by = 0.4) )
dot <- hclust(dist(x), method = "single")
par(mfrow = c(1,2))
plot(dot)
plot(x, pch = cutree(dot, k = 2))
\end{verbatim}
\onehalfspacing

\begin{figure}
\begin{center}
\includegraphics[width = 0.7\textwidth]{images/chaining}
\caption{Demonstration of ``chaining'' with single link clustering}
\label{chaining}
\end{center}
\end{figure}

\subsection{Hierarchical clustering in \R}

A typical example of a cluster analysis was reported by \cite{Jolliffe+etal:1986}. To carry out hierarchical cluster analysis in \R, create a distance object from the data (\verb+USArrests.dist+), create a hierarchical clustering object from the distance matrix (\verb+USArrests.hclust+), and plot the results.   Here we have used manhattan distance and compete linkage.  You may like to see how much difference the alternatives make.

\singlespacing
\begin{verbatim}
> USArrests.dist <- dist(USArrests, method = "manhattan")
> USArrests.hclust <- hclust(USArrests.dist, method = "complete")
> plot(USArrests.hclust)
\end{verbatim}
\onehalfspacing

\begin{figure}
\begin{center}
\includegraphics{images/hclust}
\caption{Dendrogram following complete linkage cluster analysis of US Arrests data}
\label{hclust}
\end{center}
\end{figure}

For comparison with other results, we may want to ``cut'' the dendrogram at a point which gives us a certain number of classes.   Here, we will cut the dendrogram fairly high up, and look at a possible five group structure.   In the following code, we use cutree to find the five groups, and produce draftsman plot with colours / symbols altering according to which of the five groups we think the state may belong to.

\singlespacing
\begin{verbatim}
> hc.class <- cutree(USArrests.hclust, k =5)
> plot(USArrests, col = hc.class, pch = hc.class)
\end{verbatim}
\onehalfspacing

An agreement is where both objects have a one in each position.   $b$ and $c$ denote the two possibly disagreements.

\section{Cophenetic Correlation}

The cophenetic correlation can be used as some kind of measure of the goodness of fit of a particular dendrogram.

\begin{equation}
\rho_{Cophenetic} = \frac{\sum_{i=1, j=1, i <j}^{n} (d_{ij} - \bar{d})(h_{ij} - \bar{h})}{
\left(\sum_{i=1, j=1, i <j}^{n} (d_{ij} - \bar{d})^{2}(h_{ij} - \bar{h})^{2} \right)^{0.5}}
\end{equation}

\singlespacing
\begin{verbatim}
> d1 <- dist(USArrests)
> h1 <- hclust(d1)
> d2 <- cophenetic(h1)
> cor(d1, d2)
[1] 0.7636926
> h1 <- hclust(d1, method = "single")
> d2 <- cophenetic(h1)
> cor(d1, d2)
[1] 0.5702505
\end{verbatim}
\onehalfspacing

So it is easy to obtain a measure of the cophenetic correlation, it is less clear what it means.   Certainly a value below 0.6 implies that there has been some distortion in the dendrogram.

\section{Divisive hierarchical clustering}
\label{aggclust}

Divisive clustering reverses the approach taken above.   Here, we start with one large cluster of all $n$ objects, and split until each object is unique.
[page 90] \cite{Gordon:1999} argues that divisive clustering is not likely to lead to optimal divisions in the data.   Arguably the more succesful methods are monothetic and split on one variable at each stage (see \verb+mona()+ for an R example which works with binary data).

\cite{MacNaughton-Smith+etal:1964} proposed one divisive method which has seen some use.   \cite{Kaufman+Rousseeuw:1989} liken this to splintering within a political party, where one person leaves and attracts the most like-minded people and have provided the \verb+diana()+ routine which facilitates this form of clustering.

To identify the ``splinter'', the object with the largest average dissmilarity to all other objects is selected.   Then the dissimilarities are recalculated, and any objects who are closer to the splinter than to their original group are reclassified.   Average dissimilaries can then be recalculated.

At each subsequent step of the algorithm, the cluster $C$ with the largest diameter is selected:

\begin{displaymath}
diam(C) = max_{i,j \in C} d(i,j)
\end{displaymath}

This diameter is represented as the ``height'' in the dendrogram and the banner plot.

Then $C$ is split into two subclusters, initially $A=C$ and $B=\emptyset$.   For each objects in $A$, calculate the average dissimilarity to all other objects in $A$, the object with the largest distance is moved to $B$.   This step is repeated until there are $n$ clusters.

\singlespacing
\begin{verbatim}
USArrests.dist <- dist(USArrests)
library(cluster)
USArrests.diana <- diana(USArrests.dist)
par(mfrow = c(1,2), oma = c(0,0,2,0))
plot(USArrests.diana, which = 2, cex = 0.6,
main = "Dendrogram", xlab = "")
plot(USArrests.diana, which = 1, main = "Banner plot",
nmax.lab = 50, cex.axis = 0.6, max.strlen = 12)
mtext("Divisive clustering of USArrests", outer = TRUE)
\end{verbatim}
\onehalfspacing

\begin{figure}
\begin{center}
\includegraphics[width = 0.8\textwidth]{images/diana}
\caption{Divisive clustering of USArrests data, dendrogram and bannerplot}
\label{diana}
\end{center}
\end{figure}

\section{K-means clustering}
\label{kmclust}

This is a rather different method of clustering, aimed at finding ``more homogenous'' subgroups within the data.   We specify at the start how many clusters we are looking for, and ideally provide some clue as to what might be in those clusters.

The technique was perhaps first proposed by \cite{Lloyd:1957} and somewhat codified by \cite{Forgy:1965}, early developments were also reported by \cite{MacQueen:1967}, although the default S-Plus algorithm was developed by \cite{Hartigan+Wong:1979}.   There is some confusion as to whether \emph{k-means} refers to a particular technique or a particular algorithm.

Given a number of $k$ starting points, the data are classified, the centroids recalculated and the process iterates until stable.

We could attempt to demonstrate this with the following function

\singlespacing
\begin{verbatim}
step <- function(X, mu1, mu2){
## classify according to current seed point (expectation step)
one <- sqrt(rowSums((X - t(t(t(mu1)) %*% t(ones)))^2))
two <- sqrt(rowSums((X - t(t(t(mu2)) %*% t(ones)))^2))
plot(x,y, col = 1 + as.numeric(one < two), pch = 16, xlim = xlims, ylim = ylims )
legend("topright", pch = c(16,16,2,3), col = c("red", "black"), legend = c("Group1", "Group2", "Seed 1", "Seed 2" ), cex = 0.5)
points(rbind(seed$mu1, seed$mu2), pch = c(2,3), col = c("red", "black"))
fixed <- (mu1 + mu2)/2
slope <- -(mu1[1] - mu2[1])/(mu1[2] - mu2[2])
abline(c(fixed[2] - slope * fixed[1], slope))

## calculate new seed points (maximisation step)
mu1 <- colMeans(X[one < two,])
mu2 <- colMeans(X[one >= two,])
return(seed = list(mu1 = mu1, mu2 = mu2))
}
\end{verbatim}
\onehalfspacing

And set up with:

\singlespacing
\begin{verbatim}
## simulate two clusters of data
x <- c(rnorm(20,0,1), rnorm(20,4,1))
y <- c(rnorm(20,0,1), rnorm(20,4,1))
X <- cbind(x,y)
ones <- matrix(1, dim(X)[1],1)

## set up some parameters for plotting
xlims <- c(min(x), max(x)) * 1.3
ylims <- c(min(y), max(y)) * 1.3

## plot the data
par(mfrow = c(2,2))
plot(X, xlim = xlims, ylim = ylims)

## And add some very silly seed points
mu1 <- c(0,6)
mu2 <- c(5,-2)
seed <- list(mu1 = mu1, mu2 = mu2)
points(rbind(seed$mu1, seed$mu2), pch = c(2,3),col = c("red", "black"))
legend("topright", pch = c(1,2,3), col = c("black", "red", "black"), legend = c("Data", "Seed 1", "Seed 2" ), cex = 0.5)
##mtext(paste("Seed points: \n Group 1 ", formatC(seed[[1]],2), "Group 2 ", formatC(seed[[2]],2)))

seed <- step(X, seed$mu1, seed$mu2)
seed <- step(X, seed$mu1, seed$mu2)
seed <- step(X, seed$mu1, seed$mu2)
\end{verbatim}
\onehalfspacing

\begin{figure}
\begin{center}
\includegraphics[width = 0.7\textwidth]{images/kmeansdemo}
\label{kmeansdemo}
\caption{Demonstration of kmeans algorithm, points are classified according to seed, then position of seed is recalculated}
\end{center}
\end{figure}

Having illustrated the basic principles it is quite easy to run the analysis:

\singlespacing
\begin{verbatim}
> US.km <- kmeans(USArrests, centers = 2)
> plot(USArrests, col = US.km$cluster, pch = US.km$cluster) ## not shown
> plot(prcomp(USArrests, center = TRUE)$x[,c(1,2)],
col = US.km$cluster, pch = US.km$cluster)

\end{verbatim}
\onehalfspacing
For interest, we will compare the $k$-means solution with the \verb+diana()+ classification:

\singlespacing
\begin{verbatim}
 kmclass <- as.vector(US.km$cluster)
> diana.class <- cutree(USArrests.diana, k = 2)
> xtabs(~kmclass + diana.class)
       diana.class
kmclass  1  2
      1  0 29
      2 21  0
\end{verbatim}
\onehalfspacing

So in this particular example, there is good agreement that there may be two clusters in the data.

\begin{figure}
\begin{center}
\includegraphics[width = 0.6\textwidth]{images/km2classpca}
\caption{Scatterplot of original variables denoting cluster membership for $k=2$ means clustering}
\label{hclust}
\end{center}
\end{figure}

\subsection{Partitioning around medoids}

This approach to clustering is laid out in \cite{Kaufman+Rousseeuw:1989}, and the S-PLUS implementation is very well described in \cite{Struyf+etal:1997}.   The procedure is to find $k$ \emph{representative objects} called medoids in such a way that the total dissimilarity of all objects to their nearest medoids is minimised.   One side-effect of this approach is that a data entity is identifed as a \emph{representative object} which may facilitate description rather better than a non-existing group centroid.   In other words, we identify objects that ``best'' represent their groups.   This is quite simple to do in \R, we can supply either the data matrix or a distance matrix to \verb+pam()+, below we supply the data:
\singlespacing
\begin{verbatim}
> USArrests.pam <- pam(USArrests, k = 2)
> par(mfrow = c(1,2))
> plot(USArrests.pam)
> USArrest.pam$medoids
         Murder Assault UrbanPop Rape
Michigan   12.1     255       74 35.1
Kansas      6.0     115       66 18.0
\end{verbatim}
\onehalfspacing
and it can be seen that Michigan and Kansas are in some sense the best representative examples of each of the two groups we have chosen to investigate.   The \verb+plot()+ method applied to pam objects produces a $2 \times 2$ scatterplot of the first two principal components identifying group memberships and superimposing ellipses as well as a silhouette.

\begin{figure}
\begin{center}
\includegraphics[width = 0.8\textwidth]{images/pam}
\caption{Partitioning around medoids}
\label{pam}
\end{center}
\end{figure}

With minor modification, this is a tractable algorithm for large datasets.   A wrapper, \verb+clara()+ has been written which makes partitioning around mediods available for a wide range of datasets.

\subsection{Hybrid Algorithms}

There have been a number of proposals in the literature recently for hybrid algrithms.   We consider HOPACH, essentially it is a divisive algorithm, but at each stage a merging step is incorporated to bring together similar clusters (which may be on different branches of the tree).  Again, this method produces medoids, in this case 13 were identified.

\singlespacing
\begin{verbatim}
> library(hopach)
> USArrests.hopach <- hopach(USArrests)
> row.names(USArrests)[USArrests.hopach$clustering$medoids]
 [1] "Mississippi"   "Alaska"        "New Mexico"    "Michigan"
 [5] "California"    "Illinois"      "Missouri"      "West Virginia"
 [9] "Rhode Island"  "Nebraska"      "New Hampshire" "Wisconsin"
[13] "Hawaii"
> dplot(dist(USArrests), us, labels = row.names(USArrests), main = "HOPACH of USArrests")
\end{verbatim}
\onehalfspacing

The \verb+dplot()+ method requires input of a distnce matrix and produces a heatmap of the distances ordered by the cluster structure.   Silhouette plots are also available, and further plot methodology is available from programs external to \R.

\begin{figure}
\begin{center}
\includegraphics[width = 0.7\textwidth]{images/hopach}
\caption{Heatmap of distances organised according to output of HOPACH with cluster groups denoted by dashed lines}
\label{hopach}
\end{center}
\end{figure}

\section{K-centroids}

More recent proposals in the literature have tried to remove the dependency on algorithmic dependency.   I need to add something on this shortly.

\section{Further information}

It is worth repeating that ``cluster analysis'' is a poorly delimited set of methods for unsupervised classification.   Obvious omissions include fuzzy analysis (where we accept some uncertainty in group assignments), monothetic analysis (where we split based on binary variables one at a time).   All these merit further consideration.   However, it is acknowledged that the greatest omission at present is in terms of mixture modelling, much more comprehensive information is available in \cite{McLachlan+Peel:2000}, some coverage is also given in \cite{Flury:1997}.

%%% Local Variables: ***
%%% mode:latex ***
%%% TeX-master: "../book.tex"  ***
%%% End: ***


\documentclass{article}
\usepackage{times,amsmath,amsthm,amsfonts,eucal,graphicx, amssymb}
\setlength{\oddsidemargin}{0.25 in}
\setlength{\evensidemargin}{-0.25 in}
\setlength{\topmargin}{-0.6 in}
\setlength{\textwidth}{6.5 in}
\setlength{\textheight}{8.5 in}
\setlength{\headsep}{0.75 in}
\setlength{\parindent}{0 in}
\setlength{\parskip}{0.1 in}

\newcounter{lecnum}
\renewcommand{\thepage}{\thelecnum-\arabic{page}}
\renewcommand{\thesection}{\thelecnum.\arabic{section}}
\renewcommand{\theequation}{\thelecnum.\arabic{equation}}
\renewcommand{\thefigure}{\thelecnum.\arabic{figure}}
\renewcommand{\thetable}{\thelecnum.\arabic{table}}

\newcommand{\indep}{{\bot\negthickspace\negthickspace\bot}}
\newcommand{\notindep}{{\not\negthickspace\negthinspace{\bot\negthickspace\negthickspace\bot}}}
\newcommand{\definedtobe}{\stackrel{\Delta}{=}}
\renewcommand{\choose}[2]{{{#1}\atopwithdelims(){#2}}}
\newcommand{\argmax}[1]{{\hbox{$\underset{#1}{\mbox{argmax}}\;$}}}
\newcommand{\argmin}[1]{{\hbox{$\underset{#1}{\mbox{argmin}}\;$}}}

%
% The following macro is used to generate the header.
%
\newcommand{\lecture}[4]{
   \pagestyle{myheadings}
   \thispagestyle{plain}
   \newpage
   \setcounter{lecnum}{#1}
   \setcounter{page}{1}
   \noindent
   \begin{center}
   \framebox{
      \vbox{\vspace{2mm}
    \hbox to 6.58in { {\bf EE512~Stochastic Processes
                        \hfill University of Southern California} }
    \hbox to 6.58in { {\bf Fall 2016
                        \hfill Dept. of Electrical Engineering} }
       \vspace{4mm}
       \hbox to 6.28in { {\Large \hfill Lecture #1: #2  \hfill} }
       \vspace{2mm}
       \hbox to 6.28in { {\it Lecturer: {\it Prof: Nayyar {\tt <>}} \hfill Scribe: #3} }
      \vspace{2mm}}
   }
   \end{center}
   \markboth{Lecture #1: #2}{Lecture #1: #2}
   \vspace*{4mm}
}

%
% Convention for citations is authors' initials followed by the year.
% For example, to cite a paper by Leighton and Maggs you would type
% \cite{LM89}, and to cite a paper by Strassen you would type \cite{S69}.
% (To avoid bibliography problems, for now we redefine the \cite command.)
% Also commands that create a suitable format for the reference list.
\renewcommand{\cite}[1]{[#1]}
\def\beginrefs{\begin{list}%
        {[\arabic{equation}]}{\usecounter{equation}
         \setlength{\leftmargin}{2.0truecm}\setlength{\labelsep}{0.4truecm}%
         \setlength{\labelwidth}{1.6truecm}}}
\def\endrefs{\end{list}}
\def\bibentry#1{\item[\hbox{[#1]}]}

%Use this command for a figure; it puts a figure in wherever you want it.
%usage: \fig{NUMBER}{CAPTION}{.eps FILE TO INCLUDE}{WIDTH-IN-INCHES}
\newcommand{\fig}[4]{
			\begin{center}
	                \includegraphics[width=#4,clip=true]{#3} \\
			Figure \thelecnum.#1:~#2
			\end{center}
	}
% Use these for theorems, lemmas, proofs, etc.
\newtheorem{theorem}{Theorem}[lecnum]
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{definition}[theorem]{Definition}
% \newenvironment{proof}{{\bf Proof:}}{\hfill\rule{2mm}{2mm}}

% **** IF YOU WANT TO DEFINE ADDITIONAL MACROS FOR YOURSELF, PUT THEM HERE:

\begin{document}
%FILL IN THE RIGHT INFO.
%\lecture{**LECTURE-NUMBER**}{**DATE**}{**LECTURER**}{**SCRIBE**}
\lecture{2}{Aug 25, 2016}{Saket Choudhary}
%\footnotetext{These notes are partially based on those of Nigel Mansell.}

% **** YOUR NOTES GO HERE:

% Some general latex examples and examples making use of the
% macros follow.
%**** IN GENERAL, BE BRIEF AND COMPLETE.
% **** THIS ENDS THE EXAMPLES. DON'T DELETE THE FOLLOWING LINE:

%\section{}
Independent RV $P(X=x,Y=y) = P(X=x)P(Y=y)$i or $P(X \in A, Y \in B) = P(X \in A)P(Y \in B)$

$E[X] = \sum_x xP(X=x)$ or $E[X] = \int xf(x)dx$

$Variance = E[(X-\mu)^2] = E[X^2]-(E[X])^2$

$Cov(X,Y) = E[(X-\mu_X)(Y-\mu_Y)] = E[XY]-E[X]E[Y]$

Independence $\implies$ Cov = 0

Cov =0 does not $\implies$ Indepennce.
Example: $X= \{-1,0,1\}$ and $Y=X^2$

Indicator variable: $E[I_x] = P(I_x=1)$

Moment generating function: $\psi(t) = E[e^{tX}]$

$\frac{d\psi(t)}{dt} = \frac{d}{dt} \int  e^{tx} f(x) dx =  \int xe^{tx} f(x) dx$
Thus, $\psi'(0) = E[X]$

$\psi''(t) = E[X^2]$

$\psi^n(t) = E[X^n]$

Example: Two gaussian Rv. $X,Y$, $Z=X+Y$
$M_Z = E[e^{tX+tY}] = E[e^{tX}]E[e^{tY}] = \psi_X(t) \psi_Y(t) = e^{\mu_xt + \frac{1}{2} \sigma_x^2} \times e^{\mu_yt+\frac{1}{2}\sigma_y^2}
= e^{\mu_xt+\mu_yt + \frac{1}{2}(\sigma_x^2+\sigma_y^2) }$

Joint MGF: $\psi(t_1, t_2, \dots, t_n) = E[e^{\sum_i t_i X_i}]$

\section{Conditional distrbituion and conditional expectation}

$E[X_1+X_2|Y=y] = E[X_1|Y=y] + E[X_2|Y=y]$

\section{Conditional expectation as r.v}

Y - discrete r.v.

X - discerte r.v

$E[X|Y=y_1] = \sum x P(X|Y=y_1)$

$h(y_i) = E[X|Y=y_i]$ so h(Y) is like a r.v.

$E[h(y)] = \sum_i h(y_i)P(Y=y_i) = \sum E[X|Y=y_i]P(Y=y_i)  = \sum_i \sum_j x_j P(X=x_j|Y=y_i)P(Y=y_i) = \sum_i \sum_j x_j P(X=x_j, Y=y_j)
= \sum_j \sum_i x_j P(X=x_j)$

Smoothing property: $E[E[X|Y]] = EX$

$E[E[X|Y]] = \int E[X|Y=y] f(y) dy$

  /---\  1
 A --- B 2
  \---/  3

$P(Z=i)=p_i$
$E[D|Z=i] = d_i$ $\implies$ $E[D] = E[E[D|Z]] = \sum p_id_i$

\section{Probability inequalities}

Markov's inequality:
$X$ is non negative RV $X \geq 0$ then for $a > 0$, $P(X \geq a) \leq E[X]/a$

Proof. $E[X] = \int_0^{\infty} xf(x)dx = \int_{0}^a xf(x)dx + \int_a^{\infty} xf(x)dx \geq \int_{a}^{\infty} xf(x) dx \geq aP(X\geq a) $
Alter. $X \geq a I_A$

Chebychev's Inequality:
$X$ is r.v. with mean $\mu$ and $var =\sigma^2$

$P(|X-\mu| \geq a) \leq \sigma^2 a^2$

Proof: $Y=(X-\mu)^2$ $P(Y \geq a^2) \leq EY/a^2$

Jensen's Ineuqlity:

Convex function: $f(\lambda x + (1-\lambda)y) \leq \lambda f(x) + (1-\lambda) f(y)$

If $f(x)$ is convex then $E[f(x)] \geq f(E[X])$

\end{document}


\documentclass[11pt]{article}

\textheight 22cm
\textwidth  16cm
\hoffset= -0.6in
\voffset= -0.5in
\setlength{\parindent}{0cm}
\setlength{\parskip}{10pt plus 2pt minus 2pt}
\pagenumbering{roman}
\setcounter{page}{-9}

\newcommand{\be}{\begin{equation}}
\newcommand{\ee}{\end{equation}}
\newcommand{\bd}{\begin{displaymath}}
\newcommand{\ed}{\end{displaymath}}
\begin{document}

\title{Astrophysics II: Laboratory 5}
\author{\Large Cluster HR Diagrams}
\maketitle

\baselineskip 15pt

\section{\bf Objectives}
\begin{itemize}
\item Use SDSS skyserver to download SDSS data for a cluster.
\item Plot a cluster HR diagram in Matlab.
\item Compare cluster HR diagrams to determine relative age.
\end{itemize}

\section{\bf Procedure:}

Part I:\\
You will be using data from the Sloan Digital Sky Survey to measure
the magnitudes of stars in a star cluster in different filters.  A
handy tool for visualizing the public Sloan data is at the website {\it
  http://skyserver.sdss.org/dr1/en/tools/chart/navi.asp}.  Your task
will be to plot the magnitude in the r filter versus the difference
between the g and r magnitudes to construct an HR diagram using Matlab.

1 - Load the website using a web browser available on your computer
and also open matlab, which you will be using for plotting the data you collect from the images at this website.

2 - Set the viewer (on the website) to display the coordinates - RA:
229.0128, Dec: -0.1082.

3 - Select {\it InvertImage} in the lower left hand corner, this
should make it easier to see the fainter objects. It does not particularly matter which way you view the field.

4 - Compare the current field, containing the star cluster, to another
field.  Change the declination by 0.3$^o$.  What differences do you
see between the fields?

5 - Now, move back to the original coordinates.

6 - Use the controls on the left to Zoom Out 3 times; the star cluster
should now become apparent. Take note of what you see.

7 - Return to the original Zoom of the field.  Then, try left clicking
on an object in the field, you will be given the coordinates, type
(star or galaxy), and the magnitude of the object in 5 separate
filters, in the upper right hand corner.

8 - Now you want to collect the color magnitudes for as many of the stars in this cluster as possible. In the top left corner of the webpage click on the SDSS logo to link to the SkyServer homepage. On the left side of the page select SEARCH $-->$ Radial. Fill in the coordinates of the cluster given above in RA and dec and approximate the radius of the cluster (being conservative here is fine). Do not check any of the boxes beside the filter selections, and set the min and max magnitudes to something like 0 and 30 (whatever you set the limits to, be sure to try to avoid selecting foreground stars and select stars with magnitude greater than $\sim$24). Return all entries; Format as CSV. SUBMIT. You should save the file to the desktop and name it something like cluster1.dat

Import this data using the GUI you can select on the File menu in Matlab, specifying `,' delimeter and 1 row of header. Matlab will create three variables but only the one which is a matrix containing the data of the file is important here.

9 - You can now create an HR Diagram plotting r on the Y axis and g-r on the X axis. To do this you must first find out which column of the data matrix is which. Double click on the `colheaders' variable in the Workspace window to open it in the Variable Editor. Each column here contains the label matching the data in the same column in the data matrix. (g: 11, r: 12).

Make sure you label each axis correctly and that the values run in the
correct direction. Consult http://www.astro.umd.edu/~mavara/matlab/HR.html for an example of how to reverse an axis. Also, title your graph with the coordinates of
the cluster.  Why are we plotting r and g-r on their respective axis?
What physical properties does each value represent?

10 - Draw a sketch of what you see. On your HR diagram, mark where the Main Sequence lies as well as any other apparent features.

12 - The lab instructions ask you to use the g and r magnitudes only,
but the SDSS measure magnitudes through 5 separate filters.  Could
other combinations have been used?  Why?

Part II:\\

You will now be use the magnitudes of
stars in a different star cluster to plot another HR diagram to compare.

1 - This time use coordinates RA: 151.3801, Dec: 0.072. Try viewing this cluster. Note that it is fainter and smaller on the sky.

2 - To acquire a measurement of a particular star, left click on a star and note its particular magnitudes, etc.

3 - Follow the same procedure as in Part I as many stars as possible and plot the HR diagram of r vs g-r.

{\bf Questions}

1 - Why are the plots you generate equivalent to HR diagrams, which
are normally plots of luminosity versus temperature? What are you
assuming about all the stars to make this analogy work?

2 - What color are the brightest stars on the Main Sequence in each
cluster (are they bluer or redder)?  What does this tell you about the
age of the clusters?  Which is older? Can you explain your answer?

3 - What type of cluster is each, globular or open?  Why?

4 - Suppose you were told that the brightest stars on the Main
Sequence of the first cluster (from Part I) had an absolute magnitude
in r of 6.  How far is the cluster from Earth in parsecs?  Show your work.

\end{document}


% !TEX TS-program = pdflatex
% !TEX encoding = UTF-8 Unicode

% This is a simple template for a LaTeX document using the "article" class.
% See "book", "report", "letter" for other types of document.

\documentclass[10pt,twocolumn]{article}

\usepackage[utf8]{inputenc} % set input encoding (not needed with XeLaTeX)
\usepackage{graphicx}
\usepackage{listings}
\usepackage{xcolor,colortbl}
\usepackage[section]{placeins}
\usepackage{amsthm}
\usepackage{mathtools}
\graphicspath{ {images/} }

%%% Examples of Article customizations
% These packages are optional, depending whether you want the features they provide.
% See the LaTeX Companion or other references for full information.

%%% PAGE DIMENSIONS
\usepackage{geometry} % to change the page dimensions
\geometry{a4paper} % or letterpaper (US) or a5paper or....
\geometry{margin=0.55in} % for example, change the margins to 2 inches all round
% \geometry{landscape} % set up the page for landscape
%   read geometry.pdf for detailed page layout information

\usepackage{graphicx} % support the \includegraphics command and options

% \usepackage[parfill]{parskip} % Activate to begin paragraphs with an empty line rather than an indent

%%% PACKAGES
\usepackage{booktabs} % for much better looking tables
\usepackage{array} % for better arrays (eg matrices) in maths
\usepackage{paralist} % very flexible & customisable lists (eg. enumerate/itemize, etc.)
\usepackage{verbatim} % adds environment for commenting out blocks of text & for better verbatim
\usepackage{subfig} % make it possible to include more than one captioned figure/table in a single float
\usepackage{indentfirst}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{tabularx}
% These packages are all incorporated in the memoir class to one degree or another...

%%% HEADERS & FOOTERS
\usepackage{fancyhdr} % This should be set AFTER setting up the page geometry
\pagestyle{fancy} % options: empty , plain , fancy
\renewcommand{\headrulewidth}{0pt} % customise the layout...
\lhead{}\chead{}\rhead{}
\lfoot{}\cfoot{\thepage}\rfoot{}

%%% SECTION TITLE APPEARANCE
\usepackage{sectsty}
\allsectionsfont{\sffamily\mdseries\upshape} % (See the fntguide.pdf for font help)
% (This matches ConTeXt defaults)

%%% ToC (table of contents) APPEARANCE
\usepackage[nottoc,notlof,notlot]{tocbibind} % Put the bibliography in the ToC
\usepackage[titles,subfigure]{tocloft} % Alter the style of the Table of Contents
\renewcommand{\cftsecfont}{\rmfamily\mdseries\upshape}
\renewcommand{\cftsecpagefont}{\rmfamily\mdseries\upshape} % No bold!
\setlength{\parindent}{0.5cm}

%%% END Article customizations

%%% The "real" document content comes below...

\title{Redundant Primes In Goldbach Partitions}
\author{Marcin Barylski}
\date{\small{Published: October 13, 2019 \\ The last update: July 1, 2020}}

\definecolor{Gray}{gray}{0.85}
\definecolor{LightCyan}{rgb}{0.88,1,1}
\newcolumntype{a}{>{\columncolor{Gray}}c}
\newcolumntype{b}{>{\columncolor{white}}c}

\newtheorem{theorem}{Theorem}
\newtheorem{lemma}[theorem]{Lemma}

\newcommand\bigforall{\mbox{\huge $\mathsurround0pt\forall$}}
\newcommand\bigexists{\mbox{\huge $\mathsurround0pt\exists$}}

\begin{document}
\maketitle

\begin{abstract}
Goldbach Strong Conjecture ($GSC$), still unsolved, states that all even integers $n \textgreater 2$ can be expressed as the sum of two prime numbers (Goldbach partitions of $n$). But do we need all primes to satisfy this conjecture? This work is devoted to selection of must-have primes and formulation of stronger version of $GSC$ with reduced set of primes.
\end{abstract}

\section{Problem statement}

Goldbach Strong Conjecture ($GSC$, also called binary) asserts that all positive even integer $n$ $\geq$ 4 can be expressed as the sum of two prime numbers. This hypothesis, formulated by Goldbach in 1742 in letter to Euler \cite{goldbach1742} and then updated by Euler to the form above is one of the oldest and still unsolved problems in number theory. Empirical verification showed that it is true for all $n$ $\leq$ 4 x $10^{18}$ \cite{oliveira2012} \cite{oliveira2013}.\par
The expression of a given positive even number $n$ as a sum of two primes $p_1$ and $p_2$ is called a Goldbach Partition ($GP$) of $n$.  Let's denote this relation as $GSC(n, p_1, p_2)$. Then $GSC$ can be written as (1):

\begin{equation}
\displaystyle\mathop{\bigforall}_{x \textgreater 1, x \in \mathbb{N}} \displaystyle\mathop{\bigexists}_{p_1, p_2 \in \mathbb{P}} GSC (2x, p_1, p_2)
\end{equation}

But maybe we can formulate much stronger version of (1)? The set of prime numbers $\mathbb{P}$ is dense, number of $GP(n)$ is increasing with increasing $n$, thus question if stronger version $GSC$ is possible, raised in \cite{barylski2018}, is legitimate (2):

\begin{equation}
\displaystyle\mathop{\bigforall}_{x \textgreater 1, x \in \mathbb{N}} \displaystyle\mathop{\bigexists}_{p_1, p_2 \in \mathbb{R}} GSC (2x, p_1, p_2)
\end{equation}

where $\left\vert \mathbb{R} \right\vert \textless \left\vert \mathbb{P} \right\vert$. By design $\mathbb{R}$ contains primes only.

\section{Algorithm}

Elimination of primes must start from the fact that every prime is potentially required (Lemma 1).

\begin{lemma}
Every prime is present is some Goldbach partition of even number.
\end{lemma}
\begin{proof}
Let's assume that exists such prime $p$ that is not present in any Goldbach partition - we will show that this is not possible. $p + p$ is always even number, regardless $p$ is even or odd. This means that we have $GSC(2p, p, p)$ for any prime $p$ (in other words: $p$ is always in Goldbach partition of $2p$) what is in contrary to the initial assumption.
\end{proof}

Further elimination of primes from $\mathbb{P}$ to build $\mathbb{R}$ requires appropriate algorithm $A$ which is able to resign from a given prime, even if it is present in some $GPs$. Such algorithm $A$ could look as follows:

\begin{enumerate}
\item  Let $\mathbb{R}$ is empty, a set of $\mathbb{R}_i$ is empty and $n = 2$. Let's assume that we break calculations at $n = n_{max} \textgreater 2$.
\item  \textbf{New turn}: $n = n + 2$.
\item Break calculations if $n \textgreater n_{max}$.
\item  If  $\mathbb{R}$ is sufficient to fulfill GSC for all even numbers $q$ ($ 4 \leq q \leq n$), then go to \textbf{New turn}.
\item  If  we have a set of  $\mathbb{R}_i$ and any $\mathbb{R}_j$ belonging to this set is sufficient to fulfill GSC for all even numbers $q$ ($ 4 \leq q \leq n$), then $\mathbb{R} = \mathbb{R}_j$, we forget all $\mathbb{R}_i$ and go to \textbf{New turn}.
\item  If  not, find all $GP(n)$ and build as many candidates for $\mathbb{R}$ (let's call them $\mathbb{R}_i$) as required. As a base use either (as  a first choice) $\mathbb{R}$ (if $\mathbb{R}_i$ does not exist) or all previous $\mathbb{R}_i$ (if they are present).
\item Go to \textbf{New turn}.
\end{enumerate}

\section{Results}

Table 1 presents the very first rounds of algorithm eliminating primes required to satisfy $GSC$ - it is depicting current values of both $\mathbb{R}$ and $\mathbb{R}_i$, plus additional set $\mathbb{E}$ which contains primes that were present in $GPs$ so far but can be eliminated without hurting $GSC$.

\begin{table}[ht]
\centering
\captionsetup{justification=centering}
\caption{Results of the first rounds of $A$}
\label{tablealgostart}
\begin{tabular}{lalalalal}
  \rowcolor{LightCyan}
  $n$ & $GP(n)$ & $\mathbb{R}$ & $\mathbb{R}_i$ & $\mathbb{E}$ \\
  \hline
4  & 2+2  & \{2\} & $\emptyset$ & $\emptyset$\\
  \hline
6  & 3+3 & \{2,3\} & $\emptyset$ & $\emptyset$\\
  \hline
8  & 3+5 & \{2,3,5\} & $\emptyset$ & $\emptyset$ \\
  \hline
10  & \begin{tabular}[c]{@{}l@{}}3+7\\5+5\end{tabular} & \{2,3,5\} & $\emptyset$ & \{7\} \\
  \hline
12  & 5+7 & \{2,3,5,7\} & $\emptyset$ & $\emptyset$ \\
  \hline
14  & \begin{tabular}[c]{@{}l@{}}3+11\\7+7\end{tabular} & \{2,3,5,7\} & $\emptyset$ & \{11\} \\
  \hline
16 & \begin{tabular}[c]{@{}l@{}}5+11\\3+13\end{tabular} &  $\emptyset$ & \begin{tabular}[c]{@{}l@{}}\{2,3,5,7,11\}\\\{2,3,5,7,13\}\end{tabular} & $\emptyset$ \\
  \hline
18 & \begin{tabular}[c]{@{}l@{}}5+13\\7+11\end{tabular} &  $\emptyset$ & \begin{tabular}[c]{@{}l@{}}\{2,3,5,7,11\}\\\{2,3,5,7,13\}\end{tabular} & $\emptyset$ \\
  \hline
20 & \begin{tabular}[c]{@{}l@{}}3+17\\7+13\end{tabular} & \{2,3,5,7,13\} & $\emptyset$ & \{11,17\} \\
  \hline
\end{tabular}
\end{table}

For instance, if we take into account all positive even numbers $2 \textless n \textless 16$, then we need a set of primes $\{2,3,5,7\}$ to satisfy $GSC$ for all $n$ checked so far and $\{11\}$ ($GSC(14, 3, 11)$) is our candidate for elimination.

\begin{figure}[!ht]
\centering
\captionsetup{justification=centering}
\includegraphics[width=9cm]{f_required_primes}
\caption[caption]{Number of required primes for GSC ($4 \leq n \leq 2.5 \times 10^4)$}
\label{fig:requiredprimes}
\end{figure}

Figures 1, 2 and 3 are depicting findings after analyzing even numbers $4 \leq n \leq 2.5 \times 10^4$. First of all, function of required primes increases but slower than $\Pi(n)$.

\begin{figure}[!ht]
\centering
\captionsetup{justification=centering}
\includegraphics[width=9cm]{f_number_of_possible_sets}
\caption[caption]{Number of $\mathbb{R}_i$ ($4 \leq n \leq 2.5 \times 10^4$)}
\label{fig:numberofsets}
\end{figure}

It is also interesting that algorithm $A$ has sporadic congestions in terms of number of $\mathbb{R}_i$ - generally its number at a given time is low but there are quite frequent situation when it is high, even exceeding 160000 (this means that we have 160000 subsets containing candidates for $\mathbb{R}$) - it happens when there are still some $\mathbb{R}_i$ and new $n$ requires new prime to be used which is multiplying number of $\mathbb{R}_i$ in the next round of $A$. Surprisingly, shortly after number of $\mathbb{R}_i$ is decreasing to much smaller values. Congestions could be better handled (from memory utilization standpoint) if in $A$ instead of seperate lists (paths) we have a tree (where branches/leaves represent variable part added on top of a common base).

\begin{figure}[!ht]
\centering
\captionsetup{justification=centering}
\includegraphics[width=9cm]{f_number_of_eliminated_primes}
\caption[caption]{Number of eliminated primes ($4 \leq n \leq 2.5 \times 10^4$)}
\label{fig:numberofeliminatedprimes}
\end{figure}

\begin{figure}[!ht]
\centering
\captionsetup{justification=centering}
\includegraphics[width=9cm]{f_eliminated_primes_to_partitions}
\caption[caption]{Ratio of number of eliminated primes to number of GPs, including average value ($4 \leq n \leq 2.5 \times 10^4$)}
\label{fig:numberofeliminatedprimestopartitions}
\end{figure}

\begin{figure}[!ht]
\centering
\captionsetup{justification=centering}
\includegraphics[width=9cm]{f_eliminated_primes_to_req_primes}
\caption[caption]{Ratio of number of eliminated primes to number of required primes, including average value ($4 \leq n \leq 2.5 \times 10^4$)}
\label{fig:numberofeliminatedprimestorequiredprimes}
\end{figure}

\section{Summary and next steps}

Executed experiments, run for all even $n \leq 2.5 \times 10^4$, confirmed that we do not need entire set of primes to satisfy $GSC$. Appendix A lists eliminated primes after checking all even $n \leq 2.5 \times 10^4$ - the smallest eliminated prime is $17$. Of course, exercised cases do not proof that eventually such set exists for all even $n$ but observed trends (Figure $\ref{fig:numberofeliminatedprimes}$, Figure $\ref{fig:numberofeliminatedprimestopartitions}$,  Figure $\ref{fig:numberofeliminatedprimestorequiredprimes}$) give strong foundation that such set exists indeed and conjecture (2) is true. \par
As a result of this work one integer sequence has been submitted to OEIS database: A328179 \cite{A328179}.

\begin{thebibliography}{9}
\bibitem{goldbach1742}
  Christian Goldbach,
  \emph{On the margin of a letter to Leonard Euler},
  1742.
\bibitem{oliveira2012}
  Tomás Oliveira e Silva,
  \emph{Goldbach conjecture verification.}
  http://sweet.ua.pt/tos/goldbach.html,
  2012.
\bibitem{oliveira2013}
  Tomás Oliveira e Silva, Siegfried Herzog, and Silvio Pardi,
  \emph{Empirical verification of the even Goldbach conjecture and computation of prime gaps up to 4 $\times 10^{18}$.},
  Mathematics of Computation, vol. 83, no. 288, pp. 2033-2060,
  July 2014 (published electronically on November 18, 2013).
\bibitem{barylski2018}
  Marcin Barylski
  \emph{On $6k \pm 1$ Primes in Goldbach Strong Conjecture.},
  February 2018.
 \bibitem{A328179}
  OEIS Foundation Inc. (2019), The On-Line Encyclopedia of Integer Sequences, https://oeis.org/A328179. Number of distinct primes required to satisfy the Strong Goldbach Conjecture for all even numbers $\leq 2n$.

\appendix
\section{List of redundant primes}

Based on empirical verification done for all even numbers $4 \leq n \leq 2.5 \times 10^4$:

[17, 29, 43, 67, 71, 73, 83, 89, 103, 107, 131, 137, 157, 163, 167, 173, 179, 181, 193, 199, 229, 239, 241, 251, 257, 263, 269, 271, 277, 281, 283, 313, 317, 331, 349, 353, 359, 367, 401, 421, 431, 433, 443, 449, 463, 467, 479, 487, 499, 503, 509, 521, 523, 547, 557, 563, 571, 577, 587, 599, 601, 607, 617, 619, 631, 641, 643, 647, 659, 673, 677, 683, 691, 709, 719, 727, 733, 739, 751, 757, 761, 769, 773, 787, 797, 809, 811, 827, 829, 839, 853, 857, 859, 863, 883, 907, 911, 919, 937, 941, 947, 953, 967, 971, 977, 997, 1009, 1013, 1019, 1031, 1039, 1049, 1061, 1087, 1093, 1097, 1103, 1109, 1123, 1129, 1151, 1153, 1163, 1171, 1181, 1187, 1201, 1223, 1229, 1231, 1249, 1259, 1277, 1279, 1283, 1289, 1297, 1301, 1307, 1361, 1367, 1373, 1381, 1399, 1409, 1427, 1429, 1433, 1439, 1447, 1453, 1459, 1471, 1481, 1487, 1489, 1511, 1523, 1549, 1553, 1567, 1571, 1579, 1583, 1597, 1607, 1613, 1621, 1657, 1669, 1697, 1699, 1709, 1721, 1723, 1733, 1747, 1753, 1759, 1777, 1783, 1787, 1789, 1801, 1823, 1831, 1847, 1861, 1871, 1873, 1877, 1889, 1901, 1907, 1933, 1949, 1973, 1979, 1987, 1993, 1997, 2003, 2011, 2017, 2027, 2029, 2039, 2063, 2081, 2083, 2089, 2099, 2111, 2113, 2129, 2131, 2143, 2153, 2161, 2179, 2213, 2221, 2237, 2239, 2243, 2269, 2281, 2287, 2293, 2297, 2309, 2333, 2339, 2341, 2347, 2351, 2357, 2371, 2377, 2389, 2393, 2417, 2423, 2441, 2447, 2459, 2467, 2473, 2477, 2521, 2531, 2539, 2549, 2579, 2591, 2593, 2609, 2617, 2621, 2647, 2663, 2671, 2677, 2683, 2687, 2689, 2699, 2707, 2711, 2713, 2719, 2729, 2731, 2741, 2749, 2753, 2767, 2777, 2791, 2797, 2801, 2803, 2819, 2833, 2837, 2843, 2851, 2857, 2861, 2879, 2887, 2897, 2903, 2917, 2927, 2939, 2953, 2957, 2963, 2969, 2971, 2999, 3001, 3019, 3023, 3041, 3049, 3061, 3083, 3089, 3109, 3119, 3121, 3137, 3163, 3167, 3187, 3209, 3217, 3221, 3229, 3253, 3257, 3259, 3271, 3299, 3301, 3307, 3313, 3319, 3323, 3329, 3331, 3343, 3347, 3361, 3371, 3391, 3433, 3449, 3457, 3461, 3463, 3467, 3511, 3517, 3527, 3529, 3533, 3539, 3541, 3547, 3557, 3559, 3571, 3581, 3583, 3613, 3631, 3637, 3643, 3659, 3671, 3677, 3691, 3697, 3701, 3709, 3719, 3727, 3733, 3739, 3761, 3769, 3779, 3793, 3803, 3821, 3823, 3833, 3847, 3851, 3853, 3863, 3877, 3881, 3889, 3907, 3917, 3919, 3923, 3929, 3931, 3943, 3967, 3989, 4001, 4003, 4007, 4013, 4019, 4027, 4049, 4051, 4057, 4073, 4079, 4091, 4093, 4099, 4111, 4127, 4129, 4133, 4139, 4153, 4157, 4159, 4177, 4201, 4211, 4219, 4229, 4231, 4241, 4243, 4253, 4261, 4271, 4273, 4283, 4289, 4297, 4327, 4337, 4339, 4349, 4373, 4397, 4409, 4423, 4447, 4451, 4457, 4463, 4481, 4483, 4507, 4513, 4517, 4519, 4523, 4549, 4561, 4567, 4583, 4603, 4621, 4637, 4639, 4643, 4649, 4651, 4657, 4673, 4679, 4691, 4703, 4721, 4723, 4729, 4733, 4751, 4783, 4787, 4789, 4793, 4799, 4801, 4813, 4817, 4861, 4877, 4889, 4903, 4909, 4919, 4931, 4933, 4943, 4951, 4957, 4967, 4969, 4973, 4987, 4993, 4999, 5003, 5009, 5011, 5021, 5023, 5039, 5059, 5077, 5081, 5087, 5099, 5101, 5107, 5119, 5147, 5167, 5197, 5209, 5227, 5231, 5233, 5237, 5261, 5273, 5279, 5281, 5297, 5303, 5309, 5323, 5333, 5347, 5351, 5381, 5387, 5393, 5399, 5407, 5413, 5417, 5419, 5431, 5437, 5441, 5449, 5471, 5479, 5483, 5501, 5503, 5519, 5521, 5527, 5531, 5557, 5563, 5569, 5573, 5581, 5591, 5623, 5639, 5641, 5647, 5651, 5659, 5683, 5689, 5693, 5711, 5717, 5737, 5741, 5743, 5749, 5779, 5783, 5791, 5801, 5807, 5813, 5821, 5827, 5839, 5843, 5849, 5851, 5857, 5861, 5867, 5879, 5881, 5897, 5939, 5981, 5987, 6007, 6011, 6029, 6037, 6043, 6047, 6067, 6073, 6079, 6101, 6121, 6133, 6151, 6163, 6197, 6199, 6203, 6211, 6217, 6221, 6229, 6247, 6257, 6263, 6269, 6277, 6287, 6299, 6301, 6311, 6317, 6323, 6329, 6337, 6343, 6353, 6359, 6361, 6367, 6373, 6379, 6389, 6421, 6427, 6449, 6451, 6473, 6481, 6491, 6521, 6529, 6547, 6553, 6563, 6569, 6577, 6581, 6599, 6607, 6619, 6653, 6661, 6679, 6689, 6691, 6701, 6703, 6719, 6733, 6737, 6761, 6763, 6779, 6781, 6791, 6793, 6803, 6823, 6829, 6833, 6841, 6857, 6863, 6869, 6871, 6883, 6899, 6907, 6911, 6917, 6947, 6949, 6959, 6961, 6967, 6971, 6977, 6983, 6991, 6997, 7001, 7013, 7019, 7027, 7039, 7057, 7069, 7079, 7103, 7109, 7121, 7127, 7129, 7151, 7159, 7187, 7207, 7211, 7213, 7219, 7229, 7237, 7243, 7247, 7253, 7283, 7297, 7307, 7309, 7321, 7331, 7349, 7351, 7369, 7393, 7411, 7417, 7433, 7451, 7457, 7477, 7481, 7487, 7489, 7499, 7507, 7517, 7523, 7537, 7541, 7547, 7549, 7559, 7561, 7573, 7577, 7583, 7589, 7603, 7607, 7621, 7639, 7649, 7669, 7673, 7681, 7687, 7691, 7699, 7703, 7717, 7723, 7727, 7741, 7753, 7757, 7789, 7793, 7817, 7823, 7829, 7841, 7853, 7867, 7873, 7877, 7879, 7901, 7919, 7927, 7937, 7949, 7951, 7963, 8009, 8011, 8017, 8039, 8053, 8069, 8081, 8087, 8089, 8093, 8101, 8111, 8117, 8147, 8161, 8167, 8171, 8179, 8219, 8221, 8233, 8237, 8243, 8263, 8269, 8273, 8287, 8293, 8297, 8317, 8329, 8353, 8363, 8369, 8377, 8387, 8389, 8419, 8423, 8429, 8431, 8443, 8447, 8461, 8467, 8501, 8513, 8521, 8527, 8537, 8539, 8543, 8563, 8573, 8581, 8597, 8599, 8609, 8623, 8627, 8641, 8663, 8669, 8677, 8681, 8689, 8699, 8707, 8713, 8731, 8741, 8747, 8753, 8761, 8779, 8783, 8803, 8807, 8819, 8821, 8837, 8849, 8861, 8863, 8867, 8923, 8929, 8933, 8941, 8951, 8963, 8999, 9001, 9007, 9011, 9013, 9029, 9041, 9043, 9059, 9067, 9091, 9103, 9127, 9133, 9137, 9157, 9161, 9173, 9181, 9187, 9199, 9203, 9209, 9221, 9227, 9241, 9257, 9281, 9283, 9293, 9311, 9319, 9323, 9337, 9341, 9343, 9349, 9377, 9391, 9397, 9403, 9413, 9419, 9421, 9431, 9433, 9437, 9439, 9461, 9467, 9473, 9479, 9491, 9497, 9511, 9521, 9539, 9547, 9551, 9601, 9613, 9619, 9623, 9629, 9631, 9643, 9649, 9661, 9677, 9679, 9689, 9697, 9719, 9721, 9733, 9743, 9749, 9767, 9769, 9781, 9787, 9791, 9803, 9811, 9817, 9829, 9839, 9857, 9859, 9871, 9887, 9901, 9907, 9923, 9929, 9931, 9941, 9949, 9967, 9973, 10007, 10037, 10039, 10061, 10067, 10069, 10079, 10091, 10093, 10099, 10103, 10133, 10139, 10141, 10151, 10159, 10169, 10177, 10181, 10193, 10211, 10243, 10247, 10259, 10271, 10273, 10289, 10301, 10303, 10313, 10321, 10331, 10337, 10343, 10357, 10369, 10391, 10399, 10427, 10429, 10433, 10453, 10457, 10459, 10463, 10477, 10487, 10499, 10513, 10529, 10531, 10559, 10589, 10597, 10607, 10613, 10627, 10631, 10639, 10651, 10657, 10663, 10667, 10687, 10691, 10709, 10711, 10723, 10729, 10733, 10753, 10771, 10789, 10799, 10831, 10837, 10847, 10853, 10861, 10867, 10883, 10889, 10891, 10903, 10909, 10937, 10939, 10949, 10957, 10973, 10987, 10993, 11003, 11027, 11047, 11057, 11059, 11069, 11071, 11087, 11093, 11113, 11117, 11119, 11131, 11149, 11161, 11171, 11173, 11177, 11213, 11243, 11251, 11257, 11261, 11273, 11279, 11287, 11299, 11311, 11317, 11329, 11351, 11353, 11369, 11383, 11393, 11411, 11423, 11437, 11443, 11447, 11467, 11471, 11483, 11489, 11491, 11497, 11503, 11519, 11527, 11549, 11551, 11579, 11587, 11593, 11597, 11617, 11621, 11677, 11681, 11689, 11699, 11717, 11719, 11731, 11743, 11777, 11779, 11783, 11801, 11807, 11813, 11821, 11827, 11831, 11833, 11839, 11863, 11867, 11887, 11897, 11903, 11909, 11927, 11933, 11939, 11941, 11953, 11959, 11969, 11971, 11981, 11987, 12007, 12011, 12037, 12041, 12071, 12073, 12097, 12101, 12107, 12109, 12113, 12119, 12143, 12149, 12157, 12161, 12197, 12203, 12211, 12227, 12239, 12251, 12253, 12263, 12269, 12277, 12281, 12289, 12301, 12323, 12329, 12343, 12347, 12373, 12379, 12391, 12401, 12409, 12413, 12421, 12433, 12437, 12451, 12457, 12473, 12479, 12491, 12497, 12503, 12511, 12517, 12527, 12541, 12547, 12569, 12577, 12583, 12589, 12601, 12611, 12613, 12619, 12637, 12641, 12647, 12653, 12659, 12689, 12697, 12703, 12713, 12721, 12739, 12743, 12757, 12781, 12791, 12799, 12809, 12821, 12823, 12829, 12841, 12853, 12889, 12899, 12907, 12911, 12917, 12923, 12941, 12953, 12959, 12967, 12973, 12979, 12983, 13001, 13003, 13007, 13033, 13037, 13043, 13049, 13063, 13093, 13099, 13103, 13109, 13121, 13147, 13151, 13159, 13163, 13171, 13177, 13183, 13217, 13219, 13229, 13241, 13267, 13291, 13297, 13309, 13313, 13327, 13331, 13337, 13339, 13367, 13381, 13397, 13399, 13411, 13417, 13421, 13441, 13457, 13463, 13469, 13477, 13487, 13499, 13513, 13523, 13537, 13553, 13567, 13577, 13591, 13597, 13613, 13619, 13627, 13633, 13649, 13669, 13679, 13681, 13687, 13691, 13693, 13697, 13709, 13711, 13721, 13723, 13751, 13763, 13789, 13799, 13807, 13829, 13831, 13859, 13873, 13877, 13879, 13883, 13903, 13913, 13921, 13931, 13933, 13963, 13967, 13997, 13999, 14009, 14029, 14033, 14051, 14071, 14081, 14083, 14087, 14107, 14143, 14153, 14159, 14173, 14197, 14207, 14221, 14249, 14251, 14281, 14293, 14303, 14321, 14323, 14327, 14347, 14369, 14387, 14389, 14401, 14407, 14411, 14419, 14423, 14431, 14437, 14447, 14461, 14489, 14503, 14519, 14533, 14537, 14543, 14549, 14551, 14561, 14563, 14591, 14593, 14627, 14629, 14633, 14653, 14657, 14669, 14683, 14699, 14713, 14723, 14731, 14737, 14741, 14747, 14759, 14767, 14771, 14779, 14783, 14797, 14813, 14821, 14827, 14831, 14843, 14851, 14867, 14869, 14879, 14887, 14891, 14897, 14923, 14939, 14951, 14957, 14969, 14983, 15013, 15017, 15031, 15053, 15061, 15077, 15091, 15101, 15107, 15121, 15137, 15139, 15149, 15161, 15173, 15187, 15193, 15199, 15217, 15227, 15233, 15241, 15259, 15263, 15269, 15277, 15287, 15289, 15299, 15307, 15313, 15319, 15331, 15359, 15361, 15373, 15377, 15391, 15401, 15413, 15427, 15439, 15443, 15451, 15461, 15467, 15473, 15493, 15497, 15511, 15527, 15541, 15551, 15559, 15569, 15581, 15583, 15601, 15607, 15629, 15641, 15643, 15647, 15667, 15671, 15679, 15727, 15731, 15733, 15737, 15739, 15749, 15761, 15767, 15773, 15787, 15791, 15797, 15803, 15809, 15817, 15859, 15877, 15881, 15887, 15889, 15901, 15913, 15919, 15923, 15937, 15959, 15971, 15991, 16001, 16007, 16033, 16057, 16061, 16063, 16067, 16069, 16073, 16087, 16091, 16097, 16103, 16111, 16127, 16141, 16187, 16189, 16193, 16217, 16231, 16249, 16253, 16267, 16273, 16319, 16333, 16339, 16349, 16361, 16363, 16369, 16381, 16411, 16417, 16421, 16427, 16433, 16447, 16451, 16453, 16477, 16481, 16487, 16493, 16519, 16529, 16547, 16553, 16561, 16567, 16573, 16607, 16619, 16631, 16633, 16649, 16657, 16661, 16673, 16691, 16693, 16699, 16703, 16729, 16741, 16747, 16759, 16763, 16787, 16811, 16823, 16829, 16831, 16843, 16871, 16883, 16889, 16901, 16903, 16921, 16931, 16937, 16943, 16963, 16979, 16981, 16987, 16993, 17011, 17021, 17027, 17029, 17041, 17047, 17077, 17093, 17099, 17107, 17117, 17123, 17137, 17159, 17167, 17183, 17189, 17191, 17203, 17207, 17209, 17239, 17257, 17291, 17293, 17299, 17317, 17321, 17327, 17333, 17341, 17351, 17359, 17377, 17389, 17393, 17401, 17417, 17419, 17431, 17443, 17449, 17467, 17471, 17477, 17483, 17489, 17491, 17497, 17509, 17519, 17539, 17551, 17569, 17573, 17579, 17581, 17597, 17599, 17609, 17623, 17627, 17657, 17659, 17669, 17681, 17683, 17707, 17713, 17729, 17737, 17747, 17749, 17761, 17783, 17789, 17791, 17807, 17827, 17837, 17839, 17851, 17863, 17881, 17891, 17903, 17909, 17911, 17921, 17929, 17939, 17957, 17959, 17971, 17977, 17981, 17987, 17989, 18013, 18041, 18047, 18049, 18059, 18077, 18097, 18121, 18127, 18131, 18133, 18143, 18149, 18169, 18181, 18191, 18199, 18211, 18217, 18223, 18229, 18233, 18251, 18253, 18269, 18287, 18289, 18301, 18307, 18311, 18313, 18329, 18341, 18353, 18371, 18379, 18397, 18413, 18427, 18433, 18439, 18443, 18451, 18457, 18461, 18481, 18493, 18503, 18517, 18521, 18523, 18539, 18553, 18583, 18587, 18593, 18617, 18637, 18661, 18671, 18679, 18691, 18701, 18713, 18719, 18731, 18749, 18757, 18787, 18797, 18839, 18869, 18899, 18911, 18913, 18917, 18919, 18947, 18959, 18973, 18979, 19001, 19009, 19031, 19037, 19051, 19069, 19073, 19079, 19081, 19087, 19121, 19139, 19141, 19157, 19163, 19181, 19183, 19207, 19211, 19213, 19219, 19231, 19237, 19249, 19267, 19273, 19301, 19309, 19333, 19373, 19379, 19381, 19391, 19403, 19417, 19421, 19423, 19427, 19429, 19433, 19447, 19457, 19463, 19471, 19477, 19483, 19489, 19501, 19507, 19531, 19541, 19543, 19553, 19559, 19571, 19577, 19583, 19597, 19603, 19661, 19687, 19697, 19709, 19739, 19751, 19753, 19759, 19777, 19793, 19801, 19813, 19819, 19841, 19843, 19853, 19861, 19867, 19889, 19891, 19913, 19919, 19927, 19937, 19949, 19961, 19963, 19973, 19991, 19993, 19997, 20011, 20021, 20023, 20029, 20047, 20051, 20063, 20071, 20089, 20101, 20107, 20113, 20117, 20123, 20129, 20143, 20147, 20161, 20173, 20183, 20201, 20219, 20231, 20233, 20249, 20261, 20269, 20287, 20297, 20323, 20327, 20333, 20341, 20347, 20357, 20389, 20393, 20399, 20407, 20411, 20431, 20441, 20443, 20477, 20479, 20483, 20507, 20509, 20521, 20533, 20543, 20549, 20551, 20563, 20593, 20599, 20611, 20627, 20639, 20663, 20681, 20693, 20707, 20717, 20719, 20731, 20743, 20747, 20749, 20753, 20759, 20771, 20773, 20789, 20807, 20809, 20849, 20857, 20873, 20879, 20887, 20897, 20899, 20903, 20929, 20939, 20947, 20959, 20963, 20981, 20983, 21001, 21011, 21013, 21017, 21019, 21023, 21031, 21061, 21067, 21089, 21101, 21107, 21121, 21139, 21143, 21157, 21163, 21169, 21179, 21187, 21191, 21193, 21211, 21221, 21269, 21277, 21283, 21317, 21319, 21341, 21347, 21377, 21379, 21383, 21391, 21401, 21407, 21419, 21433, 21467, 21481, 21491, 21493, 21499, 21521, 21523, 21529, 21559, 21563, 21569, 21577, 21587, 21589, 21599, 21601, 21611, 21613, 21617, 21647, 21661, 21673, 21683, 21701, 21727, 21737, 21739, 21751, 21757, 21773, 21787, 21799, 21803, 21817, 21839, 21841, 21851, 21859, 21863, 21871, 21881, 21893, 21911, 21929, 21937, 21943, 21961, 21977, 21991, 21997, 22003, 22013, 22027, 22031, 22037, 22039, 22051, 22063, 22067, 22073, 22079, 22091, 22093, 22109, 22111, 22123, 22129, 22133, 22147, 22157, 22159, 22171, 22229, 22247, 22271, 22273, 22277, 22279, 22291, 22303, 22307, 22343, 22367, 22369, 22381, 22391, 22397, 22409, 22433, 22441, 22447, 22469, 22481, 22483, 22511, 22531, 22541, 22543, 22549, 22567, 22571, 22573, 22613, 22619, 22621, 22637, 22639, 22651, 22669, 22679, 22691, 22697, 22699, 22717, 22721, 22727, 22739, 22741, 22751, 22769, 22777, 22783, 22787, 22807, 22811, 22817, 22853, 22859, 22871, 22877, 22901, 22907, 22921, 22937, 22943, 22961, 22963, 22973, 22993, 23011, 23017, 23021, 23027, 23029, 23039, 23041, 23053, 23057, 23059, 23063, 23081, 23087, 23099, 23117, 23131, 23143, 23159, 23167, 23189, 23197, 23201, 23209, 23227, 23251, 23291, 23293, 23311, 23321, 23327, 23333, 23339, 23369, 23371, 23399, 23431, 23447, 23459, 23473, 23497, 23531, 23537, 23539, 23549, 23557, 23561, 23563, 23567, 23581, 23593, 23603, 23609, 23623, 23627, 23629, 23633, 23663, 23669, 23671, 23677, 23687, 23689, 23719, 23741, 23747, 23753, 23767, 23773, 23789, 23801, 23813, 23819, 23827, 23833, 23857, 23869, 23873, 23879, 23887, 23893, 23899, 23909, 23911, 23917, 23929, 23957, 23971, 23977, 23981, 23993, 24001, 24007, 24019, 24023, 24043, 24049, 24061, 24071, 24077, 24083, 24091, 24097, 24103, 24107, 24109, 24113, 24121, 24133, 24137, 24169, 24179, 24181, 24197, 24203, 24223, 24229, 24239, 24247, 24251, 24281, 24317, 24329, 24359, 24371, 24391, 24407, 24413, 24419, 24421, 24443, 24469, 24473, 24481, 24499, 24509, 24517, 24547, 24551, 24571, 24611, 24631, 24659, 24671, 24677, 24691, 24697, 24749, 24763, 24821, 24841, 24847, 24851, 24859, 24877, 24889, 24907, 24917, 24919, 24923, 24943, 24953, 24967, 24971, 24977, 24979, 24989]

\end{thebibliography}

\end{document}