% !TeX program = pdfLaTeX
\documentclass[12pt]{article}
\usepackage{amsmath}
\usepackage{graphicx,psfrag,epsf}
\usepackage{enumerate}
\usepackage{natbib}
\usepackage{textcomp}
\usepackage[hyphens]{url} % not crucial - just used below for the URL
\usepackage{hyperref}
\providecommand{\tightlist}{%
  \setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}

%\pdfminorversion=4
% NOTE: To produce blinded version, replace "0" with "1" below.
\newcommand{\blind}{0}

% DON'T change margins - should be 1 inch all around.
\addtolength{\oddsidemargin}{-.5in}%
\addtolength{\evensidemargin}{-.5in}%
\addtolength{\textwidth}{1in}%
\addtolength{\textheight}{1.3in}%
\addtolength{\topmargin}{-.8in}%

%% load any required packages here

% Pandoc citation processing

\usepackage{amssymb}
\usepackage{amsmath}

\begin{document}

\def\spacingset#1{\renewcommand{\baselinestretch}%
{#1}\small\normalsize} \spacingset{1}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\if0\blind
{
  \title{\bf Data Augmentation with Polya-Gamma Latent Variables for
Logistic Models}

  \author{
        Shen Qu \thanks{STAT 501 Statistical Literature and Problems;
The Fariborz Maseeh Department of Mathematics + Statistics} \\
    Portland State University\\
     and \\     Supervisor Prof.~Daniel Taylor-Rodriguez \\
    Portland State University\\
      }
  \maketitle
} \fi

\if1\blind
{
  \bigskip
  \bigskip
  \bigskip
  \begin{center}
    {\LARGE\bf Data Augmentation with Polya-Gamma Latent Variables for
Logistic Models}
  \end{center}
  \medskip
} \fi

\bigskip
\begin{abstract}
In Bayesian Statistics, Data Augmentation (DA) methods can generate the
auxiliary or latent variables to solve some questions that cannot sample
directly or the probability density/mass functions doesn't have a closed
form. This paper introduces the basic DA approaches and several
examples. For binary response, both probit models and logitstic model
can use Gibb's samplers to simulate the target distribution through the
single-layer latent variables. For logitstic regression models,
generating the posterior distribution from the Pólya--Gamma family is a
fast, exact simulation method.
\end{abstract}

\noindent%
{\it Keywords:} Data Augmentation, Logistic Models, Polya-Gamma
\vfill

\newpage
\spacingset{1.45} % DON'T change the spacing!

\hypertarget{introduction}{%
\section{Introduction}\label{introduction}}

Data Augmentation (DA) is a technique of newly creating synthetic data
from existing data. In Statistics contest, DA algorithm includes, but
not limited to Markov chain Monte Carlo (MCMC) method. There are plenty
of mathematical and computational techniques such as EM algorithm which
can fertilize the DA field exploration. In Deep Learning field, the DA
techniques add some slightly modified copies of already existing data
for increasing the amount of data \citep{shortenSurveyImageData2019}.
This paper focus on the DA applications of MCMC in Bayesian statistics.
It introduces the basic ideas of MCMC algorithm through several examples
and explain its main properties and conditions, that is constructing a
successes DA Markov chain requires the condition of \emph{Harris
ergodic}, which includes three properties: irreducible, aperiodic, and
Harris recurrent.

These approaches can be applied on the binary response regression
models. As a type of Generalized Linear Models (GLMs), the response is
treated as the results of a Bernoulli process. Using a link function of
probit or logit, one can construct a model with the dependent binary
variable \(\mathbf{Y}\), the vectors of covariates \(\mathbf{X}\) and
parameters \(\boldsymbol{\beta}\).

There are many approaches fitting these models through the Gibb's
samplers. From literature, we find two simple simulation methods with
some single-layer latent variables for the probit and logistic
regression models. In probit version, the Bayesian methods generate the
location-mixture latent variables from two truncated normal
distributions. For logistic models, the latent variables is drawn from
the Polya-Gamma family. Hence, its posterior distribution is a scale
mixture with the linear part.

\hypertarget{literature}{%
\section{Literature}\label{literature}}

A main goal of Bayesian approach is to identify the posterior
distribution. There are three main methods to estimate the target
posterior density of \(\beta\): asymptotic expansions, numerical
integration, and Monte Carlo integration. When the sample size is small,
asymptotic expansions may not work well. When the canonical exponential
families with normal or conjugate priors are concave on the log scale,
the numerical method can converge quickly to estimate the posterior
joint or marginal distribution \citep{ohaganKendallAdvancedTheory2004}.
In many cases, the numerical integration is difficult for the
high-dimension models. \citet{zellnerBayesianAnalysisDichotomous1984}
proposed an importance sampling generalization of Monte Carlo
simulation. They found that Monte Carlo methods are more adequate and
efficient to achieve a good approximation.
\citet{gelfandSamplingBasedApproachesCalculating1990} develop these
methods to Gibbs sampling. When sampling from the conditional
distribution is easier, Gibbs sampler is a highly useful MCMC method to
sample from multivariate distributions. Metropolis-Hastings algorithm
\citep{tierneyMarkovChainsExploring1994} is another important progress
of MCMC. It is the generalized version of Gibb's sampler. A systematic
review can be found in \citet{agrestiBayesianInferenceCategorical2005}
and \citet{agrestiCategoricalDataAnalysis2012}.

For binary response, the generalized linear models use a link function
such as probit or logit to connect the linear part to the response.
\citet{albertBayesianAnalysisBinary1993} provide a computational simple
Bayesian approach for probit regression modeling. They assume the
relationship between the binary data and the latent variables. The
question can be solved through an underlying normal regression model.
The models have a hierarchical prior structure as follow:

\[Z\sim N_p(\boldsymbol{X\beta,I}),\quad \boldsymbol{\beta}\sim N_p(\boldsymbol{A\beta_0,\sigma^2I}), \quad (\beta_0,\sigma^2)\sim \pi(\beta_0,\sigma^2)\]

where \(Z\) is the latent variable, \(\boldsymbol{\beta}\) is the
parameter vector of the linear predictors. \(\boldsymbol{A\beta_0}\) is
the linear subspace, and assuming \(\boldsymbol{\beta_0,\sigma^2}\) are
independent and noninformative priors. This method can also extend to
ordered multinomial responses and mixture models
\citep{hoffFirstCourseBayesian2009, gelmanBayesianDataAnalysis2020}.

The Bayesian logistic models using approximation methods for quite some
time. \citet{piegorschEmpiricalBayesEstimation1996} suggest the
parametric empirical Bayes methods using marginal maximum likelihood
estimate hyperparameters and extend the link function.
\citet{greenlandPuttingBackgroundInformation2001} discuss that the
approximation may be inadequate with sparse data and they suggest to use
conjugate priors to conduct exact analysis. A new step is that
\citet{polsonBayesianInferenceLogistic2013} introduce an new method that
assume the latent variables follow a Polya-Gamma distribution. It is as
simple as \citet{albertBayesianAnalysisBinary1993} 's method, which
don't need multiple layers. This approach assign the linear part at the
scale parameter. \citet{choiPolyaGammaGibbsSampler2013} prove that this
polya-gamma Gibbs sampler for logistic model is uniformly ergodic. Some
further discussions of this DA algorithm include convergence rates of
the DA and the Haar PX-DA algorithm
\citep{choiConvergenceAnalysisGibbs2014}, a sandwich version of DA
\citep{choiComparisonTheoremData2016}, and the analysis of variance
\citep{choiAnalysisPolyaGammaGibbs2017}.

\hypertarget{appoaches}{%
\section{Appoaches}\label{appoaches}}

\subsection{Basic ideas}
\label{sec:basic}

\begin{itemize}
\tightlist
\item
  Expectation
\end{itemize}

Assume a random variable \(g(X)\) and pdf
\(f_{X}(x):\mathbb{R}^{p} \to [0,\infty)\). Denote interested
\(E[g(x)]\) is the expected value or mean of \(g(X)\)
\citep[\emph{Def2.2.1}]{casellaStatisticalInference2002}.

\[E_{f_{X}}[g(x)]= \int_{\mathbb{R}^{p}} g(x) f_{X}(x) dx\]

\begin{itemize}
\tightlist
\item
  Joint, conditional, and marginal density
\end{itemize}

A probability function with a bivariate random vector \(f(x,z)\) is
called joint pdf.

A marginal probability density function of X is the integral of joint
pdf \(f(x,z)\) on X: \(f_{X}(x)=\int_{\mathbb{R}^p} f(x,z)dz\).

The relationship among joint pdf, marginal pdf, and conditional pdf of
\(X\) given \(Z\) is: \(f(x,z)=f(x|z)f(z)\)

\begin{description}
\item[\textbf{Example 1}]
Given \(f_{X}(x)=3x^{2}I_{(0,1)}(x)\), then
\(E[X]=\int_{0}^{1} xf_{X}(x)dx=\left.\frac34x^4\right|_0^1=0.75\).
\end{description}

Suppose the joint pdf \(f(x,z)=3xI_{(0<z<x<1)}\), then we can get all
the rest of density functions:

\[\begin{aligned}
f_Z(z)&=\int_{z}^{1} f(x,z)dx =\frac32(1-z^2)\\
f_{Z|X}(z|x)&=\frac{f(x,z)}{f_Z(z)}=\frac1{x}I(0<z<x)\\
f_{X|Z}(x|z)&=\frac{f(x,z)}{f_X(x)}=\frac{2x}{1-z^2}I(z<x<1)\\
\end{aligned}\]

\begin{description}
\item[\textbf{Example 2}:]
Given a joint pdf
\(f(x,z)=(\sqrt{2}\pi)^{-1}\exp\left[-(x^2-\sqrt{2}xz+z^2)\right]\),
which is a bivariate normal density with means \(\mu_X=\mu_Z=0\),
variances \(\sigma_X=\sigma_Z=1\), and correlation \(\rho=2^{-1/2}\), we
can get \(f_{X}(x)\) by integral of \(f(x,z)\).
\end{description}

\[\begin{aligned}
f_{X}(x)&=\int_{-\infty}^\infty f_{X,Z}(x,z)\,\mathrm dz \\
&= \frac{\exp(-\frac12x^2)}{\sqrt{2\pi}}\int_{-\infty}^\infty \frac{1}{\sqrt{2\pi\frac12}}\exp\left[-\frac12(z-\frac{x}{\sqrt{2}})^2\right]\,\mathrm dz \\
&= \frac{1}{\sqrt{2\pi}}\exp(-\frac12x^2)
\end{aligned}\]

Another way is, suppose \(f(z|x)\sim N(\frac{x}{\sqrt{2}}\frac12)\), we
also can get
\(f_{X}(x)=\frac{f(x,z)}{f(z|x)}=\frac{1}{\sqrt{2\pi}}\exp{(-\frac12x^2)}\)

Then, we can utilize these properties to construct MCMC algorithm to
solve the more complex questions.

\begin{itemize}
\tightlist
\item
  Monte Carlo approximation
\end{itemize}

In some case, \emph{constructing estimates of features of probability
distributions are impossible or very difficult to compute analytically}

\emph{if we have data from an unknown probability distribution belonging
to a model we can estimate that distribution in some way.} \emph{if we
have a feature (parameter) of the true distribution we want to estimate,
use Monte Carlo to estimate the feature by random quantities generated
using the estimated distribution. }
\citep[ch.~10.1]{doksumMathematicalStatisticsBasic2015}

Regardless of the distribution, if we draw i.i.d.
\(g(X_{1}),g(X_{2}),\ldots,g(X_{m})\) from \(f_{X}(x)\), then, for large
\(m\), the empirical distribution of is arbitrarily close to the
distribution under\(f_{X}(x)\). That is
\(\frac{1}{m}\sum_{i=1}^{m} g(X_{i})\overset{a.s}{\to}E_{f_{X}}[g(x)]\).

\begin{itemize}
\tightlist
\item
  Markov Chain
\end{itemize}

When it is impossible to simulate from \(f_{X}(x)\), such as
\(f_{X}(x)\) doesn't have closed form or \(f(x,z)\) is hard to integral,
we can construct a Markov chain. Suppose
\(X = \{X_{i}\}^{\infty}_{i=0}\), with state space \(\mathbf{X}\). If
the current state of the chain is \(X_{i} = x\), then the density of the
next state, \(X_{n+1}\), is \(k(\cdot|x)\). \textbf{???}

\begin{itemize}
\tightlist
\item
  Markov Chain Monte Carlo (MCMC)
\end{itemize}

Markov Chain Monte Carlo is a homogeneous positive recurrent Markov
chain
\citep{tannerCalculationPosteriorDistributions1987, swendsenNonuniversalCriticalDynamics1987a}.
\(K(x,x')\) is called the transition kernel which has an unique
stationary density \(k(x'|x)\). The conditional density \(k(x'|x)\) is
also called the Markov transition density (Mtd).

Our goal is to find \(f_{X}(x)\). The conditions are that a joint pdf
\(f:\mathbb{R}^p\times\mathbb{R}^q \to [0,\infty)\) exists, which
includes a auxiliary real variable \(Z\). If the invariant x-marginal
pdf \(f_{X}(x)=\int_{\mathbb{R}^q} f(x,z)dz\) holds. And the associated
two conditional pdfs \(f_{X|Z}(x'|z)\) and \(f_{Z|X}(z|x)\) are
applicable for simulation.

We can shows that \(k(x'|x)\) is a pdf of a random variable \(X'\)

\[\begin{aligned}
\int_{X} k(x'|x)d x' &=\int_{X}\left[\int_{Z}f_{X|Z}(x'|z)f_{Z|X}(z|x) dz\right]dx'\\
&=\int_{Z}f_{Z|X}(z|x) \left[\int_X f_{X|Z}(x'|z)dx'\right]dz\\
&=\int_{Z}f_{Z|X}(z|x)dz=1
\end{aligned}\]

Even without the \(k(x'|x)\) in closed form, we still can simulate the
Markov chain \(X\) as follows.

\begin{quote}
Given an initial value \(x_0\)
\end{quote}

\begin{quote}
\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  Draw \(z^{(1)} \sim f_{Z |X}(\cdot|x^{(0)})\).
\end{enumerate}
\end{quote}

\begin{quote}
\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\setcounter{enumi}{1}
\tightlist
\item
  Draw \(x^{(1)} \sim f_{X|Z} (\cdot|z^{(1)})\).
\end{enumerate}
\end{quote}

\begin{quote}
Repeat the two steps \ldots{}
\end{quote}

\begin{quote}
\begin{enumerate}
\def\labelenumi{\roman{enumi}.}
\tightlist
\item
  Draw \(z^{(i)} \sim f_{Z |X}(\cdot|x^{(i-1)})\).
\end{enumerate}
\end{quote}

\begin{quote}
i+1. Draw \(x^{(i)} \sim f_{X|Z} (\cdot|z^{(i)})\).
\end{quote}

\begin{quote}
At the end, we can calculate
\(E_{f_{X}}[g(x)]\approx\frac{1}{m}\sum_{i=1}^m g(X_i)\)
\end{quote}

\begin{description}
\item[\textbf{Example 3}]
The Student's \(t_4\) with four degree of freedom.
\end{description}

Assume we don't know the target
\(f_{X}(x)=\frac38(1+\frac{x^2}4)^{-\frac52}\). We can take
\(f(x,z)=\frac{4}{\sqrt{2\pi}}z^{\frac32}\exp\{-z(\frac{x^2}{2}+2)\} I_{(0,\infty)}(z)\),
which satisfied \(f_{X}(x)=\int_{\mathbb{R}^p} f(x,z)dz\). Then, it is
easy to draw from two conditional pdf of
\((Z|X=x) \sim Gamma(\frac52,\frac{x^2}2+2)\) and
\((X|Z=z) \sim N(0,z^{-1})\).

\begin{itemize}
\tightlist
\item
  Simple slice sample
\end{itemize}

The \textbf{simple slice sampler} is a DA technique through a
\emph{stepping-out and shrinkage} procedures
\citep{nealSliceSampling2003}. If we can factorize \(f_{X}(x)\) to
\(q(x)l(x)\), supposing an auxiliary variable \(Z\) satisfies
\((Z|X=x)\sim Uniform(0,l(x))\). The joint distribution over \(x\) and
\(z\) define a region \(\{(x,z):0\le z\le l(x)\}\) under the curve of
\(l(x)\). A random state of \(x^{(0)}\) can give a vertical ``slice''
for all possible \(z\) bounded from zero to \(l(x^{(0)})\). Then an
uniformly selected \(z^{(1)}\) can give a horizontal ``slice'' for all
\(x\) inside the curve. That slice is from zero to
\(\int_{z^{(1)}}^{l(x)} f_{X|Z}(x|z)dx\), the union of intervals that
all \(x\) satisfy \(l(x)\ge z^{(1)}\). The next state \(x^{(1)}\) around
\(x^{(0)}\) is accepted only if it doesn't step out the horizontal
``slice''. Each updating will shrink \((x,z)\) until the whole region is
explored.

\begin{description}
\item[Example 1 Cont.]
Rewrite \(f_{X}(x)=3x^{2}I_{(0,1)}(x)=[3x I_{(0,1)}(x)][x]=q(x)l(x)\).
\end{description}

For \(l(x)=x\), the vertical slice is \(\{z|x: 0\le z\le x \}\).
\((Z|X=x) \sim Unif(0,x)\).

The horizontal slice is \(\{x|z: x\ge z\}\). Given
\(f_{X|Z}(x|z)=\frac{2x}{1-z^2}I(z<x<1)\), The union of intervals is
\(\int_z^x f_{X|Z}(x|z)dx=\frac{x^2-z^2}{1-z^2}\). Then
\((X|Z=z)\sim Unif(0,\frac{x^2-z^2}{1-z^2})\)

Suppose \(U \sim Unif(0,1)\), we can get \((Z|X)=xU\) and
\((X|Z)=\sqrt{U(1-z^2)+z^2}\) by transformation.

\begin{quote}
Given an initial value \(x^{(0)}\)
\end{quote}

\begin{quote}
\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\tightlist
\item
  Draw \(u^{(1)} \sim Unif(0,1)\)
\end{enumerate}
\end{quote}

\begin{quote}
\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\setcounter{enumi}{1}
\tightlist
\item
  Calculate \(z^{(1)} =x^{(0)}u^{(1)}\)
\end{enumerate}
\end{quote}

\begin{quote}
\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\setcounter{enumi}{2}
\tightlist
\item
  Calculate \(x^{(1)} =\sqrt{u^{(1)}(1-(z^{(1)})^2)+(z^{(1)})^2}\).
\end{enumerate}
\end{quote}

\begin{quote}
Update \((x^{(i)},z^{(i)})\) by repeating step 1, 2, 3, \ldots{}
\end{quote}

\begin{quote}
Calculate \(E_{f_{X}}[g(x)]\approx\frac{1}{m}\sum_{i=1}^m g(X_i)\)
\end{quote}

\hypertarget{properties}{%
\subsection{Properties}\label{properties}}

\hypertarget{conditions}{%
\subsubsection{Conditions}\label{conditions}}

A well-behaved Markov Chain should be Harris ergodic, which satisfies
three properties: irreducible, aperiodic, and recurrent. A sufficient
condition for Harris ergodicity is

\[\mathcal{K}: k(x'|x)>0\quad\forall x', x\in \mathbf{X}\]

\begin{itemize}
\tightlist
\item
  \textbf{Detailed balance}
\end{itemize}

For all \(x, x'\in \mathbf{X}\), let

\[\delta(x',x)=k(x'|x)f_{X}(x)=\int_{Z} \frac{f(x',z)}{f_Z(z)}\cdot\frac{f(z,x)}{f_X(x)}f_{X}(x) dz\]
\[\delta(x,x')=k(x|x')f_{X}(x')=\int_{Z} \frac{f(x,z)}{f_Z(z)}\cdot\frac{f(z,x')}{f_X(x')}f_{X}(x') dz\]

So \(\delta(x',x)=\delta(x,x')\) is a symmetric function satisfying

\[\delta(x,x')=f_X(x)k(x'|x)=f_X(x')k(x|x'), \forall x, x'\]

which is called the \emph{detailed balance} condition.

A type of MCMC, Metropolis Sampling Algorithm defines a symmetric Markov
kernel \(K_0(\cdot,\cdot)\) in the original Metropolis algorithm and

\[r(x,x')=\min\left(1,\frac{f_X(x')}{f_X(x)}\right)\]

If and only if \(r(x,x')=\frac{\delta(x,x')}{f_X(x)K_0(x,x')}\)

\[K(x,x')=r(x,x')K_0(x,x'),\quad x\neq x'\]

Then \(K\) is a Markov kernel satisfying detailed balance. The Markov
chain is reversible with respect to \(f_X(x)\). When the MCMC always
satisfying detailed balance \(r\equiv 1\), this algorithm is called
Gibbs' Sampler
\citep[ch.~10.4.3]{doksumMathematicalStatisticsBasic2015}.

\begin{itemize}
\tightlist
\item
  \textbf{Invariant (stationarity)}
\end{itemize}

\(f_{X}\) is an invariant density for \(K\) when

\[\int_X k(x'|x)f_{X}(x) dx= f_{X}(x')\]

which means that, if \(X_n\sim f_{X}\), then \(X_{n+1}\sim f_{X}\). The
Markov chain is time homogeneous.

For the case of the multivariate distribution
\(\boldsymbol{\beta}=(\beta_1,...,\beta_p)^T\), the Gibbs sampling
procedure requires the stationary density is an unique distribution
defined by the full conditionals, which mean each single component of
the random vector given the values of the rest.

\hypertarget{convergence}{%
\subsubsection{Convergence}\label{convergence}}

For a reversible Markov chain X satisfies Harris ergodicity, the
\textbf{Strong Law of Large Numbers} holds. The marginal density of
\(X_i\) will converge to the invariant density \(f_{X}\) no matter how
the chain is started. Denote \(P^{n}(x,A)=Pr(X_n\in A|X_0=x)\) where
\(A\) is a measurable set. \(\phi(A)=\int_A f_X(x)dx\). That is

\[\lim_{n\to\infty}||P^{n}(x,\cdot)-\phi(\cdot)||\to 0\]

where
\(||P^{n}(x,\cdot)-\phi(\cdot)||=\underset{A}{\sup}|P^{n}(x,A)-\phi(A)|\)

Let \(L^1(f_X)\) denote the set of functions
\(h: \mathbf{X}\to \mathbb{R}\) \(\int_X |h(x)|f_X(x)dx<\infty\) and
\(E_{f_{X}}h=\int_X h(x)f_X(x)dx\). For \(g\in L^1(f_X)\)

\[\bar g_n := \frac{1}{n}\sum_{i=0}^{n-1} g(X_i)\overset{a.s}{\to}E_{f_{X}}[g(x)]\]

\hypertarget{geometric-ergodicity}{%
\subsubsection{Geometric ergodicity}\label{geometric-ergodicity}}

For all \(x\in \mathbf{X}\) and all n=1,2,.., if exist a function
\(M: \mathbf{X}\to [0,\infty)\) and a constant \(0<\rho<1\)

\[||P^{n}(x,\cdot)-\phi(\cdot)||\le M(x) \rho^n\]

Then the Markov chain \(X\) is geometrically ergodic.

\begin{itemize}
\tightlist
\item
  Drift condition
\end{itemize}

Define \(V: \mathbf{X}\to[0,\infty)\) is unbounded off compact sets,
constants \(\lambda\in[0,1]\), and \(L\in\mathbb{R}\). If the
\emph{drift function} \(V\) exist and

\[E[V(X_{n+1}|X_n=x)]\le\lambda V(x)+L\]

Then the Markov chain \(X\) is geometrically ergodic.

\hypertarget{central-limit-theorems}{%
\subsubsection{Central Limit Theorems}\label{central-limit-theorems}}

Assume that \(X\) is geometrically ergodic and \(g\in L^2(f_X)\). Define
\(\sigma^2=E_{f_{X}}g^2-(E_{f_{X}}g)^2\) and
\(\kappa^2=\sigma^2+2\sum_{k=1}^\infty c_k< \infty\). As \(n\to\infty\)

\[\sqrt{n}(\bar g_n-E_{f_{X}}g)\overset{d}{\to}N(0,\kappa^2)\]

\hypertarget{minorization-regenration-and-alteernative-clt}{%
\subsubsection{Minorization regenration and alteernative
CLT}\label{minorization-regenration-and-alteernative-clt}}

\subsection{MCMC and EM algorithm}
\label{sec:latent}

We can extend data augmentation methods by some techniques both from
MCMC and EM (Expectation and Maximization) algorithm
\citep{mclachlanEMAlgorithmExtensions2008}. Define \(Y\) as the observed
data, which are sampled from a family of pdfs
\(\{p(y|\theta):\theta\in \Theta\}\). To find the feature (parameter) of
the true distribution, the target is the \(\theta\)-marginal posterior
density \(\pi(\theta|y)\) and corresponding expectations
\(E[\pi(\theta|y)]\). Suppose the \textbf{missing data},
\(z\in Z\subset \mathbb{R}^q\) can be identified. The \textbf{complete
data} posterior density as

\[\pi(\theta,z|y)=\frac{p(y,z|\theta)\pi(\theta)}{\int_\Theta\int_{Z}p(y,z|\theta)\pi(\theta)dzd\theta}=\frac{p(y,z|\theta)\pi(\theta)}{\int_{\Theta}p(y|\theta)\pi(\theta)d\theta}=\frac{p(y,z|\theta)\pi(\theta)}{c(y)}.\]

where \(p(y,z|\theta)\) represents the joint density, \(\pi(\theta)\)
denotes the prior density, \(c(y)\) is the marginal density of the data.
we can check that the \(\theta\)-marginal pdf is invariant:

\[\int_{Z}\pi(\theta,z|y)dz=\frac{\pi(\theta)}{c(y)}\int_{Z}p(y,z|\theta)dz=\frac{p(y|\theta)\pi(\theta)}{c(y)}=\pi(\theta|y).\]

This question can be solved by deterministic EM algorithm. Let
\(\theta^{i}\) denote the current guess to the mode of the observed
posterior \(p(\theta|y)\). \(p(\theta|z,y)\) is the augmented posterior.
\(p(z|\theta^{i},y)\) represents the conditional predictive distribution
of the latent data \(Z\), conditional on the current guess to the
posterior mode. In the E-step, we can find the \(Q\)-function, the
conditional expectation of the complete-data log likelihood function as

\[Q(\theta,\theta^{i})=\int_Z\log(p(\theta|z,y))p(z|\theta^{i},y)dz\]

In the M-step, \(Q\)-function is maximized with respect to \(\theta\) to
obtain \(\theta^{i+1}\) by Coordinate-Ascent Algorithm. Continue the
E-step and M-step until convergent. However, for the high-dimension
\(\theta\) the \(Q\)-function is hard to differentiate.

Once we can straightforwardly sample from two full posterior densities
\(\pi(z|\theta,y)\) and \(\pi(\theta|z,y)\), then the target density
\(\pi(\theta|y)\) can be simulated by MCMC
\citep{mengSeekingEfficientData1999}. EM algorithm works well with
exponential families when the log-concave property holds. But if
\(p(z|\theta,y)\) is not ``sufficiently smooth'', the iterates
\(\theta^{i}\) may converge to some local maxima, minima, or saddle
stationary point of \(p(\theta|y)\). In small samples, the posterior or
likelihood may not consistent with the normal approximation. Unlike the
EM algorithm, the DA methods base on the entire likelihood function or
posterior distribution. It can converge to the global maximizer and can
also improves the inference in both small and large sample situation
\citep{tannerToolsStatisticalInference1993}.

The EM algorithm converges at a linear
rate\citep{dempsterMaximumLikelihoodIncomplete1977}, with the rate
depending on the proportion of information about \(\theta\) in
\((\theta|Y)\) is observed. In contrast, MCMC can converge at a
geometric speed, More discussions of EM and DA can be found in
\citet{dykCrossFertilizingStrategiesBetter2010}.

\textbf{Example 4} The multivariate location-scale Student's \(t\)
density with known degree of freedom \(\nu>0\).

Assume the true distribution of response
\(Y_i\sim t_{\nu,\mu,\sigma^2}\),i=1,..,m.

\[p(y|\mu,\sigma^2)=\frac{\Gamma(\frac{\nu+1}2)}{\sqrt{\nu\sigma^2}\Gamma(\frac{\nu}2)}(1+\frac{(y-\mu)^2}{\nu\sigma^2})^{-\frac{\nu+1}2}\]

Let the prior density \(\pi(\mu,\sigma^2)\propto 1/\sigma^2\). The
target posterior density is

\[\begin{aligned}
\pi(\mu,\sigma^2|y)&\propto\pi(\mu,\sigma^2)\prod_{i=1}^m{p(y_i|\mu,\sigma^2)} \\
                   &\propto(\sigma^2)^{-\frac{m+2}2}\prod_{i=1}^m(1+\frac{(y_i-\mu)^2}{\nu\sigma^2})^{-\frac{\nu+1}2}
\end{aligned}\]

\citet{mengSeekingEfficientData1999} propose a solution by DA algorithm.
Let \(Z=\mathbb{R}^p_{+}\) and make a i.i.d. paired vectors
\((Y_{1:m},Z_{1:m})\), where
\((Y_{i}|Z_{i},\mu,\sigma^2)\sim N(\mu,\sigma^2/z_i)\),
\((Z_{i}|\mu,\sigma^2) \sim Gamma(\nu/2,\nu/2)\). Then the joint density
is
\(p(y,z|\mu,\sigma^2)=\prod_{i=1}^mp(y_i|z_i,\mu,\sigma^2)p(z_i|\mu,\sigma^2)\)

Checking the \((\mu,\sigma^2)\)-marginal density confirms that the
condition holds.

\[\begin{aligned}\int_Y p(z,y|\mu,\sigma^2)dz=&\prod_{i=1}^m\int_{\mathbb{R}_{+}}\frac{\sqrt{z_i}}{\sqrt{2\pi\sigma^2}}\exp\left[-\frac{z_i}{2\sigma^2}(y_i-\mu)^2\right]\frac{(\frac{\nu}2)^{(\frac{\nu}2)}}{\Gamma(\frac{\nu}2)}z_i^{\frac{\nu}2-1}\exp\left[-\frac{\nu}{2}z_i\right]dz_i\\
=&\prod_{i=1}^m\frac{\Gamma(\frac{\nu+1}2)}{\sqrt{\nu\sigma^2}\Gamma(\frac{\nu}2)}(1+\frac{(y_i-\mu)^2}{\nu\sigma^2})^{-\frac{\nu+1}2}\end{aligned}\]

Then we can rearrange the joint pdf.

\[\begin{aligned}
p(\mu,\sigma^2,z|y)&\propto\pi(\mu,\sigma^2)p(z,y|\mu,\sigma^2)\\
                     &\propto\frac{1}{\sigma^2}\prod_{i=1}^m\frac{\sqrt{z_i}}{\sqrt{2\pi\sigma^2}}\exp\left[-\frac{z_i}{2\sigma^2}(y_i-\mu)^2\right]\frac{(\frac{\nu}2)^{(\frac{\nu}2)}}{\Gamma(\frac{\nu}2)}z_i^{\frac{\nu}2-1}\exp\left[-\frac{\nu}{2}z_i\right]\\
\end{aligned}\]

and get the first full conditional pdf

\[ \pi(z|\mu,\sigma^2,y)\propto \frac{(\frac{(y-\mu)^2}{2\sigma^2}+\frac{\nu}{2})^{(\frac{\nu+1}2)}}{\Gamma(\frac{\nu+1}2)}z^{\frac{\nu+1}2-1}\exp\left[-\left(\frac{(y-\mu)^2}{2\sigma^2}+\frac{\nu}{2}\right)z\right]\]

Define \(\hat\mu=\frac{1}{z_{.}}\sum_{j=1}^my_jz_j\) The second full
conditional pdf is

\[\begin{aligned}
\pi(\mu|\sigma^2,y,z)&\propto\exp\left[-\frac{\sum z_j}{2\sigma^2}\mu^2+\frac{\sum z_jy_j}{\sigma^2}\mu\right]\\
&\propto\exp\left[-\frac{z_.}{2\sigma^2}(\mu-\frac{\sum y_jz_j}{z_{.}})^2\right]\\
&\propto\frac{\sqrt{z_.}}{\sqrt{2\pi\sigma^2}}\exp\left[-\frac{z_.}{2\sigma^2}(\mu-\hat\mu)^2\right]
\end{aligned}\]

Define \(\hat\sigma^2=\frac{1}{z_{.}}\sum_{j=1}^mz_j(y_j-\hat\mu)^2\).
The third full conditional pdf can get by

\[\begin{aligned}
\pi(\sigma^2|y,z)&= \int_{\mathbb{R}}p(\mu,\sigma^2,z|y)d\mu\\
               &\propto \frac{(\frac{z_{.}\hat\sigma^2}2)^{\frac{m+1}2}}{\Gamma(\frac{m+1}2)} (\frac1{\sigma^2})^{\frac{m+1}{2}+1}\exp\left[-\frac{z_{.}\hat\sigma^2}{2\sigma^2}\right]
\end{aligned}\]

Thus, we can simulate \((\mu,\sigma^2)\) by three steps:

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\item
  Draw \((Z_i|\mu,\sigma^2,y)\) from
  \(Gamma(\frac{\nu+1}2,\frac{1}2(\frac{(y_i-\mu)^2}{\sigma^2}+\nu))\).
\item
  Draw \((\sigma^2|y,z)\) from
  \(IG(\frac{m+1}2,\frac{z_{.}\hat\sigma^2}2)\).
\item
  Draw \((\mu|\sigma^2,y,z)\) from
  \(N(\hat\mu,\frac{\sigma^2}{z_{.}})\).
\end{enumerate}

\hypertarget{probit-models}{%
\section{Probit Models}\label{probit-models}}

Probit models is one type of Generalized Linear Models. Suppose the
binary response \(Y_1,..,Y_n\sim Bern(p_i)\) is the observed data with
sample size \(n\). \(X_i^T=(X_{i1},..,X_{ip})\) known \(p\) covariates.
\(\beta_{k\times1}\) is the unknown vector of parameters we want to
estimate.

To construct a regression model, suppose a known CDF \(H(\cdot)\) as the
link function with the linear part \(\mathbf{x}_i^T\boldsymbol{\beta}\)
Then, let \(p_i=H(\mathbf{x}_i^T\boldsymbol{\beta})\).

Denote \(\pi(\boldsymbol{\beta})\) as the prior density. The target
conditional density is \[
\pi(\boldsymbol{\beta}|y) = \frac{\pi(\boldsymbol{\beta})}{C(y)}\prod_{i=1}^{k}H (\mathbf{x}_i^T\boldsymbol{\beta})^{y_i}(1-H(\mathbf{x}_i^T\boldsymbol{\beta}))^{1-y_i}
\]

where \(c(y)\) is free from \(\beta\)
\[C(y)=\int\pi(\boldsymbol{\beta})\prod_{i=1}^{k}H (\mathbf{x}_i^T\boldsymbol{\beta})^{y_i}(1-H(\mathbf{x}_i^T\boldsymbol{\beta}))^{1-y_i}d\boldsymbol{\beta}\]

When \(H(\cdot)\) represents the standard normal CDF, it obtains the
probit model. When \(H(\cdot)\) is logistic CDF, it is the logistic
model.

While the logistic models have a close form, probit models cannot
compute in closed form and are hard to integral.

Sampling from the posterior density \(\pi(\boldsymbol{\beta}|y)\) with
high dimension is also problematic. (why)

\citet{albertBayesianAnalysisBinary1993} propose a DA algorithm to
estimate the exact posterior distribution of \(\beta\). This
simulation-based approach introduces latent variables
\(Z_1,..Z_N\overset{iid}{\sim} N(\mathbf{x}_i^T\boldsymbol{\beta},1)\).
The relationship between observed data and missing data is
\(y_i=\begin{cases}1&\text{if } z_i>0\\0&\text{if }z_i\le0\end{cases}\).
The joint mass function is

\[\begin{aligned}
\pi(\boldsymbol{\beta,z|y}) &= \frac{\pi(\boldsymbol{\beta})}{C(y)}\prod_{i=1}^{N}[p(y_i|\beta)]f(z_i|\beta)\\
&\propto\pi(\boldsymbol{\beta})\prod_{i=1}^{N}\left[\Phi (\mathbf{x}_i^T\boldsymbol{\beta})^{y_i}(1-\Phi(\mathbf{x}_i^T\boldsymbol{\beta}))^{1-y_i}\right]\phi (z_{i}|\beta)\\
&=\pi(\boldsymbol{\beta})\prod_{i=1}^{N}\left[1_{z_i>0}1_{y_i=1}+1_{z_i\le0}1_{y_i=0}\right]\phi (z_{i}|\beta)\\
\end{aligned}\]

where \(\Phi\) is a standard normal CDF, \(\phi\) is a normal pdf with
means equal \(\mathbf{x}_i^T\boldsymbol{\beta}\) and variances equal
one.

Assign a flat prior to \(\beta\), we can get the first conditional
posterior density:

\[\pi(\boldsymbol{\beta|y,Z})\propto \prod_{i=1}^{N}\phi (Z_{i};\mathbf{x}_i^T\boldsymbol{\beta},1)\]

For a multivariate linear model
\(\mathbf{Z}=\mathbf{X}\boldsymbol{\beta+\varepsilon}\) where
\(\boldsymbol{\varepsilon}\sim N_N(0,\mathbf{I})\), the least squares
estimates have
\(\hat\beta=(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Z}\) and
\(\hat\sigma^2_\beta=(\mathbf{X}^T\mathbf{X})^{-1}\). Thus, we can draw
\(\boldsymbol{\beta|y,Z}\) from
\(N_k(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Z},(\mathbf{X}^T\mathbf{X})^{-1})\)

The second conditional posterior pdf is

\[Z_i|\boldsymbol{y,\beta}\sim N(\mathbf{x}_i^T\boldsymbol{\beta},1)\begin{cases}\text{truncated at the left by 0} & \text{if } y_i=1 \\
 \text{truncated at the right by 0} & \text{if } y_i=0 \end{cases}\]

\hypertarget{logistic-models}{%
\section{Logistic Models}\label{logistic-models}}

In logistic models, the link function
\(p_{i}=H(z_i)= \frac{e^{\mathbf{x}_i^T\boldsymbol{\beta}}}{1+e^{\mathbf{x}_i^T\boldsymbol{\beta}}}\).
The log odds of success \(H^{-1}(p_i)=\log(\frac{p_i}{1-p_i})\)

The joint pmf is

\[\begin{aligned}
\pi(\boldsymbol{\beta|y}) &= \frac{\pi(\boldsymbol{\beta})}{C(y)}\prod_{i=1}^{N}[p(y_i|\beta)]\\
&\propto\pi(\boldsymbol{\beta})\prod_{i=1}^{N}\left[(\frac{e^{\mathbf{x}_i^T\boldsymbol{\beta}}}{1+e^{\mathbf{x}_i^T\boldsymbol{\beta}}})^{y_i}(1-\frac{e^{\mathbf{x}_i^T\boldsymbol{\beta}}}{1+e^{\mathbf{x}_i^T\boldsymbol{\beta}}})^{1-y_i}\right]\\
&=\pi(\boldsymbol{\beta})\prod_{i=1}^{N}\left[\frac{e^{y_i\mathbf{x}_i^T\boldsymbol{\beta}}}{1+e^{\mathbf{x}_i^T\boldsymbol{\beta}}}\right]\\
\end{aligned}\]

\hypertarget{single-layer-da-methods}{%
\subsection{Single-layer DA methods}\label{single-layer-da-methods}}

\citet{polsonBayesianInferenceLogistic2013} propose a DA approach which
only need a single layer of latent variables, which involve the
Pólya--Gamma distribution. Denote
\(W_i \sim PG(1,|\mathbf{x}^T_{i}\boldsymbol{\beta}|)\),i=1:n, then (no
absolute value in polson2013)

\[f(\omega|\beta)=\cosh{(\frac12\mathbf{x}^T\boldsymbol{\beta})}\exp\left[-\frac12(\mathbf{x}^T\boldsymbol{\beta})^2\omega\right]g(\omega)\]

where \(\cosh{c}=\frac12(e^c+e^{-c})=\frac{1+e^2c}{2e^{c}}\),
\(g(\omega)\) is free of \(\beta\) s.t.

\[g(\omega)=\sum_n^\infty(-1)^n\frac{2n+1}{\sqrt{2\pi\omega^3}}\exp[-\frac{(2n+1)^2}{8\omega}]\mathbb{I}_{(0,\infty)}(\omega)\]

Since \(y\) is observed data, \(\pi(\omega|\beta,y)=f(\omega|\beta)\).

Assign the prior of \(\boldsymbol{\beta}\) is \(\sim N_p(\mathbf{b,B})\)

\[\begin{aligned}
\pi(\boldsymbol{\beta,\omega|y})&= \frac{\pi(\boldsymbol{\beta})}{C(y)}\prod_{i=1}^{N}[p(y_i|\beta)]f(\omega_i|\beta)\\
&= \frac{\pi(\boldsymbol{\beta})}{C(y)}\prod_{i=1}^{N}\left[\frac{e^{y_i\mathbf{x}_i^T\boldsymbol{\beta}}}{1+e^{\mathbf{x}_i^T\boldsymbol{\beta}}}\right]\cosh{(\frac12\boldsymbol{x_i^T\beta})}\exp\left[-\frac12(\mathbf{x}_i^T\boldsymbol{\beta})^2\omega_i\right]g(\omega_i)\\
&\propto\phi_p(\mathbf{b,B})\prod_{i=1}^{N}\frac{e^{y_i\mathbf{x}_i^T\boldsymbol{\beta}}}{1+e^{\mathbf{x}_i^T\boldsymbol{\beta}}}\cdot\frac{1+e^{\mathbf{x}_i^T\boldsymbol{\beta}}}{2e^{\frac12\mathbf{x}_i^T\boldsymbol{\beta}}}\exp\left[-\frac12(\mathbf{x}_i^T\boldsymbol{\beta})^2\omega\right]\\
&=2^{-n}\phi_p(\mathbf{b,B})\prod_{i=1}^{N}\exp\left[(y_i-\frac12)\mathbf{x}_i^T\boldsymbol{\beta}-\frac12(\mathbf{x}_i^T\boldsymbol{\beta})^2\omega_i\right]\\
\end{aligned}\]

When \(\omega_i\) is known,
\(\pi(\boldsymbol{\beta,\omega|y})=\pi(\boldsymbol{\beta|\omega,y})\).
Let \(\boldsymbol{\Omega}=diag_n(\mathbf{\omega_i})\);
\(\boldsymbol{\kappa}=\mathbf{y_{1:n}}-\frac12\) Then we get to
conditional pmfs of \(\pi(\boldsymbol{\beta|\omega,y})\)

\[\begin{aligned}
\pi(\boldsymbol{\beta|\omega,y})&\propto\exp\left\{-\frac12\left[\boldsymbol{\beta'(\mathbf{x'\Omega x+B}^{-1})\beta-2\beta'(\mathbf{x'\kappa+B^{-1}b})}\right]\right\}\\
&\propto\exp\left\{-\frac12\mathbf{v}_\omega^{-1}[\boldsymbol{\beta'\mathbf{I}\beta-2\beta'\mathbf{v_\omega(x'\kappa+B^{-1}b})]}\right\}\\
&\propto\exp\left\{-\frac12\boldsymbol{(\beta-\mathbf{m_\omega)'v_\omega^{-1}(\beta-m}_\omega)}\right\}\\
\end{aligned}\] where
\(\mathbf{m_\omega=\underset{(p\cdot p)}{v_\omega}(\underset{(p\cdot n)}{x^T}(\mathbf{y_{1:n}}-\frac12)+B^{-1}b)}\);
\(\mathbf{v_\omega=(x^T\Omega x+B^{-1})^{-1}}\). Then we confirm that
\(\boldsymbol{\beta|\omega,y}\sim N_p(m_\omega,V_\omega)\)

To sample from the posterior distribution using the Pólya--Gamma method,
simply iterate two steps:

\[\begin{aligned}
(\omega_i|\boldsymbol{\beta})&\sim PG(n_i,\mathbf{x}_i^T\boldsymbol{\beta})\\
(\boldsymbol{\beta|y,\omega})&\sim N(\mathbf{m_\omega,v_\omega})
\end{aligned}\]

\hypertarget{other-bayesian-methods}{%
\subsection{Other Bayesian methods}\label{other-bayesian-methods}}

One method is similar with \citet{albertBayesianAnalysisBinary1993}. Let
\(y_i=\begin{cases}1&\text{if } z_i>0\\0&\text{if }z_i\le0\end{cases}\).
The \emph{latent utilities}
\(z_i=\mathbf{x}_i^T\boldsymbol{\beta}+\varepsilon_i\),
\(\varepsilon_i\sim Logistic(1)\)

The standard approach has been to add another layer of auxiliary
variables to handle the logistic error model on the latent-utility
scale. One strategy is to represent the logistic distribution as a
normal-scale mixture

\[(\epsilon_i|\phi_i)\sim N(0,\phi_i);\quad \phi_i=(2\lambda_i)^2; \lambda_i\sim KS(1)\quad\text{Kolmogorov–Smirnov distribution}\]

Alternatively, one may approximate the logistic error term as a discrete
mixture of normals.

\[(\epsilon_i|\phi_i)\sim N(0,\phi_i);\quad \phi_i=\sum_{k=1}^K\omega_k\delta_{\phi^{(k)}}\]

where \(\delta_{\phi}\) indicates a {\emph{Dirac measure}} at \(\phi\).
The weights \(\omega_k\) and the points \(\phi^{(k)}\) in the discrete
mixture are fixed for a given choice of \(k\) so that \emph{{the
Kullback--Leibler divergence}} from the true distribution of the random
utilities is minimized. Frühwirth-Schnatter and Frühwirth (2010) found
that the choice of \(K=10\) leads to a good approximation.

The discrete mixture of normals is an approximation, but it outperforms
the scale mixture of normals in terms of effective sampling rate, as it
is much faster.

\[\begin{tabular}{|l|l|l|}\hline
                 & {Albert and Chib (1993)} & Polson et al. (2013) \\\hline
{Gaussian}         & location mixture  & scale mixture \\
{Latent variables} & truncated normals & Polya-Gamma \\\hline
\end{tabular}\]

One may also arrive at the hierarchy above by manipulating the random
utility derivation of McFadden (1974)

The dRUM. One must use a table of different weights and variances
representing different normal mixtures, to approximate a finite
collection of type-III logistic distributions, and interpolate within
this table to approximate the entire family.

Another approximation: the use of a Student-t link function as a close
substitute for the logistic link. This also introduces a second layer of
latent variables, in that the Student-t error model for \(z_i\) is
represented as a scale mixture of normals.

\hypertarget{other-frequentist-methods}{%
\subsection{Other Frequentist methods}\label{other-frequentist-methods}}

\hypertarget{mle-methods}{%
\subsubsection{MLE methods}\label{mle-methods}}

\hypertarget{em-algoritm}{%
\subsubsection{EM Algoritm}\label{em-algoritm}}

\hypertarget{implementation}{%
\subsection{Implementation}\label{implementation}}

\hypertarget{summary}{%
\subsection{Summary}\label{summary}}

\emph{Our data-augmentation scheme differs from each of these approaches
in several ways.}

\emph{it does not appeal directly to the random-utility interpretation
of the logit model. Instead, it represents the logistic CDF as a mixture
with respect to an infinite convolution of gammas.}

\emph{the method is exact, in the sense of making draws from the correct
joint posterior distribution, rather than an approximation to the
posterior that arises out of an approximation to the link function.}

\emph{like the Albert and Chib (1993) method, it requires only a single
layer of latent variables.}

\emph{In binary logit models, the Pólya--Gamma is more efficient than
all previously proposed data-augmentation schemes.}

\emph{the Pólya--Gamma method always had a higher effective sample size
than the two default Metropolis samplers we tried. }

\emph{the Pólya--Gamma method truly shines when the model has a complex
prior structure.}

\bibliographystyle{agsm}
\bibliography{stat501.bib}

\end{document}


\subsection{Thread Ring}

The thread ring benchmark originates in the Computer Language
Benchmarks Game\footnote{\url{http://shootout.alioth.debian.org/}}
(formerly known as the Great Computer Language Shootout).  It is a
simple concurrency benchmark, in which a large number of threads are
created in a ring topology, and then messages are passed around the
ring.  We include it here as an example of profiling a Concurrent
Haskell program using ThreadScope, in contrast to the other case
studies which have investigated programs that use semi-explicit
parallelism.

The code for our version of the benchmark is given in
Figure~\ref{f:threadring-code}.  This version uses a linear string of
threads rather than a ring, where a number of messages are pumped in
to the first thread in the string, and then collected at the other
end.

\begin{figure}
\begin{lstlisting}
import Control.Concurrent
import Control.Monad
import System
import GHC.Conc (forkOnIO)

thread :: MVar Int -> MVar Int -> IO ()
thread inp out = do
  x <- takeMVar inp
  putMVar out $! x+1
  thread inp out

spawn cur n = do
  next <- newEmptyMVar
  forkIO $ thread cur next
  return next

main = do
  n <- getArgs >>= readIO.head
  s <- newEmptyMVar
  e <- foldM spawn s [1..2000]
  f <- newEmptyMVar
  forkIO $ replicateM n (takeMVar e) >>= putMVar f . sum
  replicateM n (putMVar s 0)
  takeMVar f
\end{lstlisting}
\caption{ThreadRing code}
\label{f:threadring-code}
\end{figure}

Our aim is to try to make this program speed up in parallel.  We
expect there to be parallelism available: multiple messages are
being pumped through the thread string, so we ought to be able to pump
messages through distinct parts of the string in parallel.

First, the sequential performance.  This is for 500 messages and 2000 threads:

\begin{verbatim}
  INIT  time    0.00s  (  0.00s elapsed)
  MUT   time    0.18s  (  0.19s elapsed)
  GC    time    0.01s  (  0.01s elapsed)
  EXIT  time    0.00s  (  0.00s elapsed)
  Total time    0.19s  (  0.21s elapsed)
\end{verbatim}

Next, running the program on two cores:

\begin{verbatim}
  INIT  time    0.00s  (  0.00s elapsed)
  MUT   time    0.65s  (  0.36s elapsed)
  GC    time    0.02s  (  0.01s elapsed)
  EXIT  time    0.00s  (  0.00s elapsed)
  Total time    0.67s  (  0.38s elapsed)
\end{verbatim}

\begin{figure*}
\begin{center}
\includegraphics[scale=0.3]{threadring1.png}
\end{center}
\caption{ThreadRing profile (no explicit placement; zoomed in)}
\label{f:threadring1}
\end{figure*}

\begin{figure*}
\begin{center}
\includegraphics[scale=0.3]{threadring2.png}
\end{center}
\caption{ThreadRing profile (with explicit placement)}
\label{f:threadring2}
\end{figure*}

\begin{figure*}
\begin{center}
\includegraphics[scale=0.3]{threadring3.png}
\end{center}
\caption{ThreadRing profile (explicit placement and more messages)}
\label{f:threadring3}
\end{figure*}

Things are significantly slower when we add a core.  Let's examine the
ThreadScope profile to see why - at first glance, the program seems to
be using both cores, but as we zoom in we can see that there are lots
of gaps (Figure~\ref{f:threadring1}).

In this program we want to avoid communication between the two
separate cores, because that will be expensive.  We want as much
communication as possible to happen between threads on the same core,
where it is cheap.  In order to do this, we have to give the scheduler
some help.  We know the structure of the communication in this
program: messages are passed along the string in sequence, so we can
place threads optimally to take advantage of that.  GHC provides a way
to place a thread onto a particular core (or HEC), using the
\codef{forkOnIO} operation.  The placement scheme we use is to divide
the string into linear segments, one segment per core (in our case
two).

This strategy gets us back to the same performance as the sequential
version:

\begin{verbatim}
  INIT  time    0.00s  (  0.00s elapsed)
  MUT   time    0.23s  (  0.19s elapsed)
  GC    time    0.02s  (  0.02s elapsed)
  EXIT  time    0.00s  (  0.00s elapsed)
  Total time    0.26s  (  0.21s elapsed)
\end{verbatim}

Why don't we actually see any speedup?
Figure~\ref{f:threadring2} shows the ThreadScope profile.
The program has now been almost linearized; there is a small amount of
overlap, but most of the execution is sequential, first on one core
and then the other.

Investigating the profile in more detail shows that this is a
scheduling phenomenon.  The runtime has moved all the messages through
the first string before it propagates any into the second string, and
this can happen because the total number of messages we are using for
the benchmark is less than the number of threads.  If we increase the
number of messages, then we do actually see more parallelism.
Figure~\ref{f:threadring3} shows the execution profile for 2000
messages and 2000 threads, and we can see there is significantly more
overlap.


\documentclass[a4paper, 11pt]{article}
\usepackage{myfile}

\newcommand{\HRule}[1]{\rule{\linewidth}{#1}} 	% Horizontal rule

\makeatletter
\def\printtitle{
	{\centering \@title\par}}
\makeatother

\title{\vspace{-4ex} \textbf{Angle Chasing Problems Showcase} }
\date{National Camp 2018}
\author{\vspace{-3ex}M. Ahsan Al Mahir \thanks{Thanks to Euclidean Geometry in Mathematical Olympiads (EGMO) - Evan Chen, Yufei Zhao Handouts for being wonderful sources for many of the problems here. They are great resources for further reading.}}

\begin{document}

	\maketitle\bigskip

	Let's assume that you have learnt all the theorems and lemmas needed for chasing angles. All of these problems can be solved using angle chasing alone (ya literally), but you are free to solve them as you like. But it is highly encouraged that you try them with angle chasing first, as it will toughen up your angle chasing skills :D

	\bigskip\section{\textbf{Required Theorems}}

	You are really gonna need at least these theorems to continue:

	\theorem{ The sum of the three angles of a triangle equals to $ 180^{\circ} $}

	\theorem{ In a triangle $ ABC $ with cicrcumcenter $ O $, we have $ \angle BOC=2\times \angle BAC $}

	\theorem{Let $ ABC $ be a triangle inscribed in a circle $ \omega $. Show that $ AC \perp CB $ if and only if $ AB $ is a diameter of $ \omega $.}

	\theorem{Let $ O $ and $ H $ denote the circumcenter and orthocenter of an acute $ \triangle ABC $, respectively. Show that $ \angle BAH = \angle CAO $. }

	\theorem{Let $ ABCD $ be a convex cyclic quadrilateral. Then $ \angle ABC + \angle CDA =
		180^{\circ} $ and $ \angle ABD = \angle ACD $}

	\theorem{Let $ ABCD $ be a convex quadrilateral. Then the
		following are equivalent:
		\begin{enumerate}
			\item  $ ABCD $ is cyclic.
			\item  $ \angle ABC + \angle CDA = 180^{\circ} $.
			\item  $ \angle ABD = \angle ACD $.
	\end{enumerate}}

	\theorem{Suppose $ \triangle ABC $ is inscribed in a circle with
		center $ O $. Let $ P $ be a point in the plane. Then the following are equivalent:
		\begin{enumerate}
			\item  $ PA $ is tangent to $ (ABC) $.
			\item  $ OA \perp AP $ .
			\item  $ \angle PAB = \angle ACB $.
	\end{enumerate}}

	\bigskip\section{\textbf{Easy}}

	\prob{Let $ ABC $ be an acute triangle with circumcenter $ O $, and let $ K $ be a
		point such that $ KA $ is tangent to $ (ABC) $ and $ \angle KCB = 90^{\circ} $. Point $ D $ lies on $ BC $ such that
		$ KD \parallel AB$. Show that line $ DO $ passes through $ A $.}

	\probl{}{Extremely Useful}{}{Angles around the centers of a triangle $ ABC $:

		\begin{enumerate}
			\item If $ I $ is the incenter of $ ABC $ then $ \angle BIC = 90^{\circ} + \frac{a}{2} , \angle IBC = \frac{b}{2} and \angle ICB = \frac{c}{2} $ .

			\item If $ H $ is the orthocenter of $ ABC $ then $ \angle BHC = 180^{\circ} - a, \angle HBC = 90^{\circ} - c $ and $ \angle HCB = 90^{\circ} - b $.

			\item If $ O $ is the circumcenter of $ ABC $ then $ \angle BOC = 2a $ and $ \angle OBC = \angle OCB = 90^{\circ} - a $.

			\item If $ I_a $ is the $ A $-excenter of $ ABC $ then $ \angle AI_aB = \frac{c}{2} , \angle AI_aC = \frac{b}{2} $ and $ \angle BI_aC = 90^{\circ} - \frac{a}{2} $

		\end{enumerate}}

	\probl{}{Extremely Useful}{}{Pedal triangles of the centers of a triangle $ ABC $:

		\begin{enumerate}
			\item  If $ DEF $ is the triangle formed by projecting the incenter $ I $ onto sides $ BC, AC $
			and $ AB $, then $ I $ is the circumcenter of $ DEF $ and $ \angle EDF = 90^{\circ} - \frac{a}{2} $.
			\item  If $ DEF $ is the triangle formed by projecting the orthocenter $ H $ onto sides $ BC,
			AC $ and $ AB $, then $ H $ is the incenter of $ DEF $ and $ \angle EDF = 180^{\circ} - 2a $.
			\item  The medial triangle of $ ABC $ is the pedal triangle of the circumcenter $ O $ of $ ABC $
			and $ O $ is its orthocenter.
	\end{enumerate}}

	\prob{In scalene triangle $ ABC $, let $ K $ be the intersection of the angle bisector of	$\angle A$ and the perpendicular bisector of $ BC $. Prove that the points $ A, B, C, K $ are concyclic.}

	\prob{Let $AB$ and $CD$ be two segments, and let lines $AC$ and $BD$ meet at $X$. Let the circumcircles of $ABX$ and $CDX$ meet again at $O$. Prove that triangles $OAB$ and $OCD$ are similar.}

	\prob{	Let $L,M,N$ are the midpoints of $BC,CA,AB$ and $AD,BE,CF$ are altitudes of $\triangle ABC$. Prove that

		\begin{itemize}
			\item $O$ is the orthocenter of $\triangle LMN$.
			\item $H$ is the incenter of $\triangle DEF$.
			\item $D,E,F,L,M,N$ all lie on a circle.
			\item Let $BO \cap \bigodot ABC=Q$. Prove that $AQCH$ is a parallelogram
			\item Prove that the reflection of $ H $ on $ BC $ lies on the circumcenter.
			\item Prove that the reflection of the \textbf{Euler Line}\footnote{It is the line joining the orthocenter and the circumcenter} on the sides of $ \triangle ABC $
			concur at the circumcirle.
	\end{itemize}}

	\prob{\textbf{(Miquel’s theorem)} Let $ABC$ be a triangle. Points $X, Y$ and $Z$ lie on sides $BC, CA$ and $AB$,	respectively. Prove that the circumcircles of triangles $AYZ, BXZ, CXY$ meet at a common point.}

	\prob{\textbf{(Simson line)} Let $ABC$ be a triangle, and let $P$ be another point on its circumcircle. Let $X; Y; Z$ be the feet of perpendiculars from $P$ to lines $BC; CA; AB$ respectively. Prove that $X; Y; Z$ are collinear.}

	\prob{Let $\angle AOB$ be a right angle, $M$ and $N$ points on rays $OA$ and $OB$, respectively. Let $MNPQ$ be a square such that $MN$ separates the points $O$ and $P$. Find the locus of the center of the square when $M$ and $N$ vary.}

	\prob{An interior point $P$ is chosen in the rectangle $ABCD$ such that $\angle APD + \angle BPC = 180$. Find $\angle DAP + \angle BCP$.}

	\prob{Let $ ABC $ be an acute triangle inscribed in circle $ \omega $. Let $ X $ be the midpoint of the arc $ BC $ of $ \omega $ not containing $ A $ and define $ Y , Z $ similarly. Show that the orthocenter of $ XYZ $ is the incenter $ I $ of $ ABC $. }

	\prob{Let $ ABC $ be an acute triangle. Let $ BE $ and $ CF $ be altitudes of $ \triangle ABC $, and denote by $ M $ the midpoint of $ BC $. Prove that $ ME, MF, $ and the line through $ A $ parallel to $ BC $ are all tangents to $ (AEF) $.}

	\prob{The incircle of $ \triangle ABC $ is tangent to $ BC, CA, AB $ at $ D, E, F $, respectively. Let $ M $ and $ N $ be the midpoints of $ BC $ and $ AC $, respectively. Ray $ BI $ meets line $ EF $ at $ K $. Show that $ BK \perp CK $. Then show $ K $ lies on line $ MN $. }

	\prob{Prove that if the orthocentre lie on the circumcircle then the triangle is a right angled triangle.}

	\prob{Prove that the isogonal conjugate of a point is a point at infinity if and only if it lies on the circumcircle.}

	\prob{Let $ ABC $ be a triangle with orthocenter $ H $. If $ P $ is a point on $ (ABC) $ then its Simson line bisects $ PH $ .}

	\prob{	Given a triangle $ABC$, let $P$ lie on the circumcircle of the triangle and be the midpoint of the arc $BC$ which does not contain $A$. Draw a straight line $l$ through $P$ so that $l$ is parallel to $AB$. Denote by $k$ the circle which passes through $B$, and is tangent to $l$ at the point $P$. Let $Q$ be the second point of intersection of $k$ and the line $AB$ (if there is no second point of intersection, choose $Q = B$). Prove that $AQ = AC$.	}

	\bigskip\section{\textbf{Medium}}

	\probl{}{IMO 2004 P4}{E}{Let $ ABC $ be an acute triangle with orthocenter $ H $, and let $ W $ be a point on the side $ BC $, between $ B $ and $ C $. The points $ M $ and $ N $ are the feet of the altitudes drawn from $ B $ and $ C $, respectively. $ \omega_1 $ is the circumcircle of triangle $ BWN $ and $ X $ is a point such that $ WX $ is a diameter of $ \omega_1 $. Similarly, $ \omega_2 $ is the circumcircle of triangle $ CWM $ and $ Y $ is a point such that $ WY $ is a diameter of $ \omega_2 $. Show that the points $ X, Y, $ and $ H $ are collinear.}

	\probl{https://artofproblemsolving.com/community/c6h418633p2361970}{IMO Shortlist G1}{E}{Let $ ABC $ be an acute triangle with $ D, E, F $ the feet of the altitudes lying on $ BC, CA, AB $ respectively. One of the intersection points of the line $ EF $ and the circumcircle is $ P $. The lines $ BP $ and $ DF $ meet at point $ Q $. Prove that $ AP = AQ $.}

	\probl{https://artofproblemsolving.com/community/c6h1508225_simple_parallelogram_geo}{IOM 2017 P1}{E}{Let $ABCD$ be a parallelogram in which angle at $B$ is obtuse and $AD>AB$. Points $K$ and $L$ on $AC$ such that $\angle ADL=\angle KBA$(the points $A, K, C, L$ are all different, with $K$ between $A$ and $L$). The line $BK$ intersects the circumcircle  $\omega$ of $ABC$ at points $B$ and $E$, and the line $EL$ intersects $\omega$ at points $E$ and $F$. Prove that $BF\parallel AC$.}

	\probl{https://artofproblemsolving.com/community/c6h587990p3480801}{All Russian 2014 Grade 10 Day 1 P4}{E}{Given a triangle $ABC$ with $AB>BC$, let $ \Omega $ be the circumcircle. Let $M$, $N$ lie on the sides $AB$, $BC$ respectively, such that $AM=CN$. Let $K$ be the intersection of $MN$ and $AC$. Let $P$ be the incentre of the triangle $AMK$ and $Q$ be the $K$-excentre of the triangle $CNK$. If $R$ is midpoint of the arc $ABC$ of $ \Omega $ then prove that $RP=RQ$.}

	\probl{https://artofproblemsolving.com/community/c6h326960p1752047}{USA TST 2000 P2}{E}{Let $ ABCD$ be a cyclic quadrilateral and let $ E$ and $ F$ be the feet of perpendiculars from the intersection of diagonals $ AC$ and $ BD$ to $ AB$ and $ CD$, respectively. Prove that $ EF$ is perpendicular to the line through the midpoints of $ AD$ and $ BC$.}

	\probl{https://artofproblemsolving.com/community/c6h1289432p6815893}{IRAN 3rd Round 2016 P1}{E}{Let $ABC$ be an arbitrary triangle, $P$ is the intersection point of the altitude from $C$ and the tangent line from $A$ to the circumcircle. The bisector of angle $A$ intersects $BC$ at $D$ . $PD$ intersects $AB$ at $K$, if $H$ is the orthocenter then prove : $HK\perp AD$}

	\probl{https://artofproblemsolving.com/community/c6t48f6h1519616_geometry}{AoPS}{E}{$I$ is the incenter of $ABC$,  $PI,QI\perp BC$, $PA,QA$ intersect $BC$ at $DE$. Prove: $IADE$ is on a circle.}

	\probl{https://artofproblemsolving.com/community/c6h1434843p8120660}{IRAN 2nd Round 2016 P6}{E}{Let $ABC$ be a triangle and $X$ be a point on its circumcircle. $Q,P$ lie on a line $BC$ such that $XQ\perp AC , XP\perp AB$. Let $Y$ be the circumcenter of $\triangle XQP$. Prove that $ABC$ is equilateral triangle if and if only $Y$ moves on a circle when $X$ varies on the circumcircle of $ABC$}

	\probl{}{BAMO 1999, P2}{E}{Let $ O = (0,0), A = (0, a) $, and $ B = (0, b) $, where $ 0 < a < b $ are reals. Let $ \gamma $ be a circle with diameter $ AB $ and let $ P $ be any other point on $ \gamma $. Line $ PA $ meets the $ x $-axis again at $ Q $. Prove that $ \angle BQP = \angle BOP $}

	\probl{}{CGMO 2012 P5}{M}{Let $ ABC $ be a triangle. The incircle of $ \triangle ABC $ is tangent to $ AB $ and $ AC $ at $ D $ and $ E $ respectively. Let $ O $ denote the circumcenter of $ \triangle BCI $. Prove that $ \angle ODB = \angle OEC $}

	\probl{}{Canada 1991 P3}{M}{Let $ P $ be a point inside circle $ \omega $. Consider the set of chords of $ \omega $ that contain $ P $. Prove that their midpoints all lie on a circle.}

	\probl{}{Russia 1996}{M}{Points $ E $ and $ F $ are on side $ BC $ of convex quadrilateral $ ABCD $ (with $ E $ closer than $ F $ to $ B $). It is known that $ \angle BAE = \angle CDF $ and $ \angle EAF = \angle FDE $. Prove that $ \angle FAC = \angle EDB $.}

	\probl{}{JMO 2011 P5}{M}{Points $ A, B, C, D, E $ lie on a circle $ \omega $ and point $ P $ lies outside the circle. The given points are such that:

		\begin{itemize}
			\item lines $ PB $ and $ PD $ are tangent to $ \omega $
			\item $ P, A, C $ are collinear
			\item $ DE\parallel AC $
		\end{itemize}

		Prove that $ BE $ bisects $ AC $.}

	\probl{}{Canda 1997 P4}{M}{The point $ O $ is situated inside the parallelogram $ ABCD $ such that $ \angle AOB + \angle COD = 180^{\circ} $. Prove that $ \angle OBC = \angle ODC $.}

	\probl{}{USAMO 2010 P1}{M}{Let $ AXYZB $ be a convex pentagon inscribed in a semicircle of diameter $ AB $. Denote by $ P , Q, R, S $ the feet of the perpendiculars from $ Y $ onto lines$  AX, BX, AZ, BZ, $ respectively. Prove that the acute angle formed by lines $ PQ $ and $ RS $ is half the size of $ \angle XOZ $, where $ O $ is the midpoint of segment $ AB $. }

	\probl{https://artofproblemsolving.com/community/c6h1597669p9926981}{Romanian Masters in Mathematics 2018 P1}{M}{Let $ABCD$ be a cyclic quadrilateral an let $P$ be a point on the side $AB.$ The diagonals $AC$ meets the segments $DP$ at $Q.$ The line through $P$ parallel to $CD$ mmets the extension of the side $CB$ beyond $B$ at $K.$ The line through $Q$ parallel to $BD$ meets the extension of the side $CB$ beyond $B$ at $L.$ Prove that the circumcircles of the triangles $BKP$ and $CLQ$ are tangent .}

	\prob{$ D $ be a point such that $ ABDC $ is a parallelogram and $ E $ be the intersection of the $ B,C $ tangents. Prove that $ D,E $ are isogonal conjugates.}

	\probl{https://artofproblemsolving.com/community/c5235_2012_china_national_olympiad}{China 2012 P1}{M}{In the triangle $ABC$, $\angle A$ is biggest. On the circumcircle of $\triangle ABC$, let $D$ be the midpoint of $\widehat{ABC}$ and $E$ be the midpoint of $\widehat{ACB}$. The circle $c_1$ passes through $A,B$ and is tangent to $AC$ at $A$, the circle $c_2$ passes through $A,E$ and is tangent $AD$ at $A$. $c_1$ and $c_2$ intersect at $A$ and $P$. Prove that $AP$ bisects $\angle BAC$.}

	\prob{	Consider a circle $C_1$ and a point $O$ on it. Circle $C_2$ with center $O$, intersects $C_1$ in two points $P$ and $Q$. $C_3$ is a circle which is externally tangent to $C_2$ at $R$ and internally tangent to $C_1$ at $S$ and suppose that $RS$ passes through $Q$. Suppose $X$ and $Y$ are second intersection points of $PR$ and $OR$ with $C_1$. Prove that $QX$ is parallel with $SY$.	}

	\bigskip\section{Not So Easy...}

	That doesn't mean you shouldn't try them...

	\probl{https://artofproblemsolving.com/community/c6h582820p3444910}{APMO 2014 P5}{H}{Circles $\omega$ and $\Omega$ meet at points $A$ and $B$. Let $M$ be the midpoint of the arc $AB$ of circle $\omega$ ($M$ lies inside $\Omega$). A chord $MP$ of circle $\omega$ intersects $\Omega$ at $Q$ ($Q$ lies inside $\omega$). Let $\ell_P$ be the tangent line to $\omega$ at $P$, and let $\ell_Q$ be the tangent line to $\Omega$ at $Q$. Prove that the circumcircle of the triangle formed by the lines $\ell_P$, $\ell_Q$ and $AB$ is tangent to $\Omega$.}

	\probl{}{(Poncelet Point)}{H}{
		\begin{enumerate}
			\item Prove that for a quadruple $ {W, X, Y, Z} $ of points, the nine point circles of the triangles formed by points in this set, the pedal cicles of the points with respect to the triangle formed by the other three are concurrent. We call this the \textbf{Poncelet Point} of the quadruple.

			\item Prove that the \textbf{Feurbach Point} is the Poncelet point of the quadruple $ {A, B, C, I} $ where $ I $ is the incenter of $ \triangle ABC $.

	\end{enumerate}	}

	\probl{https://artofproblemsolving.com/community/c6h159997p894696}{Canada 2007 P5}{H}{Let the incircle of triangle $ ABC$ touch sides $ BC,\, CA$ and $ AB$ at $ D,\, E$ and $ F,$ respectively. Let $ \omega,\,\omega_{1},\,\omega_{2}$ and $ \omega_{3}$ denote the circumcircles of triangle $ ABC,\, AEF,\, BDF$ and $ CDE$ respectively. Let $ \omega$ and $ \omega_{1}$ intersect at $ A$ and $ P,\,\omega$ and $ \omega_{2}$ intersect at $ B$ and $ Q,\,\omega$ and $ \omega_{3}$ intersect at $ C$ and $ R.$

		\begin{itemize}
			\item Prove that $ \omega_{1},\,\omega_{2}$ and $ \omega_{3}$ intersect in a common point.
			\item Show that $ PD,\, QE$ and $ RF$ are concurrent.
		\end{itemize}
	}

	\probl{https://artofproblemsolving.com/community/c6h418983p2365045}{IMO 2011 P6}{H}{Let $ABC$ be an acute triangle with circumcircle $\Gamma$. Let $\ell$ be a tangent line to $\Gamma$, and let $\ell_a, \ell_b$ and $\ell_c$ be the lines obtained by reflecting $\ell$ in the lines $BC$, $CA$ and $AB$, respectively. Show that the circumcircle of the triangle determined by the lines $\ell_a, \ell_b$ and $\ell_c$ is tangent to the circle $\Gamma$.}

	\probl{https://artofproblemsolving.com/community/c6h527557p2996617}{Canda 2013 P5}{H}{Let $O$ denote the circumcentre of an acute-angled triangle $ABC$. Let point $P$ on side $AB$ be such that $\angle BOP = \angle ABC$, and let point $Q$ on side $AC$ be such that $\angle COQ = \angle ACB$. Prove that the reflection of $BC$ in the line $PQ$ is tangent to the circumcircle of triangle $APQ$.}

	\prob{Let $ ABC $ be an acute-angled triangle, and let $ P $ and $ Q $ be two points on its side $ BC $. Construct a point $ C_1 $ in such a way that the convex quadrilateral $ APBC_1 $ is cyclic, $ QC_1 \parallel CA $, and the points $ C_1 $ and $ Q $ lie on opposite sides of the line $ AB $. Construct a point $ B_1 $ in such a way that the convex quadrilateral $ APCB_1 $ is cyclic, $ QB1 \parallel BA, $ and the points $ B_1 $ and $ Q $ lie on opposite sides of the line $ AC $.

	Prove that the points $ B_1, C_1, P, $ and $ Q $ lie on a circle}

\end{document}


\title{Micrometeorology Assignment 3}
\documentclass[11pt]{article}
\usepackage{setspace} %to set line spacing
\usepackage{fullpage} %uses up more of the page
\usepackage{titlesec} %to edit section titles
\usepackage[round]{natbib} %for harvard referencing
\usepackage{graphicx} %to import graphics
\usepackage{subcaption}
\usepackage{siunitx}
\usepackage[title]{appendix}
\usepackage{float}%for the H option in figures
\usepackage{bigstrut} %for excel to latex tabular converter
\usepackage{lscape} %for landscape pages
\usepackage{pdfpages}
\usepackage{amsmath} %\begin{align} - \end{align}
\usepackage[framed,numbered]{mcode}  %imports and formats .m files

%\myFigure{filename.jpg}{Description}{figx}{size}
%\myFigure{}{}{}{}

\newcommand{\myFigure}[4][0.8]{
\begin{figure}[H]

	\centering
	\setlength\fboxsep{1pt}
	\setlength\fboxrule{0.5pt}
%\fbox{\includegraphics[width=#1\textwidth]{#2}} %fbox draws a border.
	\includegraphics[width=#1\textwidth]{#2}
	\caption{\small #3}
	\label{#4}
\end{figure}
}

\setlength{\parindent}{0cm} %removes paragraph indentation

%\titleformat{\section}{\large\bfseries}{\thesection}{1em}{}
\titleformat{\subsection}{\normalsize\bfseries}{\thesubsection}{1em}{}
\titleformat{\subsubsection}{\normalsize\it\bfseries}{\thesubsubsection}{1em}{}
\begin{document}

\setcounter{page}{1}
\begin{Large}\begin{center}
\textbf{Introduction to Micrometeorology}

\textbf{Assignment 3 - Turbulence and Spectra}
\end{center}\end{Large}

%\setstretch{1} \\
\begin{center}
\begin{tabular}{rcl}
Jaime Liew && s141777 \\
Young Kwon && s163075 \\~\\
\end{tabular}
\end{center}
%\raggedright

\section{Introduction}
This assignment focuses on spectral analysis of time series. Up until now, the main method of analyzing turbulence in wind was through basic statistical analysis in the time domain. Spectral analysis is performed in the frequency domain, and is useful for determining the periodic behaviour of wind, and the energy content available at each frequency.

\section{Spectra and the Discrete Fourier Transform}
The power spectral density of a time series shows the energy content of a signal at each frequency. These frequencies can often be related to specific time scales. For example for a spectrum covering a year's worth of data, higher frequencies are associated with the variations in wind speeds that happen within minutes while the lower frequencies to the left of a spectrum are related to the variations that happen within days. To obtain the power spectrum of a continuous time series, the Fourier transform of the signal's autocovariance, $R_X(t)$, is taken:
$$S_X(\omega) = \frac{1}{2\pi}\int^\infty_{-\infty}R_X(\tau)e^{-i\omega\tau}d\tau$$
where $\omega$ is the angular frequency in radians per second. As the autocovariance must be calculated, this method for finding the power spectrum is computationally expensive. An equivalent way of computing the spectrum is:
$$S_X(\omega) = \frac{1}{2\pi T}\left|\int^T_{0}X(\tau)e^{-i\omega\tau}d\tau\right|^2$$
This method does not require the autocovariance, saving computational power. This assignment deals with discrete time series, which has an equivalent expression:
$$S_X(\omega) = \frac{1}{2\pi Nf_s}\left|\sum^{N-1}_{n=0}X_ne^{-i\frac{2\pi ln}{N}}d\tau\right|^2$$
where the sum term is the discrete Fourier transform (DFT). The discrete Fourier transform can be calculated using the fast Fourier transform algorithm (FFT), which is able to reduce the computational time of the transformation from $\mathcal{O}(n^2)$ time to $\mathcal{O}(n\log n)$ time.
It is worth noting that the spectrum can only be calculated up to half of the sampling frequency, $f_s = 1/\Delta t$, which is also known as the Nyquist frequency, $f_N$. Since the Fourier transform returns a symmetric spectrum from $-f_N$ to $f_N$, only the positive half is considered in this assignment. To conserve spectrum power, the one sided spectrum is doubled in all figures.
\subsection{Log Smoothing Algorithm}
Looking at Figure \ref{fig:spec1}, the blue lines represent the spectral power density of a two hour wind speed data set. As can be seen from the graph, the fluctuations of the spectrum increase dramatically as frequency increases. To mitigate this problem, the spectrum is filtered by dividing the spectrum into logarithmically spaced bins, and representing the logarithmic midpoint as the mean of each bin. For logarithmic spacing in base 10, the logarithmic mid point of a bin is located $10^{-\frac{1}{2}} \approx 31.6\%$ along the length of the bin interval. These smoothed spectral estimates are shown in orange in Figure \ref{fig:spec1}. 15 bin divisions are made for each decade.
\subsection{K Averaging}
By assuming stationarity in the time series, it is possible to split a long time series into multiple smaller series and take the average of each of their power spectra. In this case, $K$ is a predetermined value that specifies the number of divisions that are to be made:
$$S_X(\omega) = \langle S_{X,k}(\omega)\rangle$$

This strategy of $K$ averaging helps to smooth out the graph of the spectrum, especially in the high frequency range. It should be noted that taking the spectrum of shorter time series does not change the domain of the spectrum. Instead, it reduces the spectral resolution as the number of frequency bins in the spectrum is equal to the length of the time series signal. Information can be lost for large values of $K$ as the series length is less, and therefore the spectrum has a lower spectral resolution.

\section{2 Hour Sprog\o s Data Set}
Wind velocity data is collected over a 2 hour period with a sampling frequency of $f_s = 10 Hz$. As seen in Figure \ref{fig:time}, there are a number of large outlier values, which were omitted from the analysis. Ideally, data points would not be removed from a time series when performing spectral analysis as it introduces discrepancies in the sample spacing. However, this effect is considered negligible due to the large number of data points.

\myFigure{Figures/figTime}{Time series data measured at the island of Sprog\o.}{fig:time}

The spectrum is generated for the time series, and the logarithmic smoothing algorithm is used applied. Both are plotted in Figure \ref{fig:spec1}.
\myFigure{Figures/figSpec1}{Spectral power density of short term wind speed time series.}{fig:spec1}

The fluctuations in the spectrum can be mitigated by splitting the time series into $K$ parts, and averaging their spectra. Figure \ref{fig:specK} shows how the spectrum (blue line) tends to converge with larger values of $K$. Although higher $K$ values produce a smoother spectrum, it also loses information as the spectral resolution decreases.
\\~\\
The spectrum is validated by comparing the area under the spectrum to the variance. If the spectrum calculation is consistent, then it should satisfy:

$$\int_{-\infty}^\infty S_X(f)df = \sigma_X^2$$

Table \ref{table:var} shows how these quantity vary for different values of K, as well as the area under the log-smoothed spectra. The true value for the variance is calculated to be $\sigma_X^2\approx 4.129$. It is clear that the $K$-average spectra maintain the correct variance properties as the area remains relatively constant. The log-smoothed curves do not remain consistent, however. This is due to the fact that the log-smoothing takes the average at the logarithmic midpoint, and it therefore misses the first points of the spectrum which contains a large amount of energy. The algorithm would need to be modified to add these initial points. $K$-averaging produces a converging spectrum while maintaining the variance properties of the spectrum.
\begin{table}[H]
\centering
\begin{tabular}{|c |c|c|c|c|}
\hline
$K$ & 1 & 10 & 50 & 100 \\
\hline
Spectrum Area & 4.129& 4.130& 4.133&4.136\\
log-smoothed Spectrum Area & 3.85& 3.034& 2.276&1.736\\
\hline
\end{tabular}
\caption{Area under spectra and log-smoothed spectra for various values of $K$.  }
\label{table:var}
\end{table}
\myFigure{Figures/figKspecs}{Spectral power density of short term wind speeds for varying values of K.}{fig:specK}

Interestingly, the spectrum closely resembles Kolmogorov's 5/3 law. This can be seen in Figure \ref{fig:spec53} as the slope of the loglog spectrum tends to $-5/3=-1.\bar{6}$. Kolmogorov's 5/3 law states that energy per length scale is a function of the eddy length scale, and follows the power law $S_X(k) \propto k^{-5/3}$. Where $S_X(k)$ is the spacial spectrum, and $k=1/L$ where L is the length scale of the turbulence. Although Kolmogorov's 5/3 law applies to length scales, it can be shown that length scales and time scales are proportional if Taylor's frozen turbulence hypothesis is assumed.
\myFigure{Figures/fig53Law}{Spectral energy density of short term wind speed time series. For higher frequencies, the energy follows Kolmogorov’s “5/3” Law.}{fig:spec53}

\section{21 Year Sprog\o s Data Set}
The data set used in assignment 1 is used to perform spectral analysis. This data set contains 21 years of wind speeds with sampling periods of 10 minutes. The data set is initially sanitized to remove invalid readings.
\\~\\
The spectrum and the log-smoothed spectrum is plotted in Figure \ref{fig:spec2}. The frequencies of a year ($~3.17\times10^{-8}Hz$) and a day ($~1.16^{-5}Hz$) are indicated with dotted lines in order to illustrate the annual and diurnal peaks. There is a clear annual peak due to the seasons. A small diurnal peak can also be seen in the log-smoothed spectrum.

\myFigure{Figures/figSpec2}{Spectral power density of long term wind speed time series.}{fig:spec2}
From looking at Figure \ref{fig:spec2}, it is clear that the power is higher in the low frequency region and tapers off in the high frequency region of the spectrum. Intuitively this makes sense. The low frequency region represents the power from the usual expected wind speeds throughout months or years of wind at this site, and the wind speeds on these larger time scales are what comprise most of the power generated by the wind turbine. In the high frequency region, the power is related to the fluctuating wind speeds that last on smaller time scales such as minutes or hours. Since this turbulent behaviour is fleeting, the power contributed by these wind oscillations are expected to be low.

\section{Autoregressive Process}
Spectral analysis was performed on an autoregressive process. It was established in the previous assignment that the autocorrelation of an autoregressive process is:
$$R_X = \frac{\alpha^{|t|}}{1-\alpha^2}$$

The absolute value of the time lag, $t$, is taken to represent the symmetric nature of the autocorrelation function. This relation can be used to find the theoretical value for the autoregressive spectrum. The spectrum as a function of frequency, $f$, which has units of Hertz is:

$$S_X(f) = \int_{-\infty}^\infty R_X(\tau)e^{-i2\pi f\tau}d\tau$$
Substituting the expression for $R_X$ into this equation gives:
$$S_X = \int_{-\infty}^\infty \frac{\alpha^{|\tau|}}{1-\alpha^2}e^{-i2\pi f\tau}d\tau$$
This expression can be decomposed into the co-spectrum and quadrature spectrum given Euler's formula, which states $e^{ix} = cos(x) + isin(x)$.
$$S_X(f) = \int_{-\infty}^\infty \frac{\alpha^{|\tau|}}{1-\alpha^2}cos(2\pi f \tau )d\tau - i\int_{-\infty}^\infty \frac{\alpha^{|\tau|}}{1-\alpha^2}sin(2\pi f \tau )d\tau$$
Since $R_X$ is an even function, the second integral is odd. Since the bounds of integration are symmetric, this integral evaluates to zero. Similarly, the first integral can be expressed as a one sided integral as it is even. What is left is a purely real expression for the spectral power density, which is the case for all autospectra of real time series:
$$S_x(f) =\frac{2}{1-\alpha^2}\int_{0}^\infty \alpha^{\tau}cos(2\pi f \tau )d\tau$$

Given that
$$\int_0^\infty a^xcos(bx)dx = -\frac{ln(a)}{ln^2(a)+b^2} \qquad \qquad \qquad \text{for }\quad a<1$$
It can be shown that
$$S_X(f) = \frac{-2ln(\alpha)}{(1-\alpha^2)(ln^2(\alpha) + 4\pi^2f^2)}$$

The spectral power density is found for autoregressive processes with varying values of $\alpha$ (Figure \ref{fig:spec3}), and are compared with the theoretical spectrum derived above. The sampling frequency is assumed to be $f_S = 1Hz$, and the signal length is $n=10^5$.

\myFigure[0.95]{Figures/figSpec3}{Spectral power density of autoregressive processes with varying $\alpha$.}{fig:spec3}

The calculated spectrum closely follows the theoretical spectrum function. Comparing the plots in Figure \ref{fig:spec3} to the spectra in Figures \ref{fig:spec1} and \ref{fig:spec2}, it is clear that the autoregressive process does not closely match the wind speed data spectra. Wind speeds tend to have a drop in power density for higher frequencies, whereas the autoregressive process continues to increase for the majority of frequencies.

\section{Conclusion}
In this assignment, three different signals of wind speeds were evaluated using power spectrum analysis. This process involved converting the time series of each data set into the frequency domain by using the discrete Fourier transform which was implemented by using the fast Fourier transform (FFT) in the code. The validity of using the FFT to find the spectrum was proved by how close the values of the integrals of the power spectra were to the variance of the time series. It was possible to see how useful a tool spectrum analysis can be for breaking down the power contribution from the wind speeds at each time scale. For the autoregressive process, the theoretical calculation of the spectrum was confirmed to be equivalent to the Fourier transform of the autocovariance function by showing that the power spectrum of the time series using FFT converged towards the theoretical values. Like in the second assignment, it was shown again that an autoregressive process is not an accurate method of approximating wind speed data as the spectrum of the autoregressive process was quite different to that of the measured time series.
\pagebreak
\begin{appendices}
\section{Spectrum Function}
\lstinputlisting{Scripts/makeSpectrum.m}

\section{Log-Smoothing Function}
\lstinputlisting{Scripts/logSmoothing.m}

\section{$K$-Averaging Spectrum Function}
\lstinputlisting{Scripts/makeMeanSpectrum.m}

\section{Main Script}
\lstinputlisting{Scripts/Micro3.m}
%\label{App1}
%\lstinputlisting{autoRegProcess.m}
%\lstinputlisting{Assigment2.m}

\end{appendices}

\end{document}


Custom data types offer you the opportunity to define how data is organised in your program. You can create records that contain a number of fields, unions that store a single field value, and enumerations that define their own list of values. To help you understand these concepts better the following illustrations demonstrate how these values are stored in the variables in your code.

\subsection{Understanding \texttt{Read Row}} % (fold)
\label{ssub:understanding_read row}

\sref{sub:designing_small_db} \nameref{sub:designing_small_db} described the design of a Small DB program. The program allows the user to enter some values that were then stored in variables in the program, making use of records, unions, and enumerations. The design for this program included a number of functions and procedures, one of which was the \texttt{Read Row} function. This function was responsible for reading a row value from the user and returning it in a \texttt{Row} record. The flowchart for this function is shown in \fref{fig:read-row-flow-understanding}, which is a repeat of \fref{fig:read-row-flow}.

The following illustrations will show this code running to read in a value from the user. This will demonstrate how the computer stores values in record, union, and enumeration variables.

The illustrations will show the following:
\begin{enumerate}
  \item \nameref{ssub:code_is_loaded_for_small_db}
  \item \nameref{ssub:read_row_is_called_to_read_in_row_with_id_0}
  \item \nameref{ssub:step_1_stores_the_value_0_in_result_s_id_field}
  \item \nameref{ssub:a_double_value_is_entered_by_the_user}
  \item \nameref{ssub:the_double_data_is_stored_in_the_row}
  \item \nameref{ssub:the_result_row_is_returned_to_main}
  \item \nameref{ssub:this_process_is_repeated_for_each_element_of_the_array}
\end{enumerate}

\begin{figure}[p]
   \centering
   \includegraphics[width=0.80\textwidth]{./topics/type-decl/diagrams/ReadRowFlow}
   \caption{Flowchart for \texttt{Read Row}, from \fref{fig:read-row-flow}}
   \label{fig:read-row-flow-understanding}
\end{figure}

\clearpage
\subsubsection{Code is loaded for Small DB} % (fold)
\label{ssub:code_is_loaded_for_small_db}

When the program starts its code is loaded into memory and its \texttt{main} procedure is started.

\begin{figure}[htbp]
   \centering
   \includegraphics[width=0.9\textwidth]{./topics/type-decl/images/ReadRow1}
   \caption{When the program starts \texttt{Main} allocates space for its local variables, including the array}
   \label{fig:read-row-vis-1}
\end{figure}

\mynote{
\begin{itemize}
  \item In \fref{fig:read-row-vis-1} the indicated areas show the following:
  \begin{enumerate}
    \item The Program starts and \texttt{Main} is loaded onto the stack, allocating space for its local variables.
    \item The \texttt{db\_data} array is allocated space to store its values. Each element of the array has the fields declared in the \texttt{Row} record structure.
    \item Each \texttt{Row} has an \texttt{id}, \texttt{kind}, and \texttt{data} value.
    \item Each of these \texttt{data} values has \emph{either} a \texttt{int\_val}, a \texttt{dbl\_val} or a \texttt{txt\_val}.
  \end{enumerate}
  \medskip
  \item Notice that the values in the array are allocated next to each other.
  \item In the illustration the \texttt{Row}'s \texttt{data} field will have only one of its fields highlighted, indicating which field is currently stored in the union.
\end{itemize}
}

% subsubsection code_is_loaded_for_small_db (end)

\clearpage
\subsubsection{\texttt{Read Row} is called to read in row with id 0} % (fold)
\label{ssub:read_row_is_called_to_read_in_row_with_id_0}

The \texttt{Read Row} function is called to read a \texttt{row} value from the user. This will check what the user has entered and store an appropriate value in the \texttt{Row} it returns.

\begin{figure}[htbp]
   \centering
   \includegraphics[width=\textwidth]{./topics/type-decl/images/ReadRow2}
   \caption{At step 2 \texttt{Main} calls \texttt{Read Row}, getting it to read in the $i^{th}$ row from the user}
   \label{fig:read-row-vis-2}
\end{figure}

\mynote{
\begin{itemize}
  \item In \fref{fig:read-row-vis-2} the indicated areas show the following:
  \begin{enumerate}
    \item When \texttt{Read Row} is called it is allocated space on the stack.
    \item \texttt{Read Row} will be returning a \texttt{Row} value, so its result will have \texttt{id}, \texttt{kind}, and \texttt{data} values as these are what is specified in the \texttt{Row} record's definition.
  \end{enumerate}
  \medskip
  \item Each row will have the same kind of data stored within it. The details of this are specified in the \texttt{Row} record's definition.
\end{itemize}
}

% subsubsection read_row_is_called_to_read_in_row_with_id_0 (end)

\clearpage
\subsubsection{Step 1 stores the value 0 in \texttt{result}'s \texttt{id} field} % (fold)
\label{ssub:step_1_stores_the_value_0_in_result_s_id_field}

\begin{figure}[htbp]
   \centering
   \includegraphics[width=0.98\textwidth]{./topics/type-decl/images/ReadRow3}
 \caption{Step 1 of \texttt{Read Row} stores a value in \texttt{result}'s id}
 \label{fig:read-row-vis-3}
\end{figure}

\mynote{
\begin{itemize}
\item In \fref{fig:read-row-vis-3} the indicated areas show the following:
\begin{enumerate}
  \item Step 1 of \texttt{Read Row} assigns a value to \texttt{result.id}.
  \item In this code \texttt{result} refers to this variable in \texttt{Read Row}.
  \item The \texttt{id} part then refers to this field.
  \item As a result the value from the \texttt{next id} parameter is read, and its value is assigned to \texttt{result.id}.
\end{enumerate}
\medskip
\item Each part of \texttt{result.id} refers to a different kind of data.
\item \texttt{result} is a row, this has \texttt{id}, \texttt{kind}, and \texttt{data} fields.
\item \texttt{result.id} is an \texttt{Integer}, it is the id field of the \texttt{result Row}.
\end{itemize}
}

% subsubsection step_1_stores_a_value_in_result_s_id_field (end)

\clearpage
\subsubsection{A double value is entered by the user} % (fold)
\label{ssub:a_double_value_is_entered_by_the_user}

\begin{figure}[htbp]
   \centering
   \includegraphics[width=0.98\textwidth]{./topics/type-decl/images/ReadRow4}
 \caption{A double value is entered by the user, so the code must store the double in the row}
 \label{fig:read-row-vis-4}
\end{figure}

\mynote{
\begin{itemize}
\item In \fref{fig:read-row-vis-4} the indicated areas show the following:
\begin{enumerate}
  \item At Step 3 the computer reads the text entered by the user.
  \item The value read is stored in the \texttt{line} variable.
  \item At Step 7 the code determines that the data is a double, and execution will proceed to step 8.
\end{enumerate}
\medskip
\end{itemize}
}

% subsubsection a_double_value_is_entered_by_the_user (end)

\clearpage
\subsubsection{The double data is stored in the row} % (fold)
\label{ssub:the_double_data_is_stored_in_the_row}

\begin{figure}[htbp]
   \centering
   \includegraphics[width=\textwidth]{./topics/type-decl/images/ReadRow5}
 \caption{A double value is entered by the user, so the code must store the double in the row}
 \label{fig:read-row-vis-5}
\end{figure}

\mynote{
\begin{itemize}
\item In \fref{fig:read-row-vis-5} the indicated areas show the following:
\begin{enumerate}
  \item Step 8 of \texttt{Read Row} stores the double value entered by the user into the \texttt{dbl\_val} field of the \texttt{data} field of the \texttt{result Row}.
  \item Notice that the union is now shown as storing a value in its \texttt{dbl\_val} field.
  \item Step 9 stores the \texttt{DBL\_VAL} value in the \texttt{kind} field of the \texttt{result Row}. This is one of the values from the \texttt{Data Kind} enumeration.
  \item At this point all of the fields of \texttt{result} have been assigned values.
\end{enumerate}
\medskip
\end{itemize}
}

% subsubsection the_double_data_is_stored_in_the_row (end)

\clearpage
\subsubsection{The \texttt{result} row is returned to \texttt{Main}} % (fold)
\label{ssub:the_result_row_is_returned_to_main}

\begin{figure}[htbp]
   \centering
   \includegraphics[width=\textwidth]{./topics/type-decl/images/ReadRow6}
 \caption{The \texttt{result Row} is returned to \texttt{Main}}
 \label{fig:read-row-vis-6}
\end{figure}

\mynote{
\begin{itemize}
\item In \fref{fig:read-row-vis-6} the indicated areas show the following:
\begin{enumerate}
  \item At the end of \texttt{Read Row} the \texttt{result Row} is returned to \texttt{Main}.
  \item In \texttt{Main} the value is used in an assignment statement, that assigns it to the $i^{th}$ value in the \texttt{db\_data} array. As \texttt{i} is currently 0, this stores the \textbf{entire} row in \texttt{dd\_data[0]}.
  \item When this assignment occurs all of the data from the \texttt{result} of \texttt{Read Row} is copied into \texttt{db\_data[0]}.
\end{enumerate}
\medskip
\item With Records and Unions you can read/write to individual fields using the dot notation, or you can access the entire record.
\end{itemize}
}

% subsubsection the_result_row_is_returned_to_main (end)

\clearpage
\subsubsection{This process is repeated for each element of the array} % (fold)
\label{ssub:this_process_is_repeated_for_each_element_of_the_array}

\begin{figure}[htbp]
   \centering
   \includegraphics[width=0.92\textwidth]{./topics/type-decl/images/ReadRow7}
 \caption{The for loop ensures that a \texttt{Row} value is read in for each element of the array}
 \label{fig:read-row-vis-7}
\end{figure}

\mynote{
\begin{itemize}
\item In \fref{fig:read-row-vis-7} the indicated areas show the following:
\begin{enumerate}
  \item Each time through the loop the value is written to an element in the array. This is showing the last iteration where \texttt{i} is 2.
  \item The second time this loop was executed the user entered an integer value, this is now stored in the second element of the array.
  \item The third time through the loop, the result returned is storing a string value. This is returned to \texttt{Main}, and then stored in the \texttt{db\_data} array.
\end{enumerate}
\medskip
\item Notice that there is only ever one value in each \texttt{Row}'s \texttt{data}. This is a \textbf{union}, and only stores one of its field values.
\item See how the enumeration values indicate the field the data has been stored in. This is why the enumeration's constants were named in a similar\footnote{They could have been named anything, but this reflects their purpose well.} way to the union's fields.
\end{itemize}
}

% subsubsection this_process_is_repeated_for_each_element_of_the_array (end)

% subsection understanding_read row (end)
\clearpage
\subsection{Understanding \texttt{Print Row}} % (fold)
\label{sub:understanding_print row}

\texttt{Print Row} is the other key piece of logic in the Small DB program. This procedure outputs the values read from the user to the Terminal. It uses the data stored in the \texttt{Row}'s fields to determine how this value is output, and how the data can be read. The flowchart of this logic is shown in \fref{fig:print-row-flow-understanding}.

\begin{figure}[htbp]
   \centering
   \includegraphics[width=\textwidth]{./topics/type-decl/diagrams/PrintRowFlow}
   \caption{Flowchart of the steps needed to print a \texttt{Row} to the Terminal, from \fref{fig:print-row-flow}}
   \label{fig:print-row-flow-understanding}
\end{figure}

Within \texttt{Main}, \texttt{Print Row} is called once for each \texttt{Row} in the \texttt{db\_data} array. The following illustrations demonstrate the third and final call to \texttt{Print Row}.

The illustrations will show the following:
\begin{enumerate}
  \item \nameref{ssub:print_row_is_called_for_each_element_in_the_array}
  \item \nameref{ssub:the_text_value_is_output_to_the_terminal}
\end{enumerate}

You can use these details to determine how the other data was written to the Terminal.

\clearpage
\subsubsection{\texttt{Print Row} is called for each element in the array} % (fold)
\label{ssub:print_row_is_called_for_each_element_in_the_array}

This illustration starts part way through the third call to \texttt{Print Row}. At this stage the first two rows have been output to the Terminal, as has the \texttt{id} of the third \texttt{Row}.

\begin{figure}[htbp]
   \centering
   \includegraphics[width=0.92\textwidth]{./topics/type-decl/images/PrintRow1}
 \caption{\texttt{Print Row} is called for each of the \texttt{Row} elements in \texttt{db\_data}}
 \label{fig:print-row-vis-1}
\end{figure}

\mynote{
\begin{itemize}
\item In \fref{fig:print-row-vis-1} the indicated areas show the following:
\begin{enumerate}
  \item This is the third call to \texttt{Print Row}. Each time this procedure is called it receives a copy of the data from the element passed to it.
  \item The parameter is a copy of the data from the array element.
  \item The first action in \texttt{Print Row} is to output the \texttt{id} value from the \texttt{to print} \texttt{Row}.
  \item The illustration is showing the computer at the stage where it is reading \texttt{to print}'s \texttt{kind} to determine which path to take. As \texttt{to print}'s \texttt{kind} is currently \texttt{TXT\_VAL} it will take the path at Step 8.
\end{enumerate}
\medskip
\item The case statement will allow the code to output the message that matches the kind of data stored in \texttt{to print}.
\end{itemize}
}

% subsubsection print_row_is_called_for_each_element_in_the_array (end)

\clearpage
\subsubsection{The text value is output to the Terminal} % (fold)
\label{ssub:the_text_value_is_output_to_the_terminal}

\begin{figure}[htbp]
   \centering
   \includegraphics[width=0.91\textwidth]{./topics/type-decl/images/PrintRow2}
 \caption{\texttt{Print Row} is called for each of the \texttt{Row} elements in \texttt{db\_data}}
 \label{fig:print-row-vis-2}
\end{figure}

\mynote{
\begin{itemize}
\item In \fref{fig:print-row-vis-2} the indicated areas show the following:
\begin{enumerate}
  \item Step 8 reads the \texttt{txt\_val} field of \texttt{to print}'s \texttt{data} field. This reads the text value from within that field, and this is output to the Terminal.
  \item The \texttt{kind} field was used to determine which field to read from the union.
\end{enumerate}
\smallskip
\item In this example the record, union, and enumeration are all working together to enable the required functionality.
\item Without the enumeration it would not be possible to determine which field to read from the union. Reading any of the fields on the union would return data, but only the field that was written to can be relied about to have a meaningful value.
\item Without the union you would need to waste space storing all three data values, but only ever using one.
\item Without the record it would be hard to relate the values stored within a single \texttt{Row}. The row only existed because of the record's declaration.
\item Back in \texttt{Main}, the array allows you to store multiple of these values.
\end{itemize}
}

% subsubsection the_text_value_is_output_to_the_terminal (end)

% subsection understanding_print row (end)