\chapter{Signal to noise ratio of filtered photodetection signals}
\label{ch:snr}

When a silicon photomultiplier (SiPM) detects a photon, the electrical output
is a sudden current spike, with a rise time on the order of nanoseconds, which
decays slowly in some microseconds.

This signal is immersed in electrical noise, which is proportional to the
square root of the area of the SiPM, since all photodiode cells are connected
in parallel. Given the relatively large area of the SiPMs Tiles, \SI{25}{cm^2},
the amplitude of the noise can be comparable to the amplitude of the signals.

This implies that to read out the detector it is important to filter the
waveform to suppress the noise. In this chapter we will study the performance
of some common filters on a PDM output.

\section{Data}
\label{sec:snrdata}

\begin{figure}[p]

    \widecenter{\includempl{fighist2dtile57-0}}

    \widecenter{\includempl{fighist2dtile57-1}}

    \figcaption{hist2dtile57}{Time-value histogram of Tile~57 in LNGS data.
    Top panel: whole events. Bottom panel: zoomed on the laser pulses and
    synchronized with the laser trigger.}

\end{figure}

\begin{figure}[p]

    \widecenter{\includempl{figsignals}}

    \figcaption{signals}{Single laser pulses.}

\end{figure}

We use the LNGS data for Tile~57, see \autoref{sec:lngsdata}. The data file is
\nolinkurl{MB2-LF-3x/NUV-LF_3x_57/nuvhd_lf_3x_tile57_77K_64V_6VoV_1.wav}. The
Tile is operated at \SI{6.0}{V} overvoltage, to be compared to \SI{6.5}{V} in
Proto0, see \autoref{fig:pdmadcch}. The amplitude of the signals is
proportional to the overvoltage, while the noise should not depend on it, so
this difference has to be taken into account when comparing the performances
in the Proto0 and LNGS setups.

In \autoref{fig:hist2dtile57} we show the persistence plot of the waveforms,
both on the whole acquisition window and zoomed on the pulses produced by the
laser. The persistence plot is a two-dimensional histogram of the events, where
the variables are the sample number in the event and the ADC value. It is like
stacking on each other all the waveforms. We have still not explained in full
detail how a SiPM works, for now we will do with what we can see with our own
eyes in the data. We defer an in-depth explanation to \autoref{sec:analtheory}.

From the whole event visualization it is evident that there are other pulses
occurring at random times beside those produced by the laser. We will need to
measure the height of pulses, and to do that we need a reference value, i.e., a
baseline. The straightforward way to compute the baseline is taking the average
of the pre-trigger region before the laser pulse, but this average would be
biased by the presence of a pulse. So we filter away all events where there is
a pre-trigger sample below~700, choosing this threshold by looking at
\autoref{fig:hist2dtile57}. There are~72 such events out of~\num{10005}.

Looking at the zoom on the laser pulses, we see that the amplitude of the
pulses is discrete. The laser does not emit only one photon, we expect a
Poisson distributed number of photons to hit the Tile. So we interpret the
discrete amplitudes as corresponding to the number of detected photons. We use
``PE'', standing for ``photoelectron'', as a unit of measure to indicate these
steps. So the \SI{1}{PE} pulses are the lowest ones, the first non-straight
band in the figure, then there are the \SI{2}{PE} pulses, and so on.

In \autoref{ch:anal} we will see that the PE do not correspond precisely to the
number of photons, due to the presence of other avalanche-generating
mechanisms, but for now we just care about the discretization. In
\autoref{fig:signals} we show some individual pulses with different PE.

\section{Filters}
\label{sec:filters}

A filter operates by converting the original sequence of ADC samples $(x_1,
x_2, \ldots)$ to a new ``filtered'' sequence $(y_1, y_2, \ldots)$. The filters
we consider are causal, i.e., the filtered sample $y_n$ can be computed only
using the original samples up to $x_n$. This limitation is introduced because
we are interested in using the filters online, updating the filter output
continuously as samples are read.

We tested three linear filters: the moving average, the exponential moving
average (also known as autoregressive filter), and the cross correlation
filter. These are standard filters and are the ones currently considered by the
DarkSide collaboration for the implementation of the online readout system.

We will always normalize the filters to behave like an average, in the sense
that a constant input waveform is mapped to itself. This is a convenient choice
because it allows to subtract the baseline value after filtering.

All filters have a temporal scale parameter, which we will tune in the analysis.

\subsection{Moving average}

The moving average consists in taking the average of the last $N$ samples:
%
\begin{equation}
    y_n = \frac1N \sum_{i=1}^N x_{n-N+i}.
\end{equation}

It is efficient to compute because at each step the average can be updated by
subtracting the $N$-th past sample and adding the incoming one:
%
\begin{equation}
    y_{n+1} = y_n + \frac1N x_{n+1} - \frac1N x_{n-N+1}.
\end{equation}
%
The memory requirement is a buffer with the $N$ samples.

Since the SiPM output is a current signal, the moving average is computing the
charge in a given time window, so it is also called ``charge integration'' in
this context.

\subsection{Exponential moving average}

The exponential moving average weighs past samples with an exponentially
decaying coefficient, and can be written recursively as
%
\begin{equation}
    y_n = a y_{n-1} + (1 - a) x_n, \quad a \in (0, 1).
\end{equation}

The scale of the exponential decay is given by
%
\begin{align}
    \tau &= -\frac1{\log a}, \\
    &\approx \frac1{1-a} \text{ for $a$ close to 1.}
\end{align}

This is the most efficient filter possible since it processes exactly once each
sample and requires no working memory.

\subsection{Cross correlation filter}

\begin{figure}

    \widecenter{\includempl{figautocorrlngs}}

    \figcaption{autocorrlngs}{Autocorrelation of the LNGS and Proto0 noise,
    computed on the part of the events before the trigger. The autocorrelation
    at lag $t$ is the correlation between a sample and another sample occurring
    a time $t$ after the first.}

\end{figure}

The cross correlation filter consists in computing the cross correlation of
the input with the expected signal shape. In some texts this is called matched
filter, term which we reserve for a different case explained later.

Let $\mathbf h = (h_1, h_2, \ldots, h_N)$ be a \emph{template} of the signal
waveform we want to detect. This means $\mathbf h$ should ideally match the
shape of the signal waveform we want to find in the noisy data. The filter is
then the cross correlation of $\mathbf x$ with $\mathbf h$:
%
\begin{equation}
    y_n = \sum_{i=1}^N h_i x_{n-N+i}, \quad \sum_{i=1}^N h_i = 1.
\end{equation}

Under the assumption that the data is white noise plus a signal that perfectly
matches the template apart from amplitude, this filter is optimal in the sense
that in the filter output there will be a peak corresponding to the signal and
this peak will have the maximum possible height relative to the standard
deviation of the filtered noise.

The differences we have from the ideal case are:
%
\begin{enumerate}
    \item the shape of the signal probably changes a bit each time;
    \item the signal is not aligned always in the same way to the ADC clock;
    \item most importantly, the noise is not white.
\end{enumerate}

We defer to \autoref{sec:cctemplate} the calculation of the template $\mathbf
h$. See \autoref{fig:filters} for an example of filtered waveform.

\begin{figure}

    \widecenter{\includempl{figfilters}}

    \figcaption{filters}{A filtered event. The number of averaged samples in
    the moving average filter, the scale of the autoregressive filter, and the
    length of the template of the cross correlation filter are all 1024
    samples, at \SI1{GSa/s}.}

\end{figure}

\subsection{Matched filter}

With the cross correlation filter, a non-white noise spectrum can be kept into
account by appropriately transforming $\mathbf h$, in this case the filter is
called \emph{matched filter} \cite[161]{ferrante2015}. Let $\mathbf s$ and
$\mathbf w$ be the signal and noise, such that the waveform to be filtered is
$\mathbf x = \mathbf s + \mathbf w$. Let $V$ be the noise covariance matrix,
i.e., $V_{ij} = \operatorname{Cov}[w_i, w_j]$. Then the template for the
matched filter is
%
\begin{equation}
    \mathbf h_\text m = V^{-1} \mathbf h.
    \label{eq:matched}
\end{equation}

This filter is in some sense equivalent to doing a linear least squares fit of
the data with one parameter which multiplies the model $\mathbf h$. This can be
seen by writing the cross correlation with the template, $\mathbf h^\top V^{-1}
\mathbf x$, which compared to the least squares estimator \cite[628]{zyla2020}
is missing the multiplicative factor $1/(\mathbf h^\top V^{-1} \mathbf h)$.

We tried implementing the matched filter with mixed results, and we will not
report these studies in detail. We computed the noise covariance matrix on the
event samples before the signal. Since the noise is stationary, $V$ is a
Toeplitz matrix, i.e., the covariance depends only on the lag between two
samples. In \autoref{fig:autocorrlngs} we show the correlation of LNGS and
Proto0 noise.

The matched filter worked slightly better than the cross correlation filter, as
expected, but only for filter length $N$ sufficiently small, getting worse as
$N$ increased. This could be due to numerical accuracy problems in solving the
linear system $V$ in \autoref{eq:matched} as the size of the matrix increases.

\stracka{As we will see, the study on the simpler filters suggests possible
strategies to mitigate the numerical accuracy problems. \upshape Eh?}

The template obtained from the transformation is much different than the
initial signal shape. There are large cancellations, so it is more sensitive to
numerical accuracy, and may not be appropriate for an implementation on FPGA.
Moreover, it has not been considered by the rest of the collaboration, so we
did not investigate it further, and limited ourselves to the characterization
of the proposed filters.

\section{The fingerplot}
\label{sec:fingerplot}

To measure the performance of filters, we define a signal to noise ratio (SNR)
variable as follows: the SNR is the ratio of the value that, on average, the
peak of the filtered waveform yields on \SI1{PE} pulses (without noise), over
the standard deviation of the filtered noise.

We could consider other similar measures, for example we could include the
standard deviation of the peak filter output value since that influences where
we should place a threshold to discriminate signals, however it is sufficient
to use any reasonable definition for the purpose of comparing different
filters. Occasionally we computed the SNR without strictly adhering to the
definition above, and this will be signaled in the text where appropriate.

For each event we compute the filter output at a fixed temporal delay from the
leading edge of the laser trigger pulse, see \autoref{fig:filtersample}.
Then we compute the average of the samples before the trigger pulse and take
that value as ``baseline''. We subtract the baseline from the filter output,
and finally we change the sign to obtain positive values since the signals are
negative.

\begin{figure}

    \widecenter{\includempl{figfiltersample}}

    \figcaption{filtersample}{Example of how a signal amplitude is computed in
    an event for the purpose of computing the SNR. The samples in the shaded
    region are averaged to compute the baseline. The filter is evaluated at a
    fixed offset, indicated by the arrow, from the leading trigger edge. The
    amplitude then is the difference between the baseline and the filter value.
    The shaded region ends 100 samples before the trigger.}

\end{figure}

We take the list of these baseline and sign corrected filter outputs and
compute an histogram. One of these histograms is shown in
\autoref{fig:fingerplot}. It is called ``fingerplot'' due to the descending
peaks reminding of fingers.

\begin{figure}

    \widecenter{\includempl{figfingerplot}}

    \figcaption{fingerplot}{The fingerplot for the moving average filter with
    128 samples evaluated 128 samples after the trigger.}

\end{figure}

The first peak is centered in zero and thus corresponds to cases where no laser
photon is detected. Since the instant where we are evaluating the filter output
in each event is independent of the output itself, instead of, e.g., being
determined by peak finding, this peak is the distribution of the noise after
passing through the filter.

The various other peaks correspond to an increasing number of PE. The second
peak is the distribution of the filter output at a fixed instant for \SI1{PE}
signals. If the instant where we evaluate the filter is the one associated to
the highest signal response, the mean of the second peak divided by the
standard deviation of the first yields the SNR.

Since the peaks are often overlapping, and that we will have to repeat the
calculations for many fingerplots automatically without checking each one, we
use robust measures of location and scale instead of the average and standard
deviation. We run a peak finding algorithm on the fingerplot and divide the
data by putting boundaries halfway between peaks, so that we assign each
datapoint to a peak. For the second peak, we take the median instead of the
average. For the first peak, we take half of the symmetric \SI{68}\%
interquantile range instead of the standard deviation, i.e., half the
difference between the 0.84 and 0.16 quantiles.

On a Gaussian distribution these are equivalent to the mean and standard
deviation, however they are less sensible to the tails of the distribution,
which is a desirable property since the boundaries could cut away the tails and
there could be contamination from the tails of the neighboring peaks. By
definition, when the SNR is high, the peaks are well separated, so in that case
the result can be trusted to match precisely the quantiles of the single
isolated peak.

Moreover, the code always automatically checks that the median of the \SI0{PE}
peak is zero within its error. If it is not, the SNR is considered to be zero,
such that it is well visible in the results.

\section{Cross correlation filter template}
\label{sec:cctemplate}

We compute the template for the cross correlation filter by taking the average
of \SI1{PE} signals, selected using the fingerplot boundaries obtained with a
1500-samples average, counting the samples starting from the trigger position.
The baseline is computed with the median. We consider three alignment methods
for the waveforms prior to averaging:
%
\begin{itemize}

    \item unaligned, or rather aligned to the acquisition window;

    \item aligned using the laser trigger;

    \item aligned using a cross correlation filter done with the unaligned
    template, where the alignment offset is the position of the minimum in the
    filter output.

\end{itemize}

We show the obtained templates in \autoref{fig:templates}. The trigger square
pulse is sampled at \SI1{GSa/s} like the PDM output, so we assume that it gives
a precise alignment. Indeed, the pulse peak appears sharper than with the
unaligned template. The filter-aligned template has an additional sinc-like
oscillation centered on the peak. It is similar to the autocorrelation of the
noise, see \autoref{fig:autocorrlngs}, so we interpret it as the effect of a
non-null coherence in the averaged noise, due to the fact that the position of
the filter peak minimum will depend both on the position of the signal and on
the overlapping noise oscillations, which have a width similar to the one of
the signal peak.

\begin{figure}

    \widecenter{\includempl{figtemplate}}

    \figcaption{template}{Templates obtained averaging \SI1{PE} laser pulses,
    selected with a \SI{1.5}{\micro s} charge integration fingerplot, with
    different pre-averaging alignment choices. The template we pick is the
    trigger-aligned one.}

\end{figure}

The trigger-aligned template appears to be the best one, so we will use it in
this chapter and in \autoref{ch:timeres}. However \autoref{ch:anal} analyzes
data without trigger information, forcing us to use the filter-aligned template
in that case. See \cite{coakley2001} for an alignment method that may offer
some improvements over our vanilla approach.

When using the template, we normalize it to unit sum, as explained in
\autoref{sec:filters}. Since we will try various lengths of the filter
template, we have to decide how to truncate the full template. When truncating
to $N$ samples, we pick $N$ contiguous samples from the template such that
their sum of squares is maximized. Of course normalizing the template to unit
sum is done after truncation.

We did not smooth the template nor fitted it with a model. For how to fit this
kind of pulses, we refer to, e.g., \cite{luzzi2020}. We think that our vanilla
average is fine enough, but we did not actually compare it with a fitted
template.

We conclude this section with a technical detail: when computing the
filter-aligned template, we first smooth the unaligned template with a 2-sample
moving average. This is to remove a coherent noise at the Nyquist frequency,
i.e., even samples are always biased in the same direction relative to odd
samples, in all waveforms. If we do not correct this, the noise appears in the
template, then when using it the filter matches the noise in template to the
one in the waveform, and gives a peak output only on even or odd samples.

\section{SNR versus filter length}

When determining how to compute the SNR from the fingerplot we assumed that we
were evaluating the filter output at the optimal instant. Since we do not know
it a priori, we repeat the computation for a range of values of the filter
evaluation point. A simpler solution that comes to mind is taking the minimum
(the signals are negative) of the filter output in each event, however this
would bias upward the SNR measure because the minimum can yield lower values
due to noise peaks. Instead, by fixing the point independently from the data,
the random oscillation due to noise is symmetrical and the averaging recovers
the actual amplitude of the filter output for the signal.

The resulting SNR curves are shown in \autoref{fig:snrplot}, repeated for a
range of values of the filter-length parameter. For the moving average and
cross correlation filter, the length parameter is the number of samples, $N$.
For the exponential moving average, it is the scale of the exponential decay
$\tau$.

\stracka{Specifica la baseline usata per questo plot. \upshape Risposta:
viene dal fingerplot, spiegato nel suo capitolo.}

\begin{figure}

    \widecenter{\includempl{figsnrplot}}

    \figcaption{snrplot}{The SNR as a function of delay from trigger (x-axis)
    and filter length (shade of gray) for the three filters.}

\end{figure}

The maximum of each curve gives the actual SNR figure we are interested in. We
expect by intuition that the width of the maximum is approximately proportional
to the temporal resolution we could achieve if we used the filter output to
locate temporally the signal. So we do another plot, \autoref{fig:changebs}
left column, where we show the maximum SNR value and the width of the peak
versus the filter length parameter. We measure the width as the distance
between the two points where the SNR is 1 less than its maximum value.

\section{Effect of the baseline computation}

In constructing the fingerplot, we subtract from the filter output value the
baseline, i.e., the average value of the waveform in absence of signals. We
compute the baseline for each event as the average of the samples before the
signal. As a default, we average \num{8000} samples. The value obtained varies
randomly due to the noise; its standard deviation is that of the noise divided
by $\sqrt{8000}$. The maximum filter length we use is 2000, so in that case the
width of the peaks gets an additional contribution (summed in quadrature) of
roughly $\sqrt{2000/8000} = 1/2$ of the width we would have with a noiseless
baseline, i.e., the width of the first peak is $\sqrt{1 + 1/2} - 1 = \SI{22}\%$
larger, lowering the SNR by the same percentage. We note that, since the noise
is autocorrelated, the standard deviation of the mean does not scale with
$1/\sqrt N$, giving a more detrimental effect on the SNR than the one just
estimated, and the width of the \SI0{PE} peak depends on the filter.

These considerations are relevant to real-world applications, because the
baseline computation is a relevant part of the signal finding, and it requires
either a fast on-digitizer algorithm estimating it reliably or sending enough
samples to subsequent stages in the data processing chain. The value of the
baseline is expected to change in the detectors, so it may not be feasible to
compute it once with a very long average and then keep it fixed for a while.

To get an idea of the effect of the precision of the baseline calculation, we
repeat everything computing the baseline with just 1000 or 200 samples. The
result is in the central and rightmost columns of \autoref{fig:changebs}. We
see that, for example, going from 8000 to 200 baseline samples the maximum SNR
drops from 19 to~9.

It may appear strange that, with an imprecise baseline, the SNR starts dropping
with increasing filter length at somewhat short lengths. We note that the
filters behave like an average, so their peak value on the signal decreases as
they get longer. The filtered noise contribution decreases too, but the
baseline error is constant.

\begin{figure}[t]

    \widecenter{\includempl{figchangebs}}

    \figcaption{changebs}{Top row: maximum SNR achieved with each filter
    (different curves) and filter length (x-axis). Bottom row: the width of the
    maxima, computed as the length of the interval of offset from trigger with
    endpoints where the SNR is 1 less than the maximum. The cross correlation
    filter maintains almost constant width with increasing SNR, while the other
    two filters have increasing width. In each column the computation is
    repeated changing the number of pre-signal samples used to compute the
    baseline. The performance decreases with a lower number of samples because
    the error on the baseline increases and the baseline is compared to the
    filtered signal amplitude to discriminate signals from noise.}

\end{figure}

\section{Noise spectrum}
\label{sec:spectrum}

To support our previous statement that the noise is not white, in this section
we report the frequency spectrum of the noise.

We take the region of each event before the trigger pulse, compute its
periodogram, and take the median across events for each frequency. We use the
median instead of the average in case there were spurious signals or
irregularities we missed. We do the same for the same tile (57) in Proto0 data,
with no need to isolate the pre-trigger region in that case. The spectra thus
obtained are shown in \autoref{fig:spectra}. We also compute the spectrum of
the signal template and compare it to the noise in \autoref{fig:templsp}. Note
that we do not window the signal because it is not a stationary process.

\begin{figure}[t]

    \widecenter{\includempl{figspectra}}

    \figcaption{spectra}{Noise spectrum of Tile~57 in LNGS and Proto0. The
    vertical scale has arbitrary units.}

\end{figure}

\begin{figure}[b]

    \widecenter{\includempl{figtemplsp}}

    \figcaption{templsp}{Comparison of the spectra of signal and noise. The red
    curve shows the percentage of signal power below a given frequency. The
    signal spectrum is computed with the discrete Fourier transform of the
    cross correlation filter template (\autoref{fig:template}) without
    windowing.}

\end{figure}


\newcommand{\TeamNo}{31}

\newcommand{\HWno}{02}

\newcommand{\AuthorOneName}{Merve Nur Öztürk}
\newcommand{\AuthorOneID}{2311322}

\newcommand{\AuthorTwoName}{Atakan Süslü}
\newcommand{\AuthorTwoID}{2311371}

\newcommand{\AuthorThreeName}{Betül Rana Kuran}
\newcommand{\AuthorThreeID}{2311173}

\documentclass[letterpaper,12pt]{article}
\usepackage{tabularx} % extra features for tabular environment
\usepackage{amsmath}  % improve math presentation
\usepackage{amssymb}
\usepackage{xcolor}
\usepackage{float}
\usepackage[export]{adjustbox}
\usepackage{graphicx} % takes care of graphic including machinery
\usepackage[margin=1in,letterpaper]{geometry} % decreases margins
\usepackage{cite} % takes care of citations

\begin{document}
\begin{center}
AE 305, 2020-21 Fall \hfill \textbf{HW \HWno} \hfill \textbf{Team \TeamNo} \\
\noindent\rule{\textwidth}{0.4pt}
\begin{tabular}{p{0.33\textwidth} | p{0.33\textwidth} | p{0.33\textwidth} }
	\AuthorOneName&\AuthorTwoName&\AuthorThreeName\\
	\textit{\AuthorOneID}&\textit{\AuthorTwoID}&\textit{\AuthorThreeID}
\end{tabular}
\noindent\rule{\textwidth}{0.4pt}
\end{center}

%Report start

\section{Introduction}

Numerical methods, such as Runge Kutta Method, are widely used to solve differential equations when an
analytical solution is not possible or time-consuming. $4^{th}$ order Runge-Kutta Method is one of the
numerical methods which gives quite accurate results due to its small truncation error. In this homework,
a solid propellant rocket is given, and its chamber pressure $p_c$, burn rate $\dot{r}$, and the specific
impulse $I_{sp}$ are asked to be calculated with this method until the chamber pressure $p_c$ becomes equal
to the atmospheric pressure $p_a$, with different time step sizes and different nozzle throat radiuses.

This equation gives the rate of the chamber pressure of the given rocket with a constant burn temperature
$T_c$:

\begin{equation}
	\frac{dp_c}{dt} = \frac{RT_c}{V_c}(\dot{m_{bs}} - \dot{m_{ce}} - \rho_{c}\dot{V_c})
\end{equation}

It is assumed that the propellant inner surface is circular and between the chamber exit and nozzle throat,
no gas mass accumulates ($\dot{m_{ce}} = \dot{m_n}$). Moreover, writing $\dot{m_{bs}} = \rho_{p}2\pi rL\dot{r}$,
and $V_c = \pi r^{2}L$, the equation becomes:

\begin{equation}
	\frac{dp_c}{dt} = RT_c[\frac{2\dot{r}}{r}(\rho_{p} - \rho_{c}) - \frac{\dot{m_n}}{\pi r^{2}L}]
\end{equation}

It is given that $\dot{r} = ap_c^{n}$, and $\dot{m_n} = p_cA^{*}\sqrt{\frac{\gamma}{RT_c}}(\frac{\gamma +1}{2})^{-\frac{\gamma +1}{2(\gamma -1)}}$
where $A^{*}$ is the nozzle throat area. Furthermore, though at first it was assumed that the propellant
inner surface is circular, it is more of a complex shape. Thus, a propellant design dependent factor
$f_{cor}(r)$ is used to simply approach to original characteristics. Then, the equations for the rate
of chamber pressure, and the rate of equivalent radius r become:

\begin{equation}
	\boxed{\frac{dp_c}{dt} = RT_c[f_{cor}(r)\frac{2ap_c^{n}}{r}(\rho_{p} - \frac{p_c}{RT_c}) - \frac{p_cA^{*}}{\pi r^{2}L}\sqrt{\frac{\gamma}{RT_c}}(\frac{\gamma +1}{2})^{-\frac{\gamma +1}{2(\gamma -1)}}]}
\end{equation}

\begin{equation}
	\boxed{\frac{dr}{dt} = ap_c^{n}}
	\label{eqn:pres}
\end{equation}

Specific impulse is also asked to be calculated, and its formula is:

\begin{equation}
	\boxed{I_{sp} = \frac{1}{g}\sqrt{\frac{2\gamma RT_c}{\gamma -1}[1-(\frac{p_a}{p_c})^{\frac{\gamma -1}{\gamma}}]}}
	\label{eq:isp}
\end{equation}

\section{Method}
\subsection{Classical Fourth-order RK Method}
The basic form of the RK method is :
\begin{equation}
	y_{i+1} = y_i + \Phi(x_i, y_i, \Delta x) \Delta x
	\label{eqn:rk}
\end{equation}
$\Phi$ can be defined as a weighted slope function or increment function. It can be expressed as:
\begin{equation}
	\Phi = a_1k_1 + a_2k_2 + ... + a_nk_n
\end{equation}
where $k_1, k_2, ..., k_n$ are slopes, $a_1, a_2, ..., a_n$ are weights.\\
In classical fourth-order RK method, four slopes are calculated for each interval. Also, weights are equal to $1/6, 1/3, 1/3, 1/6$
, respectively. In the homework, there is the system of ordinary differential equations, $\frac{dy_i}{dt}$ and $\frac{dr_i}{dt}$.
Therefore, the RK method becomes:
\begin{eqnarray}
	y_{1,i+1}&=&y_{1,i} + \frac{1}{6}(k_{1,1} + 2k_{2,1} + 2k_{3,1} + k_{4,1})\Delta x \nonumber \\
	y_{2,i+1}&=&y_{2,i} + \frac{1}{6}(k_{1,2} + 2k_{2,2} + 2k_{3,2} + k_{4,2})\Delta x \nonumber \\
	\mbox{Stage 1: }k_{1,1}&=&f_1(x_i, y_{1,i}, y_{2,i} ) \nonumber \\
	k_{1,2}&=&f_2(x_i, y_{1,i}, y_{2,i}) \nonumber \\
	y^{*}_1 &=& y_{1,i} + k_{1,1}(p_1\Delta x) \nonumber \\
	y^{*}_2 &=& y_{2,i} + k_{1,2}(p_1\Delta x) \nonumber \\
	\mbox{Stage 2: }k_{2,1}&=&f_1(x_{i+(p_1\Delta x)} ,y^{*}_1, y^{*}_2 ) \nonumber \\
	k_{2,2}&=&f_2(x_{i+(p_1\Delta x)} ,y^{*}_1, y^{*}_2 ) \nonumber \\
	y^{**}_1&=&y_{1,i} + k_{2,1}(p_2\Delta x) \nonumber \\
	y^{**}_2&=&y_{2,i} + k_{2,2}(p_2\Delta x) \nonumber \\
	\mbox{Stage 3: }k_{3,1}&=&f_1(x_{i+(p_2\Delta x)} ,y^{**}_1, y^{**}_2 ) \nonumber \\
	k_{3,2}&=&f_2(x_{i+(p_2\Delta x)} ,y^{**}_1, y^{**}_2 ) \nonumber \\
	y^{***}_1&=&y_{1,i} + k_{1,3}(p_3\Delta x) \nonumber \\
	y^{***}_2&=&y_{2,i} + k_{2,3}(p_3\Delta x) \nonumber \\
	\mbox{Stage 4: }k_{4,1}&=&f_1(x_{i+(p_1\Delta x)} ,y^{***}_1, y^{***}_2 ) \nonumber \\
	k_{4,2}&=&f_2(x_{i+(p_3\Delta x)} ,y^{***}_1, y^{***}_2 ) \nonumber
\end{eqnarray}
where $y\prime = f(x_i,y_i)$ and $p_1=p_2=1/2$, $p_3=1$.\\
At every stage, first, the slope of $r_i$ ($k_{n,1}$) and  $p_{c_{i}}$ ($k_{n,2}$) at $p_n$ fraction of interval is calculated.
Then, temporary $p_{c_{i+(p_n\Delta x)}}$ and $r_{i+(p_n\Delta x)}$ values are determined, which enables to calculate the next
slopes $p_{n+1}$ fraction of the same interval. After four stages, weighted slopes are determined. By substituting known
variables into Equation \ref{eqn:rk}, next values $r_{i+1}$ and $p_{c_{i+1}}$ are calculated.
\subsection{Adaptive Stepping}
\label{section:adaptive}

When there is a sudden change in a region, a small step size must be used to calculate this sudden change.
But if this sudden change is in a small region and ODE changes linearly in other regions, a small time step size
is not necessary for the linear region. If a constant small time step size is used, sudden changes can be calculated correctly but
computational speed will be wasted for the linear region.
Also, small step sizes cause an increase in the number of steps, which leads to a
rise in the round off errors, whereas large step sizes cause larger truncation errors.
In order to solve this problem, adaptive step size
can be used such that step size is bigger in linear regions, and if there is a sudden change, step size gets
smaller so that results can be calculated correctly.
In adaptive step size control, first, the local truncation error, $E_o$, is calculated. In this
homework, $E_o$ is determined by using the difference between the RK4 method and the RK2 method.

\begin{equation}
	E_o = \frac{y_{RK4}-y_{RK2}}{y_{RK4}}
\end{equation}

After the local truncation error $E_o$ is determined,
the new time step can be calculated with

\begin{equation}
	\Delta x_{new} = \Delta x_{old} \hspace{0.5em} \left| \frac{E_{allowed}}{E_o}\right|^{0.20}
\end{equation}

where $E_{allowed}$ is the allowed percent error tolerance and equal to 0.0001 in this homework.

\newpage

\section{Results and Discussion}
\subsection{Calculation of $p_c$, $\dot{r}$ and $I_{sp}$ with different time step sizes}

\begin{figure} [ht]
	\centering
	\includegraphics[height = 8.5cm]{graphs/q1_pc.eps}
	\caption{Calculated $p_c$ values using $4^{th}$ order RK method with three different step sizes.}
     \label{fig:q1pc}
\end{figure}

This figure demonstrates how the chamber pressure $p_c$ changes as time progresses. As can be seen in
Figure \ref{fig:q1pc}, there are three curves, one for each step size. However,
there is no significant difference between the curves, which results from the high degree of accuracy
provided by the $4^{th}$ order RK method. This is because the truncation error for this method is so
small that its effect on the results can be nearly neglected. In addition, using smaller step sizes
helps to calculate more precisely, and it is obvious that the step sizes used for this problem are small
enough. As a result, the values obtained for the chamber pressure $p_c$ of the rocket were approximately
the same for three different step sizes.

\vspace{1em}
Figure \ref{fig:q1rdot} and Figure \ref{fig:q1Isp} demonstrates the change in the rate of equivalent
radius and the specific impulse with time, respectively.
As it was for $p_c$, there are three curves also for the rate of the equivalent radius
$\dot{r}$ in Figure \ref{fig:q1rdot} and the specific impulse $I_{sp}$ in Figure \ref{fig:q1Isp}.
Since the conditions are the same as mentioned above, the values calculated for $\dot{r}$ and $I_{sp}$ does not
experience a major change for different step sizes. The point is, the accuracy of the results obtained by using the RK4 Method
is remarkably of a high degree for the step sizes used in this problem.
\newpage

\begin{figure} [!h]
	\centering
	\includegraphics[height = 8.5cm]{graphs/q1_rdot.eps}
	\caption{Calculated $\dot{r}$ values using $4^{th}$ order RK method with three different step sizes.}
	\label{fig:q1rdot}
\end{figure}

\begin{figure} [!h]
	\centering
	\includegraphics[height = 8.5cm]{graphs/q1_isp.eps}
	\caption{Calculated $I_{sp}$ values using $4^{th}$ order RK method with three different step sizes.}
	\label{fig:q1Isp}
\end{figure}

\newpage

\subsection{Calculation of $p_c$, $\dot{r}$ and $I_{sp}$ with different nozzle throat radiuses}
\begin{figure}[!h]
	\centering
	\includegraphics[height = 8.5cm]{graphs/q2_pc.eps}
	\caption{Calculated $p_c$ values using $4^{th}$ order RK method with different nozzle throat radiuses.}
	\label{fig:q2p_cp}
\end{figure}

This figure demonstrates how the chamber pressure $p_c$ changes as time progresses with different nozzle throat radiuses.
As can be concluded from Figure \ref{fig:q2p_cp}, decreasing the radius of the nozzle throat leads to an increase in maximum chamber pressure
and reduces the time needed for the chamber pressure to reach the ambient pressure.
The reason of this is reducing the radius causes an exponential decrease in the area of the nozzle.
As the area of the nozzle decreases, mass flowing out from the nozzle decreases, and total gas mass inside the cavity increases.
This results in increased pressure inside the chamber. As the pressure increases, the propellant burns faster.
Therefore, with a lower nozzle throat radius, propellant depletes faster resulting in shorter flight time.

\newpage
\begin{figure}[!h]
	\centering
	\includegraphics[height = 8.5cm]{graphs/q2_rdot.eps}
	\caption{Calculated $\dot{r}$ values using $4^{th}$ order RK method with different nozzle throat radiuses.}
	\label{fig:q2r_dot}
\end{figure}

This figure illustrates how the burn rate $\dot{r}$ changes as time progresses with different nozzle throat radiuses.
$\dot{r}$ is proportional to $p_c$ as stated in Equation \ref{eqn:pres}; therefore, both of them give
similar responses to different throat radiuses.
Lower radiuses cause higher maximum burn rates but shorter time to reach the ambient pressure.
However, since the propellant burn rate exponent in
Equation \ref{eqn:pres} is smaller than one, the growth in the maximum burn rates in Figure \ref{fig:q2r_dot} is
less remarkable than the growth in the maximum chamber pressures in Figure \ref{fig:q2p_cp}. Also, as mentioned above,
flight time decreases because of a faster burn rate.

\newpage

\begin{figure}[!h]
	\centering
	\includegraphics[height = 8.5cm]{graphs/q2_isp.eps}
	\caption{Calculated $I_{sp}$ values using $4^{th}$ order RK method with different nozzle throat radiuses.}
	\label{fig:q2I_sp}
\end{figure}

This figure shows how the specific impulse $I_{sp}$ changes as time progresses with different nozzle throat radiuses.
Specific impulse is a function of chamber pressure $p_c$ as stated in Equation \ref{eq:isp}.
Therefore, it is not surprising to observe that the specific impulse gives similar reactions in Figure \ref{fig:q2I_sp}
as the chamber pressure gave in Figure \ref{fig:q2p_cp} in terms of relationship with increasing nozzle throat radius.
As the radius of the nozzle throat decreases, specific impulse also increases but flight time decreases. Thus, the
lowest nozzle throat radius of 2 centimeters is the most efficient, as it has the highest specific impulse.

\newpage
\subsection{Calculation of $p_c$, $\dot{r}$ and $I_{sp}$ with adaptive stepping}

\begin{figure}[!h]
	\centering
	\includegraphics[height = 8.5 cm]{graphs/bonus_pc.eps}
	\caption{Chamber pressure versus time graph with constant and adaptive step size.}
	\label{fig:bonus_cp}
\end{figure}

Figure \ref{fig:bonus_cp}, Figure \ref{fig:bonus_rdot}, and Figure \ref{fig:bonus_isp} illustrate how the chamber pressure,
the burn rate and specific impulse change as time progresses, respectively, using constant and adaptive time step sizes.
It can be seen from Figures \ref{fig:bonus_cp}, \ref{fig:bonus_rdot}, and \ref{fig:bonus_isp} that when adaptive stepping is
applied with allowed percent tolerance $E_o = 0.0001$, linear parts can be calculated with fewer steps without losing any accuracy as discussed in
Section \ref{section:adaptive}. When there is a sudden change, step size gets smaller accordingly and results can be calculated correctly in that region.

It can be seen from Table \ref{tbl:timeint} that using adaptive stepping reduces the total number of time intervals needed significantly.
Although results are calculated three times for each time interval(twice for determining the step size and once for calculating result)
using the adaptive stepping approach, the number of time intervals needed is much lower than the constant step size approach. Therefore adaptive stepping is
computationally cheaper without losing accuracy.

\begin{table}[!h]
	\begin{center}
	\caption{Total number of time intervals needed for computations.}
	\vspace{1em}
	\label{tbl:timeint}
	\begin{tabular}{|c|c|}
	\hline
	\multicolumn{1}{|c|}{\bf{Stepping Approach}} & \multicolumn{1}{c|}{\bf{Number of time intervals needed}} \\
	\hline
	Constant $\Delta t = 0.005$ &   4470 \\ \hline
	Adaptive Stepping with initial $\Delta t = 0.005$ &   462 \\ \hline
	\end{tabular}
	\end{center}
\end{table}

\newpage

\begin{figure}[!h]
	\centering
	\includegraphics[height = 8.5 cm]{graphs/bonus_rdot.eps}
	\caption{Burn rate versus time graph with constant and adaptive step size.}
	\label{fig:bonus_rdot}
\end{figure}

\begin{figure}[!h]
	\centering
	\includegraphics[height = 8.5 cm]{graphs/bonus_isp.eps}
	\caption{Specific impulse versus time graph with constant and adaptive step size.}
	\label{fig:bonus_isp}
\end{figure}

\newpage

\section{Conclusion}

Given a solid-propellant rocket, the change of chamber pressure $p_c$,
the rate of the equivalent radius $\dot{r}$, and the specific impulse $I_{sp}$ were
requested to be calculated in the homework. These calculations were
made by using the classical $4^{th}$ order RK method with three different step sizes for the first
problem, and with different nozzle throat radiuses for the second problem.

At first, comparing the results obtained, it can be observed that the $4^{th}$ order RK method gives
quite reliable results for this problem since the step sizes used are small
enough to give minor error between solutions with different step sizes, and the truncation error
caused by numerical methods is negligible for the $4^{th}$ order RK method. Moving on to the
next comparison, the results illustrate that the outcome of the usage of smaller nozzle throat
radiuses is an increase in maximum chamber pressure. As the burn rate and the specific impulse
are related proportionally to the chamber pressure, their maximum values also experience
an increase. The smaller the nozzle throat radius the harder the gas flows out from the chamber and
the pressure inside the chamber increases, which accounts for the increase in the values
mentioned before.

As a final comment on the problem, it was demonstrated that the effect of adaptive stepping
on the number of time intervals needed is extremely remarkable. It helps to increase
the calculation speed without any loss of accuracy by using a smaller step size for
sudden changes and a bigger step size for linear regions.

\end{document}


\lab{Intro to pandas II}{Intro to pandas II}

In this lab, we explore in further detail two specific areas where pandas can be a very useful tool:
analyzing sequential data, and working with large datasets that can't be stored entirely in memory.
\begin{warn}
This lab assumes pandas 0.14.0. Errors may occur if you have an earlier version.
You can check the version of pandas you are running by the following:
\begin{lstlisting}
>>> import pandas as pd
>>> pd.__version__
'0.14.0'
\end{lstlisting}
\end{warn}
\section*{Time Series Analysis}
A \emph{time series} is a particular type of data set that consists of a sequence of measurements or observations
generated at successive points in time. Examples include the yearly average temperate of a city, or
the price of a given stock measured daily.

\begin{comment}
This section needs beefing up. Topics to cover include:
    -using the date_range function to create dateTimeIndexes
    -showing different types of string reps that can be parsed, using format string to speed up parsing
    -resampling of a time series to change frequency levels.
    -slicing time series using date strings
    -the truncate method for time series
    -load in financial data, use some of the above techniques, do a fun, simple application.
\end{comment}

\section*{Working With Large Datasets}
In the real world, a data scientist is often confronted with large datasets that can't be held in memory all at once.
There are various solutions to this problem; in this section, we will explore how pandas uses the HDF5 file format
to allow us to work with datasets on disk.

HDF5, which stands for ``Hierarchical Data Format", is a data storage system especially suited for large numerical datasets.
Rich and efficient software libraries have been developed over the years to enable fast read and write operations,
which make HDF5 a competitive option for working with large datasets in many applications. The Python library pytables
is one such library, and the HDF5 capabilities in pandas are built directly on top of pytables.

We have two primary learning goals: how to get our data into the proper HDF5 format, and how to intelligently work with
the data once it's tidied up. Let's dive in.
\subsection*{Writing HDF5 Data}
The primary way we will interact with HDF5 data goes through the \li{HDFStore} object, which behaves somewhat like a dictionary.
To begin, let's instantiate such an object, and write data to it. Make sure to execute the code snippets throughout to ensure that
everything works as expected on your machine.
\begin{lstlisting}
>>> # we will create an HDFStore with the filename test_store.h5
>>> my_store = pd.HDFStore('test_store.h5')
>>> my_store
<class 'pandas.io.pytables.HDFStore'>
File path: test_store.h5
Empty
\end{lstlisting}
The file \li{'test_store.h5'} has just been created in your working directory, although it contains nothing as yet.
Write some pandas data to the store:
\begin{lstlisting}
>>> # instantiate data, write to store
>>> ts = pd.Series(index=['A', 'B', 'C', 'D'], data = np.random.randn(4))
>>> df = pd.DataFrame(index=range(6), columns=['a','b'], data=np.random.random((6,2)))
>>> my_store['ts'] = ts
>>> my_store['df'] = df
>>> my_store
<class 'pandas.io.pytables.HDFStore'>
File path: test_store.h5
/df            frame        (shape->[6,2])
/ts            series       (shape->[4])
\end{lstlisting}
Note the dict-like syntax for writing data to the store. We now see that the store contains two objects, which
can easily be retrieved in the following manner:
\begin{lstlisting}
>>> # retrieve the df from the store, check it is the same
>>> store_df = my_store['df']
>>> (df == store_df).all()
a    True
b    True
dtype: bool
\end{lstlisting}
Removing an object that you have written to the store can be accomplished as follows (although note that removing the
object doesn't necessarily free up space on the hard disk, so the file size may not decrease):
\begin{lstlisting}
>>> # let's remove the series ts
>>> del my_store['ts'] # or my_store.remove('ts')
>>> my_store
<class 'pandas.io.pytables.HDFStore'>
File path: test_store.h5
/df            frame        (shape->[6,2])
\end{lstlisting}

The read and write operations that we have explored so far work just fine for small objects that can fit entirely in memory,
but the story is different when it comes to writing large datasets to an HDF5 file. Suppose we have a CSV file containing the
large dataset that we want to work with. One basic approach to storing this data in HDF5 format is to read and write it by
\emph{chunks}, that is, move the data into an HDF5 store a few lines at a time. This ensures that we never have to read too much
of the data into memory at once.

The \li{read_csv} function in pandas allows for reading a file by chunks of rows. We simply need to specify the keyword argument
\li{chunksize}, which gives the number of rows of the file to read in each time. Most other pandas data readers have a similar
option for loading data by chunks. It is also important to specify the correct data types of each column when reading in the chunks
of data. This will ensure that a consistent data type is used when writing to the HDF5 store. To do this, create a dict that
maps each column name to its correct data type. Columns containing strings should have the \li{object} datatype, and columns
containing numerical data may be ints or floats. Any column that contains date data should NOT be included in the dictionary,
but rather should be passed as the \li{parse_dates} keyword argument (a list of the column names containing dates).

To iteratively write data to a single object in an HDF5 store, we must use the \li{append} method. This will store the data in a
particular format called a \emph{table}, an on-disk data structure geared toward efficient querying of the rows. There are a few
parameters that we must tune in order to write the data successfully.
\begin{itemize}
\item \li{key}: This is the target object in the HDF5 store to which we want to write.
\item \li{value}: This is the actual chunk of data (such as a \li{DataFrame}) that we want to append to the store.
\item \li{data_columns}: A list of the column names of the incoming data that should be indexed. You should include all
columns that you will use in subsequent queries of the data. For example, if my dataset has a column labeled `A', and I anticipate
wanting to select all rows where the value in column `A' is greater than, say, 0, then it is imperative that \li{data_columns} includes
`A'. Try to avoid including columns that aren't needed in the queries, as performance can decrease with a larger number of indexed
columns. This argument only needs to be specified for the \emph{first} chunk to be written.
\item \li{min_itemsize}: This is an int that specifies the maximum length of a string found in the dataset. If you are unsure of the
exact length of the longest string, try setting this to a high default value (say 100). If you attempt to write data containing
a string whose length exceeds \li{min_itemsize}, an error is raised. This argument is also only required for the first chunk.
\item \li{index}: This argument is a boolean flag that indicates whether the data should be indexed as it is written. By setting
\li{index=False}, the data is written much faster.
\end{itemize}
In the code below, we create a CSV file containing toy data, then write it by chunks to our HDF5 store.
\begin{lstlisting}
>>> # first create the toy CSV file
>>> n_rows = 10000
>>> n_cols = 3
>>> csv_path = 'toy_data.csv'
>>> csv_file = open(csv_path, 'w')
>>> csv_file.write("A B C\n")
>>> for i in xrange(n_rows):
>>>     for num in np.random.randn(3):
>>>         csv_file.write(' ' + str(num))
>>>     csv_file.write('\n')
>>> csv_file.close()

>>> # now iteratively write the data to the store
>>> n_chunk = 1000
>>> col_types = {'A':np.float, 'B':np.float, 'C':np.float}
>>> reader = pd.read_csv(csv_path, sep=' ', dtype=col_types, index_col=False, chunksize=n_chunk, skipinitialspace=True)
>>> first = True # a flag for the very first chunk
>>> data_cols=['B', 'C'] # queries involving cols B and C will be allowed
>>> for chunk in reader:
>>>     if first:
>>>         my_store.append('toy_data', chunk, data_columns=data_cols, index=False)
>>>         first = False
>>>     else:
>>>         my_store.append('toy_data', chunk, index=False)
>>> my_store
<class 'pandas.io.pytables.HDFStore'>
File path: test_store.h5
/df                  frame        (shape->[6,2])
/toy_data            frame_table  (typ->appendable,nrows->10000,ncols->3,indexers->[index],dc->[B,C])
\end{lstlisting}
If an error of any type occurs when writing your data, and you need to start over, it is necessary to remove
the previously written data, since the \li{append} function doesn't overwrite existing data.

Now that the data is in the HDF5 store, we should index the table, which can speed up query operations.
This is done as follows:
\begin{lstlisting}
>>> my_store.create_table_index('toy_data', optlevel=9, kind='full')
\end{lstlisting}
Obviously the toy data in this example is small enough to fit in memory, but it illustrates a basic approach
to moving large datasets into a HDF5 store.

\begin{problem}
Write the data contained in the file \li{campaign.csv} to an HDF5 store using the chunking approach.
We recommend setting the \li{chunksize} argument to 50,000. There will be just over 100 chunks for this
chunk size, so you can track your progress by printing out a counter, if you wish. The whole writing
process will likely take a few minutes.

The file \li{campaign_format.txt} contains information about the dataset, including the column names and descriptions.
Consult this file and note which, if any, columns contain date information. Use this information to appropriately
set the \li{parse_dates} argument in the \li{read_csv} function. This argument should be a list of the column names
that include date information. Note also that some of the columns contain strings, so remember the
\li{min_itemsize} argument. Finally, you will be analyzing this data in the remainder of the lab, so glance over
the problems below to determine which columns need to be indexed.

Once you have finished writing to the store, create a table index for the data, just as shown in the example above.
\end{problem}

\subsection*{Working With On-Disk Arrays}
Now that we have the data in a HDF5 table, how do we work with it? The key here is to only read in subsets
of the data, since trying to read the entire dataset at once may result in swamping the memory and crashing
the system. The \li{select} method allows us to do just this. It takes two basic arguments: first, the name
of the table in the store that we want to query, and second, a query statement. Remember, the
query statement may not reference columns that weren't included in \li{data_columns}.
\begin{lstlisting}
>>> # this query is OK
>>> my_store.select('toy_data', where = ["'B' < 0", "'C' > 0"])

>>> # this query is NOT OK
>>> my_store.select('toy_data', where = ["'A' < .5"])
\end{lstlisting}
Note that each logical statement in the \li{where} list is combined with a logical AND.

Let's look at some simple examples with our campaign dataset. We assume that the data is in a HDF5 store called
\li{store}. Follow along to make sure you get the same results.
\begin{lstlisting}
>>> # get a list of the candidates; result should be 14 candidates
>>> cands = store.select_column('campaign', 'cand_nm').unique()

>>> # find number of contributions from UT
>>> n_UT = len(store.select('campaign',where = ["'contbr_st' == 'UT'", "columns='contbr_st'"]))
>>> n_UT
50462

>>> # find number of UT contributions to Romney
>>>  n_rom_UT = len(store.select('campaign', where = ["'contbr_st' == 'UT'", "'cand_nm'=='Romney, Mitt'", "columns='contbr_st'"]))
>>> n_rom_UT
26962

>>> # proportion of UT contributions that went to Romney
>>> np.float(n_rom_UT)/n_UT
0.534303039911
\end{lstlisting}

\begin{problem}
How many contributors in California gave to Obama? To Romney?
\end{problem}

It is also possible to execute more complicated queries involving grouping operations, although some care is required.
Consider the following question: how many contributions came from each state, and what is the average contribution
from each state? To answer this question, we need to aggregate information grouped by each state. If we could hold
the data in memory, we could use a simply \li{groupby} operation, but since our data resides on disk, the solution is
a bit more involved. Fortunately, the \li{'contbr_st'} column alone is small enough to load into memory, so we do that.
We can then utilize the \li{Series} method \li{value_counts}, which counts the number of occurrences of each distinct
value in the column. In this way, we obtain a \li{Series} indexed by the states, and containing the number of
contributions from each state.
\begin{lstlisting}
>>> # load in the state column, then calculate the counts
>>> states = store.select_column('campaign','contbr_st')
>>> states = states.value_counts()
\end{lstlisting}
Next, we want to iterate through each state, and calculate the mean contribution.
\begin{lstlisting}
>>> st_contb = []
>>> for state in states.index:
>>>     grp = store.select('campaign', where=["'contbr_st'=\"{}\"".format(state), "columns='contb_receipt_amt'"])
>>>     st_contb.append(grp['contb_receipt_amt'].mean())
\end{lstlisting}
To tidy up our results, we create a \li{DataFrame} indexed by the state, containing the number of contributions
and average contribution. Then we plot top results.
\begin{lstlisting}
>>> state_info = pd.DataFrame({'Count':states, 'Avg Contb':st_contb})
>>> state_info.sort(columns='Count', ascending=False, inplace=True)
>>> plt.subplot(211)
>>> state_info['Count'][:10].plot(kind='bar')
>>> plt.subplot(212)
>>> state_info['Avg Contb'][:10].plot(kind='bar')
>>> plt.show()
\end{lstlisting}
Results are shown in Figure \ref{pandas:states}
\begin{figure}
\centering
\includegraphics[width=\textwidth]{states.pdf}
\caption{Top 10 contributing states (top), and their average contributions (bottom).}
\label{pandas:states}
\end{figure}

\begin{problem}
Calculate the total net contributions to each candidate, and plot the results in a bar graph.
Your result should match Figure \ref{pandas:cand_contb}.
\end{problem}

\begin{figure}
\centering
\includegraphics[width=\textwidth]{cand_contb.pdf}
\caption{Total net contributions to each candidate.}
\label{pandas:cand_contb}
\end{figure}

\begin{problem}
Calculate the frequency of the 20 most common occupations of contributors in the dataset.
Also calculate the average positive contribution amount for each of these 20 occupations.
Plot the results in two bar graphs. You results should match Figure \ref{pandas:occupation}.
\end{problem}

\begin{figure}
\centering
\includegraphics[width=\textwidth]{occupation.pdf}
\caption{Top 20 occupations of contributors (above) and average contribution of each
occupation (below).}
\label{pandas:occupation}
\end{figure}

What if you are interested in the contributions to a candidate as a function of time?
Let's first aim to create a \li{DataFrame} containing the contribution amount and date for
all contributions going to Ron Paul.
\begin{lstlisting}
>>> cand = 'Paul, Ron'
>>> cand_contb = store.select('campaign', where=["'cand_nm'==\"{}\"".format(cand)])[['contb_receipt_amt', 'contb_receipt_dt']]
\end{lstlisting}
Next, we need to group by date, and apply the sum function to add up contributions for each particular date.
This can be done with the \li{groupby} function as follows:
\begin{lstlisting}
>>> contb = cand_contb.groupby(by='contb_receipt_dt').sum()
\end{lstlisting}
Plotting this time series yields Figure \ref{pandas:paul}.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{paul.pdf}
\caption{Campaign Contributions to Ron Paul over time.}
\label{pandas:paul}
\end{figure}

\begin{problem}
Plot the running total of campaign contributions as a function of time
for Mitt Romney, Barack Obama, and Newt Gingrich (all on the same graph).
Your results should match Figure \ref{pandas:cand_time}.
\end{problem}
\begin{figure}
\centering
\includegraphics[width=\textwidth]{cand_time.pdf}
\caption{Running contribution totals for three candidates.}
\label{pandas:cand_time}
\end{figure}

Each data project brings with it a new set of problems and pitfalls, but a careful application of the principles
in this lab will provide a good starting point for working with large datasets in pandas.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS630: Database Management Systems
% Copyright 2014 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 2}

Write \textbf{SQL queries} for the following:

\begin{enumerate}

\item
Find distinct ages of students who took a course with name \textit{CS310}.

\item
Find the names of students who took only 4-credits course.

\item
Find the average grade over all students for those courses which enrolled at least 10 students with age greater or equal than 25.

\item
Find the names of students who took every 4-credits course.

\item Find for each course identifier (\texttt{cid}) the \texttt{sid}(s) of the students who got the highest score.

\end{enumerate}

\subsection*{Solution}

\lstset{language=sql}
\lstinputlisting[firstline=8]{
	\topDirectory/src/sql/f01/f01q02.sql
}


\documentclass{beamer}

\input{../../shared_slides.tex}

\title{Optimization for Data Science}

\begin{document}
\maketitle
\frame{\tableofcontents}

\begin{frame}
  \frametitle{Course organization}
  \begin{itemize}
    \item Lectures (contribution counts)
    \item hands on sessions on some Thursdays
    \item a small weekly problem set
    \item Project (prices for most creative, best presentation, cleanest code, etc.)
    \item oral exam
  \end{itemize}

  Find everything on github (please contribute with pull requests: typos, etc.)

  \begin{itemize}
    \item Quick introductory round?
  \end{itemize}

\end{frame}

\section{Introduction}

\begin{frame}
  \frametitle{What is Optimization}

  \begin{center}
    \textit{Given a function $f$ which represents some cost/regret/loss (or gain/profit/utility) we aim to find the argument/decision associated with the smallest cost (or largest profit).}
  \end{center}

  \begin{equation}
    \min_{x\in C } \, f(x)
  \end{equation}

  \begin{itemize}
    \item variables, parameters, candidate solutions $x$
    \item objective function $f$ (typically real-valued)
    \item typically: technical assumptions on $f$
    \item constrained set $C \subset \R^d$
    \item convexity / differentiability
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Applications of optimization}

  \begin{itemize}
    \item Economics
          \begin{itemize}
            \item Microeconomics: Agents maximizing utility
            \item Game theory and equilibria
          \end{itemize}
    \item Statistics
          \begin{itemize}
            \item maximum likelihood
          \end{itemize}
    \item Physics
          \begin{itemize}
            \item soap bubble is a sphere because it minimizes surface tension
          \end{itemize}
    \item Chemistry
          \begin{itemize}
            \item Protein folding
          \end{itemize}
    \item Inverse problems
          \begin{itemize}
            \item imaging, denoising, deblurring
          \end{itemize}
  \end{itemize}

\end{frame}

\begin{frame}
  \frametitle{Optimization for ML}
  \begin{minipage}{0.5\textwidth}
    \begin{equation}
      \min_{\beta_1, \beta_0} \sum_{i=1}^{n} {( \beta_1 x_i + \beta_0 - y_i)}^2
    \end{equation}
    For data points $(x_i, y_i)$.
  \end{minipage}
  \hfill
  \begin{minipage}{0.45\textwidth}
    \begin{figure}[ht]
      \centering
      \includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{linear_regression.jpg}
      % \caption{\label{fig:label} }
    \end{figure}
  \end{minipage}

  \begin{itemize}
    \item Loss functions express the discrepancy between the predictions of the model being trained and the actual problem instances
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Optimization for ML}
  \begin{itemize}
    \item \textcolor{blue}{Mathematical modeling}
          \begin{itemize}
            \item defining \& modeling the problem
            \item finding a good metric / what is success
            \item accuracy vs.\ solvability trade-off
          \end{itemize}
    \item \textcolor{blue}{Computational optimization}
         \begin{itemize}
           \item running an (appropriate) optimization algorithm
         \end{itemize}
    \item theory vs. practice
          \begin{itemize}
            \item libraries available, but algorithms treated as ``black box'' by practitioners
            \item we will try and understand why and how they work
          \end{itemize}
  \end{itemize}
\end{frame}

\section{Methods}%

\begin{frame}
  \frametitle{Optimization Algorithms}
  \begin{center}
    \textit{\textcolor{blue}{Simplicity} rules in the large scale setting.}
  \end{center}
  Main approaches:
  \begin{itemize}
    \item First order methods: \textbf{gradient descent}
    \item Stochastic gradient descent (SGD)
    \item Coordinate descent
  \end{itemize}

  \textbf{History}
  \begin{itemize}
    \item 1847: Cauchy proposes gradient descent
    \item 1950s: Linear programming, operations research, soon followed by nonlinear
    \item 1980s: general convergence theory
    \item 2005-today: large scale optimization, SGD, distributed optimization
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Example: Coordinate descent}
  \begin{figure}[ht]
    \centering
    \includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{coordinate_descent}
    % \caption{\label{fig:label} }
  \end{figure}
  \textcolor{blue}{\textbf{Strategy:}} Minimize along one coordinate at a time, while keeping the others fixed.
\end{frame}

\begin{frame}
  \frametitle{Example: Gradient descent}
  \begin{figure}[ht]
    \centering
    \includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{gradient_descent}
    % \caption{\label{fig:label} }
  \end{figure}
  \textcolor{blue}{\textbf{Strategy:}} Follow the direction of (local) \textbf{steepest descent}.
\end{frame}

\begin{frame}
  \frametitle{}
  \begin{figure}[ht]
    \centering
    \includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{gradient_descent_meme}
    \caption{\label{fig:label} }
  \end{figure}
\end{frame}

\begin{frame}
  \frametitle{Optimization in other settings}
  \begin{itemize}
    \item \textcolor{blue}{Second order}
          \begin{itemize}
            \item if high precision in solution is required
            \item too \textbf{expensive} in high dimensions
          \end{itemize}
    \item \textcolor{blue}{Zeroth order}
          \begin{itemize}
             \item no gradient or functional representation available
             \item only function values
             \item for simulation, hyperparameters, black box models
          \end{itemize}
    \item constrained problems
    \item discrete optimization
          \begin{itemize}
            \item involving graphs, traveling salesman
            \item scheduling
          \end{itemize}
  \end{itemize}
\end{frame}

\section{Convexity}%
\label{sec:}

\begin{frame}
  \frametitle{Convex sets}
  A set $C$ is \textcolor{blue}{convex} if the line segment between any two points remains inside $C$, i.e.\ for any $x,y \in C$ and $\lambda\in [0,1]$.
  \begin{figure}[ht]
    \centering
    \includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{convex_sets}
    % \caption{\label{} }
  \end{figure}
  \begin{center}
    Which of these sets are convex?
  \end{center}

\end{frame}

\begin{frame}
  \frametitle{Properties of convex sets}
  \begin{itemize}
    \item intersection remains convex
    \item can separated by a hyperplane
    \item projections onto them are unique
          \begin{equation}
            P_C(x) := \argmin_{y\in C} \Vert y-x \Vert
          \end{equation}
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Convex functions}
  We call a function $f \to \R\cup\{+\infty\}$ \textcolor{blue}{convex} if the function values lie below the line segment between $(x, f(x))$ and $(y, f(y))$, i.e./ for any $\lambda\in [0,1]$
  \begin{figure}[ht]
    \centering
    \includegraphics[width=\textwidth,keepaspectratio]{convex_functions}
  \end{figure}

  Sometimes we will call $\{x: f(x)< +\infty \}$ the \textit{domain} of $f$.

\end{frame}

\begin{frame}
  \frametitle{Motivation: Convex optimization}
  Are of the form
  \begin{equation}
    \begin{aligned}
      &\min_x \, f(x) \\
      &\text{such that $x\in C$}
    \end{aligned}
  \end{equation}
  where \textcolor{blue}{both}
  \begin{itemize}
    \item $f$ is a convex function
    \item $C$ is a convex set
  \end{itemize}
  Why?
  \begin{itemize}
    \item \textit{Every local minimum is a global minimum.}
    \item Not all problems are convex but can be used as approximate model.
  \end{itemize}

\end{frame}

\begin{frame}
  \frametitle{Motivation: Provably (efficiently) solving convex problems}
  For convex optimization problems, basically all algorithms
  \begin{itemize}
    \item Coordinate Descent, (Stochastic) Gradient Descent, Proj.\ GD
  \end{itemize}
  \textbf{converge} \textcolor{blue}{provably} to a global optimum including a
  \begin{itemize}
    \item \textbf{quantitative bound}.
  \end{itemize}

  \begin{block}{ Example Theorem }
    Let $f: \R^d \to \R$ be convex then the \textbf{\textcolor{blue}{convergence rate}} is proportional to $1/k$, i.e.\
    \begin{equation}
      f(x_k) - f(x^*) \le \frac{c}{k}
    \end{equation}
  \end{block}
  Explanation: The \textcolor{blue}{approximation error} converges to zero and we know how many iterations are needed to achieve given target.
\end{frame}

\begin{frame}
  \frametitle{Examples of convex functions}
  \begin{itemize}
    \item linear: $f(x) = a^T x$
    \item affine: $f(x) = a^T x + b$
    \item exponential: $f(x) = e^{\alpha x}$
    \item norms, $\Vert \cdot \Vert_1, \Vert \cdot \Vert_2, \Vert \cdot \Vert_\infty$ \hfill \textcolor{purple}{See ex. 1.(i)}
    \item composition of linear and convex: \hfill \textcolor{purple}{See ex. 1.(ii)}
          \newline for example $f(x)=\Vert Ax-b  \Vert^2$
    \item sum of two convex function $f+g$
  \end{itemize}

\end{frame}

\begin{frame}
  \frametitle{Differentiable function}
  Derivative at a point is the \textcolor{blue}{best linear approximation} of the function at this point.
  \begin{figure}[ht]
    \centering
    \includegraphics[width=0.8\textwidth,keepaspectratio]{differentiable_function}
    % \caption{\label{fig:label} }
  \end{figure}
  \begin{center}
    Graph of $f(x)+ {\nabla f(x)}^T (y-x)$ is a \textcolor{blue}{tangent hyperplane} to the graph of $f$ at $(x,f(x))$
  \end{center}
\end{frame}

\begin{frame}
  \frametitle{First-order characterization of convexity}

  \begin{figure}[ht]
    \centering
    \includegraphics[width=0.8\textwidth,keepaspectratio]{graph_above_tangent}
    % \caption{\label{fig:label} }
  \end{figure}
  If $f$ is differentiable, then
  \begin{center}
    $f$ is convex \emph{if and only if:} $f(y) \ge f(x) + {\nabla f(x)}^T (y-x)$
  \end{center}
\end{frame}

\begin{frame}
  \frametitle{Nonsmooth functions }
  do in fact play a role in practice
  \begin{itemize}
    \item ReLu, Hinge loss, norms
    \item can induce sparsity in the solution
    \item appear as the maximum over a family of functions (max pooling, or min-max)
  \end{itemize}
  \begin{figure}[ht]
    \centering
    \includegraphics[width=0.7\textwidth,keepaspectratio]{nonsmooth}
    % \caption{\label{fig:label} }
  \end{figure}

\end{frame}

\begin{frame}
  \frametitle{Second-order characterization of convexity}
  If $f$ is \textcolor{blue}{twice differentiable} then it is \textcolor{blue}{convex} \textbf{if and only if} its \emph{Hessian} $\nabla^2 f(x) \in \R^{d \times d}$, given by
  \begin{equation}
    {\nabla^2 f(x)}_{ij} := \frac{\partial^2 f}{\partial x_i \partial x_j}
  \end{equation}
  is \textcolor{blue}{positive semidefinite}, i.e.
  \begin{equation}
    \nabla^2 f(x) \succcurlyeq 0
  \end{equation}

  \textcolor{gray}{A matrix $M$ is \emph{positive semidefinite} if $x^T M x \ge 0$ for all $x$.}\\
  Also used in algorithm like \emph{Newtons} method.

\end{frame}

\begin{frame}
  \frametitle{Examples}
  \begin{itemize}
    \item \textcolor{blue}{quadratic function:} $f(x)= \frac12 x^T Q x + c^T x$, then
          \begin{equation}
            \nabla^2 f(x) = Q
          \end{equation}
          and $f$ is convex iff $Q \succcurlyeq 0$.
    \item \textcolor{blue}{least squares objective:} $f(x) = \Vert Ax -b \Vert^2$, then
          \begin{equation}
             \nabla^2 f(x) = A^T A
          \end{equation}
          is always convex for any A.
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Local minima are global}

  \begin{definition}
    A \textcolor{blue}{local minimum} of $f$ is a point $\bar{x}$ such that there exists $\epsilon>0$
    \begin{equation}
      f(\bar{x}) \le f(y) \quad \forall y: \text{s.t.} \Vert \bar{x}-y \Vert \le \epsilon
    \end{equation}
  \end{definition}

  \begin{lemma}%
    \label{lem:}
    Let $x^*$ be local minimum of a convex function $f$ then $x^*$ is a global minimum.
  \end{lemma}
\textcolor{purple}{See ex 1.(iii)}
\end{frame}

\begin{frame}
  \frametitle{Local vs. global minima}
  \begin{figure}[ht]
    \centering
    \includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{local_global_minimum}
    % \caption{\label{fig:label} }
  \end{figure}

\end{frame}

\begin{frame}
  \frametitle{Critical points are global minima}

  \begin{definition}
    We call a point $\bar{x}$ \textcolor{blue}{critical} or \textcolor{blue}{stationary} if
    $\nabla f(\bar{x}) = 0$.
  \end{definition}
  \begin{lemma}%
    If $\bar{x}$ is a stationary point of the \textbf{convex} function $f$, then $\bar{x}$ is a \textcolor{blue}{global minimizer} of $f$.
  \end{lemma}
  \textcolor{purple}{See ex. 1.(iv)}

\end{frame}

\begin{frame}
  \frametitle{Strong convexity}
  \begin{definition}
    We call $f$ \textcolor{blue}{strongly convex} if there exist $\mu> 0$ such that
    \begin{equation}
      f - \frac{\mu}{2} \Vert \cdot \Vert^2 \quad \text{is convex.}
    \end{equation}
  \end{definition}

  Equivalently:
  \begin{itemize}
    \item can be lower bounded by a quadratic
          \begin{equation}
            f(x) + \langle \nabla f(x), y-x \rangle + \frac{\mu}{2} \Vert y-x \Vert^2 \le f(y)
          \end{equation}
    \item Hessian is pos. def. everywhere
          \begin{equation}
            \nabla^2 f(x) \succ 0.
          \end{equation}
  \end{itemize}

\end{frame}

\begin{frame}
  \frametitle{Constrained minimization}

  \begin{minipage}{0.5\textwidth}
    \begin{definition}
      $x^*$ is a minimizer of $f$ \textcolor{blue}{over $C$} if
      \begin{equation}
        f(x^*) \le f(x), \forall x \in C.
      \end{equation}
    \end{definition}
    \begin{lemma}%
      $x^*$ is a minimizer of $f$ over $C$ \emph{if and only if}
      \begin{equation}
        \langle \nabla f(x^*), x-x^* \rangle \ge 0 \quad \forall x \in C.
      \end{equation}
    \end{lemma}
  % \textcolor{purple}{Prove this.}
  \end{minipage}
  \hfill
  \begin{minipage}{0.45\textwidth}
  \begin{figure}[ht]
    \centering
    \includegraphics[width=\textwidth,height=\textheight,keepaspectratio]{constrained.png}
    % \caption{\label{fig:label} }
  \end{figure}
  \end{minipage}
\end{frame}

\begin{frame}
  % \frametitle{Proof}
  \begin{lemma}%
    $x^*$ is a minimizer of $f$ over $C$ \emph{if and only if}
    \begin{equation}
      \langle \nabla f(x^*), x-x^* \rangle \ge 0 \quad \forall x \in C.
    \end{equation}
  \end{lemma}
  ``$\Leftarrow$'' From the gradient inequality we deduce
  \begin{equation}
    f(x) - f(x^*) \ge \langle \nabla f(x^*), x-x^* \rangle \onslide<2->{\ge 0} \quad \forall x \in C.
  \end{equation}
  ``$\Rightarrow$'' Assume that $f(x^*)\le f(x)$ for all $x \in C$ then $\forall t \in [0, 1]$
  \begin{equation}
    \begin{aligned}
      0 &\le f(x^* + t(x-x^*)) - f(x^*) \\
      0 &\le \lim_{t\to 0} \frac{f(x^* + t(x-x^*)) - f(x^*)}{t} \\
        &= \langle \nabla f(x^*), x-x^* \rangle,
    \end{aligned}
  \end{equation}
  where the last equality follows from the chain rule.\qed

\end{frame}

\begin{frame}
  \frametitle{Existence of a minimizer}

  In general a minimizer \textit{does not need to exist}.
  \begin{itemize}
    \item can be unbounded from below (linear)
    \item bounded but infimum is not obtained
  \end{itemize}
  \begin{center}
    \begin{tikzpicture}
      \draw[->] (-3,0) -- (1.5,0) node[right] {$x$};
      \draw[->] (0,-0.5) -- (0,2.5) node[above] {$y$};
      \draw (0,1) node[right] {$y=1$};
      \draw (1,2.7) node[right] {$f(x)=e^x$};
      \draw[scale=1,domain=-3:1,smooth,variable=\x,blue] plot ({\x}, {exp(\x)});
    \end{tikzpicture}
  \end{center}

  Typically we only consider problems where we assume a minimizer to exist (otherwise our model might be bad).

  \begin{itemize}
    \item if function is strongly convex a minimizer \textit{always} exists.
  \end{itemize}

\end{frame}

\end{document}


\documentclass[12pt]{article}
\usepackage{graphics}
\begin{document}

\section*{NYU Engineering Physics 1---In-class Exam 4}

\vfill

\paragraph{Name:} ~

\paragraph{email:} ~

\vfill

This exam consists of two problems.  Write only in this booklet.  Be
sure to show your work.

\vfill ~

\clearpage

\section*{Problem 1}

As a physics demonstration, NYU decides to hang a $M=50~\mathrm{kg}$
pendulum bob from the roof of Bobst Library, by a light cable, in the
12-story-tall lobby.  The cable has a length $\ell$ of 12 stories, and
the pendulum is set swinging with a horizontal amplitude of $x_0=5$~m.

(a) What is the period $T=(2\pi/\omega)$ you expect for the
oscillations of this pendulum?  Give a symbolic answer and a numerical
answer in s.

\vfill

(b) What is the maximum horizontal speed $v_\mathrm{max}$ of the
pendulum bob swinging with this amplitude?  Give a symbolic answer and
a numerical answer in $\mathrm{m\,s^{-1}}$.

\vfill

(c) How much total mechanical energy $E$ is stored in the oscillation?
Give a symbolic answer and a numerical answer in J.

\vfill ~

\clearpage

\section*{Problem 2}

When an olympic diver of mass $M$ walks out to the tip of the diving
board, the diver's weight $M\,g$ causes the tip to sag to a new
equilibrium position a distance $y_\mathrm{eq}$ below its ``unloaded''
equilibrium position.  Estimate, roughly, this distance
$y_\mathrm{eq}$ using the following fact: When the diver is standing
at the very end of a diving board, he or she oscillates vertically
with a period $T$ of roughly 0.5~s.  Give a symbolic answer and plug
in reasonable numbers.

\emph{Hint 1:} Assume the diving board is much lighter than the diver.

\emph{Hint 2:} Model the diving board as a massless spring.

\emph{Hint 3:} Start by getting a \emph{dimensionally correct} guess
before attempting your calculation.

\vfill ~

\clearpage

[This page intentionally left blank for calculations or other work.]

\end{document}


% - Week 1 is on the website: https://math216.wordpress.com/2020/06/27/discussion-for-this-week-starting-june-27-2020/
% - Week 2:https://math216.wordpress.com/2020/07/06/readings-and-problem-set-for-after-the-second-pseudolecture/
% - Week 3: https://math216.wordpress.com/2020/07/13/reading-and-problems-after-the-third-pseudolecture/
% Link to rising sea: https://math.stanford.edu/~vakil/216blog/FOAGnov1817public.pdf
% https://agittoc.zulipchat.com/
\documentclass{book}

%%% % https://tex.stackexchange.com/a/97128
\usepackage[sc,osf]{mathpazo}   % With old-style figures and real smallcaps.
%%% \linespread{1.025}              % Palatino leads a little more leading
%%% % Euler for math and numbers
%%% \usepackage[euler-digits,small]{eulervm}

% \usepackage{cmbright}

\usepackage{mathtools}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{minted}
\usepackage{hyperref}
\usepackage{tikz}
\usepackage{tikz-cd}
\usepackage{cancel}
\usepackage{mathtools}

% https://tex.stackexchange.com/questions/469588/strike-out-an-arrow-with-a-small-oblique-segment-like-with-nrightarrow
% \newcommand*\Neg[2][0mu]{\Neginternal{#1}{\negslash}{#2}}
% \newcommand*\sNeg[2][0mu]{\Neginternal{#1}{\snegslash}{#2}}
% \newcommand*\negslash[1]{\m@th#1\not\mathrel{\phantom{=}}}
% \newcommand*\snegslash[1]{\rotatebox[origin=c]{60}{$\m@th#1-$}}

\newcommand{\Hom}{\operatorname{Hom}}
\newcommand{\Tr}{\operatorname{Tr}}
\newcommand{\tr}{\operatorname{tr}}
\newcommand{\F}{\ensuremath{\mathcal{F}}}
\newcommand{\G}{\ensuremath{\mathcal{G}}}
\newcommand{\N}{\ensuremath{\mathbb{N}}}
\newcommand{\Z}{\ensuremath{\mathbb{Z}}}
\newcommand{\T}{\ensuremath{\mathbb{T}}}
\newcommand{\Q}{\ensuremath{\mathbb{Q}}}
\newcommand{\C}{\ensuremath{\mathbb{C}}}
\newcommand{\Cstar}{\ensuremath{\mathbb{C}^*}}
\newcommand{\R}{\ensuremath{\mathbb{R}}}
\newcommand{\CP}{\ensuremath{\mathbb{CP}}}
\renewcommand{\P}{\ensuremath{\mathbb{P}}}
\newcommand{\A}{\ensuremath{\mathbb{A}}}
\renewcommand{\O}{\ensuremath{\mathcal{O}}}
\newcommand{\RP}{\ensuremath{\mathbb{RP}}}
\newcommand{\Spec}{\operatorname{Spec}}
\newcommand{\spec}{\operatorname{Spec}}
\newcommand{\m}{\mathfrak{m}}
\newcommand{\p}{\mathfrak{p}}
\newcommand{\q}{\mathfrak{q}}
\newcommand{\mSpec}{\m\operatorname{Spec}}
\newcommand{\mspec}{\m\operatorname{Spec}}
\newcommand{\frakp}{\ensuremath{\mathfrak{p}}}
\newcommand{\fraka}{\ensuremath{\mathfrak{a}}}
\newcommand{\coker}{\operatorname{coker}}
\newcommand{\colim}{\operatorname{colim}}
\newcommand{\Span}{\operatorname{span}}
\newcommand{\spn}{\Span}
\newcommand{\inv}{{\ensuremath{-1}}}
\newcommand{\im}{\operatorname{Im}}
\newcommand{\piinv}{\ensuremath{\pi^\inv}}
\renewcommand{\k}{\mathbb{k}} % field
\newcommand{\pinv}{\piinv}
\newcommand{\piinverse}{\piinv}
\newcommand{\supp}{\operatorname{Support}}
\newcommand{\Set}{\ensuremath{\mathbf{Set}}}
\newcommand{\rad}{\sqrt} % radical
\newcommand{\ann}{\operatorname{Annhilator}} % annhilator
\newcommand{\osum}{\oplus} % I confuse it often enough
\newcommand{\tensor}{\otimes} % I confuse it often enough
\renewcommand{\c}{\complement} % got tiring
\newcommand{\geqzero}{\ensuremath{\geq 0}}
\newcommand{\gezero}{\gezero}
\newcommand{\gtzero}{\ensuremath{> 0}}
\newcommand{\sym}{\operatorname{Sym}}
\newcommand{\Sym}{\sym}
\newcommand{\nil}{\operatorname{NilRadical}}
\newcommand{\Nil}{\nil}
\newcommand{\grade}{\ensuremath{\bullet}}

\DeclarePairedDelimiter\ceil{\lceil}{\rceil}
\DeclarePairedDelimiter\floor{\lfloor}{\rfloor}
\DeclarePairedDelimiter\fracpart{\{}{\}}
\theoremstyle{definition}
\newtheorem{theorem}{Theorem}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{example}[theorem]{Example}
\newtheorem{nonexample}[theorem]{Non Example}
\newtheorem{aside}[theorem]{Aside}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{exercise}[theorem]{Exercise}
\newtheorem{note}[theorem]{Note}
\newtheorem{slogan}[theorem]{Slogan}
\newtheorem{Proposition}[theorem]{Proposition}

\begin{document}
\tableofcontents

% https://qchu.wordpress.com/2009/08/30/the-orthogonality-relations-for-representations-of-finite-groups/

\chapter{Repr Theory of finite groups: Benjamin Steinberg: Ch 3}
\begin{theorem} Semisimplicity of representations of finite groups:
Every representation can be written as a direct sum of irreducible representations.
\end{theorem}
\begin{proof}
Let $\rho : G \rightarrow GL(V)$ be a representation. If $\rho$ is irreducible,
we are done. Otherwise, let $W \subsetneq V$ be a non-trivial $G$ invariant
subspace of $V$. Now, let $\langle \cdot, \cdot \rangle$ be any
inner product (which definitely exists. Take a basis and create an inner product
from that). From this, Induce an averaged inner product:

\begin{align*}
\langle v_1, v_2 \rangle_G \equiv
\sum_{g \in G}  \langle \rho(g) v_1, \rho(g) v_2 \rangle
\end{align*}

Under this inner product, $\rho(h)$ is unitary for all $h \in H$. This is
proved as follows:

\begin{align*}
&\langle \rho(h) v_1, \rho(h) v_2 \rangle_H = \sum_{g \in G}  \langle (\rho(g)(\rho(h) v_1)), \rho(g)(\rho(h)(v_2)) \rangle \\
&= \sum_{g \in G}  \langle \rho(gh) (v_1), \rho(gh)(v_2) \rangle \\
&= \sum_{g = g' h^{-1} \in G}  \langle \rho(gh) (v_1), \rho(gh)(v_2) \rangle \\
&= \sum_{g = g' h^{-1} \in G}  \langle \rho(g' h^{-1} h) (v_1), \rho(g' h^{-1} h)(v_2) \rangle \\
&= \sum_{g = g' h^{-1} \in G}  \langle \rho(g') (v_1), \rho(g')(v_2) \rangle \\
&= \langle v_1 , v_2 \rangle_G
\end{align*}

Thus, we have $W$ as a $G$ invariant subspace of the unitary $\rho(g)$. So we
can define the orthogonal subspace $W^\perp \equiv \{ v \in V : \langle v, W \rangle_G = 0 \}$. That is:
$W^\perp \equiv \{ v \in V : \forall w \in W, \langle v, w \rangle_G = 0\}$.
This is intuitively invariant , because if it were not, then we would have some
element $g \in G$ that would pull elements of $W^\perp$ into $W$: $W^\perp \xmapsto{g} W$.
But this would mean that $W \xmapsto{g^{-1}} W^\perp$. This is impossible because $W$
is a $G$-invariant subspace.

Formally, let $w^\perp \in W^\perp$. Now note that if $\rho(g) w^\perp \in W$,
then there exists a $w \in W$ such that $\langle \rho(g)w^{\perp}, w \rangle \neq 0$.
This gives us the computation:

\begin{align*}
&\langle \rho(g)w^{\perp}, w \rangle \neq 0 \\
&=\langle \rho(g)w^{\perp}, \rho(g) \rho(g^{-1}) w \rangle \neq 0 \\
&\text{$\rho(g)$ is unitary, so can be removed from inner product computation} \\
&= \langle w^{\perp}, \rho(g^{-1}) w \rangle \neq 0 \\
&\text{(Let $w' = \rho(g^{-1})w \in W$ as $W$ is $G$-invariant)} \\
&= \langle w^{\perp}, w'\rangle \neq 0 \\
\end{align*}

But this means that $w^{\perp}$ is not orthogonal to $W$, a contradiction.

\end{proof}

\section{3.3: Equivalent 1D representations are equal}

Let $p, q : G \rightarrow \C^*$. Let there be an isomorphism $T: \C^*  \rightarrow \C^*$
such that $p = Tq T^{-1}$.

Write with respect to a basis. $p(x) = \lambda_p x, q(x) = \lambda_q x, T(x) = \lambda_Tx $.
This gives us the equation:

\begin{align*}
&p = T q T^\inv \\
&p(x) = T(q(T^\inv(x))) \\
&\lambda_p(x) = \lambda_t(\lambda_q(1/\lambda_t))x \\
&\lambda_p(x) = \lambda_q(x) \\
&\lambda_p = \lambda_q
\end{align*}

\section{3.5: average of representations is a projection}
Let $p: G \rightarrow GL(V)$ be a representation. Define the fixed
subspace $V^G = \{ v \in V : \forall g \in G, p_g(v)  = v \}$.

First we see that this subspace is $G$ invariant. Let $v \in V^G$. $p_g(v) = v$
(by definition of $V^G$), hence $p_g(v) \in V^G$.

Define $P: V \rightarrow V; P(v) \equiv 1/|G|\sum_{g \in G}p_g(v)$.
We need to show that $P : V \rightarrow V^G$ (That is, the image lies entirely in $V^G$).

Consider $v \in V$. Now compute:

\begin{align*}
&p_h(P(v)) = p_h(1/|G|\sum_{g \in G} p_g(v)) \\
& = 1/|G|\sum_{g \in G} p_h(p_g(v)) \\
& = 1/|G|\sum_{g \in G} p_{hg}(v) \\
& \text{($g \mapsto hg$ is a bijection. $\sum_{g \in G} p_{hg}$ visits each element)} \\
& = 1/|G|\sum_{g \in G} p_{g}(v) = P(v)\\
\end{align*}

$p_h(P(v)) = P(v)$ hence $\forall v \in V, P(v) \in V^G$.

Next we are to show that $P: V \rightarrow V^G$ is a projection: $P \circ P = P$.
See that the image of $P$ is entirely in $V^G$. Now we need to show that $P(v \in V^G) = v$:
\begin{align*}
&P(v \in V^G) = 1/|G| \sum_{g \in G} p_h(v) \\
&\text{(since $v \in V^G$, $v$ is fixed by all $p_h$)} \\
&P(v \in V^G) = 1/|G| \sum_{g \in G} v = 1/|G| (|G| v) = v
\end{align*}

Hence, $P \circ P = P$.

Now, we can show that $dim(V^G)$ is the rank of $P$. In the eigenbasis of $P$,
we will have a $+1$ for each basis vector in the image $V^G$ and a zero
everywhere else. Thus the rank of $P$ is the dimension of $V^G$.

Similarly, by the eigenbasis argument, the trace of $P$ will add up all the $1$s
in the eigenbasis, which is equal to the dimension of $V^G$.

Thus we conclude:

$$
dim(V^G) = rank(P) = Tr(P) = 1/|G| \sum_g Tr(p_g)
$$

\section{3.6: conjugates don't have to be equivalent}
Let $p: G \rightarrow GL(C)$ be a representation. Set $s(v) = \overline{p(v)}$.
Show that $p, s$ do not have to be equivalent.

Let $G \equiv \mathbb Z/3Z$. Let $p(1) \equiv e^{i\pi/3}$. We get $s(1) = \overline{p(1)} = \overline{e^{i \pi/3}} = e^{i 2 \pi/3}$.
These are not equivalent, because they are 1D representations and are not
isomorphic (recall that equivalent 1D representations are isomorphic).

\section{3.7: bijection between unitary 1D representations of $\Z$ and elements of $\T$}

Recall that $\T \equiv \{ e^{ir} : r \in \R \}$.

Consider a 1D representation of $\Z$, $p: \Z \rightarrow \Cstar$.
We must have that $p(z) p^{-1}(z) = 1$. Since $p$ is unitary, we have that $p^{-1} = p^\dagger = p^*$.
Thus we get $p(z)p^*(z) = 1$, or $|p(z)|^2 = 1$. Hence $p(z) \in \T$. Biject
to $T$ considering the image of $1 \mapsto p(1) \in T$, since the representation of $\Z$
is determined by the image of $1$ under $p$.

\section{3.8: Eigenvectors}
(TODO).

\chapter{Repr Theory of finite groups: Benjamin Steinberg: Ch 4}

Point of interest that I had never noticed: the space of representation morphisms/intertwinings
themselves form a vector space!

\begin{theorem}
Irreducible representations (irreps) of an abelian $G$ have degree 1
\end{theorem}
\begin{proof}
Let $\phi: G \rightarrow V$ where $G$ is abelian.
For some $h \in G$. Define $T \equiv \phi_h$. Now note that:

$$ T \phi_g = \phi_h \phi_g = \phi{hg} = \phi_g \phi_h = \phi_g T $$. So we have
a $T$ that is an intertwining of $\phi$ with $\phi$: $T_h \in \Hom(\phi, \phi)$.
Thus, by schur's lemma, we must have that $T_h = \lambda_h I$. Now let $v \in V$.
Consider some element $kv \in \C v$: $T_h(kv) = \lambda_h (kv) \in \C v$.
Hence, $T_h(\C v) \in \C v$.

Now we can see that $\C v$ is a $G$ invariant subspace, since the choice of $h$
was arbitrary.

But since the representation $\phi$ is irreducible, we must have that $\C v$
equals the whole space $V$, and thus $\dim V = 1$

\end{proof}

\begin{theorem}
representations of abelian groups are simultaneously diagonalizable.
\end{theorem}
\begin{proof}
Let $\phi: G \rightarrow GL(V)$. Since all operators $\phi_g, \phi_h$ commute
pairwise, they are all simultaneously diagonalizable with each other. Hence
there exists some invertible transform $T$ such that $T \phi_g T^{-1}$ is
diagonal for all $g \in G$ ($T$ is invariant of $G$).
\end{proof}

\begin{definition}
Group algebra: The inner product space of functions $L(G) \equiv \mathbb C^G = \{ f: G \rightarrow \C \}$,
with inner product given by $\langle f | f' \rangle \equiv  1/|G|\sum_{g \in G} f(g) \overline{f'(g)}$,
and algebra given by convolution: $(f f')(g) \equiv \sum_{hh' = g}f(h_1)f'(h')$.
\end{definition}

\begin{lemma}
All linear maps can be averaged to get intertwinings. Let $p: G \rightarrow GL(V)$
and $r: G \rightarrow GL(W)$. Let $T \in \Hom(V, W)$ be a linear map. Now
define $T^\sharp: V \rightarrow W;  T^\sharp \equiv 1/|G| \sum_{g \in G} p_g; T; r_{g^-1}$.
We will show:
\begin{enumerate}
\item $T^\sharp \in \Hom(p, r)$ is an intertwining for any linear map.
\item $T \in \Hom_G(p, r)$ then $T^\sharp = T$.
\item $( \cdot )^\sharp : \Hom(V, W) \rightarrow  \Hom_G(p, r)$ is linear.
\item $( \cdot )^\sharp : \Hom(V, W) \rightarrow  \Hom_G(p, r)$ is onto.
\end{enumerate}

Thus, the map $( \cdot)^\sharp$ is a projection from the space $\Hom(V, W)$ to the
subspace $\Hom_G(p, r)$.
\end{lemma}
\begin{proof}
We verify that $T^\sharp$ is an intertwining by direct computation. We need
to check that $p_g; T^\sharp = T^\sharp; r_g$:

\begin{align*}
&x; T^\sharp; r_g =  \\
&x; 1/|G| \sum_{x \in G} (p_g; T; r^{g^{-1}}); r_g \\
&1/|G| \sum_{x \in G} (x; p_g; T; r^{e}) \\
&1/|G| \sum_{x \in G} (x; p_g; T) \\
\end{align*}

To prove (2), if $T \in \Hom(p, r)$, then consider:

\begin{align*}
&T^\sharp(x) = 1/|G|\sum_{g \in G} r_{g^{-1}} (T(p_g (x))) \\
& = 1/|G|\sum_{g \in G} r_{g^{-1}} (r_g(T (x))) \\
& = 1/|G|\sum_{g \in G} r_{e}(T (x))) \\
& = 1/|G|\sum_{g \in G} (T (x))) \\
& = T(x)
\end{align*}

(3) To establish linearity, we compute once again:

\begin{align*}
&(c_1T_1 + c_2 T_2)^\sharp(x) = 1/|G|\sum_{g \in G} r_{g^{-1}} (c_1 T_1 + c_2 T_2) p_g \\
&(c_1T_1 + c_2 T_2)^\sharp(x) = \\
& \quad c_1 \cdot 1/|G|\sum_{g \in G} r_{g^{-1}}  T_1 \rho_g +
 c_2 \cdot 1/|G|\sum_{g \in G} r_{g^{-1}} T_2 \rho_g \\
&= c_1 T_1^\sharp + c_2 T_2^\sharp
\end{align*}

(3) immediately follows. The map is onto because $\Hom(p, r) \subseteq \Hom(V, W)$.
Hence, every element $T \in \Hom(p, r)$ is mapped onto itself by $T$ itself:
every $T \in \Hom(p, r) \in Hom(V, W)$, we have that $T^\sharp = T \in \Hom(p, r) \in \Hom(V, W)$.

Alternatively, note that $\Hom(p, r)$ is a subspace of $\Hom(V, W)$, and that
$\sharp$ is a projection. Since projections are always onto their \emph{image}, we are done.
\end{proof}

\begin{theorem}
We know exactly what $T^\sharp$ looks like.
If $p \in GL(V)$, $r \in GL(W)$, $T \in \Hom(V, W)$:
\begin{enumerate}
\item If $p \neq r$ then $T^\sharp = 0$
\item If $p = r$ then $T^\sharp = \frac{\Tr(t)}{\deg(p)} I$
\end{enumerate}
\end{theorem}
\begin{proof}
If $p \neq r$ then $\Hom(p, r) = 0$ and hence $T^\sharp \in \Hom(p, r) = 0$.

If $p = r$ then by Schur's lemma $T^\sharp = \lambda I$ for some $\lambda in \C$. Recall
the argument: Let $T$ have full eigenspectrum. Then, note that the eigenspaces
of different eigenvalues are orthogonal. But that's impossible if the
representation is irreducible. So we have only a single eigenspace, corresponding
to the single eigenvalue $\lambda$.  Thus we must have that $T^\sharp = \lambda I$.
We want to know precisely what $\lambda$ is. Since $T \in \hom(p, r)$, we have
that $V = W$ and $T \in \Hom(V, V)$. Thus,

\begin{align*}
&\Tr(T^\sharp) = \lambda \Tr(I_{V, V}) \\
& \Tr(T^\sharp) = \lambda \dim V = \lambda \deg \phi \\
& \lambda = \Tr(T^\sharp)/deg \phi
\end{align*}

We can now relate $\Tr(T^\sharp)$  to $\Tr(T)$ using the definition of $T^\sharp$,
and remember that $p = r$:
\begin{align*}
&\Tr(T^\sharp) = 1/|G| \sum_{g \in G} Tr(r_{g^{-1}} T p_g) \\
&= 1/|G| \sum_{g \in G}\Tr(p_{g^{-1}} T p_g) \\
&= 1/|G| \sum_{g \in G} \Tr(p_g p_{g^{-1}} T) \\
& = 1/|G| \sum_{g \in G} \Tr(p_{g g^{-1}} T) = 1/|G| \sum_{g \in G}\Tr(p_e T) \\
&= 1/|G| \sum_{g \in G} \Tr(I T) \\
&= \Tr(T)
\end{align*}

Hence, $T^\sharp = \lambda I = \frac{\Tr(T^\sharp)}{\deg \phi}I = \frac{\Tr(T)}{\deg \phi}I$
\end{proof}

\begin{theorem}
Let $p: G \rightarrow U_n(\C)$, $r: G \rightarrow U_m(\C)$ be unitary representations
of $G$. Let $A^x_y \equiv \delta^x_k \delta^i_y \in M_{mn}(\C)$. Then we will
show that ${A^\sharp}^l_j = \langle p^i_j, r^k_l \rangle$.
\end{theorem}
\begin{proof}
Since $r$ is unitary, $r_g^{-1} = r_g^\dagger$. Thus, $r[l][k]^{-1} = \overline{r[k][l](g)}$.
This gives us the computation:

\end{proof}

\section{Characters and class functions}

Let $\phi: G \rightarrow GL(V)$ be a representation. Define $\chi_\phi(g) \equiv \Tr(\phi(g))$.
Now $\chi_\phi: G \rightarrow \C^*$ is a 1 dimensional representation.

To be able to compute anything with characters, we need to choose a basis to
compute a trace. So we will from now on pick matrix representations, where
we will identify $GL(V)$ with $U_n(V)$ (space of unitary matrices) with an
appropriate choice of basis.

\chapter{Borcherds: Orthogonality relations}

\begin{itemize}
\item Link: \url{https://www.youtube.com/watch?v=BixUw1oHBec}
\end{itemize}

\begin{theorem} Schur's Lemma: If $V, W$ are irreducible then $\Hom_G(V, W)$ is 1D
if $V = W$ and $0D$ if $V \neq W$.
\end{theorem}
\begin{proof}
Let $T \in \Hom_G(V, W)$. The kernel of $T$ is a $G$ invariant subspace of $V$. since $V$
is irreducible either $\ker T = V$ (the map is zero) or $\ker T = 0$ the map is
injective. Similarly, $\Im(T)$ is $G$ invariant subspace  of $W$. Either $\Im(T) = 0$
and the map is zero, or $\Im(T) = W$ and the map is surjective. Hence the map
is either 0, or injective and surjective (thus bijective).

If the map is zero, then $\Hom_G(V, W) = 0$ since $T$ was arbitrary.
If the map is bijective, then every map in $\Hom_G(V, W)$ is invertible. This
makes $\Hom_G(V, W)$ a division algebra over $\C$. The complex numbers are
algebraically closed, and hence it must be isomorphic to $\C$ (why? This seems
like a very interesting argument).

This tells us that $\sum_g \chi_i(g) \chi_j(g) = 0 : i \neq j$, and
$\sum_g \chi_i(g) \chi_j(g) = |G| : i = j$. This is because $\Hom_G(V_j, V_i)$ is
$1$ or $0$ depending on whether $i = j$ or not.
\end{proof}

\chapter{Clebsh gordon theory}
Given two irreducible representations $r: G \rightarrow GL(V)$, $s: G \rightarrow GL(W)$,
their tensor product $r \tensor s: G \rightarrow GL(V \tensor W)$ need not
be irreducible. Decomposing $r \tensor s$ into irreducible representations is
the Clebsh Gordon problem.

\chapter{Fully Worked out theory for $S_3$}

\chapter{Borcherds: Schur indicator}

See that the usual averaging inner product is sesquilinear: it's linear
in the left, and conjugate linear in the right.

We want to understand the space of $G$ invariant \textbf{bilinear} forms: $V \tensor V \rightarrow \C$.

\end{document}