\documentclass[12pt]{article} \input{physics1}
\begin{document}

\noindent
Name: \rule[-1ex]{0.55\textwidth}{0.1pt}
NetID: \rule[-1ex]{0.2\textwidth}{0.1pt}

\section*{NYU Physics I---Term Exam 5}

\paragraph{\problemname~\theproblem:}\refstepcounter{problem}%
When you weigh yourself on a scale, you weigh less than your
gravitational force, because there is a buoyant-force correction. What
fraction of your weight is this correction? That is, what is the
buoyant force divided by the gravitational force, roughly?
(from Problem Set 9)

\vfill

\paragraph{\problemname~\theproblem:}\refstepcounter{problem}%
A figure skater spins in place on frictionless ice at angular speed
$\omega_i$ with hands outstretched. The skater has a total moment of inertia
$I_i$. As the skater draws his hands into his body, his moment of
inertia decreases to $I_f$ = $I_i/2$. Does the kinetic energy $K$
increase, decrease, or stay the same? Assume that there are no torques
acting.
(from Problem Set 10)

\vfill

\paragraph{\problemname~\theproblem:}\refstepcounter{problem}%
Immediately after being hit by the cue, a cue ball slides along the
felt in the $x$ direction at speed $v$. Draw a free-body diagram for
the cue ball, showing the forces acting, and clearly label the $x$
direction.
(from Problem Set 10)

\vfill
~
\clearpage

\paragraph{\problemname~\theproblem:}\refstepcounter{problem}%
What would be the length of an Earth year if the Earth was orbiting
on a circular orbit of radius $0.25\,\AU$ (instead of $1\,\AU$)?
Give your answer in units of days.
(from worksheet on orbits)

\vfill

\paragraph{\problemname~\theproblem:}\refstepcounter{problem}%
What is the relationship between linear acceleration $a$ and
angular acceleration $\alpha$ when a ball of radius $R$ is rolling
without slipping down a plane?
(from lecture 2018-11-08)

\vfill

\paragraph{\problemname~\theproblem:}\refstepcounter{problem}%
How is it that the astronauts in the Space Station are weightless?
Write a grammatically correct answer in one sentence, in fewer than 20
words.  Box your sentence!
(from lecture 2018-11-15)

\vfill
~
\end{document}


\documentclass[10pt,landscape]{article}
\usepackage{multicol}
\usepackage{calc}
\usepackage{ifthen}
\usepackage[landscape]{geometry}
\usepackage{hyperref}
\usepackage{amssymb,amsmath}
\usepackage{graphicx}
\usepackage{multirow}

% To make this come out properly in landscape mode, do one of the following
% 1.
%  pdflatex latexsheet.tex
%
% 2.
%  latex latexsheet.tex
%  dvips -P pdf  -t landscape latexsheet.dvi
%  ps2pdf latexsheet.ps

% This sets page margins to .5 inch if using letter paper, and to 1cm
% if using A4 paper. (This probably isn't strictly necessary.)
% If using another size paper, use default 1cm margins.
\ifthenelse{\lengthtest { \paperwidth = 11in}}
	{ \geometry{top=.5in,left=.5in,right=.5in,bottom=.5in} }
	{\ifthenelse{ \lengthtest{ \paperwidth = 297mm}}
		{\geometry{top=0.7cm,left=0.7cm,right=0.7cm,bottom=0.7cm} }
		{\geometry{top=0.7cm,left=0.7cm,right=0.7cm,bottom=0.7cm} }
	}

% Turn off header and footer
\pagestyle{empty}

% Redefine section commands to use less space
\makeatletter
\renewcommand{\section}{\@startsection{section}{1}{0mm}%
                                {-1ex plus -.5ex minus -.2ex}%
                                {0.5ex plus .2ex}%x
                                {\normalfont\large\bfseries}}
\renewcommand{\subsection}{\@startsection{subsection}{2}{0mm}%
                                {-1explus -.5ex minus -.2ex}%
                                {0.5ex plus .2ex}%
                                {\normalfont\normalsize\bfseries}}
\renewcommand{\subsubsection}{\@startsection{subsubsection}{3}{0mm}%
                                {-1ex plus -.5ex minus -.2ex}%
                                {1ex plus .2ex}%
                                {\normalfont\small\bfseries}}
\makeatother

% Define BibTeX command
\def\BibTeX{{\rm B\kern-.05em{\sc i\kern-.025em b}\kern-.08em
    T\kern-.1667em\lower.7ex\hbox{E}\kern-.125emX}}

% Don't print section numbers
\setcounter{secnumdepth}{0}

\setlength{\parindent}{0pt}
\setlength{\parskip}{0pt plus 0.5ex}

% -----------------------------------------------------------------------

\begin{document}

\raggedright
\footnotesize
\begin{multicols}{3}

% multicol parameters
% These lengths are set only within the two main columns
%\setlength{\columnseprule}{0.25pt}
\setlength{\premulticols}{1pt}
\setlength{\postmulticols}{1pt}
\setlength{\multicolsep}{1pt}
\setlength{\columnsep}{2pt}

\begin{center}
     \Large{\textbf{$\mathbb{MA}1505$ Cheat Sheet}}\\\small{taken in AY 2015-2016 Semester I} \\
\end{center}

\section{Functions}
\begin{tabular}{@{}ll@{}}
$(f \pm g)(x)=f(x)\pm g(x)$ & $\lim_{x\to a}(f\pm g)(x) = L \pm L'$ \\
$(fg)(x) = f(x)g(x)$  & $\lim_{x\to a}(fg)(x) = LL'$ \\
$(f/g)(x) = f(x)/g(x)$ & $\lim_{x\to a}\frac{f}{g}(x) = \frac{L}{L'}$ \\
 & $\lim_{x\to a} kf(x) = kL, k\in \mathbb{R}$\\
\end{tabular}
All polynomials are continuous at every point in $\mathbb{R}$.
All rational functions $\frac{p(x)}{q(x)}$ where $p$ and $q$ are polynomials are continuous at every point such that $q(x) \ne 0$.

Composition is given by $\circ$, e.g. $(f\circ g)(x) = f(g(x))$

\subsection{Differentiation}
\begin{tabular}{@{}ll@{}}
    Product Rule    & $(fg)'(x) = f'(x)g(x) + f(x)g'(x)$\\
    Quotient Rule   & $\Big(\frac{f}{g}\Big)'(x) = \frac{f'(x)g(x)-f(x)g'(x)}{g^2(x)}$ \\
    Chain Rule      & $(f\circ g)'(x) = f'(g(x))g'(x) = (f'\circ g)(x)g'(x)$ \\
                    & $\frac{dy}{dt} = \frac{dy}{dx}\times\frac{dx}{dt}$\\
\end{tabular}

\subsubsection{Maxima and Minima}
A function $f$ has a local/relative maximum value at a point $c$ in its domain if $f(x) \leq f(c)$ for all $x$ in the neighborhood of $c$.

The function has an absolute maximum value at $c$ if $f(x) \leq f(c)$ for all $x$ in the domain.

Reverse signs for minimum.

\subsubsection{Extreme and Critical Points}
Points where $f$ can have an extreme value are: interior points where $f'(x)=0$, interior points where $f'(x)$ does not exist and end points of the domain of $f$.

An interior point of the domain of a function $f$ where $f'$ is zero or does not exist is a \emph{critical point} of $f$.

\subsubsection{Increasing or Decreasing Functions}
$f$ is increasing on an interval $I$ when $f'(x) > 0$ for all $x\in I$.

$f$ is decreasing on an interval $I$ when $f'(x) < 0$ for all $x\in I$.

\subsubsection{Concavity}
The graph of $y=f(x)$ is concave down on any interval where $y''<0$ and concave up on any interval where $y''>0$.

A point $c$ is a point of inflection of the function $f$ if $f$ is continuous at $c$ and there is an open interval containing $c$ such that the graph of $f$ changes from concave up (or down) to concave down (or up).
The function need not be differentiable at $c$.

\subsubsection{Derivative Tests for Maxima and Minima}
\textbf{First derivative test:}
Suppose $c\in(a,b)$ is a critical point of $f$.

If $f'(x) > 0$ for $x \in (a,c)$ and $f'(x) < 0$ for $x\in (c,b)$ then $f(c)$ is a local maximum.

If $f'(x) < 0$ for $x \in (a,c)$ and $f'(x) > 0$ for $x\in (c,b)$ then $f(c)$ is a local minimum.

\textbf{Second derivative test:}
If $f'(c) = 0$ and $f''(c) < 0$ then $f$ has a local maximum at $x=c$.

If $f'(c) = 0$ and $f''(c) > 0$ then $f$ has a local minimum at $x=c$.

\subsection{Indeterminacy and L'Hopital's Rule}
If the functions $f$ and $g$ are continuous at $x=a$ but $f(a)=g(a)=0$, then the limit $\lim_{x\to a}\frac{f(x)}{g(x)}$ cannot be directly evaluated.

Suppose that $f$ and $g$ are differentiable in a neighborhood of $a$, $f(a) = g(a) = 0$ and $g'(x) \ne 0$ except possibly at $a$.
$$\lim_{x\to a} \frac{f(x)}{g(x)} = \lim_{x\to a} \frac{f'(x)}{g'(x)}$$

L'Hopital's rule can be applied multiple times in succession, e.g. if $\frac{f'(x)}{g'(x)}$ is still indeterminate.

\section{Integration}
If $f$ is continuous on $[a,b]$, then
$$F(x) = \int_a^x f(t) dt$$
$$\frac{d}{dx} F(x) = \frac{d}{dx}\int_a^x f(t) dt = f(x)$$

\subsection{Integration by Parts}
$$\int u(x)v'(x) dx = u(x)v(x) - \int u'(x)v(x) dx$$

Decide which function is $u(x)$ with the LIATE rule in this order of priority:

\resizebox{\linewidth}{!}{%
\begin{tabular}{|l@{ }c@{ }l@{ }c@{ }l@{ }c@{ }l@{ }c@{ }l|}
	\hline
	logarithmic & \multirow{2}{*}{$>$} & inverse trigo.& \multirow{2}{*}{$>$} & algebraic & \multirow{2}{*}{$>$} & trigo. 	& \multirow{2}{*}{$>$} & exponential \\
	functions 	& & functions 	& & functions	& & functions & & functions \\\hline
\end{tabular}}

\subsection{Area Bounded by Two Curves}
Area between two curves $f_2(x)$ and $f_1(x)$ where $f_1(x) \leq f_2(x)$ in $[a,b]$ is given by
$$\int_a^b f_2(x) - f_1(x) dx$$

\subsection{Partial Fraction Decomposition}
$$\frac{px+q}{(ax+b)(cx+d)} = \frac{A}{ax+b} + \frac{B}{cx+d}$$
$$\frac{px^2+qx+r}{(ax+b)(cx+d)^2} = \frac{A}{ax+b} + \frac{B}{cx+d} + \frac{C}{(cx+d)^2}$$
$$\frac{px^2+qx+r}{(ax+b)(x^2+c^2)} = \frac{A}{ax+b} + \frac{Bx+C}{x^2+c^2}$$

% \subsection{Integrals}
% \begin{tabular}{@{}rlr@{}}
%     $f(x)$              & $\int f(x) dx$                    & \\
%     $\frac{1}{x^2+a^2}$ & $\frac{1}{a}\arctan{\frac{x}{a}}$ & \\
% \end{tabular}

\section{Series}
\subsection{Arithmetic Series}
$$\sum_1^n a_n = \frac{n}{2}(a_1 + a_n)$$

\subsection{Geometric Series}
$$\sum_1^n ar^{n-1} = a\frac{1-r^n}{1-r}$$

If $|r| < 1$ then $ar^{n-1} \to 0$ as $n\to\infty$,
$$\sum_1^\infty ar^{n-1} = \frac{a}{1-r}$$

\subsection{Ratio Test}
For a series $\sum a_n$, let

$$\lim_{n\to\infty} \Big| \frac{a_{n+1}}{a_n}\Big| = \rho$$

Series is convergent if $\rho < 1$, divergent if $\rho > 1$ and no conclusion reached if $\rho = 1$.

\subsection{Power Series}
A power series has the form
$$\sum_{n=0}^\infty c_nx^n = c_0 + c_1x + c_2x^2 + ... + c_nx^n + ...$$

If the series is centered about $x=a$,
$$\sum_{n=0}^\infty c_n(x-a)^n = c_0 + c_1(x-a) + c_2(x-a)^2 + ... + c_n(x-a)^n + ...$$

\subsubsection{Standard Series}
\begin{align}
	(1+x)^r & = 1+rx+\frac{r(r-1)}{2!}x^2 + ... + \frac{r(r-1)...(r-n+1)}{n!}x^n \nonumber \\
	\frac{1}{1-x} & = 1 + x + x^2 + x^3 + ... + x^n + ... \text{for } |x| < 1 \nonumber \\
	e^x & = 1+x+\frac{x^2}{2!}+\frac{x^3}{3!} + ... + \frac{x^n}{n!} + ... \nonumber \\
	\sin{x} & = x - \frac{x^3}{3!} + \frac{x^5}{5!} - ... + \frac{(-1)^nx^{2n+1}}{(2n+1)!} + ... \nonumber \\
	\cos{x} & = 1-\frac{x^2}{2!} + \frac{x^4}{4!} - ... + \frac{(-1)^nx^{2n}}{(2r)!} \nonumber \\
	\ln{(1+x)} & = x -\frac{x^2}{2} + \frac{x^3}{3} - ... + \frac{(-1)^{n+1}x^n}{n} + ... \nonumber
\end{align}

\subsection{Radius of Convergence}
$$\lim_{n\to\infty} \Big| \frac{u_{n+1}}{u_n} \Big| < 1$$

\subsection{Taylor Series}
$$\sum_{k=0}^\infty \frac{f^{(k)}(a)}{k!}(x-a)^k = f(a) + f'(a)(x-a) + ... + \frac{f^{(n)}(a)}{n!}(x-a)^n + ... $$

A Maclaurin series is a special case of a Taylor series where $a=0$, i.e. is a Taylor series expansion of a function about $0$

\subsubsection{Taylor's Theorem}
The $n$th order Taylor polynomial of $f$ at $a$ is given by
$$P_n(x) = \sum_{k=0}^n \frac{f^{(k)}(a)}{k!}(x-a)^k = f(a) + f'(a)(x-a) + ... + \frac{f^{(n)}(a)}{n!}(x-a)^n$$

Then $f(x) = P_n(x) + R_n(x)$ where
$$R_n(x) = \frac{f^{(n+1)}(c)}{(n+1)!}(x-a)^{n+1}$$
for some $c$ between $a$ and $x$, where $R_n(x)$ is the remainder of order $n$ or the error term for the approximation of $f(x)$ by $P_n(x)$.

\section{Three Dimensional Space $\mathbb{R}^3$}
From now onwards, any math character in bold face is a vector, e.g. $\mathbf{i},\mathbf{u}$

\subsection{Dot Product}
$$\vec{v_1} \cdot \vec{v_2} =
    \begin{pmatrix}
        x_1 \\
        y_1 \\
        z_1 \\
    \end{pmatrix}\cdot
    \begin{pmatrix}
        x_2 \\
        y_2 \\
        z_2 \\
    \end{pmatrix}
    = x_1x_2 + y_1y_2 + z_1z_2
    = |\vec{v_1}|\: |\vec{v_2}| \cos{\theta} $$

\subsection{Unit Vector}
For some vector $\mathbf{u}$, its unit vector $\mathbf{\hat{u}} = \frac{1}{|\mathbf{u}|}\mathbf{u}$

\subsection{Cross Product}
$$\vec{v_1} \times \vec{v_2} =
    \begin{pmatrix}
        x_1 \\
        y_1 \\
        z_1 \\
    \end{pmatrix}
    \times
    \begin{pmatrix}
        x_2 \\
        y_2 \\
        z_2 \\
    \end{pmatrix}
    =\begin{pmatrix}
        y_1z_2 - z_1y_2\\
        -(x_1z_2 - z_1x_2)\\
        x_1y_2 - y_1x_2\\
    \end{pmatrix}
    = |\vec{v_1}|\: |\vec{v_2}| \sin{\theta}$$

The distance from a point $P(x_0,y_0,z_0)$ to a plane $\Pi:ax+by+cz=d$ is given by
$$\text{dist}(P,\Pi) =\frac{|ax_0+by_0+cz_0-d|}{\sqrt{a^2+b^2+c^2}}=\text{proj}_{\mathbf{n}}\vec{OP}$$

\subsection{Space Curves}
For some curve with the vector equation $\mathbf{r}(t) = f(t)\mathbf{i} + g(t)\mathbf{j} + h(t)\mathbf{k}$,

its arc length (if the curve is traversed once)
$$L = \int_a^b \sqrt{(f'(t))^2 + (g'(t))^2 + (h'(t))^2} dt = \int_a^b |\mathbf{r}'(t)|$$

\section{Fourier Series}
\subsection{Even or Odd?}
\begin{tabular}{rl}
	$-f(x) = f(-x)$ & function $f$ is odd \\
	$f(x) = f(-x)$	& function $f$ is even \\
\end{tabular}

\subsection{Computing the Fourier Series}
A periodic function of period $T$ can be represented by a Fourier series $f(x)$.
$$f(x) = a_0 + \sum_{n=1}^\infty (a_n\cos{nx}+b_n\sin{nx})$$

As the number of terms $n$ approaches infinity, the Fourier series begins to converge on the original function $f(x)$ more and more closely.
A perfect approximation of $f(x)$ can only occur when $n\to\infty$

Let $2L=T$.
$$a_0 = \frac{1}{2L} \int_{-L}^{L} f(x) dx$$

For the $m$th term where $m\in\mathbb{Z}^+$,
$$a_m = \frac{1}{L} \int_{-L}^{L} f(x) \cos{\frac{\pi mx}{L}} dx$$
$$b_m = \frac{1}{L} \int_{-L}^{L} f(x) \sin{\frac{\pi mx}{L}} dx$$

\textbf{If the function is even, we only need to consider cosine terms. Similarly, if the function is odd, we only need to consider sine terms.}

\subsection{Convergence of Fourier Series}
Suppose $f(x)$ is a piecewise smooth periodic function on the interval $-L\leq x\leq L$ with a Fourier series $f_s(x)$
$$f_s(x) = \left\{ \begin{array}{ll}
			f(x) & \text{if } f \text{ is continuous at } x \\
			\frac{1}{2}[f(x^-)+f(x^+)] & \text{if } f \text{ is not continuous at } x \\
			\end{array}\right\}$$

This is useful when evaluating the sum of the Fourier coefficients $a_0$, $a_n$ and $b_n$ for some $x=k,k\in\mathbb{R}$ in $f(x)$

\section{Multivariate Functions}
\subsection{Partial Derivatives}
Let $z = f(x,y)$ be a function of two variables.

The \emph{partial derivative} of a function $f(x,y)$ w.r.t. $x$ is denoted by $f_x(x,y)$ or $\frac{\partial f}{\partial x}$ where the $y$ term is taken as a constant.

$$f_{xx} = (f_x)_x = \frac{\partial^2f}{\partial x^2} \text{ and } f_{xy} = (f_x)_y = \frac{\partial^2f}{\partial x\partial y}$$
$$f_{yy} = (f_y)_y = \frac{\partial^2f}{\partial y^2} \text{ and } f_{yx} = (f_y)_x = \frac{\partial^2f}{\partial x\partial y}$$

For most functions in practice, $f_{xy}(a,b) = f_{yx}(a,b)$.

\subsection{Chained Derivatives}
Suppose $z=f(x,y,z)$ where $x=x(t)$, $y=y(t)$ and $z=z(t)$. Thus, $z = f(x(t),y(t),z(t))$.
$$ \frac{dz}{dt} = \frac{\partial f}{\partial x}\frac{dx}{dt} + \frac{\partial f}{\partial y}\frac{dy}{dt} +\frac{\partial f}{\partial z}\frac{dz}{dt}$$

Suppose $w=f(x,y,z)$ and $x=x(s,t)$, $y=y(s,t)$, and $z=z(s,t)$, giving $w=f(x(s,t),y(s,t),z(s,t))$
$$ \frac{\partial w}{\partial s} = \frac{\partial f}{\partial x}\frac{\partial x}{\partial s} + \frac{\partial f}{\partial y}\frac{\partial y}{\partial s} + \frac{\partial f}{\partial z}\frac{\partial z}{\partial s}$$

$$\frac{\partial w}{\partial t} = \frac{\partial f}{\partial x}\frac{\partial x}{\partial t} + \frac{\partial f}{\partial y}\frac{\partial y}{\partial t} + \frac{\partial f}{\partial z}\frac{\partial z}{\partial t}$$

\subsection{Directional Derivatives}
Note that $D_{\mathbf{i}} f(a,b) = f_x(a,b)$ and $D_{\mathbf{j}}f(a,b) = f_y(a,b)$ for the standard unit vectors of the $x$ and $y$ direction.

For some \underline{unit vector} $\mathbf{\hat{u}} = u_1\mathbf{i} + u_2\mathbf{j}$,
$$D_{\mathbf{u}}f(a,b) = f_x(a,b)\cdot u_1 + f_y(a,b)\cdot u_2 = \nabla f(a,b)\cdot \hat{\mathbf{u}}$$

The directional derivative $D_{\mathbf{u}}f(a,b)$ measures the change in the value $df$ of a function $f$ when moved a distance $dt$ from the point $(a,b)$ in the direction of the vector $\mathbf{u}$, where $df = D_{\mathbf{u}}f(a,b)\cdot dt$.

\subsection{Gradient Vector}
The gradient vector $\nabla f$ is given by
$$\nabla f = f_x\mathbf{i} + f_y\mathbf{j}$$
$$D_{\mathbf{u}}f(a,b) = \nabla f(a,b)\cdot \mathbf{u} = |\nabla f(a,b)|\cos{\theta}$$

The function $f$ increases most rapidly in the direction $\nabla f(a,b)$ and decreases most rapidly in the direction $-\nabla f(a,b)$.

\subsection{Maxima and Minima}
$f(x,y)$ has a local maximum at $(a,b)$ if $f(x,y) \leq f(a,b)$ for all points $(x,y)$ near $(a,b)$

$f(x,y)$ has a local minimum at $(a,b)$ if $f(x,y) \geq f(a,b)$ for all points $(x,y)$ near $(a,b)$

A function $f$ may have a local maximum or minimum at $(a,b)$ if: $f_x(a,b)=0$ and $f_y(a,b)=0$ \underline{or} $f_x(a,b)$ or $f_y(a,b)$ is not defined.
A point that satisfies either condition is known as a \emph{critical point}.

Suppose that $(a,b)$ is a critical point of $f(x,y)$. Let us define $D$ as
$$D(a,b) = f_{xx}(a,b)f_{yy}(a,b)-[f_{xy}(a,b)]^2$$

\begin{tabular}{@{}rl@{}}
    $D>0$ and $f_{xx}(a,b)>0$   & relative minimum at $(a,b)$\\
    $D>0$ and $f_{xx}(a,b)<0$   & relative maximum at $(a,b)$\\
    $D<0$                       & saddle point at $(a,b)$\\
    $D=0$                       & no conclusion reached\\
\end{tabular}

\subsubsection{Saddle Point}
At a point $(a,b)$ of $f$ where $f_x(a,b) = 0$ and $f_y(a,b) = 0$, the point $(a,b)$ is known as a \emph{saddle point} of $f$ if there are some directions along which $f$ has a local maximum at $(a,b)$ and some directions along $f$ which has a local minimum at $(a,b)$.

\subsection{Lagrange Multiplier}
Suppose a function $f(x,y)$ subject to the constraint $g(x,y)$.
$$F(x,y,\lambda) = f(x,y) - \lambda g(x,y)$$

Solve for $F_x = 0$, $F_y = 0$ and $F_{\lambda} = 0$ to solve for $\lambda$.

\section{Multiple Integrals}
For $R = R_1 \cup R_2$ where $R_1$ and $R_2$ do not overlap except maybe at their boundary,
$$\iint_R f(x,y) dA = \iint_{R_1}f(x,y) dA + \iint_{R_2}f(x,y) dA$$

Suppose a rectangular region $R$ in the $xy$-plane where $a\leq x\leq b$ and $c\leq y\leq d$, then
$$\iint_R f(x,y) dA = \int_c^d\int_a^b f(x,y)\:dx\:dy = \int_a^b\int_c^d f(x,y)\:dy\:dx$$

\subsection{Type A Regions}
Bottom and top boundaries are curves given by $y=g_1(x)$ and $y=g_2(x)$ respectively, while left and right boundaries are $x=a$ and $x=b$ respectively.
$$R: g_1(x) \leq y \leq g_2(x),\:a\leq x\leq b$$
$$\iint_R f(x,y)\:dA = \int_a^b\int_{g_1(x)}^{g_2(x)}f(x,y)\:dy\:dx$$

\subsection{Type B Regions}
Left and right boundaries are curves given by $x=h_1(y)$ and $x=h_2(y)$ and bottom and top boundaries are straight lines $y=c$ and $y=d$ respectively.
$$R: c\leq y\leq d,\: h_1(y)\leq x\leq h_2(y)$$
$$\iint_R f(x,y)\:dA = \int_c^d\int_{h_1(y)}^{h_2(y)}f(x,y)\:dx\:dy$$

\subsection{Polar Coordinates}
Circular regions/sectors can be described with polar coordinates $r$ and $\theta$.

In general, a region $R$ in polar coordinates is described by
$$R: a\leq r\leq b,\: \alpha\leq\theta\leq\beta$$

When transforming from Cartesian to polar coordinates, $(x,y)$ is transformed to $(r,\theta)$ where
$$x=r\cos{\theta} \text{ and } y=r\sin{\theta}$$
and $dA$ is changed from $dx\:dy$ to $r\:dr\:d\theta$.

\subsection{Application of Double Integrals}
Suppose $D$ is a solid region under a surface defined by $f(x,y)$ over a plane region $R$.
$$\text{Volume of $D$} = \iint_Rf(x,y)\:dA$$

If $f$ has continuous first partial derivatives on a closed region $R$ of the $xy$-plane, then the area $S$ of that portion of the surface $z=f(x,y)$ that projects onto $R$ is given by
$$S = \iint_R\sqrt{\Big( \frac{\partial z}{\partial x}\Big)^2 + \Big( \frac{\partial z}{\partial y} \Big)^2 + 1}\:dA$$

\section{Line Integrals}
\subsection{Vector Fields}
A vector field on $R$ is a vector function $\mathbf{F}$ that assigns to each point a vector $\mathbf{F}(x,y,z)$.
$$\mathbf{F}(x,y,z) = P(x,y,z)\mathbf{i} + Q(x,y,z)\mathbf{j} + R(x,y,z)\mathbf{k}$$

\subsection{Gradient Fields}
$$\nabla f(x,y,z) = f_x(x,y,z)\mathbf{i}+f_y(x,y,z)\mathbf{j} + f_z(x,y,z)\mathbf{k}$$

\subsection{Conservative Fields}
A vector field $\mathbf{F}$ is called a conservative vector field if it is the gradient of some scalar function $f$ such that $\mathbf{F} = \nabla f$, where $f$ is known as  the potential function for $\mathbf{F}$.

Let $\mathbf{F}(x,y) = P(x,y)\mathbf{i} + Q(x,y)\mathbf{j}$ be a vector field on the $xy$-plane.
$$\text{If }\frac{\partial P}{\partial y} = \frac{\partial Q}{\partial x}\text{ then $\mathbf{F}$ is conservative.}$$

Let $\mathbf{F}(x,y,z) = P(x,y,z)\mathbf{i} + Q(x,y,z)\mathbf{j} + R(x,y,z)\mathbf{k}$ be a vector field on $xyz$-space.
$$\text{If } \frac{\partial P}{\partial y} = \frac{\partial Q}{\partial x}, \frac{\partial P}{\partial z} = \frac{\partial R}{\partial x}, \frac{\partial Q}{\partial z} = \frac{\partial R}{\partial y}, \text{ then $\mathbf{F}$ is conservative.} $$

If $\mathbf{F}$ is a conservative vector field, then $\int_C \mathbf{F}\cdot d\mathbf{r}$ is independent of the path taken.

If $\mathbf{F}$ is a conservative vector field, then $\oint_l \mathbf{F}\cdot d\mathbf{r} = 0$ for any \underline{closed} curve $l$, i.e. a curve with a terminal point that coincides with its initial point.

\subsection{Line Integrals of Scalar Functions}
$$\int_Cf(x,y,z)\:ds = \int_a^b f(x(t),y(t),z(t)) ||\mathbf{r}'(t)|| dt$$
$$=\int_a^b f(x(t),y(t),z(t))\sqrt{\Big(\frac{dx}{dt}\Big)^2 + \Big(\frac{dy}{dt}\Big)^2 + \Big(\frac{dz}{dt}\Big)^2}\: dt$$

\subsection{Line Integrals of Vector Fields}
$$\int_C\mathbf{F}\cdot\: d\mathbf{r} = \int_a^b \mathbf{F}(\mathbf{r}(t))\cdot \mathbf{r}'(t) dt$$

Geometrically, the line integral of $\mathbf{F}$ over $C$ is summing up the tangential components of $\mathbf{F}$ with respect to the arc length of $C$.

$$\int_{-C} f(x,y,z) ds = \int_C f(x,y,z) ds$$
The vector equation of a curve $C$ determines the orientation or direction of $C$.

For some $\mathbf{F}(x,y,z) = P(x,y)\mathbf{i} + Q(x,y)\mathbf{j} + R(x,y)\mathbf{k}$,
$$ \int_C \mathbf{F}\cdot d\mathbf{r} = \int_C P dx + Q dy + R dz = \int_a^b P(\mathbf{r}(t))\frac{dx}{dt} + Q(\mathbf{r}(t))\frac{dy}{dt} dt$$

\subsection{Fundamental Theorem for Line Integrals}
If $f$ is a function of 2 or 3 variables whose gradient $\nabla f$ is continuous,
$$\int_C \nabla f \cdot d\mathbf{r} = f(\mathbf{r}(b))-f(\mathbf{r}(a))$$

\subsection{Green's Theorem}
Let $D$ be a bounded region in the $xy$-plane and $\partial D$ the boundary of $D$. Suppose $P(x,y)$ and $Q(x,y)$ has continuous partial derivatives on $D$. Thus,
$$\oint_{\partial D} P dx + Q dy = \iint_D \Big( \frac{\partial Q}{\partial x} - \frac{\partial P}{\partial y} \Big) dA$$

The orientation of $\partial D$ is such that, as one traverses along the boundary in this direction, the region $D$ is always on the left-hand side, i.e. the positive orientation of the boundary.

\section{Surface Integrals}
A parametric representation of a surface is given by
$$\mathbf{r}(u,v) = x(u,v)\mathbf{i} + y(u,v)\mathbf{j} + z(u,v)\mathbf{k}$$

\subsection{Standard Parametric Representation: Sphere}
For a sphere of radius $a$: $x^2+y^2+z^2=a^2$
$$\mathbf{r}(u,v) = (a\sin{u}\cos{v})\mathbf{i}+(a\sin{u}\sin{v})\mathbf{j} + (a\cos{u})\mathbf{k} $$

When $0\leq u\leq\pi$ and $0\leq v\leq 2\pi$, the representation gives a full sphere.

When $0\leq u\leq\frac{\pi}{2}$ and $0\leq v\leq 2\pi$, the representation gives the upper hemisphere.

\subsection{Standard Parametric Representation: Cylinder}
For a circular cylinder of radius $a$: $x^2+y^2=a^2$
$$\mathbf{r}(u,v) = (a\cos{u})\mathbf{i} + (a\sin{u})\mathbf{j} + v\mathbf{k}$$

Here, $u$ measures the angle from the positive $x$-axis about the $z$-axis while $v$ measures the height from the $xy$-plane along the cylinder.

The same applies for $x^2+z^2=a^2$ (cylinder about $y$-axis)
$$\mathbf{r}(u,v) = (a\cos{u})\mathbf{i} + v\mathbf{j} + (a\sin{u})\mathbf{k}$$

and $y^2+z^2=a^2$ (cylinder about $x$-axis)
$$\mathbf{r}(u,v) = v\mathbf{i} + (a\cos{u})\mathbf{j} + (a\sin{u})\mathbf{k}$$

\subsection{Tangent Planes}
Let $S$ be a surface given by the parametric representation
$$\mathbf{r}(u,v) = x(u,v)\mathbf{i} + y(u,v)\mathbf{j} + z(u,v)\mathbf{k}$$

For some position vector $\mathbf{r}_0 = \mathbf{r}(u_0,v_0)$ at a point $P_0$,

Fixing $v=v_0$ for a resulting curve $C_1$, the tangent vector of the space curve $C_1$ is given by
$$\mathbf{r}_u = \frac{\partial x}{\partial u}(u_0,v_0)\mathbf{i} + \frac{\partial y}{\partial u}(u_0,v_0)\mathbf{j} + \frac{\partial z}{\partial u}(u_0,v_0)\mathbf{k}$$

Fixing $u=u_0$ for a resulting curve $C_2$, the tangent vector of the space curve $C_2$ is given by
$$\mathbf{r}_v = \frac{\partial x}{\partial v}(u_0,v_0)\mathbf{i} + \frac{\partial y}{\partial v}(u_0,v_0)\mathbf{j} + \frac{\partial z}{\partial v}(u_0,v_0)\mathbf{k}$$

Both vectors $\mathbf{r}_u$ and $\mathbf{r}_v$ lie in the tangent plane to $S$ at $P_0$.
Thus, the cross product $\mathbf{r}_u \times \mathbf{r}_v$, assuming it is non-zero, provides a normal vector to the tangent plane to $S$ at $P_0$.

The equation of the tangent plane is described by
$$(\mathbf{r} - \mathbf{r}_0)\cdot(\mathbf{r}_u\times\mathbf{r}_v) = 0$$

\subsection{Surface Integrals of Scalar Functions}
Suppose $f(x,y,z)$ be a function defined on a surface $S$, where we can find $\mathbf{r}(u,v)=x(u,v)\mathbf{i}+y(u,v)\mathbf{j}+z(u,k)\mathbf{k}$ of $S$ over a domain $D$.
$$\iint_S f(x,y,z)\:dS = \iint_D f(\mathbf{r}(u,v))|\mathbf{r}_u\times\mathbf{r}_v|\:dA$$

\subsection{Surface Integrals of Vector Fields}
Let $\mathbf{F}$ be a continuous vector fields defined on a surface $S$ with a unit normal vector $\mathbf{n}$.
The surface integral of $\mathbf{F}$ over $S$ is given as
$$\iint_S \mathbf{F}\cdot\mathbf{n}\:dS \text{ or more simply } \iint_S \mathbf{F}\cdot\: dS$$

This integral is also known as the flux of $\mathbf{F}$ over $S$.

If $S$ is given by a parametric representation $\mathbf{r} = \mathbf{r}(u,v)$ with domain $D$,
$$\iint_S\mathbf{F}\cdot\:dS = \iint_D \mathbf{F}(\mathbf{r}(u,v))\cdot(\mathbf{r}_u\times\mathbf{r}_v)\:dA$$

\subsection{Orientation of Surfaces}
By convention, a curve has a positive orientation if it progresses counter-clockwise.

$$\iint_{-S}\mathbf{F}\cdot\:dS = -\iint_S \mathbf{F}\cdot\:dS$$

You can use the right-hand grip rule with the thumb pointed in the direction of the normal vector of $C$.
The curve has a positive orientation if your grip from knucle to fingertips progresses counter-clockwise, or negative if otherwise.

\subsection{Curl}
Let $\mathbf{F}=P\mathbf{i}+Q\mathbf{j}+R\mathbf{k}$ be a vector field in $xyz$-space.
$$\text{curl }{\mathbf{F}} = \Big(\frac{\partial R}{\partial y}-\frac{\partial Q}{\partial z}\Big)\mathbf{i} + \Big(\frac{\partial P}{\partial z}-\frac{\partial R}{\partial x}\Big)\mathbf{j} + \Big(\frac{\partial Q}{\partial x}-\frac{\partial P}{\partial y}\Big)\mathbf{k} $$

The curl of a vector field is also itself a vector field.

\subsection{Divergence}
$$\text{div }\mathbf{F} = \frac{\partial P}{\partial x} + \frac{\partial Q}{\partial y}+\frac{\partial R}{\partial z}$$

The divergence of a vector field is a scalar function.

\subsection{Del Operator}
$$\nabla = \frac{\partial}{\partial x}\mathbf{i} + \frac{\partial}{\partial y}\mathbf{j} + \frac{\partial}{\partial z}\mathbf{k} $$

The curl and divergence operations can be expressed in terms of the del operator using cross and dot product.
$$\text{curl }\mathbf{F} = \nabla\times\mathbf{F} \text{ and } \text{div }\mathbf{F} = \nabla\cdot\mathbf{F}$$

\subsection{Conservative Fields*}
Let $\mathbf{F}$ be a vector field in $xyz$-space.

If $\text{curl }\mathbf{F} = \mathbf{0}$, then $\mathbf{F}$ is a conservative field.
The converse is also true.

\subsection{Stokes' Theorem}
Let $S$ be an oriented, piecewise-smooth surface that is bounded by a closed, piecewise-smooth boundary curve $C$.
Let $\mathbf{F}$ be a vector field whose components have continuous partial derivatives on $S$. Then,
$$\oint_C\mathbf{F}\cdot\:d\mathbf{r} = \iint_S (\text{curl }\mathbf{F})\cdot\:dS$$

Stokes' Theorem can also be expressed as
$$ \oint_C P\:dx+Q\:dy+R\:dz =  $$
$$ \iint_S \Big( \frac{\partial R}{\partial y} - \frac{\partial Q}{\partial z}\Big)\,dy\,dz + \Big( \frac{\partial P}{\partial z} - \frac{\partial R}{\partial x}\Big)\,dz\,dx + \Big( \frac{\partial Q}{\partial x}-\frac{\partial P}{\partial y}\Big)\,dx\,dy $$

The orientation of $C$ must be consistent with that of $S$.

\subsection{Gauss' Theorem}
Let $E$ be a solid region, $S$ be the boundary of $E$ given with outward orientation (where the normal vector on the surface always points away from $E$).
Let $\mathbf{F}$ be a vector field whose component functions have continuous partial derivatives in $E$. Then,
$$\iint_S \mathbf{F}\cdot\:d\mathbf{S} = \iiint_E \text{div }\mathbf{F}\:dV$$

\section{Trigonometric Identities}
\subsection{Pythagorean Identities}
$$\sin^2{u} + \cos^2{u} = 1$$
$$1+\tan^2{u} = \sec^2{u}$$
$$1+\cot^2{u} = \csc^2{u}$$

\subsection{Sum and Difference Formulae}
$$\sin(u\pm v) = \sin{u}\cos{v} \pm \cos{u}\sin{v}$$
$$\cos(u\pm v) = \cos{u}\cos{v} \mp \sin{u}\sin{v}$$
$$\tan(u\pm v) = \frac{\tan{u}\pm\tan{v}}{1\mp\tan{u}\tan{v}}$$

\subsection{Double Angle Formulae}
$$\sin{2u} = 2\sin{u}\cos{u}$$
$$\cos{2u} = \cos^2{u} - \sin^2{u} = 2\cos^2{u} - 1 = 1-2\sin^2{u}$$
$$\tan{2u} = \frac{2\tan{u}}{1-\tan^2{u}}$$

\subsection{Half Angle Identities}
$$\sin^2{u} = \frac{1-\cos{2u}}{2}$$
$$\cos^2{u} = \frac{1+\cos{2u}}{2}$$
$$\tan^2{u} = \frac{1-\cos{2u}}{1+\cos{2u}}$$

\subsection{Sum $\to$ Product Identities}
$$\sin{u} + \sin{v} = 2\sin{\frac{u+v}{2}}\cos{\frac{u-v}{2}}$$
$$\sin{u} - \sin{v} = 2\cos{\frac{u+v}{2}}\sin{\frac{u-v}{2}}$$
$$\cos{u} + \cos{v} = 2\cos{\frac{u+v}{2}}\cos{\frac{u-v}{2}}$$
$$\cos{u} - \cos{v} = -2\sin{\frac{u+v}{2}}\sin{\frac{u-v}{2}}$$

\subsection{Product $\to$ Sum Identities}
$$\sin{u}\sin{v} = \frac{1}{2} [\cos{(u-v)} - \cos{(u+v)}]$$
$$\cos{u}\cos{v} = \frac{1}{2} [\cos{(u-v)} + \cos{(u+v)}]$$
$$\sin{u}\cos{v} = \frac{1}{2} [\sin{(u+v)} + \sin{(u-v)}]$$
$$\cos{u}\sin{v} = \frac{1}{2} [\sin{(u+v)} - \sin{(u-v)}]$$

\subsection{Parity Identities}
$$\sin{(-u)} = -\sin{u}$$
$$\cos{(-u)} = \cos{u}$$
$$\tan{(-u)} = -\tan{u}$$

$$\cot{(-u)} = -\cot{u}$$
$$\csc{(-u)} = -\csc{u}$$
$$\sec{(-u)} = \sec{u}$$

{\hfill \texttt{darren.wee@u.nus.edu}}

{\hfill \texttt{For MA1505 taken in AY2015/16 Sem. 1 \\\hfill \LaTeX{} compiled \today.}}
\end{multicols}
\end{document}


\documentclass[letterpaper,10pt]{article}
\usepackage[margin=2cm]{geometry}

\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage[colorlinks]{hyperref}

\newcommand{\panhline}{\begin{center}\rule{\textwidth}{1pt}\end{center}}

\title{\textbf{Logistic Regression}}
\author{Aarti Singh (Instructor), HMW-Alexander Noter)()}

\begin{document}

\maketitle

\panhline
\href{../index.html}{Back to Index}

\panhline
\tableofcontents

\section*{Resources}

\begin{itemize}
	\item \href{../../Lectures/05_LinearLogisticRegression.pdf}{Lecture}
\end{itemize}

\panhline

\begin{itemize}
	\item Output need to be [0,1]
	\item It is a linear classifier, not a real regression
\end{itemize}

Sigmoid function
$$P(Y=0|X)=\frac{1}{1+\exp(w_0+\sum{w_iX_i})}$$

Decision boundar:
\begin{itemize}
	\item Class 0: $P(Y=0|X)>P(Y=1|X)$, $\sum{w_iX_i}<0$
	\item Class 1: $P(Y=0|X)<P(Y=1|X)$, $\sum{w_iX_i}>0$
\end{itemize}

\section{Training Logistic Regression}

How to learn the parameters $w_0,\dots,w_d$ from training data $\{(X^{(i)},Y^{(i)}\}$

\subsection{Maximum Conditional Likelihood Estimate}
$$\hat{\vec{w}}_{MLE} = \arg\max_{\vec{w}} \prod_{i=1}^{n}P(X^{(i)},Y^{(i)}|\vec{w})$$

But we only know the model of $P(Y|X)$

Discriminative philosophy: Don't waste effort learning $P(X)$, focus on $P(Y|X)$ -- that's all that matters for classification!

\begin{equation}
\left.
\begin{array}{rcl}
P(Y=0|X) & = & \frac{1}{1+\exp(w_0+\sum{w_iX_i})} \\
P(Y=1|X) & = & \frac{\exp(w_0+\sum{w_iX_i})}{1+\exp(\sum{w_0+w_iX_i})} \\
\end{array}
\right\} \frac{\exp(w_0y_i+\sum{w_iX_iy_i})}{1+\exp(w_0+\sum{w_iX_i})}
\end{equation}

$$l(w) = ln\prod_i P(y^i|x^i,w) = \sum_i[y^i(w_0+\sum w_jx_j^i) - \ln (1+\exp(w_0+\sum w_jx_j^i))]$$

No close-form solution to maximize $w$, but $l(w)$ is concave function of $w$.

Gradient Ascent Algorithm:

\begin{itemize}
	\item Initialize: Pick $w$ at random
	\item Gradient: $\nabla_wl(w)= [\frac{\partial l(w)}{\partial w_0},\dots,\frac{\partial l(w)}{\partial w_d}]^T$
	\item Update rule: $\Delta w = \eta \nabla_wl(w)$, and $w_i^{t+1}\leftarrow w_i^t+\Delta w$
\end{itemize}

$$\frac{\partial l(w)}{\partial w_0}=\sum_i[y^i-P(Y^i=1|x^i,w)]$$
$$\frac{\partial l(w)}{\partial w_j}=\sum_ix^i_j[y^i-P(Y^i=1|x^i,w)]$$

\subsection{Maximum Conditional A Priori Estimate}

Define priors on $w$

$$p(w)= \prod_i \frac{1}{\kappa \sqrt{2\pi}}\exp{-\frac{w_i^2}{2\kappa^2}}$$

$$l(w) = lnP(w)\prod_i P(y^i|x^i,w) = \sum_i[y^i(w_0+\sum w_jx_j^i) - \ln (1+\exp(w_0+\sum w_jx_j^i))]$$

\section{Logistric Regression for More Than 2 classes}

for $k<K$
$$P(Y=y_k|X)=\frac{\exp(w_{k0}+\sum{w_{ki}X_i})}{1+\sum_{j=1}^{K-1}\exp(w_{j0}+\sum{w_{ji}X_i})}$$
for $k=K$
$$P(Y=y_K|X)=\frac{1}{1+\sum_{j=1}^{K-1}\exp(w_{j0}+\sum{w_{ji}X_i})}$$

The decision boundaries are still linear and they will intersect at one point. (Why?)

\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS630: Database Management Systems
% Copyright 2014 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 3}

Using the schema above, and assuming that grade is of type integer, provide the SQL statement to create a view \texttt{TopStudents} that lists the student ID, name and average grade (GPA) for students that have GPA above 3.0.

\subsection*{Solution}

\lstset{language=sql}
\lstinputlisting[firstline=8]{
	\topDirectory/src/sql/f01/f01q03.sql
}


% Third Golden Rule of Bioinformatics Exercise
%
% The aim of this exercise is to demonstrate that a simple question
% of biological relevance does not have the obvious answer. The point
% is to reinforce that even "simple" questions may require sufficient
% thought and attention, in order to answer them properly. Also, that
% "intuition" is not always sufficient for even "simple" questions.

\subsection{Golden Rule 3 Exercise}
\begin{frame}
  \frametitle{Exercise 3}
  \framesubtitle{Classification}
  \begin{itemize}
    \item Rule: If there is a vowel on one side of a card, there \textit{must} be an even number on the other side of that card.
    \item Is this rule true?
    \item Which cards, when turned over, can help determine if this rule is true? (How many are there?)
  \end{itemize}
  \includegraphics[width=0.8\textwidth]{images/wason}
\end{frame}

\subsection{Golden Rule 3 Exercise}
\begin{frame}
  \frametitle{Exercise 3}
  \framesubtitle{Classification}
  \includegraphics[width=0.8\textwidth]{images/wason}
\end{frame}

\begin{frame}
  \frametitle{Exercise 3}
  \framesubtitle{Classification}
  This is the Wason Selection Task
  \begin{itemize}
    \item<1-> If you chose \emph{E} and \emph{4}
    \begin{itemize}
      \item<2-> You are in the typical majority group
      \item<2-> You are not correct
      \item<2-> You have been a victim of confirmation bias (System 1 thinking)
    \end{itemize}
    \item<3-> If you chose \emph{E} and \emph{7}
    \begin{itemize}
      \item<4-> Congratulations!
      \item<4-> Your choice was capable of \textit{falsifying} the rule.
    \end{itemize}
  \end{itemize}
\end{frame}

\begin{frame}
  \frametitle{Exercise 3}
  \framesubtitle{Classification}
  Rule: If there is a vowel on one side of the card, there \textit{must} be an even number on the other side.
  \begin{center}
  \begin{tabular}{c|c|c}
	  Card & Outcome & Rule \\
	  \hline
	  \hline
	    \multirow{2}{*}{E} & Even & Can be true even if rule false \\
	                                & Odd & \emph{violated} \\
	  \hline
	    \multirow{2}{*}{K} & Even & na \\
	                                & Odd & na \\
	  \hline
	    \multirow{2}{*}{4} & Vowel & Can be true even if rule false \\
	                                & Consonant & na \\
	  \hline
	    \multirow{2}{*}{7} & Vowel & \emph{violated} \\
	                                & Consonant & na \\
  \end{tabular}
  \end{center}
\end{frame}

\begin{frame}
  \frametitle{Exercise 3}
  \framesubtitle{Classification}
  \begin{itemize}
    \item This is equivalent to functional classification, e.g:
    \item Rule: If there is a CRN/RxLR/T3SS domain, the protein \textit{must} be an effector.
  \end{itemize}
  \includegraphics[width=\textwidth]{images/wason_rxlr}
\end{frame}

\begin{frame}
  \frametitle{Exercise 3}
  \framesubtitle{Classification}
  \begin{itemize}
    \item Confirmation Bias (Wason Selection Task)
    \begin{itemize}
      \item An uninformative experiment is performed
      \item \url{http://en.wikipedia.org/wiki/Wason_selection_task}
    \end{itemize}
    \item Affirming the Consequent (a related formal fallacy)
    \begin{enumerate}
     \item If $P$, then $Q$
     \item $Q$
     \item Therefore, $P$
    \end{enumerate}
    \begin{itemize}
      \item Experimental results are misinterpreted
      \item \url{http://en.wikipedia.org/wiki/Affirming_the_consequent}
    \end{itemize}
  \end{itemize}
\end{frame}


\documentclass[notoc,notitlepage]{tufte-book}
% \nonstopmode % uncomment to enable nonstopmode

\usepackage{classnotetitle}

\title{STAT333 --- Applied Probability}
\author{Johnson Ng}
\subtitle{Classnotes for Winter 2017}
\credentials{BMath (Hons), Pure Mathematics major, Actuarial Science Minor}
\institution{University of Waterloo}

\input{latex-classnotes-preamble.tex}
\input{probnotation.tex}

\begin{document}
\input{latex-classnotes-header.tex}

\chapter*{Foreword}%
\label{chp:foreword}
% chapter foreword

I am transcribing this set of notes from my handwritten ones in Winter 2017,
back at a time which I have yet to organize my notes by lecture. However, I will
try my best to organize them by chapters and topics as presented in class.

I will try to be as rigourous as possible while transcribing my notes. However,
given the nature of the course and the presentation, this will not always be
possible, and I am mostly keeping these notes for ``legacy purposes'', and so I
will not put too much effort into making the notes as complete as my newer
ones.

For this course, you are expected to have basic knowledge of probability in
order to be able to understand the material. You may want to have my
\href{https://tex.japorized.ink/STAT330S18/classnotes-white.pdf}{STAT330} notes
ready and/or reviewed.

% chapter foreword (end)

\chapter{Elementary Probability Review}%
\label{chp:elementary_probability_review}
% chapter elementary_probability_review

\section{Introductions}%
\label{sec:introductions}
% section introductions

\begin{defn}[Fundamental Definition of a Probability Function]
\label{defn:fundamental_definition_of_a_probability_function}
  For each event $A$ of a sample space $S$, $P(A)$ is defined as the ``\hlnoteb{probability of the event $A$}'', satisfying these 3 conditions:
  \begin{enumerate}
    \item $0 \leq P(A) \leq 1$
    \item $P(S) = 1$ \sidenote{This can also be stated as $P(\emptyset) = 0$, where $\emptyset$ is the null event.} \label{item:funddefn_probfn_item2}
    \item $P\left( \bigcup\limits_{i = 1}^{n} A_i \right) = \sum\limits_{i=1}^{n} P(A_i)$, where $A_i \cap A_j = A_i A_j = \emptyset$ for all $i \neq j$ \sidenote{We can also say that the sequence $\{A_i\}_{i = 1}^{n}$ has mutually exclusive elements.} \label{item:funddefn_probfn_item3}
  \end{enumerate}
\end{defn}

\begin{note}
  By \cref{item:funddefn_probfn_item2} and \cref{item:funddefn_probfn_item3}, we
  have
  \begin{equation*}
    1 = P(S) = P(A \cup A^C) = P(A) + P(A^C)
  \end{equation*}
  which implies that
  \begin{equation*}
    P(A^C) = 1 - P(A).
  \end{equation*}
\end{note}

\begin{defn}[Conditional Probability]\index{Conditional Probability}\label{defn:conditional_probability}
  Given events $A$ and $B$ in a sample space $S$, the \hlnoteb{conditional
  probability of $A$ given $B$} is given by
  \begin{equation}\label{eq:conditional_probability}
    P(A \mid B) = \frac{P(A \cap B)}{P(B)} \enspace \text{ where } P(B) > 0.
  \end{equation}
\end{defn}

\begin{note}
  When $B = S$, \cref{eq:conditional_probability} becomes
  \begin{equation*}
    P(A \mid S) = \frac{P(A \cap S)}{P(S)} = \frac{P(A)}{1} = P(A).
  \end{equation*}
  Also, we have, from \cref{eq:conditional_probability}, that
  \begin{equation*}
    P(A \cap B) = P(A \mid B) \cdot P(B).
  \end{equation*}
\end{note}

\begin{thm}[Law of Total Probability]\index{Law of Total Probability}\label{thm:law_of_total_probability}
  Let $S$ be a sample space. Let $\{ B_i \}_{i = 1}^{n}$ be a sequence of
  mutually exclusive events such that
  \begin{equation*}
    S = \bigcup_{i=1}^{n} B_i.
  \end{equation*}
  We say that the sequence $\{ B_i \}_{i = 1}^{n}$ is a \hldefn{partition} of $S$. Let $A \subseteq S$ be an event. Then
  \begin{equation*}
    P(A) = \sum_{i=1}^{n} P(A \mid B_i) \cdot P(B_i)
  \end{equation*}
\end{thm}

\begin{proof}
  Observe that
  \begin{align*}
    P(A) &= P(A \cap S) = P\left( A \cap \left\{ \bigcup_{i=1}^{n} B_i  \right\} \right) \\
         &= P\left( \bigcup_{i=1}^{n} \left\{ A \cap B_i \right\} \right) = \sum_{i=1}^{n} P(A \cap B_i) \\
         &= \sum_{i=1}^{n} P(A \mid B_i) P(B_i)
  \end{align*}
  where the second last step is by \cref{item:funddefn_probfn_item3}, and the
  last step is by \cref{defn:conditional_probability}.
\end{proof}

Consequently, we have the following:

\begin{crly}[Bayes' Formula/Rule]
  \index{Bayes' Formula}\index{Bayes' Rule}
\label{crly:bayes_formula}
  Let $\{ B_i \}_{i = 1}^{n}$ be a partition of a sample space $S$. Then for any
  event $A$, we have
  \begin{equation*}
    P(B_j \mid A) = \frac{P(A \mid B_j) P(B_j)}{\sum_{i=1}^{n} P(A \mid B_i) \cdot P(B_i)}.
  \end{equation*}
\end{crly}

% section introductions (end)

\section{Random Variables}%
\label{sec:random_variables}
% section random_variables

\subsection{Discrete Random Variables}%
\label{sub:discrete_random_variables}
% subsection discrete_random_variables

\hlwarn{No formal definition of a discrete rv is given in class.}

A discrete rv $X$:
\begin{itemize}
  \item takes on either finite or countable number of possible values;
  \item has a \hldefn{probability mass function} (pmf) expressed as
    \begin{equation*}
      p(a) = P(X = a);
    \end{equation*}
  \item has a \hldefn{cumulative distribution function} (cdf) expressed as
    \begin{equation*}
      F(a) = P(X \leq a) = \sum_{x \leq a}^{p(x)}
    \end{equation*}
\end{itemize}

\begin{note}
  If $X \in \{a_1, a_2, ...\}$ where $a_1 < a_2 < \hdots$ such that $p(a_i) > 0$ for all $i \in \mathbb{N}$, then
  \begin{align*}
    p(a_1) &= F(a_1) \text{ and } \\
    p(a_i) &= F(a_i) - F(a_{i - 1}) \text{ for } i = 2, 3, 4, ....
  \end{align*}
\end{note}

\newthought{The following} are some of the most common discrete distributions.

\paragraph{Binomial Distribution}\index{Binomial Distribution} For an rv $X$
that follows a Binomial Distribution, in which we denote as $X \sim \Bin(n, p)$,
where $n \in \mathbb{N}$ and $p \in [0, 1]$, its pmf is
\begin{equation*}
  p(x) = \binom{n}{x} p^x (1 - p)^{n - x}.
\end{equation*}

\paragraph{Bernoulli Distribution}\index{Bernoulli Distribution} Following the
above distribution where $n = 1$, we have that $X$ follows what is called a
Bernoulli Distribution, denoted as $X \sim \Bernoulli(p)$.

\paragraph{Negative Binomial Distribution}\index{Negative Binomial Distribution}
For an rv $X$ that follows a Negative Binomial Distribution, in which we denote
as $X \sim \NB(k, p)$, where $k \in \mathbb{N}$ and $p \in [0, 1]$, its pmf
is\marginnote{The Negative Binomial Distribution has a model that measures the
probability that the $k$th success occurs.}
\begin{equation*}
  p(x) = \binom{x - 1}{k - 1} p^k (1 - p)^{x - k}
\end{equation*}

\paragraph{Geometric Distribution}\index{Geometric Distribution} Following the
above distribution where $k = 1$, we have that $X$ folows what is called a
Geometric Distribution, denoted as $X \sim \Geo(p)$.

\paragraph{Hypergeometric Distribution}\index{Hypergeometric Distribution} For
an rv $X$ that follows a Hypergeometric Distribution, in which we denote as $X
\sim \HG(N, r n)$, where $r, n \leq N \in \mathbb{N}$, its pmf is
\begin{equation*}
  p(x) = \frac{\binom{r}{x} \binom{N - r}{n - x}}{\binom{N}{n}}
\end{equation*}

\paragraph{Poisson Distribution}\index{Poisson Distribution} For an rv $X$ that
follows a Poisson Distribution, in which we denote as $X \sim \Poi(\lambda)$,
where $\lambda > 0$, its pmf is
\begin{equation*}
  p(x) = \frac{e^{- \lambda} \lambda^{x}}{x!}
\end{equation*}

% subsection discrete_random_variables (end)

\subsection{Continuous Random Variables}%
\label{sub:continuous_random_variables}
% subsection continuous_random_variables

\hlwarn{No formal definition of a continuous rv is given in class.}

A continuous rv $X$:
\begin{itemize}
  \item takes on a continuum of possible values
  \item has a \hldefn{probability density function} (pdf) expressed as
    \begin{equation*}
      f(x) = \frac{d}{dx} F(x)
    \end{equation*}
    where $F(x)$ is:
  \item (has a) \hldefn{cumulative distribution function} (cdf) of
    \begin{equation*}
      F(x) = P(X \leq x) = \int_{-\infty}^{x} f(y) \dif{y}
    \end{equation*}
\end{itemize}

\begin{note}
  Note that our convention is that $P(X = x) = 0$ for a continuous rv $X$.
\end{note}

\newthought{The following} are some of the most common continuous
distributions.

\paragraph{Uniform Distribution}\index{Uniform Distribution} For an rv $X$ that
follows a Uniform Distribution, in which we denote as $X \sim \Unif(a, b)$,
where $a, b \in \mathbb{R}$, its pdf is
\begin{equation*}
  f(x) = \frac{1}{b - a}.
\end{equation*}

\paragraph{Gamma Distribution}\index{Gamma Distribution} For an rv $X$ that
follows a Gamma Distribution, in which we denote as $X \sim \Gam(n, \lambda)$,
where $n \in \mathbb{N}$ and $\lambda > 0$, its pdf is
\begin{equation*}
  f(x) = \frac{\lambda^n x^{n - 1} e^{-\lambda x}}{( n - 1 )!}.
\end{equation*}

\paragraph{Exponential Distribution}\index{Exponential Distribution} Following
the above distribution where $n = 1$, we have that $X$ follows what is called an
Exponential Distribution, denoted as $X \sim \Exp(\lambda)$, where its pdf is
\begin{equation*}
  f(x) = \lambda e^{- \lambda x}.
\end{equation*}

% subsection continuous_random_variables (end)

% section random_variables (end)

\section{Moments}%
\label{sec:moments}
% section moments

\begin{defn}[Expectation]\index{Expectation}
  \label{defn:expectation}\marginnote{Note that this definition is actually the
  \href{https://en.wikipedia.org/wiki/Law_of_the_unconscious_statistician}{Law
  of the Unconscious Statistician}}
  Let $X$ be an rv. Given a function $g$ that is defined over $X$, the
  \hlnoteb{expectation} of $g(X)$ is given by
  \begin{equation*}
    E[ g(X) ] = \begin{cases}
      \sum_{x} g(x) p(x) & \text{ if } X \text{ is a discrete rv } \\
      \int_{x} g(x) f(x) & \text{ if } X \text{ is a continuous rv }
    \end{cases}.
  \end{equation*}
\end{defn}

Now if $g(X) = X^k$ for some $k \in \mathbb{N}$, we have the following notion:

\begin{defn}[Moment]\index{Moment}
\label{defn:moment}
  Let $X$ be an rv. The $k$th moment of $X$ is defined as $E[ X^k ]$.
\end{defn}

Another notion that is commonly introduced after expectation is the variance.

\begin{defn}[Variance]\index{Variance}
\label{defn:variance}
  Let $X$ be an rv. The \hlnoteb{variance} of $X$ is given by
  \begin{equation*}
    \Var(X) = E[ (X - E[X])^2 ] = E[ X^2 ] - ( E[X] )^2
  \end{equation*}
\end{defn}

In relation to the variance, we have the standard deviation.

\begin{defn}[Standard Deviation]\index{Standard Deviation}
\label{defn:standard_deviation}
  Let $X$ be an rv. The \hlnoteb{standard deviation} (sd) is given by
  \begin{equation*}
    \sd(X) = \sqrt{\Var(X)} = \sqrt{ E[X^2] - (E[X])^2 }.
  \end{equation*}
\end{defn}

We shall state the following properties without providing proof\sidenote{The
proofs are very easy, but it serves as a strengthening exercise for the
unfamiliar. Therefore,
\begin{ex}
  Proof both \cref{propo:linearity_of_the_expectation} and \cref{propo:linearity_of_the_variance}.
\end{ex}
}:

\begin{propo}[Linearity of the Expectation]
\label{propo:linearity_of_the_expectation}
  Let $X$ be an rv. Let $a, b \in \mathbb{R}$. We have that
  \begin{equation*}
    E[ aX + b ] = aE[x] + b
  \end{equation*}
\end{propo}

\begin{propo}[Linearity of the Variance]
\label{propo:linearity_of_the_variance}
  Let $X$ be an rv. Let $a, b \in \mathbb{R}$. We have that
  \begin{equation*}
    \Var(aX + b) = a^2 \Var(X).
  \end{equation*}
\end{propo}

Referring back to \cref{defn:expectation}, if $g(X) = e^{tX}$, we have
ourselves, what is called, the moment generating function.

\begin{defn}[Moment Generating Function]\index{Moment Generating Function}
\label{defn:moment_generating_function}
  Let $X$ be an rv. The \hlnoteb{moment generating function} (mgf) of $X$ is given by
  \begin{equation*}
    \phi_X(t) = E\left[ e^{tX} \right].
  \end{equation*}
\end{defn}

\begin{note}
  \begin{enumerate}
    \item Observe that $\phi_X(0) = E\left[ e^0 \right] = 1$.
    \item The reason such an expression is called a moment generating function
      is as follows: observe that
      \begin{align*}
        \phi_X(t) &= E\left[ e^{tX} \right] = E\left[ \sum_{i=0}^{\infty}
                    \frac{(tX)^i}{i!} \right] \\
                  &= E\left[ 1 + \frac{tX}{1!} + \frac{(tX)^2}{2!} + \hdots +
                    \frac{(tX)^n}{n!} + \hdots \right] \\
                  &= \frac{t^0}{0!} E[1] + \frac{t}{1!} E[X] + \frac{t^2}{2!}
                    E\left[X^2\right] + \hdots + \frac{t^n}{n!} E\left[X^n\right]
                    + \hdots
      \end{align*}
      by \cref{propo:linearity_of_the_expectation}. If we take the $k$th
      derivative wrt $t$ and set $t = 0$, we will obtain the $k$th moment of
      $X$. In other words,
      \begin{equation*}
        E[X^k] = \phi_X^{(k)}(0) = \frac{d^k}{dt^k} \phi_X (t) \at{t = 0}{}.
      \end{equation*}
      It is \hlimpo{important} to note here that $t = 0$ can be interpreted in
      the sense of a limit, i.e. $\lim\limits_{t \to 0}$.
  \end{enumerate}
\end{note}

\begin{eg}
  Suppose $X \sim \Bin(n, p)$. Find $\mgf(X)$ and use it to find $E[X]$ and
  $\Var(X)$.

  \begin{solution}
    First, observe the \hlnotea{binomial formula}:
    \begin{equation*}
      (a + b)^m = \sum_{x = 0}^{m} \binom{m}{x} a^x b^m.
    \end{equation*}
    Now
    \begin{align*}
      \phi_X(t) &= E \left[ e^{tX} \right] = \sum_{x=0}^{n} e^{tx} \binom{n}{x}
                  p^x (1-p)^{n-x} \\
                &= \sum_{x=0}^{n} \binom{n}{x} \left( pe^t \right)^x \left( 1 -
                  p \right)^{n - x} \\
                &= \left( pe^t + 1 - p \right)^n.
    \end{align*}
    We observe that this works for all $t \in \mathbb{R}$.

    To find the expectation and variance, we do
    \begin{equation*}
      E[X] = \frac{d}{dt} \phi_X(t) \at{t=0}{} = n \left( pe^t + 1 - p
      \right)^{n-1} \cdot pe^t \at{t=0}{} = np
    \end{equation*}
    and
    \begin{align*}
      E[X^2] &= \frac{d^2}{dt^2} \phi_X(t) \at{t=0}{} \\
             &= npe^t \left( pe^t + 1 - p \right)^{n-1} + (n-1)np^2e^{2t} \left( pe^t
               + 1 - p \right)^{n-2} \at{t=0}{} \\
             &= np(1 + np - p)
    \end{align*}
    and conclude that
    \begin{equation*}
      \Var(X) = np + np(n-1)p - n^2p^2 = np(1 - p).
    \end{equation*}
  \end{solution}
\end{eg}

\begin{ex}
  For $X \sim \Poi(\lambda)$, show that
  \begin{equation*}
    \mgf(X) = \phi_X(t) = e^{\lambda(e^t - 1)}, \text{ for } t \in \mathbb{R}.
  \end{equation*}
  Find $E[X]$ and $\Var(X)$ using the mgf.
\end{ex}

\begin{ex}
  For $X \sim \Exp(\lambda)$, show that
  \begin{equation*}
    \mgf(X) = \phi_X(t) = \frac{\lambda}{\lambda - t}, \text{ for } t < \lambda.
  \end{equation*}
  Find $E[X]$ and $\Var(X)$ using the mgf.
\end{ex}

\begin{note}[Important property of the MGF]\label{note:mgf_uniquely_determines_distribution}
  The mgf is important to us because it \hlimpo{uniquely determines} the
  distribution of an rv.
\end{note}

% section moments (end)

\section{Joint Distributions}%
\label{sec:joint_distributions}
% section joint_distributions

We shall only review \hlnotea{bivariate distributions}. One can easily extend
the bivariate case to a multivariate situation.

\begin{defn}[Joint CDF]\index{Joint CDF}\label{defn:joint_cdf}
  The \hlnoteb{joint cdf} is defined by
  \begin{equation*}
    F(a, b) = P(X \leq a, Y \leq b) = P(\{ X \leq a \} \cap \{ Y \leq b \})
  \end{equation*}
  for all $a, b \in \mathbb{R}$, where $X, Y$ are rvs.
\end{defn}

\begin{defn}[Marginal CDF]\index{Marginal CDF}\label{defn:marginal_cdf}
  Given a joint cdf $F$, we define the \hlnoteb{marginal cdf} as
  \begin{equation*}
    F_X(a) = P(X \leq a) \coloneqq F(a, \infty) = \lim_{b \to \infty} F(a, b).
  \end{equation*}
\end{defn}

\begin{defn}[Joint and Marginal Probability Mass Functions]\index{Joint Probability Mass Function}\index{Marginal Probability Mass Function}\label{defn:joint_pmf}
  Suppose $X, Y$ are discrete rvs. We define the \hlnoteb{joint probability mass
  function (pmf)} of $X$ and $Y$ as
  \begin{equation*}
    p(x, y) \coloneqq P(X = x, Y = y).
  \end{equation*}
  The \hlnoteb{marginal pmf} of $X$ and $Y$ are
  \begin{equation*}
    p_X(x) = P(X = x) = \sum_{y} p(x, y)
  \end{equation*}
  and
  \begin{equation*}
    p_Y(y) = P(Y = y) = \sum_{x} p(x, y),
  \end{equation*}
  respectively.
\end{defn}

We are not equipped with the knowledge to formally define a \hlnotea{probability
density function}, and so we shall only introduce it in a roundabout way.

\begin{defn}[Joint and Marginal Probability Density Function]\index{Joint Probability Density Function}\index{Marginal Probability Density Function}\label{defn:joint_pdf}
  Suppose $X, Y$ are continuous rvs. We define \hlnoteb{joint probability
  density function (pdf)} of $X$ and $Y$ as
  \sidenote{
  \begin{marginwarning}
    It is important to note that
    \begin{equation*}
      f(x, y) \neq P(X = x, Y = x),
    \end{equation*}
    and so we cannot put \cref{defn:joint_pmf} and \cref{defn:joint_pdf} as one
    definition.
  \end{marginwarning}
  }
  \begin{equation*}
    f : \mathbb{R}^2 \to \mathbb{R}.
  \end{equation*}
  The \hlnoteb{marginal pdf} of $X$ and $Y$ are
  \begin{equation*}
    f_X(x) \coloneqq \int_{\forall y} f(x, y) \dif{y}
  \end{equation*}
  and
  \begin{equation*}
    f_Y(y) \coloneqq \int_{\forall x} f(x, y) \dif{x},
  \end{equation*}
  respectively.
\end{defn}

\begin{note}
  Note that
  \begin{equation*}
    f(x, y) = \frac{\partial^2}{\partial x \partial y} F(x, y).
  \end{equation*}
\end{note}

\section{Expectation of Joint Distributions}%
\label{sec:expectation_of_joint_distributions}
% section expectation_of_joint_distributions

\begin{defn}[Expectation of Joint Distributions]\index{Expectation of Joint Distributions}\label{defn:expectation_of_joint_distributions}
  Let $X$ and $Y$ be rvs, and $g(X, Y)$ a function. We define the
  \hlnoteb{expectation of a joint distribution} of $X$ and $Y$ as
  \begin{equation*}
    E[g(X, Y)] = \begin{cases}
      \sum_{x} \sum_{y} g(x, y) p(x, y) & X, Y \text{ are jointly discrete } \\
      \int_{\forall x} \int_{\forall y} g(x, y) f(x, y) & X, Y \text{ are jointly continuous }
    \end{cases}
  \end{equation*}
\end{defn}

The following is a special case of an expectation, given for when
\begin{equation*}
  g(X, Y) = (X - E[X])(Y - E[Y]).
\end{equation*}

\begin{defn}[Covariance of Joint Distributions]\index{Covariance of Joint Distributions}\label{defn:covariance_of_joint_distributions}
  We define the \hlnoteb{covariance of a joint distribution} of $X$ and $Y$ as
  \begin{equation*}
    \Cov(X, Y) = E[(X - E[X])(Y - E[Y])] = E[XY] - E[X]E[Y].
  \end{equation*}
\end{defn}

\begin{note}
  Observe that
  \begin{equation*}
    \Cov(X, X) = \Var(X).
  \end{equation*}
\end{note}

\marginnote{
\begin{ex}
  Prove \cref{propo:linearity_of_the_expectation_over_addition}.
\end{ex}
}
\begin{propo}[Linearity of the Expectation over Addition]\label{propo:linearity_of_the_expectation_over_addition}
  Let $X$ and $Y$ be rvs. Then for $a, b \in \mathbb{R}$,
  \begin{equation*}
    E[aX + bY] = aE[X] + bE[Y].
  \end{equation*}
\end{propo}

\marginnote{
\begin{ex}
  Prove \cref{propo:variance_over_a_linear_combination_of_rvs}.
\end{ex}
}
\begin{propo}[Variance over a Linear Combination of RVs]\label{propo:variance_over_a_linear_combination_of_rvs}
  Let $X$ and $Y$ be rvs. Then for $a, b \in \mathbb{R}$,
  \begin{equation*}
    \Var(aX + bY) = a^2 \Var(X) + b^2 \Var(Y) + 2ab \Cov(X, Y).
  \end{equation*}
\end{propo}

% section expectation_of_joint_distributions (end)

% section joint_distributions (end)

\section{Independence of Random Variables}%
\label{sec:independence_of_random_variables}
% section independence_of_random_variables

\begin{defn}[Independence]\index{Independence}\label{defn:independence}
  Let $X$ and $Y$ be rvs, and $F$ their joint cdf. We say that $X$ and $Y$ are
  \hlnoteb{independent} if
  \begin{equation*}
    F(x, y) = F_X(x) \cdot F_Y(y),
  \end{equation*}
  where $F_X$ and $F_Y$ are the marginal cdfs of $X$ and $Y$, respectively. We
  shall denote this relationship between rvs as $X \bot Y$.
\end{defn}

\begin{note}[Equivalent definition of Independence]
  One may also define independence of $X$ and $Y$ by
  \begin{equation*}
    p(x, y) = p_X(x) \cdot p_Y(y)
  \end{equation*}
  if $X$ and $Y$ are discrete, and
  \begin{equation*}
    f(x, y) = f_X(x) \cdot f_Y(y)
  \end{equation*}
  if $X$ and $Y$ are continuous.
\end{note}

\begin{propo}[Expectation of Independent RVs]\label{propo:expectation_of_independent_rvs}
  Let $X$ and $Y$ be independent rvs, and $g$ a function of $X$ and $h$ a
  function of $Y$. Then
  \begin{equation*}
    E(g(X) h(Y)) = E[g(X)] \cdot E[h(Y)].
  \end{equation*}
\end{propo}

\begin{crly}[MGF of Independent RVs]\label{crly:mgf_of_independent_rvs}
  Suppose $X_1, X_2, \ldots, X_n$ are independent rvs, then consider $T =
  \sum_{i=1}^{n} X_i$. The mgf of $T$ is then given by
  \begin{equation*}
    \phi_T(t) = \prod_{i=1}^{n} \phi_{X_i}(t).
  \end{equation*}
\end{crly}

\begin{proof}
  Suppose $X_1, X_2, \ldots, X_n$ are independent rvs, then consider $T =
  \sum_{i=1}^{n} X_i$. Then the mgf of $T$ is
  \begin{align*}
    \phi_T(t) &= E\left[e^{tT}\right] = E[\exp\{t(X_1 + X_2 + \hdots + X_n)\}] \\
              &= E \left[ e^{tX_1} \cdot \hdots \cdot e^{tX_n} \right] \\
              &= E \left[ e^{tX_1} \right] \hdots E \left[ e^{tX_n} \right]
              \enspace \because \cref{propo:expectation_of_independent_rvs} \\
              &= \phi_{X_1}(t) \cdot \hdots \cdot \phi_{X_n}(t) =
              \prod_{i=1}^{n} \phi_{X_i}(t),
  \end{align*}
  which is what we want.
\end{proof}

\begin{eg}[MGF of Independent Binomial Distributions with the Same Probability]
  Let $X_1, X_2, \ldots, X_m$ be an independent sequence of rvs where $X_i \sim
  \Bin(n_i, p)$. Let $T = \sum_{i=1}^{m} X_i$. By
  \cref{crly:mgf_of_independent_rvs} mgf of $T$ is
  \begin{equation*}
    \phi_T(t) = \prod_{i=1}^{m} \phi_{X_i}(t) = \prod_{i=1}^{m} (pe^t + 1 -
    p)^{n_i} = \left( pelt + 1 - p \right)^{\sum_{i=1}^{m} n_i}.
  \end{equation*}
  We observe that
  \begin{equation*}
    T \sim \Bin \left( \sum_{i=1}^{m} n_i, p \right).
  \end{equation*}
\end{eg}

% section independence_of_random_variables (end)

% chapter elementary_probability_review (end)

\chapter{Conditional Probabilities}%
\label{chp:conditional_probabilities}
% chapter conditional_probabilities

\section{Conditional Probability for Discrete Random Variables}%
\label{sec:conditional_probability_for_discrete_random_variables}
% section conditional_probability_for_discrete_random_variables

\begin{defn}[Conditional Distribution of Discrete Random Variables]\index{Conditional Distribution}\index{Conditional Probability}\label{defn:conditional_distribution_of_discrete_random_variables}
  Let $X_1, X_2$ be discrete rvs with joint pmf $p(x_1, x_2)$ and marginal pmfs
  $p_1(x_1)$ and $p_1(x_2)$, respectively. We call $X_1 \mid X_2 = x_2$ the
  \hlnoteb{conditional distribution} of $X_1$ given $X_2 = x_2$.

  The \hlnoteb{conditional pmf} of $X_1 \mid X_2 = x_2$ is defined as
  \begin{equation*}
    p(x_1 \mid x_2) = P(X_1 = x_2 \mid X_2 = x_2) = \frac{P(X_1 = x_1 \land X_2
    = x_2)}{P(X_2 = x_2)},
  \end{equation*}
  or more succinctly,
  \begin{equation*}
    p(x_1 \mid x_2) = P(X_1 \mid X_2 = x_2) = \frac{p(x_1, x_2)}{p_2(x_2)},
  \end{equation*}
  where we require $p_2(x_2) > 0$.
\end{defn}

\begin{note}
  \begin{itemize}
    \item If $X_1$ and $X_2$ are independent, then
      \begin{equation*}
        p(x_1 \mid x_2) = \frac{p(x_1, x_2)}{p_2(x_2)} = \frac{p_1(x_1)
        p_2(x_2)}{p_2(x_2)} = p_1(x_1).
      \end{equation*}
    \item We may extend the above definition to multivariate cases.
  \end{itemize}
\end{note}

\begin{eg}
  Suppose $X_1, X_2, X_3$ are discrete rvs, with $p_3$ as the pmf of $X_3$. Then
  we define the conditional pmf of $(X_1, X_2) \mid X_3 = x_3$ as
  \begin{equation*}
    p((x_1, x_2) \mid x_3) = \frac{p(x_1, x_2, x_3)}{p_3(x_3)}.
  \end{equation*}
\end{eg}

\begin{defn}[Conditional Expectation for Discrete RVs]\index{Conditional Expectation}\label{defn:conditional_expectation_for_discrete_rvs}
  Given rvs $X_1$ and $X_2$, and a function $g$ on $X_1$, We define the
  conditional expectation of $X_1$ given $X_2 = x_2$ as
  \begin{equation*}
    E[g(X_1) \mid X_2 = x_2] = \sum_{x_1} g(x_1) p(x_1 \mid x_2).
  \end{equation*}
\end{defn}

\begin{note}
  By \cref{propo:linearity_of_the_expectation_over_addition}, we have that given
  rvs $X_1$ an $X_2$, $a, b \in \mathbb{R}$, and functions $g, h$ on $X_1$,
  \begin{align*}
    &E \left[ ag(X_1) + bh(X_1) \mid X_2 = x_2 \right] \\
    &= aE[g(X_1) \mid X_2 = x_2] + aE[h(X_1) \mid X_2 = x_2]
  \end{align*}
\end{note}

\begin{propo}[Conditional Variance of Discrete RVs]\label{propo:conditional_variance_of_discrete_rvs}
  We have that
  \begin{equation*}
    \Var(X_1 \mid X_2 = x_2) = E[X_1^2 \mid X_2 = x_2] - E[X_1 \mid X_2 =
    x_2]^2.
  \end{equation*}
\end{propo}

\begin{proof}
  Consider
  \begin{equation*}
    g(x) = [X_1 - E[X_1 \mid X_2 = x_2]]^2,
  \end{equation*}
  which is the typical definition of a variance. Then
  \begin{align*}
    &\Var(X_1 \mid X_2 = x_2) \\
    &= E[(X_1 - E[X_1 \mid X_2 = x_2])^2 \mid X_2 = x_2] \\
    &= E \Big[X_1^2 - 2X_1 E[X_1 \mid X_2 = x_2] + E[X_1 \mid X_2 = x_2]^2 \;
      \Bigg\mid X_2 = x_2 \Big] \\
    &= E[X_1^2 \mid X_2 = x_2] - 2E[X_1 \mid X_2 = x_2]^2 + E[X_1 \mid X_2 =
      x_2]^2 \\
    &= E[X_1^2 \mid X_2 = x_2] - E[X_1 \mid X_2 = x_2]^2.
  \end{align*}
\end{proof}

Even better, we have the following proposition.

\begin{propo}[Linearity of Conditional Expectation]\label{propo:linearity_of_conditional_expectation}
  Given rvs $X_1, \, X_2$ and $X_3$, we have
  \begin{equation*}
    E[X_1 + X_2 \mid X_3 = x_3] = E[X_1 \mid X_3 = x_3] + E[X_2 \mid X_3 = x_3].
  \end{equation*}
\end{propo}

\begin{proof}
  Let $p_3$ be the pmf of $X_3$. Observe that
  \begin{align*}
    &E[X_1 + X_2 \mid X_3 = x_3] \\
    &= \sum_{x_1} \sum_{x_2} (x_1 + x_2) p(x_1, x_2 \mid x_3) \\
    &= \sum_{x_1} \sum_{x_2} x_1 \frac{p(x_1, x_2, x_3)}{p_3(x_3)} + \sum_{x_1}
      \sum_{x_2} x_2 \frac{p(x_1, x_2, x_3)}{p_3(x_3)} \\
    &= \sum_{x_1} \frac{x_1}{p_3(x_3)} \sum_{x_2} p(x_1, x_2, x_3) + \sum_{x_2}
      \frac{x_2}{p_3(x_3)} \sum_{x_1} p(x_1, x_2, x_3) \\
    &= \sum_{x_1} \frac{x_1}{p_3(x_3)} p(x_1, x_3) + \sum_{x_2}
      \frac{x_2}{p_3(x_3)} p(x_2, x_3) \\
    &= \sum_{x_1} x_1 p(x_1 \mid x_3) + \sum_{x_2} x_2 p(x_2 \mid x_3) \\
    &= E[X_1 \mid X_3 = x_3] + E[X_2 \mid X_3 = x_3].
  \end{align*}
\end{proof}

\begin{crly}[General Linearity of Conditional Expectation]\label{crly:general_linearity_of_conditional_expectation}
  Given an rv $Y$, a sequence of discrete rvs $\{X_i\}_{i=1}^{n}$, and a sequence of
  scalars $\{a_i\}_{i=1}^{n} \subseteq \mathbb{R}$, we have
  \begin{equation*}
    E \left[ \sum_{i=1}^{n} a_i X_i \mid Y = y \right] = \sum_{i=1}^{n} a_i
    E[X_i \mid Y = y].
  \end{equation*}
\end{crly}

\begin{eg}
  Suppose that $X$ and $Y$ are discrete rvs having joint pmf
  \begin{equation*}
    p(x, y) = \begin{cases}
      \frac{1}{5}  & x = 1, \, y = 0 \\
      \frac{2}{15} & x = 0, \, y = 1 \\
      \frac{1}{15} & x = 1, \, y = 2 \\
      \frac{1}{5}  & x = 2, \, y = 0 \\
      \frac{2}{5}  & x = 1, \, y = 1 \\
      0            & \text{ otherwise }
    \end{cases}
  \end{equation*}
  Let us first find the conditional distribution of $X \mid Y = 1$. First,
  consider the following table:
  \begin{table}[ht]
    \centering
    \caption{Tabulating values of $p(x, y)$}
    \label{table:tabulating_values_of_p_x_y}
    \begin{tabular}{c | c c c | c}
      $X \diagdown Y$ & $0$           & $1$            & $2$            & $p_X$ \\
      \hline
      $0$             & $0$           & $\frac{2}{15}$ & $0$            & $\frac{2}{15}$ \\
      $1$             & $\frac{1}{5}$ & $\frac{2}{5}$  & $\frac{1}{15}$ & $\frac{2}{3}$ \\
      $2$             & $\frac{1}{5}$ & $0$            & $0$            & $\frac{1}{5}$ \\
      \hline
      $p_Y$           & $\frac{2}{5}$ & $\frac{8}{15}$ & $\frac{1}{15}$
    \end{tabular}
  \end{table}

  Observe that
  \begin{equation*}
    p(0, 1) = \frac{\frac{2}{15}}{\frac{8}{15}} = \frac{1}{4} \text{ and } p(1,
    1) = \frac{\frac{2}{5}}{\frac{8}{15}} = \frac{3}{4}.
  \end{equation*}
  Thus the conditional distribution of $X \mid Y = 1$ is given as in
  \cref{table:conditional_distribution_of_x_mid_y_1}.
  \begin{table}[ht]
    \centering
    \caption{Conditional distribution of $X \mid Y = 1$}
    \label{table:conditional_distribution_of_x_mid_y_1}
    \begin{tabular}{c | c c}
      $X$           & $0$           & $1$ \\
      \hline
      $p(x \mid 1)$ & $\frac{1}{4}$ & $\frac{3}{4}$
    \end{tabular}
  \end{table}

  With this we can calculate the conditional expectation and conditional
  variance of $X \mid Y = 1$. Note that
  \begin{equation*}
    X \mid Y = 1 \sim \Bernoulli \left( \frac{3}{4} \right).
  \end{equation*}
  Thus
  \begin{equation*}
    E[X \mid Y = 1] = \frac{3}{4} \text{ and } \Var(X \mid Y = 1) = \frac{3}{4}
    \cdot \frac{1}{4} = \frac{3}{16}.
  \end{equation*}
\end{eg}

\begin{eg}
  For $i = 1, 2$ suppose that $X_i \sim \Bin(n_i, p)$, where $X_1 \bot X_2$. We
  want to find the conditional distribution of $X_1$ given $X_1 + X_2 = n$, i.e.
  the conditional distribution of $X_1 \mid X_1 + X_2 = n$.

  First, note that the sum of two \hlnotea{binomial distributions} is also a
  binomial distribution, where the number of trials is the sum of the number of
  trials from each distribution. In particular, we have that $X_1 + X_2 \sim
  \Bin(n_1 + n_2, p)$.

  Let the conditional pmf of $X_1 \mid X_1 + X_2 = n$ be denoted as $p(x_1 \mid
  n)$. Then
  \begin{align*}
    p(x_1 \mid n) &= P(X_1 = x_1 \mid X_1 + X_2 = n) \\
                  &= \frac{P(X_1 = x_1, \, X_1 + X_2 = n)}{P(X_1 + X_2 = n)} \\
                  &= \frac{P(X_1 = x_1, X_2 = n - x_1)}{P(X_1 + X_2 = n)} \\
                  &= \frac{P(X_1 = x_1)P(X_2 = n - x_1)}{P(X_1 + X_2 = n)}
                  \enspace \because X_1 \bot X_2 \\
                  &= \frac{\binom{n_1}{x_1}p^{x_1}(1-p)^{n_1 - x_1}
                    \binom{n_2}{n - x_1}p^{n - x_1}(1-p)^{n_2 - n +
                    x_1}}{\binom{n_1 + n_2}{n}p^{n}(1-p)^{n_1 + n_2 - n}} \\
                  &= \frac{\binom{n_1}{x_1} \binom{n_2}{n - x_1}}{\binom{n_1 +
                    n_2}{n}},
  \end{align*}
  for $0 \leq x_1 \leq n_1$ and $0 \leq n - x_1 \leq n_2$. We observe that $X_1
  \mid X_1 + X_2 = n$ has a \hlnotea{Hypergeometric Distribution}, i.e.
  \begin{equation*}
    X_1 \mid X_1 + X_2 = n \sim \HG(n_1 + n_2, n_1, n).
  \end{equation*}

  Recall that the intuition to understanding the hypergeometric distribution in
  this case is as follows: if we say that $1, 2, \ldots, n_1$ are the indexed
  trials of $X_1$ and $1, 2, \ldots, n_2$ are those of $X_2$, then if we arrange
  the trials as
  \begin{equation*}
    1, 2, \ldots, n_1, 1, 2, \ldots, n_2,
  \end{equation*}
  we may then think that we want to calculate the probability of getting $x_1$
  successes given that the rest of the $n - x_1$ are failures.

  Using the formulas for the expectation and variance of a hypergeometric
  distribution, we have that
  \begin{equation*}
    E[X_1 \mid X_1 + X_2 = n] = \frac{n(n_1)}{n_1 + n_2}
  \end{equation*}
  and
  \begin{align*}
    \Var(X_1 \mid X_1 + X_2 = n)
    &= \frac{n(n_1)(n_1 + n_2 - n_1)(n_1 + n_2 - n)}{(n_1 + n_2)^2(n_1 + n_2 -
      1)} \\
    &= \frac{n(n_1)(n_2)(n_1 + n_2 - n)}{(n_1 + n_2)^2(n_1 + n_2 - 1)}.
  \end{align*}
\end{eg}

Let's deal with a more general case of the above, but this time with the
individual rvs following $\Poi(\Lambda_i)$.

\begin{eg}
  Let $\{X_i\}_{i=1}^{m}$ be a sequence of independent rvs where $X_i \sim
  \Poi(\Lambda_i)$, and $i = 1, 2, \ldots, m$. Let $Y = \sum_{i=1}^{m} X_i$. Let
  us try to deduce the conditional distribution of $X_j \mid Y = n$.

  First, note that a sum of independent Poisson rvs is also a Poisson
  distribution, with each of its mean (which is also its parameter) summed up,
  i.e. given any $Z_1 \sim \Poi(\lambda_1), \ldots, Z_k \sim \Poi(\lambda_k)$,
  we have
  \begin{equation*}
    \sum_{i=1}^{k} Z_i \sim \Poi \left( \sum_{i=1}^{k} \lambda_i \right).
  \end{equation*}
  Also a subtle point, note that
  \begin{equation*}
    Z_j \; \bot \; \sum_{\substack{i=1 \\ i \neq j}}^{k} Z_i.
  \end{equation*}

  Observe that
  \begin{align*}
    &P(X_j = x_j \mid Y = n) \\
    &= \frac{P(X_j = x_j, Y = n)}{P(Y = n)} \\
    &= \frac{P\left(X_j = x_j, \, \sum\limits_{\substack{i=1 \\ i \neq j}}^{m} X_i = n -
      x_j \right)}{P(Y = n)} \\
    &= \frac{P(X_j = x_j) P\left( \sum\limits_{\substack{i=1 \\ i \neq j}}^{m} X_i = n - x_j \right)}{P(Y = n)} \\
    &= \frac{\exp(-\lambda_j) \lambda_j^{x_j} \left( \frac{1}{x_j!} \right) \cdot
      \exp\left(-\left( \sum\limits_{\substack{i=1 \\ i \neq j}}^{m} \lambda_i \right)\right)
      \left( \sum\limits_{\substack{i=1 \\ i \neq j}}^{m} \lambda_i \right)^{n-x_j}
      \left( \frac{1}{(n - x_j)!} \right)}{\exp\left(-\left(
      \sum\limits_{i=1}^{m} \lambda_i \right)\right) \left( \sum\limits_{i=1}^{m}
      \lambda_i \right)^m \left( \frac{1}{n!} \right)} \\
    &= \binom{n}{x_j} \frac{\lambda_j^{x_j} \left( \sum\limits_{i=1}^{m}
      \lambda_i - \lambda_j \right)^{n - x_j}}{\left( \sum\limits_{i=1}^{m}
      \lambda_i \right)^{x_j} \left( \sum\limits_{i=1}^{m} \lambda_i
      \right)^{n-x_j}} \\
    &= \binom{n}{x_j} \left( \frac{\lambda_j}{\sum_{i=1}^{m} \lambda_i}
      \right)^{x_j} \left( 1 - \frac{\lambda_j}{\sum_{i=1}^{m} \lambda_i}
      \right)^{n-x_j},
  \end{align*}
  where $x_j \in [0, n]$. We see that
  \begin{equation*}
    X_j \mid Y = n \sim \Bin \left( n, \frac{\lambda_j}{\sum_{i=1}^{m}
    \lambda_i} \right).
  \end{equation*}
\end{eg}

\begin{ex}
  It is a straightforward exercise to compute $E[X_j \mid Y = n]$ and $\Var(X_j
  \mid Y = n)$.
\end{ex}

\begin{eg}
  Suppose $X \sim \Poi(\lambda)$ and $Y \mid X = x \sim \Bin(x, p)$. Let us
  compute the conditional distribution of $X \mid Y = y$. We have that
  \begin{align*}
    p(x \mid y)
    &= \frac{P(X = x, Y = y)}{P(Y = y)} \\
    &= \frac{p(y \mid x) \cdot P(X = x)}{\sum_{x} p(y \mid x) P(X = x)} \\
    &= \frac{\exp(-\lambda) \lambda^x \left( \frac{1}{x!} \right) \cdot
        \binom{x}{y} p^y (1-p)^{x-y}}{\sum\limits_{x = y}^{\infty}
        \exp(-\lambda)\lambda^x \left( \frac{1}{x!} \right) \cdot \binom{x}{y}
        p^y (1-p)^{x-y}} \\
    &= \frac{\exp(-\lambda) \lambda^x \cdot \frac{1}{y!(x-y)!} p^y
        (1-p)^{x-y}}{\sum\limits_{x = y}^{\infty} \exp(-\lambda)\lambda^x \cdot
        \frac{1}{y!(x-y)!} p^y (1-p)^{x-y}}
  \end{align*}
  Now notice that we may work out the denominator to be
  \begin{align*}
    & \sum_{x=y}^{\infty} e^{-\lambda} \lambda^x \frac{1}{y!(x-y)!}p^y
      (1-p)^{x-y} \\
    &= \frac{e^{-\lambda}}{y!} p^y \sum_{x=y}^{\infty} \frac{\lambda^x}{(x-y)!}
      (1-p)^{x-y} \\
    &= \frac{e^{-\lambda} p^y \lambda^y}{y!} \sum_{x=y}^{\infty}
      \frac{\lambda^{x-y}(1-p)^{x-y}}{(x-y)!} \\
    &= \frac{e^{-\lambda} p^y \lambda^y}{y!} \sum_{x-y=0}^{\infty}
      \frac{(\lambda(1-p))^{x-y}}{(x-y)!} \\
    &= \frac{e^{-\lambda + \lambda(1 - p)} p^y \lambda^y}{y!} \\
    &= \frac{e^{-\lambda p} (\lambda p)^y}{y!},
  \end{align*}
  where the second last equality follows since
  \begin{equation*}
    e^x = \sum_{n=0}^{\infty} \frac{x^n}{n!}
  \end{equation*}
  by the \hlnotea{Taylor Expansion of the exponential function}. Thus we have
  \begin{align*}
    p(x \mid y)
    &= \frac{\exp(-\lambda) \lambda^x \cdot \frac{1}{y!(x-y)!} p^y
      (1-p)^{x-y}}{\frac{(\lambda p)^y e^{\lambda p}}{y!}} \\
    &= \frac{e^{-\lambda(1 - p)}(\lambda(1-p))^{x-y}}{(x-y)!},
  \end{align*}
  for $x \in [y, \infty)$, which is a Poisson-like pmf.

  We say that
  \begin{equation*}
    X \mid Y = y \sim W + y
  \end{equation*}
  where
  \begin{equation*}
    W \sim \Poi(\lambda(1 - p)).
  \end{equation*}
  This distribution, $W + y$, is called a \hldefn{shifted Poisson}
  distribution.

  Observe that it is relatively easy to compute the expectation and variance due
  to their linearity properties: we have
  \begin{equation*}
    E[X \mid Y = y] = E[W + y] = E[W] + y = \lambda(1-p) + y
  \end{equation*}
  and
  \begin{equation*}
    \Var(X \mid Y = y) = \Var(W + y) = \Var(W) = \lambda(1 - p).
  \end{equation*}
\end{eg}

% section conditional_probability_for_discrete_random_variables (end)

\section{Conditional Probability for Continuous Random Variables}%
\label{sec:conditional_probability_for_continuous_random_variables}
% section conditional_probability_for_continuous_random_variables

In the
\hyperref[sec:conditional_probability_for_discrete_random_variables]{jointly
discrete case}, it was natural to define
\begin{equation*}
  p(x \mid y) = P(X = x \mid Y = y) = \frac{P(X = x, Y = y)}{P(Y = y)}
\end{equation*}
due to \hlnotea{Bayes' Rule}, or simply by realizing that we are trying to
determine the probability of $X = x$ and $Y = y$ given that we already know the
probability of $Y = y$.

This, however, does not make sense in the continuous case, especially since
$f(x, y) \neq P(X = x, Y = y)$, and $f_Y(y) \neq P(Y = y)$.

\sidenote{I am actually not sure how this paragraph inspires our definition,
cause some things don't seem to match up nicely.} We can use a similar
definition of a single variable continuous rv and extend it for 2 rvs: we can
consider
\begin{equation}\label{eq:conditional_prob_for_cts_rvs_inspiration}
  f(x, y) = \lim_{\substack{\dif{x} \to 0 \\ \dif{y} \to 0}} \frac{P(x \leq X
  \leq x + \dif{x}, y \leq Y \leq y + \dif{y})}{\dif{x} \dif{y}}.
\end{equation}
Notice that for small $\dif{x}$ and $\dif{y}$, we have
\begin{align*}
  P(\underbrace{x \leq X \leq x + \dif{x}}_{A} \mid \underbrace{y \leq Y \leq y
  + \dif{y}}_{B})
  &= \frac{P(A \cap B)}{P(B)} \\
  &\approx \frac{f(x,y) \dif{x} \dif{y}}{f_Y(y) \dif{y}} \\
  &= \frac{f(x,y)}{f_Y(y)} \dif{x}.
\end{align*}

\begin{defn}[Conditional Distribution of Continuous Random Variables]\index{Conditional Distribution}\label{defn:conditional_distribution_of_continuous_random_variables}
  Let $X$ and $Y$ be continuous rvs, with joint pdf $f(x, y)$, and $Y$ has the
  marginal pdf $f_Y$. We call $X \mid Y = y$ the \hlnoteb{conditional
  distribution} of $X$ given $Y = y$, whose pdf is defined as
  \begin{equation*}
    f(x \mid y) = \frac{f(x, y)}{f_Y(y)} = \lim_{\substack{\dif{x} \to 0 \\
    \dif{y} \to 0}} \frac{P(x \leq X \leq x + \dif{x}, y \leq Y \leq y +
    \dif{y})}{\dif{x}}.
  \end{equation*}
\end{defn}

\begin{eg}
  Suppose the joint pdf of $X$ and $Y$ is given by
  \begin{equation*}
    f(x, y) = \begin{cases}
      \frac{12}{5} x (2 - x - y) & 0 < x < 1, \, 0 < y < 1 \\
      0                          & \text{ otherwise }
    \end{cases}
  \end{equation*}
  \hlimpo{First}, note that the \hldefn{region of support} for $X$ and $Y$ is
  given as in \cref{fig:region_of_support_as_a_rectangle}.
  \begin{marginfigure}
    \centering
    \begin{tikzpicture}
      \draw[->] (-0.5, 0) -- (2, 0) node[right] {$x$};
      \draw[->] (0, -0.5) -- (0, 2) node[above] {$y$};
      \fill[color=blue!30!background] (0, 0) -- (1, 0) -- (1, 1) -- (0, 1) -- cycle;
      \node[none,circle,fill,inner sep=1pt,label={270:{$1$}}] at (1, 0) {};
      \node[none,circle,fill,inner sep=1pt,label={180:{$1$}}] at (0, 1) {};
    \end{tikzpicture}
    \caption{Region of support as a rectangle}\label{fig:region_of_support_as_a_rectangle}
  \end{marginfigure}
  We compute the conditional pdf of $X \mid Y = y$:
  \begin{align*}
    f(x \mid y) &= \frac{f(x,y)}{f_Y(y)} \\
                &= \frac{\cancel{\frac{12}{5}}x(2-x-y)}{\int_{0}^{1} \cancel{\frac{12}{5}}x(2-x-y)
                  \dif{x}} \\
                &= \frac{2x - x^2 - xy}{x^2 - \frac{1}{3}x^3 - \frac{1}{2}x^2 y
                  \at{x=0}{x=1}} \\
                &= \frac{2x - x^2 - xy}{1 - \frac{1}{3} - \frac{1}{2} y} \\
                &= \frac{2x - xy - x^2}{\frac{2}{3} - \frac{1}{2} y} \\
                &= \frac{12x - 6xy - 6x^2}{4 - 3y}, \text{ for } 0 < x < 1.
  \end{align*}
\end{eg}

% section conditional_probability_for_continuous_random_variables (end)

% chapter conditional_probabilities (end)

\appendix

\backmatter

\fancyhead[LE]{\thepage \enspace \textsl{\leftmark}}

\bibliography{references}

\printindex

\end{document}
% vim:tw=80:fdm=syntax


\documentclass[12pt]{article}
\usepackage[margin=1in]{geometry}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{listings}
\usepackage{mathtools}
\DeclarePairedDelimiter\ceil{\lceil}{\rceil}
\DeclarePairedDelimiter\floor{\lfloor}{\rfloor}
\newcommand{\N}{\mathbb{N}}
\renewcommand\thesubsubsection{\alph{subsubsection}}
\setlength{\parskip}{1em}
\setlength\parindent{0pt}
\renewcommand\arraystretch{1.5}
\usepackage{courier} %% Sets font for listing as Courier.
\usepackage{listings}
\usepackage[dvipsnames]{xcolor}
\definecolor{almond}{RGB}{246, 243, 199}
\lstset{
tabsize = 4, %% set tab space width
showstringspaces = false, %% prevent space marking in strings, string is defined as the text that is generally printed directly to the console
numbers = left, %% display line numbers on the left
commentstyle = \color{ForestGreen}, %% set comment color
keywordstyle = \color{blue}, %% set keyword color
stringstyle = \color{red}, %% set string color
rulecolor = \color{black}, %% set frame color to avoid being affected by text color
basicstyle = \small \ttfamily , %% set listing font and size
breaklines = true, %% enable line breaking
numberstyle = \tiny,
backgroundcolor=\color{almond}
}

% Bibliography
\usepackage[style=apa, backend=biber]{biblatex}
\addbibresource{5112.bib}

\begin{document}

\begin{titlepage}
\centering
{
	\includegraphics[scale=0.08]{./media/0.png}\\
	\vfill
	{\Large \textbf{FE5112}\par}
	{\Large \textbf{Stochastic Calculus and Quantitative Methods}\par}
	{\Large Project Report\par}
	\vfill
	{\normalsize Chen Penghao (A0122017Y)\par}
	{\normalsize Chia Kai Jit (A0122017Y)\par}
	{\normalsize Joanna See (A0122017Y)\par}
	{\normalsize Li Yihao (A0122017Y)\par}
	{\normalsize Zhang Zequn (A0156936R)\par}
	\vfill
% Bottom of the page
	{ November 2020\par}
}
\end{titlepage}

\section{Introduction}
The project aims to obtain the price of an Asian American put option by using the Least-Squared Monte Carlo simulation method.  We also have explored various techniques to reduce the variance of the results produced and to increase the stablity of the results.

\section{Problem Statement}

The market is complete and arbitrage free. The risk free interest rate is 8\%. The price of a non-dividend-paying stock $S_t$ at time $t$ is governed by the following SDE under the risk neutral probability measure $\mathcal{Q}$:

$$
\begin{cases}
dS_t = 0.08\hspace{1mm}S_t\hspace{1mm}dt + \sigma(S_t) \cdot S_t\hspace{1mm}dW_t, t > 0 \\
S_0 = 100
\end{cases}
$$

where $W_t$ is a Brownian motion under $\mathcal{Q}$ and $\sigma(\cdot)$ is a function given as the following:

$$
\sigma(x) =
\begin{cases}
0.25+ 0.02 \times \left ( 1.0 - \dfrac{x}{S_0} \right), \text{if }x\leq 100 \\
\max\left(0.001, 0.25-0.01 \times\left(\dfrac{x}{S_0} - 1\right) \right), \text{if }x>100
\end{cases}
$$

We are to price a one-year-non-exercise American Asian put option with the strike price $K=108$ and the maturity $T=2$ in years. The option is not exercisable in the first year. The option is exercisable once and only once in the second year. If the option is exercised on a day $n$, where $366 \leq n \leq 730$, the option buyer will receive a payout of $(K-A_n)^+$, where $A_n$ is given by:
$$A_n = \dfrac{1}{60} \sum_{i=n-60}^{n-1} S_{\frac{i}{365}}$$

We assume that there are 365 days in a year and all days are business days.

\newpage
\section{LSMC with no variance reduction}

\subsection{Settings}

The variables are set up as the following:

\begin{lstlisting}[language=Python]
T = 2                       # Duration in years
L = 730                     # Number of time intervals
M = 10000                     # Number of asset paths
rf = 0.08                   # Annualized risk-free rate
S0 = 100.0                  # Beginning stock price
dt = T/L                    # Time discretization (dt = 1/365)
dayEx = 730                 # Maximum days before the option exercises
discf = math.exp(-rf * dt)  # Discount factor to backwardize by one day.
\end{lstlisting}

\subsection{The Algorithm}

Our algorithm follows the following step:

\begin{enumerate}
\item Generate $M$ sample paths.
\item Generate a table of stock price data $S_L$ for 730 time intervals with 500 asset price paths. $S(i,j)$ would refer to the stock price on the $i^\text{th}$ path on the $j^\text{th}$ day.
\item Calculate the option payoff on the final day, the $L^\text{th}$ day or day 730.
\item Starting with the $(L-1)^\text{th}$ day, we do a backward propagation for each simulated path of the asset. For day i from $L-1$ back to 366:
	\begin{enumerate}
	\item Skip all the paths where it is not in the money on the $i^\text{th}$ day.
	\item If some path is in the money,
	\end{enumerate}
\item The prices on all the sample pathsa re discounted to day 0. We simply take the average of the paths to obtain the fair value of the option.
\end{enumerate}

\subsection{Analysis of results}

\subsection{Conclusion}

\section{Variance reduction methods applied to LSMC}

\subsection{Antithetic Method}

\subsection{Control Variate Method}

\begin{align}
    \Tilde{X} = X + c(Y - \mathbb{E}[Y])
\end{align}

\begin{align}
    \Tilde{c} = -\frac{Cov(X,Y)}{Var(Y)}
\end{align}

\subsection{Importance Sampling Method}

\begin{align}
    dS_t = rS_t{dt} + \sigma{S_t}dW_t
\end{align}

\begin{align}
    \theta = \frac{r-r_1}{\sigma}
\end{align}

\section{Conclusion}

\section{Final Words}

The contributions of our group members are as of the following:
\begin{itemize}
\item Kai Jit: Variance reduction with Antithetic method.
\item Penghao: Base code for LSMC and analysis.
\item Zequn: Variance reduction with importance sampling; Visualization of price paths; Refactor code base.
\item Yihao: Variance reduction with control variate.
\item Joanna:
\end{itemize}

\printbibliography

\end{document}


\section{Definition}

\subsection{Project Overview}

Object recognition is a central topic in the field of computer vision. The numerous applications can be applied in many different domains such as self-driving vehicles environment detection, skin cancer screening or reality augmented tourism.

This project focus on the automatic recognition of flower species from pictures. The potential applications are:

\begin{itemize}
	\setlength\itemsep{1pt}
	\setlength{\parskip}{0pt}
	\setlength{\parsep}{0pt}
	\item population tracking and preservation
	\item crop and food supply management
	\item toxicity detection
	\item education
	\item and more...
\end{itemize}

\begin{center}
\includegraphics[scale=1.3]{Daisy_detection.jpg}
\captionof{figure}{application example: it's a Daisy!}
\end{center}

Flowers recognitions is already used in mobile applications addressing for example to garden lovers or hitch-hikers \cite{best-apps}. Unfortunately neither the technology or performances of the models used in those products are publicly available.
However the research work on flower species identification entitled "bi-level co-segmentation method for image classification" from Yuning Chai, Andrew Zimmerman and Victor Lempitsky in 2011 \cite{Chai_BiCos}, demonstrated a robust method involving the combination of image intensive pre-processing and SVM classification training. This work is used as benchmark for this project as specified in the Chapter \fullref{subsec:benchmark}.

\subsection{Problem Statement}

Is it possible to accurately identify the variety of a flower from a picture ?

The today state of the art solution to tackle pictures classification problems is the creation and training of a deep Convolution Neural Network (CNN). The training has to be performed on a dataset  of flower pictures classified by species.

\subsection{Dataset}

The dataset used in this project is the \lq\lq{}Flowers Recognition Dataset\rq\rq{} of  \href{https://www.kaggle.com/alxmamaev/flowers-recognition}{Kaggle.com}  given by \href{https://alxmamaev.github.io/}{Alexander Mamaev}.\\

The zip archive is 230MB and contains about 4300 flower images (.jpg) classified in 5 classes: Daisy, Dandelion, Rose, Sunflower and Tulip.

\subsection{metrics}

In this project, we aim to build a model able to correctly classify flowers from five different classes. The trivial metrics to evaluate how good the model is at predicting the correct flowers classes is the  \textit{Accuracy} $ACC$ defined as follows:

\begin{center}

	$ACC = \frac{number \ of \ correct \ predictions}{total \ number \ of \ prediction} \cdot 100$ \cite{sklearn_accuracy}

\end{center}

Moreover, we will use the textit{Classification Cross-Entropy} $H(y, \hat{y})$ in order to evaluate the model progression during the training and save the weights each time a new minima is reached.

\begin{center}
$H(y, \hat{y})=-\sum_{k=1} y_i log(\hat{y_i})$ \cite{cross_enropy}

\end{center}