%<<setup-child, include = FALSE>>=
%library(knitr)
%library(qrmix)
%library(mlr)
%library(quantreg)
%library(reshape2)
%set_parent("../style/preamble.Rnw")
%@

\input{../../style/preamble}
\input{../../latex-math/basic-math}
\input{../../latex-math/basic-ml}

\usepackage{booktabs}

\newcommand{\titlefigure}{figure_man/vgg_example.png}
\newcommand{\learninggoals}{
  % \item Understand why you should care about properties of loss functions
  \item Know the concept of robustness
  \item Learn about analytical and computational properties of loss functions
  \item Understand that the loss function may influence convergence of the optimizer
}

\title{Introduction to Machine Learning}
% \author{Bernd Bi{}schl, Christoph Molnar, Daniel Schalk, Fabian Scheipl}
\institute{\href{https://compstat-lmu.github.io/lecture_i2ml/}{compstat-lmu.github.io/lecture\_i2ml}}
\date{}

\begin{document}

\lecturechapter{Properties of Loss Functions}
\lecture{Introduction to Machine Learning}

\begin{vbframe}{The role of Loss Functions}

Why should we care about how to choose the loss function $\Lxy$?

\begin{itemize}
% \item For regression, the loss usually only depends on residual $\Lxy = L\left(y - \fx\right) = L(\eps)$, this is a \emph{translation invariant} loss
\item \textbf{Statistical} properties: Choice of loss implies statistical assumptions on the distribution of $y ~|~ \xv = \xv$ (see \emph{Maximum Likelihood Estimation vs.
Empirical Risk Minimization}).
\item \textbf{Robustness} properties: Some loss functions are more robust towards outliers than others.
\item \textbf{Analytical} properties: The computational / optimization complexity of the problem.
$$
\argmin_{\thetab \in \Theta} \risket
$$
is influenced by the choice of the loss function.
\end{itemize}

\end{vbframe}

\begin{vbframe}{Basic Types of Regression Losses}

% https://davidrosenberg.github.io/mlcourse/Archive/2017/Lectures/3b.loss-functions.pdf

\begin{itemize}
  \item Regression losses usually only depend on the \textbf{ residuals}

  \vspace*{-0.5cm}

  \begin{eqnarray*}
    r &:=& y - \fx %\\
    %r^{(i)} &:=& \yi - \fxi .
  \end{eqnarray*}

  \item A loss is called \textbf{distance-based} if
  \begin{itemize}
    \item it can be written in terms of the residual
    $$
      \Lxy = \psi (r) \text{ for some } \psi: \R \to \R
    $$
    \item $\psi(r) = 0 \Leftrightarrow r = 0$ .
  \end{itemize}
  \item A loss is \textbf{translation-invariant}, if $L(y + a, \fx + a) = \Lxy$.
  \item Losses are called \textbf{symmetric} if $\Lxy = L\left(\fx, y\right)$.
\end{itemize}
  % We will see later that in case of the L2-loss, pseudo-residuals correspond to the residuals - hence the name.

  \textcolor{red}{@BB: This slide is a bit without context imo. We are also not saying anything about classification losses - should we? }

\end{vbframe}

\begin{vbframe}{Robustness}

Outliers (in $y$) have large residuals $r = y - \fx$. For some losses large residuals have a much bigger impact on the risk much more than for other losses.

\begin{table}[]
\begin{tabular}{lllll}
\toprule
$y$  & $\fxh$ & L1 & L2 & Huber ($\eps = 5$) \\ \midrule
1 & 0 & 1 & 1 & 0.5 \\
5 & 0 & 5 & 25 & 12.5 \\
10 & 0 & 10 & 100 & 37.5 \\
50 & 0 & 50 & 2500  & 237.5 \\ \bottomrule
\end{tabular}
\end{table}

As a consequence, a model is less influenced by outliers than by inliers if the loss is robust.

\framebreak

The L2 loss is an example for a loss function that is not very robust towards outliers. It penalizes large residuals more than the L1 or the Huber loss. The L1 and the Huber loss are thus regarded robust.

\begin{center}
\includegraphics[width=0.8\textwidth]{figure/robustness.png}
\end{center}

\end{vbframe}

\begin{vbframe}{Analytical Properties: Smoothness}

\begin{itemize}
    \item \textbf{Smoothness} of a function is a property that is measured by the number of continuous derivatives it has.
    \item A function is said to be $\mathcal{C}^k$ if it is $k$ times continuously differentiable. A function is $\mathcal{C}^\infty$ if it is continuously differently for all orders $k$.
    \item In contrast do derivative-free methods, derivative-based methods require a certain level of smoothness of the risk function $\risket$.
    \item Example: Gradient descent requires differentiability of the $\risket$ (existence of $\nabla \risket$), Newton-Raphson requires $\risket$ to be twice differentiable (existence of Hessian $\nabla^2 \risket$).

    \framebreak

    \item If the loss function is not smooth, the risk minimization problem is in general not smooth either.
    \item  Instead, derivative-free optimization need to be used (which might not be desirable)
\end{itemize}

\begin{center}
  \includegraphics[width = 0.5\textwidth]{figure/plot_loss_overview_classif.png} \\
  \begin{footnotesize}
    Squared loss, exponential loss, and squared hinge loss are continuously differentiable. The hinge loss is continuous but not differentiable. The 0-1-loss is not even continuous.
  \end{footnotesize}
\end{center}

%   \framebreak
%   \item Convexity of loss functions
% \end{itemize}

\end{vbframe}

\begin{vbframe}{Analytical Properties: Convexity}

\begin{itemize}
  % \item A function $\risket$ is convex if
  % $$
  %   \risk\left(t \thetab^{(1)} + (1 - t) \cdot \thetab^{(2)}\right) \le t \risk\left(\thetab^{(1)}\right) + (1 - t) \cdot \risk\left(\thetab^{(2)}\right)
  % $$
  % for all $0 \le t \le 1$ and all $\thetab^{(1)}, \thetab^{(2)} \in \Theta$
  % (strictly convex if the above equation is fulfilled with equality).
  \item In optimization, convex optimization problems are desirable because they have a number of conventient properties. In particular, it holds for convex problems: A local minimum of a convex function is also a global minimum. A strictly convex function has at most \textbf{one} global minimum (uniqueness).
  \begin{center}
  \includegraphics[width = 0.45\textwidth]{figure_man/convex-vs-nonconfex-landscape.png} \\
  \begin{footnotesize} Li et al., 2018, Visualizing the Loss Landscape of Neural Nets. The problem on the bottom right is convex, the others are not. \end{footnotesize}
  \end{center}
  \item In practical terms complexity means that we do not need to worry to get stuck in a local minimum during risk minimization.
  \item Note that convexity of $\risket$ does not only depend on convexity of the loss function: Convexity of $\risket$ is also determined by the choice of the hypothesis space!
  \item For example, if $\Lxy$ is convex in its second argument, and $\fxt$ is linear in $\thetab$, then $\risket$ is convex. If $L$ is not convex, $\risket$ might have multiple local minima (bad!).
\end{itemize}

\end{vbframe}

\begin{vbframe}{Analytical Properties: Convergence}

\textcolor{red}{@BB: Do we want to cover this case here in such a detailed manner? }

The choice of the loss function may also imply convergence behavior of the optimization problem.

\vspace*{0.2cm}

\textbf{Example: } Gradient descent will not converge if we minimize the Bernoulli loss for linearly separable data.

\vspace*{0.2cm}

First, we take a look at logistic regression for an almost linearly separable dataset consisting of the observations $\xv^{(1)}, \dots, \xv^{(8)}$.
\vfill

\begin{figure}
\includegraphics[width=0.9\textwidth]{figure_man/undet-problem01.png}\\
\end{figure}

Note: WLOG we estimate the
model without intercept, s.t. we can visualize the regression coefficient
$\bm{\theta}$ in 2D. Also, the symmetry of the data does not influence the generality of our conclusions.

\vspace*{0.2cm}

Because of the symmetry of the data, the direction\footnote[frame]{$\bm{\theta}$ is perpendicular to the decision boundary and points to the "1"-space.} of $\bm{\theta}$ is $\tilde\thetab := (\frac{1}{\sqrt{2}}, -\frac{1}{\sqrt{2}})^\top$.

\medskip

To find $\overline{\theta} := ||\bm{\theta}||_2$, we consider the empirical risk $\riske$ along $\tilde\thetab$:
\begin{align*}
\riske &= \sum^8_{i=1} \log \left[1 + \exp \left(-y^{(i)}\thetab^\top \xv^{(i)}\right)\right] \\
&= \underbrace{\sum^6_{i=1} \log \left[1 + \exp \left(- \overline{\theta} \left|\tilde{\thetab}^\top \xv^{(i)}\right|\right)\right]}_{=: \; f_{\text{cor}}(\overline{\theta}) \;\text{(correctly classified)}} +
\underbrace{\sum^8_{i=7}\log \left[1 + \exp \left( \overline{\theta} \left| \tilde{\thetab}^\top \xv^{(i)}\right|\right)\right]}_{=:\; f_{\text{incor}}(\overline{\theta}) \;\text{(incorrectly classified)}}.
\end{align*}

\end{vbframe}

\begin{vbframe}{Analytical Properties: Convergence}

Clearly, $f_{\text{cor}}$ / $f_{\text{incor}}$ are monotonically decreasing/increasing with rising length of $\thetab$:

\begin{figure}
\includegraphics[width=0.8\textwidth]{figure_man/undet-problem02.png}\\
\end{figure}

\begin{itemize}
\item By removing obs. 7 and 8, we get a linearly separable dataset. \\
\item This also means that we lose our "counterweight", i.e., if a parameter vector $\thetab$ is able to classify the samples perfectly, the vector $2\thetab$ also classifies the samples perfectly, with decreased risk.
\item Therefore, an iterative optimizer such as gradient descent will continually increase $\thetab$ and never halt (in theory).
\item In such cases, regularization can guarantee convergence (see chapter on regularization).
\end{itemize}

\end{vbframe}

% \begin{vbframe}{Visulizing Losses via Loss Plots}

% We call the plot that shows the point-wise error, i.e. the loss $\Lxy$ vs. the \textbf{residuals} $\eps := y - \fx$ (for regression), \textbf{loss plot}. The pseudo-residual corresponds to the slope of the tangent in $\left(y - \fx, \Lxy \right)$.

% \vspace*{0.5cm}

% \begin{figure}
% \includegraphics[width = 1\linewidth]{figure_man/loss.png}
% \end{figure}

% %<<echo=FALSE, warning=FALSE, message=FALSE, fig.height = 4>>=
% %   xx = seq(-2, 2, by = 0.01);
% %   yy = xx^2
% %   plot(xx, yy, type = "l", xlab = "y - f(x)", ylab = "L(y, f(x))")
% %   points(1, 1, col = "red")
% %   lines(x = c(1, 1), y = c(0, 1), col = "red")
% %   points(xx, 2 * xx - 1, type = "l", col = "red")
% %  @

% % We will define a similar plot for classification later on in this chapter.

% \end{vbframe}

% \section{Comparison of Loss Functions}
%
%
% \begin{vbframe}{Comparison of Loss Functions}
%
%
% \begin{center}
% \includegraphics[width = 10cm]{figure_man/2_7_loss_comparison_plot1.png} \\
% \end{center}
%
%
% \framebreak
%
%
% This plot shows the optimal constant model $\fx = \thetab$ (also \enquote{featureless predictor}) for the losses that have been discussed.
%
% \begin{center}
% \includegraphics[width = 10cm, height = 6.5cm]{figure_man/2_8_loss_comparison_constant_model_plot1.png} \\
% \end{center}
%
% \end{vbframe}

\endlecture

\end{document}


\documentclass[a4paper,12pt]{report}
\usepackage{color}
\usepackage{graphicx}
\usepackage{subfig}
\usepackage{listings}
\usepackage{media9}
\usepackage{hyperref}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{bbm}

\DeclarePairedDelimiter\abs{\lvert}{\rvert}%
\DeclarePairedDelimiter\norm{\lVert}{\rVert}%
\DeclarePairedDelimiter\inpr{\langle}{\rangle}%

\newcommand*{\horzbar}{\rule[.5ex]{2.5ex}{0.5pt}}

% \def\reals{{\rm I\!R}}
\def\reals{\mathbb{R}}
\def\integers{\mathbb{Z}}

\newtheorem{theorem}{Theorem}

\definecolor{dkgreen}{rgb}{0,0.6,0}
\definecolor{gray}{rgb}{0.5,0.5,0.5}
\definecolor{mauve}{rgb}{0.58,0,0.82}

\begin{document}

\title{ECE367: Matrix Algebra and Optimization}
\author{Aman Bhargava}
\date{September-December 2020}
\maketitle

\tableofcontents

\section{Introduction and Course Information}

This document offers an overview of the ECE367 course. They comprise my condensed course notes for the course. No promises are made relating to the correctness or completeness of the course notes. These notes are meant to highlight difficult concepts and explain them simply, not to comprehensively review the entire course.

\paragraph{Course Information}
\begin{itemize}
\item Professor: Stark C. Draper
\item Course: Engineering Science, Machine Intelligence Option
\item Term: 2020 Fall
\end{itemize}

\chapter{Vector Space Review}

\section{Basic Terminology}

\paragraph{Subspaces: } Set of vectors closed under addition and scaling.
\paragraph{Span: } $\text{span}(S)$ is the set of linear combinations of set $S$.
\paragraph{Linear Independence: } You cannot express one element of a set as a linear combination of the others.
\begin{equation}
\sum \alpha_i v^{(i)} = 0\,\,\,\text{iff }\alpha_i =0 \forall i\in[m]
\end{equation}

$\exists \text{ unique representation for any vector } \in \text{ span}(V)$

\paragraph{Basis: } Set $B$ is a basis iff:
\begin{enumerate}
\item $B$ is linearly independent with $d$ elements ($d$ is the number of dimensions of the space).
\item $\text{Span}(B) = V$.
\end{enumerate}

\paragraph{Dimension: } $\text{Dim}(V)$ is the \textit{cardinality} of any $\text{Basis}(V)$.

\paragraph{Cardinality: } $\text{cardinality}(S)$ is the \textbf{number of elements} in set $S$.

\paragraph{Direct Sum: }
\begin{itemize}
\item Let $W_1, W_2$ be subspaces of $V$.
\item Let $U = W_1 + W_2 = \{w_1 + w_2 | w_1 \in W_1, w_2 \in W_2\}$.
\item Then $\text{dim}(U) = \text{dim}(W_1) + \text{dim}(W_2) - \text{dim}(W_1 \cap W_2)$
\item \textbf{Direct Sum: } $U = W_1 \bigoplus W_2$ iff $W_1 \cap W_2 = \{0\}$
This implies that there is a \textbf{unique choice} for a representation of a vector in $U$.
\end{itemize}

\section{Norms}

\paragraph{Norm Definition: } The norm function $\norm{\cdot}: V\to \reals$ with the conditions:
\begin{enumerate}
\item Norm is always greater than equal to zero, equalling zero only for the zero vector.
\item $\norm{u+v} \leq \norm{u} + \norm{v}$ -- \textit{Triangle inequality}, very useful in proofs.
\item $\norm{\alpha u} = \abs{\alpha} \cdot \norm{u}$
\end{enumerate}

\paragraph{$\ell_p$ Norm Family: } \begin{equation}
\norm{x}_p = (\sum_{k=1}^{n} \abs{x_k}^p)^{\frac{1}{p}}
\end{equation}
For $1 \leq p \leq \infty$.

\begin{itemize}
\item \textbf{Euclidian Norm: } $\ell_2$ norm is our conventional measure of distance in Euclidian space.
\item \textbf{$l_1$: } Is simply the sum of the absolute values in a vector.
\item \textbf{$l_\infty$: } Is the absolute value of the maximum magnitude element in the vector.
\end{itemize}

\paragraph{Norm Ball: } $B_p = \{x\in V | \norm{x}_p \leq 1\}$.

\paragraph{Level Sets: } $\{x\in V | \norm{x}_p = c, c\in\reals\}$

\paragraph{Cardinality Function: } $\text{card}(x) = \sum_{k=1}^{n} \mathbbm{1}\{x_k \neq 0\}$ \textit{i.e. the number of non-zero elements}.

\section{Inner Products}

\paragraph{Inner Product Definition: } $\inpr{\cdot,\cdot}: x, y\in\mathbb{C}^n \to \mathbb{C}$, as \begin{equation}
\inpr{x, y} = x^T\bar{y} = \sum_{k=1}^{n} x_k \bar{y}_k
\end{equation}
Where $\bar{y}$ is the complex conjugate of $y$.

\paragraph{Cauchy-Schwartz: } $\abs{\inpr{x, y}} \leq \norm{x}_2 \norm{y}_2$ holds for \textbf{all} inner product spaces.

\paragraph{Holder's Inequality: } $\abs{\inpr{x, y}} \leq \norm{x}_p \norm{y}_q$ under the conditions $p,q \geq 1$ AND $\frac{1}{p} + \frac{1}{q} = 1$.

\paragraph{Inner Product Space: } Space where the inner product \textbf{induces} a norm (somewhat hand-wavey definition).

\paragraph{Induced Norm: } For vector space $\mathbb{C}^n$: $\norm{x}_2 = \sqrt{\inpr{x,x}}$.

\section{Projection}

\paragraph{Goal: } Find vector $y^*$ for some vector $x$ such that \begin{equation}
y^* = \Pi Y(x) = \text{argmin}_{y\in Y} \norm{y-x}
\end{equation}
Where
\begin{itemize}
\item $Y$ is the set of vectors from which you choose your $y^*$.
\item $\Pi$ is the projection operator.
\item You are trying to choose optimal $y^*$.
\end{itemize}

\subsection{Projection onto Subspace}

\begin{theorem} \textbf{Projection onto multidimensional subspaces: } Let $x\in V$ and $S \subseteq V$. Then there exists unique $y^* \in S$ such that

\begin{equation}
y^* = \text{argmin}_{y\in S} \norm{x-y}
\end{equation}

\begin{itemize}
\item We let $S = \text{span}(\{v^{(1)}, ..., v^{(n)}\})$.
\item We can now write $y^* = \sum_{i=1}^{d} \alpha_i v^{(i)}$.
\end{itemize}

\begin{equation}
\begin{bmatrix}
\inpr{v^{(1)}, v^{(1)}} & ... & \inpr{v^{(1)}, v^{(d)}}\\
\vdots & \ddots & \vdots \\
\inpr{v^{(d)}, v^{(1)}} & ... & \inpr{v^{(d)}, v^{(d)}}
\end{bmatrix}
\begin{bmatrix}
\alpha_1 \\
\vdots \\
\alpha_d
\end{bmatrix} =
\begin{bmatrix}
\inpr{v^{(1)}, x} \\
\vdots \\
\inpr{v^{(d)}, x}
\end{bmatrix}
\end{equation}

\end{theorem}

Note that for an orthonormal set of vectors $V$, the lefthand matrix becomes the identity matrix. Therefore, one can project onto an othonormal set by taking the inner products between the set elements and the vector in question.

\paragraph{Gram-Schmidt Procedure: } How to make an orthonormal basis out of a set (go from $v^{(i)}$ to orthonormal $z^{(i)}$).

\begin{enumerate}
\item Normalize $v^{(1)} \to z^{(1)}$.
\item Let $u^* = \inpr{v^{(2)}, z^{(1)}}z^{(1)}$.
\item $z^{(2)} = \frac{v^{(2)} - u^*}{\norm{v^{(2)} - u^* }}$.
\item For higher dimension: Repeat steps before where the $m$th vector $z^{(m)}$ is calculated as
$$z^{(m)} = \frac{w^{(m)}}{\norm{w^{(m)}}}$$
Where $w^{(m)} = v^{(m)} - \sum_{i=1}^{m-1} \inpr{v^{(m)} , z^{(i)}} z^{(i)}$
\end{enumerate}

\paragraph{QR Decomposition: } When you write a matrix as the product of an orthonormal basis and an upper triangular matrix.

\begin{equation}
A = \begin{bmatrix}
\vdots & & \vdots \\
v^{(1)} & \dots & v^{(m)} \\
\vdots & & \vdots
\end{bmatrix} =
\begin{bmatrix}
\vdots & & \vdots \\
z^{(1)} & \dots & z^{(m)} \\
\vdots & & \vdots
\end{bmatrix}
\begin{bmatrix}
r_{11} & r_{12} & \dots & r_{1m} \\
0 & r_{22} & & \vdots \\
\vdots & & \ddots & \\
0 & \dots & & r_{mm} \\
\end{bmatrix}
\end{equation}

Note that this is a representation of taking the Gram-Schmidt procedure in reverse (i.e. the inverse of the $R$ matrix corresponds to the process described above).

\subsection{Affine Projection}

\paragraph{Affine Set Definition: } Affine set is a shift or translation of subspace $S \subseteq \reals^n$ by some vector $x^{(0)} \in \reals$. $$A = x^{(0)} + S = \{x^{(0)} + u | u\in S\}$$

\begin{enumerate}
\item Given $x,A, x^{(0)}$. You want to project $x$ onto affine set $A = S + x^{(0)}$
\item Translate $x, A$ by $-x^{(0)}$.
\item Project onto the conventional subspace using the procedure above.
\item Translate the projection back using $+x^{(0)}$.
\end{enumerate}

\chapter{Functions and Sets}

\section{Basic Terminology}

\paragraph{Prototypical Function: } \begin{equation}
f:X\to Y
\end{equation}
Means that $f$ is a function that maps members of set $X$ to members of set $Y$.

\paragraph{Sets Related to Functions: }
\begin{itemize}
\item graph $f = \{(x, f(x)) \in \reals^{n+1} | x \in \reals^n\}$
\item epigraph $f = \{(x, t) \in \reals^{n+1} | x \in \reals^n, t \geq f(x)\}$
\item level sets: $c_f(t) = \{x\in \reals^n | f(x) = t\}$
\item sublevel sets: $l_f(t) = \{x\in\reals^n | f(x) \leq t\}$
\end{itemize}

\section{Linear and Affine Functions}

\paragraph{Linearity Conditions: }
\begin{enumerate}
\item \textbf{Homogeneity: } $f(\alpha x) = \alpha f(x)$.
\item \textbf{Additivity: } $f(x+y) = f(x) + f(y)$.
\end{enumerate}

\textbf{Resultant property: } Super position; $f(\sum_{i=1}^{n} \alpha_i x^{(i)} ) = \sum_{i=1}^{n} \alpha_i f(x^{(i)})$.

\paragraph{Affine Function: } $f: \reals^n \to \reals$ is affine if $\tilde f(x)$ (defined below) is linear

\begin{equation}
\tilde f(x) = f(x) - f(0)
\end{equation}

\textbf{Useful property: } $f:\reals^n\to\reals$ is affine \textbf{iff} $\exists\, a,b \in \reals^n)$ such that\begin{equation}
f(x) = a^T x + b
\end{equation}

Which also implies that all linear functions can be expressed as $f(x) = a^T x$.

\section{Affine Approximation}

\paragraph{Affine Approximation Definition: } The affine approximation for function $f:\reals^n\to\reals$ about point $\bar{x}$ is: \begin{equation}
f(x) \approx f(\bar{x}) + \nabla f(\bar{x})^T(x-\bar{x})
\end{equation}

\begin{itemize}
\item \textbf{First-order increment: } $\nabla f(\bar{x})(x-\bar{x})$.
\item First-order increment equal to zero (or any constant value value) results in a \textbf{hyperplane}.
\end{itemize}

\section{Chain Rule on Gradients}

\textbf{Given }
\begin{enumerate}
\item $g:\reals^n\to\reals^m$
\item $f:\reals^m\to\reals$
\item $\phi(x) = f(g(x)), x\in\reals^n$
\end{enumerate}

\textbf{Goal: } Find $\nabla\phi(x)$.

\begin{equation}
\nabla\phi(x) =
\begin{bmatrix}
\frac{\partial \phi}{\partial x_1} \\
\vdots \\
\frac{\partial \phi}{\partial x_n}
\end{bmatrix} =
\begin{bmatrix}
\frac{\partial g_1(x)}{\partial x_1} & \frac{\partial g_2(x)}{\partial x_1} & \dots & \frac{\partial g_m(x)}{\partial x_1} \\
\vdots & \ddots & & \vdots \\
\frac{\partial g_1(x)}{\partial x_n} & \dots & & \frac{\partial g_m(x)}{\partial x_n} \\
\end{bmatrix}
\nabla f(g(x))
\end{equation}

\chapter{Matrices}

\section{Matrix Introduction}

\textit{This chapter began with a basic review of matrix algebra that will not be re-covered here.}

\paragraph{Outer Product View of Matrix Multiplication: }

\begin{equation}
AB =
\begin{bmatrix}
	\vrule & & \vrule \\
	a^{(1)} & \dots & a^{(m)} \\
	\vrule & & \vrule
\end{bmatrix}
\begin{bmatrix}
	\horzbar & b^{(1)T} & \horzbar \\
	 & \vdots & \\
	\horzbar & b^{(m)T} & \horzbar
\end{bmatrix} =
\sum_{k=1}^{m} a^{(k)} (b^{(k)})^T
\end{equation}

\paragraph{Important Matrices: }
\begin{itemize}
\item \textbf{Square}
\item \textbf{Symmetric: } $A_{ij} = A_{ji}$; $A = A^T$
\item \textbf{Diagonal: } $A_{ij} = 0$ for $i \neq j$
\item \textbf{Identity}.
\item \textbf{Upper Triangular:} Square matrix with condition $A_{ij} = 0$ if $i > j$.
\end{itemize}

\subsection{Matrix Vector Spaces}

\paragraph{Inner Product: } $A, B \in \mathbb{C}^{m\times n}$
\begin{equation}
	\inpr{A, B} = \text{trace}(A^T B) = \text{trace}(BA^T) = \sum_{i}^{} \sum_{j}^{} [A]_{ij} [B]_{ij}
\end{equation}

\paragraph{Trace Operator: } For matrix $A\in \mathbb{C}^{n\times n}$:

\begin{equation}
		\text{tr}(A) = \sum_{i=1}^{n} [A]_{ii} = [A]_{11} + ... + [A]_{nn}
\end{equation}

\paragraph{Frobenius Norm: } $\norm{\cdot}_F$ where
\begin{equation}
\norm{A}_F \equiv \sqrt{\inpr{A,A}}
\end{equation}
Which is the equivalent of the $\ell_2$ norm of the vector made by flattening $A$.

These definitions and properties make proofs a lot easier when you deeply understand them.

\section{Function Approximations with Matrices}

\paragraph{Second Order (Quadratic) Approximations: } For map $f:\reals^n\to \reals^m$, we can approximate by
\begin{equation}
f(x)\approx f(\bar{x}) + \nabla f(\bar{x})^T (x-\bar{x}) + \frac{1}{2} (x-\bar{x})^T \nabla^2 f(\bar{x}) (x-\bar{x})
\end{equation}

Where $\nabla^2 f(\bar{x})$ is the \textbf{Hessian} of $f$ about point $\bar{x}$ and is defined as:
\begin{equation}
\nabla^2 f(\bar{x}) =
\begin{bmatrix}
\frac{\partial^2 f}{\partial x_1^2} & \dots & \frac{\partial^2 f}{\partial x_1\partial x_n} \\
\vdots & \ddots & \vdots \\
\frac{\partial^2 f}{\partial x_n\partial x_1} & \dots & \frac{\partial^2 f}{\partial x_n^2} \\
\end{bmatrix}
\end{equation}

Conveniently, the Hessian matrix is also symmetric.

\section{Matrices as Linear Maps}

\paragraph{Linear Map Definition: } $\text{map } f:V\to W$ is linear if
\begin{equation}
	f(\alpha_1 x^{(1)} + \alpha_2 x^{(2)} ) = \alpha_1 f(x^{(1)}) + \alpha_2 f(x^{(2)})
\end{equation}

\paragraph{Key Finding: } \textbf{ANY LINEAR MAP} can be represented by a matrix $\in \reals^{m\times n}$ for $\text{map} f : \reals^n \to \reals^m$

\subsection{Matrix Inverse}

A matrix is \textbf{NOT} invertable if $m < n$ for map represented by $A \in \reals^{m\times n}$ due to the many-to-one mapping. It is \textbf{POTENTIALLY} invertible if $m \geq n$.

\paragraph{Square Matrix Invertability: } $A\in\reals^{n\times n}$ is invertible iff there exists $B\in\reals^{n\times n}$ such that \begin{equation}
AB = BA = I
\end{equation}
If $B$ exists, it is unique and is equal to $A^{-1}$.

\textbf{Invertability conditions: } If any of these hold true, then the matrix is invertable.
\begin{enumerate}
\item $\text{det}(A) \neq 0$
\item None of $A$'s eigenvalues are equal to zero.
\end{enumerate}

\paragraph{Invertible Matrix Properties: } Let $A, B \in \reals^{n\times n}$ both be invertible. Then:
\begin{enumerate}
\item $(AB)^{-1} = B^{-1} A^{-1}$.
\item $(A^T)^{-1} = (A^{-1})^T$.
\item $\det(A) = \det(A^T) = \frac{1}{\det(A^{-1})}$
\end{enumerate}

\paragraph{Pseudo Inverse: } For non-square matrix there is no inverse, though the pseudo inverse $A^{pi}$ satisfies:
\begin{equation}
AA^{pi}A = A
\end{equation}

That is, either $AA^{pi} = I_m$ or $A^{pi}A = I_n$.

\textbf{Case I: } Tall and thin matrix $A$ with $m > n$
\begin{itemize}
\item The ``left inverse'' exists iff the \textbf{columns} of the matrix are \textbf{linearly independent}.
\item $A^{li} A = I_n$.
\item This only exists if the mapping of $A$ is unique (i.e. it is a full-rank matrix).
\end{itemize}

\textbf{Case II: } Wide and fat matrix $A$ with $m < n$
\begin{itemize}
\item The ``right inverse'' exists iff the \textbf{rows} of the matrix are linearly independent.
\item $A A^{ri} = I_m$.
\item If $A^{ri}$ exists then $A$ is ``right invertible''.
\item $A^{ri} y$ gives one of \textit{infinitely many} $x\in \reals^n$ that satisfy $Ax = y$.
\end{itemize}

\section{Orthogonal Matrices}

\paragraph{Orthogonality Conditions: } $A\in \reals^{n\times n}$ is orthogonal if:
\begin{enumerate}
\item Column vectors are all \textbf{mutually orthogonal}.
\item Column vectors are all \textbf{normalized}.
\end{enumerate}

If we let $A$ have columns $q^{(i)}$ such that $A = [q^{(1)} ... q^{(n)}]$, then

\begin{equation}
\inpr{q^{(i)}, q^{(j)}} = \begin{cases}
0, & i \neq j \\
1, & i = j
\end{cases}
\end{equation}

It follows:
\begin{itemize}
\item $A^TA = I$
\item $AA^T = I$
\item $A^T = A^{-1}$
\end{itemize}

\textbf{Necessary and sufficient condition for orthogonality: } $A^T = A^{-1}$.

\paragraph{Unitary Matrix: } If $A\in \mathbb{C}^{n\times n}$ and all of the above apply, then:

$$U^H = \text{conjugate}(U^T)$$

\paragraph{Geometric Implications of Orthogonality: }
If $Ax = y$ and $A\tilde{x} = \tilde{y}$:
\begin{itemize}
		\item $\norm{x} = \norm{y}$ -- That is, there is no \textbf{scaling} because all the columns of $A$ are normalized.
		\item $\inpr{y, \tilde{y}} = \inpr{x, \tilde{x}}$ -- that is, the \textbf{angles} between vectors are maintained.
\end{itemize}

Which implies that the transformation of $A$ can \textbf{only} be composed of (1) rotation and (2) reflection.

\section{Rank, Range, and Null Space}

Let $A\in \reals^{m\times n}$. Then $A:\reals^n \to \reals^m$ and $A^T: \reals^m \to \reals^n$.

\paragraph{Domain and Range: }
\begin{itemize}
\item \textbf{Domain: } $\text{dom}(\cdot)$ is the vector space the operator acts on.
\item \textbf{Range: } $\mathcal{R}\{\cdot\}$ is the \textbf{SPAN OF THE COLUMNS} of the matrix.
\end{itemize}

\paragraph{How to Find Basis of $\mathcal{R}(A)$: }
\begin{enumerate}
\item Manipulate $A$ into reduced row-echelon form.
\item We define \textbf{pivots} as the first non-zero entries of each row in the reduced row-echelon form.
\item Basis$(\mathcal{R}(A^T))$ are the \textbf{non-zero rows} of $A_{REF}$.
\item Basis$(\mathcal{R}(A))$ are the columns of $A$ correspondinging to the columns of the \textbf{pivots}.
\end{enumerate}

Thus $\dim(\mathcal{R}(A^T)) = \dim(\mathcal{R}(A))$

\paragraph{Nullspace of a Matrix}

\begin{equation}
\mathcal{N}(A) = \{x|Ax = 0\}
\end{equation}

It is the \textbf{orthogonal compliment} of $\mathcal{R}(A^T)$. To solve for this, you must start with $Ax = 0$ and find the implied conditions on $x$.

\paragraph{Fundamental Theorem of Linear Algebra}
\begin{enumerate}
\item $\mathcal{N}(A) \bigoplus \mathcal{R}(A^T) = \reals^n$
\item $\dim(\mathcal{N}(A)) + \text{rank}(A) = n$
\item $\mathcal{R}(A) \bigoplus \mathcal{N}(A^T) = \reals^m$
\item $\text{rank}(A) + \dim(\mathcal{N}(A^T)) = m$
\end{enumerate}

\textbf{Consequence of the Fundamental Theorem}: We can express any $x\in\reals^n$ as \begin{equation}
x = A^T y + z
\end{equation}
\begin{itemize}
\item $z\in\mathcal{N}(A)$
\item $y\in\mathcal{R}(A^T)$
\end{itemize}

\chapter{Eigendecomposition}

\section{How to Solve for Eigenthings}

$$Av = \lambda v$$
$$Av-I\lambda v = 0$$
\begin{equation}
\det(A - \lambda I) = 0
\end{equation}

Once you solve for the \textbf{characteristic polynomial roots}, you can plug them back into the initial formula and solve for the corresponding eigenvalue.

\begin{theorem}
\textbf{All} $A\in\reals^{n\times n}$ have $\geq 1$ eigenvalue.
\begin{itemize}
\item Full set of eigenvectors has the same number of eigenvectors as eigenvalues.
\item Partial set (defective matrix) has fewer eigenvectors than eigenvalues.
\end{itemize}
\end{theorem}

\paragraph{Multiplicity of Eigenvalues: } We have two types of multiplicity.
\begin{enumerate}
\item \textbf{Algebraic Multiplicity: } $AM(\lambda_i) = \mu_i$ -- number of times $\lambda_i$ is repeated in the \textbf{characteristic polynomial}.
\item \textbf{Geometric Multiplicity: } $GM(\lambda_i) = \nu_i = \dim(\mathcal{N}(A-\lambda_i I)$ -- number of eigen\textbf{vectors} corresponding to the given eigenvalue.
\end{enumerate}

$$1 \leq \nu_i = GM(\lambda_i) \leq AM(\lambda_i) = \mu_i$$

\paragraph{Eigenspace: } For each $\lambda$ there is a space $E_\lambda$:
\begin{equation}
	E_{\lambda} = \text{span}\{u|Au = \lambda u\} = \mathcal{N}(A-I\lambda)
\end{equation}
The geometric multiplicity of $\lambda$ gives the dimensionality of the eigenspace. Each $E_\lambda$ is invariant to $A$.

\section{Diagonalization}

\paragraph{Condition for Diagonalizability: } If $A$ has a \textbf{full set} of eigenvalues (i.e. distinct $\lambda_i$ for $i\in[k]$, $k \leq n$) such that $\nu_i = GM(\lambda_i) = AM(\lambda_i) = \mu_i$ for all $i \in [k]$, then $A$ is diagonalizable.

This means that we can create $n\times n$ matrix $U$ with columns equal to eigenvectors of $A$ and matrix $\Lambda= \text{diag}(\lambda_1, ..., \lambda_n)$ such that
\begin{equation}
A = U\Lambda U^{-1}
\end{equation}

\paragraph{Utility of Diagonalization: }
\begin{itemize}
\item We can take arbitrary powers of $A$ very easily now by cancelling $U$, $U^{-1}$.
\end{itemize}

\paragraph{Determinants of Diagonal Matrices}

If we can write $A = U\lambda U^{-1}$, then
\begin{equation}
|\det A| = |\prod_{i=1}^{n} \lambda_i|
\end{equation}

\chapter{Symmetric Matrices}

\section{Symmetric Matrices and Quadratic Functions}

\paragraph{Symmetric Matrix Definition: } $S^n$ is the set of all symmetric matrices and is defined as
\begin{equation}
S^n = \{A \in \reals^{n\times n} | A = A^T\}
\end{equation}

\paragraph{Quadratic Function Definition: } $q:\reals^n\to \reals$ is a quadratic function if
\begin{equation}
q(x) = x^T A x + c^T x + d
\end{equation}
where $A\in \reals^{n\times n}$, $c\in \reals^n$, $d\in\reals$.

\paragraph{Important Alternate Forms of Quadratics: }
\begin{equation}
q(x) = \frac{1}{2}x^T(A+A^T)x + c^T x + d
\end{equation}
\begin{equation}
q(x) = \frac{1}{2}
\begin{bmatrix}
	x^T & 1 \\
\end{bmatrix}
\begin{bmatrix}
	A+A^T & c \\
	c^T & 2d
\end{bmatrix}
\begin{bmatrix}
	x \\
	1
\end{bmatrix}
\end{equation}

\textbf{Important note: } The 2x2 matrix there is $\in S^{n+1\times n+1}$! This is the strong connection between quadratics and symmetric matrices.

\section{Spectral Theorem}

\textit{Tl;dr: Eigenthings are really nice and simple with symmetric matrices.}

\begin{theorem}
\textbf{Nice things about symmetrical matrices: }
Let $A\in S^n$ have $n$ (potentially non-distinct) eigenvalues. Then
\begin{enumerate}
\item $\lambda_i \in \reals$ which means that the eigenvectors are \textbf{purely real}.
\item $GM(\lambda_i) = AM(\lambda_i)$ which means there is a full set of eigenvalues $\to$ $A$ is diagonalizable!
\item All eigen spaces are \textbf{mutually orthogonal} (not just linearly independent).
\end{enumerate}
\end{theorem}

\paragraph{Spectral Decomposition: } If $A\in S^n$ then there are $n$ orthonormal eigenvectors. We let each eigenvector $u^{(i)}$ be a column in $U\in \reals^{n\times n}$. Then:
\begin{equation}
A = U\Lambda U^T
\end{equation}

\paragraph{Rayleigh Quotients: }
\begin{equation}
\lambda_{min} \leq \frac{x^T A x}{\norm{x}^2} \leq \lambda_{max}
\end{equation}

\begin{itemize}
\item The Rayliegh quotient when $x = u^{(1)}$ (the maximum $\lambda$ eigenvector) is equal to $\lambda_{max}(A)$.
\item Same goes for when $x = u^{(n)}$ with respect to the smallest eigenvalue $\lambda_n$.

\item \textbf{This connects eigenvectors and singular values.}
\end{itemize}

\paragraph{Positive Semi-Definite Matrices: } if \begin{equation}
x^T A x \geq 0
\end{equation}
For all $x\in \reals^n$, then $A\in S^n_+$ ($A$ is a positive semidefinite matrix).

\paragraph{Positive Definite Matrices: } if strict inequality holds such that
\begin{equation}
x^T A x > 0
\end{equation}
for all $x\in \reals^n$, then $A$ is a positive definite matrix ($A\in S^n_{++}$).

\begin{theorem}{Convenient things about $S_+^n$, $S_{++}^n$: }
\begin{itemize}
\item $A\in S^n_+$ iff $\lambda_i \geq 0$ for all $i\in [n]$.
\item $A\in S^n_{++}$ iff $\lambda_i > 0$ for all $i \in [n]$.
\item Any positive definite matrix is invertible. \textit{Positive semi-definite matrices are invertible iff they are positive definite}.
\end{itemize}
\end{theorem}

\section{Ellipsoids}

\paragraph{Ellipsoid definition: } An ellipsoid is a set \begin{equation}
\varepsilon = \{x \in \reals^n | (x-x^{(0)})^T P^{-1} (x-x^{(0)}) \leq 1\}
\end{equation}
Where $P\in S^n_{++}$, $x^{(0)} \in \reals^n$
\begin{itemize}
\item The condition is a quadratic function!
\item You can apply spectral decomposition to $P^{-1}$.
\end{itemize}

\paragraph{Plotting Ellipses: }
\begin{equation}
\begin{split}
1 \geq \bar{x}^T(U\Lambda U^T)^{-1} \bar{x} \\
1 \geq \tilde{x}^T \lambda^{-1} \tilde{x} \\
= \sum_{i=1}^{n} (\frac{\tilde{x}_i}{\sqrt{\lambda_i}})^2
\end{split}
\end{equation}

Where $\bar{x} = (x-x^{(0)})$, $\tilde{x} = U^T\bar{x}$.

To plot:
\begin{enumerate}
\item Generate `plot' in $\tilde{x}$ space (note the key positions).
\item Unrotate by multiplying key position vectors by $U$.
\item Un-shift by adding $x^{(0)}$
\end{enumerate}

\paragraph{Sample Mean and Covariance: } for $m$ data vectors $x^{(i)}$, $i\in[m]$:
\begin{enumerate}
\item \textbf{Sample Mean: } $\hat{x} = \frac{1}{m} \sum_{i=1}^{m} x^{(i)}$
\item \textbf{Sample Covariance: } $S = \frac{1}{m} \sum_{i=1}^{m} (x^{(i)} - \hat{x})(x^{(i)} -\hat{x})^T \in S^n$
\end{enumerate}

\textbf{Data ellipse: } $\varepsilon_\lambda = \{x\in \reals^n | (x-\hat{x})^T S^{-1} (x-\hat{x}) < \gamma\}$ where $\gamma$ is usually 1. This ellipse can help describe the geometry of the data.

\section{Matrix Square Root and Cholesky Decomposition}

\paragraph{Matrix Square Root Definition: } For any $A\in S_+^n$:
\begin{equation}
	A^{\frac{1}{2}} = U\Lambda^{\frac{1}{2}} U
\end{equation}

Where $\lambda = \text{diag}(\sqrt{\lambda_1}, ..., \sqrt{\lambda_n})$.
\begin{itemize}
\item $A^\frac{1}{2}$ is a unique PSD matrix.
\item $A$ is a PSD matrix \textbf{iff $A^{\frac{1}{2}}$ is PSD}.
\end{itemize}

\paragraph{Cholesky Decomposition: } Any PSD $A$ can be factored as
\begin{equation}
	A = B^T B
\end{equation}
Where $B = \Lambda^{\frac{1}{2}} U$ and $B^T = U\Lambda^{\frac{1}{2}}$.

$B$ also has a QR (orthogonal times upper triangular) decomposition, so:
\begin{equation}
	A = B^T B = (QR)^T(QR) = R^TQ^TQR = R^TR
\end{equation}

\textbf{To create Choleski Decomposition: }
\begin{enumerate}
\item Select $R$ such that diagonals $R_{ii} > 0$.
\item If $A$ is PD, then \textbf{there exists unique $R$} which forms the Choleski Decomposition.
\item There exist non-unique $R$ for $A$ is PSD.
\end{enumerate}

\paragraph{Solving $Ax = b$ with Choleski Decomposition: }
\begin{enumerate}
\item $A = R^TR$
\item $R^TRx = b$
\item Let $y = Rx$
\item \textbf{Forward Substitution: }
\begin{enumerate}
\item $y_1 R_{11} = b_1$ -- Everything but $y_1$ is known, so we solve for $y_1$!
\item $y_1 R_{21} + y_2 R_{22} = b_2$ -- Everything but $y_2$ is known, so we can solve for $y_2$!
\item We continue this until we know the entire $y$ vector!
\end{enumerate}
\item \textbf{Back Substitution: }
\begin{enumerate}
\item We use a similar procedure to solve from the bottom up for $Rx = y$
\end{enumerate}
\end{enumerate}

\chapter{Singular Value Decomposition}

\section{Motivation}

\begin{theorem}
We can write any linear map $A\in \reals^{m\times n}$ as
\begin{equation}
	A = U\Sigma V^T
\end{equation}
Where
\begin{itemize}
\item $U$ is $m\times m$ orthonormal matrix composed of \textbf{left singular vectors}.
\item $V$ is $n\times n$ orthonormal matrix composed of \textbf{right singular vectors}.
\item $\Sigma$ is $m\times n$ $\text{diag}(\sigma_1, ..., \sigma_r)$ singular values.
\end{itemize}
\end{theorem}

\paragraph{Utility of SVD: }
\begin{enumerate}
\item Understand rank of relevant spaces to $A$.
\item Spectral norm (direction of maximum gain).
\item Orthonormal basis for $\mathcal{N}(A), \mathcal{R}(A)$.
\item Solve systems of linear equations relating to $A$.
\end{enumerate}

\section{Steps to Creating SVD}

\paragraph{Key Starting Ideas: } For $m\times n$ real matrix $A$:
\begin{enumerate}
\item $AA^T, A^TA$ are both PSD matrices. $x^T A^TAx = \norm{Ax}_2^2$.
\item $\text{rank}(A^TA) = \text{rank}(AA^T) = \text{rank}(A)$.
\item $AA^T$ and $A^TA$ have the same $r$ eigenvalues with $r = \text{rank}(A)$.
\end{enumerate}

\paragraph{Singular Value Definition: } The singular values of $A$ are
\begin{equation}
	\sigma_i = \sqrt{\lambda_i(A^TA)} = \sqrt{\lambda_i(AA^T)} > 0
\end{equation}
Where $\lambda_i(B)$ is the $i$th eigenvalue of matrix $B$.

\paragraph{First $r$ Right Singular Values of $A$: } We take the \textbf{spectral decomposition}
of $A^TA = V\Lambda V^T$.
\begin{equation}
V =
\begin{bmatrix}
	\vrule & & \vrule & \vrule & vrule & & \vrule \\
	v^{(1)} & \dots & v^{(r)} & \vrule & v^{(r+1)} & \dots & v^{(n)} \\
	\vrule & & \vrule & \vrule & vrule & & \vrule
\end{bmatrix}
\end{equation}
\begin{itemize}
\item The first $r$ $v^{(i)}$ relate to $\lambda_i \neq 0$.
\begin{itemize}
	\item \textbf{THESE ARE OUR RIGHT SINGULAR VECTORS}.
\end{itemize}

\item They are all \textbf{mutually orthogonal}.
\item The remaining vectors are associated with $\lambda_i = 0$.
\begin{itemize}
		\item We get to choose our own arbitrary basis for $\mathcal{N}(A)$.
\end{itemize}

\end{itemize}

\paragraph{First $r$ Left Singular Values of $A$: } The left singular vectors are denoted by $u^{(i)}$ and are given by

\begin{equation}
	u^{(i)} = Av^{(i)}
\end{equation}

You could also get them by taking the spectral decomposition of $AA^T$ but you run the risk of inverting the polarity (i.e. $u^{(i)} \to -u^{(i)}$ which would mess up the entire decomposition. Just use the above formula to be safe and efficient.

Each $u^{(i)}$ are associated with the same singular values $\sigma_i = \sqrt{\lambda_i}$.

\paragraph{Compact SVD: } We can package our first $r$ right ($v$) and left ($u$) singular vectors into $V_r$ and $U_r$ respectively to decompose $A$ as follows:

\begin{equation}
	A = U_r \Sigma V_r^T
\end{equation}
Where $\Sigma = \text{diag}(\sigma_1, ..., \sigma_r)$.

\paragraph{Full SVD: } To get the `Full' SVD, we add vectors to $U_r, V_r$ such that they form bases for $\reals^m, \reals^n$ respectively.

\begin{itemize}
\item $u^{(i)} \in \mathcal{N}(A^T)$ for all $i \in \{r+1, ..., m\}$
\item $v^{(i)} \in \mathcal{N}(A)$ for all $i\in \{r+1, ..., n\}$
\item Each of the above sets are \textbf{orthonormal bases} for their respective null spaces.
\end{itemize}

Now we have full SVD:
\begin{equation}
A = U\tilde{\Sigma}V^T
\end{equation}
Where $\tilde{\Sigma}$ is the original sigma padded with zeros such that it is an $m \times n$ matrix.
\begin{equation}
\tilde{\Sigma} =
\begin{bmatrix}
\Sigma & \dots & 0 \\
\vdots & \ddots & \vdots \\
0 & \dots & 0
\end{bmatrix}
\in \reals^{m\times n}
\end{equation}

\section{Principal Component Analysis}

\paragraph{Goal: } Get minimum error when you project data vectors onto some affine set.
\paragraph{Approach: } We can optimize the variance of projected data points onto a unit vector by aligning it such that it is an \textbf{eignevalue} of the covariance matrix $\frac{1}{m} X^TX$

\section{SVD Matrix Properties}

\paragraph{Fundamental Subspaces from SVD: } Assuming that $A = U\tilde{\Sigma}V^T$, $U = [U_r, U_{nr}]$, $V = [V_r, V_{nr}]$:
\begin{enumerate}
\item Basis of $\mathcal{R}(A)$: columns of $U_r$.
\item Basis of $\mathcal{N}(A)$: columns of $V_{nr}$.
\item Basis of $\mathcal{N}(A^T)$: columns of $U_{nr}$.
\item Basis of $\mathcal{R}(A^T)$: columns of $V_r$.
\end{enumerate}

\section{Moore-Penrose Pseudo-Inverse}

\paragraph{MP Pseudo-Inverse Definition: } For rank $r$ matrix $A\in \reals^{m\times n}$ with SVD $A = U\tilde{\Sigma}V^T$,
\begin{equation}
	A^\dagger = V\tilde{\Sigma}^\dagger U^T = V_r\Sigma^{-1}U_r^T \in \reals^{n\times m}
\end{equation}

Where

\begin{equation}
\tilde{\Sigma}^\dagger =
\begin{bmatrix}
	\Sigma^-1 & \dots & 0 \\
	\vdots & \ddots & \vdots \\
	0 & \dots & 0 \\
\end{bmatrix}
\end{equation}

And $\Sigma^{-1} = \text{diag}(\frac{1}{\sigma_1}, ..., \frac{1}{\sigma_r})$

\textbf{Recall: } $B$ is a pseudo inverse of $A$ if $ABA = A$. Moore-Penrose gives proper inverse (true, left, right) given the dimensions of the matrix $A$.

\chapter{Systems of Linear Equations}

\section{Mathematical Setup}

\paragraph{General Form:}
\begin{equation}
		Ax = y
\end{equation}

\begin{itemize}
\item $A\in \mathbb{R}^{m\times n}$ is a set of \textbf{known} matrix coefficients.
\item $x\in \mathbb{R}^n$ is the \textbf{unknown} vector we want to solve for.
\item $y\in \mathbb{R}^m$ is the \textbf{known} constraint vector.
\end{itemize}

It is possible that there are no solutions, one solution, or infinite solutions.

\paragraph{Existence of Solutions: } Since $Ax \in \mathcal{R}(A)$, the solution only exists for $y\in \mathcal{R}(A)$.

\paragraph{Simple Test for Existence: } If
\begin{equation}
\text{Rank}([A\,\,y]) = \text{Rank}(A)
\end{equation}
Then the solution exists. Otherwise, it doesn't (in this case, $y$ adds to the column space and is therefore not in $\
mathcal{R}(A)$).

\paragraph{Simple Test for Uniqueness: } If $\mathcal{N}(A) = \{\emptyset\}$, then the solution is unique.

\paragraph{Consistent Solution to Least Squares: } Using the \textbf{Moore-Penrose pseudo inverse} $A^\dagger$, we have
\begin{equation}
x^* = A^\dagger y
\end{equation}

This will always work, and in the underdetermined case, will produce the solution that minimizes $\norm{x^*}_2$.

\section{Over, Under, and Uniquely Determined Least Squares}

\begin{theorem}{Normal Equation:}
The optimal solution $x^*$ to any (unconstrained) least squares problem satisfies
\begin{equation}
A^T A x^* = A^T y
\end{equation}
\end{theorem}

\subsection{Overdetermined}

For ``tall and thin'' matrix $A\in \mathbb{R}^{m\times n}$ with $m > n$, we have the dimension of the column space of $A$ (or $\mathcal{R}(A)$) limited by $n$. Since the output of $Ax$ is in $\mathbb{R}^n$, \textbf{most of the possible $y$ vectors will not be in $\mathcal{R}(A)$}.

Therefore, we are forced to search for the \textbf{best solution:}
\begin{equation}
x^* = \text{arg min}_{x\in \mathbb{R}^n} \norm{Ax-y}
\end{equation}

Which looks an awful lot like the definition of \textbf{projection} of $y \to \text{column span}(A)$. We use the \textbf{normal equation} (seen above) to solve for this.

\paragraph{Solving the Normal Equation}

To solve $A^T A x^* = A^T y$ we have several cases:

\begin{enumerate}
\item $A^T A$ is invertible (i.e. $A$ is full rank). In this case,
\begin{equation}
x^* = (A^T A)^{-1} A^T y
\end{equation}
and we can project $y$ onto the columns of $A$ to get the \textit{approximation of $y$} we achieve.
\begin{equation}
\hat y^* = A x^*
\end{equation}

\item $A^T A$ is \textbf{degenerate} (i.e. not full rank). In this case,
\begin{itemize}
\item One or more ccolumns of $A$ can be removed without detracting from $\mathcal{R}(A)$ since columns are not linearly independent due to degeneracy.
\item \textbf{Procedure:} Remove redundant columns from $A$ and follow case 1 to generate $x^*$ and $\hat y^*$.
\end{itemize}
\end{enumerate}

\paragraph{Using SVD to Solve Over Determined Least Squares: } $A = U_r \Sigma V_r^T$ where $U_r$ is a basis for the column space of $A$. Then

\begin{equation}
\hat y^* = \sum_{i=1}^{r} \inpr{u^{(i)}, y} u^{(i)}
\end{equation}

\subsection{Uniquely Determined}

If the solution is uniquely determined, $A$ has full rank, is square, and therefore has an inverse.
\begin{equation}
x^* = A^{-1} y
\end{equation}

\subsection{Underdetermined}

For $A\in \mathbb{R}^{m\times n}$, we have a ``short and fat'' matrix. In other words, $m < n$. Therefore, we have fewer constraints than we have manipulable variables. We have a \textbf{choice} between infinite $x^*$.

\paragraph{Convention:} We use the \textit{shortest} $x^*$ vector (i.e. we minimize $\norm{x^*}_2$.)

Our infinite set of possible $x^*$ can be represented by the affine set where $\bar x$ satisfies $A\bar x = y$.
\begin{equation}
\mathcal{A} = \bar x + \mathcal{N}(A)
\end{equation}

We can then project $y\to A$.

\paragraph{Solution to Underdetermined Case: }
\begin{equation}
x^* = A^T(AA^T)^{-1} y
\end{equation}

With the norm of $x^*$ given by the following
\begin{equation}
\norm{x^*}_2^2 = \sum_{i=1}^{r} \frac{1}{\sigma_i^2} \inpr{u^{(i)}, y}^2
\end{equation}

Where $A = U_r \Sigma V_r^T$ is the SVD of $A$.

\section{Regularized Least Squares}

The idea is to introduce another term to the ``cost function'' for least squares. We \textit{penalize} the solution $x^*$, generally based on some measure of its magnitude (e.g. an $\ell_p$ norm, a weighted sum, etc.)

\subsection{$\ell_2$ Regularization}

\begin{equation}
x^* = \text{arg min}_{x\in \mathbb{R}^n} \norm{y-Ax}_2^2 + \gamma \norm{x}_2^2
\end{equation}

\paragraph{Solving $\ell_2$ Regularized Least Squares:}
\begin{enumerate}
\item Let
\begin{equation}
\bar A =
	\begin{bmatrix}
		A \\
		\sqrt \gamma I_{n\times n}\\
	\end{bmatrix}
\end{equation}
\begin{equation}
\bar y \in \mathbb{R}^n = \begin{bmatrix}
y \\
0 \\
0 \\
\vdots
0 \\
\end{bmatrix}
\end{equation}

\item Solve as before, knowing that
\begin{equation}
\norm{Ax-y}_2^2 + \gamma \norm{x}_2^2 \equiv \norm{\bar A x - \bar y}_2^2
\end{equation}
\begin{equation}
x^* = (\bar A^T \bar A)^{-1} \bar A^T \bar y = (A^T A + \gamma I)^{-1} A^T y
\end{equation}
\end{enumerate}

\section{Tikhonov Regularization}

\textit{Also known as ridge regression, weight decay.}

\paragraph{Governing Formula: } We introduces matrices $W_1, W_2$ to indicate ``priority'' of minimizing different error terms (and linear combinations of error terms) and variable $x^{(0)}$ to represent the ``zero effort point'' as follows:
\begin{equation}
\min_x \norm{W_1(Ax-y)}_2^2 + \norm{W_2 (x-x^{(0)})}_2^2
\end{equation}
\begin{itemize}
\item $W_1 \in S_{++}^n$: How much do we weigh residuals (i.e. indices of $Ax-y$)?
\item $W_2$: How much do we weigh deviations of $x^*$ from the \textbf{zero effort point} $x^{(0)}$?
\end{itemize}

\paragraph{Solution: }
We find algebraic equivalence to the following unregularized least squares problem:
\begin{equation}
\min_x \norm{\bar A x - \bar y}_2^2
\end{equation}

Where
\begin{equation}
\bar A = \begin{bmatrix}
W_1 A \\
W_2 \\
\end{bmatrix}
\end{equation}

\begin{equation}
\bar y = \begin{bmatrix}
W_1 y \\
W_2 x^{(0)} \\
\end{bmatrix}
\end{equation}

\section{Theoretical Aspects of Least Squares}

For the solution of regularized least squares,
\begin{equation}
\min_x \norm{Ax - y}_2^2 + \gamma \norm{x}_2^2
\end{equation}

Our objective function can be re-written w.r.t. $x$ as the following sum of constants and quadratics:
\begin{equation}
\norm{Ax^*_{LS} - y}_2^2 + (x - x_{LS}^*)^T A^T A (x - x_{ls}^*) + \gamma x^T x
\end{equation}

One can think of it as two ``gravity wells'' or ``potential energy wells'' that pull the optimal solution $x^*$ towards them. One is centered at the solution to the un-regularized least squares problem and the other at the \textbf{zero effort point} (in the above equations, that is the origin). Since they are quadratic cost surfaces, they actually do have very similar to shape as gravity wells (other than their eccentricity).

\subsection{$\ell_1$ Regularization}

LASSO = ``Least Absolute Shrinkage and Selection Operator'' is the term for $\ell_1$ regularization. It generally produces sparsity in approximations.

\subsection{SVD Connection}

\textit{After much symbol-shunting of $\ell_2$ regularized least squares,}
\begin{equation}
y^* = \sum_{i=1}^{r} \frac{\sigma_i^2}{\sigma_i^2 + \gamma} \inpr{u^{(i)}, y} u^{(i)}
\end{equation}

\chapter{Linear Programming}

While linear and quadratic programs are numerical problems (i.e. there are no closed form solutions), they are such nice, well developed fields of study that any problem you can formulate as an LP or QP can be broadly considered ``solved''.

\section{Definition and Notation}

\begin{theorem}{Linear Program Form: }
\begin{align}
\text{arg min}_{x\in \mathbb{R}^n} \,& c^T x + d \\
\text{such that } & Gx \leq h \\
& Ax = b
\end{align}
\begin{itemize}
\item $f_0(x) = c^T x + d$ is the scalar linear objective function.
\item $Gx \leq h$ and $Ax = b$ are inequality and equality constraints respectively and lead to the \textbf{feasible set}.
\end{itemize}
\begin{equation}
p^* = \min_{x\in \mathbb{R}^n} c^T x + d
\end{equation}
Where constraints are satisfied is the minimum cost. If the feasible set is empty, we have $p^* = \infty$. If you can make it arbitrarily small, then $p^*$ DNE.
\end{theorem}

\section{Objective Function}

\begin{equation}
f_0(x) = c^T x + d
\end{equation}

This is simply a hyperplane. Therefore, we can minimize to $-\infty$ by taking taking $x = \lim_{\alpha\to\infty} \alpha(-c)$.
If $c=\vec 0$, then $x$ does not matter and the arg min is the entirety of $\mathbb{R}^n$.

\paragraph{Key Lesson: } To minimize the objective, we want to move $x$ in the $-c$ direction.

\section{Feasible Set}

\paragraph{Equality Constraints: } $Ax = b$ boils down to a bunch of intersecting hyperplanes. If we let the rows of $A$ be $\alpha^{(i)}$, then
\begin{equation}
x\in \bigcup\limits_{i=1}^{q}\{x | \inpr{\alpha^{(i)}, x} = b_i \}
\end{equation}

\section{Inequality Constraint} $Gx \leq h$ boils down to a bunch of intersecting \textbf{half spaces}. Letting rows  of $G$ be $g^{(i)}$, then
\begin{equation}
x\in \bigcup\limits_{i=1}^q \{ x | \inpr{g^{(i)}, x} \leq h_i \}
\end{equation}
Note that the normal vectors to the half space boundaries contained as rows of $G$ point \textbf{outward} from the feasible set.

\paragraph{Feasible Set: } The intersection of the sets described by the two constraints if $\mathcal{F}$, the \textbf{feasible set}. It is a \textbf{polytope} (i.e. $n$-dimensional polygon/polyhedron generalization).

\section{Solutions to Linear Programs}

\paragraph{Solution Possibilities: }
\begin{enumerate}
\item $x^*$ is unique and $p^*$ is finite. \textbf{The optimum is at a vertex} of the polytope.
\item $x^*$ is not unique and $p^*$ is finite. \textbf{The optimum is at an edge of facet} of the polytope.
\item $x^*$ does not exist:
\begin{itemize}
\item $\mathcal{F} = \emptyset$ is an \textbf{infeasible LP}. $p^* = \infty$.
\item $\mathcal{F}$ is unbounded with no minimum. $p^* = \infty$.
\end{itemize}
\end{enumerate}

\subsection{Simplex Algorithm}
The basic idea is to traverse the edges of the LP feasible set.
\begin{enumerate}
\item Start at a vertex within the feasible set.
\item Identify an edge $E$ of the feasible set where the angle $\angle c E$ is \textbf{obtuse} (i.e. you're decreasing cost by going along $E$).
\item Move along the edge as far as you can until you bump into a constraint or another vertex.
\item Repeat until no edge available has $\angle cE > 90^o$.
\end{enumerate}

\subsection{Other Forms for LP}

\paragraph{Inequality Form:}
\begin{align}
\min_x & c^T x + d \\
\text{such that } & Gx \leq h \\
& Ax \leq b \\
& -Ax \leq -b
\end{align}

\paragraph{Standard Form}
\begin{align}
\min_x \,& \tilde c^T \tilde x + \tilde d \\
\text{such that }& \tilde A \tilde x = \tilde b
& \tilde x \geq 0
\end{align}

\begin{itemize}
\item $Gx \leq h$ becomes $Gx + s = h$ with $s \geq 0$ are ``slack variables'' (common technique for formulating problems).
\item $x \to x^+ - x^-$ such that $x^+_i, x^-_i \geq 0$ for all $i \in [n]$.
\item $\tilde c^T = [c^T\,\, -c^T\,\, 0]$.
\item $\tilde d = d$.
\begin{equation}
\tilde x = \begin{bmatrix}
x^+ \\
x^- \\
s
\end{bmatrix}
\end{equation}
\begin{equation}
\tilde A = \begin{bmatrix}
G & -G & I \\
A & -A & 0 \\
\end{bmatrix}
\end{equation}
\begin{equation}
\tilde b = \begin{bmatrix}
h \\
b
\end{bmatrix}
\end{equation}
\end{itemize}

\chapter{Quadratic Programs}

\section{QP Definitions}

A quadratic program has the same feasible set definition as a linear program, but has a \textbf{quadratic} objective function w.r.t. $x$.

\begin{theorem}{Quadratic Program Definition:}
\begin{align}
\text{(arg) min }_{x\in \mathbb{R}^n} & \frac{1}{2} x^T H x + c^T x + d \\
\text{such that } & Ax = b \\
& Gx \leq h
\end{align}
\end{theorem}

\section{Geometry}

\paragraph{No Loss in Generality } for $H\in S^n$. For the objective $f_0$,
\begin{equation}
f_0 = \frac{1}{2} x^T H x + c^T + d
\end{equation}

We have $x^T H x = \frac{1}{2} x^T (H+H^T) x$. Therefore, \textbf{spectral decomposition} is valid for all quadratic objective functions.

\paragraph{Properties of $H$}
\begin{enumerate}
\item $H$ is \textbf{diagonalizable} ($H = U\Lambda U$ with orthonormal columns in $U$).
\item Eigenvalues are \textbf{real} and therefore can be ordered.
\item Eigenvectors can be chosen to be \textbf{orthogonal}.
\end{enumerate}

\paragraph{Properties of the Objective Function}
\begin{enumerate}
\item If $H \notin S_+^n$ then $p^* = -\infty$.
\begin{itemize}
\item The quadratic ``bowl'' points downward, so if we follow vector $c$, we can find an unconstrained minimum.
\end{itemize}

\item If $H\in S_{++}^n$ then $p^*$ exists and can be found (i.e. is not $\pm \infty$).
\begin{itemize}
\item No directions of infinite descent because ``bowl'' points up and asymptotically dominates the linear components.
\item The objective if a \textbf{convex function} in this case.
\end{itemize}

\item If $H\in S_+^n \notin S_{++}^n$, we have two cases:
\begin{enumerate}
\item If $c\in \mathcal{R}(H)$: $p^*$ exists and can be found.
\item If $c\notin \mathcal{R}(H)$: $p^* = -\infty$
\end{enumerate}
\end{enumerate}

\section{Quadratically Constrained Quadratic Programs}

\begin{theorem}{Form for QCQP's: }
\begin{align}
\min_{x\in \mathbb{R}^n} & \frac{1}{2} x^T H_0 x + c_0^T x + d_0 \\
\text{such that }& \frac{1}{2} x^T H_i x + c_i^T x + d_i \leq 0, & \forall i\in [m] \\
& \frac{1}{2} x^T \tilde H_i x + \tilde c_i^T x + \tilde d_i = 0, & \forall i\in [q] \\
\end{align}
\end{theorem}

Though the intersections of these constraints may lead to some non-convex feasible sets, we generally focus on \textbf{convex} QCQP's.

\begin{theorem}{Form for Convex QCQP's: }
\begin{align}
\min_{x\in \mathbb{R}^n} & \frac{1}{2} x^T H_0 x + c_0^T x + d_0 \\
\text{such that }& \frac{1}{2} x^T H_i x + c_i^T x + d_i \leq 0, & \forall i\in [m] \\
& \tilde c_i^T x + \tilde d_i = 0, & \forall i\in [q] \\
\end{align}

Where $H_0\in S_+^n$ and each $H_i \in S_+^n$.
\end{theorem}

\paragraph{QCQP Feasible Sets}
\begin{itemize}
\item These are generally the intersection of \textbf{half spaces, paraboloids, and ellipsoids}.
\item Paraboloids are kind of like half spaces but have a quadratic instead of a flat plane separating the spaces.
\end{itemize}

\chapter{Convexity, Sets, and Hulls}

\section{Hulls}

A hull is an infinite set of points that can be defined with respect to a set of vectors. For the following examples, we let $\mathcal{P}$ be some set of $m$ vectors in $\mathbb{R}^n$.

\begin{theorem}{Linear Hull Definition: }
\begin{equation}
\text{Lin}(\mathcal{P}) = \{x | x = \sum_{i = 1}^{m} \lambda_i x^{(i)}, \lambda_i \in \mathbb{R}\}
\end{equation}
\end{theorem}
\begin{itemize}
\item Which is exactly the same as the \textbf{span} of set $\mathcal P$.
\end{itemize}

\begin{theorem}{Affine Hull Definition: }
\begin{equation}
\text{Aff}(\mathcal{P}) = \{x | x = \sum_{i = 1}^{m} \lambda_i x^{(i)}, \lambda_i \in \mathbb{R}, \sum_{i=1}^{m} \lambda_i = 1 \}
\end{equation}
\end{theorem}
\begin{itemize}
\item All affine hulls are affine sets.
\item You can symbol-shunt the definition to make one of the vectors the \textbf{offset} vector and the rest create a subspace.
\item Affine hull can be written, for any $x^{(k)}$
\begin{equation}
x^{(k)} + \text{span}(\{x^{(i)} - x^{(k)} | \forall i\in [m], i \neq k\})
\end{equation}
\end{itemize}

\begin{theorem}{Convex Hull Definition: }
\begin{equation}
\text{Co}(\mathcal{P}) = \{x | x = \sum_{i = 1}^{m} \lambda_i x^{(i)}, \lambda_i \in \mathbb{R}, \lambda_i \geq 0, \sum_{i=1}^{m} \lambda_i = 1 \}
\end{equation}
\end{theorem}

\begin{itemize}
\item Given two points in a convex hull, all the points on the line connecting the two points are inside of the hull, too.
\item Very convenient property.
\end{itemize}

\begin{theorem}{Conic Hull Definition: }
\begin{equation}
\text{conic}(\mathcal{P}) = \{x | x = \sum_{i = 1}^{m} \lambda_i x^{(i)}, \lambda_i \in \mathbb{R}, \lambda_i \geq 0 \}
\end{equation}
\end{theorem}

\begin{itemize}
\item Can be re-written as
\begin{equation}
x = (\lambda_1 + \lambda_2)(\frac{\lambda_1}{\lambda_1 + \lambda_2} x^{(1)} + \frac{\lambda_2}{\lambda_1 + \lambda_2} x^{(2)})
\end{equation}
\begin{itemize}
\item The first sum being a ``gain'' term greater than zero and the fractions of lambda being any independent factors each between 0 and 1.
\item Boils down to any point between the two vectors ``zoomed'' anywhere from the origin to infinity.
\end{itemize}
\item $\text{conic}(\mathcal P) = \gamma \text{Convex}(\mathcal P)$ where $\gamma$ is some positive gain.
\end{itemize}

\section{Sets}

\begin{theorem}{Affine Set Definition: }
$A\subset \mathbb R^n$ is affine iff for $x, y\in A$,
\begin{equation}
\{z | z = \lambda x + (1-\lambda)y, \lambda \in \mathbb R\} \in A
\end{equation}
\end{theorem}
\begin{itemize}
\item This is algebraically the same as our initial definition.
\item It boils down to \textbf{all the points on an infinite line going through two point in the set must also be in the set}.
\end{itemize}

\begin{theorem}{Convex Set Definition: }
$C\subset \mathbb R^n$ is convex iff $\forall \,\, x, y \in C$,
\begin{equation}
\{ z | z = \lambda x + (1-\lambda)y, \lambda\in [0,1] \}\in C
\end{equation}
\end{theorem}
\begin{itemize}
\item All points on a line segment connecting two points in the set must also be in the set.
\item \textbf{Strict convexity} occurs when we have $\lambda \in (0,1)$ and all points in the line segment are in the \textbf{relative interior} of $C$.
\end{itemize}

\paragraph{Relative Interior: }
\begin{equation}
\text{relint}(\mathcal D) = \{ x\in D | B(x,r) \cap \text{aff}(\mathcal D) \subseteq \mathcal D \text{ for some } r > 0 \}
\end{equation}
Where $B(x,r) = \{y | \norm{y-x} \leq r\}$

\paragraph{Convexifying: } If set $S$ is not convex, we can create $S' = \text{Co}(S)$ to create minimum set that includes all $s\in S$ that is also convex.

\begin{theorem}{Cone Definition: }
$\mathcal K \subseteq \mathbb R^n$ is a cone iff $\forall \, x\in \mathcal K$,
\begin{equation}
\{\gamma x | \gamma \geq 0\} \in \mathcal K
\end{equation}
\end{theorem}
\begin{itemize}
\item \textbf{Note: } All conic hulls are cones but not all cones are conic hulls.
\end{itemize}

\section{Properties of Convexity}
\textit{These properties are stated without their proofs for ease of reference}
\begin{enumerate}
\item Intersection of multiple convex sets are always convex.
\item Affine (linear + offset) maps always preserve convexity.
\begin{itemize}
\item The \textbf{pre-image} of a set w.r.t. an affine map is also convex.
\end{itemize}
\item All \textbf{norm balls} are convex.
\item All \textbf{ellipsoids} are convex.
\item The set of PSD matrices $S_+^n$ is a \textbf{convex cone} in $S^n$.
\end{enumerate}

\section{Convex Functions}

\begin{theorem}{Convex Function Definition: }
A function $f:\mathbb R^n \to \mathbb R$ is \textbf{convex} iff for all $x, y \in \text{dom }f$,
\begin{equation}
f(\lambda x + (1-\lambda) y) \leq \lambda f(x) + (1-\lambda) f(y), \,\, \forall \lambda\in [0,1]
\end{equation}
With strict inequality indicating strict convexity.
\end{theorem}

\paragraph{Concavity: } When $-f$ is convex, $f$ is concave.

\begin{theorem}{Quasi-Convex Function Definition: }
If all \textbf{sublevel sets} are convex, then $f$ if quasi-convex.
\end{theorem}

\paragraph{Facts about Convex Functions: } \textit{Not proven here for ease of access.}
\begin{itemize}
\item If each $f_i$ if convex, then
\begin{equation}
f(x) = \sum_{i=1}^{m} \alpha_i f_i(x)\, | \, \alpha_i \geq 0
\end{equation}
Is also convex.

\item Given convex $f:\mathbb R^m \to \mathbb R$, $b\in \mathbb R^m$, $A\in \mathbb R^{m\times n}$, then
\begin{equation}
g(x) = f(Ax + b)
\end{equation}
Is also convex.

\item For a set of convex functions $\{f_i(x)\}$, then
\begin{equation}
f(x) = \max(f_i(x))
\end{equation}
Is also convex.
\end{itemize}

\section{Convexity by Restriction to Line}

\begin{theorem}{Restriction to Line: }
$f:\reals^n \to \reals$ is convex iff $g:\reals \to \reals$ is convex with respect to $t$ as defined below.
\begin{equation}
g(t) = f(x^{(0)} + tv)
\end{equation}
For \textbf{all possible choices} of $x^{(0)} \in \text{dom }f$, $v\in \reals^n$, and $t\in \reals$.
\begin{equation}
\text{dom }g = \{t | x^{(0)} +tv \in \text{dom }f \}
\end{equation}
\end{theorem}

\begin{theorem}{First Order Condition: }
Differentiable function $f:\reals^n\to \reals$ is convex iff $\forall$ $x,y\in \text{dom }f$,
\begin{equation}
f(y) \geq f(x) + \nabla f(x)^T (y-x)
\end{equation}
With strict convexity for strict inequality.
\end{theorem}
\begin{itemize}
\item This can be connected to the ``line connecting two points is in the set'' point.
\item The linear approximation of a convex function is \textbf{always underestimating}.
\end{itemize}

\begin{theorem}{Second Order Condition: }
Twice differentiable function $f:\reals^n \to \reals$ if convex iff the \textbf{Hessian}
\begin{equation}
\nabla^2 f(x) \in S_{+}^n \,\, \forall x\in \text{dom }f
\end{equation}
In other words, the quadratic approximation is a bowl facing upward.
\end{theorem}

\chapter{Convex Optimization}

\textit{This section was cut short by the reduced term length.}

\section{Convex Optimization Definitions}

\begin{theorem}{Convex Optimization Problem Form: }
\begin{align}
\text{(arg) min}_{x\in \reals^n} \, & f_0 (x) \\
\text{such that } & f_i(x) \leq 0 & \forall i\in [m] \\
& h_i(x) = 0 & \forall i \in [q]
\end{align}
Where
\begin{itemize}
\item Objective $f_0:\reals^n \to \reals$ is convex.
\item Inequality constraints $f_1:\reals^n \to \reals$ are convex.
\item Equality constraints $h_i(x):\reals^n\to \reals$ are \textbf{affine}.
\begin{itemize}
\item Can be written as $Ax = b$ because $a^{(i)T} x - b_i = 0$ is the form of an affine constraint.
\end{itemize}
\end{itemize}
\end{theorem}

\section{Theorems on Convex Optimization}

\begin{theorem}{Local Minimum Definition: }
$z$ is a local minimum if $\exists \, r > 0$ such that
\begin{equation}
f(x) \geq f(z) \forall x \in \{x | \norm{x-z} < r\}
\end{equation}
\end{theorem}

\begin{theorem}{Global Minimum Definition: }
$z$ is a global minimum if
\begin{equation}
f(x) \geq f(z) \, \forall x\in \text{dom }f
\end{equation}
\end{theorem}

\begin{theorem}{Convex Minima: }
If $f_0$ is convex and $\mathcal F$ is convex, then \textbf{any local minimum is the global minimum}.
\end{theorem}

\begin{theorem}{Convex Optimization Solution Uniqueness: }
If $f_0$ is strictly convex and $\mathcal F$ is strictly convex and the set of optimizing $x$ $X_{opt}$ is non-empty, then \textbf{there is a unique solution}.
\end{theorem}

\begin{theorem}{Up Hill Point Theorem: }
Given a convex function $f_0(x)$ optimization with convex feasible fest $\mathcal F$, the point $x^*\in \mathcal F$ is optimal iff, for all $y\in \mathcal F$,
\begin{equation}
\nabla f_0(x^*)^T (y-x^*) \geq 0
\end{equation}
\end{theorem}
Think of this in terms of $\nabla f_0(x^*)$ pointing uphill from the optimal point. The product is the inner product between the uphill vector and the vector from $x$ to $y$, so it intuitively makes sense that it would be an acute angle.

\paragraph{Gradients on Inequality Constraints: } All gradients on inequality constraints $\nabla f_i(x)$ point \textbf{outward} from the feasible set.

\end{document}


\documentclass[11pt,a4paper]{article}

% These are extra packages that you might need for writing the equations:
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{booktabs}
\usepackage{hyperref}
\usepackage{listings}
\usepackage{xcolor}
\usepackage{outlines}
\usepackage{mathtools}

\lstset {language=C++,
		 basicstyle=\ttfamily,
         keywordstyle=\color{blue}\ttfamily,
         stringstyle=\color{red}\ttfamily,
         commentstyle=\color{purple}\ttfamily,
         morecomment=[l][\color{magenta}]{\#},
       	 basicstyle=\tiny}

% You need the following package in order to include figures in your report:
\usepackage{graphicx}

% With this package you can set the size of the margins manually:
\usepackage[left=2cm,right=2cm,top=2cm,bottom=2cm]{geometry}

\begin{document}

% Enter the exercise number, your name and date here:
\noindent\parbox{\linewidth}{
 \parbox{.25\linewidth}{ \large ICP, Exercise 10 }\hfill
 \parbox{.5\linewidth}{\begin{center} \large Beat Hubmann \end{center}}\hfill
 \parbox{.2\linewidth}{\begin{flushright} \large Dec 03, 2018 \end{flushright}}
}
\noindent\rule{\linewidth}{2pt}

\section{Introduction}

An explicit Runge-Kutta method was implemented to solve the equation of motion
for an inclined throw with air resistance in two dimensions.

\section{Algorithm Description}
The Runge-Kutta family of methods to solve ordinary differential equations is well known and described.
The algorithm implemented is the classical and most used Runge-Kutta method usually referred to as RK4 or
something closely similar.

\section{Results}

The program was implemented as described above and submitted with this report.

\subsection{Task 1}
For a fixed friction coefficient $\gamma = 1.0$, trajectories were calculated for launch velocities $v \in \{5, 10, 20, 40\}\ \text{m/s}$ and
over a range of launch angles $10 \le \alpha \le 80\ \text{deg}$ over 15'000 time steps of size
$\Delta t = 0.001\ \text{s}$. The results are shown in figures~\ref{fig:1_1}, \ref{fig:1_2},
\ref{fig:1_3} and~\ref{fig:1_4}.

\begin{figure}[ht]
\begin{center}
\includegraphics[scale=1.2]{figure1_1.eps}
\end{center}
\caption{Flight trajectories for $\gamma = 1.0$, $v= 5.0$.}
\label{fig:1_1}
\end{figure}

\begin{figure}[ht]
\begin{center}
\includegraphics[scale=1.2]{figure1_2.eps}
\end{center}
\caption{Flight trajectories for $\gamma = 1.0$, $v= 10.0$.}
\label{fig:1_2}
\end{figure}

\begin{figure}[ht]
\begin{center}
\includegraphics[scale=1.2]{figure1_3.eps}
\end{center}
\caption{Flight trajectories for $\gamma = 1.0$, $v= 20.0$.}
\label{fig:1_3}
\end{figure}

\begin{figure}[ht]
\begin{center}
\includegraphics[scale=1.2]{figure1_4.eps}
\end{center}
\caption{Flight trajectories for $\gamma = 1.0$, $v= 40.0$.}
\label{fig:1_4}
\end{figure}

\subsection{Task 2}
For a fixed launch velocity $v = 40.0\ \text{m/s}$, the friction coefficient was varied such
that $0.25 \le \gamma \le 10.0$ and the maximum obtained flight distance $x$ recorded.
The result is shown in figure~\ref{fig:2}.

\begin{figure}[ht]
    \begin{center}
    \includegraphics[scale=1.2]{figure2.eps}
    \end{center}
    \caption{Launch angle $a_{max}$ for maximum range as function of friction coeffcient $\gamma$, $v= 40.0$.}
    \label{fig:2}
\end{figure}

\section{Discussion}
The results seem physically plausible. No error analysis was performed.\\
Considerable effort went into templating the solver for generic use for the
purpose of practice following the example of~\cite{Gottschling}.

\begin{thebibliography}{99}

% \bibitem{metropolis}
% Metropolis, N.,
% Rosenbluth, A.W.,
% Rosenbluth, M.N.,
% Teller, A.H.,
% Teller, E.\\
% \emph{Equations of State Calculations by Fast Computing Machines},\\
% Journal of Chemical Physics. 21 (6): 1087,\\
% 1953.

% \bibitem{herrmann}
% 	Herrmann, H. J.,
% 	Singer, H. M.,
% 	Mueller L.,
% 	Buchmann, M.-A.,\\
% 	\emph{Introduction to Computational Physics - Lecture Notes},\\
% 	ETH Zurich,\\
% 	2017.

\bibitem{Gottschling}
Gottschling, Peter\\
\emph{Discovering Modern C++},\\
Addison-Wesley,\\
2016.

\end{thebibliography}

\end{document}


%%
%% Automatically generated file from DocOnce source
%% (https://github.com/hplgit/doconce/)
%%
%%

%-------------------- begin preamble ----------------------

\documentclass[%
oneside,                 % oneside: electronic viewing, twoside: printing
final,                   % draft: marks overfull hboxes, figures with paths
10pt]{article}

\listfiles               %  print all files needed to compile this document

\usepackage{relsize,makeidx,color,setspace,amsmath,amsfonts,amssymb}
\usepackage[table]{xcolor}
\usepackage{bm,ltablex,microtype}

\usepackage[pdftex]{graphicx}

\usepackage[T1]{fontenc}
%\usepackage[latin1]{inputenc}
\usepackage{ucs}
\usepackage[utf8x]{inputenc}

\usepackage{lmodern}         % Latin Modern fonts derived from Computer Modern

% Hyperlinks in PDF:
\definecolor{linkcolor}{rgb}{0,0,0.4}
\usepackage{hyperref}
\hypersetup{
    breaklinks=true,
    colorlinks=true,
    linkcolor=linkcolor,
    urlcolor=linkcolor,
    citecolor=black,
    filecolor=black,
    %filecolor=blue,
    pdfmenubar=true,
    pdftoolbar=true,
    bookmarksdepth=3   % Uncomment (and tweak) for PDF bookmarks with more levels than the TOC
    }
%\hyperbaseurl{}   % hyperlinks are relative to this root

\setcounter{tocdepth}{2}  % levels in table of contents

% --- fancyhdr package for fancy headers ---
\usepackage{fancyhdr}
\fancyhf{} % sets both header and footer to nothing
\renewcommand{\headrulewidth}{0pt}
\fancyfoot[LE,RO]{\thepage}
% Ensure copyright on titlepage (article style) and chapter pages (book style)
\fancypagestyle{plain}{
  \fancyhf{}
  \fancyfoot[C]{{\footnotesize \copyright\ 1999-2020, "Computational Physics I FYS3150/FYS4150":"http://www.uio.no/studier/emner/matnat/fys/FYS3150/index-eng.html". Released under CC Attribution-NonCommercial 4.0 license}}
%  \renewcommand{\footrulewidth}{0mm}
  \renewcommand{\headrulewidth}{0mm}
}
% Ensure copyright on titlepages with \thispagestyle{empty}
\fancypagestyle{empty}{
  \fancyhf{}
  \fancyfoot[C]{{\footnotesize \copyright\ 1999-2020, "Computational Physics I FYS3150/FYS4150":"http://www.uio.no/studier/emner/matnat/fys/FYS3150/index-eng.html". Released under CC Attribution-NonCommercial 4.0 license}}
  \renewcommand{\footrulewidth}{0mm}
  \renewcommand{\headrulewidth}{0mm}
}

\pagestyle{fancy}

% prevent orhpans and widows
\clubpenalty = 10000
\widowpenalty = 10000

% --- end of standard preamble for documents ---

% insert custom LaTeX commands...

\raggedbottom
\makeindex
\usepackage[totoc]{idxlayout}   % for index in the toc
\usepackage[nottoc]{tocbibind}  % for references/bibliography in the toc

%-------------------- end preamble ----------------------

\begin{document}

% matching end for #ifdef PREAMBLE

\newcommand{\exercisesection}[1]{\subsection*{#1}}

% ------------------- main content ----------------------

% ----------------- title -------------------------

\thispagestyle{empty}

\begin{center}
{\LARGE\bf
\begin{spacing}{1.25}
Project 5, deadline  December 16, 2020
\end{spacing}
}
\end{center}

% ----------------- author(s) -------------------------

\begin{center}
{\bf \href{{http://www.uio.no/studier/emner/matnat/fys/FYS3150/index-eng.html}}{Computational Physics I FYS3150/FYS4150}}
\end{center}

    \begin{center}
% List of all institutions:
\centerline{{\small Department of Physics, University of Oslo, Norway}}
\end{center}

% ----------------- end author(s) -------------------------

% --- begin date ---
\begin{center}
Fall semester 2020
\end{center}
% --- end date ---

\vspace{1cm}

\subsection*{Studies of social interactions using the Ising model}

The Ising model is one of the most
frequently used models of statistical mechanics. Recently, this
model has also become widely studied in many other disciplines, including mathematics, statistics, biology, \href{{https://www.springer.com/gp/book/9783319477046}}{economy} and \href{{https://www.springer.com/gp/book/9781461420316}}{sociology}.

\textbf{For this project you can hand in collaborative reports and programs.}
This project (together with projects 3 and 4) counts 1/3 of the final mark.

\paragraph{Introduction.}
The aim of this project is to use the Ising model from project 4 and
apply it to the modeling of electoral patterns and social interactions.
This project aims at studying the work of \href{{https://arxiv.org/abs/cond-mat/0101130}}{Katarzyna Sznajd-Weron and
Jozef Sznajd}, see also their
published work in \href{{https://www.worldscientific.com/doi/abs/10.1142/S0129183100000936}}{Int. J. Mod. Phys. C11,(2000) 1157}.

We repeat
In its simplest form
the energy of the Ising model from project 4 without an externally applied magnetic field,
\[
E=-J\sum_{< kl >}^{N}s_ks_l
\]
with
$s_k=\pm 1$. The quantity $N$ represents the total number of spins and $J$ is a coupling
constant expressing the strength of the interaction between
neighboring spins.  The symbol $<kl>$ indicates that we sum over
nearest neighbors only.
ordering, viz $J> 0$.

We will  consider as our model a community which time and again should
take a stand in some matter, for example vote on a president in
a two-party system. If each member of the
community can take only two attitudes ($A$ or $B$) then in several
votes one expects some difference $m$ of voters for $A$ and against.
We assume three limiting cases:

\begin{enumerate}
\item all members of the community vote for $A$ (an \textbf{all} $A$ state),

\item all members of the community vote for $B$ (an \textbf{all} $B$ state),

\item $50\%$ vote for $A$ and $50\%$ vote for $B$,
\end{enumerate}

\noindent
where the latter is meant to  be the stable solutions of our model.

The aim here is to analyze the time evolution of $m$. To model the
above mentioned system we consider an Ising spins chain
($S_i;i=1,2,\ldots N$) with the following dynamic rules:

\begin{itemize}
\item if $S_iS_{i+1}=1$ then $S_{i-1}$ and $S_{i+2}$ take the direction of the pair (i,i+1),

\item if $S_iS_{i+1}=-1$ then $S_{i-1}$ takes the direction of $S_{i+1}$ and $S_{i+2}$ the direction of $S_i$.
\end{itemize}

\noindent
These rules describe the influence of a given pair on the decision of
its nearest neighbours. When members of a pair have the same opinion
then their nearest neighbours agree with them. On the contrary, when
members of a pair have opinions different then the nearest neighbour
of each member disagrees with her/him.  These dynamic rules lead to
the three steady states above.  However, the third steady state ($50\%$
for $A$ and $50\%$ for $B$) is realized in a very special way.  Every
member of the community disagrees with her/his nearest neighbour (it
is easy to see that the Ising model with only next nearest neighbour
interaction has such fixed points: ferro- and antiferromagnetic
state).  According to the authors of the above mentioned article,
this rule is in accordance with the well known sentence
\emph{united we stand divided we fall} (USDF-model).

\paragraph{Project 5a):}
To investigate our model we perform a standard Monte Carlo simulation
with random updating.  Consider a chain of $N$ Ising spins with free
boundary conditions. In our simulation you could use values from
$N=1000$ up to $N=10000$. We start from a totally random initial state
i.e.~to each site of the chain we assign an arrow with a randomly
chosen direction: up or down (Ising spin).  Show that you obtain as a
state, one of the three fixed points (1-3, i.e.~AAAA, BBBB, ABAB) with
probability 0.25,0.25 and 0.5, respectively. The typical relaxation
time for $N=1000$ is $\sim 10^4$ Monte Carlo steps (MCS). Plot the
spatial distribution of spins from the initial to a steady state and
compare your results with figure 1 of \href{{https://arxiv.org/abs/cond-mat/0101130}}{Katarzyna Sznajd-Weron and
Jozef Sznajd}. Can you see
whether there is a formation of clusters?  Comment your results.

\paragraph{Project 5b): Magnetization and Social Mood.}
Let us define the decision as a magnetization, i.e.:
\begin{equation}
m=\frac{1}{N}\sum_{i=1}^N S_i.
\label{em}
\end{equation}

Compute the magnetization and compare your results with figure 2 of \href{{https://arxiv.org/abs/cond-mat/0101130}}{Katarzyna Sznajd-Weron and
Jozef Sznajd}.

Without any external stimulation decision can change
dramatically in a relatively short time.
Such strongly non-monotonic behaviour of the change of $m$
is typically observed in the USDF-model when the system evolves
towards the third steady state (total disagreement or in magnetic
language the antiferromagnetic state). Comment your results.

\paragraph{Project 5c): Autocorrelation Function.}
To measure the time correlation of $m$ one can employ the
classical autocorrelation function:

\begin{equation}
G(\Delta t) = \frac
{\sum \left( m(t)-<m>\right) \left( m(t+ \Delta t)- <m>\right)}
{\sum (m(t)-<m>)^2}.
\end{equation}

In the work of Sznajd-Weron and Snajzd, there is a comparison of
simulation results with empirical data as shown in their figure 3. Make a plot similar to their figure 3 and comment your results.
Following the changes of one particular individual, the dynamics seem to lead to some interesting effects. If an individual changes her/his opinion at time $t$ she/he will probably
change it also at time $t+1$.

 On the other hand
an individual can stay for a long time without changing her/his decision.
Let us denote by $\tau$ the time needed by an individual to change  her/his
opinion. From the autocorrelation function  it can be seen that $\tau$ is usually very short,
but sometimes can be very long. The distribution of $(\tau)$ ($P(\tau)$)
follows seeminingly a power law behavior
with an exponent $-3/2$. Plot this distribution and compare your results with
figure 4 Sznajd-Weron and Snajzd. Comment your results.

\paragraph{Project 5d): Initial Conditions.}
We will now study the influence of the initial conditions on the
evolution of the system by considering  two different ways -
randomly and in clusters.  In both cases you could start from an initial
concentration $c_B$ of opinion $B$.  In the random setup $c_B*N$
individuals are randomly (uniformly) chosen out of all $N$
individuals. In the cluster setup simply the first $c_B*N$ individuals
are chosen.

Study whether the distribution of decision time $\tau$ still follows
the power law with the same exponent as you found in the previous
part.  A non-monotonic behaviour of decision change is still typical
and sometimes even much stronger.  However, it is obvious that if
initially there is more $A$'s then $B$'s the final state should be
more often "all $A$" then "all $B$".  Compare your results to figure 5
of Sznajd-Weron and Snajzd and comment your results.

\paragraph{Project 5e): Information Noise.}
It is well known that the changes of opinion are determined by the
social impact. Till now we have considered a community in which a
change of an individuals opinion is caused only by a contact with its
neighbours. It was the simplest social impact one can imagine.  Now,
we introduce to our model noise $p$, which is the probability that an
individual, instead of following the dynamic rules, will make a random
decision. We start from an "all $A$" state to investigate if there is
a $p \in (0,1)$ which does not throw the system out of this state.

See if you can reproduce figures 6 and 7 of the above authors and
comment your final results. Their discussion section contains several interesting observations you may consider discussing. The text by \href{{https://www.springer.com/gp/book/9781461420316}}{Serge Galam} contains several other interesting obeservations which can be useful in the present analysi.

\subsection*{Introduction to numerical projects}

Here follows a brief recipe and recommendation on how to write a report for each
project.

\begin{itemize}
  \item Give a short description of the nature of the problem and the eventual  numerical methods you have used.

  \item Describe the algorithm you have used and/or developed. Here you may find it convenient to use pseudocoding. In many cases you can describe the algorithm in the program itself.

  \item Include the source code of your program. Comment your program properly.

  \item If possible, try to find analytic solutions, or known limits in order to test your program when developing the code.

  \item Include your results either in figure form or in a table. Remember to        label your results. All tables and figures should have relevant captions        and labels on the axes.

  \item Try to evaluate the reliabilty and numerical stability/precision of your results. If possible, include a qualitative and/or quantitative discussion of the numerical stability, eventual loss of precision etc.

  \item Try to give an interpretation of you results in your answers to  the problems.

  \item Critique: if possible include your comments and reflections about the  exercise, whether you felt you learnt something, ideas for improvements and  other thoughts you've made when solving the exercise. We wish to keep this course at the interactive level and your comments can help us improve it.

  \item Try to establish a practice where you log your work at the  computerlab. You may find such a logbook very handy at later stages in your work, especially when you don't properly remember  what a previous test version  of your program did. Here you could also record  the time spent on solving the exercise, various algorithms you may have tested or other topics which you feel worthy of mentioning.
\end{itemize}

\noindent
\subsection*{Format for electronic delivery of report and programs}

The preferred format for the report is a PDF file. You can also use DOC or postscript formats or as an ipython notebook file.  As programming language we prefer that you choose between C/C++, Fortran2008 or Python. The following prescription should be followed when preparing the report:

\begin{itemize}
  \item Use \textbf{Canvas} to hand in your projects, log in  at  \href{{https://www.uio.no/english/services/it/education/canvas/}}{\nolinkurl{https://www.uio.no/english/services/it/education/canvas/}} with your normal UiO username and password.

  \item Upload \textbf{only} the report file!  For the source code file(s) you have developed please provide us with your link to your github domain.  The report file should include all of your discussions and a list of the codes you have developed.  Do not include library files which are available at the course homepage, unless you have made specific changes to them. Alternatively, you can just upload the address to your GitHub or GitLab repository.

  \item In your git repository, please include a folder which contains selected results. These can be in the form of output from your code for a selected set of runs and input parameters.

  \item In this and all later projects, you should include tests (for example unit tests) of your code(s).

  \item Comments  from us on your projects, approval or not, corrections to be made  etc can be found under your \textbf{Canvas} domain and are only visible to you and the teachers of the course.
\end{itemize}

\noindent
Finally,
we encourage you to work two and two together. Optimal working groups consist of
2-3 students. You can then hand in a common report.

% ------------------- end of main content ---------------

\end{document}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[utf8, left=2.5cm, right=2.5cm, top=2.5cm, bottom=2.5cm]{geometry}
\usepackage[english]{babel}
\setlength{\parindent}{2em}
\setlength{\parskip}{1em}
\renewcommand{\baselinestretch}{1}
\usepackage{graphicx}
\usepackage{gensymb}
\usepackage{natbib}
\usepackage{amssymb}
\usepackage{xcolor}

\title{CTA200 Assignment 2 Question 4}
\author{Patricia Golaszewska}
\date{May $8^{th}$ 2021}
\begin{document}

\maketitle

\newpage

\begin{center}
    \section*{Question 1}
\end{center}

\section*{Methods}

In this problem, we aim to analyze two methods of differentiation. The first method is often referred to as the forward difference method, and the second as the centered difference method. I began by computing the value of the derivative of the function $sin(x)$ for h values of 0.001, 0.01, and 0.1 to establish a baseline of the accuracy. From there, I wanted to introduce the analytical derivative of $sin(x)$ and contrast the error between the analytical and numerical approaches. In order to do this, I needed to modify the original equations for the forward and centered difference slightly. They would now take a range of h values, versus feeding them individual values. From there, I could create a function which would illustrate the error between the analytical result and the forward difference method, and another which illustrated the error between the analytical result and the centered difference method. Afterwards, I plotted the error on a loglog plot, and Figure 1 was obtained.

\section*{Analysis}

In reference to the graph presented (Figure 1), the error between the two methods and analytical solution are presented. This graph is a loglog plot, generated as the value of h varies. The slope of each line indicates the growth of error between the methods and the analytical solution. As the value of h approaches zero, the error is much smaller. But as the value of h (the separation distance) gets larger, the error of the centered difference grows at a faster rate than that of the forward difference. I assume that this is because of the difference in the dependence oh h in both methods. In regards to evaluating the numerical derivative for different values of h, we obtain a more accurate value for the derivative.

\begin{figure}[htp]
    \centering
    \includegraphics[width=12cm]{Q1.pdf}
    \caption{Error for centered and forward difference methods}
    \label{fig:Q1}
\end{figure}

\newpage

\begin{center}
    \section*{Question 2}
\end{center}

\section*{Methods}

In this problem, we are working with points in the complex plane. I first started by defining an array of x and y values in the specified range. I could then create the points in the complex plane, c. From there, we could then iterate the Mandelbrot set, given that we first created a mesh grid for a and y. An image was generated by filtering the z values and limiting them. Had I provided a longer set of iteration numbers, we would have seen a more defined image. We expect values that do not diverge, and so they must be bounded to not exceed infinity. We can see from the image that the unique boundary of the Mandelbrot set is generated, and the classic shape is obtained. Admittedly, I ran out of time to complete this assignment. Had I had more time, I would have generated another map.

\section*{Analysis}

Studies in nonlinear dynamics and chaos show us the relationship between the Mandelbrot set and the bifurcation diagram of the logistic map. Our points are contained in a closed and compact set. Our points are generated by a quadratic recurrence equation, and the iteration of it in our code provides us with exactly the result we expect. Because the solution is meant to be bounded, there is some value for which the iterations cannot be larger than. Our simple formula gives us a very complex result, and our graph provides a visual representation of how quickly our values can diverge.

\begin{figure}[htp]
    \centering
    \includegraphics[width=12cm]{Q2.pdf}
    \caption{Mandelbrot Set}
    \label{fig:Q2}
\end{figure}

\newpage

\begin{center}
    \section*{Question 3}
\end{center}

\section*{Methods}

In order to solve this system of ordinary differential equations, I chose to employ the dopri5 integrator. In order to solve our problem, I needed to first create a function which contained our equations. From there, I was able to feed that function into an integrator. In order to produce numerical results, I needed to provide the integrator with a set of initial conditions and function parameters. Values were chosen for $\beta$, $N$, and $\gamma$, and the integration process could begin. First, an empty array was created as a means of storing the solution to the numerical integral, and a counter was set up. This counter would allow for us to store a solution for each point we passed through as we took the integral. From there, we could plot the solutions for each variable (S, I, or R) on the same graph. This process was repeated 3 times for the SIR model. We then introduced a fourth variable for D, and adapted the model to include death. This model was then solved in a similar fashion, but a 4-D approach was taken versus a 3-D approach.

\section*{Analysis}

In the models, values of $\beta$ and $\gamma$ were chosen to reflect a likely transmission and infection pathway. The variable $\gamma$ represents the recovery rate of people infected with this disease. This implies that $\gamma$ is bounded between 0 and 1. If $\gamma$ is 1, this means that every individual will recover, versus 0, where no one will recover. Similar conditions apply to $\beta$; not every interaction is guaranteed to result in disease transmission, but there has to be some transmission taking place. In all models, we see that the number of susceptible people will drop, and won't return to the initial value. This is because these individuals now have immunity. The number of infected people will fluctuate, as these are the individuals infected at a given time. The recovery rate differs in the SIR models versus the SIRD model, as now the infected individuals may succumb to their disease rather than recover.

\begin{figure}[htp]
    \centering
    \includegraphics[width=12cm]{Q3.1.pdf}
    \caption{$\beta$ = 0.5 and $\gamma$ = 0.2}
    \label{fig:Q3.1}
\end{figure}

\begin{figure}[htp]
    \centering
    \includegraphics[width=12cm]{Q3.2.pdf}
    \caption{$\beta$ = 0.8 and $\gamma$ = 0.09}
    \label{fig:Q3.2}
\end{figure}

\begin{figure}[htp]
    \centering
    \includegraphics[width=12cm]{Q3.3.pdf}
    \caption{$\beta$ = 0.7 and $\gamma$ = 0.1}
    \label{fig:Q3.3}
\end{figure}

\begin{figure}[htp]
    \centering
    \includegraphics[width=12cm]{Q3.4.pdf}
    \caption{$\beta$ = 0.8, $\gamma$ = 0.3, and $\mu$ = 0.05}
    \label{fig:Q3.4}
\end{figure}

\end{document}