\documentclass[12pt,letterpaper]{article}

\usepackage{assignments}
\usepackage{minted}

\begin{document}

\title{\vspace{-4ex}ECE521: Inference Algorithms and Machine Learning \\
University of Toronto\\ \  \\
Assignment 3: \\Unsupervised Learning and Probabilistic Models}
\date{\vspace{-8ex}TA: Use Piazza for Q\&A \\ Due date: Mar. 24 11:59 pm, 2017 \\ Electronic submission to: \href{mailto:ece521ta@gmail.com}{ece521ta@gmail.com} }

\maketitle

\section*{General Note:}
\begin{itemize}
\item In this assignment, you will implement learning and inference procedures for some of the probabilistic models described in class, apply your solutions to some simulated datasets, and analyze the results.
\item Full points are given for complete solutions, including justifying the choices or assumptions you made to solve the question. Both complete source code and program outputs should be included in the final submission.
\item Homework assignments are to be solved in the assigned groups of two or three. You are encouraged to discuss the assignment with other students, but
you must solve it within your own group. Make sure to be closely involved in all aspects of the assignment.
\end{itemize}

\begin{mycomments}
\section{Part 1}
\end{mycomments}
\section{K-means}

K-means clustering is one of the most widely used data analysis algorithms. It is used to summarize data by discovering a set of data prototypes that represent clusters of data. The data prototypes are usually referred to as cluster centers. Usually, K-means clustering proceeds by alternating between assigning data points to clusters and then updating the cluster centers. In this assignment, we will investigate a different learning algorithm that directly minimizes the K-means clustering loss function.

\subsection{Learning K-means [8 pt.]}

The $K$ cluster centers can be thought of as $K$, $D$-dimensional parameter vectors and we can place them in a $K \times D$ parameter matrix $\boldsymbol{\mu}$, where the $k^{th}$ row of the matrix denotes the $k^{th}$ cluster center $\boldsymbol{\mu}_k$. The goal of K-means clustering is to learn $\boldsymbol{\mu}$ such that it minimizes the loss function, $\mathcal{L}(\boldsymbol{\mu}) = \sum_{n=1}^B \min_{k=1}^K \|\bold{x}_n - \boldsymbol{\mu}_k\|_2^2$. Even though the loss function is not smooth due to the ``$\min$'' operation, one may still be able to find its solutions through iterative gradient-based optimization. The ``$\min$'' operation leads to discontinuous derivatives, in a way that is similar to the effect of the ReLU activation function, but nonetheless, a good gradient-based optimizer can work effectively.

\begin{enumerate}
\item Is the loss function $\mathcal{L}(\boldsymbol{\mu})$ convex in $\boldsymbol{\mu}$? Why or why not? [2 pt.]
\item For the dataset \textit{data2D.npy}, set $K=3$ and find the K-means clusters $\boldsymbol{\mu}$ by minimizing the $\mathcal{L}(\boldsymbol{\mu})$ using the gradient descent optimizer. The parameters $\boldsymbol{\mu}$ should be initialized by sampling from the standard normal distribution. Include a plot of the loss vs the number of updates. Hints: you may want to use the Adam optimizer for this assignment with following hyper-parameter \textit{tf.train.AdamOptimizer(LEARNINGRATE, beta1=0.9, beta2=0.99, epsilon=1e-5)}. The learning should converge within a few hundred updates. [2 pt.]
\item Run the algorithm with $K={1,2,3,4,5}$ and for each of these values of $K$, compute and report the percentage of the data points belonging to each of the $K$ clusters. Comment on how many clusters you think is ``best'' and why? (To answer this, it may be helpful discuss this value in the context of a 2D scatter plot of the data.) Include the 2D scatter plot of data points colored by their cluster assignments. [2 pt.]
\item Hold 1/3 of the data out for validation. For each value of $K$ above, cluster the training data and then compute and report the loss for the validation data. How many clusters do you think is best? [2 pt.]
\end{enumerate}

\section{Mixtures of Gaussians}

Mixtures of Gaussians (MoG) can be interpreted as a probabilistic version of K-means clustering. For each data vector, MoG uses a latent variable $z$ to represent the cluster assignment and uses a joint probability model of the cluster assignment variable and the data vector: $P(\bold{x}, z) = P(z)P(\bold{x} \given z)$. For $B$ IID training cases, we have $P({\bold{X},\bold{z}}) = \prod_{n=1}^B P(\bold{x}_n,z_n)$. The Expectation-Maximization (EM) algorithm is the most commonly used technique to learn a MoG. Like the standard $K$-means clustering algorithm, the EM algorithm alternates between updating the cluster assignment variables and the cluster parameters. What makes it different is that instead of making hard assignments of data vectors to cluster centers (the ``$\min$'' operation above), the EM algorithm computes probabilities for different cluster centers, $P(z|\bold{x})$. These are computed from $P(z|\bold{x}) = P(\bold{x},z)/\sum_{z'=1}^K P(\bold{x},z')$.

While the EM algorithm is guaranteed to converge to a local optimum, it suffers from slow convergence. In this assignment, we will explore a different learning algorithm that makes use of gradient descent.

\subsection{The Gaussian cluster model [6 pt.]}

Each of the $K$ mixture components in the MoG model occurs with probability $\pi^k = P(z=k)$. The data model is a multivariate Gaussian distribution centered at the cluster mean (data center) $\boldsymbol{\mu}^k \in \real^D$. We will consider a MoG model where it is assumed that for the multivariate Gaussian for cluster $k$, different data dimensions are independent and have the same standard deviation, ${\sigma^k}$.

\begin{enumerate}

\item Derive the expression for the latent variable posterior distribution of a data point $P(z\given\bold{x})$ in terms of the MoG parameters, $\{\boldsymbol{\mu}^k, \sigma^k, \pi^k\}$. [2 pt.]

\item Modify the K-means distance function we derived above to compute the \underline{log} probability density function for cluster $k$: $\log \mathcal{N}(\bold{x} \, ; \, \boldsymbol{\mu}^k, {\sigma^k}^2)$ for all pair of $B$ data points and $K$ clusters. Include the snippets of the Python code [2 pt.]

\item Write a \underline{vectorized} Tensorflow Python function that computes the \underline{log} probability of the cluster variable $z$ given the data vector $\bold{x}$: $\log P(z|\bold{x})$. The log Gaussian pdf function implemented above should come in handy. The implementation should use the provided \textit{utils.logsumexp} function. Include the snippets of the Python code and comment on why it is important to use the log-sum-exp function instead of using \textit{tf.reduce\_sum}. [2 pt.]
\end{enumerate}

\subsection{Learning the MoG [12 pt.]}

The marginal data likelihood for the MoG model is as follows (here ``marginal'' refers to summing over the cluster assignment variables):
\bean
P(\bold{X}) = \prod_{n=1}^B P(\bold{x}_n) &= \prod_{n=1}^B \sum_{k=1}^K P(z_n=k)P(\bold{x}_n \given z_n=k)\\&= \prod_n \sum_k \pi^k \mathcal{N}(\bold{x}_n\, ; \, \boldsymbol{\mu}^k, {\sigma^k}^2)
\eean
The loss function we will minimize is the negative log likelihood $\mathcal{L}(\boldsymbol{\mu}, \sigma, \pi) = - \log P(\bold{X})$. The maximum likelihood estimate (MLE) is a set of the model parameters $\boldsymbol{\mu}, \sigma, \pi$ that maximize the log likelihood or, equivalently, minimize the negative log likelihood.

\begin{enumerate}
  \item Direct gradient-based optimization appears to learn the MoG parameters without inferring the cluster assignment variables, that is, without computing $P(z|\bold{x})$. In fact, this inference is implicit in the gradient computation. Show that for a single training example, the gradient of the marginal log likelihood function is the expected gradient of the log joint probability under its posterior distribution, $\grad_{\boldsymbol{\mu}} \log P(\bold{x}) = \sum_k P(z=k\given \bold{x}) \grad_{\boldsymbol{\mu}} \log P(\bold{x}, z=k)$. [2 pt.]

{\color{red}
\textbf{Answer: }
    \[ \nabla \log P(\mathbf x) = \frac{\nabla P(x)}{P(x)} = \frac{\nabla \sum_{k=1}^{K}{P(x, z=k)}}{P(x)} = \frac{\sum_{k=1}^{K}{\nabla P(x, z=k)}}{P(x)} = \]
    \[= \frac{\sum_{k=1}^{K}{P(x,z=k)\frac{\nabla P(x, z=k)}{P(x,z=k)}}}{P(x)} = \frac{\sum_{k=1}^{K}{P(x,z=k)\nabla \log P(x, z=k)}}{P(x)} = \]
    \[= \sum_{k=1}^{K}{\frac{P(x,z=k)}{P(x)}\nabla \log P(x, z=k)} = \sum_{k=1}^{K}{P(z=k\, |\, x)\nabla \log P(x, z=k)}\]
}

\item Implement the loss function using log-sum-exp function and perform MLE by directly optimizing the log likelihood function using gradient descent in Tensorflow. Note that the standard deviation has the constraint of $\sigma \in [0, \infty)$. One way to deal with this constraint is to replace $\sigma^2$ with $\exp(\phi)$ in the math and the software, where $\phi$ is an unconstrained parameter. In addition, $\pi$ has a simplex constraint, that is $\sum_k \pi^k = 1$. We can again replace this constrain with unconstrained parameter $\psi$ through a softmax function $\pi^k = \exp(\psi^k)/\sum_{k'}\exp(\psi^{k'})$. A log-softmax function is provided for convenience, \textit{utils.logsoftmax}. For the dataset \textit{data2D.npy}, set $K=3$ and report the best model parameters it has learnt. Include a plot of the loss vs the number of updates. [6 pt.]

\item Hold out 1/3 of the data for validation and for each value of $K={1,2,3,4,5}$, train a MoG model. For each $K$, compute and report the loss function for the validation data and explain which value of $K$ is best. Include a 2D scatter plot of data points colored by their cluster assignments. [2 pt.]

\item Run both the K-means and the MoG learning algorithms on \textit{data100D.npy}. Comment on how many clusters you think are within the dataset and compare the learnt results of K-means and MoG.  [2 pt.]
\end{enumerate}

\section{Discover Latent Dimensions }

\subsection{Factor Analysis [Bonus: 8 pt.]}

So far we have considered K-means and MoG for clustering the data. In both of these cases we assume that each data point `belongs to' or `is generated by' one of K prototypes or causes. In K-means, we make a hard decision about the prototype that each point belongs to, allowing to choose only 1. In MoG, we assign points to clusters in a soft way, reflecting our uncertainty about the underlying cause of each point by modelling the softmax distribution. However, these soft assignments merely represent a probabilistic view over which of the K latent causes generated each point; we still believe that only 1 cause generated it. In this question, we use Factor Analysis to relax this constraint: there is now no restriction on the number of latent causes that generate each point.

Let $\bold{s}_i$ be a vector of real-valued latent variables that have generated point $\bold{x}_i$. We assume that $p(\bold{s}_i) = \mathcal{N}(\mathbf{s}_i\, ;\, \mathbf{0}, I)$.
Our $\bold{x}_i$'s are also continuous, therefore allowing us to model the likelihood with a Gaussian as well:
$p(\bold{x}_i | \bold{s}_i) = \mathcal{N}(\bold{x}_i\, ; \, W\mathbf{s}_i + \boldsymbol{\mu}, \Psi)$.

%Something along the line of 1) deriving the marginal log likelihood for factor analysis, 2) learning the factor analysis model through directly optimizing the marginal log likelihood. 3) compare the result with PCA

Consider the marginal likelihood defined as:
\bean
P(\bold{X}) = \prod_{n=1}^B P(\bold{x}_n) &= \prod_{n=1}^B \int_{\mathbf{s}_n} P(\mathbf{s}_n)P(\bold{x}_n \given \mathbf{s}_n)\\&= \prod_{n=1}^B \int_{\mathbf{s}_n} \mathcal{N}(\mathbf{s}_n\, ;\, \mathbf{0}, I) \mathcal{N}(\bold{x}_n\, ; \, W\mathbf{s}_n + \boldsymbol{\mu}, \Psi)
\eean

\begin{enumerate}
  \item Derive the marginal log likelihood of the factor analysis model for a single training example  $\log P(\mathbf{x}) = \log \int_{\mathbf{s}} P(\mathbf{x} \given \mathbf{s})P(\mathbf{s}) = \log \mathcal{N}(\mathbf{x}\, ;\, \boldsymbol{\mu}, \Psi + WW^T) $

{\color{red}
\textbf{Answer: }
Finish the following using joint covariance $\to$ marginal covariance
\bean
&s^\T s + (Ws - x)^\T \Psi^{-1} (Ws - x) \\
=&s^\T s + s^TW^\T\Psi^{-1} Ws + x^\T \Psi^{-1} x - 2x^\T\Psi^{-1} Ws \\
=&s^\T(I + W^\T\Psi^{-1} W) s - 2(x^\T\Psi^{-1} W) s + x^\T \Psi^{-1} x
\eean
}
  \item Write a TensorFlow implementation that learns Factor Analysis models by directly maximizing the log likelihood function. And learn latent space on notMNIST or MNIST dataset. {\color{red} Need to try this algorithm out to see if it works and decide which dataset gives better results. }
  %\item Learning Factor Analysis through an approximate inference model, linear VAE.
\item Geoffrey Hinton's explanation on PCA and FA: Generate a toy dataset of 200 3-dimensional data points $\{\mathbf{x}^{(1)}, \dots, \mathbf{x}^{(200)}\}$ by first generating the latent states $\mathbf{s}$ from a 3-D multivariate Gaussian distribution with zero mean and identity covariance matrix $\mathbf{s} \sim \mathcal{N}(\mathbf{s}\, ; \, \mathbf{0}, I), \quad \mathbf{s} = \begin{bmatrix} s_1\\ s_2\\ s_3 \end{bmatrix} \in \mathbb{R}^3$. Now transform the latent states to 3-dimensional observations $\mathbf{x} = \begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}$ using the following formula:
    \begin{gather*}
       x_1 = s_1 \\
       x_2 = s_1 + 0.001s_2 \\
       x_3 = 10s_3
    \end{gather*}
    Use such dataset to train a PCA with a single principle component and a factor analysis model with a single latent dimension. Show that PCA learns the maximum variance direction (i.e. $x_3$ direction) while FA learns the maximum correlation direction(i.e. $x_1+x_2$ direction).
\end{enumerate}

\end{document}


\documentclass{beamer}

\usepackage[utf8]{inputenc}
\usepackage[english]{babel}

\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{mathtools}
\usepackage{wrapfig}
\usepackage{amsfonts}
\usepackage{csquotes}
\usepackage{epigraph}
\usepackage{float}
\usepackage{lipsum}
\usepackage{blindtext}
\usepackage{multirow}
\usepackage[ruled,vlined]{algorithm2e}
\usepackage{bm}
\usepackage{xcolor}
\usepackage{upgreek}
\usepackage{calc}
\usepackage{lscape}
\usepackage[font={scriptsize}]{caption}
\usepackage{systeme}

% TYPESETTING SUGAR
\DeclareMathOperator*{\argmax}{argmax}
\DeclareMathOperator*{\argmin}{argmin}
\DeclareMathOperator*{\Find}{Find}
\DeclareMathOperator*{\find}{find}
\DeclareMathOperator*{\NNet}{NNet}
\DeclareMathOperator*{\EGR}{EGR}
\def\leq{\leqslant}
\def\geq{\geqslant}

\usetheme{Madrid}
\usecolortheme[rgb={0.0,0.4,0.5}]{structure}
%-----------------------------------------------%
%-----------------------------------------------%

%
% TITLE SLIDE
%

\makeatletter
\setbeamertemplate{footline}{%
  \leavevmode%
  \hbox{\begin{beamercolorbox}[wd=.4\paperwidth,ht=2.5ex,dp=1.125ex,leftskip=.20cm plus1fill,rightskip=.20cm]{author in head/foot}%
    \usebeamerfont{author in head/foot}\insertshortauthor
  \end{beamercolorbox}%
  \begin{beamercolorbox}[wd=.6\paperwidth,ht=2.5ex,dp=1.125ex,leftskip=.20cm,rightskip=.20cm plus1fil]{title in head/foot}%
    \usebeamerfont{title in head/foot}\insertshorttitle \hspace*{1.2ex}
    \insertframenumber{}/\inserttotalframenumber
  \end{beamercolorbox}}%
  \vskip0pt%
}
\makeatother

\title[\textit{Neural ODEs} for \textit{real-world}, irregularly-sampled, time series prediction]{\textit{Neural ODEs} \\ for \textit{real-world}, irregularly-sampled, time series prediction}

\author[E. Ballarin, M. N. Plasencia Palacios, A. Tasciotti]{Emanuele Ballarin \\ Milton Nicolás Plasencia Palacios \\ Arianna Tasciotti}
\institute[]{Department of Mathematics and Geosciences, University of Trieste}
\date[]{\textit{Statistical Machine Learning} course \\ Final Project $\sim$ June 2020}

\titlegraphic{\includegraphics[width=1.25cm,height=1.3cm,keepaspectratio]{dssclogo.png}
	\hspace*{93.04mm}~% Horizontal space is tailored for 1.3cm square logos
	\includegraphics[width=1.3cm,height=1.25cm,keepaspectratio]{logouniv.png}
}

%-----------------------------------------------%
%-----------------------------------------------%

%
% ADAPTIVE TABLE OF CONTENTS
%

\AtBeginSection[]
{
	\begin{frame}
		\frametitle{\textit{We are here...}}
		\tableofcontents[currentsection]
	\end{frame}
}

%-----------------------------------------------%
%-----------------------------------------------%

%
% DOCUMENT
%

\begin{document}

% Insert title slide
\frame{\titlepage}

%-----------------------------------------------%

%% TIME SERIES
%\section{Introduzione al paradigma del \textit{Deep Learning}}{
\section{\textit{Theory first!}}{

%% SLIDE 1.1:
\begin{frame}
\frametitle{Time Series}

A \textbf{time series} is a sequence of observations labeled in time order.  If the time-dependent variables are more than one we are talking about \textbf{multivariate time series}.\\~\

This type of data is highly exploited in economics, business and finance where it is important to predict future outcomes that depends on past observations.

\end{frame}
%---------------------------------------------%
%% SLIDE 1.2:
\begin{frame}
	\frametitle{RNN}
	%-----------------------------------------%
	To study a time series we can use a \textbf{Recurrent Neural Network} (RNN), which is a type of neural network in which the current hidden state is computed using the previous hidden state and the current input.

	 \includegraphics[scale = 0.25]{rnn1.png}
\end{frame}
%---------------------------------------------%
%%Slide 1.3.1
\begin{frame}
\frametitle{LSTM}

The \textbf{Long Short Term Memory Unit} is an improvement of the recurrent unit in which the information pass through a \textbf{don't forget gate} which decides to keep or throw away data. Input and output data are processed respectively by \textbf{input} and \textbf{output gate}.

\begin{center}
\includegraphics[scale = 0.5]{lstm.png}
\end{center}

\end{frame}
%---------------------------------------------%
%% SLIDE 1.3.2:
\begin{frame}
	\frametitle{GRU}
	\textbf{Gated Recurrent Unit} (GRU) is an improved version (in terms of efficiency) of the LSTM. GRU uses \textbf{update and reset gates} in order to modulate what information should be passed to the output.

	\begin{center}
	\includegraphics[scale = 0.6]{gru.png}
         \end{center}
\end{frame}
%---------------------------------------------%
%% SLIDE 1.4:
\begin{frame}
	\frametitle{ResNet}
	\textbf{Residual Neural Networks} (ResNet) use residual blocks and skip connections to solve vanishing gradient problem.

	\begin{center}
	\includegraphics[scale = 0.4]{resnet.png}
         \end{center}

\end{frame}
%---------------------------------------------%
%% SLIDE 1.5:
\begin{frame}
	\frametitle{ODE}
	The ResNet model:

	\begin{equation}
	\textbf{h}_{l+1} =\textbf{h}_l + f(\textbf{h}_l, \theta_l)
	\end{equation}

	can be seen as  the Euler method for solving ordinary differential equations. We can exploit this new relation by adding more layers into the network and take smaller time steps.  In the limit we can describe our new neural network with the following model (\textbf{Neural ODE}):

	\begin{equation}
	\frac{d \textbf{h}(t)}{dt} = f(\textbf{h}(t), t, \theta)
	\end{equation}

	 with the function $f$ parameterized by a neural network, and solve it with a differentiable discretization method for ODEs.
\end{frame}
%---------------------------------------------%

%% SLIDE 1.6.1:
\begin{frame}
	\frametitle{RNN-ODE}
	\framesubtitle{Standard RNNs}
	\begin{figure}
	\includegraphics[height=0.24\linewidth]{standard_RNN.png}
	\caption*{'Latent ODEs for Irregularly-Sampled Time Series', Yulia Rubanova and Ricky T. Q. Chen and David Duvenaud, 2019.}
	\end{figure}
	\break
	\pause \begin{itemize}
	\item{Constant or undefined hidden states between observations}
	\end{itemize}
	\begin{itemize}
	\item{Hard to adapt for sparse irregular data...}
	\end{itemize}
\end{frame}
%---------------------------------------------%
%% SLIDE 1.6.2:
\begin{frame}
	\frametitle{RNN-ODE}
	\framesubtitle{RNNs for sparse data}
	\begin{itemize}
	\item{Include the time gap $\Delta_{t}=t_{i}-t_{i-1}$ between observations into RNN update function:}
	\end{itemize}
	$$h_{i} = RNNCell(h_{i-1}, \Delta_{t}, x_{i})$$
	\begin{itemize}
	\pause \item{Exponential decay of the hidden state:}
	\end{itemize}
	$$h_{i} = RNNCell(h_{i-1}, \exp{(-\tau \Delta_{t})}, x_{i})$$
	\begin{figure}
	\includegraphics[height=0.18\linewidth]{RNN_decay.png}
	\caption*{'Latent ODEs for Irregularly-Sampled Time Series', Yulia Rubanova and Ricky T. Q. Chen and David Duvenaud, 2019.}
	\end{figure}
\end{frame}
%---------------------------------------------%
%% SLIDE 1.6.3:
\begin{frame}
	\frametitle{RNN-ODE}
	\framesubtitle{RNNs and Neural ODEs}
	Equivalence between exponential decay approach and solving an initial value problem involving an ODE:
	\[ \systeme[\frac{dh(t)}{dt} h(t_{0})]{\frac{dh(t)}{dt}=-\tau h, h(t_{0})=h_{0}} \]
	\hfill \break
	\pause ODE solution: $h(t)=h_{0}e^{-\tau \Delta_{t}}$
	\hfill \break
	\begin{itemize}
	\pause \item{hidden state $h(t)$ is an unknown function of time and is the solution of an ODE:}
	\end{itemize}
	$$\frac{dh(t)}{dt}=f_{\theta}(h(t),t)$$
	\begin{itemize}
	\pause \item{$h(t)$ can be evaluated at any time using a numerical ODE solver.}
	\end{itemize}
\end{frame}
%---------------------------------------------%
%% SLIDE 1.6.4:
\begin{frame}
	\frametitle{GRU-ODE architecture}
	The state between observations is the solution of an ODE:
	$$h_{i}^{'}=ODESolve(f_{\theta}, h_{i-1}, (t_{i-1},t_{i}))$$
	\hfill \break
	\pause For each observation, the hidden state is updated using a GRU update:
	$$h_{i}=GRUCell(h_{i}^{'},x_{i})$$
	\begin{figure}
	\includegraphics[height=0.16\linewidth]{ODE_RNN.png}
	\caption*{'Latent ODEs for Irregularly-Sampled Time Series', Yulia Rubanova and Ricky T. Q. Chen and David Duvenaud, 2019.}
	\end{figure}
\end{frame}
%-----------------------------------------------%
%% SLIDE 1.7:
\begin{frame}
	\frametitle{Latent ODE model}
	\begin{itemize}
	\item{Time series is represented by a latent trajectory;}
	\pause \item{Each trajectory is determined from a local initial state $z_{0}$ and a global set of latent dynamics shared across time series.}
	\end{itemize}
	\hfill \break
	\pause Two tasks:
	\begin{itemize}
	\item{extrapolation $\rightarrow$ forwards in time}
	\item{interpolation $\rightarrow$ backwards in time}
	\end{itemize}

%We investigate the ability of the latent ODE model to fit and extrapolate time series. The recognition network is an RNN with 25 hidden units. We use a 4-dimensional latent space. We parameterize the dynamics function f with a one-hidden-layer network with 20 hidden units. The decoder computing p(xti |zti ) is another neural network with one hidden layer with 20 hidden units. Our baseline was a recurrent neu- ral net with 25 hidden units trained to minimize negative Gaussian log-likelihood. We trained a second version of this RNN whose inputs were concatenated with the time difference to the next observation to aid RNN with irregular observa- tions.

\end{frame}
%---------------------------------------------%
%% SLIDE Intermezzo:
\begin{frame}
	\frametitle{Uncertainty quantifier}
	 \center $x_{in}$, $x_{out}$ observed, $z$ latent, $p(x_{in}, x_{out}, z)$ joint
	\pause \center $q(z_{0}|x_{in}, x_{out}) =  \mathcal{N}(\mu_{z_{0}},\sigma_{z_{0}})$
\end{frame}
}
%---------------------------------------------%
%% SLIDE 1.8:
\begin{frame}
	\frametitle{Uncertainty quantifier}
	\textbf{Feed-forward NN}
	\begin{itemize}
	 \item{Input: final hidden state of GRU-ODE}
	 \item{Output: $\mu_{z_{0}}$ and $\sigma_{z_{0}}$}
	\end{itemize}
	\hfill \break
	\pause \textbf{Sampler}
	\begin{itemize}
	 \item{Input: $\mu_{z_{0}}$, $\sigma_{z_{0}}$}
	 \item{Output: $\bar{z_{0}}$}
	\end{itemize}
\end{frame}
%---------------------------------------------%
%% SLIDE 1.9:
\begin{frame}
	\frametitle{Architectural overview}
	\center $h_{i}^{'}=ODESolve(f, h_{i-1}, (t_{i-1},t_{i}))$
	\center \pause $h_{i}=GRUCell(h_{i}^{'},x_{i})$
	\center \pause $\mu_{z_{0}}, \sigma_{z_{0}} =g(h_{i})$
	\center \pause $z_{0} \sim \mathcal{N}(\mu_{z_{0}},\sigma_{z_{0}})$
	\center \pause $\{z_{i}\}=ODESolve(z_{0}, f, (t_{0},...,t_{N}))$
	\center \pause $\widetilde{x_{i}}=OutputNN(z_{i})$

%Let x be a (set of) observed variable(s), z a (set of) latent variable(s) and let p(x, z) be the parametric model of their joint distribution, called the generative model defined over the variables. Given a dataset X = {x1 , ..., xN } we typically wish to perform maximum marginal likelihood learning of its parameters, i.e. to maximize XN i=1 but in general this marginal likelihood is intractable to compute or differentiate directly for flexible generative models, e.g. when components of the generative model are parameterized by neural networks. A solution is to introduce q(z|x), a parametric inference model defined over the latent variables, and optimize the variational lower bound on the marginal log-likelihood of each observation x:log p(x)  Eq(z|x) [log p(x, z)  log q(z|x)] = L(x; ✓)
\end{frame}
%---------------------------------------------%
\begin{frame}
	\frametitle{A \textit{bird's eye} architectural recap (1)}
	\framesubtitle{A \textit{lato sensu} encoder-decoder structure}
	\begin{block}{\center $h_{i}^{'}=ODESolve(f, h_{i-1}, (t_{i-1},t_{i}))$ \center $h_{i}=GRUCell(h_{i}^{'},x_{i})$}
		\centering
		\textit{Encoder}. Encodes an entire multidimensional, variable-length time series to fixed-low-dimensional latent representation, loop after loop. Outputs the present \textit{latent state}.
	\end{block}
	\center $\mu_{z_{0}}, \sigma_{z_{0}} =g(h_{i})$
	\center $z_{0} \sim \mathcal{N}(\mu_{z_{0}},\sigma_{z_{0}})$
	\center $\{z_{i}\}=ODESolve(z_{0}, f, (t_{0},...,t_{N}))$
	\center $\widetilde{x_{i}}=OutputNN(z_{i})$

\end{frame}
%---------------------------------------------%
\begin{frame}
	\frametitle{A \textit{bird's eye} architectural recap (2)}
	\framesubtitle{A \textit{lato sensu} encoder-decoder structure}
	\center $h_{i}^{'}=ODESolve(f, h_{i-1}, (t_{i-1},t_{i}))$
	\center $h_{i}=GRUCell(h_{i}^{'},x_{i})$

	\begin{block}{\center $\mu_{z_{0}}, \sigma_{z_{0}} = g(h_{i})$ \center $z_{0} \sim \mathcal{N}(\mu_{z_{0}},\sigma_{z_{0}})$}
		\centering
		\textit{Uncertainty estimator}. $\approx$ a VAE. Essentially \textit{BBVI} tricked into making uncertainty estimation (only). Outputs the present \textit{processed} latent state.
	\end{block}

	\center $\{z_{i}\}=ODESolve(z_{0}, f, (t_{0},...,t_{N}))$
	\center $\widetilde{x_{i}}=OutputNN(z_{i})$

\end{frame}
%---------------------------------------------%
\begin{frame}
	\frametitle{A \textit{bird's eye} architectural recap (3)}
	\framesubtitle{A \textit{lato sensu} encoder-decoder structure}
	\center $h_{i}^{'}=ODESolve(f, h_{i-1}, (t_{i-1},t_{i}))$
	\center $h_{i}=GRUCell(h_{i}^{'},x_{i})$
	\center $\mu_{z_{0}}, \sigma_{z_{0}} = g(h_{i})$
	\center $z_{0} \sim \mathcal{N}(\mu_{z_{0}},\sigma_{z_{0}})$

	\begin{block}{\center $\{z_{i}\}=ODESolve(z_{0}, f, (t_{0},...,t_{N}))$ \center $\widetilde{x_{i}}=OutputNN(z_{i})$}
	\centering
	\textit{Decoder}. Emits one entire predicted future-time-series, from present \textit{processed} latent state.
	\end{block}
\end{frame}
\section{\textit{Deep learning works great, except when it doesn't.}}{
%---------------------------------------------%
\begin{frame}
	\frametitle{The theory seems sound...}
	\framesubtitle{So sound it doesn't even work}
	\begin{figure}
		To make such architecture work in practice, we need to pull some (theoretically-grounded) tricks out of the $\widehat{trick}^*$!
	\end{figure}
\hfill\break
[* trick hat]
\end{frame}
%---------------------------------------------%
\begin{frame}
	\frametitle{\textit{Batch Normalisation} and \textit{Layer Normalisation}}
	\framesubtitle{A \textit{batch-wise} and a \textit{feature-wise} approach to training regularisation}

	Introduced to fight the problem of \textit{internal covariate shift}, they mainly act as \textit{noise-injection regularisers} \textit{(Leslie, 2019)}.
\hfill\break
	$$
	\hat{x}^{(j)}_i = \frac{x^{(j)}_i - \mu^{(j)}}{\sqrt{\sigma^{(j)^2} + \epsilon}}
	$$
	\begin{itemize}
		\item{\textit{BatchNorm (Ioffe, 2015)} acts dimension-wise over a minibatch, during training;}
		\item{\textit{LayerNorm (Ba, 2016)} acts dimension-wise and feature-wise, online, also during evaluation.}
	\end{itemize}
\end{frame}
%---------------------------------------------%
\begin{frame}
	\frametitle{Gradient Norm Clipping}
	\framesubtitle{Keeping seatbelts fastened}
	The recurrent nature of GRUs (and RNNs in general) may lead to exponentially-varying weights, and subsequently gradients. \\ $\rightarrow$ Training instability!
\hfill\break

\begin{itemize}
	\item{\textit{Vanishing gradients} $\rightarrow$ Gating (Schmidhuber, 1992; Cho, 2015), ReLU networks;}
	\item{\textit{Exploding gradients} $\rightarrow$ $ ||\nabla \boldsymbol{\theta}|| \coloneqq \text{min}(c, ||\nabla \boldsymbol{\theta}||)$ and rescale \textit{(Graves, 2013)}.}
\end{itemize}
\hfill\break
Also: no side effects if unneeded.

\end{frame}
%---------------------------------------------%
\begin{frame}
	\frametitle{AdamW Optimiser (1)}
	\framesubtitle{\textit{Ridge penalty} and \textit{weight decay} are not the same thing}
	All the \textit{Adam-family} optimisers:
	\begin{itemize}
		\item{Are Hessian-free;}
		\item{Use single-parameter-adaptive, upper-bound, SGD learning;}
		\item{Exploit $2^{nd}$-order gradient-momentum information.}
	\end{itemize}
\hfill\break
However, they regularise differently!
	\begin{itemize}
	\item{Adam: $\mathcal{L}_r = \mathcal{L} + \omega \frac{||\boldsymbol{\theta}||_2^2}{2}$ \textit{(Kingma, 2014)};}
	% w = w - lr * w.grad - lr * wd * w
	\item{AdamW: $\boldsymbol{\theta} \coloneqq \boldsymbol{\theta} - \boldsymbol{\rho}\cdot \nabla \boldsymbol{\theta} - \omega \boldsymbol{\rho}\cdot \boldsymbol{\theta}$ \textit{(Loshchilov, 2018)}.}
	\end{itemize}
\end{frame}
%---------------------------------------------%
\begin{frame}
	\frametitle{AdamW Optimiser (2)}
	\framesubtitle{A deeper dive into Adam \& friends}

	For each scalar weight $w$, at iteration $t$, in parallel:

	$$
	v_t = \beta_1 v_{t-1} - (1-\beta_1) \nabla w
	$$
	$$
	s_t = \beta_2 s_{t-1} - (1-\beta_2) (\nabla w)^2
	$$
	$$
	\rho = \text{min}\left({\text{lr}, \text{lr} \frac{v_t}{\sqrt{s_t + \epsilon}}}\right)
	$$
\hfill\break
	And then, the usual $\boldsymbol{\theta} \coloneqq \boldsymbol{\theta} - \boldsymbol{\rho}\cdot \nabla \boldsymbol{\theta}$.
\hfill\break
\hfill\break
Per-parameter estimation makes the two regularisations different.

\end{frame}

%-----------------------------------------------%
\begin{frame}
	\frametitle{$L_1$ loss}
	\framesubtitle{Uniformity is the key}
	An outright impopular choice for a loss function, however:
	\hfill\break
\begin{itemize}
	\item{The per-epoch loss average was $\approx 1$;}
	\item{$\mathcal{L}^2 < \mathcal{L}$ if $\mathcal{L} < 1$, whereas $\mathcal{L}^2 > \mathcal{L}$ if $\mathcal{L} > 1$, thus ruling out as potentially unstable and biased $L_2$ and \textit{Huber's smooth} losses;}
	\item{Also: more straightforward training monitoring (``$>2.5$'' heuristic)}
\end{itemize}
\end{frame}
}

\section{\textit{No data, no learning!}}{

%---------------------------------------------%
%% SLIDE 1.14:
\begin{frame}
	\frametitle{Dataset}
	The dataset contains informations about stock prices of many US companies in the period from 2 January 1962 to 27 March 2018.

	\begin{center}
	\includegraphics[scale = 0.40]{stock.jpg}
         \end{center}

\end{frame}
%---------------------------------------------%
\begin{frame}
	\frametitle{Windowing}
	\framesubtitle{Changing everything to make nothing change}

	Presenting data in overlapping-window form to RNNs is the first step to effectively \textit{teacher-force} them \textit{(Lamb, 2016; Hafner, 2018)}.

\begin{gather*}
a\ b\ c\ |\ d\ e\ f\ |\ g\ h\ i\ |\ \ \dots \\
b\ c\ d\ |\ e\ f\ g\ |\ h\ i\ j\ |\ \ \dots \\
c\ d\ e\ |\ f\ g\ h\ |\ i\ j\ k\ |\ \ \dots
\end{gather*}
(note: data ordered by column!)

\hfill\break

However, we often used very large windows to further improve learning on the most recent data.

\end{frame}
%---------------------------------------------%
\begin{frame}
	\frametitle{Data batching}
	\framesubtitle{You probably always made batches for RNNs wrong!}
	The second (and last) step is doing the batches in the right manner! \textit{(Lamb, 2016; Hafner, 2018)}.

\begin{gather*}
\alert{a\ b\ c}\ |\ \alert{d\ e\ f}\ |\ g\ h\ i\ |\ \ \dots \\
\alert{b\ c\ d}\ |\ e\ f\ g\ |\ h\ i\ j\ |\ \ \dots \\
\alert{c\ d\ e}\ |\ f\ g\ h\ |\ \alert{i\ j\ k}\ |\ \ \dots
\end{gather*}

\begin{itemize}
	\item{The usual (and almost-always unchangeable) \textit{first-K-and-batch} approach leads to hidden-state dissociation!}
	\item{Hidden-state averaging can be limiting and slow;}
\end{itemize}
\hfill\break
{$\rightarrow$ Use of the \textit{multi-head-disk seek algorithm}.}

\end{frame}
}

\section{\textit{So, what?}}{
%---------------------------------------------%
%% SLIDE 1.17.1:
\begin{frame}
	\frametitle{Results}
	\begin{columns}
	\begin{column}{.5\textwidth}
	\centering
	\includegraphics[height=0.45\textheight]{open_AAPL.png}
	Prediction of AAPL open price.
	\end{column}%

	\begin{column}{.5\textwidth}
	\centering
	\includegraphics[height=0.45\textheight]{close_FOX.png}
	Prediction of FOX close price.
	\end{column}
	\end{columns}

	\hfill \break

	\centering
	\begin{tabular}{|c|c|c|} \hline
	& MAE on Median I.C. & MAE on Median E.C. \\
	\hline
	AAPL & 0.0051 & 0.0066 \\
	\hline
	FOX & 0.1836 & 0.3016 \\
	\hline
 	\end{tabular}

\end{frame}
%---------------------------------------------%
%% SLIDE 1.17.2:
\begin{frame}
	\frametitle{Results}
	\begin{columns}
	\begin{column}{.5\textwidth}
	\centering
	\includegraphics[height=0.45\textheight]{open_IBM.png}
	Prediction of IBM open price.
	\end{column}%

	\begin{column}{.5\textwidth}
	\centering
	\includegraphics[height=0.45\textheight]{close_F.png}
	Prediction of F close price.
	\end{column}
	\end{columns}

	\hfill \break

	\centering
	\begin{tabular}{|c|c|c|} \hline
	& MAE on Median I.C. & MAE on Median E.C. \\
	\hline
	IBM & 0.0086 & 0.0068 \\
	\hline
	F & 0.0311 & 0.0347 \\
	\hline
 	\end{tabular}

\end{frame}
%---------------------------------------------%
%% SLIDE 1.18:
\begin{frame}
	\frametitle{Final remarks and future work (1)}
	\framesubtitle{Just one -- even successful -- try is never enough!}

	\begin{displayquote}
		“We never wanted to build an infallible oracle, but just to ease risk management.”
		\begin{flushright}Ritchie Ng (Chief of AI -- Hessian Matrix Capital) \end{flushright}
	\end{displayquote}

\end{frame}

\begin{frame}
	\frametitle{Final remarks and future work (2)}
	\framesubtitle{Just one -- even successful -- try is never enough!}

	Overall, a moderately-successful application of \textit{state-of-the-art} NN methods to a real-world problem, despite the lack of settled knowledge on the matter and training instabilities.
	\hfill \break
	\begin{itemize}
		\item{Fairly \textit{close-to-reality} predictions provided a successful training;}
		\item{Usable uncertainty estimation for \textit{limit-case} scenarios;}
		\item{Fast and introspectable training ($\leftarrow$ heuristics).}
	\end{itemize}
	\hfill \break
	Not a perfect one, though...
	\hfill \break
	\begin{itemize}
		\item{Extreme brittleness of hyperparameter tuning;}
		\item{Too broad uncertainty for finer-grained analysis;}
		\item{Deep $\mathcal{L}$ local minima may require training restart.}
	\end{itemize}

\end{frame}
}

%% THANKS!
\begin{frame}
	\frametitle{Greetings}
	\center Thank you for your attention!
	%\hfill\break
%	\hfill\break
	\center \includegraphics[scale = 0.35]{xkcdml.png}
	\center [\url{https://github.com/emaballarin/cathode}]
\end{frame}

%-----------------------------------------------%
%-----------------------------------------------%
\end{document}


\documentclass[letterpaper,10pt]{article}
\usepackage[margin=2cm]{geometry}

\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{multicol}
\usepackage{listings}
\usepackage{color}

\definecolor{mygreen}{rgb}{0,0.6,0}
\definecolor{mygray}{rgb}{0.5,0.5,0.5}
\definecolor{mymauve}{rgb}{0.58,0,0.82}

\lstset{ %
  backgroundcolor=\color{white},   % choose the background color; you must add \usepackage{color} or \usepackage{xcolor}
  basicstyle=\footnotesize,        % the size of the fonts that are used for the code
  breakatwhitespace=false,         % sets if automatic breaks should only happen at whitespace
  breaklines=true,                 % sets automatic line breaking
  captionpos=b,                    % sets the caption-position to bottom
  commentstyle=\color{mygreen},    % comment style
  deletekeywords={...},            % if you want to delete keywords from the given language
  escapeinside={\%*}{*)},          % if you want to add LaTeX within your code
  extendedchars=true,              % lets you use non-ASCII characters; for 8-bits encodings only, does not work with UTF-8
  frame=single,	                   % adds a frame around the code
  keepspaces=true,                 % keeps spaces in text, useful for keeping indentation of code (possibly needs columns=flexible)
  keywordstyle=\color{blue},       % keyword style
  language=Matlab,                 % the language of the code
  otherkeywords={},           % if you want to add more keywords to the set
  numbers=left,                    % where to put the line-numbers; possible values are (none, left, right)
  numbersep=5pt,                   % how far the line-numbers are from the code
  numberstyle=\tiny\color{mygray}, % the style that is used for the line-numbers
  rulecolor=\color{black},         % if not set, the frame-color may be changed on line-breaks within not-black text (e.g. comments (green here))
  showspaces=false,                % show spaces everywhere adding particular underscores; it overrides 'showstringspaces'
  showstringspaces=false,          % underline spaces within strings only
  showtabs=false,                  % show tabs within strings adding particular underscores
  stepnumber=1,                    % the step between two line-numbers. If it's 1, each line will be numbered
  stringstyle=\color{mymauve},     % string literal style
  tabsize=4,	                   % sets default tabsize to 2 spaces
  title=\lstname                   % show the filename of files included with \lstinputlisting; also try caption instead of title
}

\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator*{\argmax}{arg\,max}

\title{\textbf{Pattern Recognition Theory 18794 Homework 2}}
\author{Mengwen He (Alex)}

\begin{document}

\maketitle

\section*{Problem 1}

PCA vs LDA. Given the following dataset for a 2-class problem with 2-D features:

\begin{equation}
\nonumber
c_1=\left\{\left[\begin{array}{c}
2 \\ 1
\end{array}\right],\left[\begin{array}{c}
2 \\ 2
\end{array}\right],\left[\begin{array}{c}
2 \\ 3
\end{array}\right]\right\},~
c_2=\left\{\left[\begin{array}{c}
4 \\ 3
\end{array}\right],\left[\begin{array}{c}
5 \\ 3
\end{array}\right],\left[\begin{array}{c}
6 \\ 4
\end{array}\right]\right\}
\end{equation}

\begin{enumerate}
\item Using global PCA, find the best direction onto which the data will be projected on.\\
\lstinputlisting[language=Matlab]{./matlab/P_1_1.m}
\begin{enumerate}
	\item Express the equation of the line along this direction passing	through the samples mean in the following form: $\vec{w}^T\vec{x} + w_0 = 0$, where
	$$\vec{x}=\left[\begin{array}{c}
	x_1 \\ x_2
	\end{array}\right]\text{~and~}\vec{w}=\left[\begin{array}{c}
	w_1 \\ w_2
	\end{array}\right]$$
	and $w_0$ is a scalar known as the bias. Plot the $\vec{w}^T\vec{x} + w0 = 0$ line
	along with all the sample points as well the line along $\vec{w}$ which recall starts from the mean.
	$$\vec{w}=\left[\begin{array}{c}
	 0.4541 \\
	 -0.8910 \\
	\end{array}\right],~w_0=0.7866$$

	\item Project and reconstruct all the sample points. Plot the reconstructed points.
	\begin{center}
	\includegraphics[width=0.65\textwidth]{./matlab/PCA.eps}
	\end{center}

	\item Find the total mean square error MSE for all sample points (between the original points and the reconstructed points).
	$$MSE=0.2943$$

	\item Find the Fisher Ratio for this projection defined by
	$$FR=\frac{(m_1-m_2)^2}{\sigma_1^2+\sigma_2^2}$$
	where $m_i$ is the mean of the projected samples of class $i$, and $\sigma_i^2$ is the equivalent variance. You can compute the $FR$ on the projected 1-D points (rather than the reconstructed points which are 2-D vectors).
	$$FR=10.9421$$

\end{enumerate}

\item Using Fisher Linear Discriminant Analysis (LDA), determine the best one-dimensional space onto which the above data should be projected.\\
\lstinputlisting[language=Matlab]{./matlab/P_1_2.m}
\begin{enumerate}
	\item Express the equation of the line along this direction passing
	through the samples mean in the following form: $\vec{w}^T\vec{x}+w0 = 0$. Plot
	the $\vec{w}^T\vec{x} + w0 = 0$ line along with all the sample points.
	$$\vec{w}=\left[\begin{array}{c}
		 -0.0499 \\
		    -0.9988 \\
		\end{array}\right],~w_0=2.8381$$

	\item Project and reconstruct all the sample points. Plot the reconstructed points. Make sure they lie on a line.\\
	\begin{center}
		\includegraphics[width=0.65\textwidth]{./matlab/LDA.eps}
	\end{center}

	\item Find the total MSE for all sample points (between the original points and the reconstructed points). Compare that to the total MSE found in the first question.
	$$MSE=1.0095$$

	\item Find the Fisher Ratio for this projection. You can compute the FR on the projected points (rather than the reconstructed points which are 2-D vectors). Compare that to the $FR$ found in part 1. Interpret your result.
	$$FR=13.5385$$

\end{enumerate}
\end{enumerate}

\section*{Problem 2}

Another goal of PCA is to obtain the linear subspace which minimizes the projection error caused by dimension reduction. The two goals, maximum variance and minimum error, have the same formulation as PCA.

\begin{enumerate}
\item Assume all data samples $\{\vec{x}_1,\dots,\vec{x}_n\}$ are centered. Formulate the optimization problem to solve for a vector $\vec{w}$ spanning the linear subspace which minimizes the sum of the squared reconstruction errors:
$$\sum_{i=1}^{n}{d_i^2}$$
where $d_i$, the reconstruction error of $\vec{x}_i$, is defined as the distance between $\vec{x}_i$ and its projection onto the linear subspace. Assume that $\vec{w}$ has a unit norm since we are interested only in the direction of the vector $\vec{w}$. The answer must be written in terms of $\vec{w}$ and $\vec{x}_i$ without $d_i$.\\
\textbf{Answer:}\\
$\because$
\begin{equation}
\nonumber
\begin{array}{rcl}
d_i^2(\vec{w}) & = & ||\vec{x}_i-\vec{w}\vec{x}_i^T\vec{w}||_2^2 \\
 & = & (\vec{x}_i-\vec{w}\vec{x}_i^T\vec{w})^T(\vec{x}_i-\vec{w}\vec{x}_i^T\vec{w}) \\
 & = & \vec{x}_i^T\vec{x}_i-2\vec{x}_i^T\vec{w}\vec{x}_i^T\vec{w}+\vec{w}^T\vec{x}_i\vec{x}_i^T\vec{w}\\
 & = & \vec{x}_i^T\vec{x}_i-2\vec{w}^T(\vec{x}_i\vec{x}_i^T)\vec{w}+\vec{w}^T(\vec{x}_i\vec{x}_i^T)\vec{w}\\
 & = & \vec{x}_i^T\vec{x}_i-\vec{w}^T(\vec{x}_i\vec{x}_i^T)\vec{w}\\
 \sum_{i=1}^{n}{d_i^2(\vec{w})} & = & \sum_{i=1}^{n}{\vec{x}_i^T\vec{x}_i}-\vec{w}^T\sum_{i=1}^{n}{(\vec{x}_i\vec{x}_i^T)}\vec{w} \\
 \text{where}~~||\vec{w}|| & = & 1 \\
 & = & \vec{w}^T\vec{w}=1
\end{array}
\end{equation}
$\therefore$ use Lagrange Multipliers to get the optimized $\vec{w}$:
\begin{equation}
\nonumber
\begin{array}{rcl}
L(\vec{w},\lambda) & = & \sum_{i=1}^{n}{d_i^2(\vec{w})} - \lambda(1-\vec{w}^T\vec{w}) \\
 & = & \sum_{i=1}^{n}{\vec{x}_i^T\vec{x}_i}-\vec{w}^T\sum_{i=1}^{n}{(\vec{x}_i\vec{x}_i^T)}\vec{w} - \lambda(1-\vec{w}^T\vec{w}) \\
 \frac{\partial L}{\partial \lambda} & = & \vec{w}^T\vec{w}-1 = 0 \\
 \frac{\partial L}{\partial \vec{w}} & = & -2(\sum_{i=1}^{n}{\vec{x}_i\vec{x}_i^T})\vec{w}+2\lambda\vec{w} = 0\\
 & \Rightarrow & \left(\sum_{i=1}^{n}{\vec{x}_i\vec{x}_i^T}\right)\vec{w}=\lambda\vec{w} \\
\end{array}
\end{equation}
$\therefore$ $\lambda$s are the eigenvalues, and $\vec{w}$s are the corresponding eigenvectors of $\sum_{i=1}^{n}{\vec{x}_i\vec{x}_i^T}$.\\
$\because$
$$\left(\sum_{i=1}^{n}{\vec{x}_i\vec{x}_i^T}\right)\vec{w}=\lambda\vec{w}\Rightarrow\vec{w}^T\left(\sum_{i=1}^{n}{\vec{x}_i\vec{x}_i^T}\right)\vec{w}=\lambda$$
and
$$\sum_{i=1}^{n}{\vec{x}_i^T\vec{x}_i}\text{ is constant}$$
$\therefore$ to minimize $\sum_{i=1}^{n}{d_i^2(\vec{w})}$, we need to choose the $\vec{w}^*$ to maximize $\vec{w}^T\left(\sum_{i=1}^{n}{\vec{x}_i\vec{x}_i^T}\right)\vec{w}=\lambda=\lambda_{max}$.

\item From your answer in (a), show that the optimization in part (a) is equivalent to the PCA optimization using the following equivalence:
$$\Sigma = \sum_{i}^{n}{\vec{x}_i\vec{x}_i^T}$$
\textbf{Answer:}\\
$\because$ PCA is to choose the $\vec{w}^*$ to maximize the variant after projection $\vec{w}^T\Sigma\vec{w}=\lambda=\lambda_{max}$.\\
$\therefore$ they are equivalent.
\end{enumerate}

\section*{Problem 3}

You are to implement PCA from scratch and not use the MATLAB built in
functions. You are allowed to use SVD and/or eigenvalue decomposition. Please
submit a print out of your code aw well.

\begin{enumerate}
	\item Plot the mean image of digit 1 and plot the first 5 global PCA vectors
	corresponding to the dataset corresponding to two cases: using Gram
	Matrix Trick and without Gram Matrix trick. Do not forget to remove
	the mean before you run PCA. Measure the time taken for the PCA vector
	computation in each case.\\
	\lstinputlisting[language=Matlab]{./matlab/P_3_1.m}
	\begin{center}
	\includegraphics[width=0.9\textwidth]{./matlab/digit_1.eps}\\
	The first row is the digit 1's mean image; the second row is the first 5 eigenvector images without Gram trick; the third row is the first 5 eigenvector images with Gram trick.
	\end{center}

	\item Comment on the time taken in each case. Does it make sense to use the
	Gram trick for this particular dataset?\\
	\textbf{Answer:}
	\begin{itemize}
		\item When samplenum=1000 (total number is 1000*10=10000): w/o Gram costs 0.100550s, and w/ Gram costs 4.579334s.
		\item When samplenum=100 (total number is 100*10=1000): w/o Gram costs 0.037760s, and w/ Gram costs 0.038084s.
		\item When samplenum=10 (total number is 10*10=100): w/o Gram costs 0.013963s, and w/ Gram costs 0.042149s.
		\item When samplenum=1 (total number is 10*10=100): w/o Gram costs 0.008933s, and w/ Gram costs 0.003651s.
	\end{itemize}
	Therefore, if $N<<d$, Gram trick is helpful for the PCA algorithm speed.

	\item  Pick any random image from the dataset which can be any image from any
	class and project onto the eigen space (the global PCA eigen space that
	you obtained in part 1 of this problem) and reconstruct it using the first n
	eigen vectors (not just the nth vector but all of them up until n) (for the
	two cases, Gram trick and without it) where $n = \{1, 2, 5, 10, 20\}$. Do not
	forget to remove the mean before projecting the image, and also to add it
	back once it has been reconstructed using only the first n eigen vectors.

	For each of the 5 reconstructions, compute the mean square error (MSE)
	of the reconstructed image. Note that the MSE between two vectors $\vec{a}$
	and $\vec{b}$ is given by
	$$MSE(\vec{a},\vec{b})=||\vec{a}-\vec{b}||_2^2$$
	Display all reconstructed images and the original in one figure, with the
	corresponding MSE value as the title of each sub-figure\\
	\lstinputlisting[language=Matlab]{./matlab/P_3_3.m}
	\newpage
	\begin{center}
	\includegraphics[width=0.95\textwidth]{./matlab/WoG_N01.eps}
	\includegraphics[width=0.95\textwidth]{./matlab/WG_N01.eps}
	\includegraphics[width=0.95\textwidth]{./matlab/WoG_N02.eps}
	\includegraphics[width=0.95\textwidth]{./matlab/WG_N02.eps}
	\includegraphics[width=0.95\textwidth]{./matlab/WoG_N05.eps}
	\includegraphics[width=0.95\textwidth]{./matlab/WG_N05.eps}
	\includegraphics[width=0.95\textwidth]{./matlab/WoG_N10.eps}
	\includegraphics[width=0.95\textwidth]{./matlab/WG_N10.eps}
	\includegraphics[width=0.95\textwidth]{./matlab/WoG_N20.eps}
	\includegraphics[width=0.95\textwidth]{./matlab/WG_N20.eps}
	\end{center}

\end{enumerate}

\end{document}


Welcome to the \castro\ User's Guide!

In this User's Guide we describe how to download and run \castro, a
massively parallel code that solves the multicomponent compressible
hydrodynamic equations for astrophysical flows including self-gravity,
nuclear reactions and radiation.  \castro\ uses an Eulerian grid and
incorporates adaptive mesh refinement (AMR).  Our approach to AMR uses
a nested hierarchy of logically-rectangular grids with simultaneous
refinement in both space and time, utilizing the
\amrex\ library\footnote{earlier versions of \castro\ used the
  \boxlib\ library}.

The core algorithms in \castro\ are described in a series of papers:
\begin{itemize}
\item {\it CASTRO: A New Compressible Astrophysical Solver. I. Hydrodynamics and Self-gravity},
  A.~S.~Almgren, V.~E.~Beckner, J.~B.~Bell, M.~S.~Day, L.~H.~Howell, C.~C.~Joggerst, M.~J.~Lijewski,
  A.~Nonaka, M.~Singer, \& M.~Zingale, 2010, ApJ, 715, 1221\newline
  \url{http://dx.doi.org/10.1088/0004-637X/715/2/1221}

\item {\it CASTRO: A New Compressible Astrophysical Solver. II. Gray Radiation Hydrodynamics},
  W.~Zhang, L.~Howell, A.~Almgren, A.~Burrows, \& J.~Bell, 2011, ApJS, 196, 20\newline
  \url{http://dx.doi.org/10.1088/0067-0049/196/2/20}

\item {\it CASTRO: A New Compressible Astrophysical Solver. III. Multigroup Radiation Hydrodynamics},
  W.~Zhang, L.~Howell, A.~Almgren, A.~Burrows, J.~Dolence, \& J.~Bell, 2013, ApJS, 204, 7\newline
  \url{http://dx.doi.org/10.1088/0067-0049/204/1/7}

\end{itemize}

Improvements to the gravity solver and rotation were described in:
\begin{itemize}
\item {\it Double White Dwarf Mergers on Adaptive Meshes I. Methodology
       and Code Verification, }
  M.~P.~Katz, M.~Zingale, A.~C.~Calder, F.~D.~Swesty, A.~S.~Almgren, W.~Zhang
  2016, ApJ, 819, 94.\newline
  \url{http://dx.doi.org/10.3847/0004-637X/819/2/94}
\end{itemize}

The development of \amrex\ library is led by the
Center for Computational Sciences and Engineering / Lawrence Berkeley
National Laboratory.  \castro\ development is done collaboratively,
including the CCSE and Stony Brook University.

\castro\ {\em core developers} are those who have made substantial
contributions to the code.  The process for becoming a core developer
is described in the {\tt README.md} in the \castro\ root directory.
Current \castro\ core developers are:

% git shortlog -sn
\begin{quote}
Ann Almgren\newline
Maria G.\ Barrios Sazo\newline
John Bell\newline
Vince Beckner\newline
Marc Day\newline
Max Katz\newline
Mike Lijewski\newline
Chris Malone\newline
Andy Nonaka\newline
Don Willcox\newline
Weiqun Zhang\newline
Michael Zingale
\end{quote}

All \castro\ development takes place on the project's github
page\\[0.5em] \url{https://github.com/AMReX-Astro/Castro}\\[0.5em]
External contributions are welcomed.  Fork the \castro\ repo, modify
your local copy, and issue a pull-request to the {\tt
  AMReX-Astro/Castro} project.  Further guidelines are given in the
{\tt README.md} file.

To get help, subscribe to the {\em castro-help} google group mailing list:
\url{https://groups.google.com/forum/#!forum/castro-help}

\section*{Acknowledging and Citing \castro}

If you use \castro\ in your research, we would appreciate it if you
cited the relevant code papers describing its design, features, and
testing.  A list of these can be found in the
\href{https://github.com/AMReX-Astro/Castro/blob/master/CITATION}{\tt
  CITATION} file in the root {\tt Castro/} directory.

The development \castro\ is supported by the science application
interests of the contributors.  There is a lot of effort behind the
scenes: testing, optimization, development of new features, bug
fixing, $\ldots$, that is often done under the radar.  Nevertheless,
we are happy to volunteer our time to help new users come up to speed
with \castro.  When significant new development / debugging for you
application is provided by a member of the \castro\ development
community, we would appreciate consideration of inviting the
developer(s) for co-authorship on any science paper that results.


\section{Reorganization energy}
\label{sec:reorganization}

The reorganization energy\index{reorganization energy} $\lambda_{ij}$ takes into account the change in  nuclear (and dielectric) degrees of freedom as the charge moves from donor $i$ to acceptor $j$. It has two contributions: intramolecular, $\lambda^\text{int}_{ij}$, which is due to reorganization of nuclear coordinates of the two molecules forming the charge transfer complex, and intermolecular (outersphere), $\lambda^\text{out}_{ij}$, which is due to the relaxation of the nuclear coordinates of the environment. In what follows we discuss how these contributions can be calculated.

\subsection{Intramolecular reorganization energy}
\label{sec:eintramolecular}
\index{reorganization energy!intramolecular}
If intramolecular vibrational modes of the two molecules are treated classically, the rearrangement of their nuclear coordinates after charge transfer results in the dissipation of the internal reorganization energy, $\lambda_{ij}^\text{int}$. It can be computed from four points on the potential energy surfaces (PES) of both molecules in neutral and charged states, as indicated in \fig{parabolas}.

\begin{figure}
   \centering
   \includegraphics[width=0.6\linewidth]{fig/reorganization_energy/monomer_parabolas}
    \caption{Potential energy surfaces of (a) donor and (b) acceptor in charged and neutral states. After the change of the charge state both molecules relax their nuclear coordinates. If all vibrational modes are treated classically, the total internal reorganization energy and the internal energy difference of the electron transfer reaction are $\lambda_{ij}^\text{int} = \lambda_{i}^{cn} + \lambda_{j}^{nc}$ and $\Delta E_{ij}^\text{int} =  \Delta U_i - \Delta U_j$, respectively.}
   \label{fig:parabolas}
\end{figure}

Adding the contributions due to discharging of molecule $i$ and charging of molecule $j$ yields~\cite{bredas_charge-transfer_2004}
\begin{equation}
\lambda_{ij}^\text{int} =\lambda_{i}^{cn}+\lambda_{j}^{nc}=U_{i}^{nC}-U_{i}^{nN}+U_{j}^{cN}-U_{j}^{cC}\,.
\label{equ:lambdas}
\end{equation}
Here $U_{i}^{nC}$ is the internal energy of the neutral molecule $i$ in the geometry of its charged state (small $n$ denotes the state and capital $C$ the geometry). Similarly, $U_{j}^{cN}$ is the energy of the charged molecule $j$ in  the geometry of its neutral state.
%If the PES of neutral and charged states are different for the same molecule, that is  $\lambda^{cn}_i \neq \lambda^{nc}_i$, the rate for the bimolecular charge transfer is no longer a simple sum. If, as before, both modes are treated classically, the rate is an integral over the charge detachment and attachment spectrum of molecules $i$ and $j$~\cite{kakitani_comprehensive_1987}. For most systems, however, the reorganization energies for charging or discharging of the same molecule do not deviate by more than a few percent and the rate is given by~\equ{marcus}.
%
Note that the PES of the donor and acceptor are not identical for chemically different compounds or for conformers of the same molecule. In this case $\lambda_{i}^{cn} \ne \lambda_{j}^{cn}$ and  $\lambda_{i}^{nc} \ne \lambda_{j}^{nc}$. Thus $\lambda_{ij}^\text{int}$ is a property of the charge transfer complex, and not of a single molecule.

Intramolecular reorganization energies for discharging ($\lambda^{cn}$) and charging ($\lambda^{nc}$) of a molecule need to be determined using quantum-chemistry and given in \xmlcsg. The values are written to the \sqlstate using the calculator \calc{einternal} (see also \slink{sec:einternal}{internal energy}):
\votcacommand{Intramolecularl reorganization energies}{\cmdeint}

\subsection{Outersphere reorganization energy}
\index{reorganization energy!outersphere}
\label{sec:eoutersphere}
During the charge transfer reaction, also the molecules outside the charge transfer complex reorient and polarize in order to adjust for changes in electric potential, resulting in the outersphere contribution to the reorganization energy. $\lambda_{ij}^\text{out}$ is particularly important if charge transfer occurs in a polarizable environment. Assuming that charge transfer is much slower than electronic polarization but much faster than nuclear rearrangement of the environment, $\lambda_{ij}^\text{out}$ can be calculated from the electric displacement fields created by the charge transfer complex~\cite{may_relationship_2011}
\begin{equation}
\lambda_{ij}^\text{out}=
\frac{c_p}{2\epsilon_0}\int_{V^\text{out}}d V
\left[ \vec{D}_I(\vec{r}) - \vec{D}_F(\vec{r}) \right]^2\,,
\label{equ:lambda_outer1}
\end{equation}
where $\epsilon_0$ is the the permittivity of free space, $\vec{D}_{I,F}(\vec{r})$ are the electric displacement fields created by the charge transfer complex in the initial (charge on molecule $i$) and final (charge transferred to molecule $j$) states,  $V^\text{out}$ is the volume outside the complex, and $c_p=\frac{1}{\epsilon_\text{opt}}-\frac{1}{\epsilon_\text{s}}$ is the Pekar factor, which is determined by the low ($\epsilon_\text{s}$) and high ($\epsilon_\text{opt}$) frequency dielectric permittivities.

\Equ{lambda_outer1} can be simplified by assuming spherically symmetric charge distributions on molecules $i$ and $j$ with total charge $e$. Integration over the volume $V^\text{out}$ outside of the two spheres of radii $R_i$ and $R_j$ centered on molecules $i$ and $j$ leads to the classical Marcus expression for the outersphere reorganization energy
\begin{equation}
\lambda_{ij}^\text{out}=\frac{c_{p}e^2}{4\pi\epsilon_0}\left(\frac{1}{2 R_i}+\frac{1}{2 R_j}-\frac{1}{r_{ij}} \right)\,,
\label{equ:lambda_outer2}
\end{equation}
where $r_{ij}$ is the molecular separation.  While \equ{lambda_outer2} captures the main physics, e.g. predicts smaller outer-sphere reorganization energies (higher rates) for molecules at smaller separations, it often cannot provide quantitative estimates, since charge distributions are rarely spherically symmetric.

Alternatively, the displacement fields can be constructed using the atomic partial charges. The difference of the displacement fields at the position of an atom $b_k$ outside the charge transfer complex (molecule $k \ne i,j$)  can be expressed as
\begin{eqnarray}
\label{equ:disp_atom}
\vec{D}_I(\vec{r}_{b_k}) - \vec{D}_F(\vec{r}_{b_k})  =
\sum_{a_i} \frac{q_{a_i}^c - q_{a_i}^n}{4\pi } \frac{ (\vec{r}_{b_k} - \vec{r}_{a_i} ) }
                                            {|\vec{r}_{b_k}-\vec{r}_{a_i}|^3}+
\sum_{a_j} \frac{q_{a_j}^n - q_{a_j}^c}{4\pi } \frac{ (\vec{r}_{b_k}-\vec{r}_{a_j} ) }
                                            {|\vec{r}_{b_k}-\vec{r}_{a_j}|^3}\,,
%\nonumber
\end{eqnarray}
where $q^n_{a_i}$ ($q^c_{a_i}$) is the partial charge of atom $a$ of the neutral (charged) molecule $i$ in vacuum. The partial charges of neutral and charged molecules are obtained by fitting their values to reproduce the electrostatic potential of a single molecule (charged or neutral) in vacuum.
%
Assuming a uniform density of atoms, the integration in~\equ{lambda_outer1} can be rewritten as a density-weighted sum over all atoms excluding those of the charge transfer complex.

The remaining unknown needed to calculate $\lambda_{ij}^\text{out}$ is the Pekar factor\index{Pekar factor}, $c_p$. In polar solvents $\epsilon_\text{s}\gg\epsilon_\text{opt}\sim 1$ and $c_p$ is of the order of 1. In most organic semiconductors, however, molecular orientations are fixed and therefore the low frequency dielectric permittivity is of the same order of magnitude as $\epsilon_\text{opt}$. Hence, $c_p$ is small and its value is very sensitive to differences in the permittivities.

Outersphere reorganization energies for all pairs of molecules in the \slink{sec:neighborlist}{neighbor list} can be computed from the atomistic trajectory by using the \calc{eoutersphere} \calculator.

Two methods can be used to compute $\lambda_{ij}^\text{out}$. The first method uses the atomistic partial charges of neutral and charged molecules from files specified in \xmlcsg and \equ{lambda_outer1}. The Pekar factor $c_p$ and a cutoff radius  based on molecular centers of mass have to be specified in the \xmloptions file.

If this method is computationally prohibitive, $\lambda_{ij}^\text{out}$ can be computed using \equ{lambda_outer2}, which assumes spherical charge distributions on the molecules. In this case the radii of these spheres are specified in \xmlsegments, while the Pekar factor $c_p$ is given in the \xmloptions file and no cutoff radius is needed.

The outer sphere reorganization energies are saved to the \sqlstate file:
\votcacommand{Outersphere reorganization energy}{\cmdouter}


\documentclass[a4paper]{article}

% Language and Font Encoding
\usepackage[english]{babel}
%\usepackage[utf8x]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{listings}
\usepackage{float}
\usepackage{caption}         % For captions
\usepackage{fancyhdr}        % For header and footer
\usepackage{rotating}
\usepackage{amssymb}
%\usepackage{varwidth}
\usepackage[colorlinks=true, allcolors=blue]{hyperref}

% Page size and margins
\usepackage[a4paper,top=3cm,bottom=2cm,left=2cm,right=2cm,marginparwidth=1.75cm]{geometry}

% Useful Packages
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{amsthm}
%\usepackage{bbm}
\usepackage{siunitx}
\usepackage{cite}
\DeclareMathOperator*{\argmax}{argmax}
\DeclareMathOperator*{\argmin}{argmin}
\DeclareMathOperator*{\softmax}{softmax}

\newcommand{\squeezeup}{\vspace{-2.5mm}}
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}

\pagestyle{fancy}
\lhead{Appendix: Functions}
\rhead{}

\begin{document}

\title{Optimization on Manifolds, Appendix: Functions}
\author{Ibrahim Akbar}
\maketitle

\section{Definition}

There is no generally accepted manner for defining a function, it's domain, or it's codomain, thus this section is rather used as a means to denote the book's conventions for how these terms are defined. Furthermore to provide a more thorough introdution into functions there is a more abstract definition, \textbf{morphisms}.\\

\noindent A \textbf{function}, $f$, is a mapping from one set to another set,
$$
f: A\rightarrow B
$$
such that $f$ is a set of ordered pairs $(a,b)$ where $a\in A$ and $b\in B$ and has the condition that if $(a,b)\in f$ and $(a,c)\in f$ then $b=c$. This means that an element in the set $A$ may only be the 1\textsuperscript{st} element of 1 ordered pair. Otherwise the function would be ambigious allowing an input to map to multiple outputs. If $(a,b)\in f$ then we can say $b = f(a)$. Note that $f(a)$ need not be defined for all $a\in A$ in this book. Thus this definition of a function aligns more with the concept of \textbf{partial functions} which may be seen as a generalization of a function by not requiring it to map every element of $A$ to $B$.\\

\noindent The \textbf{domain} of a function $f$, is also denoted as the input and can be defined as the elements in $A$ such that there exists an element in $B$ in order for the ordered pair $(a,b)$ to be in $f$.
$$
dom(f) := \{a\in A: \exists b\in B: (a,b)\in f\}
$$

\noindent The \textbf{range} of a function $f$, also known as the image can be seen as the resulting set based on applying the function to the elements in $A$.
$$
range(f) := \{b\in B:\exists a\in A:(a,b)\in f\}
$$

\noindent The \textbf{pre-image} of a function $f$ we say is $f(Y)^{-1}$ and is defined as $f(Y)^{-1} := \{a\in A:f(a)\in Y:Y\subseteq B\}$. This means that the pre-image is a subset of the domain.\\

\noindent A \textbf{surjective} function is one that maps \textit{onto} a set. More specifically for every element $b\in B$ there must exists an element $a\in A$ such that $f(a) = b$. This is equivalent to saying that a functions range is equivalent to the codomain.
$$
\forall b\in B\hspace{2mm}\exists a\in A : f(a) = b
$$
A \textbf{codomain} is the set that $f$ can map to similar to the domain which is the set that $f$ maps from.\\

\noindent An \textbf{injective} function $f$ such that if $a_{1}\neq a_{2}$ then $f(a_{1})\neq f(a_{2})$. Simple examples for visualization of injective and non-injective would be $x\mapsto x$ and $x\mapsto x^{2}$, respectively. A \textbf{bijective} function is one that is both surjective and injective and thus has a one-to-one correspondence between the sets. Note, that in this book, the codomain and range are not differentiated (commonly done in set-theory or graph definitions) and thus a bijective function need only be injective since every function is surjective.\\

\noindent A \textbf{morphism} is a structure-preserving map between algebraic structures. That means for set theory it is functions, linear algebra uses linear transformations, topology uses continuous functions, etc.\\

\noindent A \textbf{homomorphism} is a morphism between algebraic structure of the same type.\\

\noindent An \textbf{isomorphism} is a homomorphism that admits an inverse mapping. This means that $f: A\rightarrow B$ is an isomorphism if and only if there exists a function $g$ such that if $f(a) = b$ then $g(b) = a$. For a morphism/function $f$ to admit an inverse $f$ must be bijective as otherwise the inverse function $g$ would not hold the original property nor would $g$ be on the set $B$.\\

\noindent A \textbf{homeomorphism} is an isomorphism that is continuous. For topological spaces a definition of a \textbf{continuous function} $f$ is a function such that for every open set $V\in B$ the pre-image is an open set in $A$.\\

\noindent A \textbf{diffeomorphism} is an isomorphism between smooth manifolds. Smooth simply means that the structure belongs to $C^{\infty}$ and hence can be differentiated infinitely many times. This is not to be confused with a \textbf{differentiable manifold} which is a generalization of a smooth manifold.\\
\end{document}


%--------------------------------------------------------------------
% NE 155 (intro to numerical simulation of radiation transport)
% Spring 2021

% Exam Template from UMTYMP and Math Department courses
%
% Using Philip Hirschhorn's exam.cls: http://www-math.mit.edu/~psh/#ExamCls
%
% run pdflatex on a finished exam at least three times to do the grading table on front page.
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% These lines can probably stay unchanged, although you can remove the last
% two packages if you're not making pictures with tikz.
\documentclass[12pt]{article}
\RequirePackage{amssymb, amsfonts, amsmath, latexsym, verbatim, xspace, setspace}
\RequirePackage{tikz, pgflibraryplotmarks}
\usetikzlibrary{arrows}

% By default LaTeX uses large margins.  This doesn't work well on exams; problems
% end up in the "middle" of the page, reducing the amount of space for students
% to work on them.
\usepackage[margin=1in]{geometry}
\usepackage{enumerate}
\usepackage{graphicx}

\newcommand{\nth}{n\ensuremath{^{\text{th}}} }
\newcommand{\ve}[1]{\ensuremath{\mathbf{#1}}}
\newcommand{\Macro}{\ensuremath{\Sigma}}
\newcommand{\vOmega}{\ensuremath{\hat{\Omega}}}

% For an exam, single spacing is most appropriate
\singlespacing
% \onehalfspacing
% \doublespacing

% For an exam, we generally want to turn off paragraph indentation
\parindent 0ex

\begin{document}

\begin{flushright}
\begin{tabular}{p{5in} r l}
NE 155 & Spring 2021 \\
2D DE Reflecting Boundaries
\end{tabular}
\end{flushright}
\rule[1ex]{\textwidth}{.1pt}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% See http://www-math.mit.edu/~psh/#ExamCls for full documentation, but the questions
% below give an idea of how to write questions [with parts] and have the points
% tracked automatically on the cover page.
%
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Derive the matrix expressions each reflecting boundary in the 2-D diffusion equation formulation derived with the finite volume method and describe the associated equations.

In 2-D:
\[-\frac{\partial}{\partial x}D(x,y)\frac{\partial}{\partial x} \phi(x,y) -\frac{\partial}{\partial y}D(x,y)\frac{\partial}{\partial y} \phi(x,y) + \Sigma_a(x,y) \phi(x,y) = S(x,y)\]
%
with $x \in [0,a]$ and $y \in [0,b]$.

%-----------------------------------------------------------------------
\subsection*{LEFT}
The left side boundary condition is:
\[\frac{\partial }{\partial x} \phi(x,y)\big|_{x=0} = 0\]

\begin{minipage}{0.45\textwidth}
\begin{tikzpicture}
% grid
\draw[step=2cm] (0,0) grid (2,4);
% volume labels
\node at (1,1) {$V_{1,j}$};
\node at (1,3) {$V_{1,j+1}$};
% surface labels
\node[left] at (-.25,3) {$s_8$};
\node[right] at (2,3) {$s_5$};
\node[left] at (-.25,1) {$s_1$};
\node[right] at (2,1) {$s_4$};
\node[above] at (1,4) {$s_6$};
\node[below] at (1,0) {$s_3$};
% surrounding grid and markers
\draw plot[mark=*, mark options={fill=black}] (0,2) node[left] {$(0,j)$};
\draw (2,2) -- plot[mark=*, mark options={fill=black}] (4,2) node[right] {$(1,j)$};
\draw (0,0)-- plot[mark=*, mark options={fill=black}] (0,-2) node[below] {$(0,j-1)$};
\draw (0,4)-- plot[mark=*, mark options={fill=black}] (0,6) node[above] {$(0,j+1)$};
% arrows
\draw[<->](3.5,2) -- (3.5,6);
\node[right] at (3.5,4) {$\epsilon_{j+1}$};
\draw[<->](3.5,-2) -- (3.5,2);
\node[right] at (3.5,0) {$\epsilon_{j}$};
\draw[<->](0, -1.5) -- (4,-1.5);
\node[below] at (2, -1.5) {$\delta_{1}$};
\end{tikzpicture}
\end{minipage}
%
\begin{minipage}{0.5\textwidth}
To implement the boundary condition, we integrate in the first half-cell, so from $x =0$ to $x=\delta_1 / 2$.

Recall that when we considered the streaming term we converted the volume integral into a surface integral:
\begin{align}
&-\int_V d\vec{r}\:\bigl[\nabla \cdot \bigl(D(\vec{r})\nabla \phi(\vec{r})\bigr)\bigr] \nonumber \\
 &= -\int_S d\vec{S} \:D(\vec{r})\frac{\partial}{\partial \hat{n}}\phi(\vec{r}) \nonumber
\end{align}
%
and we defined the partial derivative w.r.t.\ direction on each surface. We now have two fewer surfaces (no $s_2$ or $s_7$), and we apply the zero current boundary condition to $s_1$ and $s_8$.
\end{minipage}

\begin{align}
\frac{\partial}{\partial \hat{n}}\phi(\vec{r}) &= \frac{\phi_{0,j-1} - \phi_{0,j}}{\epsilon_j} \qquad \text{on } S_3 \nonumber \\
%
&= \frac{\phi_{0,j+1} - \phi_{0,j}}{\epsilon_{j+1}} \qquad \text{on } S_6 \nonumber \\
%
&= \frac{\phi_{1,j} - \phi_{0,j}}{\delta_{1}} \qquad \text{on } S_4 \:, S_5 \nonumber \\
%
&= 0 \qquad \qquad \qquad \text{on } S_1 \:, S_8 \nonumber
\end{align}
%
We then use the midpoint rule for the integration and integrate along each surface. Recall that the physics values are cell-centered while the flux is edge-centered. The two terms that are different are for the top and bottom of the cell:
%
\begin{align*}
- \int_{S_3} d\vec{S} \:D(\vec{r})\frac{\partial}{\partial \hat{n}}\phi(\vec{r}) &= \frac{\phi_{0,j} - \phi_{0,j-1}}{\epsilon_{j}} \biggl(\frac{D_{1,j} \delta_{1}}{2}\biggr) \: \\
%
- \int_{S_6} d\vec{S} \:D(\vec{r})\frac{\partial}{\partial \hat{n}}\phi(\vec{r}) &= \frac{\phi_{0,j+1} - \phi_{0,j}}{\epsilon_{j+1}} \biggl(\frac{D_{1,j+1} \delta_{1}}{2}\biggr) \:\\
%
S_1+S_8 &= 0\:,\\
%
S_4+S_5 &= \frac{\phi_{1,j} - \phi_{0,j}}{\delta_{1}} \biggl(\frac{D_{1,j} \epsilon_{j} + D_{1,j+1} \epsilon_{j+1}}{2}\biggr)\:.
\end{align*}

The absorption and streaming terms now become:
\begin{align}
\int \int dx dy\:\Sigma_a(x,y) \phi(x,y) &= \boxed{\phi_{0,j}\bigl(\Sigma_{a,1,j} V_{1,j} + \Sigma_{a,1,j+1} V_{1,j+1} \bigr) \equiv \Sigma^*_{a,0j}}\:, \nonumber \\
%
\int \int dx dy \: \:S(x,y) &= \boxed{S_{1,j} V_{1,j} + S_{1,j+1} V_{1,j+1} \equiv S^*_{0j}}\:. \nonumber
\end{align}

%-----------------------------------
\vspace*{2em}
Collecting all of the terms and separating them, we get a 4-point difference equation for $i=0$; $j=1,\dots,m-1$:
%
\[\boxed{a_{1,j}^{0j}\phi_{1,j} + a_{0,j-1}^{*,0j}\phi_{0,j-1} + a_{0,j+1}^{*,0j}\phi_{0,j+1} +  a_{0,j}^{*,0j}\phi_{0,j} = S^*_{0j}} \]
%
Recall: the lower index is the cell to which you are coupling, and the upper index is which cell you are in. The coefficients how change to become
\begin{align}
a_{1,j}^{0j} &= -\frac{D_{1,j} \epsilon_{j} + D_{1,j+1} \epsilon_{j+1}}{2 \delta_{1}}  \nonumber \\
%
a_{0,j-1}^{*,0j} &= -\frac{D_{1,j} \delta_{1}}{2 \epsilon_{j}}  \nonumber \\
%
a_{0,j+1}^{*,0j} &= -\frac{D_{1,j+1} \delta_{1}}{2 \epsilon_{j+1}}  \nonumber \\
%
a_{0,j}^{*,0j} &= \Sigma^*_{a,0j} - \bigl(a_{1,j}^{0j} + a_{0,j-1}^{*,0j} + a_{0,j+1}^{*,0j} \bigr)
 \:.\nonumber
\end{align}

%-----------------------------------------------------------------------
%\subsection*{RIGHT}
%The right side boundary condition is:
%\[\frac{\partial }{\partial x} \phi(x,y)\big|_{x=a} = 0\]
%
%\begin{minipage}{0.45\textwidth}
%\begin{tikzpicture}
%% grid
%\draw[step=2cm] (2,0) grid (4,4);
%% volume labels
%\node at (3,1) {$V_{n,j}$};
%\node at (3,3) {$V_{n,j+1}$};
%% surface labels
%\node[left] at (2,3) {$s_8$};
%\node[right] at (4,3) {$s_5$};
%\node[left] at (2,1) {$s_1$};
%\node[right] at (4,1) {$s_4$};
%\node[above] at (3,4) {$s_7$};
%\node[below] at (3,0) {$s_2$};
%% surrounding grid and markers
%\draw plot[mark=*, mark options={fill=black}] (0,2) node[left] {$(n-1,j)$};
%\draw (0,2) -- plot[mark=*, mark options={fill=black}] (4,2) node[right] {$(n,j)$};
%\draw (4,0)-- plot[mark=*, mark options={fill=black}] (4,-2) node[below] {$(n,j-1)$};
%\draw (4,4)-- plot[mark=*, mark options={fill=black}] (4,6) node[above] {$(n,j+1)$};
%% arrows
%\draw[<->](0.5,2) -- (0.5,6);
%\node[left] at (0.5,4) {$\epsilon_{j+1}$};
%\draw[<->](0.5,-2) -- (0.5,2);
%\node[left] at (0.5,0) {$\epsilon_{j}$};
%\draw[<->](0, -1.5) -- (4,-1.5);
%\node[below] at (2, -1.5) {$\delta_{n}$};
%\end{tikzpicture}
%\end{minipage}
%%
%\begin{minipage}{0.5\textwidth}
%To implement the boundary condition, we integrate in the last half-cell, so from $x=\delta_n / 2$ to $x=a$.
%
%Recall that when we considered the streaming term we converted the volume integral into a surface integral:
%\begin{align}
%&-\int_V d\vec{r}\:\bigl[\nabla \cdot \bigl(D(\vec{r})\nabla \phi(\vec{r})\bigr)\bigr] \nonumber \\
% &= -\int_S d\vec{S} \:D(\vec{r})\frac{\partial}{\partial \hat{n}}\phi(\vec{r}) \nonumber
%\end{align}
%%
%and we defined the partial derivative w.r.t.\ direction on each surface. We now have two fewer surfaces (no $s_3$ or $s_6$), and we apply the zero current boundary condition to $s_4$ and $s_5$.
%\end{minipage}
%
%\begin{align}
%\frac{\partial}{\partial \hat{n}}\phi(\vec{r}) &= \frac{\phi_{n,j-1} - \phi_{n,j}}{\epsilon_j} \qquad \text{on } S_2  \nonumber \\
%%
%&= \frac{\phi_{n,j+1} - \phi_{n,j}}{\epsilon_{j+1}} \qquad \text{on } S_7 \nonumber \\
%%
%&= \frac{\phi_{n-1,j} - \phi_{n,j}}{\delta_{n}} \qquad \text{on } S_1 \:, S_8 \nonumber \\
%%
%&= 0 \qquad \qquad \qquad \text{on } S_4 \:, S_5 \nonumber
%\end{align}
%%
%We then use the midpoint rule for the integration and integrate along each surface. Recall that the physics values are cell-centered while the flux is edge-centered. The two terms that are different are for the top and bottom of the cell:
%%
%\begin{align}
%- \int_{S_2} d\vec{S} \:D(\vec{r})\frac{\partial}{\partial \hat{n}}\phi(\vec{r}) &= \frac{\phi_{n,j} - \phi_{n,j-1}}{\epsilon_{j}} \biggl(\frac{D_{n,j} \delta_{n}}{2}\biggr) \nonumber \\
%%
%- \int_{S_7} d\vec{S} \:D(\vec{r})\frac{\partial}{\partial \hat{n}}\phi(\vec{r}) &= \frac{\phi_{n,j+1} - \phi_{n,j}}{\epsilon_{j+1}} \biggl(\frac{D_{n,j+1} \delta_{n}}{2}\biggr) \nonumber
%\end{align}
%
%The absorption and streaming terms now become:
%\begin{align}
%\int \int dx dy\:\Sigma_a(x,y) \phi(x,y) &= \boxed{\phi_{n,j}\bigl(\Sigma_{a,n,j} V_{n,j} + \Sigma_{a,n,j+1} V_{n,j+1} \bigr) \equiv \Sigma^*_{a,nj}}\:, \nonumber \\
%%
%\int \int dx dy \: \:S(x,y) &= \boxed{S_{n,j} V_{n,j} + S_{n,j+1} V_{n,j+1} \equiv S^*_{nj}}\:. \nonumber
%\end{align}
%
%%-----------------------------------
%\vspace*{2em}
%Collecting all of the terms and separating them, we get a 4-point difference equation for $i=n$; $j=1,\dots,m-1$:
%%
%\[\boxed{a_{n-1,j}^{*,nj}\phi_{n-1,j} + a_{n,j-1}^{*,nj}\phi_{n,j-1} + a_{n,j+1}^{*,nj}\phi_{n,j+1} +  a_{n,j}^{*,nj}\phi_{n,j} = S^*_{nj}} \]
%%
%Recall: the lower index is the cell to which you are coupling, and the upper index is which cell you are in. The coefficients how change to become
%%
%\begin{align}
%a_{n-1,j}^{*,nj} &= -\frac{D_{n,j} \epsilon_{j} + D_{n,j+1} \epsilon_{j+1}}{2 \delta_{n}}  \nonumber \\
%%
%a_{n,j-1}^{*,nj} &= -\frac{D_{n,j} \delta_{n}}{2 \epsilon_{j}}  \nonumber \\
%%
%a_{n,j+1}^{*,nj} &= -\frac{D_{n,j+1} \delta_{n}}{2 \epsilon_{j+1}}  \nonumber \\
%%
%a_{n,j}^{*,nj} &= \Sigma_{a,nj} - \bigl(a_{n-1,j}^{*,nj} + a_{n,j-1}^{*,nj} + a_{n,j+1}^{*,nj} \bigr)
% \:.\nonumber
%\end{align}

%-----------------------------------------------------------------------
\newpage
\subsection*{TOP}
The top boundary condition is:
\[\frac{\partial }{\partial y} \phi(x,y)\big|_{y=b} = 0\]

\begin{tikzpicture}
% grid
\draw[step=2cm] (2,2) grid (6,4);
% volume labels
\node at (3,3) {$V_{i,m}$};
\node at (5,3) {$V_{i+1,m}$};
% surface labels
\node[left] at (2,3) {$s_1$};
\node[below] at (3,2) {$s_2$};
\node[below] at (5,2) {$s_3$};
\node[right] at (6,3) {$s_4$};
\node[above] at (5,4) {$s_6$};
\node[above] at (3,4) {$s_7$};
% surrounding grid and markers
\draw plot[mark=*, mark options={fill=black}] (0,4) node[above] {$(i-1,m)$};
\draw (0,4) -- plot[mark=*, mark options={fill=black}] (4,4) node[above] {$(i,m)$};
\draw (4,4)-- plot[mark=*, mark options={fill=black}] (4,0) node[below] {$(i,m-1)$};
\draw (4,4)-- plot[mark=*, mark options={fill=black}] (8,4) node[above] {$(i+1,m)$};
% arrows
\draw[<->](0,0.5) -- (4,0.5);
\node[below] at (2,0.5) {$\delta_{i}$};
\draw[<->](4,0.5) -- (8,0.5);
\node[below] at (6,0.5) {$\delta_{i+1}$};
\draw[<->](0.5, 0) -- (0.5,4);
\node[left] at (0.5,2) {$\epsilon_{m}$};
\end{tikzpicture}

To implement the boundary condition, we integrate in the last half-cell, so from $y=b-\epsilon_m / 2$ to $y=b$.

Recall that when we considered the streaming term we converted the volume integral into a surface integral:
\begin{align}
&-\int_V d\vec{r}\:\bigl[\nabla \cdot \bigl(D(\vec{r})\nabla \phi(\vec{r})\bigr)\bigr] \nonumber \\
 &= -\int_S d\vec{S} \:D(\vec{r})\frac{\partial}{\partial \hat{n}}\phi(\vec{r}) \nonumber
\end{align}
%
and we defined the partial derivative w.r.t.\ direction on each surface. We now have two fewer surfaces (no $s_8$ or $s_5$), and we apply the zero current boundary condition to $s_7$ and $s_6$.

\begin{align}
\frac{\partial}{\partial \hat{n}}\phi(\vec{r}) &= \frac{\phi_{i-1,m} - \phi_{i,m}}{\delta_{i}} \qquad \text{on } S_1  \nonumber \\
%
&= \frac{\phi_{i+1,m} - \phi_{i,m}}{\delta_{i+1}} \qquad \text{on } S_4 \nonumber \\
%
&= \frac{\phi_{i,m-1} - \phi_{i,m}}{\epsilon_m} \qquad \text{on } S_2 \:, S_3 \nonumber \\
%
&= 0 \qquad \qquad \qquad \text{on } S_7 \:, S_6 \nonumber
\end{align}

We then use the midpoint rule for the integration and integrate along each surface. Recall that the physics values are cell-centered while the flux is edge-centered. The two terms that are different are for the left and right of the cell:
%
\begin{align}
- \int_{S_1} d\vec{S} \:D(\vec{r})\frac{\partial}{\partial \hat{n}}\phi(\vec{r}) &= \frac{\phi_{i,m} - \phi_{i-1,m}}{\delta_{i}} \biggl(\frac{D_{i,m} \epsilon_{m}}{2}\biggr) \nonumber \\
%
- \int_{S_4} d\vec{S} \:D(\vec{r})\frac{\partial}{\partial \hat{n}}\phi(\vec{r}) &= \frac{\phi_{i+1,m} - \phi_{i,m}}{\delta_{i+1}} \biggl(\frac{D_{i+1,m} \epsilon_{m}}{2}\biggr) \nonumber
\end{align}

The absorption and streaming terms now become:
\begin{align}
\int \int dx dy\:\Sigma_a(x,y) \phi(x,y) &= \boxed{\phi_{n,j}\bigl(\Sigma_{a,i,m} V_{i,m} + \Sigma_{a,i+1,m} V_{i+1,m} \bigr)  \equiv \Sigma^*_{a,im}}\:, \nonumber \\
%
\int \int dx dy \: \:S(x,y) &= \boxed{S_{i,m} V_{i,m} + S_{i+1,m} V_{i+1,m} \equiv S^*_{im}}\:. \nonumber
\end{align}

%-----------------------------------
\vspace*{2em}
Collecting all of the terms and separating them, we get a 4-point difference equation for $i=1,\dots,n-1$, $j=m$:
%
\[\boxed{a_{i-1,m}^{*,im}\phi_{i-1,m} + a_{i,m-1}^{*,im}\phi_{i,m-1} + a_{i+1,m}^{*,im}\phi_{i+1,m} +  a_{i,m}^{*,im}\phi_{i,m} = S^*_{im}} \]
%
Recall: the lower index is the cell to which you are coupling, and the upper index is which cell you are in. The coefficients how change to become
%
\begin{align}
a_{i-1,m}^{*,im} &= -\frac{D_{i,m} \epsilon_{m}}{2\delta_{i}}  \nonumber \\
%
a_{i,m-1}^{*,im} &= -\frac{D_{i,m} \delta_{i} + D_{i+1,m} \delta_{i+1}}{2 \epsilon_{m}}  \nonumber \\
%
a_{i+1,m}^{*,im} &= -\frac{D_{i+1,m} \epsilon_{m}}{2\delta_{i+1}} \nonumber \\
%
a_{n,j}^{*,im} &= \Sigma_{a,im} - \bigl(a_{i-1,m}^{*,im} + a_{i,m-1}^{*,im} + a_{i+1,m}^{*,im} \bigr)
 \:.\nonumber
\end{align}

\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% UMB-CS110-2015S: Introduction to Computing
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/UMB-CS110-2015S
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\def \topDirectory {../..}
\def \texDirectory {\topDirectory/src/main/tex}

\documentclass[10pt, compress]{beamer}

\usepackage{\texDirectory/template/style/directives}
\input{\texDirectory/template/config}
\usepackage{\texDirectory/template/style/beamerthemeUmassLecture}
\usepackage[school]{\texDirectory/template/pgf-umlcd/pgf-umlcd}
\doc{number}{12}
%\setbeamertemplate{footline}[text line]{}

\begin{document}
\prepareCover

\section{Course Administration}

\begin{frame}[fragile]
\frametitle{Course Administration}
Assignment 4 released. Due on April 16, 2015 at 17:30 PM.
\end{frame}

\begin{frame}[fragile]
  \frametitle{Overview}
  \begin{itemize}
    \item[] Abstraction
    \begin{itemize}
      \item[] Introduction
      \item[] Abstract Classes and Methods
    \end{itemize}
  \end{itemize}
\end{frame}

\plain{}{Abstraction}

\section{Introduction}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{How Do We Think?}
    \begin{quote}
      - Hi Sara! What's up?\\
      + Hey Mike! What's up?
    \end{quote}
  \end{block}
  \begin{block}{How Do We Really Think?}
    \begin{quote}
      What did you do today?
    \end{quote}
  \end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{Objective}
    Write a program \texttt{Geometry.java} in which you can create cricles and squares and get their area, perimeter, color and whether they are filled or not.
  \end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{UML Diagram for Class \texttt{Shape}}
  \begin{figure}
  \centering
    \begin{tikzpicture}
      \begin{class}[]{Shape}{0, 0}
        \attribute{- numShapes: int}
        \attribute{- filled: boolean}
        \attribute{- color: String}
        \operation{+ Shape()}
        \operation{+ getNumShape(): int}
        \operation{+ isFilled(): boolean}
        \operation{+ setFilled(boolean filled)}
        \operation{+ getColor(): String}
        \operation{+ setColor(String color)}
      \end{class}
    \end{tikzpicture}
  \end{figure}
  \end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{UML Diagram for Class \texttt{Circle}}
  \begin{figure}
  \centering
    \begin{tikzpicture}
      \begin{class}[]{Circle}{0, 0}
        \attribute{- numCircles: int}
        \attribute{- radius: double}
        \operation{+ Circle()}
        \operation{+ getPerimeter(): double}
        \operation{+ getArea(): double}
        \operation{+ getNumCircles(): int}
        \operation{+ getRadius(): double}
        \operation{+ setRadius(double radius)}
      \end{class}
    \end{tikzpicture}
  \end{figure}
  \end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{UML Diagram for Class \texttt{Square}}
  \begin{figure}
  \centering
    \begin{tikzpicture}
      \begin{class}[]{Square}{0, 0}
        \attribute{- numSquares: int}
        \attribute{- length: double}
        \operation{+ Square()}
        \operation{+ getPerimeter(): double}
        \operation{+ getArea(): double}
        \operation{+ getNumSquares(): int}
        \operation{+ getLength(): double}
        \operation{+ setLength(double radius)}
      \end{class}
    \end{tikzpicture}
  \end{figure}
  \end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{\texttt{Shape.java} (v2.0) (Part 1)}
    \begin{minted}[fontsize=\small,tabsize=8, linenos, firstnumber=1]{java}
public class Shape {
  // attributes and fields
  private static int numShapes = 0;
  private String color;
  private boolean filled;
  // constructors
  public Shape() {
    numShapes++;
  }
  // setter and getter for color
  public void setColor(String someColor) {
    this.color = someColor;
  }
  public String getColor() {
    return color;
  }
    \end{minted}
  \end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{\texttt{Shape.java} (v2.0) (Part 2)}
    \begin{minted}[fontsize=\small,tabsize=8, linenos, firstnumber=17]{java}
  // setter and getter for filled
  public void setFilled(boolean state) {
    this.filled = state;
  }
  public boolean isFilled() {
    return filled;
  }
  // setter and getter for numShapes
  public static int getNumShapes() {
    return numShapes;
  }
}
    \end{minted}
  \end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{\texttt{Circle.java} (v2.0) (Part 1)}
    \begin{minted}[fontsize=\small,tabsize=8, linenos, firstnumber=1]{java}
public class Circle extends Shape {
  // fields and attributes
  private static int numCircles = 0;
  private double radius;
  // methods
  public double getArea() {
    return Math.PI*Math.pow(radius,2);
  }
  public double getPerimeter() {
    return 2*Math.PI*radius;
  }
  // constructors
  public Circle(double someRadius) {
    radius = someRadius;
    numCircles++;
  }
    \end{minted}
  \end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{\texttt{Circle.java} (v2.0) (Part 2)}
    \begin{minted}[fontsize=\small,tabsize=8, linenos, firstnumber=17]{java}
  // setter and getter for radius
  public void setRadius(double someRadius) {
    this.radius = someRadius;
  }
  public double getRadius() {
    return radius;
  }
  // setter and getter for numCircles
  public static int getNumCircles() {
    return numCircles;
  }
}
    \end{minted}
  \end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{\texttt{Square.java} (v2.0) (Part 1)}
    \begin{minted}[fontsize=\small,tabsize=8, linenos, firstnumber=1]{java}
public class Square extends Shape {
  // fields and attributes
  private static int numSquares = 0;
  private double length;
  // methods
  public double getArea() {
    return Math.pow(length,2);
  }
  public double getPerimeter() {
    return length*4;
  }
  // constructors
  public Square(double someLength) {
    length = someLength;
    numSquares++;
  }
    \end{minted}
  \end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{\texttt{Square.java} (v2.0) (Part 2)}
    \begin{minted}[fontsize=\small,tabsize=8, linenos, firstnumber=17]{java}
  // setter and getter for length
  public void setLength(double someLength) {
    this.length = someLength;
  }
  public double getLength() {
    return length;
  }
  // setter and getter for numSquares
  public static int getNumSquares() {
    return numSquares;
  }
}
    \end{minted}
  \end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{\texttt{Geometry.java} (v2.0)}
    \begin{minted}[fontsize=\small,tabsize=8, linenos, firstnumber=1]{java}
public class Geometry {
  public static void main(String[] args) {
    Circle myCircle = new Circle(5);
    myCircle.setColor("Red");
    myCircle.setFilled(true);
    double result1 = myCircle.getPerimeter();
    System.out.println("Perimeter of circle: "+result1);
    Square mySquare = new Square(4);
    mySquare.setColor("Blue");
    mySquare.setFilled(false);
    double result2 = mySquare.getArea();
    System.out.println("Area of square: "+result2);
    System.out.println(Shape.getNumShapes());
  }
}
    \end{minted}
  \end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{\texttt{UML Diagram (v2.0)}}
  \begin{figure}
  \centering
    \begin{tikzpicture}
      \begin{class}[]{Shape}{0, 0}
        \attribute{numShapes: int}
        \attribute{filled: boolean}
        \attribute{color: String}
        \operation{Shape()}
      \end{class}
      \begin{class}[]{Circle}{-3, -3.25}
        \inherit{Shape}
        \attribute{numCircles: int}
        \attribute{radius: double}
        \operation{Circle()}
        \operation{getPerimeter(): double}
        \operation{getArea(): double}
      \end{class}
      \begin{class}[]{Square}{3, -3.25}
        \inherit{Shape}
        \attribute{numSquares: int}
        \attribute{length: double}
        \operation{Square()}
        \operation{getPerimeter(): double}
        \operation{getArea(): double}
      \end{class}
    \end{tikzpicture}
  \end{figure}
  \end{block}
\end{frame}

\section{Abstract Classes and Methods}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{Problem Statement}
    \begin{itemize}
      \item[] Superclass can be instantiated.
      \item[] Subclasses may not implement \texttt{getPerimeter()} and \texttt{getArea()} methods.
    \end{itemize}
    \begin{minted}[fontsize=\small,tabsize=8, linenos, firstnumber=1]{java}
public class Evil {
  public static void main(String[] args) {
    Shape myShape = new Shape();
    myShape.setColor("Red");
    myShape.setFilled(true);
  }
}
    \end{minted}
  \end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{Proposed Solution}
    Disallow instantiation of the \texttt{Shape} class.

    Force all subclasses of \texttt{Shape} class to implement \texttt{getArea()} and \texttt{getPerimeter()} methods.
  \end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{Definition}
    Abstraction is the act of hiding internal details of implementation of behaviors of object and providing functionalities of the object instead.
  \end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{Abstract Methods}
    Abstract methods define signature of methods that must be implemented in subclasses.
    \begin{minted}[fontsize=\small,tabsize=8, linenos, firstnumber=1]{java}
public abstract void getArea();
public abstract void getPerimeter();
    \end{minted}
  \end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{Abstract Classes}
    Classes declared as abstract may not be instantiated.

    Abstract classes may or may not have abstract methods.

    Classes with abstract methods must be declared as abstract.
  \end{block}
    \begin{minted}[fontsize=\small,tabsize=8, linenos, firstnumber=1]{java}
public abstract class Shape {
  public abstract double getArea();
  public abstract double getPerimeter();
  // other class members
}
    \end{minted}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{Remember}
    Subclasses extending an abstract class must implement its abstract methods or be declared as abstract.
  \end{block}
    \begin{minted}[fontsize=\small,tabsize=8, linenos, firstnumber=1]{java}
public class Circle {
  private double radius;
  @Override
  public double getArea() {
    return Math.PI*Math.pow(radius,2);
  }
  @Override
  public double getPerimeter() {
    return this.radius*Math.PI*2;
  }
  // other class members
}
    \end{minted}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{\texttt{Shape.java} (v3.0) (Part 1)}
    \begin{minted}[fontsize=\small,tabsize=8, linenos, firstnumber=1]{java}
public abstract class Shape {
  // fields
  private static int numShapes = 0;
  // attributes
  private String color;
  private boolean filled;
  // constructors
  public Shape() {
    numShapes++;
  }
  // abstract methods
  public abstract double getPerimeter();
  public abstract double getArea();
    \end{minted}
  \end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{\texttt{Shape.java} (v3.0) (Part 2)}
    \begin{minted}[fontsize=\small,tabsize=8, linenos, firstnumber=14]{java}
  // setters and getters
  public static int getNumShapes() {
    return numShapes;
  }
  public void setColor(String someColor) {
    this.color = someColor;
  }
  public String getColor() {
    return color;
  }
  public void setFilled(boolean state) {
    this.filled = state;
  }
  public boolean isFilled() {
    return filled;
  }
}
    \end{minted}
  \end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{\texttt{Circle.java} (v3.0) (Part 1)}
    \begin{minted}[fontsize=\small,tabsize=8, linenos, firstnumber=1]{java}
public class Circle extends Shape {
  // fields
  private static int numCircles = 0;
  // attributes
  private double radius;
  // methods
  @Override
  public double getArea() {
    return Math.PI*Math.pow(radius,2);
  }
  @Override
  public double getPerimeter() {
    return 2*Math.PI*radius;
  }
    \end{minted}
  \end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{\texttt{Circle.java} (v3.0) (Part 2)}
    \begin{minted}[fontsize=\small,tabsize=8, linenos, firstnumber=15]{java}
  // constructors
  public Circle(double someRadius) {
    radius = someRadius;
    numCircles++;
  }
  // setter and getter for radius
  public void setRadius(double someRadius) {
    this.radius = someRadius;
  }
  public double getRadius() {
    return radius;
  }
  // setter and getter for numCircles
  public static int getNumCircles() {
    return numCircles;
  }
}
    \end{minted}
  \end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{\texttt{Square.java} (v3.0) (Part 1)}
    \begin{minted}[fontsize=\small,tabsize=8, linenos, firstnumber=1]{java}
public class Square extends Shape {
  // fields and attributes
  private static int numSquares = 0;
  private double length;
  // methods
  @Override
  public double getArea() {
    return Math.pow(length,2);
  }
  @Override
  public double getPerimeter() {
    return length*4;
  }
    \end{minted}
  \end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{\texttt{Square.java} (v3.0) (Part 2)}
    \begin{minted}[fontsize=\small,tabsize=8, linenos, firstnumber=15]{java}
  // constructors
  public Square(double someLength) {
    length = someLength;
    numSquares++;
  }
  // setters and getters
  public void setLength(double someLength) {
    this.length = someLength;
  }
  public double getLength() {
    return length;
  }
  public static int getNumSquares() {
    return numSquares;
  }
}
    \end{minted}
  \end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Abstraction}
  \begin{block}{\texttt{Geometry.java} (v3.0)}
    \begin{minted}[fontsize=\small,tabsize=8, linenos, firstnumber=1]{java}
public class Geometry {
  public static void main(String[] args) {
    Circle myCircle = new Circle(5);
    myCircle.setColor("Red");
    myCircle.setFilled(true);
    double result1 = myCircle.getPerimeter();
    System.out.println("Perimeter of circle: "+result1);
    Square mySquare = new Square(4);
    mySquare.setColor("Blue");
    mySquare.setFilled(false);
    double result2 = mySquare.getArea();
    System.out.println("Area of square: "+result2);
    System.out.println(Shape.getNumShapes());
  }
}
    \end{minted}
  \end{block}
\end{frame}

\plain{}{Keep Calm\\and\\Think Object-Oriented}

\end{document}


\chapter{To verify the truth table for AND Gate}
%\ref{sec:background}.

%\section{Aim}
%\label{sec:objectives}
%	To verify the truth table for AND Gate.

\section{Apparatus}
%\label{sec:objectives}
	\begin{itemize}
		\tightlist
		\item Kit for realization of gates
		\item Connecting Leads
	\end{itemize}

\section{Theory}
	The AND gate is an electronic circuit that gives a high output (1) only if all its inputs are high. A dot (.) is used to show the AND operation i.e. A.B or can be written as AB
	\begin{figure}[h]
		\centering
		\includegraphics{img/exp1/1}
		\caption{Symbol for AND gate}
		\label{fig:and_gate}
	\end{figure}
	\begin{figure}[h]
		\centering
		\includegraphics{img/exp1/2}
		\caption{Truth Table for AND gate}
		\label{fig:and_gate_table}
	\end{figure}
	A simple 2-input logic AND gate can be constructed using RTL (Resistor-Transistor-Logic) switches connected together as shown below with the inputs connected directly to the transistor bases. Both transistors must be saturated “ON” for an output at Q.

	\begin{figure}[h]
		\centering
		\includegraphics{img/exp1/3}
		\caption{Circut for making AND gate}
		\label{fig:and_diag}
	\end{figure}

\section{Procedure}
	\subsubsection{Simulator 1}
	\begin{itemize}
		\tightlist
		\item Connect the supply(+5V) to the circuit.
		\item Press the switches for inputs "A" and "B".
		\item The bulb does not glow if any one or both the switches (2 and 3) are OFF and glows only if both the switches (2 and 3) are ON.
		\item Repeat step-2 and step-3 for all state of inputs.
	\end{itemize}

	\subsubsection{Simulator 2}
	\begin{itemize}
		\tightlist
		\item Enter the Boolean input "A" and "B".
		\item Enter the Boolean output for your corresponding inputs.
		\item Click on "Check" Button to verify your output.
		\item Click "Print" if you want to get print out of Truth Table.
	\end{itemize}

\section{Observations}
	\begin{figure}[h]
		\centering
		\includegraphics[width=0.9\linewidth]{img/exp1/4}
		\caption{}
		\label{fig:1:4}
	\end{figure}
		\begin{figure}[h]
		\centering
		\includegraphics[width=0.9\linewidth]{img/exp1/5}
		\caption{}
		\label{fig:1:5}
	\end{figure}
		\begin{figure}[h]
		\centering
		\includegraphics[width=0.9\linewidth]{img/exp1/6}
		\caption{}
		\label{fig:1:6}
	\end{figure}
		\begin{figure}[h]
		\centering
		\includegraphics[width=0.9\linewidth]{img/exp1/7}
		\caption{}
		\label{fig:1:7}
	\end{figure}

\section{Conclusion}
Thus we can conclude that if both inputs are high then only the output is high but if even one of the inputs is low the output is low.

\section{Precautions}
	\begin{enumerate}
		\tightlist
		\item Make the connections when power supply is OFF.
		\item Ensure that the connections are tight.
		\item Change the status of inputs only when power supply is OFF.
	\end{enumerate}


\section{Proof Writing Guidelines}
\input{Ch4/proof1}

\section{Indirect Proof}
\input{Ch4/proof2}

\section{Why do we need proofs?}
\input{Ch4/whyproof}

\section{Direct Proofs}
\input{Ch4/directproof}

\section{Structure of Logical Claims}
\input{Ch4/proofforms}