\documentclass[11pt]{article}

\usepackage{alltt,fullpage,graphics,color,epsfig,amsmath, amssymb}
\usepackage{hyperref}
\usepackage{boxedminipage}
\usepackage[ruled,vlined]{algorithm2e}
\usepackage{amsmath}
\usepackage{commath}
\usepackage{graphicx}
\usepackage{amssymb,amsmath}
\usepackage{tikz}

\newcommand{\circleland}{
  \mathbin{
    \mathchoice
      {\buildcircleland{\displaystyle}}
      {\buildcircleland{\textstyle}}
      {\buildcircleland{\scriptstyle}}
      {\buildcircleland{\scriptscriptstyle}}
  }
}
\graphicspath{ {.} }
\newcommand{\floor}[1]{\lfloor #1 \rfloor}
\newcommand{\ceil}[1]{\lceil #1 \rceil}
\newcommand\buildcircleland[1]{%
  \begin{tikzpicture}[baseline=(X.base), inner sep=0, outer sep=0]
    \node[draw,circle] (X)  {$#1 x$};
  \end{tikzpicture}%
}
\title{CS 512 Midterm 2}
\author{Daniel Campos}
\date{April 30th,2021}
\begin{document}
\maketitle
\section{Short Questions}
\subsection{True Or False}
\subsubsection{For the Susceptible-Infected-Susceptible (SIS) model on network, whether there will be an epidemic is related with the selection of initial infected nodes.}
False, initial selection of infected nodes can effect if there is an epidemic but an epidemic for SIS is determined by the infection rate $\beta$ and the recovery rate $\delta$ if the $\beta$ is high and the $\delta$ is low few initalizations can effect epidemic or not and alternatively if $\beta$ is low and $\delta$ is high few initialization can cause an epidemic.
\subsubsection{Node attribute can provide effective assistance for network alignment task}
True. Methods such as Final: Attributed Network Alignment leverages the intuition that similar node-pairs share similar node attributes to predict alignment.
\subsubsection{Initializing all the weights with 0 during training will prevent the neural network from learning.}
False. If all weights are initialized to zero then the derivative for each weight in the network will be identical. Since each weight is identical each weight will receive the same update and the values in the network will always be equal $w_i = w_j \forall j \in N$. As a result, the network will produce poor results for any constant initialization. The network is technically learning as weight values are updating but it is very unlikely to learn a good representation of the data.
\subsubsection{Stochastic gradient descent (SGD) can help getting out of local minimums of the loss function}
True. SGD is more able to avoid getting stuck at local minima for the entire dataset because it is randomly sampling a batch from the dataset. Since it is not optimizing on the entire dataset at each step it is possible that the network has a loss on the entire dataset it cannot escape but eventually a random batch of the data could be different causing the network to leave the local minimum.
\subsubsection{If we randomly shuffle the order of nodes in a graph, we can have different results (e.g., node classification) from the GCN model.}
True. In a graph random shuffling of a graph does not change the graph but since GCN's rely on convolution shuffling node order will cause different outcomes. It worth noting that there are solutions to such as Simple Permutation-Invariant Graph
Convolutional Network and more regularly applied node ordering.
\subsection{Short Answer Questions}
\subsubsection{What problems might occur if the learning rate is fixed during training?}
The learning rate may be either too big or too small. Too big of a learning rate will cause the model to overshoot the global minimum during training. Too small will cause the model to move very slowly and can cause it to get stuck at local minima. While there could be a theoretically optimal fixed learning rate for each task without knowing that having some variability in learning rate will allow optimal exploration.
\subsubsection{List at least three challenges for network alignment.}
Complexity, variety, and Heterogeneity. Network alignment can be too complex in terms of space and time complexity which causes issues as network size grows. Network alignment can lack variety as networks can be rich in terms of nodes and edge attributes which can be forgotten in alignment. Network alignment can not be heterogeneous enough as networks may have different behaviors for each type of node but the network is aligning to the average.
\subsubsection{From the formula of Netshield, intuitively explain that why Netshield does not only select nodes with high eigen-scores to be vaccinated.}
Given the formula $Sv(s) = \sum_{i \in S} 2\lambda u (i)^2-\sum_{i,j \in S} A(i,j)u(i)u(j)$ we can pay attention to the second summation as it measures diversity between nodes. We do want to select the nodes with the highest eigen-scores but want to make sure they are diverse amongst themselves to maximize the impact on the overall network.
\subsubsection{Based on the idea of IONE (Lecture 8, before and including page 35), explain with your own words that whether we can learn node embeddings from two networks independently and align node pairs based on a distance measure (say cosine similarity}
We can learn the alignment between two networks independently as long as we assume that their distribution and relation is similar. Given the KL-Divergence we can learn some form of alignment between the two networks such that we minimize the divergence between the two networks. At its core the idea is to transform each network into a vector space and minimize the distance between the two. This makes perfect sense as we can see the real effect in how languages work. Each language is effectively a network of concepts which are connected by shared attributes (Cats and dogs are animals and pets but only cats are felines). If we transform and align the embeddings of the words to recognize that cats in English should be close to gato in Spanish then we can find an effective way of align the networks using embedding similarity vs alignment.
\subsubsection{In order to capture long-term dependency, what modifications does LSTM introduce to RNN? Briefly explain how it works to improve learning.}
LSTM's introduce learn able gating parameters to control the information that passes in the hidden state. RNNs suffer from a vanishing gradient where information from previous states disappears the as more information comes into the hidden state. LSTM's have a forget, input, and output gate which control what information is updated for the hidden state. This means the network learns what information to keep and what information to discard and as a result is able to learn longer distance dependencies.
\section{Deep Learning}
\subsection{Show how tanh can be transformed from sigmoid through shifting and re-scaling. }
As the sigmoid function is symmetric at origin and return [0,1] we can write $1 - \sigma(I) = \sigma(-I)$ which is then simplified into $1- \frac{1}{1+e^{-I}} = \frac{1}{1+e^{I}}$. \\
Similarly we can rearrange tanh function becomes $0 = \frac{e^I-e^{-I} -2e^{-I}}{e^I + e^{-I}}$ which becomes $= 1+ \frac{-2^{-I}}{e^I+e^{-I}} = 1 - \frac{2}{e^{2I}+1}$ .\\
If we simplify from the sigmoid perspective $tanh(I) = 1 - \frac{2}{e^{2I}+1} = 1-2\sigma(-2I)$ which in turn  $tanh(I)= 1-2(1-\sigma(2I)) = 1-2 + 2 \sigma (2I) = 2 \sigma (2I) -1$
which means tanh is shifted (the constant $-1$) and rescaled ($2 \sigma (2I)$) sigmoid.
\subsection{How many possible distinct and connected networks are there?}
Mathematically there could be as many as $2^N$ networks where N is the number of non output neurons. Since we have 3 layers that would mean that possible network combinations are $2^{(d_0)*(d_0)(d_1)(d_1)(d_2)(d_2)}$ but since the output layer must be fully connected we drop the final $d_2$. Additionally, since in each layer there must be at least 1 neuron connecting layers we update combinations to $2^{((d_0)-1)*((d_0)(d_1)-1)((d_1)(d_2)-1)}$ which becomes $2^{d_{0}^2d_{1}^2d_{2}-d_0^2d_1-d_0d_1d_2-d_0-d_0d_1^2d_2+d_0d_1-d_1d_2-1}$ unique subnetworks with dropout.
\subsection{Compute the input and the output of units 4, 5 and 6}
Results can be found in table \ref{tab:outputs}
\begin{table}[]
\begin{tabular}{|l|l|l|} \hline
Unit,j & Net Input, $I_j$ & Output, $O_j$             \\ \hline
4      & 0.6+0-0.3= 0.3   & $\max \{0,(0.3-0.2)\}=0.1$ \\  \hline
5      & 0.3+0+0.2 = 0.5  &  $\max \{0,(0.5+0.3)\}=0.8$ \\  \hline
6      & 0.03 + 0.56 = 0.59& $\max \{0,(0.59-0.1)\}=0.49$ \\  \hline
\end{tabular}
\caption{Inputs and outputs of neurons 4,5,6}
\label{tab:outputs}
\end{table}
\subsection{Compute the error of units 4, 5 and 6}
Results can be found in table \ref{tab:errors}
\begin{table}[]
\begin{tabular}{|l|l|} \hline
Unit,j & Error $\delta_j$ \\ \hline
6 & 0.49(0-0.49)(0-0.49) = 0.49(0.2401) = 0.117649 \\ \hline
5 & 0.8(0-0.8)(0.117649)(0.7) = -0.052706752 \\ \hline
4 & 0.1(0-0.1)(0.117649)(0.3) = -0.000352947 \\ \hline
\end{tabular}
\caption{Errors of  4,5,6}
\label{tab:errors}
\end{table}
\subsection{compute the new weights, $w_{1,5}$ and $w_{4,6}$, after one-step update.}
Results can be found in table \ref{tab:newweight}
\begin{table}[]
\begin{tabular}{|l|l|} \hline
weight & New Value \\ \hline
$w_{4,6}$ & 0.3 -(0.1)(0.117649)(0.1)= 0.29882351\\ \hline
$w_{1,5}$ & 0.3-(0.1)(-0.052706752)(1) = 0.3052706752 \\ \hline
\end{tabular}
\caption{New Weights for 4,5,6}
\label{tab:newweight}
\end{table}
\subsection{Suppose we do not apply zero-paddings, what are the dimensions of the feature maps after hidden layer 4? Please show the key steps of calculation}
After the first layer our feature map becomes 28*28*16 due to the the 16 filters and loosing 2 on each side of the 5x5 convolution. After max pool the size is 14x14*16 as max pool reduces 2x2 to 1x1. After the second convolutional layer the feature map is 12*12*32 because our 3x3 convolutions loses 1 on each dimension. Layer 4 then applies max pool which gives a size of 6x6x32 which is the answer to the question. Layer 5 produces a 1x1152
\subsection{How many weights does the architecture have? What advantage of CNN do you observe compared with fully-connected layer during the computation?}
The parameters we have are $p(CCN1) + p(CNN2) +p(FFN) + output$ since max pool and flatten have no parameters. This means $params = 16*5*5*3+32*3*3*16+1152*256+256*1= 1200+4608+294912+256 = 300,976$ parameters. \\ The advantages of a CNN vs FFN is faster training/inference because the relative sparsity of the network. Instead of having fully connected layers at each level there are sparsely connected layers which means the number of parameters and computations is orders of magnitude less.
\subsection{ Compute the output features after applying one spectral-based GCN layer}
$Z = D^{-0.5}AD^{-0.5}X\theta$ where $D = [[2,0,0,0,0],[0,3,0,0,0],[0,0,3,0,0],[0,0,0,3,0],[0,0,0,0,3]]$ and $A=[[0,1,0,0,1],[1,0,1,1,0],[0,1,0,1,1],[0,1,1,0,1],[1,0,1,1,0]]$ so Z = [[-0. ,  0.6],
       [-0.3,  0. ],
       [-0.3,  0.6],
       [-0.3,  0.6],
       [-0.3,  0. ]]
\subsection{Show how the categorical cross-entropy loss is calculated given T and O}
Starting with the Binary cross entropy $L(T,O) = -T \log O - (1-T) log(1-O)$ we can see that this is actually 2 class cross entropy losses but since we know that T and O have to be O or one we can write $T'=1-T$. Knowing that lets rewrite this as a 2 class cross entropy loss. First, our T goes becomes a one hot vector ($T_v$) as we make $T_V = [T, 1-T]$ and we do the same for our Output. As a result our loss function $L(T_v,O_v) = \sum_{i=0}^1 -T_i \log(O_i)$. Now, since we have to ensure that $\sum_{i=0}^1 T_i \log(O_i) = 1$ irregardless of how many classes we add we leverage softmax function and rewrite our CE loss as $CE(T,O) = - \sum t_i \log \frac{e^{O_i}}{\sum_j^C e^{O_j}}$
\section{Outlier Detection}
\subsection{which value is a potential outlier}
First we calculate $\mu=16.33636363636364$ as $\mu$ is the mean. Next we calculate $\sigma= \sqrt{\frac{\sum_i (x_i-\mu)^2}{N}}= 1.4846320188534794$\\
Next we calculate the z scores \\$=[ 0.24493,0.1592,0.3122,0.37964,3.0555,0.37964,0.1775, 0.3122,0.851,0.0244,0.581]$. \\
Looking at Z scores we can see that 11.8 is an outlier with a z score of over 3.
\subsection{Compute the local reach ability density of B and F}
First off we calculate the $dist_3 = [2,3,2,3,3,5,5]$ and the neighborhoods are \\
$[[b,c,d],[a,c,e],[a,b,d],[a,c,e,g],[b,c,d],[c,d,e],[a,c,d]]$.\\
Using this we calculate the Local reachability density for each item with results:\\
$= [0.13636,0.11538, 0.16666, 0.2, 0.12, 0.08823, 0.08571$ which means : \\
$lrd_3(b)=0.11538$ and  $lrd_3(f)=00.08823$
\subsection{compute the local outlier factor of A and F}
$LOF_3(A) = 1.1783475783475785$ and $LOF_3(F)= 1.8385185185185182$
\subsection{what are the possible outliers from the perspective of local outlier factor and why?}
The LOF for all points are \\$= 1.1783, 1.222, 0.903, 0.6359, 1.339, 1.838, 1.956$ \\ which makes G and F likely outliers as their coefficient is highest.
\section{Graph Connectivity Optimization }
\subsection{Report $\zeta_0$ and $p_1$}
Since probability all starts the same $\zeta_0$ is $0.5*(1-.2)+(1-0.5) = (0.5)0.8 + 0.5 = 0.9$ raised to their degree which is $= [2,3,2,3,4]$ giving $\zeta_0 = [0.81  , 0.729 , 0.81  , 0.729 , 0.6561]$.\\
Then the infection probability $1-p_{t+1}[i] = \sigma p_t[i] + (1-p_t[i])\zeta_t[i]$ can be rewritten as $p_{t+1}[i] = 1-( \sigma p_t[i] + (1-p_t[i])\zeta_t[i])$ which gives
$p_1= [0.545  , 0.5855 , 0.545  , 0.5855 , 0.62195]$. \\
\subsection{If we have a vaccination to immune one node which is best. Intuitively explain the answer of (1) with your own words}
If we don't vaccinate any node the infection probability is 0.5766. If we vaccinate 0 the infection probability is 0.5652 , for 1 it is 0.5439, for 2 it is 0.5652, for 3 it is 0.5439, and for 4 it is 0.5225. \\
As a result, our best node to vaccinate is node 4. Intuitively this makes sense as 4 is the node that has the highest degree in our graph. By removing the node that is most connected in the network we essentially remove one 0.5 from our non infection probability which ensures that it stays $\frac{1}{0.9}$ bigger for all. This follows much of what we studied in class where the highest eigen value node is the optimal vaccination as long as the high degree nodes are separated. In the case of this problem since we have few nodes and are only vaccinating one it is not a concern.
\section{Network Alignment}
\subsection{What is the time complexity (in big O notation) for the computation}
First off, the kronecker product of $A_1$ and $A_2$ takes $O(n^2m^2)$ to run as each element must be multiplied. Next, as the power method has linear convergence we must run this algorithm K times. Since this runs just like random walk algorithm but with two sets of nodes we have $N$ which represents the nodes in each of our matrixes of $nXm$. Thus the complexity is $O(K*2N) = O(KN)$ where N in number of nodes in each adjacency matrix.
\subsection{Explain with your own words that why this component can capture the topology consistency}
Topology consistency means the same node has a consistent connectivity structure across different networks. This is captured by the equation for Final because it is trying to minimize the the value that $J$ has with $S$. Since the main term in the equation is $\frac{S(x,a)}{\sqrt{f(x,a)}}-\frac{S(y,b)}{\sqrt{f(y,b)}}$ The goal of this equation is to minimize the difference between $S(x,a)$ and $S(y,B)$ an alignment that minimizes this has connected nodes that share network structure and attributes. Since this equation not only places value on the attributes but what they are connected to, it thus captures and maximizes an alignment that preserves topology consistency.
\subsubsection{what is the approximated closed-form solution of s?  What is the time complexity of the approximated solution ?}
First, we include the approximations in the closed form formula $s = (1-\alpha(I-\alpha D_n^{-\frac{1}{2}} N(U_1 \Lambda_1 U_1^T \circleland U_2 \Lambda_2 U_2^T)N D_N^{-\frac{1}{2}})^{-1} h$ by using the Sherman Morrison Lemma. We then move things around to simply to $s = (1-\alpha(I-\alpha D_n^{-\frac{1}{2}} N(U_1 \circleland U_2 *\Lambda_1 \circleland \Lambda_2 *U_1^T \circleland U_2^T)N D_N^{-\frac{1}{2}})^{-1} h$ which is further improved to $s = (1-\alpha(I+\alpha D_n^{-\frac{1}{2}} N(U_1 \circleland U_2 *\Lambda_1 \circleland \Lambda_2 *U_1^T \circleland U_2^T)^{-1} N D_N^{-\frac{1}{2}}) h$. This is finalized with $s = (1-\alpha(I-\alpha D_n^{-\frac{1}{2}} N*U_1 \circleland U_2((\Lambda_1 \circleland \Lambda_2)^-1 *\alpha U_1^T \circleland U_2^T N D_N^-1 N U_1 \circleland U_2 )^{-1} *U_1^T \circleland U_2^T *N D_N^{-\frac{1}{2}}) h$.  \\
The complexity of the approximate solution from one is $O(n^2r^4)$ but since we do not have to calculate the approximations our solution is $O(N^2)$ as only random walk needs to be executed.
\end{document}


\documentclass[handout]{beamer}
\usepackage{textpos}
\usepackage{listings}
\usepackage{hyperref}

\usepackage{caption}
\captionsetup[table]{labelformat=empty}

\usepackage{xcolor}
\definecolor{mygreen}{rgb}{0,0.6,0}
\definecolor{mygray}{rgb}{0.5,0.5,0.5}

\usepackage{graphicx}% http://ctan.org/pkg/graphicx
\usepackage{booktabs}

\lstset{language=C++,
           basicstyle=\ttfamily\scriptsize,
           keywordstyle=\color{blue}\ttfamily,
           stringstyle=\color{red}\ttfamily,
           commentstyle=\color{mygreen}\ttfamily,
          breaklines=true,
          captionpos=b,
          numbers=left,
          numbersep=5pt,
          numberstyle=\tiny\color{mygray},
          rulecolor=\color{black},
          xleftmargin=\parindent,
          frame=single,
          backgroundcolor=\color{white}
}

%\hypersetup{%
%colorlinks=true,% hyperlinks will be black
%linkcolor=blue
%}

% \usepackage{beamerthemesplit} // Activate for custom appearance

\setbeamercolor{normal text}{fg=black,bg=white}
\definecolor{beamer@blendedblue}{rgb}{0,0,0}
\setbeamercolor{structure}{fg=beamer@blendedblue}

\title{CUDA Fundementals}
\author{
	\includegraphics[width=3cm]{../media/logo/NVLogo_2D.eps}
	\vspace{0.75cm}
	\\}
\date{\today}

\begin{document}

\frame{\titlepage}

%\section[Outline]{}
\begin{frame}{Outline}
\tableofcontents
\end{frame}

\addtobeamertemplate{frametitle}{}{%
\begin{textblock*}{200mm}(.75\textwidth,-0.35cm)
\includegraphics[width=3cm]{../media/logo/NVLogo_2D_H.eps}
\end{textblock*}}

\addtobeamertemplate{navigation symbols}{}{%
    \usebeamerfont{footline}%
    \usebeamercolor[fg]{footline}%
    \hspace{1em}%
    \insertframenumber/\inserttotalframenumber
}

\section{SIMD Programing Paradigm}
\begin{frame}{Parallel Algorithms in SIMD}
\begin{itemize}
\itemsep1em
\setbeamercovered{transparent}
	\item<1->Recall that GPU hardware is designed for maximizing thread execution throughput.
	\item<1->And, this throughput-oriented design is effective since all the GPU threads are working a single CUDA kernel in parallel.
	\item<1->Each GPU thread execute the same kernel.  Once each individual thread has finished sequentially executing the kernel code, the kernel task as a whole is complete.
	\item<1->That is, each thread executes the same kernel but on it's own chunk or piece of the data.  This style of programing is called \emph{single instruction multiple data} (SIMD) pronounced ``sim-dee''
\end{itemize}
\end{frame}

\begin{frame}{Parallel Algorithms in SIMD}
\begin{itemize}
\setbeamercovered{transparent}
	\item<1->In SIMD, each thread executes the same kernel but only on it's own small piece of the data.
	\item<1->Each thread must have some kind of index information to provide context for which piece of the data to work on.
	\item<1->Threads in SIMD use there own index info as a key to determine which data to access.
	\item<1->This means that in massively parallel SIMD algorithms, thread organization plays a very important role.
	\item<1->Kernel thread organization will typically closely mirror the data dimensions so that thread index is aligned with data index.
\end{itemize}
\end{frame}

\section{Vector Addition Example}
\begin{frame}{Vector Addition Definition}
\begin{itemize}
\itemsep1em
	\item<1>Suppose we have two vectors of type double with matching dimensions: $\textbf{a}$ and $\textbf{b}$ both of size $1\times N$
	\item<1>Our goal is to create a resultant vector $\textbf{c}$ of doubles having size $1\times N$ where $\textbf{c}[i] = \textbf{a}[i]+\textbf{b}[i]$
	\item<1>Using a traditional sequential programing approach a simple {\fontfamily{qcr}\selectfont for}-loop would solve this nicely:
\end{itemize}

\end{frame}

\begin{frame}[fragile]{Traditional Vector Addition in C}
\lstinputlisting[language=C++,caption={Vanilla vector addition in C}]{../src/vecAdd.c}
\end{frame}

\begin{frame}{Embarrassingly Parallel Problems}
\begin{itemize}
\itemsep1em
	\item<1->Vector addition, as defined, makes a good initial parallel programing exercise since each element of the output vector {\fontfamily{qcr}\selectfont \textbf{c}} can be computed independent of all other elements.
	\item<1->Problems (or algorithms) which exhibit perfect data parallelism, such as vector addition, are called \emph{embarrassingly parallel}.
	\item<1->Launching a CUDA kernel typically generates a large number of threads to exploit data parallelism.
\end{itemize}
\end{frame}

\begin{frame}[fragile]{SIMD Vector Addition with CUDA}
\begin{lstlisting}[caption={A SIMD example for vector addition using CUDA}]
#define N 256

// Kernel definition (i.e. device code executed on the GPU)
__global__ void VecAdd(float* a, float* b, float* c){

    // get local thread context
    int i = threadIdx.x;

    // perform operation on data
    c[i] = a[i] + b[i];
}
// host code executed by the CPU
int main(){
    ...
    // launch configuration
    dim3 gridSize (1,1,1), blockSize(N,1,1);

    // kernel launch
    VecAdd<<<gridSize, blockSize>>>(a, b, c);
    ...
}
\end{lstlisting}
\end{frame}

\begin{frame}{SIMD Vector Addition Visualization}
\begin{figure}
\begin{center}
\includegraphics[width=5.5cm]{../media/vector_add_cuda.pdf}
\caption{Thread visualization of vector addition CUDA kernel.  Here thread0 calculates {\fontfamily{qcr}\selectfont \textbf{c}[0]}, thread1 calculates {\fontfamily{qcr}\selectfont \textbf{c}[1]}, and so on.  Each thread performs the same set of operations, but on its own piece of data.}
\end{center}
\end{figure}
\end{frame}

\begin{frame}{Kernel Launch Thread Configurations}
\centering

\begin{table}
\resizebox{\linewidth}{!}{
\begin{tabular}{ | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c |}
\hline
  0 & 1 & 2 & 3 & 4 & 5  & 6 & 7 & 8 & 9 & 10 &11 & 12 & 13 & 14 & 15 & 16 & 17 & 18 & 19 & 20 & 21  & 22 & 23 & 24 & 25 & 26 & 27 & 28 & 29 & 30 & 31  \\
\hline
\end{tabular}
}
\caption{{\fontfamily{qcr}\selectfont dim3 blockSize(32,1,1)}}
\end{table}

\begin{table}
\resizebox{\linewidth}{!}{
\begin{tabular}{ | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c | c |}
\hline
  0 & 1 & 2 & 3 & 4 & 5  & 6 & 7 & 8 & 9 & 10 &11 & 12 & 13 & 14 & 15   \\
\hline
 16 & 17 & 18 & 19 & 20 & 21  & 22 & 23 & 24 & 25 & 26 & 27 & 28 & 29 & 30 & 31 \\
\hline
\end{tabular}
}
\caption{{\fontfamily{qcr}\selectfont dim3 blockSize(16,2,1)}}
\end{table}

\begin{table} `
\resizebox{!}{!}{
\begin{tabular}{ | c | c | c | c | c | c | c | c |}
\hline
  0 & 1 & 2 & 3 & 4 & 5  & 6 & 7  \\
\hline
  8 & 9 & 10 &11 & 12 & 13 & 14 & 15 \\
\hline
 16 & 17 & 18 & 19 & 20 & 21  & 22 & 23  \\
\hline
  24 & 25 & 26 & 27 & 28 & 29 & 30 & 31 \\
\hline
\end{tabular}
}
\caption{{\fontfamily{qcr}\selectfont dim3 blockSize(8,4,1)}}
\end{table}

\end{frame}

\begin{frame}{Kernel Thread Organization}
\centering
\begin{figure}
\label{fig:gridorg}
\includegraphics[width=5.5cm]{../media/grid_org.pdf}
\caption{dim3 gridSize(4,2,1) with dim3 blockSize(N,M,1).}
\end{figure}
\end{frame}

\begin{frame}{Kernel Thread Configurations}
\begin{itemize}
	\item<1->Grids can have up to 65535x65535 blocks on SM 1.x and 65535x65535x65535 blocks with SM 2.x hardware.
	\item<1->Thread-blocks are further divided into \href{http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html\#simt-architecture}{\color{blue}\emph{warps}} of 32 threads.
	\item<1->A warp is the fundamental unit of granularity on the SM.
	\item<1->Therefore, thread-block size should be specified modulo 32.
	\item<1->Special read-only registers give each \href{http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html\#thread-hierarchy}{\color{blue}thread context} via built-in variables \emph{threadIdx}, \emph{blockIdx}, \emph{gridDim} and \emph{blockDim}.
	\item<1->Taken together these variables can be used to compute which part of a problem the thread will operate on.
	\item<1->Note that there is no performance benefit with 2D or 3D blocks and grids, but they often match better to the data organization.
\end{itemize}
\end{frame}

%\begin{frame}{A Note on CUDA Header Files}
%\begin{itemize}
%\itemsep1em
%	\item<1->{There are three key header files when programing in CUDA:}
%	\break
%	\begin{itemize}
%	\itemsep1em
%		\item<1->{\fontfamily{qcr}\selectfont cuda.h} defining types and host functions for the CUDA \emph{driver} API.
%		\item<1->{\fontfamily{qcr}\selectfont cuda\_runtime\_api.h} which defines types and host functions and types for the CUDA \emph{runtime} API.
%		\item<1->{\fontfamily{qcr}\selectfont cuda\_runtime.h} contains a superset of definitions including everything from {\fontfamily{qcr}\selectfont cuda\_runtime\_api.h}, as well as built-in type definitions, function overlays for the CUDA language extensions, and device intrinsic functions.
%	\end{itemize}
%	\item<1->Notice that when compiling with \textbf{{\fontfamily{qcr}\selectfont nvcc}} the appropriate CUDA headers are included automatically.
%\end{itemize}
%
%\end{frame}

\begin{frame}{Understanding CUDA Kernel Signatures}
\begin{itemize}
\itemsep1em
	\item<1->A kernel function \emph{must} have a {\fontfamily{qcr}\selectfont void} return type.
	\item<1-> There are three kernel function type qualifiers which signal \textbf{{\fontfamily{qcr}\selectfont nvcc}} for how the function should be compiled:
	\begin{itemize}
	\itemsep1em
		\item<1->{\fontfamily{qcr}\selectfont \_\_global\_\_} says that the function will be \emph{executed} on the device (i.e. it's a kernel) but callable (launched) from the host.
		\item<1->{\fontfamily{qcr}\selectfont \_\_device\_\_} also says that the function will be executed on the device but is \emph{only callable from the device}.  Think kernel helper functions here.
		\item<1->Finally, {\fontfamily{qcr}\selectfont \_\_host\_\_} tells \textbf{{\fontfamily{qcr}\selectfont nvcc}} that this function will only be executable and callable from the host (i.e. this is just a regular C function!).
	\end{itemize}
\end{itemize}
\end{frame}

\begin{frame}{Kernel Restrictions}
There are a few kernel coding restrictions to keep in mind:
\break
\begin{itemize}
\itemsep1em
	\item<1->Kernel functions can \emph{only} access device memory\footnote{Exception when using \emph{mapped pinned memory} (more later)\hfill\break}
	\item<1->Again, kernel functions must have {\fontfamily{qcr}\selectfont void} return type
	\item<1->No support for variable number of arguments
	\item<1->No support for static variables
	\item<1->No support for function pointers
	\item<1->Exhibit asynchronous behavior
\end{itemize}
\end{frame}

\begin{frame}[fragile]{Kernel Launch Runtime Syntax}
When using the CUDA runtime API, the generic kernel launch triple-angle-bracket syntax is:
\hfill\break
\begin{lstlisting}[caption={Kernel launch syntax using runtime API}]
KERNEL<<<gridSize, blockSize, sharedMem, stream>>>(args)
\end{lstlisting}
\begin{itemize}
	\item<1->{\fontfamily{qcr}\selectfont KERNEL} specifies the kernel function to launch
	\item<1->{\fontfamily{qcr}\selectfont gridSize} specified the number of thread-blocks in the form of a {\fontfamily{qcr}\selectfont dim3} structure
	\item<1->{\fontfamily{qcr}\selectfont blockSize} specifies the size of each thread-block in three dimensions as a {\fontfamily{qcr}\selectfont dim3}.
	\item<1->{\fontfamily{qcr}\selectfont sharedMem} specifies the amound of additional shared memory to reserve for each thread-block. Default: 0
	\item<1->{\fontfamily{qcr}\selectfont stream} specifies the logical execution queue for the kernel
\end{itemize}
\end{frame}

\begin{frame}[fragile]{{\fontfamily{qcr}\selectfont dim3} Structure}
The {\fontfamily{qcr}\selectfont dim3} structure is used to specify the grid and thread-block sizes and has 3 members ({\fontfamily{qcr}\selectfont x}, {\fontfamily{qcr}\selectfont y}, and {\fontfamily{qcr}\selectfont z}) with a default constructor which initializes {\fontfamily{qcr}\selectfont y} and {\fontfamily{qcr}\selectfont z} to 1.
\begin{lstlisting}[caption={{\fontfamily{qcr}\selectfont dim3} structure definition defined in {\fontfamily{qcr}\selectfont vector\_types.h}}]
struct __device_builtin__ dim3
{
    unsigned int x, y, z;
#if defined(__cplusplus)
    __host__ __device__ dim3(
    unsigned int vx = 1,
    unsigned int vy = 1,
    unsigned int vz = 1) : x(vx), y(vy), z(vz) {}
    __host__ __device__ dim3(uint3 v):x(v.x),y(v.y),z(v.z){}
    __host__ __device__ operator uint3(void){
        uint3 t;  t.x = x;   t.y = y;  t.z = z;  return t;
    }
#endif
};
\end{lstlisting}
\end{frame}

\begin{frame}[fragile]{Kernel Launch with Driver API}
Kernels can be launched from the CUDA driver API using the {\fontfamily{qcr}\selectfont cuLaunchKernel()} call
\begin{lstlisting}[caption={{\fontfamily{qcr}\selectfont cuLaunchKernel()} in the CUDA driver API}]
CUresult CUDAAPI cuLaunchKernel(
    CUfunction f,
    unsigned int gridDimX,
    unsigned int gridDimY,
    unsigned int gridDimZ,
    unsigned int blockDimX,
    unsigned int blockDimY,
    unsigned int blockDimZ,
    unsigned int sharedMemBytes,
    CUstream hStream,
    void **kernelParams,
    void **extra
);
\end{lstlisting}
\end{frame}

\begin{frame}[fragile]{Kernel Launch with Driver API}
However, note that kernel launches using the runtime API get translated into even lower driver API calls using {\fontfamily{qcr}\selectfont cudaLaunch()}, {\fontfamily{qcr}\selectfont cudaConfigureCall()}, {\fontfamily{qcr}\selectfont cudaSetupArgument()}.  For example, the {\fontfamily{qcr}\selectfont VecAdd} kernel launch
\begin{lstlisting}
VecAdd<<<1, N>>>(a, b, c);
\end{lstlisting}
generates ({\fontfamily{qcr}\selectfont vecAdd.cudafe1.stub.c}, using {\fontfamily{qcr}\selectfont nvcc} {\fontfamily{qcr}\selectfont --keep})
\begin{lstlisting}[]
void __device_stub__Z6vecAddPfS_S_(
    float *__par0,
    float *__par1,
    float *__par2)
{
        __cudaSetupArgSimple(__par0,  0UL);
        __cudaSetupArgSimple(__par1,  8UL);
        __cudaSetupArgSimple(__par2, 16UL);
        __cudaLaunch(
            ((char *)((void ( *)(float *, float *, float *))vecAdd))
        );
}
\end{lstlisting}
\end{frame}

\begin{frame}{{\fontfamily{qcr}\selectfont C++} Class Restrictions}
All {\fontfamily{qcr}\selectfont C++} classes participating in a kernel launch must be ``plain old data'' (POD) with the following characteristics:
\hfill\break
\begin{itemize}
\itemsep1em
	\item<1->No user-declared constructors or destructors
	\item<1->No user-defined copy assignment operator
	\item<1->No non-static data members that are not themselves POD
	\item<1->No private or protected non-static data
	\item<1->No virtual functions or base classes
\end{itemize}
\hfill \break
Classes that violate these rules my be used in CUDA, or even in CUDA kernels; they simply cannot be used for kernel launch.
\end{frame}

\begin{frame}{Kernel Execution}
\begin{itemize}
	\item<1->Once a kernel is launched, it runs as a \emph{grid} of \emph{blocks} of \emph{threads}.
	\item<1->Each thread block is assigned to a streaming multiprocessor (SM) on the device.
	\item<1->Not all thread blocks run concurrently, necessarily.
	\item<1->The CUDA programming model makes \emph{no guarantees whatsoever as to the order of execution} or whether certain blocks or threads can run concurrently.
	\item<1->Again, CUDA developers can never assume that all the threads in a kernel launch are executing concurrently.
	\item<1->It is easy to launch a kernel with more threads than the device can hold and therefore some will not start executing until others have finished.
\end{itemize}
\end{frame}

%\begin{frame}{Kernel Execution Continued}
%\begin{itemize}
%	\item<1->Thread-blocks are further divided into \emph{warps} of 32 threads.
%	\item<1->Each thread in a warp is called a \emph{lane}.
%	\item<1->A warp is the fundamental unit of granularity on the SM.
%	\item<1->Grids can have up to 65535x65535 blocks on SM 1.x and 65535x65535x65535 blocks with SM 2.x hardware.
%	\item<1->Thread-blocks do not move from SM to SM.  A thread-block resides on an SM for the lifetime of the kernel.
%	\item<1->There can be many active warps from different blocks at the same time on a single SM.
%	\item<1->Warp schedulers dispatch instructions as requested resources become available.
%	\item<1->All 32 threads execute the same instruction, each thread using its own set of registers to perform the requested operation.
%\end{itemize}
%\end{frame}

\begin{frame}{Asynchronous Error Handling}
\begin{itemize}
	\item<1->Kernel launches are \emph{asynchronous}.
	\item<1->As soon as a kernel is submitted to the hardware, it begins executing in parallel with the CPU.
	\item<1->On most platforms, the kernel will start executing on the GPU microseconds after the CPU has finished processing the launch command\footnote{With the Windows Display Driver Model (WDDM), it may take longer to launch a kernel because the driver must perform an OS \href{https://msdn.microsoft.com/en-us/library/windows/hardware/ff565689(v=vs.85).aspx}{\color{blue}kernel thunk} in order to submit the launch to the hardware.\hfill\break}.
	\item<1->If the kernel encounters an error (e.g. invalid memory access, etc), the error is reported to the driver (and the application) some time after the kernel launch.
	\item<1->The best way to check for such errors is to synchronize with the GPU using {\fontfamily{qcr}\selectfont cudaDeviceSynchronize()} or {\fontfamily{qcr}\selectfont cuCtxSynchronize()}
\end{itemize}
\end{frame}

\begin{frame}[fragile]{Error Handling Continued}
\begin{itemize}
	\item<1->Most CUDA runtime API calls return type {\fontfamily{qcr}\selectfont cudaError\_t}
	\item<1->It is standard practice to define error-handling macro to wrap API calls.  For example
\end{itemize}
\begin{lstlisting}[caption={Error handling macro for CUDA}]
#include <cuda_runtime.h>
#define CHECK(call)
{
    const cudaError_t error = call;
    if (error != cudaSuccess){
        printf("Error: %s:%d",__FILE__,__LINE__);
        printf("code:%d, reason: %s\n", error, cudaGetErrorString(error));
        exit(1);
    }
}
\end{lstlisting}
\end{frame}

\begin{frame}[fragile]{Error Handling Continued}
\begin{itemize}
\itemsep1em
	\item<1->The error-handling macro can be use around {\fontfamily{qcr}\selectfont cudaDeviceSynchronize()} after a kernel invocation to check for kernel errors
	\item<1->If an error in the kernel execution has occurred, the error string "unspecified launch failure" is returned.
\end{itemize}
\begin{lstlisting}[caption={Catching kernel failures with CHECK macro}]
    kernel_function<<<gridsz,blocksz>>>(args);
    CHECK(cudaDeviceSynchronize());
\end{lstlisting}
\begin{itemize}
	\item<1->See also: {\fontfamily{qcr}\selectfont cudaGetLastError()}
\end{itemize}
\end{frame}

\begin{frame}{Kernel Timeouts}
\begin{itemize}
	\item<1->Because the GPU is not able to context-switch in the midst of kernel execution, a long-running CUDA kernel may negatively impact the interactivity of a system that uses the GPU to interact with the user (i.e. graphics output).
	\item<1->Many CUDA capable systems implement a ``timeout'' that resets the GPU if it runs too long without context switching.
	\item<1->On \href{https://en.wikipedia.org/wiki/Windows_Display_Driver_Model}{\color{blue}Windows Display Driver Model} (WDDM), the timeout is enforced by the operating system.  Microsoft calls this ``Timeout Detection and Recovery'' (TDR).  TDR can be safely disabled using the \href{http://http.developer.nvidia.com/ParallelNsight/2.1/Documentation/UserGuide/HTML/Content/Tesla_Compute_Cluster.htm}{\color{blue}Tesla Compute Cluster} (TCC) driver, though the TCC is not available for all hardware.
	\item<1->On Linux, the NVIDIA driver enforces a default timeout of 2 seconds.  But keep in mind that no timeout is enforced on GPUs that are not being used for display.
\end{itemize}
\end{frame}

\begin{frame}{Host/Device Cooperation: Events}
\begin{itemize}
    %\item<1->While a majority of the commands issued by the driver to the GPU device are either kernel launches or memory copies, there exists an important subclass of commands for helping the CUDA driver track progress of the GPU in processing the command buffer.
    \item<1->It can not be known a-priori how long a kernel will take to complete execution.  Therefore, it is the device \emph{itself} that must report execution progress back to the host.
    \item<1->The driver keeps a monotonically increasing command counter called the ``progress-value''.
    \item<1->Major commands issued by the driver are followed by another command for the device to write the new progress-value to a shared portion of host memory called the ``sync-location''.
    \item<1->The driver can read the sync-location at any time to determine which commands have been completed by the GPU.
    \item<1->For example, when calling {\fontfamily{qcr}\selectfont cudaThreadSynchronize()} the driver simply waits for the sync-location value to be greater than or equal to the current progress-value.
    \item<1->CUDA \emph{events} expose these interactions more explicitly.
\end{itemize}
\end{frame}

\begin{frame}{Host/Device Cooperation: Events}
\begin{itemize}
	\item<1->CUDA runtime events are of type {\fontfamily{qcr}\selectfont cudaEvent\_t} and are initialized with {\fontfamily{qcr}\selectfont cudaEventCreate()}
	\item<1->The asynchronous command {\fontfamily{qcr}\selectfont cudaEventRecord()} pushes a command to the execution buffer to write a progress-value to the sync-location.  The progress-value to be written is tracked by the event.
	\item<1->The host can synchronize with a recorded event via the blocking call {\fontfamily{qcr}\selectfont cudaEventSynchronize()}
	\item<1->CUDA events are routinely used to profile execution performance of asynchronous commands on the GPU.  The elapsed time between recored events can be determined using {\fontfamily{qcr}\selectfont cudaEventElapsedTime()}.
	\item<1->Timing of CUDA events is based on 64-bit values provided by a high-resolution GPU-based clock and are less subject to system overhead such as page-faults etc.
\end{itemize}
\end{frame}

\begin{frame}[fragile]{Profiling Asynchronous Commands}
\begin{lstlisting}[caption={Pattern for timing kernels and other asynchronous API calls}]

    // initialize events
    cudaEventCreate(start); cudaEventCreate(stop);

    // record, launch, record (asynchronous)
    cudaEventRecord(start);
    kernel <<<gridSz, blockSz>>>(args);
    cudaEventRecord(stop);

    // wait for end event (synchronous)
    cudaSynchronizeEvent(stop);

    // figure out time between events
    cudaEventElapsedTime(&elapsedTime, start, stop);

    // clean up
    cudaEventDestroy(start); cudaEventDestroy(stop);
}
\end{lstlisting}
\end{frame}

\begin{frame}{Kernel Performance: Occupancy}
\begin{itemize}
	\item<1->When launching a kernel, each thread-block is assigned to a specific SM for the lifetime of the kernel.
	\item<1->These thread-blocks are decomposed into \emph{warps} of 32 threads.
	\item<1->The term \emph{occupancy} is defined as the number of warps per SM divided by maximum possible number of warps .
\end{itemize}
\begin{center}
$\text{occupancy} =\frac{\text{\# warps per SM}}{\text{Max warps per SM}}$
\end{center}

\begin{itemize}
	\item<1->The maximum number of warps per SM is a fixed constant that depends only on the compute capability of the device.
        \item<1->The numerator of the expression is a function of the following
        \begin{itemize}
        		\item<1->Compute capability (1.0, 1.1, 1.2, 1.3, 2.0, 2.1, 3.0, 3.5)
		\item<1->Threads per block
		\item<1->Registers per thread
		\item<1->Shared memory configuration
		\item<1->Shared memory per block
        \end{itemize}
\end{itemize}
\end{frame}

\begin{frame}{Blocks v.s. Warps}
It is important to keep in mind the central distinctions between blocks and warps
\hfill\break
\begin{itemize}
\itemsep1em
	\item<1->Thread-blocks are the unit of \emph{cooperation}
	\item<1->Warps are the unit of \emph{execution}
\end{itemize}
\hfill\break
Thread-blocks are a CUDA programing model abstraction where as warps are the fundamental quanta of the CUDA execution model.
\end{frame}

\begin{frame}{Kernel Performance: Occupancy}
Kernel sizing is an important consideration for CUDA performance.
\begin{itemize}
	\item<1->Resources on the SM (registers, shared memory) are finite.
	\item<1->These SM resources are allocated on a per-block basis.
	\item<1->Utilizing too many resources in kernel will reduce occupancy.
\end{itemize}
\hfill\break
Don't let all the terms confuse you.  As a simple analogy: there is a finite amount of space (seat belts?) in your car.  This space constraint will limit the number of kids you can driver to soccer practice.  The larger the kids the fewer of them you can take.
\hfill\break
\begin{itemize}
	\item<1->If the kernel execution requires many SM resources then thread-block size should, generally speaking, be smaller.
\end{itemize}
\end{frame}

\begin{frame}{Kernel Performance: Occupancy}
\begin{itemize}
\itemsep1em
	\item<1->Q: Do we really need to figure out the register count and KB of shared memory for every kernel launch (that doesn't sound like much fun)?
	\item<1->The short answer is that it is important to be mindful of kernel size and finite device resources.  The micro-architecture of the streaming multiprocessor is quite advanced and can typically do a great job at warp scheduling etc.
	\item<1->However, to truly maximize kernel performance, roll up those sleeves and get out the calculator.
	\item<1->Note the option {\fontfamily{qcr}\selectfont \textbf{nvcc} --ptxas options=v} provides details on number of registers used per thread for each kernel.
	\item<1->There is more info in the \href{http://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html\#occupancy}{\color{blue}docs} to facilitate occupancy analysis with details on the occupancy calculator spreadsheet included in the CUDA Toolkit.
\end{itemize}
\end{frame}

\begin{frame}{Kernel Performance: Occupancy API}
As of CUDA 6.5 there are several new runtime functions to aid in occupancy calculations and launch configuration.  There are new core occupancy calculator API calls: \begin{center}
{\fontfamily{qcr}\selectfont cudaOccupancyMaxPotentialBlockSize()}\break
{\fontfamily{qcr}\selectfont cudaOccupancyMaxActiveBlocksPerMultiprocessor()}
\end{center}
which can be use to produce an occupancy prediction based on the block size and shared memory usage of a kernel.
\newline\newline
Note that multiplying by the number of warps per block yields the number of concurrent warps per multiprocessor; further dividing concurrent warps by max warps per multiprocessor gives the occupancy as a percentage (see examples).
\end{frame}

\begin{frame}[fragile]{Occupancy API Example}
It is often the case that changes to kernel code (i.e. resource footprint) require modification of the associated launch configuration.  Using the occupancy API allows reasonable kernel launch configurations to be determined dynamically at runtime.
\begin{lstlisting}[caption={Use CUDA runtime API to determine kernel lunch configuration}]
  int arraySize = 1024*1024;
  int blockSize, minGridSize, gridSize;

  // make call to runtime API for thread block size
  cudaOccupancyMaxPotentialBlockSize(
    &minGridSize, &blockSize, MyKernel, 0, 0);

  // round up gridSize according to array size
  gridSize = (arraySize + blockSize - 1) / blockSize;

  // launch kernel with appropriate configuration
  MyKernel<<<gridSize,blockSize>>>(array, arraySize);
\end{lstlisting}
\end{frame}

\begin{frame}[fragile]{Occupancy API Example}
Having computed the appropriate blockSize and gridSize from the previous example, we can further calculate the theoretical kernel occupancy for given the launch configuration.
\begin{lstlisting}[caption={Use CUDA runtime API to calculate kernel occupancy}]
  int maxActiveBlocks;

  // compute max number active thread-blocks for this kernel
  cudaOccupancyMaxActiveBlocksPerMultiprocessor(
    &maxActiveBlocks, MyKernel, blockSize, 0);

  // query device properties
  int device;
  cudaDeviceProp props;
  cudaGetDevice(&device);
  cudaGetDeviceProperties(&props, device);

  // calculate theoretical occupancy
  float occupancy =
         (maxActiveBlocks * blockSize / props.warpSize) /
  (float)(props.maxThreadsPerMultiProcessor/props.warpSize);
\end{lstlisting}
\end{frame}

%\begin{frame}{Thread Divergence}
%\begin{itemize}
%	\item<1->Instructions are issued per warp.
%	\item<1->All threads in a warp execute same instruction.
%	\item<1->If an operand is not ready then the warp will stall.
%\end{itemize}
%\end{frame}

\end{document}


\chapter{The second chapter}
\label{cha:a_chapter}

This is the first paragraph of the Softcover Markdown template produced with the \softcover{} command-line interface. It shows how to write a document in Markdown, a lightweight markup language, augmented with the \href{http://kramdown.gettalong.org/}{kramdown} converter and some custom extensions, including support for embedded \PolyTeX{}, a subset of the powerful \LaTeX{} typesetting system.\footnote{Pronunciations of {}``LaTeX{}'' differ, but \emph{lay}-tech is the one I prefer.} For more information, see \href{http://manual.softcover.io/book}{\emph{The Softcover Book}}. To learn how to easily publish (and optionally sell) documents produced with Softcover, visit \href{http://softcover.io/}{Softcover.io}.

This is the \emph{second} paragraph, showing how to emphasize text.\footnote{This is a footnote. It is numbered automatically.} You can also make text \textbf{bold} or \emph{emphasize a second way}. Via embedded \PolyTeX, Softcover also supports colored text, such as \coloredtext{red}{red}, \coloredtext{CornflowerBlue}{cornflower blue}, and \coloredtexthtml{E8AB3A}{arbitrary HTML colors}.

\section{A section}

\label{sec:a_section}

This is a section. You can refer to it using the \LaTeX{} cross-reference syntax, like so: Section~\ref{sec:a_section}.

\subsection{Source code}

This is a subsection.

You can typeset code samples and other verbatim text using four spaces of indentation:

\begin{verbatim}
def hello
  puts "hello, world"
end
\end{verbatim}

Softcover also comes with full support for syntax-highlighted source code using kramdown{}'s default syntax, which combines the language name with indentation:

%= lang:ruby
\begin{code}
def hello
  puts "hello, world"
end
\end{code}

Softcover{}'s Markdown mode also extends kramdown to support so-called {}``code fencing{}'' from GitHub-flavored Markdown:

%= lang:ruby
\begin{code}
def hello
  puts "hello, world!"
end
\end{code}

The last of these can be combined with \PolyTeX{}'s \kode{codelisting} environment to make code listings with linked cross-references (Listing~\ref{code:hello}).

\begin{codelisting}
\codecaption{Hello, world.}
\label{code:hello}
%= lang:ruby
\begin{code}
def hello
  puts "hello, world!"
end
\end{code}
\end{codelisting}

\subsection{Mathematics}

Softcover{}'s Markdown mode supports mathematical typesetting using \LaTeX{} syntax, including inline math, such as \( \phi^2 - \phi - 1 = 0, \) and centered math, such as
\[ \phi = \frac{1+\sqrt{5}}{2}. \]
It also supports centered equations with linked cross-reference via embedded \PolyTeX{} (Eq.~\eqref{eq:phi}).

\begin{equation}
\label{eq:phi}
\phi = \frac{1+\sqrt{5}}{2}
\end{equation}

Softcover also supports an alternate math syntax, such as \(\phi^2 - \phi - 1 = 0\), and centered math, such as

\[\phi = \frac{1+\sqrt{5}}{2}.\]

The \LaTeX{} syntax is strongly preferred, but the alternate syntax is included for maximum compatibility with other systems.

\subsection{Exercises}

\beforeFirstExercises{ch2}

\begin{exercises}{ch2}

	\begin{exercise}
		Do the following vectors represent probability distributions?
		\begin{exparts*}
			\partsitem	$\left(\tfrac{1}{2}, \tfrac{1}{2}, \tfrac{1}{2} \right)^T$
			\partsitem	$\left(\tfrac{1}{4}, \tfrac{1}{4}, \tfrac{1}{4}, \tfrac{1}{4} \right)^T$
			\partsitem	$\left(0.3, 0.3, -0.1, 0.5  \right)^T$
		\end{exparts*}

		\begin{eanswer}\begin{ansparts*}
					\partsitem No; weights don't add to one.
					\partsitem Yes.
					\partsitem No; contains a negative number.
					\end{ansparts*}\end{eanswer}

		\begin{esolution}
			In each case we check if each probability weight is positive and
			the total probability sums to one.
		\end{esolution}
	\end{exercise}

\end{exercises}

\afterLastExercises{ch2}

\section{Images and tables}

This is the second section.

Softcover supports the inclusion of images, like this:

\image{figures/math/conic_sections_four-shapes.png}

Using \LaTeX{} labels, you can also include a caption (as in Figure~\ref{fig:captioned_image}) or just a figure number (as in Figure~\ref{fig:figure_number}).

\begin{figure}[H]
\begin{center}
\image{figures/math/conic_sections_four-shapes.png}
\end{center}
\caption{The four conic sections.\label{fig:captioned_image}}

\end{figure}

\begin{figure}[H]
\begin{center}
\image{figures/math/conic_sections_four-shapes.png}
\end{center}
\caption{\label{fig:figure_number}}

\end{figure}

\subsection{Tables}

Softcover supports raw tables via a simple table syntax:

\begin{longtable}{|l|l|l|l|}
\hline
\textbf{HTTP request} & \textbf{URL} & \textbf{Action} & \textbf{Purpose}\\
\kode{GET} & /users & \kode{index} & page to list all users\\
\kode{GET} & /users/1 & \kode{show} & page to show user with id \kode{1}\\
\kode{GET} & /users/new & \kode{new} & page to make a new user\\
\kode{POST} & /users & \kode{create} & create a new user\\
\kode{GET} & /users/1/edit & \kode{edit} & page to edit user with id \kode{1}\\
\kode{PATCH} & /users/1 & \kode{update} & update user with id \kode{1}\\
\kode{DELETE} & /users/1 & \kode{destroy} & delete user with id \kode{1}\\
\hline
\end{longtable}

See \href{http://manual.softcover.io/book/softcover_markdown#sec-embedded_tabular_and_tables}{\emph{The Softcover Book}} to learn how to make more complicated tables.

\section{Command-line interface}

Softcover comes with a command-line interface called \kode{softcover}. To get more information, just run \kode{softcover help}:

%= lang:console
\begin{code}
$ softcover help
Commands:
  softcover build, build:all           # Build all formats
  softcover build:epub                 # Build EPUB
  softcover build:html                 # Build HTML
  softcover build:mobi                 # Build MOBI
  softcover build:pdf                  # Build PDF
  softcover build:preview              # Build book preview in all formats
  .
  .
  .
\end{code}

\noindent You can run \kode{softcover help \textless{}command\textgreater{}} to get additional help on a given command:

%= lang:console
\begin{code}
$ softcover help build
Usage:
  softcover build, build:all

Options:
  -q, [--quiet]   # Quiet output
  -s, [--silent]  # Silent output

Build all formats
\end{code}

\section{Miscellanea}

This is the end of the template---apart from two mostly empty chapters. In fact, let’s include the last chapter in its entirety, just to see how mostly empty it is:
%= <<(chapters/yet_another_chapter.tex, lang: text)

Visit \href{http://manual.softcover.io}{\emph{The Softcover Book}} to learn more about what Softcover can do.

\section{Probability problems}

\begin{problems}{ch2}

	\begin{problem}		\label{problem:geometric_distr_biased_coin_until_heads}
		You have a biased coin which lands on \texttt{heads} with probability $p$,
		and consequently lands on \texttt{tails} with probability $(1-p)$.
		Suppose you want to flip the coin until you get \texttt{heads}.
		Define the random variable $N$ as the number of tosses required until the first \texttt{heads} outcome.
		What is the probability mass function $P_N(n)$ for success on the $n$\textsuperscript{th} toss?
		Confirm that the formula is a valid probability distribution by showing $\sum_{n=1}^\infty P_N(n) = 1$.

		\begin{hint}
			Find the probabilities for cases $n=1,2,3,\ldots$ and look for a pattern.
		\end{hint}

		\begin{answer}$P_N(n) = (1-p)^{n-1}p$\end{answer}

		\begin{solution}
			The biased coin flip is modelled by a random variable $Y$,
			and different coin flips correspond to random variables $Y_1$, $Y_2$, $Y_3$, \ldots which are independent copies of $Y$.
			The probability of getting \texttt{heads} on the first flip is $P_N(1)=\textrm{Pr}\!\left( \{ Y_1=\texttt{heads} \} \right)\! =p$.
			The probability of getting \texttt{heads} on the second flip corresponds
			to the event $\{Y_1=\texttt{tails}\} \; \texttt{AND} \; \{Y_2=\texttt{heads} \}$.
			We assumed the coin flips are independent so % the probability of this event is the product
			$P_N(2)=(1-p)p$.
			Similarly $P_N(3) = (1-p)^2p$.
			The general formula is $P_N(n) = (1-p)^{n-1}p$.
		\end{solution}
	\end{problem}

\end{problems}


\section{Related readings}
\label{sec:related-readings}

\begin{itemize}
	\item Closed queueing networks: \cite{lazowska1984quantitative,menasce2001capacity}

	\item Comparison Open/Closed queueing networks: \cite{bondi1986influence,schroeder2006open,tay2013analytical,whitt1984open}

	\item Stability conditions for queueing networks: \cite{bramson2008stability}

	\item Generalization of Little's Law: \cite{el2012sample,miyazawa1994rate}.

	\item Markov Chains:
	\cite{ross2014introduction}.

	\item Generalization of PASTA:
	\cite{wolff1982poisson}.

	\item insensitivity property:
	\cite{tijms2003first}.

	\item reverse-chains and time-reversibility:
	\cite{bertsekas1992data,ross1996stochastic}.

	\item Burke theorem:
	\cite{burke1956output}

	\item Jackson Networks:
	\cite{jackson1963jobshop}.

	\item normalizing constants for closed systems:
	\cite{harrison1985technical,choudhury1995calculating}.

	\item Mean Value Analysis:
	\cite{reiser1980mean,bolch2006queueing,menasce2001capacity,allen2014probability}.

	\item Scheduling Policies
	\cite{wierman2007scheduling}.

	\item Processor-Sharing:
	\cite{aalto2007beyond}.

	\item Foreground-Background:
	\cite{nuijens2004foreground,rai2003analysis}.
\end{itemize}


%

\documentclass[twoside]{article}
\setlength{\oddsidemargin}{0.25 in}
\setlength{\evensidemargin}{-0.25 in}
\setlength{\topmargin}{-0.6 in}
\setlength{\textwidth}{6.5 in}
\setlength{\textheight}{8.5 in}
\setlength{\headsep}{0.75 in}
\setlength{\parindent}{0 in}
\setlength{\parskip}{0.1 in}

%
% ADD PACKAGES here:
%

\usepackage{amsmath,amsfonts,graphicx}

%
\newcounter{lecnum}
\renewcommand{\thepage}{\thelecnum-\arabic{page}}
\renewcommand{\thesection}{\thelecnum.\arabic{section}}
\renewcommand{\theequation}{\thelecnum.\arabic{equation}}
\renewcommand{\thefigure}{\thelecnum.\arabic{figure}}
\renewcommand{\thetable}{\thelecnum.\arabic{table}}

%
% The following macro is used to generate the header.
%
\newcommand{\lecture}[4]{
   \pagestyle{myheadings}
   \thispagestyle{plain}
   \newpage
   \setcounter{lecnum}{#1}
   \setcounter{page}{1}
   \noindent
   \begin{center}
   \framebox{
      \vbox{\vspace{2mm}
    \hbox to 6.28in { {\bf EE302 - Feedback Systems
	\hfill Spring 2019} }
       \vspace{4mm}
       \hbox to 6.28in { {\Large \hfill Lecture #1 \hfill} }
       \vspace{2mm}
       \hbox to 6.28in { {\it Lecturer: #2 \hfill } }
      \vspace{2mm}}
   }
   \end{center}
   \markboth{Lecture #1}{Lecture #1}

   \vspace*{4mm}
}

%Use this command for a figure; it puts a figure in wherever you want it.
%usage: \fig{NUMBER}{SPACE-IN-INCHES}{CAPTION}
\newcommand{\fig}[3]{
			\vspace{#2}
			\begin{center}
			Figure \thelecnum.#1:~#3
			\end{center}
	}
% Use these for theorems, lemmas, proofs, etc.
\newtheorem{theorem}{Theorem}[lecnum]
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{definition}[theorem]{Definition}
\newenvironment{proof}{{\bf Proof:}}{\hfill\rule{2mm}{2mm}}

% **** IF YOU WANT TO DEFINE ADDITIONAL MACROS FOR YOURSELF, PUT THEM HERE:

\begin{document}

% Lecture Details
\lecture{19}{Asst. Prof. M. Mert Ankarali}

\par

\section{Lead Compensator Design}

The lead-compensator is a controller which has the form of a
first-order high-pass filter
%
\begin{align*}
  G_{c}(s) = K_{c} \frac{T s + 1}{T \alpha s + 1}
  \quad \alpha \in (0,1)
\end{align*}
%
In general, we first design $K_{c}$ based on the setady-state
requirements of the system, and then design $T$ and $\alpha$
based on the phase-margin requirement.  First let's illustrate the bode-plots of a unity gain lead-compensator
to understand how we can utilize its properties for the
design process.

     \begin{center}
 \begin{minipage}[h]{\linewidth}
     \begin{center}
       \includegraphics[width=0.9\textwidth]{leadnew}
     \end{center}
 \end{minipage}
     \end{center}

We can see that (both from the actual and approximate bode-plots),
phase-lead compensator is a type of high-pass filter for which
the cut-off (mid) frequency is $\omega_c = \frac{1}{\sqrt{\alpha} T}$.
The low and high frequency gains are respectively $0$ dB and $-20
\mathrm{log}_{10} \alpha$ dB. The lead-compensator phase,
$\phi_{max}$, peaks at its cut-off frequency, and basically we will
try to use this positive maximum phase shift to improve the phase
margin of the feedback
system.

Let's derive a formula for $\phi_{max}$, which we will need during the
design phase
%
\begin{align*}
    \phi_{max} &= \angle [ G_c( j \omega_c) ] = \angle \left[ 1 + T
  \omega_c j \right] - \angle \left[ 1 + \alpha T
  \omega_c j \right] = \arctan(T \omega_c) - \arctan(\alpha  T \omega_c)
  \\
  &= \arctan \left( \frac{1}{\sqrt{\alpha}}  \right) - \arctan \left(
    \sqrt{\alpha} \right)
    \\
  &= \pi/2 - 2 \arctan \left( \sqrt{\alpha} \right)
\\
\sin \phi_{max} &=  \cos \left( 2 \arctan \left( \sqrt{\alpha} \right)
                  \right)
                  = 1 - 2 \left[ \sin\left( \arctan \left( \sqrt{\alpha} \right)
                  \right) \right]^2 = 1 - 2 \frac{\alpha}{1 + \alpha}
                  \\
&= \frac{1 - \alpha}{1 + \alpha}
  \\
    \phi_{max} &= \arcsin \left( \frac{1 - \alpha}{1 + \alpha} \right)
        \         \Rightarrow \ \alpha = \frac{1 - \sin \phi_{max}}{1
                 + \sin \phi_{max}}
\end{align*}
%
As expected when $\alpha = 1$, $\phi_{max} = 0$ since numerator
and denominator time constants becomes equal in this
case. Accordingly, we can see that
%
\begin{align*}
  \alpha \searrow \ \Rightarrow \ \phi_{max} \nearrow
\end{align*}
%
Theoretical maximum value for $\phi_{max}$ is $90^o$,
however practically $\phi_{max} < 75^o$ for analog
lead compensator circuits. Another important factor that
we will need to pay attention is the gain-shift of
the lead-compensator at the cut-off frequency
%
\begin{align*}
  M_{dB}(j \omega_c) = -10 \mathrm{log}_{10} \alpha \quad \& \quad
  | G(j \omega_c) | = \frac{1}{\sqrt{\alpha}}
\end{align*}
%

We will illustrate the design process on an example

\vspace{6pt}

\textbf{Ex:} Consider the following feedback system illustrated below.
It is given that $G(s) = \frac{1}{s (s+1)}$ and we want to
design a lead-compensator, $G_c(s) = K_c \frac{T s + 1}{T \alpha s +
  1}$, such that unit-ramp steady-state error satisfies, $e_{ss} = 0.1$ and phase-margin of the compensated system satisfies $\phi_m^* =
\left[ 45^0 , 55^0 \right]$.

\vspace{6pt}

     \begin{center}
 \begin{minipage}[h]{\linewidth}
     \begin{center}
       \includegraphics[width=0.6\textwidth]{ex1block}
     \end{center}
 \end{minipage}
     \end{center}

\vspace{6pt}

\textbf{Solution:}

\textbf{Step 1:} We design/compute $K_c$ based on the steady-state
requirement on the unit-ramp error.
%
\begin{align}
   e_{ss} = \frac{1}{K_v} = \frac{1}{K_c} = 1 \ \rightarrow \ K_c = 10
\end{align}

\textbf{Step 2:} Embed $K_c$ into $G(s)$, i.e.
%
\begin{align*}
  \bar{G}(s) = K_c G(s) = \frac{10}{s (s+1)} ,
\end{align*}
%
draw the bode-plot for $\bar{G}(s)$, and compute the gain-crosover
frequency, $\omega_{gc}$, and the phase margin, $\phi_M$, for the
uncompensated $\bar{G}(s)$. The bode plot of $\bar{G}(s)$ and its
approximation is illustrated in the Figure below.

     \begin{center}
 \begin{minipage}[h]{\linewidth}
     \begin{center}
       \includegraphics[width=0.85\textwidth]{bodesystem}
     \end{center}
 \end{minipage}
     \end{center}

If we concentrate on the approximate bode-plots, we can estimate
the gain-crosover frequency and the phase-margin as
%
\begin{align*}
  \omega_{gc} &\approx \sqrt{10} \ rad/s = 3.16 \ rad/s
  \\
  \phi_m &\approx 22.5^o
\end{align*}
%
If we compute the actual values from the actual Bode plot, we obtain
%
\begin{align*}
  \omega_{gc} &= 3.08 \ rad/s
  \\
  \phi_m &\approx 18^o
\end{align*}
%
We can see that approximation is pretty good for the
$\omega_c$, however a little crude for $\phi_m$.
So let's compute $\angle [ G(j \omega_{gc}) ] $ for
$\omega_{gc} \approx \sqrt{10}$ and estimate the
phase-margin based on this frequency
%
\begin{align*}
  \angle [ G(j \omega_{gc}) ] &= \angle [ G(j \sqrt{10}) ] =
 -90^0 - \arctan \left( \sqrt{10} \right) =  -162.5^o
  \\
 \phi_M \approx 17.5^o
\end{align*}

\textbf{Step 3:} Compute the required phase increment, $\Delta \phi$ to be
added by the compensator and compute $\alpha$
%
\begin{align*}
   \phi_{max} &\approx \Delta \phi = \phi_M^* - \phi_M + \epsilon
  \\
  \epsilon &= 5^0 - 10^0
\end{align*}
%
So for the given problem, we can compute $\phi_{max}$
and $\alpha$ as
%
\begin{align*}
   \phi_{max} &\approx 47.5^o - 17.5^0 + 7^0 = 37^0
                \\
  \alpha &= \frac{1 - \sin \phi_{max}}{1 + \sin \phi_{max}} \approx \frac{1}{4}
\end{align*}

\textbf{Step 4:} Estimate the ``new'' gain-crossover frequency,
$\hat{\omega}_{gc}$, and place the peak point of the lead-compensator,
at this estimated $\hat{\omega}_{gc}$.

We already know that, lead-compensator at its
center/cutoff frequency shifts the bode magnitude
by $-10 \mathrm{log}_{10} \alpha$ (or increases the gain
by $\frac{1}{\sqrt{6}}$) and this causes a shift in
gain-crossover frequency.
For this reason, we can estimate the new gain-crossover
frequency as the point where the bode magnitude of
$\bar{G}(s)$ crosses $10 \ \mathrm{log}_{10} \alpha$,
i.e.
%
\begin{align*}
  M_{dB}( j \hat{\omega}_{gc} ) = 10 \ \mathrm{log}_{10} \alpha
\quad or \quad
  |G(j \hat{\omega}_{gc})| = \sqrt{\alpha}
\end{align*}
%
In our problem,
%
\begin{align*}
  M_{dB}( j \hat{\omega}_{gc} ) \approx -6 \ dB
\quad or \quad
  |G(j \hat{\omega}_{gc})| = \frac{1}{2}
\end{align*}
%
We can indeed estimate the new gain-crossover frequency graphically from the
bode-plot. The figure below, illustrates how we can find
the new-gain crossover frequency graphically

     \begin{center}
 \begin{minipage}[h]{\linewidth}
     \begin{center}
       \includegraphics[width=0.85\textwidth]{gaincros}
     \end{center}
 \end{minipage}
     \end{center}

where $\hat{\omega}_{gc} \approx 4.5 \ rad/s$.
We can also estimate the new-gain crosover frequency
numerically
%
\begin{align*}
  |G(j \hat{\omega}_{gc})| = \frac{1}{2}
  \\
  \frac{100}{\hat{\omega}_{gc}^2 \left( \hat{\omega}_{gc}^2 + 1
  \right)} = \frac{1}{4}
  \\
  \hat{\omega}_{gc}^4 + \hat{\omega}_{gc}^2 - 400 = 0
  \\
   \hat{\omega}_{gc} \approx 400^{1/4} = 4.47 \ rad/s
\end{align*}
%
which is very close to the graphical estimation.

\textbf{Step 5:} Compute $T$ using $\hat{\omega}_{gc}$ and
check if the lead-compensator meets the phase-margin
requirement. Otherwise, repeat the process with a higher
$\Delta \phi$ angle.
%
\begin{align*}
  \hat{\omega}_{gc} = \omega_c = \frac{1}{\sqrt{\alpha} T}
                      \quad \Rightarrow \quad
  T = \frac{1}{\sqrt{\alpha} \hat{\omega}_{gc}}
\end{align*}
%
In our example
%
\begin{align*}
     T = \frac{1}{\frac{4.5}{2}} \approx 0.45
         \quad \Rightarrow \quad
   G_c(s) = 10 \frac{0.45 s + 1}{0.1125 s + 1} = 40 \frac{s + 2.25}{s + 9}
\end{align*}
%
The Figure below illustrates the bode plots of both
compensated and uncompensated ($\bar{G}(s) = K_c G(s)$) systems.
Compensated systems has a phase margin of $\phi_m = 49.7^o$
which meets the requirements.
%
     \begin{center}
 \begin{minipage}[h]{\linewidth}
     \begin{center}
       \includegraphics[width=0.85\textwidth]{comparison}
     \end{center}
 \end{minipage}
     \end{center}
%
Now let's compute new phase margin directly using the
new gain-crossover frequency
%
\begin{align*}
	\phi_{M} &= -180^o + \angle [ G_c(j 4.45 \ rad/s) \bar{G}( j 4.45 \ rad/s) ]
		= -180^o + 37^o + \angle [ \bar{G} ( j 4.45 \ rad/s) ]
		\\
		&= -180^o + 37^o - 90^o - \angle [ j 4.45 + 1 ]
	= -180^o + 37^o - 90^o - 77^0
		\\
		&= 50^o
\end{align*}
%
which is consistent with the phase-margin on the bode-plot.
In Figure below, we compare the closed-loop step responses
of both uncompensated and compensated closed-loop
transfer functions. We can clearly see that, the lead-compensator
substantially improves both the settling time
and over-shoot performance.

     \begin{center}
 \begin{minipage}[h]{0.7\linewidth}
     \begin{center}
       \includegraphics[width=\textwidth]{step}
     \end{center}
 \end{minipage}
     \end{center}

\paragraph{Phase-Lead Design Guideline:}
\begin{enumerate}
	\item Design/compute $K_c$ based on the steady-state requirements
	%
	\item Embed $K_c$ into $G(s)$, i.e. $ \bar{G(s)} = K_c G(s)$, then
	draw the bode-plots for $\bar{G}(s)$, and compute the gain-crossover
	frequency, $\omega_{gc}$, and the phase margin, $\phi_M$, for the
	uncompensated $\bar{G}(s)$.
	%
	\item Compute the required phase increment, $\Delta \phi$ to be
	added by the compensator and design/compute $\alpha$
	\item Estimate the ``new'' gain-crossover frequency,
	$\hat{\omega}_{gc}$, and place the peak point of the lead-compensator,
	at this estimated $\hat{\omega}_{gc}$.
	\item Check the phase-margin if it does not meet the requirements
	increase $\Delta \phi$ and repeat the process.
\end{enumerate}

\newpage

\section{Lag Compensator Design}

The lag-compensator is a controller which has the form of a
first-order low-pass filter
%
\begin{align*}
  G_{c}(s) = K_{c} \frac{T s + 1}{T \alpha s + 1}
  \quad \alpha \in (1,\infty)
\end{align*}
%
In general, we first design $K_{c}$ based on the steady-state
requirements of the system, then design $\alpha$
based on the phase-margin requirement, and finally
choose a $T$ such that phase-lag of the compensator
does not interfere with the gain-crossover frequency.

First let's illustrate the bode-plots of a unity gain lag-compensator
to understand how we can utilize its properties for the
design process.

     \begin{center}
 \begin{minipage}[h]{\linewidth}
     \begin{center}
       \includegraphics[width=0.8\textwidth]{lag}
     \end{center}
 \end{minipage}
     \end{center}

In lag-compensator design, we basically use the negative-gain shift of
the compensator in the high-frequency region and we try to push the
low and mid frequency region to the left (in frequency axis) such that
they don't interfere with the gain-crossover frequency. For this
reason, design process easier compared to the lead-compensator.

We will illustrate the lag-compensator design process on an the same example

\vspace{6pt}

\textbf{Ex:} Consider the feedback system that we analyzed
previously in lead compensatory case. Plant is same,
$G(s) = \frac{1}{s (s+1)}$. However now we want to
design a lag-compensator, $G_c(s) = K_c \frac{T s + 1}{T \alpha s + 1}
, \ a \in (1,\infty))$, such that unit-ramp steady-state error satisfies, $e_{ss}
0.1$ and phase-margin of the compensated system satisfies $\phi_m^*
> 40^o$.

\textbf{Solution:}

\textbf{Step 1:} Same as the lead-design, we design/compute $K_c$ based on the steady-state
requirement on the unit-ramp error.
%
\begin{align}
   e_{ss} = \frac{1}{K_v} = \frac{1}{K_c} = 1 \ \rightarrow \ K_c = 10
\end{align}

\textbf{Step 2:} Embed $K_c$ into $G(s)$, i.e.
%
\begin{align*}
  \bar{G(s)} = K_c G(s) = \frac{10}{s (s+1)} ,
\end{align*}
%
draw the bode-plot for $\bar{G}(s)$.

\textbf{Step 3:} Compute/find the required gain-crossover
frequency, $\omega_{gc}^*$, based on
the required phase-margin, $\phi^*_M$, and compute/find
the magnitude of $G(s)$ at $\omega_{gc}^*$, i.e.
$| \bar{G}(j  \omega_{gc}^*) | $ or $20 \mathrm{log}_20 | \bar{G}(j  \omega_{gc}^*) |$.
%
\begin{align*}
  \angle [ \bar{G}(j  \omega_{gc}^*) ] &= -180^o + \phi_M^* + \epsilon
  \\
  \epsilon &\approx 5^0
\end{align*}
%
Note that at high-frequency
region, lag-compensator acts like negative gain
shift on magnitude plot while not affecting the
phase response. Such a gain shift technically
changes the gain-crossover frequency, which
can be potentially used for changing the phase
margin.

For our example, required phase-margin, $\phi^*_M$
can be computed as
%
\begin{align*}
  \angle [ \bar{G}(j  \omega_{gc}^*) ] \approx 45^p
  \quad \Rightarrow \quad \omega_{gc}^* \approx 1 \ rad/s
\end{align*}
%
The bode plot of $\bar{G(s)}$,
required gain-crossover frequency, and
the magnitude of $\bar{G(s)}$ at the desired
gain-crossover frequency are
illustrated in the Figure below.

     \begin{center}
 \begin{minipage}[h]{\linewidth}
     \begin{center}
       \includegraphics[width=0.8\textwidth]{laggaincross}
     \end{center}
 \end{minipage}
     \end{center}

From the bode-plots we can observe that
at the desired  $\omega_{gc}^* \approx 1 \ rad/s$,
bode-plot approximation has a magnitude of
$20 \ dB$, whereas the magnitude in the actual
bide plot is approximately $17 \ dB$. In this example,
let's use the magnitude of the approximation in the
next Step.

\textbf{Step 4:} Compute $\alpha$ to compensate the
the magnitude at the new-gain crossover frequency
%
\begin{align*}
  20 log_{10} \alpha =  20 \mathrm{log}_{10} |\bar{G}(j  \omega_{gc}^*) |
  	\quad \mathrm{or} \quad
     \alpha = | \bar{G}(j  \omega_{gc}^*) |
\end{align*}
%
In our example, $\alpha$, can be computed as
%
\begin{align*}
	20 log_{10} \alpha \approx 20 \ dB
	\ \Rightarrow \ \alpha \approx 10
\end{align*}
%

\textbf{Step 4:} Choose $T$ such that ``$10/T \leq \omega_{gc}^*$''.
Note that $10/T$ is the frequency where the phase of the compensator
approximately re-approaches to zero.
This is required in order the
          negative phase bump of the compensator does not affect the
          phase-margin.
%
\begin{align*}
	\frac{1}{T} \approx 0.1 \omega_{gc}^*
		\ \Rightarrow \ T \approx 10
		\\
G_c(s) = 10 \frac{10 s + 1}{100 s + 1} = \frac{s + 0.1}{s + 0.01}
\end{align*}
%
The Figure below illustrates the bode plots of
the uncompensated ($\bar{G}(s) = K_c G(s)$) system,
designed lag-compensator, and the compensated system.
Compensated systems has a phase margin of $\phi_m = 45^o$
which meets the requirements.

     \begin{center}
 \begin{minipage}[h]{\linewidth}
     \begin{center}
       \includegraphics[width=0.75\textwidth]{complag}
     \end{center}
 \end{minipage}
     \end{center}

In Figure below, we compare the closed-loop step responses
of both uncompensated and compensated closed-loop
transfer functions. We can see that, similar to the the lead-compensator,
lag-compensator improves the over-shoot performance.
However, the settling-time of the compensated system is approximately
two times of the original system. This is major the drawback of the lag-compensator.

     \begin{center}
 \begin{minipage}[h]{0.65\linewidth}
     \begin{center}
       \includegraphics[width=\textwidth]{lagstep}
     \end{center}
 \end{minipage}
     \end{center}

The reason behind the reduced convergence speed performance is that
new gain-crossover frequency is less than the original gain-crossover frequency.
PM is closely related with over-shoot performance. On the other hand
gain-crossover frequency determines the band-with of the system
which is closely related with the rise and settling times.

\paragraph{Phase-Lag Design Guideline:}
\begin{enumerate}
	\item Design/compute $K_c$ based on the steady-state requirements
	%
	\item Compute/find the required gain-crossover
	frequency, $\omega_{gc}^*$, based on
	the required phase-margin, $\phi^*_M$, and compute/find
	the magnitude of $G(s)$ at $\omega_{gc}^*$
	%
	\item Compute $\alpha$ to compensate the
	the magnitude at the new-gain crossover frequency
	%
	\item Chose $T$ at such that ``$10/T \leq \omega_{gc}^*$''. This is required in order the
          negative phase bump of the compensator does not affect the
          phase-margin. Note that $10/T$ is the frequency where the phase of the compensator
          approximately re-approaches to zero.
	%
	\item Check the phase-margin if it does not meet the requirements
	change $\Delta \phi$ and repeat the process.
\end{enumerate}

% **** This ENDS THE EXAMPLES. DON'T DELETE THE FOLLOWING LINE:
\end{document}


%% -*- coding: utf-8 -*-
\documentclass{beamer}

\input{common}
%%% frontmatter
\input{frontmatter}
\subtitle
{Functors, Applicatives, and Parsers}

\begin{document}

\begin{frame}
  \titlepage
\end{frame}

\begin{frame}
  \frametitle{Introduction}
  \begin{itemize}
  \item Functors and applicatives are concepts from \alert{category
      theory}
  \item A very general and abstract theory about structures and maps between them
  \item So general that mathematicians call it ``general abstract
    nonsense''
  \item Yields very useful abstractions for functional programming
  \item We only review them specialized for Haskell
  \end{itemize}
\end{frame}

\section{Functors}
\begin{frame}[fragile]
  \frametitle{Functors}
  \begin{block}<+->{Definition}
    A \alert{Functor} is a mapping \lstinline{f} between types such that for
    every pair of type \lstinline{a} and \lstinline$b$ there is a
    function
    \lstinline|fmap :: (a -> b) -> (f a -> f b)|
    such that the \alert{functorial laws} hold:
    \begin{enumerate}
    \item the identity function on \lstinline$a$ is mapped to the
      identity function on \lstinline$f a$:\\
      \lstinline$fmap id fx == id fx$, \quad for all \lstinline$fx$ in
      \lstinline$f a$
    \item \lstinline$fmap$ is compatible with function composition\\
      \lstinline$fmap (f . g) == fmap f . fmap g$, \quad for all
      \lstinline$f :: b -> c$ and \lstinline$g :: a -> b$
    \end{enumerate}
  \end{block}
  \begin{alertblock}<+->{Functions on types}
    \begin{itemize}
    \item \lstinline{Int}, \lstinline{Bool}, \lstinline{Double} etc
      are types.
    \item parameterized types like \lstinline$[a]$,
      \lstinline$BTree a$, \lstinline$IO a$ can be considered as a
      type constructor (i.e., \lstinline$[]$, \lstinline$BTree$,
      \lstinline$IO$) applied to a type
    \item We can express that formally by writing \alert{kindings}:
      \lstinline{Int :: *},
      \lstinline{Bool :: *},
      \lstinline{Double :: *}, but
      \lstinline{[] :: * -> *},
      \lstinline{BTree :: * -> *},
      \lstinline{IO :: * -> *}
    \end{itemize}
  \end{alertblock}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Functors in Haskell}
\begin{alertblock}<+->{The functor class}
\begin{lstlisting}
class Functor f where
  fmap :: (a -> b) -> (f a -> f b)
\end{lstlisting}
    \begin{itemize}
  \item \alert{\textbf{NEW:}} \texttt{f} is a type variable that can
    stand for \textbf{type
      constructors} (ie, functions on types) like  \texttt{IO},
    \texttt{[]}, and others.  So \lstinline$f :: * -> *$!
  \end{itemize}
\end{alertblock}
\begin{exampleblock}<+->{Good news}
  We already know a couple of functors!
\end{exampleblock}
\end{frame}

\begin{frame}
  \frametitle{List is a functor}
  \begin{itemize}[<+->]
  \item To make list an instance of functor, we need to instantiate
    the type \lstinline$f$ in the type of \lstinline|fmap| by
    \lstinline|[]|, the list type constructor
  \item \lstinline|fmap :: (a -> b) -> (f a -> f b)|
  \item \lstinline|fmap :: (a -> b) -> ([a] -> [b])|
  \item Looks familiar?
  \item It's the type of \lstinline|map|
  \item It remains to check the functorial laws on \lstinline|map|
  \end{itemize}
\end{frame}
\begin{frame}[fragile]
  \frametitle{Functorial laws for list}
  \begin{block}<+->{\lstinline$fmap id fx == id fx$}
    \lstinline$fx$ is a list, so we must proceed by induction
    \begin{itemize}
    \item \lstinline|map id [] == [] == id []|
    \item \lstinline$map id (x:xs) == id x : map id xs == x : xs == id (x : xs)$
    \end{itemize}
  \end{block}
  \begin{block}<+->{\lstinline$fmap (f . g) == fmap f . fmap g$}
    Must hold when applied to any list \lstinline$fx$
    \begin{itemize}
    \item \lstinline$map (f . g) [] == [] == map f (map g [])$
    \item \lstinline$map (f . g) (x : xs) == (f . g) x : map (f . g) xs$ \\
      \lstinline$== f (g x) : (map f . map g) xs$ by function
      composition and induction \\
      \lstinline$== f (g x) : map f (map g xs)$ by function composition
      \\
      \lstinline$== map f (g x : map g xs)$ by \lstinline$map f$
      \\
      \lstinline$== map f (map g (x : xs))$ by \lstinline$map g$
      \\
      \lstinline$== (map f . map g) (x : xs)$
    \end{itemize}
  \end{block}
\end{frame}
\begin{frame}
  \frametitle{Maybe is a functor}
  \begin{itemize}[<+->]
  \item Reminder: \lstinline$data Maybe a = Nothing | Just a$
  \item To make \lstinline$Maybe$ an instance of functor, we need to instantiate
    the type \lstinline$f$ in the type of \lstinline|fmap| by the type constructor
    \lstinline|Maybe|
  \item \lstinline|fmap :: (a -> b) -> (f a -> f b)|
  \item \lstinline|mapMaybe :: (a -> b) -> (Maybe a -> Maybe b)|
  \item There is actually no real choice for its definition
  \item \lstinline|mapMaybe g Nothing = Nothing|
  \item \lstinline|mapMaybe g (Just a) = Just (g a)|
  \item Second equation could return \lstinline$Nothing$, but that
    would violate the functorial laws
  \item It remains to check the functorial laws on \lstinline|mapMaybe|
  \end{itemize}
\end{frame}
\begin{frame}[fragile]
  \frametitle{Functorial laws for Maybe}
  \begin{block}<+->{\lstinline$fmap id fx == id fx$}
    \lstinline$fx$ is a Maybe, so we must proceed by induction (cases)
    \begin{itemize}
    \item \lstinline$mapMaybe id Nothing == Nothing == id Nothing$
    \item \lstinline$mapMaybe id (Just x) == Just x == id (Just x)$
    \end{itemize}
  \end{block}
  \begin{block}<+->{\lstinline$fmap (f . g) == fmap f . fmap g$}
    Must hold when applied to any Maybe \lstinline$fx$
    \begin{itemize}
    \item \lstinline$mapMaybe (f . g) Nothing == Nothing == map f (map g Nothing)$
    \item \lstinline$mapMaybe (f . g) (Just x)$ \\
      \lstinline$== Just ((f . g) x)$
      \\
      \lstinline$== Just (f (g x))$ by function composition
      \\
      \lstinline$== mapMaybe f (Just (g x))$ by \lstinline$map f$
      \\
      \lstinline$== mapMaybe f (mapMaybe g (Just x))$ by \lstinline$map g$
      \\
      \lstinline$== (mapMaybe f . mapMaybe g) (Just x)$
    \end{itemize}
  \end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{BTree is a functor}
  \begin{itemize}[<+->]
  \item Reminder:
    \lstinline$data BTree a = Leaf | Node (BTree a) a (BTree a)$
  \item To make \lstinline$BTree$ an instance of functor, we need to instantiate
    the type \lstinline$f$ in the type of \lstinline|fmap| by the type constructor
    \lstinline|BTree|
  \item \lstinline|fmap :: (a -> b) -> (f a -> f b)|
  \item \lstinline|mapBTree :: (a -> b) -> (BTree a -> BTree b)|
  \item There is actually no real choice for its definition
\begin{lstlisting}
mapBTree g Leaf = Leaf
mapBTree g (Node l a r) = Node (mapBTree g l) (g a) (mapBTree g r)
\end{lstlisting}
  \item In the second equation we need to transform the data at the
    node by \lstinline$g$ and the subtrees of type
    \lstinline$BTree a$ recursively to \lstinline$BTree b$ using the
    \lstinline$mapBTree$ function
  \item It remains to check the functorial laws on
    \lstinline|mapBTree|, but we'll leave this inductive proof to you.
  \end{itemize}
\end{frame}

\section{Applicatives}
\begin{frame}
  \frametitle{Applicatives}
  \begin{itemize}
  \item An applicative (functor) is a special kind of functor
  \item It has further operations and laws
  \item We motivate it with a couple of examples
  \end{itemize}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Applicative}
  \begin{alertblock}<+->{Example 1: sequencing IO commands}
\begin{lstlisting}
sequence :: [IO a] -> IO [a]
sequence []       = return []
sequence (io:ios) = do x <- io
                       xs <- sequence ios
                       return (x:xs)
\end{lstlisting}
  \end{alertblock}
  \begin{alertblock}<+->{Alternative way}
\begin{lstlisting}
sequence []       = return []
sequence (io:ios) = return (:) `ap` io `ap` sequence ios

return :: Monad m => a -> m a
ap     :: Monad m => m (a -> b) -> m a -> m b
\end{lstlisting}
  \end{alertblock}
\end{frame}

\begin{frame}[fragile]
 \frametitle{Applicative}
 \begin{exampleblock}<+->{Example 2: transposition}
\begin{lstlisting}
transpose :: [[a]] -> [[a]]
transpose [] = repeat []
transpose (xs:xss) = zipWith (:) xs (transpose xss)
\end{lstlisting}
 \end{exampleblock}
 \begin{alertblock}<+->{Rewrite}
\begin{lstlisting}
transpose []       = repeat []
transpose (xs:xss) = repeat (:) `zapp` xs `zapp` transpose xss

zapp :: [a -> b] -> [a] -> [b]
zapp fs xs = zipWith ($) fs xs
\end{lstlisting}
 \end{alertblock}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Applicative Interpreter}
  \begin{block}<+->{A datatype for expressions}
\begin{lstlisting}
data Exp v
  = Var v               -- variables
  | Val Int             -- constants
  | Add (Exp v) (Exp v) -- addition
\end{lstlisting}
  \end{block}
  \begin{exampleblock}<+->{Standard interpretation}
\begin{lstlisting}
eval :: Exp v -> Env v -> Int
eval (Var v) env = fetch v env
eval (Val i) env = i
eval (Add e1 e2) env = eval e1 env + eval e2 env

type Env v = v -> Int
fetch :: v -> Env v -> Int
fetch v env = env v
\end{lstlisting}
\end{exampleblock}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Applicative Interpreter}
  \begin{alertblock}{Alternative implementation}
\begin{lstlisting}
eval' :: Exp v -> Env v -> Int
eval' (Var v) = fetch v
eval' (Val i) = const i
eval' (Add e1 e2) = const (+) `ess` (eval' e1) `ess` (eval' e2)

ess a b c = (a c) (b c)
\end{lstlisting}
\end{alertblock}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Applicative}
\begin{exampleblock}{Extract the common structure}
\begin{lstlisting}
class Functor f => Applicative f where
  pure  :: a -> f a
  (<*>) :: f (a -> b) -> f a -> f b
\end{lstlisting}
\end{exampleblock}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Applicative}
  \begin{block}{Laws}
  \begin{itemize}
  \item Identity
\begin{lstlisting}
pure id <*> v == v
\end{lstlisting}
  \item Composition
\begin{lstlisting}
pure (.) <*> u <*> v <*> w = u <*> (v <*> w)
\end{lstlisting}
  \item Homomorphism
\begin{lstlisting}
pure f <*> pure x = pure (f x)
\end{lstlisting}
  \item Interchange
\begin{lstlisting}
u <*> pure y = pure ($ y) <*> u
\end{lstlisting}
  \end{itemize}
  \end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Instances of Applicative}
  \begin{itemize}
  \item<+-> List, Maybe, and IO are also applicatives
  \end{itemize}
  \begin{block}<+->{Lists}
\begin{lstlisting}
instance Applicative [] where
  -- pure :: a -> [a]
  pure a = [a]
  -- (<*>) :: [a -> b] -> [a] -> [b]
  fs <*> xs = concatMap (\f -> map f xs) fs
\end{lstlisting}
  \end{block}
  \begin{block}<+->{Maybe}
\begin{lstlisting}
instance Applicative Maybe where
  -- pure :: a -> Maybe a
  pure a = Just a
  -- (<*>) :: Maybe (a -> b) -> Maybe a -> Maybe b
  Just f <*> Just a = Just (f a)
  _ <*> _ = Nothing
\end{lstlisting}
  \end{block}
\end{frame}

\section{Parsers}
\begin{frame}[fragile]
  \frametitle{An interesting example for Applicatives}
  \begin{block}<+->{Simple arithmetic expressions}
\begin{lstlisting}
data Term  = Con Integer
           | Bin Term Op Term
             deriving (Eq, Show)

data Op    = Add | Sub | Mul | Div
             deriving (Eq, Show)
\end{lstlisting}
\end{block}
\begin{alertblock}<+->{Parsing expressions}
  \begin{itemize}
  \item Read a string like \texttt{"3+42/6"}
  \item Recognize it as a valid term
  \item Return \texttt{Bin (Con 3) Add (Bin (Con 42) Div (Con 6))}
  \end{itemize}
\end{alertblock}
\end{frame}
\begin{frame}[fragile]
  \frametitle{Parsing}
\begin{block}{The type of a simple parser}
\begin{lstlisting}
type Parser token result = [token] -> [(result, [token])]
\end{lstlisting}
\end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Combinator parsing}
  \begin{block}{Primitive parsers}
\begin{lstlisting}
pempty :: Parser t r
succeed :: r -> Parser t r
satisfy :: (t -> Bool) -> Parser t t
msatisfy :: (t -> Maybe a) -> Parser t a
lit :: Eq t => t -> Parser t t
\end{lstlisting}
  \end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Combinator parsing II}
  \begin{block}{Combination of parsers}
\begin{lstlisting}
palt :: Parser t r -> Parser t r -> Parser t r
pseq :: Parser t (s -> r) -> Parser t s -> Parser t r
pmap :: (s -> r) -> Parser t s -> Parser t r
\end{lstlisting}
\end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{A taste of compiler construction}
\begin{block}{A lexer}
 A lexer partitions the incoming list of
 characters into a list of tokens. A token is either a single symbol,
 an identifier, or a number. Whitespace characters are removed.
\end{block}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Underlying concepts}
  \begin{alertblock}{Parsers have a rich structure}
    \begin{itemize}
    \item parsing illustrates functors, applicatives, as well as
      monads that we already saw in the guise of IO instructions
    \end{itemize}
  \end{alertblock}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Parsing is \dots}
  \begin{block}{A functor}
    Check the functorial laws!
  \end{block}
  \begin{block}{An applicative}
    Check applicative laws!
  \end{block}
  \begin{block}{A monad}
    Check the monad laws (upcoming)!
  \end{block}
  \begin{alertblock}{Consequence}
    Can use \texttt{do} notation for parsing!
  \end{alertblock}
\end{frame}

\begin{frame}[fragile]
  \frametitle{Parsers are Applicative!}
  \begin{alertblock}{}
\begin{lstlisting}
instance Applicative (Parser' token) where
  pure = return
  (<*>) = ap

instance Alternative (Parser' token) where
  empty = mzero
  (<|>) = mplus
\end{lstlisting}
  \end{alertblock}
\end{frame}

\begin{frame}
  \frametitle{Wrapup}
  \begin{itemize}[<+->]
  \item what if there are multiple applicatives?
  \item they just compose (unlike monads)
  \item applicative do notation
  \item applicatives cannot express dependency
  \item enable more clever parsers
  \end{itemize}

\end{frame}

\end{document}