\lab{Windows 64 bit Installation}{Windows 64 bit}
\label{win64install}

\objective{Install fundamental package requirements for the labs on a Windows 64 bit machine.}

\begin{warn}
The information provided in this appendix is for convenience only.
The reader assumes all the liability and risk involved in making any change to configuration mentioned in this appendix.
The authors of this appendix are not responsible for any damage that may result from any changes covered in following material.
Some of the changes could cause unexpected behavior in the computer system.
\end{warn}

There are a wide variety of ways to install most of the Python packages required for these labs.
There are, however, some problems with Cython and F2py since the distutils package included in Python is currently not very good for building new extension modules on 64 bit Windows machines.
Installing these libraries can be difficult; the following are ways that have worked for us.
They still may not work on every computer.

% We suggest students use Anaconda because it includes nearly all the packages we use.
% They also tend to update their packages more often.
% It is also much less intrusive on the host filesystem.
\section*{Installing Using the Anaconda Python Distribution}
\begin{enumerate}
\item Note: you can get an academic license for more of the features of the Anaconda Distribution.
The primary benefit is that it will allow many basic linear algebra routines to run much faster.
To obtain an academic license, see \url{https://store.continuum.io/cshop/academicanaconda}.
To activate your academic license, follow the steps in the confirmation email.

\item Install the Anaconda Python Distribution from \url{http://www.continuum.io/downloads}

\item Add the main \li{Anaconda} directory and the \li{bin}, \li{libs}, and \li{include} subdirectories to your system path if they aren't already there.
You can find this option by opening the file explorer to "Computer", right clicking on the background and clicking on "Properties", then clicking on "Advanced Settings", then clicking on "Environment Variables", then add the following to the end of the path if it isn't already there:

\li{;C:\\Anaconda\\; C:\\Anaconda\\Scripts; C:\\Anaconda\\libs; C:\\Anaconda\\include}

This should be done to the path variable in the box for "System Variables" (the system path instead of the user path).

% This is what allows f2py to work on 64 bit windows.
% The version of MinGW that comes as a part of Anaconda is
% currently a 64 bit version of gcc and gfortran 4.7.
% Using MinGW64 with f2py is not officially supported, but it works.
\item Open \li{C:\\Anaconda\\Lib\\site-packages\\numpy\\distutils\\fcompiler\\gnu.py} and comment out the two lines that read
\begin{lstlisting}
else:
    raise NotImplementedError("Only MS compiler supported with gfortran on win64")
\end{lstlisting}
They are probably somewhere around line $330$.
Once you have modified them, they should read
\begin{lstlisting}
#else:
#    raise NotImplementedError("Only MS compiler supported with gfortran on win64")
\end{lstlisting}

% The first definition eliminates the issue with "unable to find vcvarsall.bat" error.
% It is currently already included as a part of Anaconda.
% The second, as of this writing, is not currently included in Anaconda.
% It is needed for pyximport to work properly.
\item Open the file \li{C:\\Anaconda\\Lib\\distutils\\distutils.cfg} (create it if it doesn't already exist) and make sure it contains the following two definitions
\begin{lstlisting}
[build]
compiler=mingw32

[build_ext]
compiler=mingw32
\end{lstlisting}

% These are miscellaneous packages used in different parts of the labs.
% The map reduce frameworks are not included here.
% Those are designed to be run from Unix servers.
\item Install and install the following additional packages from \url{http://www.lfd.uci.edu/~gohlke/pythonlibs/}
\begin{enumerate}
	% Recommended for several different labs.
	% Strictly speaking, scipy.fftpack is good enough for what we do,
	% but pyfftw is faster and more common.
	\item pyfftw
	% Used in the wavelets labs.
	\item pywavelets
	% Used in the optimization labs.
	\item cvxopt
	% Used in the parallel computing labs.
	\item mpi4py
	% This is a requirement in our custom matplotlibrc file.
	% This package is what allows us to render plots from matplotlib as
	% a vector graphic instead of a pixellated image.
	\item pycairo (optional for students, required for anyone that contributes to these labs)
\end{enumerate}

\item Install the \li{scikits.bvp_solver} package by opening a command terminal and running the command
\begin{lstlisting}[style=ShellInput]
pip install scikits.bvp_solver
\end{lstlisting}

% This is used later on as a turnin system.
% It is most important for people that contribute to these labs though.
\item Download and install the Git version control system.
You can get an installer at \url{http://git-scm.com/downloads}.

% Useful for students.
% Absolutely required for contributors.
\item Download and install MikTex from \url{http://miktex.org/download}.
You can use either the 32 or 64 bit version in this case.

\end{enumerate}

\section*{Updating packages}
As a general rule, you update your packages using the same system you used to install them.
If you want to update a package included by default in Anaconda, you can run (in the command terminal) the command
\begin{lstlisting}[style=ShellInput]
conda update <packagename>
\end{lstlisting}
where \li{<packagename>} is the name of the package you want to update.

If you installed a package via an online installer, get an updated version from the same source.

If you installed a package via pip (a Python package manager), you can upgrade it using the command
\begin{lstlisting}[style=ShellInput]
pip install --upgrade <packagename>
\end{lstlisting}
where \li{<packagename>} is the name of the package you want to update.

If you want to update all of the packages that are included in anaconda, you can run the two commands
\begin{lstlisting}[style=ShellInput]
conda update conda
conda update anaconda
\end{lstlisting}
The fist of these commands updates Anaconda's package manager.
The second updates all packages so that they match with the current release version of Anaconda.

\begin{info}
If you update NumPy, you will need to modify the file \li{C:\\Anaconda\\Lib\\site-packages\\numpy\\distutils\\fcompiler\\gnu.py} the same way you did when you first installed everything.
\end{info}

\begin{warn}
Be careful not to try to update a Python package while it is in use.
It is safest to update packages while there is no Python interpreter currently running.
\end{warn}

\begin{info}
All the packages used in these labs have Python 3 versions available, but the code in these labs is currently all written for Python 2.7.
\end{info}


\subsection{Dutch auctions}

In a Dutch auction the auctioneer decreases the price until a single bid is received.

These are open auctions. Each bid is visible to all participants.

The price paid is the bid.


\chapter{Chapter 1. Introduction}

\section*{1.6 Self-Study Questions}

\emph{Explanations}:

	\begin{enumerate}

	\item[1.6.1] \begin{enumerate}[(a)]

		\item If in every situation, the premises are false, there can't be a situation in which the premises are true and the conclusion is false, i.e. the inference can't be invalid. Hence the inference is valid.

		\item If in some situation the premises are true and the conclusion not, the conclusion is false. By definition, this means that the inference is invalid.

		\item If in no situation, the premises are true and the conclusion not, then the inference can't be invalid. So it must be valid.

		\item If in no situation the conclusion is false, there can't be a situation in which the premises are true and the conclusion is false, i.e. the inference can't be invalid. Hence the inference is valid.

		\item If in every situation the conclusion is false, then it's still possible that there is no situation in which the premises are true. But if there's no situation where the premises are true, then the inference would be valid.e

		\item If in every situation where the conclusion is false, at least one of the premises is false, then there can't be a situation in which the premises are true and the conclusion is false. For then, all the premises would be true, but also at least one of them would be false, which can't be.

	\end{enumerate}

	\item[1.6.2] \begin{enumerate}[(a)]

		\item Even if there is a situation in which both premises and conclusion are false, there could still be no situation in which the premises are \emph{true} and the conclusion is false.

		\item There could be \emph{no} situation in which the conclusion is false. Then, trivially, in every situation where the conclusion is false, the premises are true. But, at the same time, the inference would be valid, since there  couldn't be a situation in which the premises are true and the conclusion is false, i.e. the inference can't be invalid.

		\item Suppose the conclusion is true in no situation. This means the conclusion is false in \emph{every} situation. But if then, there's a situation in which the premises are true, in that situation the conclusion must be false. Hence, there's a situation in which the premises are true and the conclusion is false, so the inference is invalid.

		\item This is just the definition of what it means for an inference to be invalid.

		\item It could still be that there is no situation in which the premises are true. Then, trivially, in every situation in which the premises are true, the conclusion is false. But, at the same time, there couldn't be a situation in which the premises are true and the conclusion is false, the argument would be valid.

		\item There could still be a situation in which the premises are true and the conclusion is false, all we're given is that there is no situation in which the premises and the conclusion are both true. This just means that in every situation in which the premises are true, the conclusion is false (see previous option).

	\end{enumerate}

	\end{enumerate}

\section*{1.7 Exercises}

	\begin{enumerate}

		\item[1.7.1] \begin{enumerate}[(a)]

			\item The inference is not valid. To see this, note that we can have a situation in which the premises are true and the conclusion is false. Think of a situation in which there are two whales, Moby and Dick, and one more fish, the clownfish Nemo. Moby is a blue whale and Dick is a grey whale, Nemo is orange and white. For argument's sake, suppose that all whales are fish. Surely then, all blue fish are whales, since there's only one blue fish, the whale Moby. But there's a whale, Dick, which is a fish but grey. So, Dick is not a blue fish. All in all, in the situation, the premises are true and the conclusion is false, the inference is invalid.

			\item The inference is valid. To see this, suppose that we're in a situation in which you didn't not miss your train. Can it be, in such a situation, that you still didn't miss your train? Well, that would mean that some statement, viz. you didn't miss your train, is both true and not true. But this is impossible. So we can't have a situation in which you didn't not miss your train but still didn't miss the train. This means the inference can't be invalid, so it has to be valid.

			\item The inference is valid. The case is very similar to the case with the letters in the drawer. Suppose that it's true in some situation that if you'd checked your mail, then you'd have seen my message, and you didn't see it. Can it be that you checked your mail in that situation? Well, then you'd have seen my mail and you didn't. So it can't be that, in the situation, you checked your mail. So, in every situation where the premises are true, the conclusion needs to be true as well, i.e. the inference is valid.

			\item The inference is invalid. Think of a possible situation in which there are no roses at all, e.g. because a rose disease wiped them out. In such a situation, trivially, every rose would be red (can you show me a non-red rose in the situation?). But there would be no rose and certainly not a red one. So there's a possible situation in which the premises are true and the conclusion is false, the inference is invalid.

			\item The inference is invalid. For concreteness sake, let's suppose that ``that'' is that you jumped over the Eiffel tower. Now think of a possible situation in which you have superhuman strength and, at the same time, pigs have wings and can fly (what a beautiful world it would be). Well, in such a situation, we know that no matter whether you jumped over the Eiffel tower, pigs can fly. So certainly, \emph{if} you did it, pigs can fly (how can the statement be false?). Now suppose that in our magic land, you indeed jumped over the Eiffel tower. Then the conclusion is false---you did actually do it. So, we have a situation in which if you did it, then pigs can fly and you did it---the premises are true and the conclusion is false.

			If you were to add the premise that pigs \emph{don't} fly, the argument would become valid. Check that for yourself. This shows that usual figure of speech involved here is elliptic. It's assumed, in the background, that pigs can't fly.

		\end{enumerate}

	\item[1.7.2] Suppose that an inference is valid. By definition, this means that in every situation where the premises are true, the conclusion is true as well. Suppose you add some premises to the inference. Can that inference be invalid? Well there would need to be a situation in which all the premises of the new inference are true but the conclusion is false. But all the premises of the old argument are also premises of the new inference, so in such a situation also all the premises of the old inference would need to be true. But since the old inference is valid, this means that the conclusion would be true in the situation. So for the new inference to be invalid, the conclusion would need to be both false and true in some situation, which is impossible. Hence the new inference is valid.

	\item[1.7.3] Take some invalid inference. Here are two ways of making it valid:

		\begin{enumerate}[(1)]

			\item Add the conclusion of the inference as a premise. It's easy to see that the new inference can't be invalid, for there would need to be a situation in which the new premises, which now include the conclusion, are all true but the conclusion is false. So the conclusion would need to be both true and false in some situation which can't be. So the inference is valid.

			\item Add any contradiction, such as ``the rose is red and not red'' to the premises. We've already seen that any inference with inconsistent premises is valid (1.3.3), so the inference will be valid, too.

		\end{enumerate}

	\end{enumerate}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "../../logic.tex"
%%% End:


\documentclass[11pt, oneside]{article}   	% use "amsart" instead of "article" for AMSLaTeX format

% \usepackage{draftwatermark}
% \SetWatermarkText{Draft}
% \SetWatermarkScale{5}
% \SetWatermarkLightness {0.9}
% \SetWatermarkColor[rgb]{0.7,0,0}

\usepackage{geometry}                		% See geometry.pdf to learn the layout options. There are lots.
\geometry{letterpaper}                   		% ... or a4paper or a5paper or ...
%\geometry{landscape}                		% Activate for for rotated page geometryhttps://www.washingtonpost.com/world/europe/amid-impeachment-probe-gordon-sondland-is-overseeing-a-renovation-of-his-residence-that-has-cost-1-million-in-taxpayer-money/2019/10/16/d0eece92-ef86-11e9-bb7e-d2026ee0c199_story.html?tid=sm_tw
%\usepackage[parfill]{parskip}    		% Activate to begin paragraphs with an empty line rather than an indent
\usepackage{graphicx}				% Use pdf, png, jpg, or eps� with pdflatex; use eps in DVI mode
								% TeX will automatically convert eps --> pdf in pdflat						\label{thm:integral_domain}

								% TeX will automatically convert eps --> pdf in pdflatex
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{mathrsfs}
\usepackage[hyphens,spaces,obeyspaces]{url}
\usepackage{url}
\usepackage{hyperref}
\usepackage{subcaption}
\usepackage{authblk}
\usepackage{mathtools}
\usepackage{graphicx}
\usepackage[export]{adjustbox}
\usepackage{fixltx2e}
\usepackage{hyperref}
\usepackage{alltt}
\usepackage{color}
\usepackage[utf8]{inputenc}
\usepackage[english]{babel}
\usepackage{float}
\usepackage{bigints}
\usepackage{braket}
\usepackage{siunitx}
\usepackage{mathtools}
\usepackage{xcolor}

\usepackage[hyphenbreaks]{breakurl}

\newtheorem{thm}{Theorem}[section]
% \newtheorem{defn}[thm]{Definition}
\theoremstyle{definition}
\newtheorem{definition}{Definition}[section]
\newtheorem{proposition}{Proposition}[section]
\newtheorem{lemma}{Lemma}[section]
\newtheorem{example}{Example}[section]

\newcommand{\veq}{\mathrel{\rotatebox{90}{$=$}}}
\DeclareMathOperator{\bda}{\Big \downarrow}

\DeclareMathOperator{\E}{\mathbb{E}}
\newcommand{\argmax}{\operatornamewithlimits{argmax}}
\newcommand{\argmin}{\operatornamewithlimits{argmin}}

\title{Fermat's Christmas Theorem}
\author{David Meyer \\ dmm@\{1-4-5.net,uoregon.edu\}}

\date{Last update: December 26, 2020}

\begin{document}
\maketitle

\section*{Fermat's Christmas Theorem}

\noindent
Fermat's Christmas Theorem \cite{wiki:christmas_theorem} is a beautiful and simply stated theorem. It is called Fermat's Christmas Theorem because Fermat announced
a proof of the theorem in a letter to Mersenne dated December 25, 1640. And of course, Fermat didn't include a proof in the letter.

\bigskip
\noindent
Fermat's Christmas Theorem (aka Fermat's theorem on sums of two squares) states that an odd prime number $p$ can be expressed as

\bigskip
\begin{center}
\scalebox{2.00} {$p = r^{2} + s^{2}$}
\end{center}
\bigskip

\noindent
where $r,s \in \mathbb{N}$, if and only if $p \equiv 1 \textrm{ (mod  $4$)}$.

\bigskip
\noindent
For example, the primes 5, 13, 17, 29, 37 and 41 are all congruent to 1 modulo 4 and can be expressed as sums of two squares in the following ways:

\begin{equation*}
\begin{array}{rcll}
5   &=& 1^{2} + 2^{2}   \\
13 &=& 2^{2} + 3^{2}   \\
17 &=&1^{2}  + 4^{2}    \\
29 &=& 2^{2} + 5^{2}    \\
37 &=& 1^{2} + 6^{2}    \\
41 &=& 4^{2} + 5^{2}
\end{array}
\end{equation*}

\bigskip
\noindent
The prime numbers $p$ for which Fermat's Christmas Theorem is true are called Pythagorean primes. See \cite{wiki:pythagorean_primes} for more on Pythagorean primes.

\bigskip
\noindent
A variety of proofs of Fermat's Christmas Theorem can be found in \cite{wiki:christmas_theorem_proofs}.

\bibliographystyle{plain}
\bibliography{/Users/dmm/papers/bib/qc}

\end{document}


\documentclass[12pt]{article}
% page layout
\usepackage[margin=1in]{geometry}
\usepackage[doublespacing]{setspace}

% useful package
\def\studynote{}
\usepackage{../cmd}

% only for this document
\setcounter{section}{-1}
\newcommand{\yhat}{\hat{y}}
\newcommand{\thetabmhat}{\hat{\thetabm}}

% document starts here
\title{Machine Learning Study Note}
\author{Long Wang}
\begin{document}
\maketitle
\tableofcontents

\newpage

\section{Pipeline}

\subsection{Data Collection}

\subsection{Data Preprocessing}

\begin{itemize}
	\item \textbf{Missing data}
	\begin{itemize}
		\item Drop (only when using)
		\item Imputation
		\begin{itemize}
			\item Mean, median, mode, KNN, interpolation
			\item Carry forward/backward
		\end{itemize}
	\end{itemize}
	\item \textbf{Categorical}
	\begin{itemize}
		\item \textbf{Nomial encoding}
		\begin{itemize}
			\item One-hot encoding ($ [1,0,0,\dots] $): fast, but increase dimensions
			\item Count/Mean encoding: replace by mean of counts
		\end{itemize}
		\item \textbf{Ordinal encoding}
		\begin{itemize}
			\item Require known relationship between categories
		\end{itemize}
		\item Text: count vectorization
	\end{itemize}
	\item \textbf{Numeric}
	\begin{itemize}
		\item \textbf{Outlier}
		\begin{itemize}
			\item Remove/replace
			\item Robust models
		\end{itemize}
		\item \textbf{Transformation}
		\begin{itemize}
			\item Normalization: $ \xbm - \xbm_{\min} / (\xbm_{\max} - \xbm_{\min}) \in [0,1] $
			\item Standardization: $ (\xbm - \mubm) / \sigma $
			\item Whitening: $ \hat{\Xbm} \to (\Xbm_{n\times p} - \mubm) \Wbm_{p \times p} $ with $ \E{\hat{\Xbm}} = \zerobm $ and $ \Var(\hat{\Xbm}) = \Ibm $
			\item Quantile normalization: make feature distributions identical
			\item Robust scalar: remove median and scale by quantile range
			\begin{itemize}
				\item \pyinline{sklearn.preprocessing.RobustScaler}
			\end{itemize}
			\item Other: $ \log(x + 1), \sqrt{x} $
		\end{itemize}
	\end{itemize}
	\item \textbf{Imbalanced data}
	\begin{itemize}
		\item Collect more data
		\item Under/Over-sampling, ensemble (different models on different data subset)
		\begin{pyminted}
			import pandas as pd
			data_0 = df.query('Class == 0')
			data_1 = df.query('Class == 1')
			balanced_df = pd.concat([data_0, data_1.sample(len(data_0), random_state=1)])
			balanced_df = balanced_df.sample(frac=1, random_state=1) # shuffle
		\end{pyminted}
		\item Assign weights
		\item Synthetic Minority Oversampling Technique (SMOTE)
		\begin{itemize}
			\item For each point, connect with its $k$-nearest neighbor
			\item Generate points on the edges
		\end{itemize}
	\end{itemize}
\end{itemize}

\subsection{Model Building}

\begin{itemize}
	\item Classification
	\item Regression
	\item Clustering
	\item Dimension Reduction
%	\item Rankings
\end{itemize}

\subsection{Model Evaluation}

\subsubsection{Classification}

\textbf{Confusion matrix} for classification
\begin{table}[htbp]
	\centering
	\begin{tabular}{ |c|c|c|c| }
		\hline
		& Positive (Actual) & Negative (Actual) &\\
		\hline
		Positive (Predict) & $ TP $ & $ FP $ & $\hat{P} = TP + FP $\\
		\hline
		Negative (Predict) & $ FN $ & $ TN $ & $ \hat{N} = FN + TN $\\
		\hline
		& $ P = TP + FN $ & $ N = FP + TN $ &\\
		\hline
	\end{tabular}
\end{table}

\begin{itemize}
	\item \textbf{Accuracy:} correct\% among all $ = \frac{TP + TN}{\hat{P} + \hat{N}}$
	\begin{itemize}
		\item Error rate = 1 $-$ accuracy
	\end{itemize}
	\item \textbf{Precision:} correct\% among $ \hat{P} = \frac{TP}{\hat{P}} = \frac{TP}{TP + FP} $
	\begin{itemize}
		\item FP is important (wrong positive samples/labeling user as robot)
		\item Use when imbalanced data where $ P \gg N $
	\end{itemize}
	\item \textbf{Recall/Sensitive/TPR:} correct\% among $ P = \frac{TP}{P} = \frac{TP}{TP + FN} $
	\begin{itemize}
		\item FN is important (missing negative sample/disaster)
		\item Imbalanced data where $ P \ll N $
	\end{itemize}
	\item \textbf{$F_\beta$ Score:} (1 + $\beta^2$) Precision * Recall / ($\beta^2$ Precision + Recall)
	\begin{itemize}
		\item Balance FP and FN
		\item High $ \beta \Rightarrow $ FN/Recall is important
	\end{itemize}
	\item \textbf{Specificity:} correct\% among $ N = \frac{TN}{N} = \frac{TN}{FP + TN} $
	\item \textbf{ROC/AUC}: plot of y-x (TPR-FPR)
	\begin{itemize}
		\item TPR $ = \frac{TP}{P} = $ Recall; FPR $ = \frac{FP}{N} $
		\begin{itemize}
			\item $ T(t) = \Prob{\hat{p}(x) > t | x \in P} $
			\item $ F(t) = \Prob{\hat{p}(x) > t | x \in N} $
		\end{itemize}
		\item Critical point
		\begin{itemize}
			\item Top-left 1-0 (perfect), bottom-right 0-1 (worst)
			\item Bottom-left 0-0 (all negative), top-right 1-1 (all positive)
		\end{itemize}
		\item AUC = $ \Prob{\text{correctly ranking a random pos-neg pair}} $
		\begin{itemize}
			\item AUC $ = \Prob{\hat{p}(x) > \hat{p}(x') | x \in P, x' \in N} $
			\item AUC $ = \frac{\sum_{i \in P} \rank(i) - (1 + 2 + \cdots + n_P)}{n_P + n_N} $
		\end{itemize}
		\item \textbf{Pro:} not threshold, robust (imbalanced data)
		\item \textbf{Cons:} only ranking, not precision or recall
	\end{itemize}
\end{itemize}

\subsubsection{Feature/Model Selection}
\textbf{Bias-Variance tradeoff}
\begin{itemize}
	\item Simple model: large bias, small variance, underfitting
	\item Complex model: small bias, large variance, overfitting
\end{itemize}

\textbf{Feature selection}
\begin{itemize}
	\item \textbf{Filter method (before training)}
	\begin{itemize}
		\item Feature statistics
		\begin{itemize}
			\item Drop low variance features: \pyinline{sklearn.feature_selection.VarianceThreshold}
			\item Drop high correlation features
		\end{itemize}
		\item Classification: \pyinline{sklearn.feature_selection.SelectKBest(score_func, k)}
		\begin{itemize}
			\item $\chi^2$-test: \pyinline{score_func=chi2}
			\begin{itemize}
				\item Drop features independent of labels
				\item $ T = \sum \frac{(O_i - E_i)^2}{E_i} \sim \chi^2_{(r-1)(c-1)}$
			\end{itemize}
			\item ANOVA F-test: \pyinline{score_func=f_classif}
			\begin{itemize}
				\item For each feature: $ F = \frac{\text{dist between classes}}{\text{dist within classes}} = \frac{\sum_k n_k(\mu_k - \mu)^2}{[\sum_k \sum_i (x_i^{(k)} - \mu_k)^2] / \sum_k (n_k - 1)} $
			\end{itemize}
			\item Mutual infor: \pyinline{score_func=mutual_info_classif}
		\end{itemize}
		\item Categorical feature and one-hot encoding
		\begin{itemize}
			\item \textbf{Weight of Evidence (WOE):} $ \log(\frac{\text{non-event \%}}{\text{event \%}}) $
			\item \textbf{Information Value (IV):} $ \sum (\text{non-event \%} - \text{event \%}) \times \text{WOE} $
		\end{itemize}
	\end{itemize}
	\item \textbf{Wrapper method (after training)}
	\begin{itemize}
		\item Cross-validation: directly estimate testing error
		\item Forward selection, backward elimination: cheap comparisons
		\item Exhaustive selection, recursive elimination
	\end{itemize}
	\item \textbf{Embedded method (with training)}
	\begin{itemize}
		\item AIC/BIC: $ L_0 $ penalty $ \Rightarrow $ sparse
		\item Lasso: $ L_1 $ penalty $ \Rightarrow $ reduce variance, shrink and sparse
		\item Ridge: $ L_2 $ penalty $ \Rightarrow $ reduce variance, shrink
		\item Early stopping: implicit regularization
		\item Decision tree, random forest
	\end{itemize}
\end{itemize}

\textbf{Ensemble methods:} combine base models to improve
\begin{itemize}
	\item \textbf{Bagging (bootstrap aggregating):} train base models \textbf{in parallel} (independently)
	\begin{itemize}
		\item Base model (small bias, large variance) $ \Rightarrow $ ensemble model (smaller variance)
		\item Training: bootstrap samples, select subset of features to improve independence
		\item Prediction: average of all model predictions
	\end{itemize}
	\item \textbf{Boosting:} train base models \textbf{in sequence} (train on error of previous model)
	\begin{itemize}
		\item Base model (large bias, small variance) $ \Rightarrow $ ensemble model (smaller bias)
		\item Prediction: weighted average of all model predictions
	\end{itemize}
	\item Stacking: train any models \textbf{in parallel}
	\begin{itemize}
		\item Split data in two folds, train based models on the first and test on the second
		\item Train meta-model on the second using predictions of base models
	\end{itemize}
\end{itemize}

\subsection{Other}
\subsubsection{Loss function}
\textbf{Classification:} $ y \in \{\pm1\} $ and $ m(y, \xbm, \wbm) = y (\xbm \wbm) $
\begin{itemize}
	\item Zero-one loss: $ \ind{m(y, \xbm, \wbm) \leq 0} $
	\item Hinge loss: $ \max(1 - m(y, \xbm, \wbm), 0) $
	\item Logistic loss: $ \log(1 + e^{1 - m(y, \xbm, \wbm)}) $
\end{itemize}

\textbf{Regression:}
\begin{itemize}
	\item Squared loss: $ (y - \yhat)^2 $
	\item Absolute deviation loss: $ \abs{y - \yhat} $
\end{itemize}

\subsubsection{Kernel}
\textbf{Feature map:} $ \xbm \to \varphi(\xbm) $ (higher dimension)

\textbf{Kernel function:} $ k(\xbm, \xbm') = \ip{\varphi(\xbm)}{\varphi(\xbm')}_{\Vcal} $
\begin{itemize}
	\item \textbf{Kernel matrix:} $ \Kbm_{ij} = k(\xbm_i, \xbm_j) $
	\item \textbf{Mercer's theorem:} an implicit $ \varphi(\cdot) $ exists for PD $ \Kbm $
\end{itemize}

\textbf{Common Kernel}
\begin{itemize}
	\item \textbf{Linear Kernel:} $ k(\xbm, \xbm') = \ip{\xbm}{\xbm'} $
	\item \textbf{Polynomial Kernel:} $ k(\xbm, \xbm') = (\gamma \ip{\xbm}{\xbm'} + r)^d $
	\item \textbf{Sigmoid Kernel:} $ k(\xbm, \xbm') = \tanh(\gamma \ip{\xbm}{\xbm'} + r) $
	\item \textbf{Radial basis function (RBF) Kernel:} $ k(\xbm, \xbm') = \exp(-\gamma \norm{\xbm - \xbm'}^2) $
	\item \textbf{Laplacian Kernel:} $ k(\xbm, \xbm') = \exp(-\gamma \norm{\xbm - \xbm'}_1) $
	\item \textbf{Chi-squared Kernel:} $ k(\xbm, \xbm') = \exp(-\gamma \sum_i (x_i - x'_i)^2 / (x_i + x'_i)) $
\end{itemize}

\subsection{Information Theory}

\textbf{Entropy:} average level of information or uncertainty in the possible outcomes
\begin{itemize}
	\item $ H(P) = -\sum_x p(x) \log p(x) $
	\item Binary: $ -p \log(p) - q \log q $, maximized at $ p = 1/2 $
	\item Conditional entropy: $ H(X|Y) = - \sum_{x,y} p(x,y) \log p(x|y) $
\end{itemize}

\textbf{Kullback-Leibler divergence:} different between two distributions
\begin{itemize}
	\item $ D_{KL}(P||Q) = \sum_x p(x) \log \frac{p(x)}{q(x)} $
\end{itemize}

\textbf{Cross-entropy}
\begin{itemize}
	\item $ H(P, Q) = - \sum_x p(x) \log q(x) $
	\item $ H(P, Q) = \E{-\log Q} = H(P) + D_{KL} (P || Q) $
\end{itemize}

\newpage
\section{Classification}
\subsection{Preliminaries}

\textbf{Method}
\begin{itemize}
	\item \textbf{Direct:} KNN
	\item \textbf{Distribution-based:} naive Bayes, LDA, QDA
	\item \textbf{Model-based:} logistic regression, SVM
	\item \textbf{Tree-based:} decision tree
\end{itemize}

\textbf{Multiclass:} Classify into one of multiple classes
\begin{itemize}
	\item \textbf{One-vs-rest}
	\begin{itemize}
		\item $K$ binary classifier for each class (require real-value probability score)
		\item \textbf{Prediction:} Class with the highest probability
		\item \textbf{Cons:} Scale of probability score may differ, always unbalanced data
	\end{itemize}
	\item \textbf{One-vs-One}
	\begin{itemize}
		\item $ K(K-1)/2 $ classifier
		\item \textbf{Prediction:} majority vote
		\item \textbf{Cons:} same votes
	\end{itemize}
	\item \textbf{Extension from binary:} KNN, Naive Bayes, SVM, Decision tree, NN (softmax)
	\item \textbf{Hierarchical classification}
\end{itemize}

\subsection{KNN (K Nearest Neighbor)}
\textbf{Prediction:} majority vote

\textbf{Pros:} model (nonlinear), fitting (no training), robust (outlier)

\textbf{Cons:} data (balanced), model (distance metric, poor interpretation), fitting (slow)

\textbf{Extension}
\begin{itemize}
	\item Locality-sensitive hashing (LSH): hash similar inputs together with high prob
	\begin{itemize}
		\item LSH family $ \Fcal = \cu{h : \Xcal \to S} $ (map input $ \xbm $ to bucket $s$)
		\begin{itemize}
			\item Hash similar/different point into same bucket with high/low prob
		\end{itemize}
		\item Family $ \Gcal = \cu{g(\xbm) = [h_1(\xbm), \dots, h_k(\xbm)]} \Rightarrow L $ hash tables with $g_1, \dots, g_L$
		\item Pre-process: hash all $n$ data points into $L$ hash tables
		\item Query $ \xbm^* $: examine points in bucket $ g(\xbm^*) $ of each hash tables
	\end{itemize}
\end{itemize}

\subsection{Naive Bayes}

\textbf{Model:} $ p(y = k | \xbm) \propto \pi_k p(\xbm | y = k) $
\begin{itemize}
	\item Prior: $ \hat{\pi}_k = $ proportion of samples with label $ y = k $
	\item Independent features: $ p(\xbm | y = k) = \prod_i p(x_i | y = k) $
	\item Gaussian with \textbf{diagonal} covariance $ \Lambdabm $
	\item Prediction: $ \hat{k} = \argmax_k \pi_k p(\xbm | y = k) $
\end{itemize}

\textbf{Pros:} data (auto scaling)

\textbf{Cons:} model (linear, diagonal Gaussian)

\subsection{Discriminant Analysis}

\textbf{LDA:} $ p(\xbm | y= k) \sim \Ncal(\mubm_k, \Sigmabm) $, \textbf{identical} covariance
\begin{itemize}
	\item Linear: $ \hat{k} = \argmin_k (\xbm - \mubm_k)^T \Sigmabm^{-1} (\xbm - \mubm_k) - 2 \log(\pi_k) = a_k + \bbm_k^T \xbm + \text{const} $
\end{itemize}

\textbf{QDA:} $ p(\xbm | y = k) \sim \Ncal(\mubm_k, \Sigmabm_k) $, \textbf{non-identical} covariance
\begin{itemize}
	\item Quadratic: $ \hat{k} = \argmin_k (\xbm - \mubm_k)^T \Sigmabm_k^{-1} (\xbm - \mubm_k) - 2 \log(\pi_k) = a_k + \bbm_k^T \xbm + \xbm^T \cbm_k \xbm +  \text{const} $
\end{itemize}

\subsection{Logistic Regression}
\textbf{Model:} $ y_i \sim \Ber(p_i) $ where $ p_i = \sigma(\xbm_i \betabm) = \frac{1}{1 + e^{-\xbm_i \betabm}} $
\begin{itemize}
	\item \textbf{Cross-entropy loss}
	\begin{itemize}
		\item $ \ell(y_i, p_i) = - [y_i \log(p_i) + (1 - y_i) \log(1 - p_i)] $
		\item Matrix form: $ - [\ybm^T \log(\sigma(\Xbm \betabm)) + (1 - \ybm)^T \log(1 - \sigma(\Xbm \betabm))] $
		\item Gradient: $ -\Xbm^T (\ybm - \sigma(\Xbm \betabm)) $
	\end{itemize}
	\item \textbf{Logistic loss}
	\begin{itemize}
		\item $ \ell(y_i, \yhat_i) = \log(1 + \exp(-y_i \yhat_i)) $ where $ \yhat_i = \xbm_i \betabm $
	\end{itemize}
	\item \textbf{Link functions:} $ \xbm_i \betabm = f(p_i) $
	\begin{itemize}
		\item $ \text{logit}(p_i) = \log(\frac{p_i}{1 - p_i}) $
		\item $ \text{probit}(p_i) = \Phi^{-1}(p_i) $
		\item $ \text{log-log}(p_i) = \log(-\log(1 - p_i)) $
	\end{itemize}
\end{itemize}

\textbf{Pros:} model (prob scores), fitting (fast)

\textbf{Cons:} data (normalize), model (linear)

\textbf{Extension}
\begin{itemize}
	\item Multinomial: $ \log(\frac{p_{i,k}}{1 - p_{i,1} - \cdots - p_{i,K-1}}) = \xbm_i \betabm_k $ for $ k = 1, ..., K - 1 $
\end{itemize}

\textbf{Implementation}
\begin{pyminted}
	from sklearn.linear_model import LogisticRegression
	# x.shape = (n_samples, n_features), y.shape = (n_samples,)
	clf = LogisticRegression.fit(x_train, y_train)

	clf.score(x_train, y_train) # training accuracy
	clf.intercept_; clf.coef_ # coef_.shape = (1, n_features)
	clf.predict(x_test); clf.predict_proba(x_test)

	from sklearn.metrics import classification_report
	# report on validation data
	classification_report(y_val, logistic_model.predict(x_val))
\end{pyminted}

\begin{pyminted}
	import numpy as np
	# generate data
	n, d = 100, 3
	X = np.random.normal(size=(n, d - 1)); all_ones = np.ones((n, 1))
	X = np.hstack((all_ones, X)) # shape = (n, d)
	y = np.random.binomial(1, 0.2, size=(n, 1)) # shape = (n, 1)

	def sigmoid(beta, X):
		return 1 / (1 + np.exp(- X @ beta)) # shape = (n, 1)

	def log_loss(beta, X, y):
		p = sigmoid(beta, X)
		return -(y.T @ np.log(p) + (1 - y).T @ np.log(1 - p))

	def gradient(beta, X, y):
		p = sigmoid(beta, X)
		return X.T @ (p - y)

	# initialization
	beta_0 = np.random.randn(d, 1) * 0.1 # shape = (d, 1)
	loss_0 = log_loss(beta_0, X, y)

	# gradient descent
	beta_k = beta_0
	beta_prev, loss_prev = beta_0, loss_0
	for k in range(1000):
		beta_k = beta_k - 0.1 * gradient(beta_k, X, y)
		loss_k = log_loss(beta_k, X, y)
		if (abs(loss_k - loss_prev) < 1e-6): break
		beta_prev, loss_prev = beta_k, loss_k
\end{pyminted}

\subsection{SVM}
\textbf{Model:} Minimize $ \frac{1}{2} \norm{\wbm}^2 $ s.t. $ y_i (\xbm_i \wbm + b) \geq 1 $
\begin{itemize}
	\item Linear separable: $ \xbm_i \wbm > 0 $ or $ \xbm_i \wbm < 0 $
	\begin{itemize}
		\item $ y_i \in \{\pm1\} $ with $ \xbm_i \wbm + b \geq 1 $ or $ \leq -1 \Rightarrow y_i (\xbm_i \wbm + b) \geq 1 $
	\end{itemize}
	\item Minimize $ \frac{1}{2} \norm{\wbm}^2 $
	\begin{itemize}
		\item Maximize margin $ = (\xbm_{\text{right}} - \xbm_{\text{left}}) \frac{\wbm}{\norm{\wbm}} = \frac{1-b}{\norm{\wbm}} - \frac{-1-b}{\norm{\wbm}} = \frac{2}{\norm{\wbm}} $
		\item Center line: $ \xbm \wbm + b = 0 $; boundary line: $ \xbm \wbm + b = \pm1 $
		\item Dist($\xbm$, center line) $ = \frac{|\xbm \wbm + b|}{\norm{\wbm}} $
	\end{itemize}
\end{itemize}

\textbf{Algorithm}
\begin{itemize}
	\item Lagrangian: minimize $ L(\wbm, b) = \frac{1}{2} \norm{\wbm}^2 - \sum_i \lambda_i (y_i(\xbm_i \wbm + b) - 1) $
	\begin{itemize}
		\item $ \fracpartial{L}{\wbm} = \zerobm, \fracpartial{L}{b} = 0 \Rightarrow \min_\alphabm \frac{1}{2} \alphabm^T \Qbm \alphabm - \ebm^T \alphabm $ with $ \Qbm_{ij} = y_i y_j k(\xbm_i, \xbm_j) $
		\item Depending only on $ \ip{\xbm_i}{\xbm_j} $
	\end{itemize}
\end{itemize}

\textbf{Soft-margin:} not linear separable data
\begin{itemize}
	\item Minimize $ \frac{1}{2} \norm{\wbm}^2 + C \sum_i \varepsilon_i $ s.t. $ y_i (\xbm_i \wbm + b) \geq 1 - \varepsilon_i $ and $ \varepsilon_i \geq 0 $
	\begin{itemize}
		\item \textbf{Hinge loss:} Minimize $ \frac{1}{2} \norm{\wbm}^2 + C \sum_i \max(0, 1 - y_i (\xbm_i \wbm + b)) $
		\item Larger $ C \Rightarrow $ stronger penalty for violation
	\end{itemize}
	\item Dual: $ \min_\alphabm \frac{1}{2} \alphabm^T \Qbm \alphabm - \ebm^T \alphabm $ s.t. $ \ybm^T \alphabm = 0 $ and $ 0 \leq \alpha_i \leq C $
	\item Prediction: $ \sgn(\sum_{i \in \text{SV}} \ybm_i \alpha_i k(\xbm_i, \xbm^*) + b) $
\end{itemize}

\textbf{Pros:} data (high dimension, small sample)

\textbf{Cons:} data (only boundary matters),
model (linear), fitting (tuning)
\begin{itemize}
	\item Kernel SVM: avoid linear model
\end{itemize}

\begin{pyminted}
	from sklearn.svm import LinearSVC
	svm = LinearSVC(class_weight='balanced') # for imbalanced data
	svm.fit(X_train, y_train)
	print(classification_report(y_val, svm.predict(X_val), target_names=['Good', 'Bad']))
\end{pyminted}

\begin{pyminted}
	from sklearn.svm import SVC
	# kernel = 'rbf', 'ploy', 'sigmoid'
	# C, gamma: regularization parameter
	svm = SVC(kernel = 'rbf', C = 0.1, gamma = 0.1)
	svm.fit(X_train, y_train)
\end{pyminted}

\subsection{Classification Tree}

\textbf{Model:} Recursively partition with the best split
\begin{itemize}
	\item Splitting criteria: minimize \textbf{impurity}
	\begin{itemize}
		\item \textbf{Entropy:} $ H(\text{data}) = -\sum_k p_k \log p_k $, where $ p_k = $ \% of data in class $k$
		\begin{itemize}
			\item Tend to have more small nodes ($\Rightarrow$ use Gain Index)
		\end{itemize}

		\item \textbf{Gini:} $ H(\text{data}) = 1 - \sum_k p_k^2 $
		\begin{itemize}
			\item Easier to compute
		\end{itemize}
		\item \textbf{Misclassification rate:} $ 1 - \max_k p_k $
		\item Regression: $ H(\text{data}) = $ variance
	\end{itemize}
	\item Stopping criteria
	\begin{itemize}
		\item Max depth
		\item Information gain: $ I(\text{before split}) - I(\text{after split}) < $ threshold
		\item Bottom-up pruning
		\begin{itemize}
			\item Reduced Error Pruning (REP): stop if $ \Delta pred > 0 $
			\item Minimum Cost Complexity Pruning (MCCP): $ \frac{\Delta pred}{\Delta comp} $
			\item Minimum Error Pruning (MEP)
		\end{itemize}
		\item Top-down pruning: Pessimistic Error Pruning (PEP),
	\end{itemize}
	\item Algorithm
	\begin{itemize}
		\item \textbf{ID3:} select attribute (A) to maximize \textbf{Information Gain (IG)}
		\begin{itemize}
			\item $ IG(S, A) = H(S) - \sum_{t \in T} p(t) H(t) $, where $ p(t) = $ \% of data in subset $ t $
			\item \textbf{Cons:} local optimal, favor attribute with many values
		\end{itemize}
		\item \textbf{C4.5}
	\end{itemize}
\end{itemize}

\textbf{Pros:} data (allow missing data, auto scaling, no distance metric, high dimension), model (nonlinear, interpretation)

\textbf{Cons:} data (need balance, sensitive to data change), model (linear boundary, overfitting), fitting(no online learning)

\subsubsection{Ensemble Methods}

\textbf{Bagging:} reduce variance
\begin{itemize}
	\item \textbf{Base tree:} many depths $\Rightarrow$ small bias, large variance
	\item \textbf{Random forest:} bootstrap samples and features $\Rightarrow$ increase independence
	\begin{pyminted}
		from sklearn.ensemble import RandomForestClassifier
		rf = RandomForestClassifier(max_depth=2, n_jobs=-1)
		rf.fit(x_train, y_train)
		print(classification_report(y_val, rf.predict(x_val), target_names=['Good', 'Bad']))
	\end{pyminted}
\end{itemize}

\textbf{Boosting:} reduce bias
\begin{itemize}
	\item \textbf{Base tree:} few depths $\Rightarrow$ large bias, small variance
	\item \textbf{Adaboost:} update \textbf{weights} of samples
	\begin{itemize}
		\item Prediction: $ \yhat^{(t)} = \sum_{l=1}^{t} w_l f_l(\xbm) = \yhat^{(t-1)} + w_t f_t(x) $
		\begin{itemize}
			\item Find $ \{w_t, f_t(\cdot)\} = \argmin \sum_i \ell(y_i, \yhat_i^{(t-1)} + w_t f_t(x_i)) $
			\item Update weights of samples based on $ \ell(y_i, \yhat_i) $
			\item Train $ f_t(\cdot) $ and update $ w_t $
		\end{itemize}
	\end{itemize}
	\item \textbf{Gradient boosting:} update \textbf{values} of samples
	\begin{itemize}
		\item Generalize Adaboost by approximately optimizing $ f_t(\cdot) $
		\item Prediction: $ \yhat^{(t)} = \yhat^{(t-1)} + f_t(x) $
	\end{itemize}
	\begin{itemize}
		\item \textbf{XGBoost}
		\item Split with positive gain
		\begin{itemize}
			\item Loss: $ \ell^{(t)} = \sum_i \ell(y_i, \yhat_i^{(t-1)} + f_t(x_i)) + \Omega(f_t) $
			\begin{itemize}
				\item Model: $ f_t(x_i) = \omega_{q(x_i)} $, score at the $ q(x_i) $-th leaf
				\item Complexity: $ \Omega(f) = \gamma J + \frac{1}{2} \lambda \sum_{j=1}^J \omega_j^2 $
			\end{itemize}
			\item Approx: $ \ell(y_i, \yhat_i^{(t)}) \approx \ell(y_i, \yhat_i^{(t-1)}) + g_i f_t(x_i) + \frac{1}{2} h_i f_t^2(x_i) $
			\begin{itemize}
				\item $ G_j = \sum_{q(x_i) = j} g_i $ and $ H_j = \sum_{q(x_i) = j} h_i $
			\end{itemize}
			\item Minimize $ \sum_i g_i \omega_{q(x_i)} + \frac{1}{2} h_i \omega_{q(x_i)}^2 + \gamma J + \frac{1}{2} \lambda \sum_j \omega_j^2 = \sum_j G_j \omega_j + \frac{1}{2} (H_j + \lambda) \omega_j^2 + \lambda J $
			\begin{itemize}
				\item Optimal leaf: $ \omega_j^* = - \frac{G_j}{H_j + \lambda} \Rightarrow \ell^{(t)} = -\frac{1}{2} \sum_j \frac{G_j^2}{H_j + \lambda} + \gamma J $
				\item Gain on split: $ \frac{1}{2} \sq{\frac{G_L^2}{H_L + \lambda} + \frac{G_R^2}{H_R + \lambda} - \frac{(G_L + G_R)^2}{H_L + H_R + \lambda}} - \gamma $
			\end{itemize}
			\item Optional step-size $ \alpha_t $ such that $ \yhat^{(t)} = \yhat^{(t-1)} + \alpha_t f_t(x) $
		\end{itemize}
		\begin{pyminted}
			from sklearn.ensemble import GradientBoostingClassifier
			gbc = GradientBoostingClassifier(n_estimators=50, learning_rate=1.0, max_depth=1, random_state=0)
			gbc.fit(x_train, y_train)
			print(classification_report(y_val, gbc.predict(x_val), target_names=['Good', 'Bad']))
		\end{pyminted}
	\end{itemize}
	\item \textbf{LightGBM}
	\begin{itemize}
		\item Best-first: grow the best leaf leading to imbalanced tree
		\item \textbf{Gradient-based one-side sampling} (GOSS)
	\end{itemize}
	\item \textbf{CatBoost}
	\begin{itemize}
		\item Balanced tree: select best feature and apply to all nodes at current level
		\item \textbf{Minimal variance sampling} (MVS)
	\end{itemize}
\end{itemize}

\textbf{Rotation forest:} trained by first applying PCA on a random subset of features

\newpage

\section{Regression}

\subsection{Summary}

\begin{itemize}
	\item Model-based: OLS, LARS, LOESS
	\item Tree-based: decision tree
\end{itemize}

\subsection{Linear Regression}

\subsubsection{Simple Linear Regression (SLR)}

\textbf{Model:} $ y = \beta_0 + \beta_1 x + \varepsilon $ where $ \varepsilon \sim N(0, \sigma^2) $
\begin{itemize}
	\item $ SST = \sum(y_i - \bar{y})^2, SSR = \sum(\yhat_i - \bar{y})^2, SSE = \sum(y_i - \yhat_i)^2 , SST = SSR + SSE $
	\item df: $ (n-1) = (2-1) + (n-2) $
	\item $ S_{xy} = \sum(x_i-\bar{x})(y_i-\bar{y}) $ and $ S_{xx} =  \sum(x_i-\bar{x})^2 $
\end{itemize}

\textbf{Theory}
\begin{itemize}
	\item $ \hat{\beta}_1 = \frac{S_{xy}}{S_{xx}} $, Distribution: $ \frac{\hat{\beta}_1 - \beta_1}{\sqrt{\sigma^2/S_{xx}}} \sim \Ncal(0,1),  \frac{\hat{\beta}_1 - \beta_1}{\sqrt{MSE/S_{xx}}} \sim t_{n-2} $
	\item $ \hat{\beta}_0 = \bar{y} - b\bar{x} $, Distribution:  $ \frac{\hat{\beta}_0 - \beta_0}{\sqrt{\sigma^2~ \overline{x^2}/S_{xx}}} \sim \Ncal(0,1), \frac{\hat{\beta}_0 - \beta_0}{\sqrt{MSE~\overline{x^2}/S_{xx}}} \sim t_{n-2} $
	\item $ \yhat^*: \frac{\yhat - (\beta_0 + \beta_1 x^*)}{\sqrt{MSE\paren{\frac{1}{n} + \frac{(x^* - \bar{x})^2}{S_{xx}}}}} \sim t_{n-2} $
	\item Residual: $ \hat{\varepsilon}_i = y_i - \yhat_i $
	\begin{itemize}
		\item Constraint: $ \sum \hat{\varepsilon}_i = 0 $ and $ \sum \hat{\varepsilon}_i x_i = 0 $
		\item $ \var(\hat{\varepsilon}_i) = \sigma^2(1 - \Hbm_{i,i}) $ with $ \Hbm = \Xbm(\Xbm^T \Xbm)^{-1} \Xbm^T $
		\begin{itemize}
			\item Variance decreases as $ x_i $ gets farther from $ \bar{x} $ (fitting is better since point are more sensitive to slope) $\Rightarrow$ need \textbf{studentized residual}
		\end{itemize}
		\item Studentized residual: $ t_i = \frac{\hat{\varepsilon}_i}{\hat{\sigma} \sqrt{\Hbm_{i,i}}} \sim \sqrt{\nu} \frac{t}{\sqrt{t^2 + \nu - 1}}$ (tau distribution)
	\end{itemize}
\end{itemize}

\textbf{Testing}
\begin{itemize}
	\item $ H_0: \beta_1 = 0 $; Stat: $ F = \frac{SSR/(2-1)}{SSE/(n-2)} = \frac{MSR}{MSE} \sim F_{1, n-2} $
	\begin{itemize}
		\item Identical: $ H_0: \beta_1 = 0 $, Stat: $ T = \frac{\hat{\beta}_1}{\sqrt{MSE/S_{xx}}} \sim t_{n-2} $
	\end{itemize}
	\item $  H_0: \beta_0 = 0 $, Stat: $ T = \frac{\hat{\beta}_0}{\sqrt{MSE~\overline{x^2}/S_{xx}}} \sim t_{n-2} $
\end{itemize}

\textbf{Coefficient of determination}
\begin{itemize}
	\item $ R^2 = \frac{SSR}{SST} = \frac{S_{xy}^2}{S_{xx}S_{yy}} = \frac{\text{variation between}~\yhat~\andtext ~\bar{y}}{\text{variation between}~y~\andtext ~\bar{y}} = \text{variation explained by line} $
	\begin{itemize}
		\item High $ R^2 $: might overfit
	\end{itemize}
\end{itemize}

\subsubsection{Ordinary Least Squares Regression (OLS)}

\textbf{Model:} $\ybm_{n \times 1} = \Xbm_{n \times p} \betabm_{p \times 1} + \varepsilonbm \sim \Ncal(\Xbm \betabm, \sigma^2 \Ibm)$

\textbf{Assumption}
\begin{itemize}
	\item Linearity
	\item Homescedasticity (constant variance) + normal noise
	\item Independent covariates and observations
	\begin{itemize}
		\item Collinearity $ \Rightarrow $ Variable Inflation Factors (\textbf{VIF}) $ \frac{1}{1 - R_j^2} $ to remove features
	\end{itemize}
\end{itemize}

\textbf{Theory}
\begin{itemize}
	\item $ \betabm = (\Xbm^T \Xbm)^{-1} \Xbm^T \ybm \sim \Ncal(\betabm^*, \sigma^2 (\Xbm^T \Xbm)^{-1}) $
	\begin{itemize}
		\item \textbf{Gauss-Markov:} best linear unbiased estimator (BLUE)
		\item Columns of $ \Xbm $ are linearly independent $ \Rightarrow \Xbm^T \Xbm $ is invertible
		\item Prediction: $ \hat{\ybm} = \Xbm (\Xbm^T \Xbm)^{-1} \Xbm^T \ybm \Rightarrow $ linear combination of $ \ybm $
		\begin{itemize}
			\item $ \hat{\ybm} = \Hbm \Ybm $, where $ \Hbm^2 = \Hbm $ is a project matrix $\Rightarrow$ $ \hat{\ybm} $ is an orthogonal projection of $ \ybm $ onto the space spanned by columns of $ \Xbm $
		\end{itemize}
	\end{itemize}
	\item Sample variance: $ s^2 = \frac{1}{n - p} \norm{\ybm - \hat{\ybm}}^2 $
	\item $ R^2 = 1 - \frac{\norm{\Ybm - \hat{\Ybm}}^2}{\norm{\Ybm - \bar{\Ybm}}^2} = 1 - \frac{\text{variation outside line}}{\text{total variation}} $
	\begin{itemize}
		\item Adjusted-$R^2 = 1 - \frac{(1 - R^2)(n-1)}{n - p - 1}$
	\end{itemize}
\end{itemize}

\textbf{Numerical solution:} $ L(\betabm) = \frac{1}{2} \norm{\ybm - \Xbm \betabm}^2 $
\begin{itemize}
	\item \textbf{Gradient Descent}
	\begin{itemize}
		\item $ \gbm(\betabm) = \Xbm^T (\Xbm \betabm - \ybm) $
	\end{itemize}
	\item \textbf{Coordinate Descent}
	\begin{itemize}
		\item $ g_i(\thetabm) = \Xbm_i^T (\Xbm_i \betabm_i + \Xbm_{-i}\betabm_{-i} - \ybm) = 0 $
		\item $ \beta_i = \frac{\Xbm_i^T(\ybm - \Xbm_{-i} \betabm_{-i})}{\Xbm_i^T \Xbm_i} = \frac{\Xbm_i^T (\ybm - \Xbm_1 \beta_1 - \cdots - \Xbm_p \beta_p + \Xbm_i \beta_i) }{\norm{\Xbm_i}^2} = \frac{\Xbm_i^T (\rbm + \Xbm_i \beta_i) }{\norm{\Xbm_i}^2} $
		\begin{itemize}
			\item $ O(n) $ to update $ \rbm $ $ \Rightarrow $ $ O(n) $ to update $ \beta_i $
			\item OK if only have $ \Xbm^T \Xbm, \Xbm^T \ybm $
			\item Elastic Net: $ \lambda_1  \norm{\betabm}_1 + \lambda_2 \norm{\betabm}^2 \Rightarrow \beta_i = \frac{S_{\lambda_1}(\cdot)}{\norm{\Xbm_i}^2 + \lambda_2} $
		\end{itemize}
	\end{itemize}
\end{itemize}

\textbf{Variant}
\begin{itemize}
	\item \textbf{Generalized least squares (GLS)}
	\begin{itemize}
		\item $ \Cov(\varepsilonbm | \Xbm) = \Sigmabm \Rightarrow \hat{\beta} = (\Xbm^T \Sigmabm^{-1} \Xbm)^{-1} \Xbm^T \Sigmabm^{-1} \ybm $
		\item \textbf{Weighted least squares (WLS):} $ \Sigmabm $ is diagonal, $ \Wbm = \Sigmabm^{-1} $
	\end{itemize}
	\item \textbf{Poisson regression}
	\begin{itemize}
		\item Count data: $ y_i \sim \Poi(\lambda_i) $ and $ \log (\E{y_i | \xbm_i}) = \log \lambda_i = \xbm_i \cdot \betabm $
	\end{itemize}
	\item \textbf{Ordinal regression}
	\begin{itemize}
		\item Ordinal data: $ y_1 \leq \cdots \leq y_n $ and $ \Prob{y \leq i | x} = \frac{1}{1 + \exp(-(\theta_i - \xbm \cdot \beta))} $
		\item OLS: $ y^* = \xbm \cdot \betabm + \varepsilon \Rightarrow y = i $ if $ \theta_{i-1} < y^* \leq \theta_i $
	\end{itemize}
	\item \textbf{Bayesian regression}
	\begin{itemize}
		\item Prior: $ \pi(\betabm, \sigma^2) = \pi(\betabm | \sigma^2) \pi(\sigma^2) = \Ncal(\mubm_0, \sigma^2 \Lambdabm_0^{-1})~\text{Inv-Gamma}(\nu_0/2, \nu_0 s_0^2/2) $
		\item Posterior: $ p(\betabm, \sigma^2 | \ybm, \Xbm) = \Ncal(\mubm_n, \sigma^2 \Lambdabm_n^{-1})~\text{Inv-Gamma}(a_n, b_n) $
		\item Model selection: indicator, spike and slab, adaptive shrinkage, reversible jump MCMC
	\end{itemize}
\end{itemize}

\textbf{Model selection}
\begin{itemize}
	\item \textbf{Mallow's Cp:} $ \frac{1}{n} (\text{SSR} + 2(p + 1) \hat{\sigma}^2) $
	\item \textbf{AIC:} $ -2\log L(\betabm) + 2p $
	\item \textbf{BIC:} $ -2\log L(\betabm) + p \log(n) $
	\item \textbf{Lasso regression}
	\begin{itemize}
		\item $ L(\betabm) = \norm{\ybm - \Xbm \betabm}^2 + \lambda \norm{\betabm}_1 $
		\item Equivalent to Bayesian with Laplace prior $ \pi(\beta) = \frac{\lambda}{2} \exp(-\lambda |\beta_j|) $ and posterior mode
	\end{itemize}
	\item \textbf{Ridge regression}
	\begin{itemize}
		\item $ L(\betabm) = \norm{\ybm - \Xbm \betabm}^2 + \lambda \norm{\betabm}_2^2 $
		\item $ \hat{\betabm}_\lambda = (\Xbm^T \Xbm + \lambda \Ibm)^{-1} \Xbm^T \ybm $ $\Rightarrow$ avoid singular $ \Xbm^T \Xbm $
		\begin{itemize}
			\item $ \E{\hat{\betabm}_\lambda} = (\Xbm^T \Xbm + \lambda \Ibm)^{-1} \Xbm^T \Xbm \betabm $
			\item $ \Var(\hat{\betabm}_\lambda) = \sigma^2 (\Xbm^T \Xbm + \lambda \Ibm)^{-1} \Xbm^T \Xbm (\Xbm^T \Xbm + \lambda \Ibm)^{-1} $
			\item Shrinkage factor: $ \frac{d_j^2}{d_j^2 + \lambda} $ with $ \Xbm = \Ubm \Dbm \Vbm^T $ (SVD)
		\end{itemize}
		\item Equivalent to Bayesian with Gaussian prior $ \beta \sim \Ncal(0, \sigma^2 / \lambda) $
		\item Choose $ \lambda $: leave-one-out CV with candidate $ \lambda $ values
		\item \textbf{Cons:} not scale invariant
	\end{itemize}
	\item \textbf{Elastic Net:} $ L(\betabm) = \norm{\ybm - \Xbm \betabm}^2 + \lambda [(1 - \alpha) \norm{\betabm}_1 + \alpha \norm{\betabm}_2^2] $
\end{itemize}

\subsubsection{Least-Angle Regression (LARS)}

\textbf{Model}
\begin{itemize}
	\item Iteratively pick the most correlated $ \Xbm_j $ with the residual
	\item Gradually update $ \beta_j $ and record residual $ \rbm = \ybm - \hat{\ybm} $
	\item Include a new $ \Xbm_k $ when its correlation with $ \rbm $ is larger than $ \Xbm_j $'s
	\item Gradually update $ (\beta_j, \beta_k) $ and stop until all predictor are included
\end{itemize}

\textbf{Properties}
\begin{itemize}
	\item Pros: fitting (fast, selection path, effective for $ p \gg n  $)
	\item Cons: data (sensitive to noise and collinearity)
\end{itemize}

\subsubsection{Local Regression (LOESS)}

\textbf{Model}: Piecewise SLM with rolling window
\begin{itemize}
	\item Smoothing: more weight on the current point, less weights on bad fitted points
\end{itemize}

\subsection{Regression Tree}

\subsection{Gaussian Process Regression}
\textbf{Model:} $ f(\xbm) = \text{GP}(m(\xbm), k(\xbm, \xbm') + \delta_{ij} \sigma_n^2) $
\begin{itemize}
	\item $ k(\xbm, \xbm') = \sigma_f^2 \exp(-\norm{\xbm - \xbm'}^2 / (2\ell^2)) $
	\item $ \parenmatrix{\ybm\\\ybm*} = \Ncal\parenmatrix{\sqmatrix{\mubm\\\mubm^*},\sqmatrix{\Kbm(\Xbm, \Xbm) + \sigma_n^2 \Ibm & \Kbm(\Xbm, \Xbm^*)\\\Kbm(\Xbm^*, \Xbm) & \Kbm(\Xbm^*, \Xbm^*)}} \Rightarrow \ybm^* | \ybm \sim \Ncal(\cdot, \cdot) $
\end{itemize}

\newpage

\section{Clustering}

\subsection{Summary}

\begin{itemize}
	\item Direct: K-Means, SOM, DBCAN
	\item Distribution-based: GMM
	\item Graph-based: spectral clustering
\end{itemize}

\textbf{Model selection}
\begin{itemize}
	\item \textbf{Elbow:} plot explained variation vs. $k$
	\begin{itemize}
		\item Between-group variance / total variance
		\item Between-group variance / within-group variance (ANOVA F-test)
	\end{itemize}
	\item \textbf{Silhouette coefficient}
	\begin{itemize}
		\item Dist within cluster: $ a_i = \frac{1}{|C_i| - 1} \sum_{j \in C_i, j \neq i} d(i,j) $
		\item Dist with (closest) neighbor cluster: $ b_i = \min_{k \neq i} \frac{1}{|C_k|} \sum_{j \in C_k} d(i,j) $
		\item $ s_i = \frac{b_i - a_i}{\max\{a_i, b_i\}} \in [-1, 1] $ (prefer $a_i \ll b_i \Rightarrow s_i \approx 1$ )
	\end{itemize}
	\item \textbf{Calinkshi-Harabaz index}
	\begin{itemize}
		\item Between/Within-cluster dispersion matrix: $ \Bbm_k, \Wbm_k$
		\item $ s(k) = \frac{\tr(\Bbm_k)}{\tr(\Wbm_k)} \frac{N - k}{k - 1} $
	\end{itemize}
	\item Hierarchical clustering
	\begin{itemize}
		\item Ward linkage: minimize within cluster distance
		\item Average linkage: minimize avg distance between cluster pairs
		\item Complete linkage: minimize maximum distance of between cluster pairs
	\end{itemize}
\end{itemize}

\subsection{K-Means}

\textbf{Goal:} repeatedly find centroids and assign cluster based on centroids

\textbf{Model:} $ \argmin_{\mubm_k} \sum_k \sum_{\xbm \in C_k} \norm{\xbm - \mubm_k}^2 $, where $ \mubm_k = \frac{1}{|C_k|} \sum_{\xbm \in C_k} \xbm $
\begin{itemize}
	\item Compute $ \mubm_i \Rightarrow $ assign cluster $ \Rightarrow $ repeat until converge
\end{itemize}

\textbf{Pros:} fitting (efficient)

\textbf{Cons:} model (distance metric gives need distance spherical clusters), fitting (initialization matters)

\subsection{Self-Organizing Map (SOM)}

\textbf{Goal:} fit NN using unsupervised learning

\textbf{Model}
\begin{itemize}
	\item Initialize Kohonen layer (neurons with Gaussian neighbors)
	\item Connect samples to every neuron with weights: dim(weights) = dim(data)
	\item \textbf{Competition:} pick a sample and select winner neuron based on distance
	\item \textbf{Collaboration:} update neuron weights with neighbors
\end{itemize}

SOM (move weights) is constrained $K$-mean (move centroids)

\subsection{Density-Based Clustering of App with Noise (DBCAN)}

\textbf{Goal:} cluster data in high-density area separated by low-density area

\textbf{Model:} Identify core (enough neighbors), border (reachable by core), and outlier samples

\textbf{Pros:} data (robust to outliers), model (arbitrarily shaped, no $ K $)

\textbf{Cons:} fitting (need radius and neighbor count, data order matters)

\subsection{Gaussian Mixture Model (GMM)}

\textbf{Goal:} fit data to GMM using EM

\textbf{Model:} $ p(\xbm) \sim \sum_k \pi_k \Ncal(\mubm_k, \Sigmabm_k) $
\begin{itemize}
	\item Latent variable: $ p(\xbm_i | z_i = k) \sim \Ncal(\mubm_k, \Sigmabm_k) $ and $ p(z_i = k) = \pi_k $
\end{itemize}

\textbf{EM:} $ w_{ik} \Rightarrow n_k, \pi_k, \mu_k, \Sigmabm_k \Rightarrow $ repeat until converge
\begin{itemize}
	\item \textbf{E-step:} estimate cluster prob $ w_{ik} $
	\begin{itemize}
		\item $ w_{ik} = p(z_i = k | \xbm_i) = \frac{p(z_i = k) p(\xbm_i | z_i = k)}{\sum_k p(z_i = k) p(\xbm_i | z_i = k)} =  \frac{\pi_k \Ncal(\xbm_i; \mubm_k, \Sigmabm_k)}{\sum_k \pi_k \Ncal(\xbm_i; \mubm_k, \Sigmabm_k)}$
	\end{itemize}
	\item \textbf{M-step:} maximize parameters given $ \zbm_i $ and $ \xbm_i $
	\begin{itemize}
		\item Cluster size $ n_k = \sum_i w_{ik} $ and prob $ \pi_k = \frac{n_k}{n} $
		\item Cluster mean $ \mubm_k = \frac{1}{n_k} \sum_{i=1}^{n} w_{ik} \xbm_i $
		\item Cluster variance: $ \Sigmabm_k = \frac{1}{n_k} \sum_{i=1}^{n} w_{ik} (\xbm_i - \mubm_k) (\xbm_i - \mubm_k)^T $
	\end{itemize}
\end{itemize}

\subsection{Spectral Clustering}

\textbf{Goal:} Convert data to graph and min-cut

\textbf{Model:} Eigen-decomp on Laplacian matrix
\begin{itemize}
	\item \textbf{Adjacency matrix} $ \Abm $
	\begin{itemize}
		\item Edge $ A_{ij} = 1 $ if $ \norm{\xbm_i - \xbm_j} < \varepsilon $
		\begin{itemize}
			\item Pros: symmetric, geometrically intuitive
			\item Cons: disconnected components, need $ \varepsilon $
		\end{itemize}
		\item Edge $ A_{ij} = 1 $ if $ i \in $ $n$-nearest neighbors of $j$, or vice versa
		\begin{itemize}
			\item Pros: symmetric, tend to have one connected component
			\item Cons: not geometrically intuitive
		\end{itemize}
	\end{itemize}
	\item \textbf{Weight matrix} $ \Wbm $
	\begin{itemize}
		\item Simple: $ W_{ij} = 1 $ if $ A_{ij} = 1 $
		\item Heat Kernel: $ W_{ij} = e^{-\norm{\xbm_i - \xbm_j}^2 / t} $ if $ A_{ij} = 1 $
	\end{itemize}
	\item \textbf{Laplacian:} $ \Lbm = \Dbm - \Wbm $
	\begin{itemize}
		\item Degree matrix: diagonal, $ D_{ii} = \sum_j W_{ij} $
		\item Eigen-decomp on $ \Lbm = \Vbm \Lambdabm \Vbm^T $
		\begin{itemize}
			\item More edges $ \Rightarrow $ larger $ \lambda_i $; \{\# of connected comp\} $ = $ \{\# of zero $ \lambda_i\}$
			\item \textbf{Spectral gap}: 1st smallest non-zero $ \lambda_i = $ graph density
			\item \textbf{Fiedler value}: 2nd smallest $ \lambda_i \Rightarrow $ partition points
			\item $K$-means cluster on $ \Vbm_{n \times K} $, where each row is a data point
		\end{itemize}
		\item Variants:
		\begin{itemize}
			\item Normalized Laplacian: $ \Lbm^{\text{Norm}} = \Ibm - \Dbm^{-1/2} \Wbm \Dbm^{-1/2} = \Dbm^{-1/2} \Lbm \Dbm^{-1/2} $
			\item Relaxed Laplacian: $ \Lbm_p = \Lbm - \rho \Dbm $
		\end{itemize}
	\end{itemize}
\end{itemize}

\newpage

\section{Dimension Reduction/Subspace Learning}

\subsection{Summary}

\begin{itemize}
	\item Variance-based: PCA
	\item Distance-based: MDS, LLE, auto-encoder, t-SNE
\end{itemize}

\subsection{Independent Component Analysis (ICA)}

\subsection{Principal Component Analysis (PCA)}

\textbf{Model:} find $ \vbm_1 = \argmax_{\norm{\vbm} = 1} \norm{\Xbm_{n \times p} \vbm_{p \times 1}}^2 $, $ \vbm_2 = \argmax_{\norm{\vbm} = 1} \norm{(\Xbm - \Xbm \vbm_1 \vbm_1^T) \vbm}^2 $
\begin{itemize}
	\item \textbf{Samples:} $ \Xbm_{n \times p} $ (0 column mean) and $ \Sigmabm_{p \times p} = \frac{1}{n - 1} \Xbm^T \Xbm $
	\item Eigen-decomp: $ \Sigmabm = \Vbm \Lambdabm \Vbm^T = \sum_{i=1}^p \lambda_i \vbm_i \vbm_i^T $ with $ \vbm_i $ being the $i$-th column
	\item SVD: $ \Xbm_{n \times p} = \Ubm_{n \times n} \Dbm_{n \times p} \Vbm_{p \times p}^T $
	\begin{itemize}
		\item $ \Xbm^T \Xbm = \Vbm_{p \times p} [\Dbm^T \Dbm]_{p \times p} \Vbm_{p \times p}^T $
		\item Eigenpair $ (\vbm_1, d_1^2) \Rightarrow \zbm_1 = \Xbm \vbm_1 $ with $ \var(\zbm_1) = \frac{d_1^2}{n} $ (largest variance)
	\end{itemize}
	\item Dim Reduct: $ \hat{\Xbm}_{n \times K} = \Xbm_{n \times p} \Vbm_{p \times K} $
	\begin{itemize}
		\item Subspace Proj: $ \hat{\Xbm}_{n \times p} = \Xbm_{n \times p} [\Vbm_{p \times K} \Vbm_{K \times p}^T] $
		\item Coordinate: $ \xbm_i = x_{i1} \ebm_1^T + \cdots + x_{ip} \ebm_p^T = (\xbm_i \ebm_1) \ebm_1^T + \cdots + (\xbm_i \ebm_p) \ebm_p^T $
		\item New coordinate: $ \xbm_i = (\xbm_i \vbm_1) \vbm_1^T + \cdots + (\xbm_i \vbm_p) \vbm_p^T $
	\end{itemize}
	\item Total variance: sum of variance of each dim (``re-distributed" by PCA)
\end{itemize}

\textbf{Choose $K$}
\begin{itemize}
	\item Elbow: Percentage of total variance
	\item Choose $ \lambda_i \geq \bar{\lambda} $ (choose $ \lambda_i \geq 1 $ if correlation matrix)
\end{itemize}

\textbf{Variant}
\begin{itemize}
	\item \textbf{Kernel PCA:} $ \xbm_i \to \Phibm(\xbm_i) $
\end{itemize}

\textbf{Application}
\begin{itemize}
	\item Compress image
\end{itemize}

\begin{pyminted}
	import numpy as np
	from sklearn.decomposition import PCA
	X = np.random.normal(size=(n, p))
	pca = PCA(n_components=2)
	pca.fit(X)

	print(pca.explained_variance_ratio_)
	X_new = pca.fit_transform(X)
\end{pyminted}

\subsection{Multidimensional Scaling (MDS)}

\textbf{Model:} preserve distance with low-dim points
\begin{itemize}
	\item Given distance matrix $ \Dbm $, find points such that $ d_{ij} \approx \norm{\xbm_i - \xbm_j} $
	\item Exist unique solution $ \Bbm_{n \times n} = \Xbm_{n \times n} \Xbm_{n \times n}^T = -\frac{1}{2} \Cbm \Dbm^{(2)} \Cbm $
	\item Low-dim points: $ \Xbm_{n \times p} = \Vbm_{n \times p} \Lambdabm_{p \times p}^{1/2} $ via eigen-decomp of $ \Bbm = \Vbm \Lambdabm \Vbm^T $
\end{itemize}

\textbf{Variants}
\begin{itemize}
	\item Sammon's mapping: more weights on small $ d_{ij} $
	\begin{itemize}
		\item Sammon's stress: $ \frac{1}{\sum_{i < j} d_{ij}} \sum_{i < j} \frac{(d_{ij} - \hat{d}_{ij})}{d_{ij}} $
	\end{itemize}
	\item (Non-)Metric MDS: $ d_{ij} \to f(d_{ij}) \approx \norm{\xbm_i - \xbm_j} $
	\item SOM: $ d_{ij} = \norm{\wbm_i - \wbm_j} $ (distance between neuron weights)
	\item Isomap: update $ \Dbm $ via shortest path
\end{itemize}

\subsection{LLE (Locally-Linear Embedding)}
\textbf{Model:} construct points by neighbors
\begin{itemize}
	\item Reconstruction error: $ \Wbm = \argmin \sum_i \norm{\xbm_i - \sum_j W_{ij} \xbm_j}^2 $
	\item Low-dim points: $ \hat{\xbm}_{1:n} = \argmin \sum_i \norm{\hat{\xbm}_i - \sum_j W_{ij} \hat{\xbm}_j}^2 $
\end{itemize}

\subsection{Auto-encoder}

\textbf{Model:} minimize $ \norm{\xbm_i - \hat{\xbm}_i}^2 $ where $ \hat{\xbm}_i = (\psi \circ \phi) (\xbm_i) $
\begin{itemize}
	\item Encoder $ \psi: \Rbb^p \to \Rbb^d $ and Decoder $ \phi: \Rbb^d \to \Rbb^p $
\end{itemize}

\subsection{t-SNE (t-dist Stochastic Neighbor Embedding)}

\textbf{Model:} preserve similarities with low-dim points
\begin{itemize}
	\item Similarity prob: $ p_{j | i} = \frac{\exp(-\norm{\xbm_j - \xbm_i})^2 / (2\sigma_i^2)}{\sum_{k \neq i} \exp(-\norm{\xbm_k - \xbm_i})^2 / (2\sigma_i^2)} $ and $ p_{ij} = (p_{j|i} + p_{i|j}) / 2 $
	\item Low-dim similarity prob: $ q_{ij} = \frac{(1 + \norm{\ybm_i - \ybm_j}^2)^{-1}}{\sum_k \sum_{l \neq k} (1 + \norm{\ybm_k - \ybm_l}^2)^{-1}} $
	\item Minimize KL divergence: $ KL(P || Q) = \sum_{i \neq j} p_{ij} \log \frac{p_{ij}}{q_{ij}} $ via gradient descent
\end{itemize}

\textbf{Cons:} $ p_{ij} $ are all similar in high-dim

\section{Rankings}

\subsection{Comparison}

Goal: compare ranked lists

\textbf{Kendall's Correlation}
\begin{itemize}
	\item Assumption: lists contain identical elements, but different rankings
	\item Count matching and un-matching pairs
	\item \textbf{Cons:} identical elements, unweighted
\end{itemize}

\textbf{Rank Biased Overlap (RBO)}
\begin{itemize}
	\item $ (1 - p) \sum p^{\text{depth}-1} \frac{ \text{count of matchings}}{\text{depth}} $
\end{itemize}

\newpage
\section{Neural Network}
\subsection{Preliminares}
\subsubsection{Loss}
\begin{itemize}
	\item Squared loss: $ \ell(y, \yhat) = \frac{1}{2}(y - \yhat)^2 $
	\item Hinge loss: $ \ell(y, \yhat) = (1 - y \yhat)_+ $ with $ y \in \{\pm1\} $
	\begin{itemize}
		\item SVM: $ \yhat = \wbm \xbm + b $
		\item Multi-class SVM: $ \ell(y, \yhat) = (1 + \max_{\yhat \neq y} \wbm_{\yhat} \xbm - \wbm_y \xbm)_+ $
	\end{itemize}
	\item Ranking loss
	\item Cross-entropy loss (classification): $ \ell(y, \yhat) = -[y \log \yhat + (1-y) \log(1-\yhat)] $
\end{itemize}

\subsubsection{Activation}
\begin{itemize}
	\item Sigmoid: $ \sigma(z) = \frac{1}{1 + e^{-z}} \in (0,1) $
	\begin{itemize}
		\item Used for output layer in binary classification
		\item Gradient: $ \sigma'(z) = \sigma(z) (1 - \sigma(z)) \in (0, 1/4] $
		\item \textbf{Cons:} vanishing gradient for large $ |z| $
	\end{itemize}
	\item Tanh: $ \sigma(z) = \tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}} \in (-1, 1) $
	\begin{itemize}
		\item $ \sigma'(z) = 1 - \sigma(z)^2 \in (0, 1) $ (faster than sigmoid)
		\item \textbf{Cons:} vanishing gradient for large $ |z| $
	\end{itemize}
	\item ReLU: $ \sigma(z) = \max(0, z) \in [0, \infty) $
	\begin{itemize}
		\item \textbf{Pros:} fast, no vanishing gradient, unbounded
		\item \textbf{Cons:} dead neuron ($\Rightarrow$ leaky ReLU $ \sigma(z) = \max(\varepsilon z, z) $), non-smooth ($\Rightarrow$ GELU)
	\end{itemize}
	\item SoftPlus: $ \sigma(z) = \log(1 + e^z) \in (0, \infty) $
	\item Softmax: $ \sigma(z)_i = \frac{e^{z_i}}{\sum_j e^{z_j}} \in [0,1] $
	\begin{itemize}
		\item Used for output layer to get class probabilities
	\end{itemize}
\end{itemize}

\subsubsection{Special Module}
\begin{itemize}
	\item Skip connection (ResNet)
	\begin{itemize}
		\item Avoid vanishing gradient, mitigate degradation
	\end{itemize}
\end{itemize}

\subsubsection{Fitting}
\textbf{Initialization}
\begin{itemize}
	\item All 0: bad
	\item $ \text{randn}(n) / \sqrt(n) $
	\item Sparse initialization
	\item All 0.01
\end{itemize}

\textbf{Improve generalization}
\begin{itemize}
	\item Normalization
	\begin{itemize}
		\item \textbf{Local Response Normalization}
		\begin{itemize}
			\item CNN: normalize within or across feature maps
		\end{itemize}
		\item \textbf{Batch Normalization}
		\begin{itemize}
			\item Normalize neuron input with $ \mu_B $ and $ \sigma_B $ over a mini-batch
			\item $ x_i \leftarrow \gamma \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \varepsilon}} + \beta $
		\end{itemize}
		\item \textbf{Weight Normalization}
	\end{itemize}
\end{itemize}

\textbf{Regularization}
\begin{itemize}
	\item $L1$, $L2$
	\item Dropout
	\begin{itemize}
		\item Training: drop neurons, update weights and scale by dropout rate (0.5)
		\item Prediction: use all nodes
	\end{itemize}
\end{itemize}

\textbf{Hyper-Parameter}
\begin{itemize}
	\item Manual, random, grid, Bayesian
	\item Model-based (Gaussian process, tree-based model)
\end{itemize}

\subsection{Feedforward Neural Network (NN)}

\textbf{Implementation}
\begin{pyminted}
	from tensorflow.keras.models import Sequential
	from tensorflow.keras.layers import InputLayer, Dense, BatchNormalization
	from tensorflow.keras.callbacks import ModelCheckpoint

	shallow_nn = Sequential()
	shallow_nn.add(InputLayer((x_train.shape[1],)))
	shallow_nn.add(Dense(2, 'relu'))
	shallow_nn.add(BatchNormalization())
	shallow_nn.add(Dense(1, 'sigmoid'))

	checkpoint = ModelCheckpoint('shallow_nn', save_best_only=True)
	shallow_nn.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
	shallow_nn.summary()

	shallow_nn.fit(x_train, y_train, validation_data=(x_valid, y_valid), epochs=5, callbacks=checkpoint)
\end{pyminted}

\textbf{Build from scratch}
\begin{pyminted}
	def init_nn(n_inputs, n_hidden, n_outputs):
		network = list()
		hidden_layer = list()
		for i in range(n_hidden):
			hidden_layer.append({'weights' : [random() for i in range(n_inputs + 1)]})
\end{pyminted}

\subsection{Deep Neural Network (DNN)}

\subsection{Convolution Neural Network (CNN)}

\textbf{Architecture}
\begin{itemize}
	\item Convolution layer (CONV): filter size $F$, stride $S$ $ \to $ output $O$ (feature/activation map)
	\item Pooling (POOL)
	\begin{itemize}
		\item Max pooling: preserve detected features, most common
		\item Average pooling: down samples feature map, used in LeNet
	\end{itemize}
	\item Fully connected (FC)
\end{itemize}

\textbf{Classic net}
\begin{itemize}
	\item LeNet-5, AlexNet, Conv3/4/5, FC6/7/8,
	\item ZFNet, VGGNet, GoogLeNet
	\item Inception
\end{itemize}

\subsection{Recurrent Neural Network (RNN)}

\textbf{Gates}
\begin{itemize}
	\item Input gates: write to cell or not
	\item Forget gates: erase a cell or not
	\item Gate: how much to write to cell
	\item Output gate: how much to reveal cell
\end{itemize}

\textbf{Issues}
\begin{itemize}
	\item Vanishing/Exploding gradient
\end{itemize}

Bag of words $ \Rightarrow $ RNN $ \Rightarrow $ LTSM $ \Rightarrow $ Transformer

\textbf{Tricks:}
\begin{itemize}
	\item Grad clipping
	\item LSTM self-loops
	\item Gate
	\item Delay
\end{itemize}

\subsection{Long Short-Term Memory (LSTM)}

\subsection{Generative Adversarial Network (GAN)}

\newpage

\section{Reinforcement Learning}
\textbf{Terminology}
\begin{itemize}
	\item State: $ s_i $
	\item Action: $ a \in \Acal(s_i) $
	\item Transition prob $ s \overset{a}{\to} s' $: $ T(s, a, s') $, , s.t. $ \sum_{s'} T(s, a, s') = 1 $
	\item Rewards $ s \overset{a}{\to} s' $: $ R(s, a, s') $
	\item Discount factor: $ \gamma $
	\item Policy: $ \pi: s \mapsto a $
	\item Utility: $ u(s_0, \dots, s_k) = r_1 + \gamma r_2 + \dots + \gamma^{k-1} r_k = \sum_i \gamma^{i-1} r_i $
\end{itemize}

\textbf{$Q$-value}
\begin{itemize}
	\item $ Q_\pi(s_0, a) = \sum_{s'} T(s, a, s') [R(s, a, s') + \gamma V_\pi(s')] $
	\item Policy value: $ V_\pi(s) = Q_\pi(s, \pi(s)) $
\end{itemize}

\textbf{Policy evaluation}
\begin{itemize}
	\item Initialization: $ V_\pi^{(0)}(s) \leftarrow 0 $
	\item Iteration
	\begin{itemize}
		\item $ Q_\pi^{(t-1)}(s, \pi(s)) = \sum_{s'} T(s, \pi(s), s') [R(s, \pi(s), s') + \gamma V_\pi^{(t-1)}(s')] $
		\item For all $s$, $ V_\pi^{(t)}(s) \leftarrow Q_\pi^{(t-1)}(s, \pi(s)) $
	\end{itemize}
	\item Optimization
	\begin{itemize}
		\item Optimal $Q$-value: $ Q_{opt}(s_0, a) = \sum_{s'} T(s, a, s') [R(s, a, s') + \gamma V_{opt}(s')] $
		\item Optimal value: $ V_{opt}(s) = \max_a Q_{opt}(s, a) $
		\item Optimal policy: for all $s$, $ \pi_{opt}(s) = \argmax_a Q_{opt}(s, a) $
	\end{itemize}
\end{itemize}

\textbf{State-action-reward-state-action} (SARSA)
\begin{itemize}
	\item $ \hat{Q}_\pi(s, a) \leftarrow (1 - \eta) \hat{Q}_\pi(s, a) + \eta [r + \gamma \hat{Q}_\pi(s', a')] $
\end{itemize}

\textbf{Q-learning}
\begin{itemize}
	\item $ \hat{Q}_{opt}(s, a) \leftarrow (1 - \eta) \hat{Q}_{opt}(s, a) + \eta [r + \gamma \max_{a'} \hat{Q}_{opt}(s', a')] $
\end{itemize}

\textbf{$\epsilon$-greedy}
\begin{itemize}
	\item $ \pi_{act}(s) =
	\begin{cases}
		\argmax_a \hat{Q}_{opt}(s, a) & w.p. ~ 1 - \epsilon\\
		\text{random}~a & w.p. ~ \epsilon
	\end{cases} $
\end{itemize}

\newpage
\section{Other}

\subsection{Apriori}
\textbf{Data:} unlabeled data with categorical features (data: feature1, feature2, ...)

\textbf{Goal:} select most frequent set of features

\textbf{Algorithm}
\begin{itemize}
	\item Count frequency of each feature, eliminate ones below threshold
	\item Generate all (feature A, feature B) from remaining features, count frequency and eliminate ones below threshold
	\item Generate all (feature A, feature B, feature B) from ...
\end{itemize}

\subsection{Recommendation System }
\subsubsection{Collaborative filtering}
\textbf{Goal:} Model similar users together

\textbf{Algorithm}
\begin{itemize}
	\item KNN
	\item Item-item-based
	\begin{itemize}
		\item Model-building stage: similarity scores between users (correlation, cosine)
		\item Recommendation stage: recommend unrated items (highly rated by others)
	\end{itemize}
	\item Memory-based
	\begin{itemize}
		\item
	\end{itemize}
	\item User-based algorithm
	\item Kernel Mapping Recommender
\end{itemize}

\textbf{Problem}
\begin{itemize}
	\item Cold start: no enough data to make good recommendations
	\begin{itemize}
		\item Multi-armed bandit algorithm
	\end{itemize}
	\item Scalability: large computation power for millions of users and items
	\item Sparsity: too many items where most are unrated
\end{itemize}

\textbf{Pros:} data (no need to understand item)

\textbf{Cons:} data (need large data to model user)

\subsubsection{title}

\textbf{Content-based filtering}
\begin{itemize}
	\item Model content and re-weight features based on user
\end{itemize}

\textbf{Concepts}
\begin{itemize}
	\item
	\item
	\item Knowledge-based system
\end{itemize}

\textbf{Methods and challenges}

\textbf{Implementation}

\textbf{Research}

\section{Machine Learning System Design}

\textbf{Project flow}
\begin{itemize}
	\item Project setup
	\begin{itemize}
		\item Goals: maximize revenue
		\item User experience: when and how the predictions are used (offline or real-time)
		\item Performance constraints: accuracy vs. speed, FP vs. FN
		\item Evaluation
		\begin{itemize}
			\item Training: loss function
			\item Inferencing: users' reaction
		\end{itemize}
		\item Personalization: individual vs. whole model
		\item Project constraints: time, computing power, systems
	\end{itemize}
	\item Data pipeline
	\begin{itemize}
		\item Data availability and collection: kind, size, annotation
		\item User data: what, how to collect, feedback
		\item Storage: where, data structures, update frequency
		\item Preprocessing: feature engineering, normalization, missing values
		\item Challenges
		\item Privacy: anonymizing, store data in server or user device
		\item Biases: correct biases
	\end{itemize}
	\item Modeling and training
	\begin{itemize}
		\item Model selection
		\begin{itemize}
			\item Baseline: random baseline, human baseline, simple heuristic
		\end{itemize}
		\item Training: start from simple, overfitting a single batch, set random seed
		\item Scaling: SSGD, ASGD
	\end{itemize}
	\item Serving
\end{itemize}

\hspace{1in}

\textbf{Clarify Requirements}
\begin{itemize}
	\item Data size: simply model for small dateset
	\item Computing power constraint
	\item Speed vs. accuracy
\end{itemize}

\textbf{Metric}
\begin{itemize}
	\item Offline metric
	\begin{itemize}
		\item Dateset: train, valid, test
		\item Metric: AUC, $F1$, $R^2$, MSE
	\end{itemize}
	\item Online metric
	\begin{itemize}
		\item Metric: click-through rate, browsing time
	\end{itemize}
	\item Non-functional metric
	\begin{itemize}
		\item Training speed and scalability
		\item Extensibility: new techniques
		\item Easy to implement
	\end{itemize}
\end{itemize}

\textbf{Architecture Design}
\begin{itemize}
	\item Data
	\begin{itemize}
		\item Identify target variable and how to collect
		\item Identify features
		\item Feature engineering
		\begin{itemize}
			\item Train/valid/test split
			\item Missing data/outliers
			\item Data re-balancing: under/over-sample, SMOTE
			\item Date normalization
			\item Storage: Amazon S3, cloud, MySQL (database), FEAST (feature), DVC (version)
		\end{itemize}
	\end{itemize}
	\item Model
	\item Serving
\end{itemize}
\end{document}


\documentclass[12pt]{article}

\title{CSC 455 - Assignment 2}
\author{Robert Krency}
\date{\today}

\usepackage{tikz}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{tabularx}
\usepackage{multicol}
\usepackage{algpseudocode}
\usepackage{algorithm}
\usepackage{setspace}

% Geometry
\usepackage{geometry}
\geometry{letterpaper, left=20mm, top=20mm, right=20mm, bottom=20mm}

% Add vertical spacing to tables
\renewcommand{\arraystretch}{1.4}

\onehalfspacing

% Macros
\newcommand{\definition}[1]{\underline{\textbf{#1}}}

\newenvironment{rcases}
  {\left.\begin{aligned}}
  {\end{aligned}\right\rbrace}

% Begin Document
\begin{document}

\maketitle

Using the following grammer, identify which of the sentences is valid.

\begin{flalign*}
    <S> & \to\ <A> a <B> b &&\\
    <A> & \to\ <A> b\ \vert\ b &&\\
    <B> & \to\ <B> a\ \vert\ a &&
\end{flalign*}

\begin{enumerate}
    \item baab
    \item bbbab
    \item bbaaaaa
    \item bbaab
\end{enumerate}

\subsection*{Solution:}

The two valid sentences are ``baab'' and ``bbaab''.

The first grammar rule indicates that any valid sentence must end with `b'.
The second grammar rule recursively indicates any valid sentence must begin with a series of `b', with length $\geq 1$.
The third grammer rule recursively indicates any valid sentence must contain a series of `a' between the `b', with length $\geq 1$ immediately preceded by an `a'.

The sentence ``bbbab'' breaks the third rule, as there are not enough `a'.

The sentence ``bbaaaaa'' breaks the first rule, as there is no `b' at the end.

\end{document}


\chapter{Background \& Review}
Everything mentioned in this chapter should already be familiar to you from other math classes.
These topics span two major areas: algebra/pre-calculus and limits.
Ideas from these topics will often be used implicitly or without a passing reference. \\

If you are unfamiliar with anything mentioned, you can use man of the great online resources like Khan Academy, to familiarize yourself before moving forward.

\input{./backgroundReview/algebraPreCalc/algebraPreCalc.tex} % Algebra and Pre-Calc


\subsection{Knowledge Areas (in terms of application)}

\begin{itemize}
    \item Robotics
    \item Automation
    \item Design (mechanical, electrical)
    \item Control
    \item Data analysis
    \item Computational geometry
\end{itemize}

\subsection{Course Delivery}

The lectures are given every week, followed by practical sessions (labs). During the sessions, students are required to develop their own code based on the knowledge acquired during lectures and the self-study.

\subsection{Prerequisite courses}

\begin{itemize}
    \item Strong prerequisites: Linear Algebra.
    \item Weak prerequisites: Calculus, Control Theory.
    \item Required background knowledge: Python (alternatively Matlab or any other language suitable to work with linear algebra-heavy problems)
\end{itemize}

\subsection{Expected Learning outcomes}

The course will provide an opportunity for participants to:
\begin{itemize}
    \item Understand and learn to use Convex Programming (CP): Quadratic programming, Linear Programming, SOCP, SDP (LMI), as well as Mixed-integer CP.
    \item Learn how to use convex optimization solvers, especially CVX.
    \item Being able to implement optimization on practical problems, especially in robotics, control, computational geometry.
\end{itemize}

\subsection{Expected acquired core competencies}

\begin{itemize}
    \item Developing efficient linear algebra-based code for control and robotics applications.
    \item Developing solutions using numeric optimization, especially convex optimization.
    \item Optimization-based methods in Robotics.
\end{itemize}

\subsection{Reference Materials}
\begin{itemize}
    \item Annotated slides
    \item Online materials
    \item Educational videos
\end{itemize}

\subsection{Computer Resources}
Students will need to run computer experiments on a laptop and/or on lab computers.

\subsection{Laboratory Exercises}
There are a series of labs and electronic handouts prepared for the course.

\subsection{Laboratory Resources}
Students will be required to use and modify a software tool written in Python which run on multiple platforms (Linux, Microsoft Windows, and Mac OS). The tool requires freely available software libraries.

\subsection{Cooperation Policy and Quotations}
We encourage intensive discussion and collaboration in this class. You should feel free to discuss all aspects of the class with classmates and work with them to complete your assignments and project report. However, if you are working together, you must provide details of your contribution and that of others.


\documentclass[12pt]{article}
\usepackage[noanswer]{exercise}
\usepackage[]{amsthm}
\usepackage[]{amsmath}
\usepackage[]{enumitem}
\usepackage[]{hyperref}

\newtheorem{theorem}{Theorem}
\newtheorem{lemma}{Lemma}
\newtheorem{proposition}{Proposition}
\newtheorem{corollary}{Corollary}

\theoremstyle{definition}
\newtheorem{definition}{Definition}

\let\set\mathbf

% I don't like how the "exercise" package normally lays things out.
% Feel free to copy this if you agree.
\renewcommand{\ExerciseHeader}{\noindent\textbf{\ExerciseName\ \ExerciseHeaderNB}\ \ExerciseHeaderTitle\ \ExerciseHeaderOrigin\smallskip}
\renewcommand{\ExerciseHeaderOrigin}{(\ExerciseOrigin)}
\renewcommand{\AnswerHeader}{\bigskip\noindent\textbf{Solution \ExerciseHeaderNB}\newline}

\title{Workshop Notes}
\author{Robert Dougherty-Bliss}
\date{3 February 2021}

\begin{document}

\maketitle

% Adding the * gets rid of the section numbering.
\section*{Logistics}%
\label{sec:logistics}

Nothing to say here.

\section*{New stuff}%
\label{sec:new_stuff}

Since I last saw you, you learned about cardinality and sequences. Sequences
will get properly confusing when we get to some theorems, but for now I imagine
thatt cardinality is very weird for everyone. Let's remind ourselves of what's
going on.

One way to measure a set is to just count all the things in it;
\begin{equation*}
    |\{a, b, c\}| = 3.
\end{equation*}
Easy. Another way is to ``line up'' one set with another. For example:
\begin{align*}
    &\{a, b, c\} \\
    &\{1, 2, 3\}
\end{align*}
These sets are the same size because we can line them up. If we agree that
$\{1, 2, 3\}$ has size $3$, then $\{a, b, c\}$ has size $3$ as well.

This second way seems like a really stupid way to measure sets. Until you learn
about infinite sets. You can't ``count'' infinite sets. It doesn't make any
sense. You could say that every infinite set has size $\infty$, but you're
going to miss a \emph{weird} thing that happens: infinite sets can have
different sizes if you measure using the second way.

Formally, we say that two sets $A$ and $B$ have the same cardinality if there
exists a bijection between them (a way to line them up). That is, there's a
function $f \colon A \to B$ such that:
\begin{enumerate}
    \item For all $b \in B$, there is some $a \in A$ such that $f(a) = b$ (surjectivity); and
    \item If $f(a_1) = f(a_2)$, then $a_1 = a_2$ (injectivity).
\end{enumerate}

For example, $\{1, 2, 3, \dots\}$ has the same size as $\{2, 3, 4, \dots\}$,
because we can define
\begin{equation*}
    f(n) = n + 1,
\end{equation*}
which is a bijection. (Exercise!)

More interestingly, $\{1, 2, 3, \dots \}$ has the same size as $\{2, 4, 6, 8,
\dots\}$. (Define $f(n) = 2n$.) That is, the even natural numbers have the same
size as all the natural numbers themselves! The odds \emph{also} have the same
size as the natural numbers. (Define $f(n) = 2n + 1$.) If we say that
$\aleph_0$ is the ``size'' of $\{1, 2, 3, \dots\}$ then we have an equation
like
\begin{equation*}
    \aleph_0 + \aleph_0 = \aleph_0.
\end{equation*}
That's weird. It's not so weird if we think of $\aleph_0$ as $\infty$, but
there are \emph{other} infinities! Famously, $\mathbf{R}$ is
\emph{uncountable}, in the sense that there is no bijection from $\mathbf{N}$
to $\mathbf{R}$. (See Abbott.) But clearly $\mathbf{R}$ is bigger than
$\mathbf{N}$\dots\ this is all very weird.

I won't say anything more---that's what Abbott is for---but I'm sure that you
have lots of questions about cardinality and homework. Fire away!

% Hidden secrets.

% I want everyone to have their own questions. If they don't, then I'm going to
% walk us through a few exercises.

% Algebraic numbers are countable.

% 1.5.5.
% 1.5.6. Hard!
% 1.5.8. Very hard!

% If lim_n (a(n) + b(n)) = 0, does lim_n a(n) = - lim_n b(n)?

\subsection*{Promises}%
\label{sub:promises}

I forgot to prove something for you that I promised I would. To make it up to
you, I'll show you something fancier to whet your appetite for things to come.
(I won't talk about this at all during the workshop. This is just a bonus read
if you're interested.)

\begin{theorem}
    If $S$ is a nonempty, bounded set of integers, then $\sup S$ is contained
    in $S$.
\end{theorem}

This is actually a special case of a more general result. First, I need to
define a new object.

\begin{definition}
    A point $x$ is a \emph{limit point} of a set $S$ if every
    $\epsilon$-neighborhood contains a point of $S$ not equal to $x$.
\end{definition}

For example, $0$ is a limit point of $\{1, 1 / 2, 1 / 3, 1 / 4, \dots\}$.

A nice fact about sups and infs is that they're either in the underlying set or
limit points of it.

\begin{proposition}
    If $\sup S \notin S$, then $\sup S$ is a limit point of $S$.
\end{proposition}

\begin{proof}
    For every $\epsilon > 0$, we can find some $s \in S$ such that $\sup S -
    \epsilon < s < \sup S$, the last inequality being strict since $\sup S
    \notin S$. Thus every $\epsilon$-neighborhood of $\sup S$ contains a point
    of $S$ not equal to $\sup S$, so $\sup S$ is a limit point of $S$.
\end{proof}

\begin{corollary}
    Let $S$ be a nonempty set, bounded from above, without limit points. Then
    $\sup S \in S$.
\end{corollary}

\begin{proof}
    Since $S$ has no limit points, we must have $\sup S \in S$ by the previous
    proposition.
\end{proof}

Now Theorem~1 is easy!

\begin{proof}[Proof of Theorem 1]
    If $S$ is a set of integers, then $S$ has no limit points. (Every element
    is at least $1$ away from every other element.) Thus, if $\sup S$ exists,
    $\sup S \in S$ by the previous corollary.
\end{proof}

All of this is the application of general \emph{metric space theory} to
$\set{R}$. It turns out that a lot of arguments we make about sequences, sets,
and so on, generalize to other situations without any modifications. If this
sounds interesting, go look up ``metric spaces.''

\end{document}


\section{Data}
\label{sec:data}
Throughout this project we make use of three datasets for housing price prediction publicly available from Kaggle competitions. We choose them under assumption that testing the GBDT models on various datasets with distinctive properties is important for reasoning about their performance in housing price prediction area.

The first dataset we utilize is a classic but tiny Boston dataset \cite{boston1978housing} collected in 70's. It encloses 333 instances described by 13 features, only one of which is categorical in itself. Despite the modest size this dataset can still be employed as a rapid and reasonable initial step towards understanding the models' performance in regression problems.

We also adopt the Ames housing dataset \cite{de2011ames} incorporating 1460 observations and 80 explanatory variables involved in assessing home price: 43 categorical and 37 ordinal and continuous. It describes the sale of individual residential property in Ames, Iowa from 2006 to 2010 and covers much greater information diversity than the Boston dataset.

The third dataset is provided by Russian bank Sberbank \cite{sberbank2017housing} and not only includes housing specifications but also comprises macroeconomic patterns. It contains 12231 training instances each of which represented by 293 parameters with 18 categorical among them. This dataset is the most diverse among all the considered datasets which imposes a significant challenge to the models.

Comparative analysis shows that the Ames housing dataset offers the greatest variety of categorical features which are of the most interest taking into account specific CatBoost treatment towards them. The Sberbank dataset seems to be the most appealing in the sense of being true to life as it is mostly collected by the real estate companies which, on the other hand, makes it more noisy, and therefore, more demanding for model training.


% Copyright (C) 1992, Digital Equipment Corporation
% All rights reserved.
% See the file COPYRIGHT for a full description.
% Last modified on Tue Jan 25 16:45:08 PST 1994 by mhb

\documentstyle{article}
\begin{document}

%\documentstyle[hmwk]{handout}
%\hmwknumber{5}
%\topic{{\bf Transformations and Clipping}}
%\duedate{Tuesday, 19 November 1991}
%\demodate{Thursday, 21 November 1991}
%\begin{document}
%\maketitle

\section{Overview}

In this assignment you will learn about 3-D viewing and clipping.
You'll implement both a 3-D viewing package and a 3-D clipping package
and use these packages to display an animation of a rotating cube.
You'll need to be familiar with 3D matrix transformations (Chapter
5 in FvDFH) and with orthographic and perspective 3D viewing (parts
of Chapter 6 in FvDFH).

\section{Implementation}

Your program will be demonstrated with a simple animation: a cube
rotating about its main diagonal. Allow the user to choose {\em
orthographic\/} or {\em perspective\/} projections, and to select
the number of degrees of rotation per frame. The skeleton program
we provide shows how to use SRGP for achieving smooth animation (of
a spinning triangle) by double buffering. You may start with our
code, or build a more ergonomic interface using SUIT.

For the orthographic projection, you don't need to do any clipping
and you can assume that the viewer is looking straight on. However,
you may be happier with the animation --- and it won't be hard ---
if you first rotate the cube $-30\deg$ about the $x$ axis and
$30\deg$ about the $y$ axis.

For the perspective projection, you need to let the user set a viewing
position and clipping planes. However, you do not need to give the
user all the degrees of freedom that are discussed in the book. In
particular, you do not need to support oblique screen views, so no
shearing is needed. Also, you can define the location of the cube yourself,
so you can clip in 3D rather than in homogeneous coordinates. The
3D clipping is a straightforward extension of the 2D clipping algorithm
discussed the first week of class. Be sure that a user of your program
can select perspective views that adequately demonstrate the effects
of clipping to all six sides of the viewing pyramid.

The cube should be represented on the screen by a line-drawing of
its twelve edges.  In addition, draw an additional edge along the
cube's main diagonal, since this edge will remain invariant under
the correct rotation.  Try to come up with an efficient data structure
for storing the cube's set of vertices and edges, so that a minimum
number of operations will be required to rotate the cube.  Of course,
your implementation should concatenate the rotation matrices before
any transformations take place, so that each vertex may be rotated
with a single matrix multiply.

There is no test data for this program.

\section{Suggestion}

This assignment is not very long, but it can be tricky. Get started
right away (but keep in mind that we won't be talking about the
mathematics underlying perspective transformations until next Tuesday.)
There are basically four parts to the assignment: the driver, the
orthographic projection, the basic perspective projection, and the
perspective projection with clipping. I suggest that you implement
the four parts in this order.

\section{What to Hand In}

Turn in the usual.

\section{Bells \& Whistles}

This is for those of you who wish to explore material beyond the
normal assignment. No extra credit will be given.  Be sure you have
completed all the normal parts of the homework before attempting
bells and whistles.

Implement a more general system that can perform arbitrary affine
transformations on a more general class of line-drawn objects.  Your
program should read a script containing a sequence of commands of
the form
\begin{verbatim}
    Translate dx, dy, dz
    Scale sx, sy, sz
    Rotate {A,B,C}{X,Y,Z}, angle
      % rotation axis is the line from {A,B,C} to {X,Y,Z}
\end{verbatim}
and use the script to create an animation. The program should
also be capable of reading a description of an object (or set of
objects) from a file.  These bells \& whistles should not be difficult
to implement:  you already have the tools at your disposal for
performing all the transformations; you just need to write a more
sophisticated driver.

If you are feeling really ambitious, you may want to try implementing
a 4-D viewing and clipping package and display a rotating hypercube
in addition to the ordinary cube.

On the other hand, if you are feeling unambitious (but want to do
something with your copious free time), try rotating line drawings
other than a cube. Block letters of your name, for instance.

Finally, if spinning block letter don't excite you, try eliminating
the hidden lines. Another easy hack with some sex appeal is to
eliminate the hidden faces by painting each face with a unique opaque
color. The crux of the matter is to figure out the order in which
to paint the faces.

\end{document}