\newif\ifshowsolutions
\showsolutionstrue
\input{../preamble}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% HEADER
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\chead{
  {\vbox{
      \vspace{2mm}
      \large
      Machine Learning \& Data Mining \hfill
      Caltech CS/CNS/EE 155 \hfill \\[1pt]
      Set 2\hfill
      January $13^\text{th}$, 2021 \\
    }
  }
}

\begin{document}
\pagestyle{fancy}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% POLICIES
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Policies}
\begin{itemize}
  \item Due 9 PM, January $20^\text{th}$, via Moodle.
  \item You are free to collaborate on all of the problems, subject to the collaboration policy stated in the syllabus.
  \item In this course, we will be using Google Colab for code submissions. You will need a Google account.
  \item We ask that you use Python 3 (set that as your Colab runtime's Python version) and sklearn version 0.22 (should be the default version for Python 3 in Colab) for your code, and that you comment your code such that the TAs can follow along and run it without any issues.
\end{itemize}

\section*{Submission Instructions}
\begin{itemize}
   \item You are highly encouraged to use the submission template: \url{https://github.com/lakigigar/Caltech-CS155-2021/blob/main/psets/set2/set2template.tex}
   \item Submit your report as a single .pdf file to Gradescope (entry code N8XV6Z), under "Set 2 Report".
   \item In the report, \textbf{include any images generated by your code} along with your answers to the questions.
   \item Submit your code by \textbf{sharing a link in your report} to your Google Colab notebook for each problem (see naming instructions below). Make sure to set sharing permissions to at least "Anyone with the link can view". \textbf{Links that can not be run by TAs will not be counted as turned in.} Check your links in an incognito window before submitting to be sure.
   \item For instructions specifically pertaining to the Gradescope submission process, see \url{https://www.gradescope.com/get_started#student-submission}.

\end{itemize}

\section*{Google Colab Instructions}

For each notebook in the course gitub \url{https://github.com/lakigigar/Caltech-CS155-2021}, you need to save a copy to your drive.

\begin{enumerate}
	\item Open the github preview of the notebook, and click the icon to open the colab preview.
	\item On the colab preview, go to File $\rightarrow$ Save a copy in Drive.
	\item Edit your file name to “lastname_firstname_originaltitle”, e.g.”cui_alexander_set2-prob3.ipynb”
\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% PROBLEM 1
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\newpage
\section{Comparing Different Loss Functions [30 Points]}
\materials{lecture 3}

We've discussed three loss functions for linear models so far:
\begin{itemize}
\item Squared loss: $L_\text{squared} = (1 - y\mathbf{w}^T\mathbf{x})^2$
\item Hinge loss: $L_\text{hinge} = \max(0, 1 - y\mathbf{w}^T\mathbf{x})$
\item Log loss: $L_\text{log} = \ln(1 + e^{-y\mathbf{w}^T\mathbf{x}})$
\end{itemize}
where $\mathbf{w} \in \mathbb{R}^n$ is a vector of the model parameters, $y \in \{-1,1\}$ is the class label for datapoint $\mathbf{x} \in \mathbb{R}^n$, and we're including a bias term in $\mathbf{x}$ and $\mathbf{w}$.  The model classifies points according to $\text{sign}(\mathbf{w}^T\mathbf{x})$.

Performing gradient descent on any of these loss functions will train a model to classify more points correctly, but the choice of loss function has a significant impact on the model that is learned.

\problem[3]
Squared loss is often a terrible choice of loss function to train on for classification problems.  Why?

\begin{solution}
\end{solution}

\problem[9]
A dataset is included with your problem set: \texttt{problem1data1.txt}. The first two columns represent $x_1, x_2$, and the last column represents the label, $y \in \{-1,+1\}$.

On this dataset, train both a logistic regression model and a ridge regression model to classify the points.  (In other words, on each dataset, train one linear classifier using $L_\text{log}$ as the loss, and another linear classifier using $L_\text{squared}$ as the loss.) For this problem, you should use the logistic regression and ridge regression implementations provided within scikit-learn
(\href{http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html}{logistic regression documentation})
(\href{http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html}{Ridge regression documentation})
instead of your own implementations. Use the default parameters for these classifiers except for setting the regularization parameters so that very little regularization is applied.

For each loss function/model, plot the data points as a scatter plot and overlay them with the decision boundary defined by the weights of the trained linear classifier.  Include both plots in your submission. The template notebook for this problem contains a helper function for producing plots given a trained classifier.

What differences do you see in the decision boundaries learned using the different loss functions? Provide a qualitative explanation for this behavior.

\begin{solution}
\end{solution}

\problem[9]
Leaving squared loss behind, let's focus on log loss and hinge loss. Consider the set of points $S = \{(\frac{1}{2}, 3), (2, -2), (-3, 1)\}$ in 2D space, shown below, with labels $(1, 1, -1)$ respectively.

Given a linear model with weights $w_0 = 0, w_1 = 1, w_2 = 0$ (where $w_0$ corresponds to the bias term), compute the gradients $\nabla_{w}L_{\text{hinge}}$ and $\nabla_{w}L_{\text{log}}$ of the hinge loss and log loss, and calculate their values for each point in S.

\begin{center}
  \includegraphics[width=.8\textwidth]{./plots/SimpleDatasetWithDecisionBoundary.png}
\end{center}
\begin{small}
  The example dataset and decision boundary described above. Positive instances are
  represented by red x's, while negative instances appear as blue dots.
\end{small}

\begin{solution}
\end{solution}

\problem[4]
Compare the gradients resulting from log loss to those resulting from hinge loss. When (if ever) will these gradients converge to 0? For a linearly separable dataset, is there any way to reduce or altogether eliminate training error without changing the decision boundary?

\begin{solution}
\end{solution}

\problem[5]
Based on your answer to the previous question, explain why for an SVM to be a ``maximum margin'' classifier, its learning objective must not be to minimize just $L_\text{hinge}$, but to minimize $L_\text{hinge} + \lambda\Vert w \Vert^2$ for some $\lambda > 0$.

(You don't need to prove that minimizing $L_\text{hinge} + \lambda\Vert w \Vert^2$ results in a maximum margin classifier; just show that the additional penalty term addresses the issues of minimizing just $L_\text{hinge}$.)

\begin{solution}

\end{solution}

\newpage
\section{Effects of Regularization}
\textit{Relevant materials: Lecture  4}

For this problem, you are required to implement everything yourself and submit code.
\indent\problem[4] % indent for consistency
In order to prevent over-fitting in the least-squares linear regression problem, we add a regularization penalty term.
Can adding the penalty term decrease the training (in-sample) error?
Will adding a penalty term always decrease the out-of-sample errors?
Please justify your answers. Think about the case when there is over-fitting while training the model.

\begin{solution}
\end{solution}

\problem[4]
$\ell_1$ regularization is sometimes favored over $\ell_2$ regularization due to its ability to generate a sparse $w$ (more zero weights).
In fact, $\ell_0$ regularization (using $\ell_0$ norm instead of $\ell_1$ or $\ell_2$ norm) can generate an even sparser $w$, which seems favorable in high-dimensional problems.
However, it is rarely used.  Why?

\begin{solution}
\end{solution}

\subsection{Implementation of \texorpdfstring{$\ell_2$}{L2} regularization:}

We are going to experiment with regression for the Red Wine Quality Rating data set. The data set is uploaded on the course website, and you can read more about it here: \url{https://archive.ics.uci.edu/ml/datasets/Wine}. The data relates 13 different factors (last 13 columns) to wine type (the first column). Each column of data represents a different factor, and they are all continuous features. Note that the original data set has three classes, but one was removed to make this a binary classification problem.

Download the data for training and testing.
There are two training sets, wine\_training1.txt (100 data points) and wine\_training2.txt (a proper subset of wine\_training1.txt containing only 40 data points), and one test set, wine\_testing.txt (30 data points). You will use the wine\_testing.txt dataset to evaluate your models.

We will train a \emph{$\ell_2$-regularized logistic regression} model on this data. Recall that the unregularized logistic error (a.k.a. log loss) is
\[E = -\sum_{i=1}^{N}\log(p(y_i | \mathbf{x}_i))\]

where $p(y_i = -1 | \mathbf{x}_i)$ is
\[\frac{1}{1 + e^{\mathbf{w}^T\mathbf{x}_i}}\]

and $p(y_i = 1 | \mathbf{x}_i)$ is
\[\frac{1}{1 + e^{-\mathbf{w}^T\mathbf{x}_i}},\]
where as usual we assume that all $\mathbf{x}_i$ contain a bias term.
The $\ell_2$-regularized logistic error is
\begin{align*}
  E &= -\sum_{i=1}^{N}\log(p(y_i | \mathbf{x}_i)) + \lambda\mathbf{w}^T\mathbf{w} \\
    &= -\sum_{i=1}^{N} \log \left( \frac{1}{1 + e^{-y_i\mathbf{w}^T\mathbf{x}_i}} \right) + \lambda\mathbf{w}^T\mathbf{w} \\
    &= -\sum_{i=1}^{N} \left(\log \left( \frac{1}{1 + e^{-y_i\mathbf{w}^T\mathbf{x}_i}} \right) - \frac{\lambda}{N}\mathbf{w}^T\mathbf{w}\right).
\end{align*}
Implement SGD to train a model that minimizes the $\ell_2$-regularized logistic error, i.e. train an $\ell_2$-regularized logistic regression model.  Train the model with 15 different values of $\lambda$ starting with $\lambda_0 = 0.00001$ and increasing by a factor of 5, i.e. $$\lambda_0 = 0.00001, \lambda_1 = 0.00005, \lambda_2 = 0.00025, ..., \lambda_{14} = 61,035.15625.$$
\par

Some important notes: Terminate the SGD process after 20,000 epochs, where each epoch performs one SGD iteration for each point in the training dataset. You should shuffle the order of the points before each epoch such that you go through the points in a random order (hint: use \texttt{numpy.random.permutation}). Use a learning rate of $5e-4$ (note: this number is in E notation), and initialize your weights to small random numbers.

You are highly encouraged to check that your loss and update functions pass the provided unit tests first. The SGD takes quite a while to run, and it is difficult to debug if your loss or update functions are incorrect.

You may run into numerical instability issues (overflow or underflow). One way to deal with these issues is by normalizing the input data $X$, so you are required to normalize the input data in this assignment. Given the column for the $j$th feature, $X_{:,j}$, you can normalize it by setting $X_{ij} = \frac{X_{ij} - \overline{X_{:,j}} }{\sigma(X_{:,j})}$ where $\sigma(X_{:,j})$ is the standard deviation of the $j$th column's entries, and $\overline{X_{:,j}}$ is the mean of the $j$th column's entries. Normalization may change the optimal choice of $\lambda$; the $\lambda$ range given above corresponds to data that has been normalized in this manner. When you normalize the test data, you should normalize it using the mean and standard deviation of the training set. If you treat the input data differently, simply plot enough choices of $\lambda$ to see any trends.

\problem[16]
Do the following for both training data sets (wine\_training1.txt and wine\_training2.txt) and attach your plots in the homework submission (use a log-scale on the horizontal axis):

\subproblem Plot the average training error ($E_\text{in}$, averaged over the set of datapoints after model convergence on the training set) versus different $\lambda$s.

\subproblem Plot the average test error ($E_\text{out}$, averaged over the set of datapoints after model convergence on the training set) versus different $\lambda$s using wine\_testing.txt as the test set.

\subproblem Plot the $\ell_2$ norm of $\mathbf{w}$ versus different $\lambda$s. \medskip

 You should end up with three plots, with two series (one for wine\_training1.txt and one for wine\_training2.txt) on each plot. Note that the $E_\text{in}$ and $E_\text{out}$ values you plot should not include the regularization penalty --- the penalty is only included when performing gradient descent.

\begin{solution}

\end{solution}

\problem[4]
Given that the data in wine\_training2.txt is a subset of the data in wine\_training1.txt, compare errors (training and test) resulting from training with wine\_training1.txt (100 data points) versus wine\_training2.txt (40 data points). Briefly explain the differences.

\begin{solution}
\end{solution}

\problem[4]
Briefly explain the qualitative behavior (i.e. over-fitting and under-fitting) of the training and test errors with different $\lambda$s while training with data in wine\_training1.txt.

\begin{solution}

\end{solution}

\problem[4]
Briefly explain the qualitative behavior of the $\ell_2$ norm of $\textbf{w}$ with different $\lambda$s while training with the data in wine\_training1.txt.

\begin{solution}

\end{solution}

\problem[4]
If the model were trained with wine\_training2.txt, which $\lambda$ would you choose to train your final model? Why?

\begin{solution}

\end{solution}

% Question 2
\newpage
\section{Lasso (\texorpdfstring{$\ell_1$}{L1}) vs. Ridge (\texorpdfstring{$\ell_2$}{L2}) Regularization}
\textit{Relevant materials: Lecture 3}

For this problem, you may use the scikit-learn (or other Python package) implementation of Lasso and Ridge regression --- you don't have to code it yourself.

The two most commonly-used regularized regression models are Lasso ($\ell_1$) regression and Ridge ($\ell_2$) regression.
Although both enforce ``simplicity'' in the models they learn, only Lasso regression results in sparse weight vectors.
This problem compares the effect of the two methods on the learned model parameters.

\problem[12] % Specific problem (linear regresion - P(w|D) gaussion), run lasso and ridge, and plot parameters vs lambda (linear scale for lambda is more interpretable)
The tab-delimited file problem3data.txt on the course website contains 1000 9-dimensional datapoints.  The first 9 columns contain $x_1,\ldots,x_9$, and the last column contains the target value $y$.

\subproblem
Train a linear regression model on the problem3data.txt data with Lasso regularization for regularization strengths $\alpha$ in the vector given by \texttt{numpy.linspace(0.01, 3, 30)}.
On a single plot, plot each of the model weights $w_1, ..., w_9$ (ignore the bias/intercept) as a function of $\alpha$.

\subproblem
Repeat \textbf{i.} with Ridge regression, and this time using regularization strengths $\alpha \in \{1, 2, 3, \ldots, 1e4\}$.

\subproblem
As the regularization parameter increases, what happens to the number of model weights that are exactly zero with Lasso regression?
What happens to the number of model weights that are exactly zero with Ridge regression?

\medskip
\lstset{
  basicstyle=\small\ttfamily,
  breaklines=true,
  columns=fullflexible
}

\begin{solution}
\end{solution}

\problem[9]

\subproblem
Given a dataset containing $N$ datapoints each with $d$ features, solve for
\[\underset{\mathbf{w}}{\argmin} \Vert\mathbf{y} - \mathbf{X}\mathbf{w}\Vert^2 + \lambda\Vert\mathbf{w}\Vert_2^2
\]
where $\mathbf{X} \in \mathbb{R}^{N \times d}$ is the matrix of datapoints and $\mathbf{y} \in \mathbb{R}^N$ is the  vector of all output values for these datapoints. Do so for arbitrary $d$ and $\lambda \geq 0$.

This is linear regression with Ridge regularization.

\begin{subsolution}
\end{subsolution}

\subproblem In this question, we consider Ridge regularization in 1-dimension. Suppose that when $\lambda = 0$, $w \neq 0$. Does there exist a value for $\lambda > 0$ such that $w = 0$? If so, what is the smallest such value?

\begin{subsolution}
\end{subsolution}

\problem[9] (Extra Credit)

\subproblem
In the case of 1-dimensional data, Lasso regression admits a closed-form solution.
Given a dataset containing $N$ datapoints each with $d$ features, where $d = 1$, solve for
\[\underset{w}{\argmin} \Vert\mathbf{y} - \mathbf{x}w\Vert^2 + \lambda\Vert w\Vert_1,
\]
where $\mathbf{x} \in \mathbb{R}^{N}$ is the vector of datapoints and $\mathbf{y} \in \mathbb{R}^N$ is the  vector of all output values corresponding to these datapoints. Just consider the case where $d = 1$, $\lambda \geq 0$, and the weight $w$ is a scalar.

This is linear regression with Lasso regularization.

\begin{subsolution}
\end{subsolution}

\subproblem
In this question, we continue to consider Lasso regularization in 1-dimension. Now, suppose that when $\lambda = 0$, $w \neq 0$. Does there exist a value for $\lambda$ such that $w = 0$? If so, what is the smallest such value?

\begin{subsolution}
\end{subsolution}

\end{document}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End:


\documentclass{article}\usepackage[]{graphicx}\usepackage[]{color}
%% maxwidth is the original width if it is less than linewidth
%% otherwise use linewidth (to make sure the graphics do not exceed the margin)
\makeatletter
\def\maxwidth{ %
  \ifdim\Gin@nat@width>\linewidth
    \linewidth
  \else
    \Gin@nat@width
  \fi
}
\makeatother

\definecolor{fgcolor}{rgb}{0.345, 0.345, 0.345}
\newcommand{\hlnum}[1]{\textcolor[rgb]{0.686,0.059,0.569}{#1}}%
\newcommand{\hlstr}[1]{\textcolor[rgb]{0.192,0.494,0.8}{#1}}%
\newcommand{\hlcom}[1]{\textcolor[rgb]{0.678,0.584,0.686}{\textit{#1}}}%
\newcommand{\hlopt}[1]{\textcolor[rgb]{0,0,0}{#1}}%
\newcommand{\hlstd}[1]{\textcolor[rgb]{0.345,0.345,0.345}{#1}}%
\newcommand{\hlkwa}[1]{\textcolor[rgb]{0.161,0.373,0.58}{\textbf{#1}}}%
\newcommand{\hlkwb}[1]{\textcolor[rgb]{0.69,0.353,0.396}{#1}}%
\newcommand{\hlkwc}[1]{\textcolor[rgb]{0.333,0.667,0.333}{#1}}%
\newcommand{\hlkwd}[1]{\textcolor[rgb]{0.737,0.353,0.396}{\textbf{#1}}}%

\usepackage{framed}
\makeatletter
\newenvironment{kframe}{%
 \def\at@end@of@kframe{}%
 \ifinner\ifhmode%
  \def\at@end@of@kframe{\end{minipage}}%
  \begin{minipage}{\columnwidth}%
 \fi\fi%
 \def\FrameCommand##1{\hskip\@totalleftmargin \hskip-\fboxsep
 \colorbox{shadecolor}{##1}\hskip-\fboxsep
     % There is no \\@totalrightmargin, so:
     \hskip-\linewidth \hskip-\@totalleftmargin \hskip\columnwidth}%
 \MakeFramed {\advance\hsize-\width
   \@totalleftmargin\z@ \linewidth\hsize
   \@setminipage}}%
 {\par\unskip\endMakeFramed%
 \at@end@of@kframe}
\makeatother

\definecolor{shadecolor}{rgb}{.97, .97, .97}
\definecolor{messagecolor}{rgb}{0, 0, 0}
\definecolor{warningcolor}{rgb}{1, 0, 1}
\definecolor{errorcolor}{rgb}{1, 0, 0}
\newenvironment{knitrout}{}{} % an empty environment to be redefined in TeX

\usepackage{alltt}

% latex packages
\usepackage{graphicx}
\usepackage[margin=0.75in]{geometry}
\usepackage[backend=bibtex]{biblatex}
\addbibresource{refs.bib}

% title information
\title{Simple Regression}
\author{Gaston Sanchez}
\IfFileExists{upquote.sty}{\usepackage{upquote}}{}
\begin{document}
\maketitle

\begin{abstract}
We analyzed the relationship between height and weight of some inviduals from a galaxy far, far away.
\end{abstract}

\section{Introduction}
This is just a minimal example of a simple regression analysis in R. The goal is to practice creating a reproducible report using several tools such as Makefile, R, knitr, Rnw files, plots, etc.

\section{Data}
We curated a data set from the wookieepedia. Current descriptions in the website do not longer contain information about weight.

% latex table generated in R 3.2.3 by xtable 1.8-2 package
% Mon Mar 14 07:54:48 2016
\begin{table}[ht]
\centering
\begin{tabular}{rlrr}
  \hline
 & name & height & weight \\
  \hline
1 & Anakin Skywalker & 188 &  84 \\
  2 & Padme Amidala & 165 &  45 \\
  3 & Luke Skywalker & 172 &  77 \\
  4 & Leia Skywalker & 150 &  49 \\
  5 & Obi-Wan Kenobi & 182 &  77 \\
  6 & Han Solo & 180 &  80 \\
  7 & R2-D2 &  96 &  32 \\
  8 & C-3PO & 167 &  75 \\
  9 & Yoda &  66 &  17 \\
  10 & Chewbacca & 228 & 112 \\
   \hline
\end{tabular}
\end{table}

There are 10 individuals, with an average height of 159.4 cms, and an average weight of 64.8 kgs. The summary statistics are displayed in the table below:
% latex table generated in R 3.2.3 by xtable 1.8-2 package
% Mon Mar 14 07:54:48 2016
\begin{table}[ht]
\centering
\begin{tabular}{rll}
  \hline
 &     height &     weight \\
  \hline
1 & Min.   : 66.0   & Min.   : 17.00   \\
  2 & 1st Qu.:153.8   & 1st Qu.: 46.00   \\
  3 & Median :169.5   & Median : 76.00   \\
  4 & Mean   :159.4   & Mean   : 64.80   \\
  5 & 3rd Qu.:181.5   & 3rd Qu.: 79.25   \\
  6 & Max.   :228.0   & Max.   :112.00   \\
   \hline
\end{tabular}
\end{table}

\subsection{EDA}
We looked at the distributions of height and weight. As suggested by \cite{ductavis}, this relation is almost perfectly linear among human Jedis.

\begin{figure}[h!]
\includegraphics[width=1\textwidth]{../images/histograms.pdf}
\end{figure}

There is a strong linear relationship between height and weight. This is evidenced in the scatterplot, and calculation the correlation 0.9412865.

\begin{figure}[h!]
    \centering
	\includegraphics[width=0.9\textwidth]{../images/scatterplot.pdf}
\end{figure}

\section{Conclusions}
Our analysis agrees with Yoda's seminal article \cite{yoda}. But more work needs to be carried out like collecting additional samples in future surveys. May the Force be with you.

\printbibliography
\end{document}


\documentclass{article}
\usepackage{amsmath}
\usepackage{graphicx}

\begin{document}

\section{This is a test}

\begin{figure}
	\includegraphics[width=0.4\textwidth]{test.png}
	\caption{The LaTeX logo}
	\label{fig:latex}
\end{figure}

The above figure (\ref{fig:latex}) shows the LaTeX logo.

You can see an example of some reference math (equation \ref{eq:test}) below.

\begin{equation}\label{eq:test}
	\int \cos x \rm d x = \sin x + k
\end{equation}

This code even handles citations correctly. Here's an example, \cite{Lilley:2009zz}.

Now, let's look at Tables:

\begin{table}
	\centering
	\begin{tabular}{c|c|c}
		a & b & c \\
		d & e & f \\
		g & h & i \\
	\end{tabular}
	\caption{What have we here? A table!}\label{tbl:test}
\end{table}

This table is referenced here: \ref{tbl:test}.

\end{document}