%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%     Declarations (skip to Begin Document, line 101, for parts you fill in)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\documentclass[10pt]{article}

\usepackage{geometry}  % Lots of layout options.  See http://en.wikibooks.org/wiki/LaTeX/Page_Layout
\geometry{letterpaper}  % ... or a4paper or a5paper or ...
\usepackage{fullpage}  % somewhat standardized smaller margins (around an inch)
\usepackage{setspace}  % control line spacing in latex documents
\usepackage[parfill]{parskip}  % Activate to begin paragraphs with an empty line rather than an indent

\usepackage{url}

\usepackage{amsmath,amssymb}  % latex math
\usepackage{empheq} % http://www.ctan.org/pkg/empheq
\usepackage{bm,upgreek}  % allows you to write bold greek letters (upper & lower case)

% allows strikethroughs in math via \cancel{math text goes here}
\usepackage{cancel}

% provides adjustwidth environment to temporarily change margins
% \begin{adjustwidth}{left change length}{\right change length}
\usepackage{changepage}

% for typsetting algorithm pseudocode see http://en.wikibooks.org/wiki/LaTeX/Algorithms_and_Pseudocode
\usepackage{algorithmic,algorithm}

\usepackage{graphicx}  % inclusion of graphics; see: http://en.wikibooks.org/wiki/LaTeX/Importing_Graphics
% allow easy inclusion of .tif, .png graphics
\DeclareGraphicsRule{.tif}{png}{.png}{`convert #1 `dirname #1`/`basename #1 .tif`.png}

\usepackage{subfigure}  % allows subfigures in figure
%\usepackage{caption}
%\usepackage{subcaption}

\usepackage{xspace}
\newcommand{\latex}{\LaTeX\xspace}

\usepackage{color}  % http://en.wikibooks.org/wiki/LaTeX/Colors

\long\def\ans#1{{\color{blue}{\em #1}}}
\long\def\ansnem#1{{\color{blue}#1}}
\long\def\boldred#1{{\color{red}{\bf #1}}}
\long\def\boldred#1{\textcolor{red}{\bf #1}}
\long\def\boldblue#1{\textcolor{blue}{\bf #1}}
\long\def\todo#1{\textcolor{red}{\bf TODO: #1}}

% misc math defs
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}

% Useful package for syntax highlighting of specific code (such as python) -- see below
\usepackage{listings}  % http://en.wikibooks.org/wiki/LaTeX/Packages/Listings
\usepackage{textcomp}

%%% The following lines set up using the listings package
\renewcommand{\lstlistlistingname}{Code Listings}
\renewcommand{\lstlistingname}{Code Listing}

%%% Specific for python listings
\definecolor{gray}{gray}{0.5}
\definecolor{green}{rgb}{0,0.5,0}

\lstnewenvironment{python}[1][]{
\lstset{
language=python,
basicstyle=\footnotesize,  % could also use this -- a little larger \ttfamily\small\setstretch{1},
stringstyle=\color{red},
showstringspaces=false,
alsoletter={1234567890},
otherkeywords={\ , \}, \{},
keywordstyle=\color{blue},
emph={access,and,break,class,continue,def,del,elif ,else,%
except,exec,finally,for,from,global,if,import,in,i s,%
lambda,not,or,pass,print,raise,return,try,while},
emphstyle=\color{black}\bfseries,
emph={[2]True, False, None, self},
emphstyle=[2]\color{green},
emph={[3]from, import, as},
emphstyle=[3]\color{blue},
upquote=true,
morecomment=[s]{"""}{"""},
commentstyle=\color{gray}\slshape,
emph={[4]1, 2, 3, 4, 5, 6, 7, 8, 9, 0},
emphstyle=[4]\color{blue},
literate=*{:}{{\textcolor{blue}:}}{1}%
{=}{{\textcolor{blue}=}}{1}%
{-}{{\textcolor{blue}-}}{1}%
{+}{{\textcolor{blue}+}}{1}%
{*}{{\textcolor{blue}*}}{1}%
{!}{{\textcolor{blue}!}}{1}%
{(}{{\textcolor{blue}(}}{1}%
{)}{{\textcolor{blue})}}{1}%
{[}{{\textcolor{blue}[}}{1}%
{]}{{\textcolor{blue}]}}{1}%
{<}{{\textcolor{blue}<}}{1}%
{>}{{\textcolor{blue}>}}{1},%
%framexleftmargin=1mm, framextopmargin=1mm, frame=shadowbox, rulesepcolor=\color{blue},#1
framexleftmargin=1mm, framextopmargin=1mm, frame=single,#1
}}{}
%%% End python code listing definitions

\DeclareMathOperator{\diag}{diag}
\DeclareMathOperator{\cov}{cov}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%     Begin Document
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{document}

\begin{center}
    {\Large {\bf ISTA 421 / INFO 521 - Homework 1}} \\
    \boldred{Due: Friday, August 31, 5pm} \\
    24 points total (same for undergraduates and graduates) \\
    \vspace{1cm}
\end{center}

\begin{flushright}
STUDENT NAME  %% Fill in your name here

Undergraduate / Graduate %% select which you are!
\end{flushright}

\vspace{1cm}

{\Large {\bf Instructions}}

On this (and other ``regular'' -- i.e., non-midterm/final assignments) you may {\em discuss} problems with others in the class, BUT your code and written solutions \boldred{MUST BE YOUR OWN}.  Do not copy code or written solutions by others.  Copying is cheating.

{\bf If you work with others, you must include in your submission a list of people you talked with.  If you reference other material (fine to do!), you must cite those resources.}

Include in your final submission the pdf of written answers along with separate files for any python scripts that you write in support of answering the questions.
You are required to create either a .zip or tarball (.tar.gz / .tgz) archive of all of the files for your submission and submit the archive to the D2L dropbox by the date/time deadline above.
Refer to the course submission instructions at: \\
\url{http://w3.sista.arizona.edu/~clayton/courses/ml/submissions.html}

In future problem sets, there will be additional or different problems that graduate students must complete, but in this case, everyone must do the same problems for the same points.

Mathematical content: This is a mathematical subject and there is a wide variance in backgrounds of students who take this course. For example, there may be problems in the assignments which seem more difficult than they really are simply because you are not (yet) used to the kind of problem.

The purpose of this assignment is to learn (or remember) how to write short programs to explore mathematical ideas. It was written on the assumption that you will use Python for your assignments (recommended). If you wish do some of the tasks in a different language you may need to adjust the questions a little bit, but what goes into the PDF (e.g., plots, answers) should be the same.

Most of the problems below have multiple parts.  In most cases, what is being asked for is clearly indicated by the explicit question(s) or request(s) (e.g., exercise 5 clearly asks you to show two different equations between scalar and vector notation hold).  Some exercises (e.g., exercise 6) have a flavor of a ``tutorial''.  For these problems, I add a (\$) for each part of the exercise that requires an explicit answer.  Each (\$) is worth an equal proportion of the total points for that exercise; e.g., exercise 6 is worth 5 points total and there are 10 (\$)s, so each is worth 0.5 points.

In the following, FCMA refers to the course text: Simon Rogers and Mark Girolami (2012), {\em A First Course in Machine Learning}, first edition.  (All questions from the book are reproduced here in full, so you do not need the book to proceed.)

For general notes on using latex to typeset math, see: {\tt http://en.wikibooks.org/wiki/LaTeX/Mathematics}

{\bf Always start assignments ASAP, do {\em NOT} wait until the last couple of days.}

\vspace{.5cm}

%%%%%%%%%%%%%%%%
%%%     Problems
%%%%%%%%%%%%%%%%

\newpage
\begin{enumerate}

%%%   Problem 1
\item \label{prob:1} [0 points]
If you haven't already, setup your programming environment!  Python is highly recommended:\\
\url{http://w3.sista.arizona.edu/~clayton/courses/ml/python_setup.html}

If python is new to you, work through the short python tutorial:\\
\url{http://w3.sista.arizona.edu/~clayton/courses/ml/tutorial.html}

See the Appendix at the end of these instructions for additional information about numpy, arrays and matrices.

%%%   Problem 2
\item \label{prob:2} [1 point]
{\bf Exercise 1.1} from FCMA p.35

\begin{figure}[htb]
\begin{center}
\includegraphics[width=6cm]{figures/figure1-1_p2}
\caption{Reproduction of figure 1.1, Olympic men's 100m data}
\end{center}
\end{figure}
By examining Figure 1.1 [from p. 2 of FCMA, reproduced here], estimate (by hand / in your head) the kind of values we should expect for $w_0$ (y-intercept) and $w_1$ (slope) as parameters of a line fit to the data (e.g., High? Low?  Positive?  Negative?).  (No computer or calculator calculation is needed here -- just estimate!)

{\bf Solution.} $<$Solution goes here$>$\\

\newpage
%%% Note for next three problems:
NOTE: The following three exercises (3, 4 and 5) review basic linear algebra concepts.
% \begin{enumerate}

\item[] Notation conventions:
\begin{itemize}
\item Script variables, such as $x_{n2}$ and $w_1$ represent scalar values
\item Lowercase bold-face variables, such as $\mathbf{x}_n$ and $\mathbf{w}$, represent vectors
\item Uppercase bold-face variables, such as $\mathbf{X}$, represent $n$ (rows) $\times ~m$ (columns) matrices
\item Note that because all indexes in the following are either a single digit integers (0, 1, ..., 9), or a single letter representing an integer index, e.g., $n$, I am representing multiple dimension indexes without a comma, as it is unambiguous; e.g., $x_{32}$ is the element scalar value of $\mathbf{X}$ at row 3, column 2.  When we have to refer to specific index values greater than 9, we'll use commas, such as $x_{32,3}$ is the scalar value in the 32nd row and 3rd column.
\item `$\top$' in expressions like $\mathbf{w}^\top$ indicates the {\em transpose} operator.
\item Unless stated otherwise, we will assume that all vectors {\em without} the transpose, $\top$, are {\em column} vectors, so $\mathbf{w}^\top$ is a {\em row vector}.
\item It is sometimes convenient to express an example vector as a bracketed list of elements (e.g., in a sentence): $[x_1, x_2, ..., x_n]$.  In general I am going to try to be careful about the orientation of vectors, so the previous example would be a {\em row} vector.  To make it a column vector, I'll add a transpose: $[x_1, x_2, ..., x_n]^\top$.
\end{itemize}

%%%   Problem 3
\item \label{prob:3} [2 points]
{\bf Exercise 1.3} from FCMA p.35

Show that:
\begin{eqnarray*}
\mathbf{w}^\top\mathbf{X}^\top\mathbf{X}\mathbf{w} = w_0^2 \left( \sum_{n=1}^N x_{n1}^2 \right) + 2w_0w_1 \left( \sum_{n=1}^N x_{n1}x_{n2} \right) + w_1^2 \left( \sum_{n=1}^N x_{n2}^2 \right),
\end{eqnarray*}
where
\begin{eqnarray*}
\mathbf{w} =
    \begin{bmatrix}
    w_0 \\[0.3em]
    w_1
    \end{bmatrix}
    ,
\mathbf{X} =
    \begin{bmatrix}
    x_{11} & x_{12} \\[0.3em]
    x_{21} & x_{22} \\[0.3em]
    x_{31} & x_{32} \\[0.3em]
    \vdots & \vdots \\[0.3em]
    x_{N1} & x_{N2}
    \end{bmatrix}
    .
\end{eqnarray*}
(Hint -- it's probably easiest to do the $\mathbf{X}^\top\mathbf{X}$ first!)

{\bf Solution.} $<$Solution goes here$>$\\

%%%   Problem 4
\item \label{prob:4} [1 point]
{\bf Exercise 1.4} from FCMA p.35

Using $\mathbf{w}$ and $\mathbf{X}$ as defined in the previous exercise, show that ${(\mathbf{X}\mathbf{w})}^\top = {\mathbf{w}}^\top{\mathbf{X}}^\top$ by multiplying out both sides.

{\bf Solution.} $<$Solution goes here$>$\\

%%%   Problem 5
\item \label{prob:5} [2 points]
{\bf Exercise 1.5} from FCMA p.35  % [1pt]

When multiplying a scalar by a vector (or matrix), we multiply each element of the vector (or matrix) by that scalar.  For $\mathbf{x}_n = {[ x_{n1}, x_{n2} ]}^\top$, $\mathbf{t} = {[ t_1,...,t_N ]}^\top$, $\mathbf{w} = {[ w_0, w_1 ]}^\top$, and
\begin{eqnarray*}
\mathbf{X} =
    \begin{bmatrix}
    {\mathbf{x}_{1}}^\top \\[0.3em]
    {\mathbf{x}_{2}}^\top \\[0.3em]
    \vdots \\[0.3em]
    {\mathbf{x}_{N}}^\top
    \end{bmatrix}
\end{eqnarray*}
show that
\begin{eqnarray*}
\sum_{n} \mathbf{x}_n t_n = \mathbf{X}^\top\mathbf{t}
\end{eqnarray*}
and
\begin{eqnarray*}
\sum_{n} \mathbf{x}_n \mathbf{x}_n ^\top \mathbf{w} = \mathbf{X}^\top\mathbf{X} \mathbf{w}
\end{eqnarray*}

{\bf Solution.} $<$Solution goes here$>$\\

\newpage
%%%   Problem 6
\item \label{prob:6} [5 points]
Reading and Displaying Numpy Arrays:

Write a script, called `hw1.py', that uses the numpy function {\tt loadtxt} to load the contents of {\tt humu.txt} (available in the {\tt data/} subdirectory of the release) into a variable (\$).

{\em Hint: If {\tt loadtxt} is new to you, visit} \\
\url{http://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.loadtxt.html} {\em ; \\ in general python, numpy, scipy and matplotlib all have great online documentation!  Also, keep in mind that you will need to {\tt import numpy} at the top of your script in order to access numpy functions; I recommend avoiding using {\tt from numpy import *} as that pollutes your namespace and can lead to very hard to debug errors. }

The default output of {\tt loadtxt} is a numpy ndarray.  You can show this by getting the {\tt type} of the variable.  From the python terminal you could do this by entering {\tt type(var)}, or if you are executing your script and want to see the output, add a print statement: {\tt print(type(var))}.  In python 3.6, the output should be {\tt <class 'numpy.ndarray'>} .  Have your script print the type (\$).

You can get the total size of a numpy object using the {\tt size} field: {\tt var.size}.  {\tt size} holds the total number of elements in the array.  Google (or other search) ``numpy size'' to see the documentation about this field.  Have your script print the size of the loaded humu data (\$).

Knowing the (total) size is helpful, but in this case we are working with a 2-dimensional array, so it would be useful to get the array dimensions.  The numpy ndarray field {\tt shape} holds a tuple representing size of each dimension of the array.  Look at the documentation for numpy {\tt shape}.  Have your script print the shape of the loaded humu data (\$).

{\em Hint: To store both the width and height into variables in one go, try: {\tt h, w = var.shape}.  This is an example of ``unpacking'', and you can even do this with nested patterns: {\tt a, (b, c) = [1, [2, 3]]}.  You could alternately do: {\tt h = var.shape[0]} and {\tt w = var.shape[1]}.}

You will now use matplotlib to plot the image.  To do so, you need to do the following:
\begin{enumerate}
\item Add {\tt import matplotlib.pyplot as plt} to the top of your file.
\item Add {\tt plt.figure()} to create a new figure object
\item Use {\tt imshow} to display your original humu array.
\end{enumerate}
If you successfully do the above and run your script, it will likely open the figure very quickly and then close it right after (if it shows anything at all).  To get the figure to hang around, add: {\tt plt.show()}.  Note: the figure will stay open and python will not continue further script execution until after you close the figure.

Why does the plot look so strange?  Print the current colormap: {plt.cm.cmapname} -- this will print the default colormap name (which is likley `afmhot', matplotlib's version of Matlab's {\tt jet}, or `Vega20c\_r').  This is the default colormap that is useful for data visualization.  Try plotting your figure again, but this time, select the `gray' scale colormap by adding the following argument to {\tt imshow}: {\tt cmap='gray'} (\$).

{\em Note: specifying the colormap in imshow does not change the default colormap.}

{\em Tip: visit} \url{http://matplotlib.org/1.2.1/examples/pylab_examples/show_colormaps.html} {\em to see a list of available colormaps.  You can also create your own colormaps:}\\
\url{http://matplotlib.org/examples/pylab_examples/custom_cmap.html}\\
\url{http://stackoverflow.com/questions/16834861/create-own-colormap-using-matplotlib-and-plot-color-scale}

Now make a grayscale image that is the same size as the one we have been using, except create the values to be uniformly random (see the function {\tt numpy.random.random()}).  Does it look like what you expected?  Create a second such image.  Does it look much different? (\$)

Write the uniformly random array to a file called `random.txt' using the numpy {\tt savetxt} function.  Confirm (1) that you can read the result by opening the file in a text editor, and (2) that you can load it back into another variable and display it to get the same image as before (\$).

(For fun: Anyone know what `humu' refers to?  Give the full name!)

\textbf{PART B}

In this last part of the exercise you will load a 1-dimensional array of float values in {\tt walk.txt} (of the {\tt data/} subdirectory of the release) representing a bounded random walk, where bounded means the walk does not venture outside of some range of values.  What is the range of values contained within the array?  Use the {\tt min} and {\tt max} functions to figure this out.  Have your script print the min and max values of the array (\$).

{\em Hint: take a look at the related {\tt amin} and {\tt amax} functions for full documentation.}

Use the {\tt min} and {\tt max} values to {\em linearly} scale the values of the {\tt walk.txt} array so that they lie in the range $[0, 1]$ -- store this in a new array and save the result as {\tt walk\_scale01.txt} -- you will include this file in our submission (\$).  Verify that the new, scaled array has the same dimensions as the original (print the shape!) but that the new values are in the range $[0, 1]$ (print the min and max of the new array!)

Also plot the original {\tt walk.txt} and your scaled version {\tt walk\_scale01.txt}.  To plot you can use the following very simple function that uses the matplotlib plot function:
\begin{verbatim}
import matplotlib.pyplot as plt

def plot_1d_array(arr):
    plt.figure()
    plt.plot(arr)
\end{verbatim}
Include these two plots, with appropriate captions, in your results (\$).

{\em Note: the shapes of the original {\tt walk.txt} and {\tt walk\_scale01.txt} should be identical, but the y-axis ranges should reflect the original and scaled ranges.}

{\bf Solution.} $<$Solution goes here$>$\\

%%%   Problem 7
\item \label{prob:7} [1 point]
Functions:

You can define new python functions with:\\
\hspace*{1cm} {\tt def <function\_name>(<argument1>, <argument2>, ...):}\\
\hspace*{1cm}\hspace{2em} {\tt <function\_body>...}

Modify your script from problem~\ref{prob:6} so that all of the code between loading the `humu.txt' to saving and reading the random `out.txt' file are within a single function called {\tt exercise6} and takes two arguments, {\tt infile} and {\tt outfile}, which are used instead of the hard-coded filenames.  If you execute your script now, nothing will appear to happen -- in fact, python does execute and defines the function, but the function is not called.  Add the line {\tt exercise6('humu.txt', 'out.txt')} to the end of the script, and now the function will be called when you execute the script (\$).

{\em Tip: Python functions can also have {\bf optional} arguments; for example, we could have specified the exercise6 argument list as: {\tt exercise6(infile='humu.txt', outfile='out.txt')}}

%%%   Problem 8
\item \label{prob:8} [2 points]
Documenting:

{\bf You must always document your code!}  Documenting your code is {\color{red} required} for this class -- you will loose points if you do not document your code.  Python in-line comments can be added using the \# character -- anything following will be ignored by python.  Python also uses a special idiom for documenting functions: Right after function signature line add documentation within a triple-quote body, e.g.,:
\begin{verbatim}
def scale01(arr):
    """
    Linearly scale the values of an array in the range [0,1]
    :param a: input ndarray
    :return: scaled ndarray
    """
    <function_body>...
\end{verbatim}
Beyond making your source code easier to understand and maintain, you also get the benefit of making documentation available within the python console, once functions are defined within the python instance.  For example, once I've executed the above function definition within the python console, I can execute {\tt help(scale01)} as follows:
\begin{verbatim}
>>> help(scale01)
Help on function scale01:

scale01(arr)
    Linearly scale the values of an array in the range [0,1]
    :param arr: input ndarray
    :return: scaled ndarray
\end{verbatim}

Document your code (with inline comments) and provide function docstrings for each function you write in this homework (\$).

%%%   Problem 9
\item \label{prob:9} [2 points]
Random Numbers:

Create a new function in hw1.py called {\tt exercise9} (no arguments), and add the following functionality.

Set the random seed to 8 using the {\tt seed()} function: {\tt numpy.random.seed(seed=8)}.  Use random {\tt numpy.random.randint()} to produce 1000 throws of two (6-sided) die.  ({\em Note: {\tt randint() is zero-based!}})  Use the result to estimate the probability of double sixes.  Report what you did and the result (\$).

Now run your estimation procedure 9 more times, making sure that the random number generator is not reset.  Report the results, and comment on how many times you got the same estimate as the first time (\$).

Finally, set the seed to 8 a second time, rerun your estimation procedure 10 times again and report whether you get the same result as the first 10 times (\$).

Explain why it is often important to have random number sequences that are not really random, and can be controlled (\$).

{\bf Solution.} $<$Solution goes here$>$\\

%%%   Problem 10
\item \label{prob:10} [5 points] Random Numbers, Vectors, Matrices, and Operations

{\bf Part \ref{prob:10}a:} Create a new function in hw1.py called {\tt exercise10} (no arguments), and add the following functionality.

Write a short script that initializes the random number generator
\begin{itemize}
\item[] python: {\tt numpy.random.seed(seed=5)}
\end{itemize}
Followed by creating two three-dimensional column vectors using
\begin{itemize}
\item[] python: {\tt numpy.random.rand} (used in the context of code to generate the vectors)
\end{itemize}
Here we're going to be explicit about the numpy arrays having 3 rows and 1 column dimension!
Represent the random variables as $\mathbf{a}$ and $\mathbf{b}$ (be sure you issue the call to set the random seed immediately before creating these variables).  Print them at the terminal and copy-and-paste the result here. (If using \LaTeX, use the {\tt verbatim} environment to display). (\$)

{\bf Solution~\ref{prob:10}a.} $<$Solution goes here$>$\\

{\bf Part \ref{prob:10}b:} Using the values of $\mathbf{a}$ and $\mathbf{b}$, compute the following and display the result two ways: (1) copy-and-paste the output (from the python interpreter/terminal; again, in \LaTeX~use the {\tt verbatim} environment), (2) typeset the output (e.g., using the \LaTeX~math environment).
\begin{enumerate}
\item[1.] $\mathbf{a} + \mathbf{b} = ~?$ ~~~(\$)
\item[2.] $\mathbf{a} \circ \mathbf{b} = ~?$  (element-wise multiply; Note: the notation $\mathbf{a} \circ \mathbf{b}$ is also known as the Hadamard product, the entrywise product, or the Schur product.) ~~~(\$)
\item[3.] $\mathbf{a}^\top \mathbf{b} = ~?$  (also called the dot-product) ~~~(\$)
\end{enumerate}

{\bf Solution~\ref{prob:10}b.} $<$Solution goes here$>$\\

{\bf Part \ref{prob:10}c:} Now, set the random seed to 2 and immediately generate a random $3 \times 3$ matrix $\mathbf{X}$.  In your solution, display the value of $\mathbf{X}$.  Using $\mathbf{X}$ and the earlier values of $a$ and $b$, compute the following in python and typeset the results in two ways, as before.
\begin{enumerate}
\item[4.] $\mathbf{a}^\top\mathbf{X} = ~?$ ~~~(\$)
\item[5.] $\mathbf{a}^\top\mathbf{X}\mathbf{b} = ~?$ ~~~(\$)
\item[6.] $\mathbf{X}^{-1} = ~?$ ~~~(\$)
\end{enumerate}

{\bf Solution~\ref{prob:10}c.} $<$Solution goes here$>$\\

%%%   Problem 11
\item \label{prob:11} [3 points] Simple Plotting

Use the provided python script {\tt plotlinear.py} and plot three lines (parameters of your choosing).  Place the output graphic in your pdf submission and provide a descriptive caption that indicates the intercept and slope values you used to generate the lines. (\$)

{\bf Solution~\ref{prob:11}a.} $<$Solution goes here$>$\\

Now, for another plot: create a function called exercise11 in hw1.py.  Add the following functionality:

Generate a vector whose entries are the values of $\sin(\mathbf{x})$ for $\mathbf{x}$ in the range $[0,10]$ in steps of $0.01$, and plot it.  Make it so that the horizontal and vertical axes of the plot are labeled and include a plot title: label the y-axis `sin(x)', the x-axis 'x values' and provide a title for the plot, `Sine Function for x from 0.0 to 10.0' (look at pyplot xlabel, ylabel and title).  (In this course, you must {\bf always} label you plot axes!)  Include your plot in the pdf submission. (2\$)

{\bf Solution~\ref{prob:11}b.} $<$Solution goes here$>$\\

\end{enumerate}

\newpage

%%% APPENDIX
\section*{Appendix: Numpy Arrays and Linear Algebra}

Here are some notes on the relation of numpy arrays to the linear algebra math of ``vectors'' and ``matrices'' we have been using.  The bottom line is that vectors are, in an important sense, just limit cases of matrices that are two-dimensional, but one of the dimensions is size 1.  In all code examples I will use numpy arrays as our base data structure to represent vectors and matrices.

Numpy does provide matrix objects; these build on top of the numpy.ndarray data structure.  While they are nice for reducing some of the verbosity of code, we can do {\em everything} with plain old ndarrays.  (numpy.matrix objects introduce (at least when you are first getting used to them) potential confusion about what operators are being used, due to operator overloading.)  In this course, for pedagogical reasons I'll stick with numpy arrays for all representation of vectors and matrices.

There are a variety of ways to create arrays (here is a helpful overview: \url{http://docs.scipy.org/doc/numpy/user/basics.creation.html} ). Numpy arrays can be n-dimensional.  The dimensionality of an array can be accessed by the attribute {\tt shape}, which is represented as a tuple where each position in the tuple represents the number of indices in the corresponding dimension.  Here are three arrays and their shape:
\begin{verbatim}
>>> a = numpy.array([1, 2, 3])  # This creates a 1-dimensional array
>>> a
array([1, 2, 3])
>>> a.shape
(3,)  # this shows the array is 1-d with three elements in the first dimension
>>> b = numpy.array([[1, 2, 3], [4, 5, 6]])  # This creates a 2-dimensional array
>>> b.shape                  # We see this array is two-dimensional with 2 indices
(2, 3)                       # in the first dimension and 3 in the second.
>>> b                        # We generally interpret the first dimension as row
array([[1, 2, 3],            # indices, the second as column indices.
       [4, 5, 6]])
>>> b[1, 2]       # We can use the bracket notation to index into the array;
6                 # keep in mind that python indices are 0-based, so b[1, 2] is
                  # picking out the second row, third column
>>> c = numpy.array([[1], [2], [3]])  # this defines a 2-d array
>>> c.shape
(3, 1)
>>> c
array([[1],
       [2],
       [3]])
\end{verbatim}
We can create higher dimensional arrays by nesting more lists specifying elements, but for representing vectors and matrices, we'll stick to 1 and 2-dimensional arrays.  Now for the connection with the linear algebra we have used.  It turns out that 1-d arrays are equivalent to column vectors: as noted in the comments, the convention is that the first dimension (even for 1-d arrays) is interpreted as indexing the rows of an array -- so all 1-d array indices can be interpreted as spread across rows -- i.e., their natural interpretation is as a column vector.  One thing that is a little odd is that taking the transpose of a 1-d numpy array has no effect: there is only one dimension so the transpose cannot swap dimensions.  For 2-d arrays, however, we can explicitly transpose, and therefore we can make an explicit distinction between column and row vectors (as in c vs. c1, below):
\begin{verbatim}
>>> a1 = a.T  # This "transposes" the 1-d array, which does nothing
>>> a1
array([1, 2, 3])
>>> a1.shape
(3,)  # this shows the transpose of a is still a 1-d array,
      # safely interpreted as a vector with three rows: a column vector
>>> b1 = b.T  # This transposes the b 2-d array
>>> b1.shape                # We now see this array is two-dimensional with 3 rows
(3, 2)                      # and 2 columns (i.e., 2 elements per row)
>>> b1
array([[1, 4],
       [2, 5],
       [3, 6]])
>>> c1 = c.T  # now we transpose c, which was a 2-d array
>>> c1.shape
(3, 1)   # c1 now has 1 row with 3 columns.
>>> c1   # This is most naturally interpreted as an explicit row vector
array([[1, 2, 3]])
\end{verbatim}
You will see in the provided code that I generally follow the (popular) convention that as long as I am only working with a column vector that does not need to be transposed into a row vector, I will use a 1-d array.  IF, however, I know that I will need to switch between column and row vector forms, then I will use the explicit 2-d array as in c and c1 (where one of the dimensions has only 1 index).

Here is a handy trick for converting a 1-d array into an explicit 2-d array as a column vector:
\begin{verbatim}
>>> a.shape    # recall that a is a 1-d array (with 3 row indices)
(3, )
>>> d = numpy.array(a[:, numpy.newaxis])  # The `:' is the slice operator
>>> d.shape     # The numpy.newaxis allows us to add a new axis
(3, 1)          # (1 more dimension) to our array object ... which just
>>> d           # makes explicit that our previously "implicit" column
array([[1],     # vector is now an explicit column vector, which can then
       [2],     # be transposed...
       [3]])
\end{verbatim}
The creation of the array d is built from indexing into the vector a, using a slice operator to refer to all elements along the first dimension, and then effectively adding a second dimension (of size 1) using the numpy.newaxis as the specifier to the second dimension index.

See here for more information about indexing: \\
\url{http://docs.scipy.org/doc/numpy/user/basics.indexing.html}\\
and\\
\url{http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html}

You will also want to look at the numpy linalg package for linear algebra operators (esp. for the {\tt dot} and {\tt inv} operators):\\
\url{http://docs.scipy.org/doc/numpy/reference/routines.linalg.html}

\end{document}


\newsection{Static}

\subsection{Motivation}
Now that we have created instance methods in the last chapter, we want to create class methods, which do not involve any instance state.

\subsection{Static}
The keyword \verb|static| in Java is meant to mean ``only one of these exists". If I have a \verb|static| variable in a class (in the same place an instance variable will be), there will only be one copy of that variable, regardless of how many instances there are of that class.

\par For example, let's say we have a class called \verb|B| that has a \verb|static| field called \verb|numInstances| of type \verb|int| which is incremented every time the constructor is called. We would design the code like this:
\begin{lstlisting}
public class B {
     private static int numInstances = 0; // start with a value
     public B() {
          numInstances++;
     }
     public int getNumInstances() {
          return numInstances;
     }
}
\end{lstlisting}
If we call this from main:
\begin{lstlisting}
B b1 = new B();
System.out.println(b1.getNumInstances()); // prints 1
B b2 = new B();
System.out.println(b2.getNumInstances()); // prints 2
\end{lstlisting}
Now if we change the code to not have the instance variable be static, then we have:
\begin{lstlisting}
public class B {
     private int numInstances = 0; // remove "static"
     public B() {
          numInstances++;
     }
     public int getNumInstances() {
          return numInstances;
     }
}
\end{lstlisting}
And the outputs are:
\begin{lstlisting}
B b1 = new B();
System.out.println(b1.getNumInstances()); // prints 1
B b2 = new B();
System.out.println(b2.getNumInstances()); // prints 1
\end{lstlisting}
which is not the behavior that we want.

\par Now we can get a larger understanding of various conventions we have been using:
\begin{itemize}
\item The main method's use of \verb|static| in its signature shows that only 1 copy of that method can exist at a time in a Java program. Therefore, there is no ambiguity as to where execution starts.
\item For other code, such as \verb|Math.PI|, we can have an understanding as to possibly how it and some of its static methods are implemented:
\begin{lstlisting}
public class Math {
     // ...
     public final static double PI = 3.1415926;
     // ...
     public static double pow(double base,
          double exponent) { ... }
     // ...
     public static double sqrt(double num) { ... }
}
\end{lstlisting}

\end{itemize}

\subsection{Written Exercises}

\begin{enumerate}

\item What is a static variable? What is a static method?

\item Using the code below, how many copies of the variable \verb|number| exist after instantiating 374 different \verb|AmazingClass| objects?
\begin{lstlisting}
public class AmazingClass {
     private static int number;
     public AmazingClass(int a) {
          number = a;
     }
     public int twice() {
          number *= 2;
          return number;
     }
}
\end{lstlisting}

\item Using the code from above, what is the value of \verb|number| after each of the following statements? (For each line below, assume the preceding ones have already been executed).
\begin{lstlisting}
AmazingClass ac1 = new AmazingClass(3);
AmazingClass ac2 = new AmazingClass(7);
ac1.twice();
ac2.twice();
\end{lstlisting}

\end{enumerate}


\documentclass[12pt]{article}
\usepackage[margin=1in]{geometry}
\usepackage[T1]{fontenc}
\usepackage[USenglish]{babel}
\usepackage[nodayofweek,level]{datetime}
\usepackage{amsfonts}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{tikz}
\usetikzlibrary{intersections,arrows.meta}
\usepackage{pgfplots}
\usepackage[scr]{rsfso}
\usepackage{array}
\usepackage{stackengine}
\stackMath

\usepackage{tikz,pgfplots}

\pgfplotsset{compat=1.10}
% Uncomment to use "fillbetween" function
% otherwise leave commented because
% the red highlighting is annoying
%\usepgfplotslibrary{fillbetween}

% Change for each new hw week
\newcommand{\dueDate}{\formatdate{3}{11}{2017}} % day/month/year
\newcommand{\hwNum}{5}

\begin{document}

\selectlanguage{USenglish}
%------------------------ Title Code ------------------------
\title{Homework: Week \hwNum}
\author{Joseph Ismailyan}
\date{}
\maketitle
\begin{flushleft}
Math 100 \\
Due: \dueDate \\
Professor Boltje \\
MWF 9:20a-10:25a
\end{flushleft}

%------------------------ Begin Page 1 ------------------------
\begin{minipage}[t]{0.40\textwidth}

\section*{Chapter 5}
\subsection*{6.}

\textbf{Proposition}:  \\If $ x\leq -1 $ then $ x^3-x\geq 0 $.
\\\textit{Proof.} (Contrapositive) Suppose $ x \in \mathbb{R} $, and it is not the case that $ x>-1 $, so $ x\leq -1 $. Then $ x^3 $ and $ x $ are less than or equal to $ -1 $. Therefore $ x^3-x\leq0 $. Thus it is not true that $ x^3-x>0 $.
$ \hfill\blacksquare $

\subsection*{18.}
\textbf{Proposition:} For any $ a,b\in \mathbb{Z}, $ it follows that $ (a+b)^3 \equiv a^3+b^3 (\mathrm{mod}\ 3)$.
\\\textit{Proof.} Consider $ (a+b)^3 -  a^3+b^3 $. By the process of simplification, we get the following steps: $ (a+b)^3 - (a^3+b^3) $
 $= (a^3+3a^2b+3ab^2+b^3) -a^3-b^3 $
 $= 3a^2b+3ab^2 $. Since  $ a,b\in \mathbb{Z} $, we can say that $ 3(a^2b+ab^2) $ is an integer. Then $ (a^2b+ab^2) $had the property of $ (a+b)^3 -  a^3+b^3 = 3(a^2b+ab^2) $. Then it is proper to say $ 3|((a+b)^3 -  a^3+b^3) $. Therefore $ (a+b)^3 \equiv a^3+b^3 (\mathrm{mod}\quad n)$.
$ \hfill\blacksquare $
\end{minipage}
% Creates verticle line
\hfill\vline\hfill
\begin{minipage}[t]{0.45\textwidth}

\subsection*{22.}
\textbf{Proposition:} Let $ a\in \mathbb{Z}, n\in \mathbb{N} $. If $ a $ has a remainder $ r $ when divided by $ n $, then $ a\equiv r(\mathrm{mod}\ n) $.
\\\textit{Proof.} Suppose $ a $ has a remainder $ r $ when divided by $ n $. Then it follows that $ a=qn+r $ for some integer $ q $.  We can simplify this into $ \frac{a}{n}=q+\frac{r}{n}\rightarrow -q=\frac{r}{n}-\frac{r}{n}$ which becomes $ -qn=r-a $ and since $ q $ is an arbitrary integer, we can say $ c,c\in \mathbb{Z}, c=-q $. So then $ -qn=r-a $ becomes $ cn=r-a $, which clearly shows that $ r-a $ is divisible by $ n $, or just $ n|r-a $. Therefore $ a\equiv r(\mathrm{mod}\ n) $.
$ \hfill\blacksquare $

\end{minipage}
\pagebreak

%------------------------ End Page 1 ------------------------

%------------------------ Begin Page 2 ------------------------

\begin{minipage}[t]{0.40\textwidth}

\section*{Chapter 6}
\subsection*{6.}
\textbf{Proposition:} If $ a,b\in \mathbb{Z} $, then \\ $ a^2-4b-2\neq0 $.
\\\textit{Proof.} For the sake of contradiction, suppose $ a^2-4b-2=0 $. Then $ a^2=4b+2\rightarrow a^2=2(2b+1) $ which means $ a^2 $ is even, which implies $ a $ is even. Then $ a=2n $ for some integer $ n $. Substituting back into our original equation, we get $ (2n)^2-4b-2=0$ and through we series of simplification we get $ 4n^2-4b-2=0\rightarrow4(n^2-b)=2\rightarrow2(n^2-b)=1$, which implies that $ 1 $ is even because it's the result of multiplying two even numbers, but it is really not even so we've reached a contradiction.
$ \hfill\blacksquare $
\subsection*{16.}
\textbf{Proposition:} If $ a $ and $ b $ are positive real numbers, then $ a+b\geq 2\sqrt{ab} $.
\\\textit{Proof.} For the sake of contradiction, suppose $ a+b< 2\sqrt{ab} $. Then through a series of simplification we get \\$ (a+b)^2< 4ab \\\rightarrow a^2+2ab+b^2< 4ab \\\rightarrow a^2+b^2<2ab\\\rightarrow a^2-2ab-b^2<0\\\rightarrow (a-b)^2<0$ which is a contradiction because $ a $ and $ b $ are positive real numbers and when their difference is squared, it cannot be less than 0.
$ \hfill\blacksquare $

\end{minipage}
% Creates verticle line
\hfill\vline\hfill
\begin{minipage}[t]{0.45\textwidth}

\subsection*{20.}
\textbf{Proposition:} The curve $ x^2+y^2-3=0 $ has no rational points.
\\\textit{Proof.} For the sake of contradiction, suppose $ x^2+y^2-3=0 $ has a rational point. Now suppose that the positive numbers $ x,y $ have a rational point $ (x,y) $ line on the curve. Then $ x $ and $ y $ can be rewritten as $ x=\frac{a}{b}, y=\frac{c}{d}, 0<a, b, c, d\in \mathbb{N} $ where $ a $ and $ b $ are relatively prime so then $ gcd(a,b)=1 $ and $ gcd(c,d)=1 $. Then by substation, we get $ (\frac{a}{b})^2+(\frac{c}{d})^2-3=0\rightarrow (ad)^2+(bc)^2-3(bd)^2=0\\\rightarrow (ad)^2+(bc)^2=3(bd)^2 $. Then we must have $ (ad)^2+(bc)^2\equiv 0(\mathrm{mod}\ 3)$. But recall we must have $ x\equiv 0(\mathrm{mod}\ 3) $ or $  x\equiv \pm1(\mathrm{mod}\ 3)$, thus $ x^2\equiv 0(\mathrm{mod}\ 3) $ or $ x^2\equiv 1(\mathrm{mod}\ 3) $ by properties of congruency. Then $ (ad)^2+(bc)^2 \equiv 0(\mathrm{mod}\ 3) \implies (ad)^2\equiv 0(\mathrm{mod}\ 3)$ and $ (bc)^2\equiv 0(\mathrm{mod}\ 3) $ again by properties of congruency. Then $ (ad)\equiv 0(\mathrm{mod}\ 3)$ and $ (bc)\equiv 0(\mathrm{mod}\ 3) $. Since 3 is prime, we must have $ a\equiv0(\mathrm{mod}\ 3) $ or $ d\equiv0(\mathrm{mod}\ 3) $, and $ b\equiv0(\mathrm{mod}\ 3)$ or $ c\equiv0(\mathrm{mod}\ 3) $. Then if $ b\equiv0(\mathrm{mod}\ 3) $ then $ a\not\equiv0(\mathrm{mod}\ 3) $ since $ gcd(a,b)=1 $. Then since $ b\equiv0(\mathrm{mod}\ 3) $ then $ d\equiv0(\mathrm{mod}\ 3) $. It follows that $ b=3n, d=3m, 0<n,m\in \mathbb{N} $. Therefore, $ (ad)^2+(bc)^2=3(bd)^2 \implies (3am)^2+(3cn)^2=3(9mn)^2 $. Thus, we have $ (am)^2+(cn)^2=27(mn)^2 $ so $ 3|(am)^2+(cn)^2 $ which means $gcd(a,b)\geq3 $ and $ gcd(c,d)\geq3 $.
$ \hfill\blacksquare $

\subsection*{24.}
\textbf{Proposition:} $ \log_2 3 $ is irrational.
\\\textit{Proof.} For the sake of contradiction, suppose $ \log_2 3 $ is rational. By definition of rational numbers, that means $ \log_2 3 $ can be written in the form $ \frac{a}{b} $ for arbitrary integers $ a $ and $ b $.  Therefore we can say $ 2^\frac{a}{b} = \frac{a}{b}$ using the definition of logarithms. Then we can simplify to $ 2^a=3^b $ which we know is false because an even number raised to an integer cannot be equal to an odd number raised to an integer.
$ \hfill\blacksquare $

\end{minipage}
\pagebreak

%------------------------ End Page 2 ------------------------

\end{document}


%GiG
\documentclass{beamer}
\usetheme{Copenhagen}
\setbeamertemplate{navigation symbols}{}
\setbeamertemplate{headline}{}
\DeclareMathOperator*{\argmax}{arg\,max}

\usepackage{hyperref}

\definecolor{azure}{rgb}{0.0, 0.5, 1.0}
%\newcommand{\tblue}[1]{\textcolor{blue}{#1}}
\newcommand{\tblue}[1]{{\Large {\textcolor{azure}{#1}}}}
\newcommand{\thblue}[1]{{\Huge {\textcolor{azure}{#1}}}}
\newcommand{\hred}[1]{{\textcolor{red}{#1}}}
\newcommand{\furl}[1]{{\footnote{\url{#1}}}}

\newcommand{\mypause}{\pause}
%\newcommand{\mypause}{}

\title[Saravanan Thirumuruganathan]
{Lecture 7: Decision Trees}

\author[CSE 5334]
{Instructor: Saravanan Thirumuruganathan}

\date[]

\begin{document}

\begin{frame}
  \titlepage
\end{frame}

%\begin{frame}{Outline}
%  \tableofcontents
%  % You might wish to add the option [pausesections]
%\end{frame}

\section{Outline}

\begin{frame}
\frametitle {Outline}
\begin{enumerate}
\item Geometric Perspective of Classification
\item Decision Trees
\end{enumerate}
\end{frame}

%\begin{frame}{In-Class Quizzes}
%\begin{itemize}
%\item {\Large {\bf URL:}} {\LARGE \bf \url{http://m.socrative.com/}}
%\item {\Large {\bf Room Name:} {\LARGE \bf 4f2bb99e}}
%\end{itemize}
%\end{frame}

\section{Geometric Perspective of Classification}
\begin{frame}{}
    \begin{center}
        \thblue{Geometric Perspective of Classification}
    \end{center}
\end{frame}

\begin{frame}{Perspective of Classification}
    \begin{itemize}
        \item Algorithmic
        \item Geometric
        \item Probabilistic
        \item $\ldots$
    \end{itemize}
\end{frame}

\begin{frame}{Geometric Perspective of Classification}
    \begin{itemize}
        \item Gives some intuition for model selection
        \item Understand the distribution of data
        \item Understand the expressiveness and limitations of various classifiers
    \end{itemize}
\end{frame}

\begin{frame}{Feature Space\footnote{DMA Book}}
    \begin{center}
        \includegraphics[scale=0.15]{geometricView.png}
    \end{center}
    \begin{itemize}
        \item {\bf Feature Vector:} $d$-dimensional vector of features describing the object
        \item {\bf Feature Space:} The vector space associated with feature vectors
    \end{itemize}
\end{frame}

\begin{frame}{Feature Space in Classification}
    \begin{center}
        \includegraphics[scale=0.5]{featureSpace.png}
    \end{center}
\end{frame}

\begin{frame}{Geometric Perspective of Classification}
    \begin{itemize}
        \item {\bf Decision Region:} A partition of feature space such that all feature vectors in it are assigned to same class.
        \item {\bf Decision Boundary:} Boundaries between neighboring decision regions
    \end{itemize}
\end{frame}

\begin{frame}{Geometric Perspective of Classification}
    \begin{itemize}
        \item Objective of a classifier is to {\em approximate} the ``real'' decision boundary as much as possible
        \item Most classification algorithm has specific expressiveness and limitations
        \item If they align, then classifier does a good approximation
    \end{itemize}
\end{frame}

\begin{frame}{Linear Decision Boundary}
    \begin{center}
        \includegraphics[scale=0.75]{linearDecisionBoundary.png}
    \end{center}
\end{frame}
\begin{frame}{Piecewise Linear Decision Boundary\footnote{ISLR Book}}
    \begin{center}
        \includegraphics[scale=0.6]{piecewiseLinearDecisionBoundary.png}
    \end{center}
\end{frame}
\begin{frame}{Quadratic Decision Boundary\footnote{Figshare.com}}
    \begin{center}
        \includegraphics[scale=0.5]{quadraticDecisionBoundary.png}
    \end{center}
\end{frame}
\begin{frame}{Non-linear Decision Boundary\footnote{ISLR Book}}
    \begin{center}
        \includegraphics[scale=0.5]{nonlinearDecisionBoundary.png}
    \end{center}
\end{frame}
\begin{frame}{Complex Decision Boundary\footnote{ISLR Book}}
    \begin{center}
        \includegraphics[scale=0.5]{complexDecisionBoundary.png}
    \end{center}
\end{frame}

\begin{frame}{Classifier Selection Tips}
    \begin{itemize}
        \item If decision boundary is linear, most {\em linear} classifiers will do well
        \item If decision boundary is non-linear, we sometimes have to use kernels
        \item If decision boundary is piece-wise, decision trees can do well
        \item If decision boundary is too complex, $k$-NN might be a good choice
    \end{itemize}
\end{frame}

\begin{frame}{$k$-NN Decision Boundary\footnote{ISLR Book}}
    \begin{center}
        \includegraphics[scale=0.2]{kNNVaringK.png}
    \end{center}

    \begin{itemize}
       \item Asymptotically Consistent: With infinite training data and large enough $k$, $k$-NN approaches the best possible classifier (Bayes Optimal)
       \item With infinite training data and large enough $k$, $k$-NN could approximate most possible decision boundaries
    \end{itemize}
\end{frame}

\section{Decision Trees}
\begin{frame}{}
    \begin{center}
        \thblue{Decision Trees}
    \end{center}
\end{frame}

\begin{frame}{Strategies for Classifiers}
    \begin{itemize}
        \item {\bf Parametric Models: } Makes some assumption about data distribution such as density and often use explicit probability models
        \item {\bf Non-parametric Models: } No prior assumption of data and determine decision boundaries directly.
        \begin{itemize}
            \item $k$-NN
            \item Decision tree
        \end{itemize}
    \end{itemize}
\end{frame}

\begin{frame}{Tree\furl{http://statweb.stanford.edu/~lpekelis/talks/13_datafest_cart_talk.pdf}}
    \begin{center}
        \includegraphics[scale=0.4]{tree.png}
    \end{center}
\end{frame}
\begin{frame}{Binary Decision Tree\furl{http://statweb.stanford.edu/~lpekelis/talks/13_datafest_cart_talk.pdf}}
    \begin{center}
        \includegraphics[scale=0.4]{binaryDTree.png}
    \end{center}
\end{frame}
\begin{frame}{20 Question Intuition\furl{http://www.idiap.ch/~fleuret/files/EE613/EE613-slides-6.pdf}}
    \begin{center}
        \includegraphics[scale=0.4]{20QAsDTree.png}
    \end{center}
\end{frame}
\begin{frame}{Decision Tree for Selfie Stick\footnote{The Oatmeal Comics}}
    \begin{center}
        \includegraphics[scale=0.30]{selfie_stick.jpg}
    \end{center}
\end{frame}

\begin{frame}{Decision Trees and Rules\furl{http://artint.info/slides/ch07/lect3.pdf}}
    \begin{center}
        \includegraphics[scale=0.50]{egDTrees.png}
    \end{center}
\end{frame}
\begin{frame}{Decision Trees and Rules\furl{http://artint.info/slides/ch07/lect3.pdf}}
    \begin{columns}
        \column{0.58\linewidth}
            \begin{itemize}
                \item long $\rightarrow$ skips
                \item short $\wedge$ new $\rightarrow$ reads
                \item short $\wedge$ follow Up $\wedge$ known $\rightarrow$ reads
                \item short $\wedge$ follow Up $\wedge$ unknown $\rightarrow$ skips
            \end{itemize}

        \column{0.38\linewidth}
            \centering
            \includegraphics[scale=0.40]{egDTrees.png}
    \end{columns}
\end{frame}

\begin{frame}{Building Decision Trees Intuition\furl{http://spark-summit.org/wp-content/uploads/2014/07/Scalable-Distributed-Decision-Trees-in-Spark-Made-Das-Sparks-Talwalkar.pdf}}
\begin{center}
    \begin{table}
        \begin{tabular}{| c | c | c |}
            \hline
            {\bf Horsepower} & {\bf Weight} & \textcolor{blue}{{\bf Mileage}} \\ \hline
                95 & low & low \\ \hline
                90 & low & low \\ \hline
                70 & low & high \\ \hline
                86 & low & high \\ \hline
                76 & high & low \\ \hline
                88 & high & low \\ \hline
        \end{tabular}
        \caption{Car Mileage Prediction from 1971}
    \end{table}
\end{center}
\end{frame}
\begin{frame}{Building Decision Trees Intuition}
\begin{center}
    \begin{table}
        \begin{tabular}{| c | c | c |}
            \hline
            {\bf Horsepower} & {\bf Weight} & \textcolor{blue}{{\bf Mileage}} \\ \hline
                95 & low & low \\ \hline
                90 & low & low \\ \hline
                70 & low & high \\ \hline
                86 & low & high \\ \hline
                \textcolor{green}{76} & \textcolor{green}{high} & \textcolor{green}{low} \\ \hline
                \textcolor{green}{88} & \textcolor{green}{high} & \textcolor{green}{low} \\ \hline
        \end{tabular}
        \caption{Car Mileage Prediction from 1971}
    \end{table}
\end{center}
\end{frame}
\begin{frame}{Building Decision Trees Intuition}
    \begin{center}
        \includegraphics[scale=0.5]{dTreeEg1.png}
    \end{center}
\end{frame}
\begin{frame}{Building Decision Trees Intuition}
\begin{center}
    \begin{table}
        \begin{tabular}{| c | c | c |}
            \hline
            {\bf Horsepower} & {\bf Weight} & \textcolor{blue}{{\bf Mileage}} \\ \hline
                95 & low & low \\ \hline
                90 & low & low \\ \hline
                70 & low & high \\ \hline
                86 & low & high \\ \hline
        \end{tabular}
        \caption{Car Mileage Prediction from 1971}
    \end{table}
\end{center}
\end{frame}
\begin{frame}{Building Decision Trees Intuition}
    \begin{center}
        \includegraphics[scale=0.3]{dTreeEg2.png}
    \end{center}
\end{frame}
\begin{frame}{Building Decision Trees Intuition}
    \begin{center}
        \includegraphics[scale=0.3]{dTreeEg3.png}
    \end{center}
\end{frame}
\begin{frame}{Building Decision Trees Intuition}

    {\bf Prediction:}
    \begin{center}
        \includegraphics[scale=0.3]{dTreeEg4.png}
    \end{center}
\end{frame}
\begin{frame}{Building Decision Trees Intuition}

    {\bf Prediction:}
    \begin{center}
        \includegraphics[scale=0.3]{dTreeEg5.png}
    \end{center}
\end{frame}

\section{Learning Decision Trees}
\begin{frame}{}
    \begin{center}
        \thblue{Learning Decision Trees}
    \end{center}
\end{frame}

\begin{frame}{Decision Trees}
    \begin{itemize}
        \item Defined by a {\bf hierarchy} of rules (in form of a tree)
            \begin{center}
                \includegraphics[scale=0.3]{dTreeExpl.png}
            \end{center}
        \item Rules form the internal nodes of the tree (topmost internal node = root)
        \item Each rule (internal node) tests the value of some property the data
        \item Leaf nodes make the prediction
    \end{itemize}
\end{frame}

\begin{frame}{Decision Tree Learning}

    {\bf Objective:}
    \begin{itemize}
        \item Use the training data to construct a good decision tree
        \item Use the constructed Decision tree to predict labels for test inputs
    \end{itemize}
\end{frame}

\begin{frame}{Decision Tree Learning}
    \begin{itemize}
        \item Identifying the region (blue or green) a point lies in
        \begin{itemize}
            \item A classification problem (blue vs green)
            \item Each input has 2 features: co-ordinates $\{x_1, x_2\}$ in the 2D plane
        \end{itemize}
        \item Once learned, the decision tree can be used to predict the region (blue/green) of a new test point
    \end{itemize}
\end{frame}

\begin{frame}{Decision Tree Learning}
    \begin{center}
        \includegraphics[scale=0.23]{dTreeLearning1.png}
    \end{center}
\end{frame}

\begin{frame}{Expressiveness of Decision Trees}
    \begin{columns}
        \column{0.48\linewidth}
            \centering
            \includegraphics[scale=0.40]{dTreeExpressiveness1.png}

        \column{0.48\linewidth}
            \centering
            \includegraphics[scale=0.40]{dTreeExpressiveness2.png}
    \end{columns}
\end{frame}

\begin{frame}{Expressiveness of Decision Trees}
    \begin{itemize}
        \item Decision tree divides feature space into {\bf axis-parallel} rectangles
        \item Each rectangle is labelled with one of the $C$ classes
        \item Any partition of feature space by recursive binary splitting can be simulated by Decision Trees
    \end{itemize}
\end{frame}

\begin{frame}{Expressiveness of Decision Trees}
    \begin{columns}
        \column{0.48\linewidth}
            \centering
            \includegraphics[scale=0.36]{dTreeExpressiveness1.png}

        \column{0.48\linewidth}
            \centering
            \includegraphics[scale=0.40]{dTreeExpressiveness3.png}
    \end{columns}

    \pause Feature space on left can be simulated by Decision tree but not the one on right.
\end{frame}

\begin{frame}{Expressiveness of Decision Tree}
    \begin{columns}
        \column{0.48\linewidth}
            \begin{itemize}
                \item Can express any logical function on input attributes
                \item Can express {\bf any} boolean function
                \item For boolean functions, path to leaf gives truth table row
                \item Could require exponentially many nodes
                \item $cyl=3 \vee (cyl=4 \wedge (maker=asia \vee maker=europe)) \vee \ldots $
            \end{itemize}

        \column{0.48\linewidth}
            \centering
            \includegraphics[scale=0.30]{dTreeFunctions.png}
    \end{columns}
\end{frame}

\begin{frame}{Hypothesis Space}
    \begin{itemize}
        \item Exponential search space wrt set of attributes
        \item If there are $d$ boolean attributes, then the search space has $2^{2^d}$ trees
        \begin{itemize}
            \item If $d=6$, then it is approximately $18,446,744,073,709,551,616$ (or approximately $1.8 \times 10^{18}$)
            \item If there are $d$ boolean attributes, each truth table has $2^d$ rows
            \item Hence there must be $2^{2^d}$ truth tables that can take all possible variations
            \item Alternate argument: the number of trees is same as
            \begin{itemize}
                \item number of bolean functions with $d$ variables
                \item = number of distinct truth tables with $2^d$ rows = $2^{2^d}$
            \end{itemize}
        \end{itemize}
        \item NP-Complete to find optimal decision tree
        \item Idea: Use greedy approach to find a locally optimal tree
    \end{itemize}
\end{frame}

\begin{frame}{Decision Tree Learning Algorithms}
    \begin{itemize}
        \item 1966: Hunt and colleagues from Psychology developed first known algorithm for human concept learning
        \item 1977: Breiman, Friedman and others from Statistics developed CART
        \item 1979: Quinlan developed proto-ID3
        \item 1986: Quinlan published ID3 paper
        \item 1993: Quinlan's updated algorithm C4.5
        \item 1980's and 90's: Improvements for handling noise, continuous attributes, missing data, non-axis parallel DTs, better heuristics for pruning, overfitting, combining DTs
    \end{itemize}
\end{frame}

\begin{frame}{Decision Tree Learning Algorithms}

    Main Loop:
    \begin{enumerate}
        \item Let $A$ be the ``best'' decision attribute for next node
        \item Assign $A$ as decision attribute for node
        \item For each value of $A$, create a new descendent of node
        \item Sort training examples to leaf nodes
        \item If training examples are perfectly classified, then STOP else iterate over leaf nodes
    \end{enumerate}
\end{frame}

\begin{frame}{Recursive Algorithm for Learning Decision Trees}
    \begin{center}
        \includegraphics[scale=0.4]{dTreeLearningAlgo.png}
    \end{center}
\end{frame}

\begin{frame}{Decision Tree Learning}
    \begin{itemize}
        \item Greedy Approach: Build tree, top-down by choosing one attribute at a time
        \item Choices are locally optimal and may or may not be globally optimal
        \item Major issues
        \begin{itemize}
            \item Selecting the next attribute
            \item Given an attribute, how to specify the split condition
            \item Determining termination condition
        \end{itemize}
    \end{itemize}
\end{frame}

\begin{frame}{Termination Condition}

    Stop expanding a node further when: \pause
    \begin{itemize}
        \item It consist of examples all having the same label
        \item Or we run out of features to test!
    \end{itemize}
\end{frame}

\begin{frame}{How to Specify Test Condition?}
    \begin{itemize}
        \item Depends on attribute types
        \begin{itemize}
            \item Nominal
            \item Ordinal
            \item Continuous
        \end{itemize}
        \item Depends on number of ways to split
        \begin{itemize}
            \item 2-way split
            \item Multi-way split
        \end{itemize}
    \end{itemize}
\end{frame}

\begin{frame}{Splitting based on Nominal Attributes}
    \begin{center}
        \includegraphics[scale=0.4]{splittingNominalAttr.png}
    \end{center}
\end{frame}
\begin{frame}{Splitting based on Ordinal Attributes}
    \begin{center}
        \includegraphics[scale=0.4]{splittingOrdinalAttr.png}
    \end{center}
\end{frame}
\begin{frame}{Splitting based on Continuous Attributes}

    How to split continuous attributes such as Age, Income etc
    \pause
    \begin{itemize}
        \item {\bf Discretization} to form an ordinal categorical attribute
        \begin{itemize}
            \item {\em Static:} discretize once at the beginning
            \item {\em Dynamic:} find ranges by equal interval bucketing, equal frequency bucketing, percentiles, clustering etc
        \end{itemize}
        \item {\bf Binary Decision:} $ (A < v) or (A \geq v)$
        \begin{itemize}
            \item Consider all possible split and find the best cut
            \item Often, computationally intensive
        \end{itemize}
    \end{itemize}
\end{frame}
\begin{frame}{Splitting based on Continuous Attributes}
    \begin{center}
        \includegraphics[scale=0.5]{splittingContAttr.png}
    \end{center}
\end{frame}

\begin{frame}{Choosing the next Attribute - I}
    \begin{center}
        \includegraphics[scale=0.4]{choosingNextAttr1.png}
    \end{center}
\end{frame}
\begin{frame}{Choosing the next Attribute - II\furl{http://www.cedar.buffalo.edu/~srihari/CSE574/Chap16/Chap16.1-InformationGain.pdf}}
    \begin{center}
        \includegraphics[scale=0.3]{choosingNextAttr2.png}
    \end{center}
\end{frame}
\begin{frame}{Choosing the next Attribute - III}
    \begin{center}
        \includegraphics[scale=0.45]{choosingNextAttr3.png}
    \end{center}
\end{frame}

\begin{frame}{Choosing an Attribute}
    \begin{itemize}
        \item Good Attribute \pause
        \begin{itemize}
            \item for one value we get all instances as positive
            \item for other value we get all instances as negative
        \end{itemize}
        \item Bad Attribute \pause
        \begin{itemize}
            \item it provides no discrimination
            \item attribute is immaterial to the decision
            \item for each value we have same number of positive and negative instances
        \end{itemize}
    \end{itemize}
\end{frame}

\begin{frame}{How to Find the Best Split?}
    \begin{center}
        \includegraphics[scale=0.4]{howToFindBestSplit.png}
    \end{center}
\end{frame}

\begin{frame}{Measures of Node Impurity}
    \begin{itemize}
        \item Gini Index
        \item Entropy
        \item Misclassification Error
    \end{itemize}
\end{frame}

\begin{frame}{Gini Index}
    \begin{itemize}
        \item An important measure of statistical dispersion
        \item Used in Economics to measure income inequality in countries
        \item Proposed by Corrado Gini
    \end{itemize}
\end{frame}
\begin{frame}{Gini Index}
    \begin{center}
        \includegraphics[scale=0.4]{giniIndex2.png}
    \end{center}
\end{frame}
\begin{frame}{Gini Index}
    \begin{center}
        \includegraphics[scale=0.4]{giniIndex1.png}
    \end{center}
\end{frame}
\begin{frame}{Splitting Based on Gini}
    \begin{itemize}
        \item Used in CART, SLIQ, SPRINT
        \item When a node $p$ is split into $k$ partitions (children), the quality of split is computed as,
                $$ Gini_{split} = \sum_{i=1}^{k} \frac{n_i}{n} Gini(i) $$
        \begin{itemize}
            \item $n_i$ =  number of records at child $i$
            \item $n$ =  number of records at node $p$
        \end{itemize}
    \end{itemize}
\end{frame}
\begin{frame}{Gini Index for Binary Attributes}
    \begin{center}
        \includegraphics[scale=0.4]{giniIndex3.png}
    \end{center}
\end{frame}
\begin{frame}{Gini Index for Categorical Attributes}
    \begin{center}
        \includegraphics[scale=0.4]{giniIndex4.png}
    \end{center}
\end{frame}

\begin{frame}{Entropy and Information Gain}
    \begin{itemize}
        \item You are watching a set of independent random samples of a random variable $X$
        \item Suppose the probabilities are equal: $P(X=A) = P(X=B) = P(X=C) = P(X=D) = \frac{1}{4}$
        \item Suppose you see a text like $BAAC$
        \item You want to transmit this information in a {\bf binary} communication channel
        \item How many bits will you need to transmit this information? \pause
        \item Simple idea: Represent each character via 2 bits: $A=00, B=01, C=10, D=11$
        \item So, $BAAC$ becomes $01000010$
        \item Communication Complexity: $2$ on average bits per symbol
    \end{itemize}
\end{frame}

\begin{frame}{Entropy and Information Gain}
    \begin{itemize}
        \item Suppose you knew probabilities are unequal: $P(X=A) = \frac{1}{2}, P(X=B) = \frac{1}{4}, P(X=C) = P(X=D) = \frac{1}{8}$
        \item It is now possible to send information $1.75$ bits on average per symbol \pause
        \item Choose a frequency based code!
        \item $A=0, B=10, C=110, D=111$ \pause
        \item $BAAC$ becomes $1000110$
        \item Requires only 7 bits for transmitting $BAAC$
    \end{itemize}
\end{frame}

\begin{frame}{Entropy}
    \begin{center}
        \includegraphics[scale=0.4]{entropy1.png}
    \end{center}
\end{frame}
\begin{frame}{Entropy}
    \begin{center}
        \includegraphics[scale=0.4]{entropy2.png}
    \end{center}
\end{frame}
\begin{frame}{Entropy}
    \begin{center}
        \includegraphics[scale=0.4]{entropy3.png}
    \end{center}
\end{frame}
\begin{frame}{Entropy}
    \begin{center}
        \includegraphics[scale=0.4]{entropy4.png}
    \end{center}
\end{frame}

\begin{frame}{Splitting based on Classification Error}
    \begin{itemize}
        \item Classification error at node $t$ is $$Error(t) = 1 - max_{i} P(i|t)$$
        \item Measures misclassification error made by a node.
        \begin{itemize}
            \item Minimum ($0.0$) when all records belong to one class, implying  most interesting information
            \item Maximum ($1 - \frac{1}{n_c}$) when records are equally distributed among all classes, implying least interesting information
        \end{itemize}
    \end{itemize}
\end{frame}
\begin{frame}{Classification Error}
    \begin{center}
        \includegraphics[scale=0.4]{classificationError.png}
    \end{center}
\end{frame}

\begin{frame}{Comparison among Splitting Criteria}
    \begin{center}
        \includegraphics[scale=0.4]{comparingSplittingCritieria.png}
    \end{center}
\end{frame}
\begin{frame}{Splitting Criteria}
    \begin{itemize}
        \item Gini Index (CART, SLIQ, SPRINT):
        \begin{itemize}
            \item select attribute that minimize impurity of a split
        \end{itemize}
        \item Information Gain (ID3, C4.5)
        \begin{itemize}
            \item select attribute with largest information gain
        \end{itemize}
        \item Normalized Gain ratio (C4.5)
        \begin{itemize}
            \item normalize different domains of attributes
        \end{itemize}
        \item  Distance normalized measures (Lopez de Mantaras)
        \begin{itemize}
            \item define a distance metric between partitions of the data
            \item chose the one closest to the perfect partition
        \end{itemize}
        \item $\chi^2$ contingency table statistics (CHAID)
        \begin{itemize}
            \item measures correlation between each attribute and the class label
            \item select attribute with maximal correlation
        \end{itemize}
    \end{itemize}
\end{frame}

\begin{frame}{Overfitting in Decision Trees}
    \begin{itemize}
        \item Decision trees will always overfit in the absence of label noise
        \item Simple strategies for fixing:
        \begin{itemize}
            \item Fixed depth
            \item Fixed number of leaves
            \item Grow the tree till the gain is above some threshold
            \item Post pruning
        \end{itemize}
    \end{itemize}
\end{frame}

\begin{frame}{Trees vs Linear Models}
    \begin{center}
        \includegraphics[scale=0.4]{treesVsLinearModels.png}
    \end{center}
\end{frame}

\begin{frame}{Advantages and Disadvantages}
    \begin{itemize}
        \item Very easy to explain to people.
        \item Some people believe that decision trees more closely mirror human decision-making
        \item Trees can be displayed graphically, and are easily interpreted even by a non-expert (especially if they are small)
        \item Trees can easily handle qualitative predictors without the need to create dummy variables.
        \item Inexpensive to construct
        \item Extremely fast at classifying new data
        \item Unfortunately, trees generally do not have the same level of predictive accuracy as other classifiers
    \end{itemize}
\end{frame}

\section{Summary}
\begin{frame}{Summary}

\tblue{Major Concepts:}
\begin{itemize}
    \item Geometric interpretation of Classification
    \item Decision trees
\end{itemize}
\end{frame}

\begin{frame}{Slide Material References}
\begin{itemize}
    \item Slides from ISLR book
    \item Slides by Piyush Rai
    \item Slides for Chapter 4 from ``Introduction to Data Mining'' book by Tan, Steinbach, Kumar
    \item Slides from Andrew Moore, CMU
    \item See also the footnotes
\end{itemize}
\end{frame}

\end{document}


\section{Standing Up the Z80}

Let's start this process by making sure we can get our Z80 processor to work properly. Building incrementally, and learning as we go,
is key to how we are approaching the creation of our Retro Computer. Following a similar approach will be helpful for you as you explore new components and systems of your own design.

We want to minimize the number of failure points, so we are going to take our first steps with the Z80 using the Arduino as our bootstrap. We know we are working with a +5V processor expecting TTL-level logic signals, so the Arduino will interface without having to do any special level shifting. What we need to do is to ascertain the \textit{bare minimum} we need to do to the Z80 to make it function (and, of course, we need some way of confirming that the processor \textit{is} doing something!).

What's the simplest thing we can have the Z80 do? Without any memory, we can't provide the Z80 with any kind of interesting program to run. Without any I/O devices, we can't enter data or see the results of any program anyway. Instead, we will \textit{trick} the Z80 into thinking it is reading a program from memory - but it will be the world's most boring program, and endless series of NOP instructions.

NOP stands for \textit{no operation}. It's an instruction that effectively does nothing, other than to tell the processor to move on to the next instruction. An endless series of of NOPs will cause the processor to run forever, doing 'nothing' but looking at memory address after memory address, finding NOP after NOP. What helps us out here is that the binary code for NOP is 00000000.

We don't have to build any kind of memory into this exiprement; we simply need to ensure that whenever the Z80 tries to read from memory looking for an instruction, that all of the data lines are 0. It will think it has read from memory and found a NOP.

Since the Z80 will execute the NOP then move to the next address, we can check to see if the Z80 is working by looking at the address bus. If we connect some LED's to the bus, we should see them turn on and off in an ascending binary pattern every few clock ticks (it takes the Z80 multiple clock ticks to execute an instruction).

Reviewing the datasheet, we see have a number of pins on the Z80 that we will need to do something with:

\resizebox{\textwidth}{!}{%
\begin{tabular}{|c|c|l|}
\hline
\textbf{Pin} & \textbf{Name} & \textbf{Notes} \\
\hline
$A_{0}$ - $A_{15}$ & Address Bus & We will use the address bus to view the actions of the processor. \\
\hline
$D_{0}$ - $D_{7}$ & Data Bus & We need to tie these low to simulate a NOP instruction. \\
\hline
$M_{1}$ & Machine Cycle One & This is an output pin, which we can ignore. \\
\hline
$MREQ$ & Memory Request & This is an output pin, which we can ignore. \\
\hline
$IORQ$ & Input/Output Request & This is an output pin, which we can ignore. \\
\hline
$RD$ & Memory Read & This is an output pin, which we can ignore. \\
\hline
$WR$ & Memory Write & This is an output pin, which we can ignore. \\
\hline
$REFSH$ & Refresh & This is an output pin, which we can ignore. \\
\hline
$HALT$ & Halt & This is an output pin, which we can ignore. \\
\hline
$WAIT$ & Wait & We need to turn this pin off or the Z80 will enter into a waiting state. \\
\hline
$INT$ & Interrupt Request & We need to turn this input pin off. \\
\hline
$NMI$ & Non-Maskable Interrupt & This also needs to be turned off. \\
\hline
$RESET$ & Reset & We will need to temporarily activate this pin to get the Z80 started properly. \\
\hline
$BUSRQ$ & Bus Request & Another input pin that needs to be turned off. \\
\hline
$BUSAK$ & Bus Acknowledge & This is an ouput pin, which we can ignore. \\
\hline
$CLK$ & Clock & We need to provide a clock signal to the Z80. \\
\hline
\end{tabular}}

\begin{center}
\includegraphics[height=6cm]{Z80_pinout.png}
\end{center}

Some comments:

\begin{itemize}
\item The WAIT, INT, NMI and BUSRQ pins are \textbf{active low}. This means that the pin's logic levels are effectively inverted; a 'low' is ON or 1, a 'high' is OFF or 0. To turn these pins off, we must actually provide +5V.
\item The Data Bus is not inverted; it is \textbf{active high}. With any IC, though, we can't leave an input pin unconnected and just assume that it will be a zero (the Data Bus are tri-state, input/output pins). We need to explicitly set it low, which we do by connecting the pin to ground through a small resistor.
\end{itemize}

It looks like there are a number of pins we can either safely ignore, or simply tie to either +5V or ground. And although we didn't list them, we do need to provide +5V of power and make a ground connection.

\subsection{Schematic}
\begin{center}
\includegraphics[height=10cm]{first_z80_schematic.png}
\end{center}

Let's walk through the schematic connection by connection:

\begin{itemize}
\item Address lines $A_{0}$ through $A_{3}$ are connected to a resistor then to an LED. These pins will provide $+5V$ when on, so we need to be sure we connect to the proper side of the LED (the longer wire on an LED is positive). The other side of the LED goes into ground\footnote{The resistor can vary depending on the LED, but assuming you are using a standard red LED, a resistor of around 220 ohms should be fine. The purpose of the resistor is to drop some of the voltage, as putting $+5V$ directly on the LED can and probably will burn it out, imediatly if not eventually.}.
\item We need to simulate a zero on each data line, so we tie each line to ground. We can't leave the line open; it will float around and may or may not end up as a low or zero. We could just tie it directly to ground, but we insert a 330 ohm resistor to limit the amount of current that the Z80 has to try and provide. So - data lines $D_{0}$ through $D_{7}$ each go to a resistor then to ground.
\item $WAIT$, $INT$, $NMI$, and $BUSRQ$ All need to be turned off. They are active \textbf{low}, so we need to provide a logic level high to turn them off. They are all connected through a resistor to the $+5V$ supply\footnote{In this case, the resistor is not really necessary annd if fact will drop the voltage a little but I left it in to be consistent with what I really did while building this.}.
\item In order to help debug, I added a LED to the $HALT$ line. This is also active \textbf{low}, so when the LED is lit, HALT is actually \textit{off}. Like the address lines, it is tied through a voltage dropping resistor then the positive side of an LED, then on to ground.
\item The $CLK$ is driven directly by the Arduino; see the Arduino code below for more information.
\item The $RESET$ is also driven by the Arduino. Note that in this case, we need to pull the line to zero to enable reset (again, active \textbf{low}). Turning the Arduino pin "on" turns off reset by providing $+5V$, but turning the Arduino pin "off" doesn't pull the pin low, it just stops sending $+5V$. We need to tie the pin to ground, through a current limiting resistor, to enable reset\footnote{Remember, input pins float when unconnected, and when the Arduino output is "off", the pin is effectively unconnected. This resistor pulls it low when the Arduino pin is "off", and is basically ignored when the Arduino pin is "on".}.
\item $GND$ and $VCC$ are connected to ground and $+5V$ respectively, through a decoupling capacitor\footnote{The purpose of the capacitor is to provide smoother current to the Z80. As the Z80 processes instructions, it's current needs will rise and fall. The capacitor acts as buffer to ensure there is enough current available at all times. At the slow speeds we are running, yes, the Z80 will probably work without it, but as we speed up and as we add more components, we need this capacitor there to increase reliability.}.
\end{itemize}

Ground and $+5V$ are supplied to the Z80 by the Arduino itself. The Z80 doesn't require much power and the Arduino can easily source enough current.

\subsection{Arduino Code}
The following code is loaded onto the Arduino:

\lstset{numbers=left, frame=single, breaklines=true, basicstyle=\ttfamily\scriptsize}
\lstinputlisting{z80_first_bootsrap.ino}


\documentclass[a4paper]{article}

%% Language and font encodings
\usepackage[english]{babel}
\usepackage[utf8x]{inputenc}
\usepackage[T1]{fontenc}

%% Sets page size and margins
\usepackage[a4paper,top=3cm,bottom=2cm,left=3cm,right=3cm,marginparwidth=1.75cm]{geometry}

%% Useful packages
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{listings}
\usepackage{graphicx}
\usepackage[colorinlistoftodos]{todonotes}
\usepackage[colorlinks=true, allcolors=blue]{hyperref}

\title{Judson's Abstract Algebra: Chapter 4}
\date{}

\begin{document}
\maketitle

\section*{1}

\section*{2}

\section*{3}

\section*{4}

\section*{5}

Find the order of every element in $\mathbb{Z}_{18}$.

\vspace{\baselineskip}

\begin{tabular}{ l | c }
  Element & Order \\
  \hline
  0 & 1 \\
  1 & 18 \\
  2 & 9 \\
  3 & 6 \\
  4 & 9 \\
  5 & 18 \\
  6 & 3 \\
  7 & 18 \\
  8 & 9 \\
  9 & 2 \\
  10 & 9 \\
  11 & 18 \\
  12 & 3 \\
  13 & 18 \\
  14 & 9 \\
  15 & 6 \\
  16 & 9 \\
  17 & 18 \\
\end{tabular}

This can be generated using a simple python script

\begin{verbatim}
for i in range(18):
    for j in range(1, 19):
        if (i*j) % 18 == 0:
            print i,j
            break
\end{verbatim}

\section*{6}

\section*{7}

\section*{8}

\section*{9}

\section*{10}

Find all elements of finite order in each of the following groups. Here the "*" indicates the set with 0 removed.

\vspace{\baselineskip}

In $\mathbb{Z}$ the single element with finite order is $0.$

In $\mathbb{Q}^*$ the elements with finite order are $1, -1.$

In $\mathbb{R}^*$ the elements with finite order are $1, -1.$

\section*{11}

If $a^{24} = e$ in a group $G$, what are possible orders of $a$?

\vspace{\baselineskip}

Possible orders of $a$ are factors of 24. Therefore $|a| \in \{ 1, 2, 3, 4, 6, 8, 12, 24 \}$.

\section*{12}

\section*{13}

\section*{14}

\section*{15}

Evaluate each of the following

$$(3-2i) + (5i-6) = -3 + 3i$$
$$(4-5i) - \overline{(4i-4)} = 8 - i$$
$$(5-4i)(7+2i) = 35 + 10i - 28i + 8 = 43 - 18i$$
$$(9-i)\overline{(9-i)} = 9^2 + 1^2 = 82$$
$$i^{45} = i^{44} i = i(i^4)^{11} = i 1^{11} = i$$
$$(1+i) + \overline{(1+i)} = 2$$

\section*{16}

Convert the following complex numbers to the form $a + bi$.

$$2 \text{cis} (\pi/6) = 2(\cos (\pi/6) + i \sin(\pi/6)) = \sqrt{3} + i$$
$$5 \text{cis} (9\pi/4) = 5(\cos (9\pi/4) + i \sin(9\pi/4)) = \frac{5 + 5i}{2}$$
$$4 \text{cis} (\pi) = 4(\cos (\pi) + i \sin(\pi)) = -4$$
$$\text{cis} (7\pi/4) = \cos (7\pi/4) + i \sin(7\pi/4) = \frac{1-i}{/sqrt{2}}$$

\section*{17}

Convert the following complex numbers to polar representation.

$$ 1-i = \sqrt{2} \text{cis}(\pi/4)$$
$$ -5 = 5(\text{cis}(\pi)$$
$$ 2 + 2i = 2\sqrt{2}(\text{cis}(\pi/4)$$
$$ \sqrt{3} + i = 2(\text{cis}(\pi/6)$$
$$ -3i = 3(\text{cis}(-\pi/2)$$
$$ 2i + 2 \sqrt{3} = 4(\text{cis}(\pi/6)$$

\section*{18}

Calculate each of the following expressions

$$(1+i)^{-1} = 1/2 - i/2$$
$$(1-i)^6 = 8i$$
$$(\sqrt{3} + i)^5 = 16 (-\sqrt(3) + i)$$
$$(-i)^{10} = -1$$
$$ ((1-i)/2)^4 = -1/4$$
$$(-\sqrt{2} - \sqrt{2} i)^{12} = -4096$$
$$(-2 + 2i)^{-5} = (1/i)/256$$

\section*{19}

Prove each of the following statements.

$$|z| = |\overline{z}|$$
$$z \overline{z} = |z|^2$$
$$z^{-1} = \overline{z} / |z|^2$$
$$|z+w| \leq |z| + |w|$$
$$|z-w| \geq ||z| - |w||$$
$$|zw| = |z||w| $$

These statements are proved in order, but some proofs may rely on statements yet to be proved. Let $z = a + bi$ and $w = c + di$. Consider

\begin{align*}
|z| &= |a + bi| \\
&= \sqrt{a^2 + b^2} \\
&= \sqrt{a^2 + (-b)^2} \\
&= |a - bi| \\
&= |\overline{z}|.
\end{align*}

This proves the first statement. Consider

\begin{align*}
z \overline{z} &= (a + bi)(a - bi) \\
&= a^2 + b^2 \\
&= |z|^2.
\end{align*}

This proves the second statement. Consider, that by the previous statement,

\begin{align*}
z \overline{z} / |z|^2 &= |z|^2 / |z|^2 \\
&= 1.
\end{align*}

This proves the third statement. To prove the fourth statement first consider that

\begin{align*}
z \overline{w} + w \overline{z} &= z \overline{w} + \overline{z \overline{w}} \\
&= 2 \text{Re} (z \overline{w}) \\
&\leq 2 |z \overline{w}| = |z||\overline{w}| = |z||w|.
\end{align*}

Using this fact

\begin{align*}
|z+w|^2 &= (z+w) \overline{(z+w)} \\
&= (z+w) (\overline{z} + \overline{w}) \\
&= z \overline{z} + z \overline{w} + w \overline{z} + w \overline{w} \\
&= |z|^2 + z \overline{w} + w \overline{z} + |w|^2 \\
&\leq |z|^2 + 2|z||w| + |w|^2 = (|z| + |w|)^2.
\end{align*}

Since the modulus is a greater than or equal to 0 we can take the square root of both sides. This proves the fourth statement. Assume that $|z| \geq |w|$. Consider

\begin{align*}
|z| &= |(z-w) - w| \\
 &\leq |z-w| + |-w| \\
 & \leq |z-w| + |w|
\end{align*}

Rearranging yields

$$ |z| - |w| \leq |z-w|$$

and since $|z| > |w|$

$$ ||z| - |w|| \leq |z-w|.$$

The case where $|w| \geq |z|$ is identical, but uses the fact that

$$|w - z| = |-(z-w)| = |z-w|.$$

This proves the fifth statement. Consider

\begin{align*}
|zw|^2 &= (zw)\overline{(zw)} \\
&= zw \overline{z} \overline{w} \\
&= |z|^2 |w|^2
\end{align*}

Since the modulus is a greater than or equal to 0 we can take the square root of both sides. This proves the sixth statement.

\section*{23}

Let $a,b \in G$. Prove the following statements

The order of $a$ is the same as the order of $a^{-1}$.

For all $g \in G, |a| = |g^{-1} a g|.$

The order of $ab$ is the same as the order of $ba$.

\vspace{\baselineskip}

Let $n$ be the order of $a$. Consider

$$a^n a^{-n} = e$$
$$e a^{-n} = e$$
$$(a^{-1})^{n} = e.$$

We will prove the first statement. We must now show that $n$ is the least positive integer such that this is true for $a^{-n}$. Assume that $m \in \mathbb{Z} : 0 < m < n$ and  $(a^{-1})^{m} = e$. Consider

\begin{align*}
a^m a^{-m} &= e \\
a^m e &= e \\
a^m &= e \\
\end{align*}

which is clearly a contradiction. Hence $|a^{-1}| = n$.

\vspace{\baselineskip}

We will prove the second statement. The proof is similar to that of the previous statement. Let $g \in G$. Consider

\begin{align*}
(g^{-1} a g)^n &= g^{-1} a g g^{-1} a ... a g g^{-1} a g \\
&= g^{-1} a^n g \\
&= g^{-1} e g \\
&= g^{-1} g \\
&= e
\end{align*}

We must now show that $n$ is the least positive integer such that this is true for $a^{-n}$. Assume that $m \in \mathbb{Z} : 0 < m < n$ and  $(g^{-1} a g)^{m} = e$. Consider

\begin{align*}
(g^{-1} a g)^{m} &= e \\
g^{-1} a^m g &= e \\
a^m g &= g \\
a^m &= gg^{-1} \\
a^m &= e
\end{align*}

which is clearly a contradiction.

\vspace{\baselineskip}

We will prove the third statement. The proof is similar to the other two. Let $p$ be the order of $ab$. Consider

\begin{align*}
(ba)^{p+1} &= b (ab)^p a \\
&= b e a \\
&= ba \\
\end{align*}

hence $(ba)^p = e$. We must now show that $p$ is the least positive integer such that this is true for $ba^{p}$. Assume that $q \in \mathbb{Z} : 0 < q < p$ and  $(ba)^{q} = e$. Consider

\begin{align*}
(ab)^{q+1} &= a (ba)^q b \\
&= a e b \\
&= ab
\end{align*}

hence $(ab)^q = e$ which is a contradiction.

\section*{24}

Let $p$ and $q$ be distinct primes. How many generators does $\mathbb{Z}_{pq}$ have?

\vspace{\baselineskip}

Recall Corollary 4.7: The generators of $\mathbb{Z}_n$ are the integers $r$ such that $1 \leq r < n$ and $\gcd(r,n) = 1$. From this and the fact that $p$ and $q$ are the prime factors of $pq$ we know that each $r$ that is not a multiple of $p$ and not a multiple of $q$ is a generator. There are $q$ multiples of $p$ between 0 and $pq$ and there are $p$ multiples of $q$ between 0 and $pq$. This double counts 0 so we must subtract it. Therefore, there are $pq - p - q + 1$ generators in $\mathbb{Z}_n$.

\section*{25}

Let $p$ be prime and $r$ be a positive integer. How many generators does $\mathbb{Z}_{p^r}$ have?

\vspace{\baselineskip}

Recall Corollary 4.7: The generators of $\mathbb{Z}_n$ are the integers $r$ such that $1 \leq r < n$ and $\gcd(r,n) = 1$. The proof is similar to that of exercise 24. The generators are those elements of $\mathbb{Z}_{p^r}$ that are not multiples of $p$. There are $p^r / p = p^{r-1}$ multiples of $p$ in $\mathbb{Z}_{p^r}$. Therefore $\mathbb{Z}_{p^r}$ has $p^r - p^{r-1} = (p-1)p^{r-1}$ generators.

\section*{26}

Prove that $\mathbb{Z}_p$ has no nontrivial subgroups if $p$ is prime.

\vspace{\baselineskip}

Recall Corollary 4.7: The generators of $\mathbb{Z}_n$ are the integers $r$ such that $1 \leq r < n$ and $\gcd(r,n) = 1$. Note that except for 0 all elements  $x \in \mathbb{Z}_p$ have $\gcd(x, p) = 1$. That means for all subgroups except for $\{ 0 \}$ that if the subgroup contains a non-zero element the subgroup contains the entire group. This shows that there are no nontrivial subgroups of $\mathbb{Z}_p$ if $p$ is prime.

\section*{29}

Prove that $\mathbb{Z}_n$ has an even number of generators for $n > 2$.

\vspace{\baselineskip}

Let $x \in \mathbb{Z}_n$ be a generator of $\mathbb{Z}_n$. First note that in exercise 23 we showed that $x^{-1}$ has the same order as $x$ this means that $|x^{-1} = x = n$ and hence $x^{-1}$ is also a generator. This shows that generators always come in pairs. The exception would be the case where $x = x^{-1}$. This contradicts our assumptions however because if $x = x^{-1}$ then

$$\langle x \rangle \{x, e \}$$

which clearly contradicts $n > 2$.

\section*{31}

Let $G$ be an abelian group. Show that the elements of finite order in $G$ form a subgroup. This subgroup is called the \textit{torsion subgroup} of $G$.

\vspace{\baselineskip}

Denote the torsion subgroup of $G$ by $T$. Note that $e$ has order 1 and hence $e \in T$. Let $t \in T$. Let $n$ be the order of $t$ such that $t^n = e$. Notice that since $tt^{n-1} = e$ it is true that $t^{-1} = t^{n-1}$ and thus $t^{-1} \in T$. Let $y \in T$ with the order of $y$ being $m$. Notice that $(ty)^{mn} = t^{mn}y^{mn} = e$. This shows $ty \in T$. This proves that $T$ is a subgroup of $G$.

\section*{37}

Prove that if $G$ has no proper nontrivial subgroups, then $G$ is a cyclic group.

\vspace{\baselineskip}

Assume $G$ has no proper nontrivial subgroups. Recall Theorem 4.1: Let $G$ be a group and $a$ be any element in $G$. Then the set

$$\langle a \rangle = \{ a^k : k \in \mathbb{Z} \}$$

is a subgroup of $G$. If $G$ only has a single element then it is trivially cyclic. Assume $G$ has more than 1 element. Let $a \in G$ such that $a \neq e$. Note that $\langle a \rangle$ is a subgroup of $G$. Since $G$ has no proper nontrivial subgroups either $\langle a \rangle = \{ e \}$ (which is immediately a contradiction) or $\langle a \rangle = G$. Since the second case must be true it immediately follows that $G$ is cyclic.

\section*{38}

Prove that the order of an element in a cyclic group $G$ must divide the order of the group.

\vspace{\baselineskip}

Read prop 4.5 for idea of proof

\section*{41}

Prove that the circle group is a subgroup of $\mathbb{C}^*$.

\vspace{\baselineskip}

Note that since $|1| = 1$ the identity is in $\mathbb{T}$. Let $x,y \in \mathbb{T}$. Notice that $|xy| = |x||y| = 1$. This implies that $\mathbb{T}$ is closed under the group operation. Consider

\begin{align*}
x x^{-1} &= 1 \\
|x x ^{-1}| &= |1| \\
|x x ^{-1}| &= 1 \\
|x| |x ^{-1}| &= 1 \\
1 | x ^{-1}| &= 1 \\
|x ^{-1}| &= 1 .\\
\end{align*}

This shows $x^{-1} \in \mathbb{T}$. This shows the circle group is a subgroup of $\mathbb{C}^*$.

\end{document}