%
% API Documentation for Peach - Computational Intelligence for Python
% Module peach.ga.fitness
%
% Generated by epydoc 3.0beta1
% [Mon Dec 21 08:51:36 2009]
%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%                          Module Description                           %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

    \index{peach \textit{(package)}!peach.ga \textit{(package)}!peach.ga.fitness \textit{(module)}|(}
\section{Module peach.ga.fitness}

    \label{peach:ga:fitness}

Basic definitions and base classes for definition of fitness functions for use
with genetic algorithms.

Fitness is a function that rates higher the chromosomes that perform better
according to the objective function. For example, if the minimum of a function
needs to be found, then the fitness function should rate better the chromosomes
that correspond to lower values of the objective function. This module gives
support to use common Python functions as fitness functions in genetic
algorithms.

The classes defined in this sub-module take a function and use some algorithm to
rank a population. There are some different ranking functions, some are provided
in this module. There is also a class that can be subclassed to generate other
fitness methods. See the documentation of the corresponding class for more
information.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%                               Variables                               %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

  \subsection{Variables}

\begin{longtable}{|p{.30\textwidth}|p{.62\textwidth}|l}
\cline{1-2}
\cline{1-2} \centering \textbf{Name} & \centering \textbf{Description}& \\
\cline{1-2}
\endhead\cline{1-2}\multicolumn{3}{r}{\small\textit{continued on next page}}\\\endfoot\cline{1-2}
\endlastfoot\raggedright \_\-\_\-d\-o\-c\-\_\-\_\- & \raggedright \textbf{Value:}
{\tt \texttt{...}}&\\
\cline{1-2}
\end{longtable}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%                           Class Description                           %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

    \index{peach \textit{(package)}!peach.ga \textit{(package)}!peach.ga.fitness \textit{(module)}!peach.ga.fitness.Fitness \textit{(class)}|(}
\subsection{Class Fitness}

    \label{peach:ga:fitness:Fitness}
\begin{tabular}{cccccc}
% Line for object, linespec=[False]
\multicolumn{2}{r}{\settowidth{\BCL}{object}\multirow{2}{\BCL}{object}}
&&
  \\\cline{3-3}
  &&\multicolumn{1}{c|}{}
&&
  \\
&&\multicolumn{2}{l}{\textbf{peach.ga.fitness.Fitness}}
\end{tabular}

\textbf{Known Subclasses:} peach.ga.fitness.Ranking

Base class for fitness function classifiers.

This class is used as the base of all fitness functions. However, even if
it is intended to be used as a base class, it also provides some
functionality, described below.

A subclass of this class should implement at least 2 methods:
\begin{quote}
\begin{description}
%[visit_definition_list_item]
\item[{{\_}{\_}init{\_}{\_}(self, f)}] %[visit_definition]

Initialization method. The initialization procedure should take at least
one parameter, \texttt{f}, which is the function to be maximized. Other
parameters could be used, but sensible default values should be
available.

%[depart_definition]
%[depart_definition_list_item]
%[visit_definition_list_item]
\item[{{\_}{\_}call{\_}{\_}(self, population)}] %[visit_definition]

This method is called to calculate population fitness. There is no
recomendation about the internals of the method, but its signature is
expected as defined above. This method receives a population -{}- please,
consult the \texttt{ga} module for more information on populations -{}- and
should return a vector or list with the fitness value for each
chromosome in the same order that they appear in the population.

%[depart_definition]
%[depart_definition_list_item]
\end{description}

This class implements the standard normalization fitness, as described in
every book and article about GAs. The rank given to a chromosome is
proportional to its objective function value.
\end{quote}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%                                Methods                                %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

  \subsubsection{Methods}

    \vspace{0.5ex}

    \begin{boxedminipage}{\textwidth}

    \raggedright \textbf{\_\_init\_\_}(\textit{self}, \textit{f})

    \vspace{-1.5ex}

    \rule{\textwidth}{0.5\fboxrule}

Initializes the operator.
    \vspace{1ex}

      \textbf{Parameters}
      \begin{quote}
        \begin{Ventry}{x}

          \item[f]

The cost function to be maximized. If you need to minimize a
function, negate the return value.
        \end{Ventry}

      \end{quote}

    \vspace{1ex}

      Overrides: object.\_\_init\_\_

    \end{boxedminipage}

    \label{peach:ga:fitness:Fitness:__call__}
    \index{peach \textit{(package)}!peach.ga \textit{(package)}!peach.ga.fitness \textit{(module)}!peach.ga.fitness.Fitness \textit{(class)}!peach.ga.fitness.Fitness.\_\_call\_\_ \textit{(method)}}

    \vspace{0.5ex}

    \begin{boxedminipage}{\textwidth}

    \raggedright \textbf{\_\_call\_\_}(\textit{self}, \textit{population})

    \vspace{-1.5ex}

    \rule{\textwidth}{0.5\fboxrule}

Calculates the fitness for all individuals in the population.
    \vspace{1ex}

      \textbf{Parameters}
      \begin{quote}
        \begin{Ventry}{xxxxxxxxxx}

          \item[population]

The population to be processed. Please, consult the \texttt{ga} module
for more information on populations. This method calculates the
fitness according to the traditional normalization technique.
        \end{Ventry}

      \end{quote}

    \vspace{1ex}

      \textbf{Return Value}
      \begin{quote}

A vector containing the fitness value for every individual in the
population, in the same order that they appear there.
      \end{quote}

    \vspace{1ex}

    \end{boxedminipage}

    \label{object:__delattr__}
    \index{object.\_\_delattr\_\_ \textit{(function)}}

    \vspace{0.5ex}

    \begin{boxedminipage}{\textwidth}

    \raggedright \textbf{\_\_delattr\_\_}(\textit{...})

    \vspace{-1.5ex}

    \rule{\textwidth}{0.5\fboxrule}

x.{\_}{\_}delattr{\_}{\_}('name') {\textless}=={\textgreater} del x.name
    \vspace{1ex}

    \end{boxedminipage}

    \label{object:__getattribute__}
    \index{object.\_\_getattribute\_\_ \textit{(function)}}

    \vspace{0.5ex}

    \begin{boxedminipage}{\textwidth}

    \raggedright \textbf{\_\_getattribute\_\_}(\textit{...})

    \vspace{-1.5ex}

    \rule{\textwidth}{0.5\fboxrule}

x.{\_}{\_}getattribute{\_}{\_}('name') {\textless}=={\textgreater} x.name
    \vspace{1ex}

    \end{boxedminipage}

    \label{object:__hash__}
    \index{object.\_\_hash\_\_ \textit{(function)}}

    \vspace{0.5ex}

    \begin{boxedminipage}{\textwidth}

    \raggedright \textbf{\_\_hash\_\_}(\textit{x})

    \vspace{-1.5ex}

    \rule{\textwidth}{0.5\fboxrule}

hash(x)
    \vspace{1ex}

    \end{boxedminipage}

    \label{object:__new__}
    \index{object.\_\_new\_\_ \textit{(function)}}

    \vspace{0.5ex}

    \begin{boxedminipage}{\textwidth}

    \raggedright \textbf{\_\_new\_\_}(\textit{T}, \textit{S}, \textit{...})

      \textbf{Return Value}
      \begin{quote}
\begin{alltt}
a new object with type S, a subtype of T
\end{alltt}

      \end{quote}

    \vspace{1ex}

    \end{boxedminipage}

    \label{object:__reduce__}
    \index{object.\_\_reduce\_\_ \textit{(function)}}

    \vspace{0.5ex}

    \begin{boxedminipage}{\textwidth}

    \raggedright \textbf{\_\_reduce\_\_}(\textit{...})

    \vspace{-1.5ex}

    \rule{\textwidth}{0.5\fboxrule}

helper for pickle
    \vspace{1ex}

    \end{boxedminipage}

    \label{object:__reduce_ex__}
    \index{object.\_\_reduce\_ex\_\_ \textit{(function)}}

    \vspace{0.5ex}

    \begin{boxedminipage}{\textwidth}

    \raggedright \textbf{\_\_reduce\_ex\_\_}(\textit{...})

    \vspace{-1.5ex}

    \rule{\textwidth}{0.5\fboxrule}

helper for pickle
    \vspace{1ex}

    \end{boxedminipage}

    \label{object:__repr__}
    \index{object.\_\_repr\_\_ \textit{(function)}}

    \vspace{0.5ex}

    \begin{boxedminipage}{\textwidth}

    \raggedright \textbf{\_\_repr\_\_}(\textit{x})

    \vspace{-1.5ex}

    \rule{\textwidth}{0.5\fboxrule}

repr(x)
    \vspace{1ex}

    \end{boxedminipage}

    \label{object:__setattr__}
    \index{object.\_\_setattr\_\_ \textit{(function)}}

    \vspace{0.5ex}

    \begin{boxedminipage}{\textwidth}

    \raggedright \textbf{\_\_setattr\_\_}(\textit{...})

    \vspace{-1.5ex}

    \rule{\textwidth}{0.5\fboxrule}

x.{\_}{\_}setattr{\_}{\_}('name', value) {\textless}=={\textgreater} x.name = value
    \vspace{1ex}

    \end{boxedminipage}

    \label{object:__str__}
    \index{object.\_\_str\_\_ \textit{(function)}}

    \vspace{0.5ex}

    \begin{boxedminipage}{\textwidth}

    \raggedright \textbf{\_\_str\_\_}(\textit{x})

    \vspace{-1.5ex}

    \rule{\textwidth}{0.5\fboxrule}

str(x)
    \vspace{1ex}

    \end{boxedminipage}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%                              Properties                               %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

  \subsubsection{Properties}

\begin{longtable}{|p{.30\textwidth}|p{.62\textwidth}|l}
\cline{1-2}
\cline{1-2} \centering \textbf{Name} & \centering \textbf{Description}& \\
\cline{1-2}
\endhead\cline{1-2}\multicolumn{3}{r}{\small\textit{continued on next page}}\\\endfoot\cline{1-2}
\endlastfoot\raggedright \_\-\_\-c\-l\-a\-s\-s\-\_\-\_\- & \raggedright \textbf{Value:}
{\tt {\textless}attribute '\_\_class\_\_' of 'object' objects{\textgreater}}&\\
\cline{1-2}
\end{longtable}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%                          Instance Variables                           %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

  \subsubsection{Instance Variables}

\begin{longtable}{|p{.30\textwidth}|p{.62\textwidth}|l}
\cline{1-2}
\cline{1-2} \centering \textbf{Name} & \centering \textbf{Description}& \\
\cline{1-2}
\endhead\cline{1-2}\multicolumn{3}{r}{\small\textit{continued on next page}}\\\endfoot\cline{1-2}
\endlastfoot\raggedright f\- & Objective function to be maximized. Handle with care -{}- although it
can be changed, it might cause trouble if you do so.&\\
\cline{1-2}
\end{longtable}

    \index{peach \textit{(package)}!peach.ga \textit{(package)}!peach.ga.fitness \textit{(module)}!peach.ga.fitness.Fitness \textit{(class)}|)}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%                           Class Description                           %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

    \index{peach \textit{(package)}!peach.ga \textit{(package)}!peach.ga.fitness \textit{(module)}!peach.ga.fitness.Ranking \textit{(class)}|(}
\subsection{Class Ranking}

    \label{peach:ga:fitness:Ranking}
\begin{tabular}{cccccccc}
% Line for object, linespec=[False, False]
\multicolumn{2}{r}{\settowidth{\BCL}{object}\multirow{2}{\BCL}{object}}
&&
&&
  \\\cline{3-3}
  &&\multicolumn{1}{c|}{}
&&
&&
  \\
% Line for peach.ga.fitness.Fitness, linespec=[False]
\multicolumn{4}{r}{\settowidth{\BCL}{peach.ga.fitness.Fitness}\multirow{2}{\BCL}{peach.ga.fitness.Fitness}}
&&
  \\\cline{5-5}
  &&&&\multicolumn{1}{c|}{}
&&
  \\
&&&&\multicolumn{2}{l}{\textbf{peach.ga.fitness.Ranking}}
\end{tabular}

Ranking fitness for a population

Ranking gives fitness values equally spaced between 0 and 1. The fittest
individual receives fitness equals to 1, the second best equals to 1 - 1/N,
the third best 1 - 2/N, and so on, where N is the size of the population.
It is important to note that the worst fit individual receives a fitness
value of 1/N, not 0. That allows that no individuals are excluded from the
selection operator.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%                                Methods                                %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

  \subsubsection{Methods}

    \vspace{0.5ex}

    \begin{boxedminipage}{\textwidth}

    \raggedright \textbf{\_\_init\_\_}(\textit{self}, \textit{f})

    \vspace{-1.5ex}

    \rule{\textwidth}{0.5\fboxrule}

Initializes the operator.
    \vspace{1ex}

      \textbf{Parameters}
      \begin{quote}
        \begin{Ventry}{x}

          \item[f]

The cost function to be maximized. If you need to minimize a
function, negate the return value.
        \end{Ventry}

      \end{quote}

    \vspace{1ex}

      Overrides: peach.ga.fitness.Fitness.\_\_init\_\_

    \end{boxedminipage}

    \vspace{0.5ex}

    \begin{boxedminipage}{\textwidth}

    \raggedright \textbf{\_\_call\_\_}(\textit{self}, \textit{population})

    \vspace{-1.5ex}

    \rule{\textwidth}{0.5\fboxrule}

Calculates the fitness for all individuals in the population.
    \vspace{1ex}

      \textbf{Parameters}
      \begin{quote}
        \begin{Ventry}{xxxxxxxxxx}

          \item[population]

The population to be processed. Please, consult the \texttt{ga} module
for more information on populations. This method calculates the
fitness according to the equally spaced ranking technique.
        \end{Ventry}

      \end{quote}

    \vspace{1ex}

      \textbf{Return Value}
      \begin{quote}

A vector containing the fitness value for every individual in the
population, in the same order that they appear there.
      \end{quote}

    \vspace{1ex}

      Overrides: peach.ga.fitness.Fitness.\_\_call\_\_

    \end{boxedminipage}

    \label{object:__delattr__}
    \index{object.\_\_delattr\_\_ \textit{(function)}}

    \vspace{0.5ex}

    \begin{boxedminipage}{\textwidth}

    \raggedright \textbf{\_\_delattr\_\_}(\textit{...})

    \vspace{-1.5ex}

    \rule{\textwidth}{0.5\fboxrule}

x.{\_}{\_}delattr{\_}{\_}('name') {\textless}=={\textgreater} del x.name
    \vspace{1ex}

    \end{boxedminipage}

    \label{object:__getattribute__}
    \index{object.\_\_getattribute\_\_ \textit{(function)}}

    \vspace{0.5ex}

    \begin{boxedminipage}{\textwidth}

    \raggedright \textbf{\_\_getattribute\_\_}(\textit{...})

    \vspace{-1.5ex}

    \rule{\textwidth}{0.5\fboxrule}

x.{\_}{\_}getattribute{\_}{\_}('name') {\textless}=={\textgreater} x.name
    \vspace{1ex}

    \end{boxedminipage}

    \label{object:__hash__}
    \index{object.\_\_hash\_\_ \textit{(function)}}

    \vspace{0.5ex}

    \begin{boxedminipage}{\textwidth}

    \raggedright \textbf{\_\_hash\_\_}(\textit{x})

    \vspace{-1.5ex}

    \rule{\textwidth}{0.5\fboxrule}

hash(x)
    \vspace{1ex}

    \end{boxedminipage}

    \label{object:__new__}
    \index{object.\_\_new\_\_ \textit{(function)}}

    \vspace{0.5ex}

    \begin{boxedminipage}{\textwidth}

    \raggedright \textbf{\_\_new\_\_}(\textit{T}, \textit{S}, \textit{...})

      \textbf{Return Value}
      \begin{quote}
\begin{alltt}
a new object with type S, a subtype of T
\end{alltt}

      \end{quote}

    \vspace{1ex}

    \end{boxedminipage}

    \label{object:__reduce__}
    \index{object.\_\_reduce\_\_ \textit{(function)}}

    \vspace{0.5ex}

    \begin{boxedminipage}{\textwidth}

    \raggedright \textbf{\_\_reduce\_\_}(\textit{...})

    \vspace{-1.5ex}

    \rule{\textwidth}{0.5\fboxrule}

helper for pickle
    \vspace{1ex}

    \end{boxedminipage}

    \label{object:__reduce_ex__}
    \index{object.\_\_reduce\_ex\_\_ \textit{(function)}}

    \vspace{0.5ex}

    \begin{boxedminipage}{\textwidth}

    \raggedright \textbf{\_\_reduce\_ex\_\_}(\textit{...})

    \vspace{-1.5ex}

    \rule{\textwidth}{0.5\fboxrule}

helper for pickle
    \vspace{1ex}

    \end{boxedminipage}

    \label{object:__repr__}
    \index{object.\_\_repr\_\_ \textit{(function)}}

    \vspace{0.5ex}

    \begin{boxedminipage}{\textwidth}

    \raggedright \textbf{\_\_repr\_\_}(\textit{x})

    \vspace{-1.5ex}

    \rule{\textwidth}{0.5\fboxrule}

repr(x)
    \vspace{1ex}

    \end{boxedminipage}

    \label{object:__setattr__}
    \index{object.\_\_setattr\_\_ \textit{(function)}}

    \vspace{0.5ex}

    \begin{boxedminipage}{\textwidth}

    \raggedright \textbf{\_\_setattr\_\_}(\textit{...})

    \vspace{-1.5ex}

    \rule{\textwidth}{0.5\fboxrule}

x.{\_}{\_}setattr{\_}{\_}('name', value) {\textless}=={\textgreater} x.name = value
    \vspace{1ex}

    \end{boxedminipage}

    \label{object:__str__}
    \index{object.\_\_str\_\_ \textit{(function)}}

    \vspace{0.5ex}

    \begin{boxedminipage}{\textwidth}

    \raggedright \textbf{\_\_str\_\_}(\textit{x})

    \vspace{-1.5ex}

    \rule{\textwidth}{0.5\fboxrule}

str(x)
    \vspace{1ex}

    \end{boxedminipage}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%                              Properties                               %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

  \subsubsection{Properties}

\begin{longtable}{|p{.30\textwidth}|p{.62\textwidth}|l}
\cline{1-2}
\cline{1-2} \centering \textbf{Name} & \centering \textbf{Description}& \\
\cline{1-2}
\endhead\cline{1-2}\multicolumn{3}{r}{\small\textit{continued on next page}}\\\endfoot\cline{1-2}
\endlastfoot\raggedright \_\-\_\-c\-l\-a\-s\-s\-\_\-\_\- & \raggedright \textbf{Value:}
{\tt {\textless}attribute '\_\_class\_\_' of 'object' objects{\textgreater}}&\\
\cline{1-2}
\end{longtable}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%                          Instance Variables                           %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

  \subsubsection{Instance Variables}

\begin{longtable}{|p{.30\textwidth}|p{.62\textwidth}|l}
\cline{1-2}
\cline{1-2} \centering \textbf{Name} & \centering \textbf{Description}& \\
\cline{1-2}
\endhead\cline{1-2}\multicolumn{3}{r}{\small\textit{continued on next page}}\\\endfoot\cline{1-2}
\endlastfoot\raggedright f\- & Objective function to be maximized. Handle with care -{}- although it
can be changed, it might cause trouble if you do so.&\\
\cline{1-2}
\end{longtable}

    \index{peach \textit{(package)}!peach.ga \textit{(package)}!peach.ga.fitness \textit{(module)}!peach.ga.fitness.Ranking \textit{(class)}|)}
    \index{peach \textit{(package)}!peach.ga \textit{(package)}!peach.ga.fitness \textit{(module)}|)}


% Style from https://stackoverflow.com/questions/1911516/how-to-make-cheat-sheets-in-latex/36768704#36768704
\documentclass[10pt,landscape,a4paper]{article}
\usepackage[utf8]{inputenc}
\usepackage[english]{babel}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{mathrsfs}
\usepackage{tikz,pgf}
\usepackage{pgfplots}
\usetikzlibrary{angles,shapes,positioning,arrows,fit,calc,graphs,graphs.standard}
\usepackage{multicol}
\usepackage{wrapfig}
\usepackage[top=0mm,bottom=1mm,left=0mm,right=1mm]{geometry}
\usepackage[framemethod=tikz]{mdframed}
\usepackage{microtype}

\let\bar\overline

\DeclareMathOperator{\re}{Re}
\DeclareMathOperator{\im}{Im}
\DeclareMathOperator{\caparg}{Arg}
\DeclareMathOperator{\dom}{Dom}
\DeclareMathOperator{\codom}{Codom}
\DeclareMathOperator{\img}{Img}
\DeclareMathOperator{\Ind}{Ind}
\DeclareMathOperator{\Res}{Res}
\DeclareMathOperator{\Log}{Log}

\newcommand{\floor}[1]{\lfloor #1 \rfloor}      % simplifying the writing of a floor function
\newcommand{\ceiling}[1]{\lceil #1 \rceil}      % simplifying the writing of a ceiling function
\newcommand{\dotp}{\, \cdotp}             % dot product to distinguish from \cdot
\newcommand{\qed}{\hfill\ensuremath{\square}}   % Q.E.D sign
\newcommand{\abs}[1]{\left|#1\right|}           % absolute value
\newcommand{\at}[2]{\Big|_{#1}^{#2}}
\newcommand{\Arg}[1]{\caparg #1}
\renewcommand{\bar}[1]{\mkern 1.5mu \overline{\mkern -1.5mu #1 \mkern -1.5mu} \mkern 1.5mu}
\newcommand{\quotient}[2]{\faktor{#1}{#2}}
\newcommand{\cyclic}[1]{\left\langle #1 \right\rangle}
  % highlighting shortcuts
\newcommand{\hlimpo}[1]{\textcolor{base16-eighties-red}{\textbf{#1}}}
\newcommand{\hlwarn}[1]{\textcolor{base16-eighties-yellow}{\textbf{#1}}}
\newcommand{\hldefn}[1]{\textcolor{base16-eighties-blue}{\textbf{#1}}}
\newcommand{\hlnotea}[1]{\textcolor{base16-eighties-green}{\textbf{#1}}}
\newcommand{\hlnoteb}[1]{\textcolor{base16-eighties-lightblue}{\textbf{#1}}}
\newcommand{\hlnotec}[1]{\textcolor{base16-eighties-brown}{\textbf{#1}}}
\newcommand{\WTP}{\textcolor{base16-eighties-brown}{WTP}}
\newcommand{\WTS}{\textcolor{base16-eighties-brown}{WTS}}
\newcommand{\ind}[2]{\Ind_{#2}(#1)}
\newcommand{\res}[2]{\underset{#2}{\Res} #1 }
\renewcommand{\epsilon}{\varepsilon}
\newcommand*\dif{\mathop{}\!d}

\pgfplotsset{compat=1.15}
\usepgfplotslibrary{fillbetween}
\pgfplotsset{four quads/.append style={axis x line=middle, axis y line=
middle, xlabel={$x$}, ylabel={$y$}, axis equal }}
\pgfplotsset{four quad complex/.append style={axis x line=middle, axis y line=
middle, xlabel={$\re$}, ylabel={$\im$}, axis equal }}

\definecolor{myblue}{cmyk}{1,.72,0,.38}

\def\firstcircle{(0,0) circle (1.5cm)}
\def\secondcircle{(0:2cm) circle (1.5cm)}

\colorlet{circle edge}{myblue}
\colorlet{circle area}{myblue!5}

\tikzset{filled/.style={fill=circle area, draw=circle edge, thick},
    outline/.style={draw=circle edge, thick}}

\pgfdeclarelayer{background}
\pgfsetlayers{background,main}

\everymath\expandafter{\the\everymath \color{myblue}}
\everydisplay\expandafter{\the\everydisplay \color{myblue}}

\renewcommand{\baselinestretch}{.8}
\pagestyle{empty}

\global\mdfdefinestyle{header}{%
linecolor=gray,linewidth=1pt,%
leftmargin=0mm,rightmargin=0mm,skipbelow=0mm,skipabove=0mm,
}

\newcommand{\header}{
\begin{mdframed}[style=header]
\footnotesize
\sffamily
PMATH352W18 Cheatsheet\\
by~Johnson~Ng,~page~\thepage~of~1
\end{mdframed}
}

\makeatletter
\renewcommand{\section}{\@startsection{section}{1}{0mm}%
                                {.2ex}%
                                {.2ex}%x
                                {\color{myblue}\sffamily\small\bfseries}}
\renewcommand{\subsection}{\@startsection{subsection}{1}{0mm}%
                                {.2ex}%
                                {.2ex}%x
                                {\sffamily\bfseries}}

\def\multi@column@out{%
   \ifnum\outputpenalty <-\@M
   \speci@ls \else
   \ifvoid\colbreak@box\else
     \mult@info\@ne{Re-adding forced
               break(s) for splitting}%
     \setbox\@cclv\vbox{%
        \unvbox\colbreak@box
        \penalty-\@Mv\unvbox\@cclv}%
   \fi
   \splittopskip\topskip
   \splitmaxdepth\maxdepth
   \dimen@\@colroom
   \divide\skip\footins\col@number
   \ifvoid\footins \else
      \leave@mult@footins
   \fi
   \let\ifshr@kingsaved\ifshr@king
   \ifvbox \@kludgeins
     \advance \dimen@ -\ht\@kludgeins
     \ifdim \wd\@kludgeins>\z@
        \shr@nkingtrue
     \fi
   \fi
   \process@cols\mult@gfirstbox{%
%%%%% START CHANGE
\ifnum\count@=\numexpr\mult@rightbox+2\relax
          \setbox\count@\vsplit\@cclv to \dimexpr \dimen@-1cm\relax
\setbox\count@\vbox to \dimen@{\vbox to 1cm{\header}\unvbox\count@\vss}%
\else
      \setbox\count@\vsplit\@cclv to \dimen@
\fi
%%%%% END CHANGE
            \set@keptmarks
            \setbox\count@
                 \vbox to\dimen@
                  {\unvbox\count@
                   \remove@discardable@items
                   \ifshr@nking\vfill\fi}%
           }%
   \setbox\mult@rightbox
       \vsplit\@cclv to\dimen@
   \set@keptmarks
   \setbox\mult@rightbox\vbox to\dimen@
          {\unvbox\mult@rightbox
           \remove@discardable@items
           \ifshr@nking\vfill\fi}%
   \let\ifshr@king\ifshr@kingsaved
   \ifvoid\@cclv \else
       \unvbox\@cclv
       \ifnum\outputpenalty=\@M
       \else
          \penalty\outputpenalty
       \fi
       \ifvoid\footins\else
         \PackageWarning{multicol}%
          {I moved some lines to
           the next page.\MessageBreak
           Footnotes on page
           \thepage\space might be wrong}%
       \fi
       \ifnum \c@tracingmulticols>\thr@@
                    \hrule\allowbreak \fi
   \fi
   \ifx\@empty\kept@firstmark
      \let\firstmark\kept@topmark
      \let\botmark\kept@topmark
   \else
      \let\firstmark\kept@firstmark
      \let\botmark\kept@botmark
   \fi
   \let\topmark\kept@topmark
   \mult@info\tw@
        {Use kept top mark:\MessageBreak
          \meaning\kept@topmark
         \MessageBreak
         Use kept first mark:\MessageBreak
          \meaning\kept@firstmark
        \MessageBreak
         Use kept bot mark:\MessageBreak
          \meaning\kept@botmark
        \MessageBreak
         Produce first mark:\MessageBreak
          \meaning\firstmark
        \MessageBreak
        Produce bot mark:\MessageBreak
          \meaning\botmark
         \@gobbletwo}%
   \setbox\@cclv\vbox{\unvbox\partial@page
                      \page@sofar}%
   \@makecol\@outputpage
     \global\let\kept@topmark\botmark
     \global\let\kept@firstmark\@empty
     \global\let\kept@botmark\@empty
     \mult@info\tw@
        {(Re)Init top mark:\MessageBreak
         \meaning\kept@topmark
         \@gobbletwo}%
   \global\@colroom\@colht
   \global \@mparbottom \z@
   \process@deferreds
   \@whilesw\if@fcolmade\fi{\@outputpage
      \global\@colroom\@colht
      \process@deferreds}%
   \mult@info\@ne
     {Colroom:\MessageBreak
      \the\@colht\space
              after float space removed
              = \the\@colroom \@gobble}%
    \set@mult@vsize \global
  \fi}

\makeatother
\setlength{\parindent}{0pt}

\begin{document}
\small
\begin{multicols*}{5}

\section{Complex Numbers and Their Properties}
\subsection*{Complex Plane as a Set}
$\mathbb{C} = \mathbb{R}^2 = \left\{\begin{pmatrix} x \\ y \end{pmatrix} : x, y \in \mathbb{R} \right\}$
\subsection*{Real and Imaginary Part}
$\forall z = x + iy \in \mathbb{C} \; x, y \in \mathbb{R} \\ \re(z) = x \; \im(z) = y$
\subsection*{Product}
$\forall z = a + ib, w = c + id \in \mathbb{C} \; a, b, c, d \in \mathbb{R} \\
zw = (ac - bd) + i (ad + bc)$
\subsection*{Inverse of a Complex Number}
$\forall z = a + ib \in \mathbb{C} \; a, b \in \mathbb{R} \\
\exists z^{-1} = \frac{a}{aa^2 + b^2} - i \frac{b}{a^2 + b^2} \in \mathbb{C}$
\subsection*{Conjugate}
$\forall z = a + ib \in \mathbb{C} \; a, b \in \mathbb{R} \\
\exists \bar{z} = a - ib \in \mathbb{C}$
\subsection*{Modulus}
$\forall z = x = iy \in \mathbb{C} \; x, y \in \mathbb{R} \\
\abs{z} = \sqrt{x^2 + y^2} \in \mathbb{R}$
\subsection*{Basic Inequalities}
$\forall z, w \in \mathbb{C},$
\begin{enumerate}
  \item $\abs{\re(z)} \leq \abs{z}$
  \item $\abs{\im(z)} \leq \abs{z}$
  \item $\abs{z + w} \leq \abs{z} + \abs{w}$
  \item $\abs{z + w} \geq \abs{ \; \abs{z} + \abs{w} \; }$
\end{enumerate}
\subsection*{Region of a set of Complex Numbers}
Describe $\{z \in \mathbb{C} : \abs{z - a} < \abs{z - b}\}$.
\resizebox{\columnwidth}{!}{
  \begin{tikzpicture}
    \begin{axis}[four quad complex, xmin=-12, xmax=12, ymin=-12, ymax=12]
      \draw[line width=0pt,dashed,fill=black,fill opacity=0.25](-20,20)--(-20,-20)--(20,-20)--(20,-10)--(-5,15);
      \draw[color=black] (-1,13) node {$f$};
      \draw[line width=2pt,solid](2,2)--(8,8);
      \draw[color=black] (3.5,5) node {$g$};
      \node[label={0:{$a$}},circle,fill,inner sep=2pt] at (axis cs:2,2) {};
      \node[label={0:{$b$}},circle,fill,inner sep=2pt] at (axis cs:8,8) {};
    \end{axis}
  \end{tikzpicture}
}
\subsection*{Every complex number has exactly 2 roots}
$\forall z = x = iy \in \mathbb{C} \; x, y \in \mathbb{R} \\
\exists w_{1, 2} = u + iv \in \mathbb{C} \; u, v \in \mathbb{R}$
\begin{equation*}
  \resizebox{\columnwidth}{!}{%
    $w = \begin{cases}
      \pm \left[ \left(\frac{x + \sqrt{x^2 + y^2}}{2} \right)^{\frac{1}{2}} + i \left( \frac{-x + \sqrt{x^2 + y^2}}{2} \right)^{\frac{1}{2}} \right] & y > 0 \\
      \pm \left[ \left(\frac{x + \sqrt{x^2 + y^2}}{2} \right)^{\frac{1}{2}} - i \left(\frac{-x + \sqrt{x^2 + y^2}}{2} \right)^{\frac{1}{2}} \right] & y < 0 \\
      \pm \sqrt{x} & y = 0, x > 0 \\
      \pm i \sqrt{x} & y = 0, x < 0
    \end{cases}$
  }
\end{equation*}
\subsection*{Quadratic Formula}
$\forall a, b, c \in \mathbb{C} \; a \neq 0 \; az^2 + bz + c = 0$
\begin{equation*}
  z = \frac{-b + \sqrt{b^2 - 4ac}}{2a}
\end{equation*}
\subsection*{Argument}
\begin{tikzpicture}
  \tikzset{>=stealth}
  \draw[->] (-1, 0) -- ++(3, 0) coordinate (X) node[below] {$\re$};
  \draw[->] (0, -1) -- ++(0, 3) node[left] {$\im$};
  \coordinate (O) at (0, 0);

  \draw[->] (O) -- (1,1) coordinate (z) node[above] {$z$};
  \path (X) -- (O) -- (z) pic [draw,->,pic text=$\Arg{z}$, angle eccentricity=2.0] {angle=X--O--z};
\end{tikzpicture}
\subsection*{Polar Form}
$\forall z \in \mathbb{C} \; \exists r, \theta \in \mathbb{R} \; \theta \in [0, 2\pi)$ \\
$z = re^{i\theta}$
\subsection*{Polar to Cartesian}
$x = r \cos \theta \quad y = r \sin \theta$
\subsection*{Cartesian to Polar}
$r = \abs{z} \quad \tan \theta = \frac{x}{y}$
\subsection*{Conjugate in Polar Form}
$z = re^{i\theta} \iff \bar{z} = re^{-i\theta}$
\subsection*{Inverse in Polar Form}
$z = re^{i \theta} \land z \neq 0 \\
\implies z^{-1} = \frac{1}{r} e^{-i \theta}$
\subsection*{Product in Polar Form}
\begin{itemize}
  \item $z_1 z_2 = r_1 r_2 e^{i (\theta_1 + \theta_2)}$
  \item $\forall n \in \mathbb{Z} \enspace (re^{in}) = r^n e^{in\theta}$
\end{itemize}
\subsection*{nth Roots of a Complex Number}
$\left\{ r^{\frac{1}{n}} e^{i \left(\frac{\theta + 2 \pi k}{n} \right)} : k = 0, 1, ..., n - 1 \right\}$
\subsection*{nth Roots of Unity}
$\left\{ e^{i \left(\frac{2 \pi k}{n} \right)} : k = 0, 1, ..., n - 1 \right\}$

\section{Complex Functions} % (fold)
\label{sec:complex_functions}

\subsection{Convergence} % (fold)
\label{sub:convergence}
$\forall \{z_n\}_{n \in \mathbb{N}} \subseteq \mathbb{C} \enspace \land \enspace z \in \mathbb{Z} \\
  (n \to \infty \implies z_n \to z) \iff \lim_{n \to \infty} \abs{z_n - z} = 0$

May also write as $\lim_{n \to \infty} z_n = z$
% subsection convergence (end)

\subsection{Convergence for Complex Functions} % (fold)
\label{sub:convergence_for_complex_functions}
$\forall \Omega \subseteq \mathbb{C} \enspace \forall f : \Omega \to \mathbb{C} \enspace z_0 \in \mathbb{C} \enspace \exists L \in \mathbb{C} \enspace \forall \{z_n\}_{n \in \mathbb{N}} \subseteq \Omega \setminus \{z_0\}$

$(z_n \to z_0 \implies f(z_0) \to L) \implies \lim_{z \to z_0} f(z) = L$
% subsection convergence_for_complex_functions (end)

\subsection{Continuity} % (fold)
\label{sub:continuity}

$\forall f : \Omega \subseteq \mathbb{C} \to \mathbb{C}$

$f$ is continuous on $z_0 \implies$
\begin{enumerate}
  \item $\forall \{z_0\}_{n \in \mathbb{N}} \enspace z_n \to z_0 \implies f(z_n) \to f(z_0)$
  \item $\forall z \in \Omega \; \forall \epsilon > 0 \; \exists \delta > 0 \enspace \abs{z - z_0} < \delta \implies \abs{f(z) - f(z_0)} < \epsilon$
\end{enumerate}

% subsection continuity (end)

\subsection{Real and Imaginary Parts of a Function} % (fold)
\label{sub:real_and_imaginary_parts_of_a_function}

$f(z) = u(x, y) + iv(x, y)$

% subsection real_and_imaginary_parts_of_a_function (end)

% section complex_functions (end)

\section{Differentiation} % (fold)
\label{sec:differentiation}

\subsection{Neighbourhood} % (fold)
\label{sub:neighbourhood}

$\forall z_0 \in \mathbb{C} \; r \in \mathbb{R} \; D(z_0, r) := \{z in \mathbb{C} : \abs{z - z_0} < r\}$ is the neighbourhood of radius $r$ around $z_0$.

% subsection neighbourhood (end)

\subsection{Differentiation/Holomorphic} % (fold)
\label{sub:differentiation_holomorphic}

Let $z_0 \in \mathbb{C} \; r \in \mathbb{R} \; \exists D(z_0, r) \subseteq \mathbb{R}$. $\forall f : D(z_0, r) \to \mathbb{C} \; \forall h \in \mathbb{C}$

$\exists \lim_{h \to 0} \frac{f(z_0 + h) - f(z_0)}{h} \implies$ $f$ is differentiable/holomorphic $\land \; f'(z_0) = \lim_{h \to 0} \frac{f(z_0 + h) - f(z_0)}{h}$

% subsection differentiation_holomorphic (end)

\subsection{Properties of Holomorphic Functions} % (fold)
\label{sub:properties_of_holomorphic_functions}
$f, g$ are holomorphic at $z \in \mathbb{C} \implies$
\begin{enumerate}
  \item $(f + g)' = f' + g'$
  \item $(fg)' = f'g + fg'$
  \item $(g \neq 0 \implies (\frac{f}{g})' = \frac{f'g - fg'}{g^2} )$
\end{enumerate}

% subsection properties_of_holomorphic_functions (end)

\subsection{Cauchy-Riemann Equations} % (fold)
\label{sub:cauchy_riemann_equations}

$\forall z_0 = x_0 + iy_0 \in \mathbb{C} \; x_o, y_o \in \mathbb{R}$. $f(z)$ is holomorphic at $z_0 \implies$ at $(x_0, y_0)$
\begin{equation*}
  \frac{\partial u}{\partial x} = \frac{\partial v}{\partial y} \; \land \; \frac{\partial v}{\partial x} = - \frac{\partial u}{\partial y}
\end{equation*}

% subsection cauchy_riemann_equations (end)

\subsection{Conditional Converse of CRE} % (fold)
\label{sub:conditional_converse_of_cre}
Let $z_0 = z_0 + iy_0 \in \mathbb{\Omega} \subseteq \mathbb{R} \; x_0, y_0 \in \mathbb{R} \; u, v : \mathbb{R^2} \to \mathbb{R} \; f = u + iv : \Omega \to \mathbb{C}$.
\begin{enumerate}
  \item partials of $u, v$ exist in nbd of $x_0, y_0)$
  \item partials of $u, v$ are cont' at $(x_0, y_0)$
  \item $\frac{\partial u}{\partial x} = \frac{\partial v}{\partial y} \; \land \; \frac{\partial v}{\partial x} = - \frac{\partial u}{\partial y}  $
\end{enumerate}
$\implies f$ is holo at $z_0$.

% subsection conditional_converse_of_cre (end)

\subsection{Power Series} % (fold)
\label{sub:power_series}

Infinite series of the form $\sum_{n \in \mathbb{N}} c_n z^n$

% subsection power_series (end)

\subsection{Convergence for Power Series} % (fold)
\label{sub:convergence_for_power_series}

We will usually aim for absolute convergence, for

\begin{equation*}
  \abs{\sum_{n=0}^{N} c_n z^n} \leq \sum_{n=0}^{N} \abs{c_n} \abs{z}^n
\end{equation*}

% subsection convergence_for_power_series (end)

\subsection{Hadamard's Formula} % (fold)
\label{sub:hadamard_s_formula}

$\frac{1}{R} := \limsup_{n \to \infty} \abs{c_n}^\frac{1}{n}$.

% subsection hadamard_s_formula (end)

\subsection{Limit Supremum} % (fold)
\label{sub:limit_supremum}
\begin{equation*}
  \limsup_{n \to \infty} a_n := \lim_{n \to \infty} \sup_{m \geq n} a_m
\end{equation*}
% subsection limit_supremum (end)

\subsection{limsup Property} % (fold)
\label{sub:limsup_property}
$\forall \{a_n\}_{n \in \mathbb{N}} L:=\limsup_{n \to \infty} a_n \implies$

$\forall \epsilon > 0 \; \exists N > 0 \; \forall n > N$

$\abs{a_0 - L} < \epsilon$

% subsection limsup_property (end)

\subsection{Radius of Convergence} % (fold)
\label{sub:radius_of_convergence}
$\forall \sum_{n \in \mathbb{N}} c_n z^n \; \exists 0 \leq R < \infty$
\begin{enumerate}
  \item $\abs{z} < R \implies$ absolute convergence
  \item $\abs{z} > R \implies$ divergence
\end{enumerate}
% subsection radius_of_convergence (end)

\subsection{Power Function and its Holomorphic Function share the same Region of Convergence} % (fold)
\label{sub:power_function_and_its_holomorphic_function_share_the_same_region_of_convergence}

$f(z) = \sum_{n \in \mathbb{N}} c_n z^n $ had a rad of conv $R \in \mathbb{R} \implies \forall \{z : \abs{z} < R\}$

\begin{equation*}
  f'(z) = \sum_{n=1}^{\infty} nc_n z^{n - 1}
\end{equation*}

rad of conv of $f'$ is $R$.

% subsection power_function_and_its_holomorphic_function_share_the_same_region_of_convergence (end)

\subsection{Entire Function} % (fold)
\label{sub:entire_function}
$f$ is said to be entire if $f$ is holomorphic in the entire $\mathbb{C}$.
% subsection entire_function (end)

% section differentiation (end)

\section{Integration} % (fold)
\label{sec:integration}

\subsection{Curves} % (fold)
\label{sub:curves}
A curve in $\mathbb{C}$ is a cont' fn $\gamma : [a, b] \subseteq \mathbb{R} \to \mathbb{C}$. Image of $\gamma$ is called $\gamma^*$.
% subsection curves (end)

\subsection{Equivalent Parameterization} % (fold)
\label{sub:equivalent_parameterization}
Let $\gamma_1 : [a, b] \subseteq \mathbb{R} \to \mathbb{C} \; \gamma_2 : [c, d] \subseteq \mathbb{R} \to \mathbb{C}$ desc path $\gamma^*$. $\gamma_1, \gamma_2$ are equiv if $\exists h : [a, b] \to [c, d]$, bijective and cont', s.t. $\forall t \in \dom(h) \; \gamma_1 (t) = \gamma_2(h(t))$.
% subsection equivalent_parameterization (end)

\subsection{Smooth Curve} % (fold)
\label{sub:smooth_curve}
$\gamma$ is smooth if $\exists \gamma'$ is cont' on $\dom(\gamma) \; \land \; \forall t \in \dom(\gamma) \; \gamma'(t) \neq 0$.
% subsection smooth_curve (end)

\subsection{Piecewise Smooth Curve} % (fold)
\label{sub:piecewise_smooth_curve}
$\gamma$ is piecewise smooth if $\gamma$ is smooth on $\dom(\gamma)$ except on finitely many pts.
% subsection piecewise_smooth_curve (end)

\subsection{Integral over path} % (fold)
\label{sub:integral_over_path}
Let $\gamma: [a, b] \to \mathbb{C} \; \land \; f : \mathbb{C} \to \mathbb{C}$ con' on $\gamma$. Integral $f$ along $\gamma$ is
\begin{equation*}
  \int_{\gamma} f(z) dz = \int_{a}^{b} f(\gamma(t))\gamma'(t) dt
\end{equation*}
Integral over a curve $\gamma^*$ is independent of the path chosen.
% subsection integral_over_path (end)

\subsection{Integral Properties} % (fold)
\label{sub:integral_properties}
\begin{enumerate}
  \item (Linearity) $\int_{\gamma} (\alpha f + \beta g) = \alpha \int_{\gamma} f + \beta \int_{\gamma} g$
  \item \begin{enumerate}
    \item $\abs{\int_{a}^{b} g} \leq \int_{a}^{b} \abs{g}$
    \item $\abs{\int_{\gamma} f dz} \leq \sup_{z \in \Omega} f(z) \cdot \int_{a}^{b} \abs{\gamma'(t)} dt$
  \end{enumerate}
  \item $\gamma^-$ is in opposite orientation of $\gamma \implies \int_{\gamma^-} f = - \int_{\gamma} f$
\end{enumerate}
% subsection integral_properties (end)

\subsection{Fundamental Theorem of Calculus} % (fold)
\label{sub:fundamental_theorem_of_calculus}
Let $(\gamma : [a, b] \subseteq \mathbb{R} \to \mathbb{C}) \in \Omega \subseteq \mathbb{C}$. $f$ cont' on $\gamma \; \exists F' = f$ holo on $\Omega \implies \int_{\gamma} f = F(\gamma(b)) - F(\gamma(a))$
% subsection fundamental_theorem_of_calculus (end)

\subsection{Corollary of FTC} % (fold)
\label{sub:corollary_of_ftc}
If $F \in H(\Omega), \, \Omega \subseteq \mathbb{C}, \, \gamma \subseteq \Omega$ that is a closed path, then
\begin{equation*}
  \int_{\gamma} F'(z) \dif{z} = 0
\end{equation*}
% subsection corollary_of_ftc (end)

\subsection{Goursat's Theorem} % (fold)
\label{sub:goursat_s_theorem}
Let $\Omega \subseteq \mathbb{C}$ be open. Sps $\Delta \subseteq \Omega$ is a closed triangle, and $\Delta^0 \subseteq \Omega$, and let $f \in H(\Omega)$. Then
\begin{equation*}
  \int_{\Delta} f(z) \dif{z} = 0
\end{equation*}
% subsection goursat_s_theorem (end)

\subsection{Convex Set} % (fold)
\label{sub:convex_set}
A set $S \subseteq \mathbb{C}$ is a convex set if the line segment joining any pair of points in $S$ lies entirely in $S$.
% subsection convex_set (end)

\subsection{Cauchy's Theorem for Convex Set} % (fold)
\label{sub:cauchy_s_theorem_for_convex_set}
Let $\Omega \subseteq \mathbb{C}$ be a convex open set, and $f \in H(\Omega)$. Then
\begin{enumerate}
  \item $f = F'$ for some $F \in H(\Omega)$
  \item $\int_{\gamma} f(z) \dif{z} = 0$ for any closed path $\gamma \in \Omega$
\end{enumerate}
% subsection cauchy_s_theorem_for_convex_set (end)

\subsection{Cauchy's Integral Formula 1} % (fold)
\label{sub:cauchy_s_integral_formula_1}
Let $\Omega \subseteq \mathbb{C}$ be a convex open set, and $C$ be a closed circle path in $\Omega$. If $w \in \Omega \setminus \partial C$, and $f \in H(\Omega)$, then
\begin{equation*}
  f(w) \ind{w}{C} = \frac{1}{2 \pi i} \int_{C} \frac{f(z)}{z - w} \dif{z}
\end{equation*}
where
\begin{equation*}
  \ind{w}{C} = \frac{1}{2 \pi i} \int_{\gamma} \frac{\dif{z}}{z - w}
\end{equation*}
% subsection cauchy_s_integral_formula_1 (end)

\subsection{Holomorphic Functions as Power Series} % (fold)
\label{sub:holomorphic_functions_as_power_series}
Let $\Omega \subseteq \mathbb{C}$ be an open set, $f \in H(\Omega)$. Then $f$ can be expressed as a power series.
% subsection holomorphic_functions_as_power_series (end)

\subsection{Cauchy's Integral Formula 2} % (fold)
\label{sub:cauchy_s_integral_formula_2}
Let $\Omega \subseteq \mathbb{C}$ be open, $f \in H(\Omega)$. Then
\begin{enumerate}
  \item $\forall w \in \Omega, \, f$ has a power series expansion at $w$
  \item $f$ is differentiable infinitely many times in $\Omega$
  \item $\forall C \subseteq \Omega$ that is a closed circle oriented anticlockwise, $\forall w \in C^0$,
  \begin{equation*}
    f^{(n)}(w) = \frac{n!}{2 \pi i} \int_{C} \frac{f(z)}{(z - w)^{n + 1}} \dif{z}
  \end{equation*}
\end{enumerate}
% subsection cauchy_s_integral_formula_2 (end)

\subsection{Taylor Expansion of Entire Functions} % (fold)
\label{sub:taylor_expansion_of_entire_functions}
If $f$ is entire, then $\forall z_0 \in \mathbb{C}$,
\begin{equation*}
  f(z) = \sum_{n=0}^{\infty} \frac{f^{(n)}(z_0)}{n!} (z - z_))^n
\end{equation*}
% subsection taylor_expansion_of_entire_functions (end)

\subsection{Analytic Functions} % (fold)
\label{sub:analytic_functions}
$f$ is analytic in $\Omega$ if $f$ has a power series expansion $\forall z \in \Omega$.
% subsection analytic_functions (end)

\subsection{Analyticity \& Holomorphicity} % (fold)
\label{sub:analyticity_&_holomorphicity}
It is an iff relation
% subsection analyticity_&_holomorphicity (end)

\subsection{Cauchy's Inequality} % (fold)
\label{sub:cauchy_s_inequality}
$\forall z_0 \in \mathbb{C} \; \forall R > 0 \in \mathbb{R} \; \forall f \in H(C = D(z_0, R))$
\begin{equation*}
  f^{(n)}(z_0) \leq \frac{n!}{R^n} \cdot \sup_{z \in C} \abs{f(z)}
\end{equation*}
% subsection cauchy_s_inequality (end)

\subsection{Liouville's Theorem} % (fold)
\label{sub:liouville_s_theorem}
A bounded entire function $f : \mathbb{C} \to \mathbb{C}$ is a constant.
% subsection liouville_s_theorem (end)

\subsection{Parseveal's Theorem} % (fold)
\label{sub:parseveal_s_theorem}
$\Omega \subseteq \mathbb{C}$ be open, $f \in H(\Omega)$, $\bar{D(z_0, R)} \subseteq \Omega \implies$

$\forall z \in \bar{D(z_0, R)}, \, f(z) = \sum_{n=0}^{\infty} c_n (z - z_0)^n \implies$

$\forall z \in \bar{D(z_0, R)} \enspace f(z_0 + re^{i \theta}) = \sum_{n=0}^{\infty} c_n (re^{i \theta})^n$
% subsection parseveal_s_theorem (end)

\subsection{Parseval's Indentity} % (fold)
\label{sub:parseval_s_indentity}
Same setup as above,
\begin{equation*}
  \frac{1}{2 \pi} \int_{0}^{2 \pi} \abs{f(z_0 = re^{i \theta})}^2 \dif{\theta} = \sum_{n=0}^{\infty} \abs{c_n}^2 r^{2n}
\end{equation*}
% subsection parseval_s_indentity (end)

\subsection{Principle of Analytic Continuation} % (fold)
\label{sub:principle_of_analytic_continuation}
$\Omega \subseteq \mathbb{C}$ open \& connected, $f \in H(\Omega)$. $Z(f) := \{a \in \Omega : f(a) = 0 \}$. Then either $Z(f) = \Omega$ or $Z(f)$ has no limit point (i.e. points where $f = 0$ are isolated)
% subsection principle_of_analytic_continuation (end)

\subsection{Maximum Modulus Principle} % (fold)
\label{sub:maximum_modulus_principle}

$\Omega \subseteq \mathbb{C} \, f \in H(\Omega) \, \exists r > 0 \, D_{z_0} = \bar{D(z_0, r)} \subseteq \Omega \implies$

$\abs{f(z_0)} \leq \max_{z \in \partial D_{z_0}} \abs{f(z)}$ and

$\abs{f(z_0)} = \max_{z \in \partial D_{z_0}} \iff f $ is constant on $\Omega$

% subsection maximum_modulus_principle (end)

\subsection{Fundamental Theorem of Algebra} % (fold)
\label{sub:fundamental_theorem_of_algebra}
$\forall P(z) \in \mathbb{C}[z] \, \deg P(z) = n \in \mathbb{N} \, \exists \alpha_1, \alpha_2, ..., \alpha_n \in \mathbb{C} \, \land \exists A \in \mathbb{C}$
\begin{equation*}
  P(z) = A(z - \alpha_1)(z - \alpha_2) \hdots (z - \alpha_n)
\end{equation*}
% subsection fundamental_theorem_of_algebra (end)

\subsection{Uniqueness of an analytically continued function} % (fold)
\label{sub:uniqueness_of_an_analytically_continued_function}
$\Omega \subseteq \mathbb{C}$ open \& connected, $\forall f, g \in H(\Omega)$

For any $\Omega' \subseteq \Omega$, $\forall z \in \Omega' \; f(z) = g(z) \implies$

$\forall z \in \Omega \; f(z) = g(z)$
% subsection uniqueness_of_a_function (end)

\subsection{Morera's Theorem} % (fold)
\label{sub:morera_s_theorem}
Def $\Omega \subseteq \mathbb{C}$ an open set, $f$ be continuous on $\Omega$. Sps for any triangular path $\Delta \in \Omega$, $\int_{\Delta} f = 0$. Then $f \in H(\Omega)$.
% subsection morera_s_theorem (end)

% section integration (end)

\section{Winding Numbers} % (fold)
\label{sec:winding_numbers}

\subsubsection{Winding Numbers} % (fold)
\label{ssub:winding_numbers}
Def $\gamma$ be a closed path oriented anti-clockwise, $\gamma^*$ the image of $\gamma$ in $\mathbb{C}$, $\Omega = \mathbb{C} \setminus \gamma^*$. $\forall w \in \Omega$

$\ind{w}{\gamma} := \frac{1}{2 \pi i} \int_{\gamma} \frac{\dif{z}}{z - w}$

% subsubsection winding_numbers (end)

\subsection{Theorem for Winding Numbers} % (fold)
\label{sub:theorem_for_winding_numbers}
$\ind{w}{\gamma}$ is
\begin{enumerate}
  \item $\in \mathbb{Z}$;
  \item constant on any connected component $\Omega$; and
  \item $0$ on the unbounded component of $\Omega$.
\end{enumerate}
% subsection theorem_for_winding_numbers (end)
% section winding_numbers (end)

\section{Singularities} % (fold)
\label{sec:singularities}

\subsection{Isolated Singularity} % (fold)
\label{sub:isolated_singularity}
$\forall a \in \mathbb{C}$, $(\exists r > 0, \, f \in H \big(B*(a, r) \big) \, \land \, f(a)$ is undefined $) \iff f$ has an isolated singularity at $z = a$.
% subsection isolated_singularity (end)

\subsection{Classification of Singularities} % (fold)
\label{sub:classification_of_singularities}
Let $z_0 \in \mathbb{C}$ be a singularity of $f$.\\
\textbf{(Removable)} $\exists r > 0 \; \forall z \in D = B(z_0, r) \; \exists g(z) \in H(D) \; \forall z \in B*(z_0, r) \; g(z) = f(z)$\\
\textbf{(Pole)} $\exists r > 0 \; \forall z \in D = B^0(z_0, r) \; \exists A, B \in H(D) \; A(z_0) \neq 0 \, \land \, B(z_0) = 0 \; f(z) = \frac{A(z)}{B(z)}$\\
\textbf{(Essential)} $z_0$ is not removable and not a pole.
% subsection classification_of_singularities (end)

\subsection{Factorization} % (fold)
\label{sub:factorization}
$\Omega \subseteq \mathbb{C}$ open \& connected, $f \in H(\Omega)$, $f \neq 0$ on $\Omega$, $z_0 \in Z(f) \subset \Omega$.\\
$\exists r > 0 \; \forall z \in D = B(z_0, r) \; \exists g \in H(D) \; g(z_0) \neq 0$ \\
$\exists! n \in \mathbb{N} \enspace f(z) = (z - z_0)^n g(z)$
% subsection factorization (end)

\subsection{Criterion for Removable Singularities} % (fold)
\label{sub:criterion_for_removable_singularities}
$f \in H(\Omega \setminus \{z_0\}) \, \land \, \lim_{z \to z_0} (z - z_0) f(z) = 0$\\
$\implies z_0$ is removable.
% subsection criterion_for_removable_singularities (end)

\subsection{Factorization when Poles Exist} % (fold)
\label{sub:factorization_when_poles_exist}
Sps $z_0$ is a pole for $f$, then $\exists r > 0 \; \exists h \in H(B(z_0, r))$ and $\exists! n \in \mathbb{N}$ such that
\begin{equation*}
  f(z) = (z - z_0)^{-n} h(z)
\end{equation*}
% subsection factorization_when_poles_exist (end)

\subsection{Dealing with Poles} % (fold)
\label{sub:dealing_with_poles}
Let $z_0$ be a pole of $f$. Then $\exists r > 0 \; \exists G \in H(B(z_0, r)) \forall z \in B^0(z_0, r)$
\begin{equation*}
  f(z) = \sum_{j=1}^{n} \frac{c_{-j}}{(z - z_0)^j} + G(z)
\end{equation*}
% subsection dealing_with_poles (end)

\subsection{Principal Part} % (fold)
\label{sub:principal_part}
From the above, the sum
\begin{equation*}
  \sum_{j=1}^{n} \frac{c_{-j}}{(z - z_0)^j}
\end{equation*}
% subsection principal_part (end)

\subsection{Residue} % (fold)
\label{sub:residue}
From the above, the coefficient $c_{-1}$, denoted $\res{f(z)}{z = z_0}$.
% subsection residue (end)

\subsection{Casorati-Weierstrass} % (fold)
\label{sub:casorati_weierstrass}
Let $z_0 \in \Omega$ and $f \in H(\Omega \setminus \{z_0\})$. Sps $z_0$ is a singularity of $f$. Then either one of the following is true:
\begin{enumerate}
  \item $z_0$ is removable;
  \item $\exists m \in \mathbb{N} \; \exists \{c_j\}_{j = 1}^{m} \subseteq \mathbb{C} \; f(z) - \sum_{j=1}^{m} \frac{c_j}{(z - z_0)^j}$ has a removable singularity at $z_0$;
  \item (essential singularity) $\forall r > 0 \; \exists B(z_0, r) \subseteq \Omega$ such that $f \big( B^0 (z_0, r) \big)$ is dense in $\mathbb{C}$.
\end{enumerate}
% subsection casorati_weierstrass (end)

% section singularities (end)

\section{Residue Theorem} % (fold)
\label{sec:residue_theorem}

\subsection{Meromorphic Functions} % (fold)
\label{sub:meromorphic_functions}
$f$ is meromorphic if $\exists \mathscr{A} \subseteq \Omega \subseteq \mathbb{C}$ such that
\begin{itemize}
  \item $\mathscr{A}^* = \emptyset$;
  \item $f \in H(\Omega \setminus \mathscr{A})$;
  \item $\forall z \in \mathscr{A}$, $f$ has a pole (of finite order) on $z$.
\end{itemize}

% subsection meromorphic_functions (end)

\subsection{Cauchy's Residue Theorem} % (fold)
\label{sub:cauchy_s_residue_theorem}
\begin{equation*}
  \frac{1}{2 \pi i} \int_{\gamma} f = \sum_{a \in \mathscr{A}} \res{f}{z = z_0} \ind{z_0}{\gamma}
\end{equation*}
% subsection cauchy_s_residue_theorem (end)

% section residue_theorem (end)

\subsection{Argument Principle} % (fold)
\label{sub:argument_principle}

\subsubsection{Argument Principle} % (fold)
\label{ssub:argument_principle}
Sps $\Omega \in \mathbb{C}$ is open \& connected, $f$ is meromorphic in $\Omega$, $\gamma$ is a closed path s.t. $\gamma^* \in \subseteq \Omega \setminus \big(\mathscr{A} \cup Z(f) \big)$,
\begin{itemize}
  \item $\forall w \notin \Omega \; \ind{w}{\gamma} = 0$; and
  \item $\forall w \in \Omega \setminus \gamma^* \; \ind{w}{\gamma} = 0$ or $1$.
\end{itemize}
Then
\begin{equation*}
  \frac{1}{2 \pi i} \int_{\gamma} \frac{f'}{f} = \abs{Z(f) \cap \gamma^0} - \abs{\mathscr{A} \cap \gamma^0}
\end{equation*}
counted with multiplicity.
% subsubsection argument_principle (end)

\subsection{Rouch\'e's Theorem} % (fold)
\label{sub:rouche_s_theorem}
Let $\Omega \subseteq \mathbb{C}$ be open \& connected, $f, g \in H(\Omega)$, $\gamma$ a closed path on $\Omega$ with
\begin{itemize}
  \item $\forall w \notin \Omega \; \ind{w}{\gamma} = 0$; and
  \item $\forall w \in \Omega \setminus \gamma^* \; \ind{w}{\gamma} = 0$ or $1$.
\end{itemize}
$\forall z \in \gamma^* \enspace \abs{g(z) - f(z)} < \abs{f(z)}$\\
$\implies \abs{Z(f) \cap \gamma^0} - \abs{\mathscr{A}_f \cap \gamma^0} = \abs{Z(f) \cap \gamma^0} - \abs{\mathscr{A}_g \cap \gamma^0}$
% subsection rouche_s_theorem (end)

\subsection{Open Mapping Theorem} % (fold)
\label{sub:open_mapping_theorem}
$f \in H(\Omega)$ is non-constant $\implies$ $f$ maps open sets to open sets.
% subsection open_mapping_theorem (end)

% subsection argument_principle (end)

\section{Logarithms} % (fold)
\label{sec:logarithms}

\subsection{Principal Branch} % (fold)
\label{sub:principal_branch}
\begin{equation*}
  \Log z = \log r + i \theta
\end{equation*}
% subsection principal_branch (end)

\subsection{Gamma Function} % (fold)
\label{sub:gamma_function}
\begin{equation*}
  \Gamma(s) = \int_{0}^{\infty} e^{-t} t^s \frac{\dif{t}}{t}
\end{equation*}
% subsection gamma_function (end)

\subsection{Analytic Continuation of Gamma Function} % (fold)
\label{sub:analytic_continuation_of_gamma_function}
\begin{equation*}
  \Gamma(s) = \begin{cases}
    \int_{0}^{\infty} e^{-t} t^s \frac{\dif{t}}{t} & s \in \mathbb{N} \\
    \frac{\Gamma(0)}{\prod_{j=0}^{\abs{s}} (s - j)} & s \in \mathbb{Z} \setminus \mathbb{N}
  \end{cases}
\end{equation*}
% subsection analytic_continuation_of_gamma_function (end)

\subsection{Function without zeros} % (fold)
\label{sub:function_without_zeros}
$f \in H(\mathbb{C}) \, \land \, Z(f) = \emptyset$\\
$\implies \exists g \in H(\mathbb{C}) \; A \in \mathbb{C} \enspace f = Ae^g$
% subsection function_without_zeros (end)

% section logarithms (end)

\section{Infinite Products} % (fold)
\label{sec:infinite_products}

\subsection{Infinite Product} % (fold)
\label{sub:infinite_product}
\begin{equation*}
  \prod_{j=1}^{\infty} (1 + u_j),
\end{equation*}
with $\{u_j\}_{j \in \mathbb{N}} \subseteq \mathbb{C}$, is called an infinite product if it converges.
% subsection infinite_product (end)

\subsection{Bounds of the Partial Product} % (fold)
\label{sub:bounds_of_the_partial_product}
Let $\{u_j\}_{j \in \mathbb{N}} \subseteq \mathbb{C}$ and $P_N^* = \prod_{j=1}^{N} \big( 1 + \abs{u_j} \big)$ be the partial product. Then
\begin{itemize}
  \item $P_N^* \leq \exp{\left( \sum_{j=1}^{N} \abs{u_j} \right)}$;
  \item $\forall N \in \mathbb{N} \; \abs{P_N - 1} \leq P_N^* - 1$
\end{itemize}
% subsection bounds_of_the_partial_product (end)

\subsection{Criterion for Convergence} % (fold)
\label{sub:criterion_for_convergence}
$\sum_{n=1}^{\infty} \abs{u_n}$ converges $\implies \prod_{n=1}^{\infty} (1 + u_n)$ converges.\\
Moreover, $\sum_{n=1}^{\infty} (1 + u_n) \to 0 \iff \exists n_0 \in \mathbb{N} \; u_{n_0} = -1$.\\
\textbf{Note:} $\sum u_n$ also needs to converge uniformly.
% subsection criterion_for_convergence (end)

\subsection{Notations} % (fold)
\label{sub:notations}
$P_n(z) := z + \frac{z^2}{2} + \frac{z^3}{3} + \hdots + \frac{z^n}{n}$\\
$E_n(z) := (1 - z) \exp{\big( P_n (z) \big)}$\\
$E_0(z) := (1 - z)$
% subsection notations (end)

\subsection{An Inequality} % (fold)
\label{sub:an_inequality}
For $\abs{z} \leq 1$, $\abs{1 - E_n(z)} \leq \abs{z}^{n + 1}$.
% subsection an_inequality (end)

\subsection{Weierstrass Product} % (fold)
\label{sub:weierstrass_product}
Let $\abs{a_n}_{n \in \mathbb{N}} \subseteq \mathbb{C}$ with $a_n \to \infty$ as $n \to \infty$ and $0 \notin \{a_n\}_{n \in \mathbb{N}}$. The Weierstrass Product
\begin{equation*}
   f(z) = \prod_{n=1}^{\infty} E_n \left( \frac{z}{a_n} \right)
\end{equation*}
is entire and has zeros at exactly each element of $\{a_n\}_{n \in \mathbb{N}}$.
% subsection weierstrass_product (end)

% section infinite_products (end)

\section{Other notes} % (fold)
\label{sec:other_notes}

\begin{itemize}
  \item Use Rouch\'e's Theorem to look for number of zeros in a radius for a polynomial.
  \item Can use Cauchy's Residue Theorem to solve certain tricky real integrals.
\end{itemize}

\subsection{Commonly Used Identities} % (fold)
\label{sub:commonly_used_identities}

\begin{itemize}
  \item $\frac{1}{1 - z} = \sum_{k=0}^{\infty} z^k$
  \item $e^z = \sum_{n=0}^{\infty} \frac{z^n}{n!}$
\end{itemize}

% subsection commonly_used_identities (end)

\subsection{Useful Proven Results} % (fold)
\label{sub:useful_proven_results}

\begin{itemize}
  \item $\int_{0}^{\infty} \cos t^2 \dif{t} = \int_{0}^{\infty} \sin t^2 \dif{t} = \frac{1}{2} \sqrt{\frac{\pi}{2} }$
  \item $\lim_{e \to 0} \int_{C(\epsilon)} f = - \pi i \res{f}{z = 0}$ for a \textbf{clockwise} semi-circular arc $C(\epsilon)$
  \item L'H\^opital's rule works for complex functions
\end{itemize}

% subsection useful_proven_results (end)

% section other_notes (end)

% The following are dummies to squeeze the last column to look good
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\
$ $ \\

\end{multicols*}
\end{document}


\documentclass[twocolumn]{article}
\usepackage{amssymb}
\usepackage[english]{babel}
\usepackage{hyperref}
\usepackage{mathtools}
\usepackage[sections,textmath,displaymath]{preview}
\setlength{\parskip}{1em}
\begin{document}
\title{Deep Learning}
\author{Jorge Diego Hernandez Medina}
\maketitle

\begin{abstract}
This talk gives a brief history of Deep Learning as well as a rough overview of
the current techniques and future developments in the field. The talk was
written for  Nearsoft's October Talks of 2015.
\end{abstract}

\section{Introduction}
(Slide 2) Good morning, my name is Jorge Hernandez and I'm a software
engineer here at Nearsoft. I learned to program 32 years ago and have been doing
it professionally for the last 15 years.

I became a Computer Scientist because I've always been fascinated by the
possibility of creating artificial minds. Today I'm going to talk about one of
the most promising technologies that we have to make that happen: Deep Learning.

\section{What is Deep Learning?}
(Slide 4) Representation learning is a set of methods that allow a machine to be
fed with raw data and to automatically discover the representations needed for
detection or classification. Deep-learning methods are representation-learning
methods with multiple levels of representation, obtained by composing simple but
non-linear modules that each transform the representation at one level (starting
with the raw input) into a representation at a higher, slightly more abstract
level. With the composition of enough such transformations, very complex
functions can be learned. For classification tasks, higher layers of
representation amplify aspects of the input that are important for
discrimination and suppress irrelevant variations.

An image, for example, comes in the form of an array of pixel values, and the
learned features in the first layer of representation typically represent the
presence or absence of edges at particular orientations and locations in the
image. The second layer typically detects motifs by spotting particular
arrangements of edges, regardless of small variations in the edge positions. The
third layer may assemble motifs into larger combinations that correspond to
parts of familiar objects, and subsequent layers would detect objects as
combinations of these parts. The key aspect of deep learning is that these
layers of features are not designed by human engineers: they are learned from
data using a general-purpose learning procedure.

\section{Neural Nets}
(Slide 6) Frank Rosenblatt was a psychologist at the Cornell Aeronautical
Laboratory. He was inspired by the work of Donald O. Hebb, who a decade earlier
had predicted how learning might work: As one neuron fires and activates
another, repeatedly, the cells improve their joint efficiency. ``The cells that
fire together, wire together'', as cognitive scientists like to say. This simple
idea, Rosenblatt thought, was enough to build a machine that could learn to
identify objects. In 1957 he built it, and called it the Perceptron.

(Slide 7) The Perceptron took up an entire lab room, it worked in three
layers. At one end, a square cluster of 400 light sensors simulated a retina;
the sensors connected multiple times to an array of 512 electrical triggers,
each of which fired, like a neuron, when it passed a certain adjustable
threshold of excitement. These triggers then connected to the last layer, which
would signal if an object matched what the Perceptron had been trained to see.

(Slide 8) The Perceptron algorithm is a linear classifier.

1. Initialize the weights and the threshold. Weights may be initialized to 0 or
to a small random value. In the example below, we use 0.

2. For each example \(j\) in our training set \(D\), perform the following steps
over the input \(x_{j}\), and desired output \(d_{j}\):

a. Calculate the actual output:

\(y_j(t) = f[\mathbf{w}(t)\cdot\mathbf{x}_j] = f[w_0(t) + w_1(t)x_{j,1} +
w_2(t)x_{j,2} + \dotsb + w_n(t)x_{j,n}]\)

b. Update the weights:

\(w_i(t+1) = w_i(t) + \alpha (d_j - y_j(t)) x_{j,i} \) for all \(0 \leq i \leq
n\), where \(0 < \alpha \leq 1\) is the learning rate.

(Slide 9) It was a thrilling development, and Rosenblatt wasn't afraid to share
it. In the summer of 1958, he held a news conference with his sponsor, the
U.S. Navy. As so often happens in science, he began to talk about the future. To
researchers then, he sounded foolish; heard today, prescient. The New York Times
caught the gist of it:

``The Navy revealed the embryo of an electronic computer today that it expects
will be able to walk, talk, see, write, reproduce itself and be conscious of its
existence. … Later Perceptrons will be able to recognize people and call out
their names and instantly translate speech in one language to speech and writing
in another language.''

Rosenblatt’s fame irked his peers, many of whom had opted to pursue rules-based
artificial intelligence; both sides were chasing after the same military
research dollars.

(Slide 10) In 1969 Minsky and Seymour Papert (both at MIT)
published \textit{Perceptrons: An Introduction to Computational Geometry}
criticizing Rumelhart's connectionist approach.

(Slide 11)  The book proved that not only can't a single layer perceptron
emulate XOR or XNOR, but also that in three-layered feed-forward perceptrons
(with a so-called  ``hidden'' or ``intermediary'' layer), it is not possible to
compute some predicates unless at least one of the neurons in the first layer of
neurons (the  ``intermediary'' layer) is connected with a non-null weight to
each and every input. Furthermore perceptrons are unable to solve the
connectivity problem.

The effect of the book was devastating: virtually no research at all was done
in connectionism for 10 years. Eventually, a new generation of researchers would
revive the field and thereafter it would become a vital and useful part of
artificial intelligence. Rosenblatt would not live to see this, as he died in a
boating accident shortly after the book was published.

(Slide 12) In the mid-80s neural networks made a comeback, when David
E. Rumelhart and others found a way of teaching more complex functions to a
neural net. To accomplish this they designed an algorithm they called
backpropagation.

(Slide 13) The backpropagation algorithm looks for the minimum of the error
function in weight space using the method of gradient descent:

\(E(\mathbf{w}) = \displaystyle\sum_{j} {(y_j - f_N(\mathbf{x_j}))}^2\)

\(\nabla E = (\frac{\partial E}{\partial w_{1}},\dotsc,\frac{\partial
  E}{\partial w_{n}})\)

Where \(f_N(\mathbf{x_j}) = \sigma(\mathbf{x_j})\), \(\sigma(x) =
\frac{1}{1+e^{-x}}\), and \(\frac{\partial E}{\partial w_{i}}\) is the partial
derivative of the error function with respect to the \(i^{th}\) weight. For a
fixed set of weights \(\nabla E\) is a vector in \(\mathbb{R}^n\) that points in
the direction of steepest ascent of the function \(E\).

\(\displaystyle \frac{\partial E}{\partial w_{k,j}} = -\sum_i (y_i -
o_i)\frac{\partial o_i}{\partial w_{k,j}}\)

(Slide 14) Where \(o_i = f_{N_{1,j}}(\dots) = \sigma(f(\dots))\) represents the
value of the impulse function at each of the output neurons. And after applying
the chain rule:

\(\displaystyle \frac{\partial E}{\partial w_{k,j}} = -\sum_i (y_i -
o_i)\sigma'(f(\dots)) w_{j,i}\)

And since \(\sigma'(x) = \sigma(x)(1 - \sigma(x))\) our update rule (for the
weights of \(N_j\)) is:

\(\displaystyle \mathbf{w} = \mathbf{w} + \eta o_j(1-o_j) \left ( \sum_i
    w_{j,i} (y_i - o_i) \right ) \mathbf{z}\)

Where by \(\mathbf{z}\) we denote the vector of inputs to the neuron in
question, and \(\eta\) is the learning parameter.

(Slide 15) So then the backpropagation algorithm works as follows:

1. Initialize the weights in the network at random.

2. Evaluate an input \(\mathbf{x}\) by feeding it forward through the network
and recording at each internal node the output value \(o_j\), and call the final
output \(o\).

3. Compute the error for that output value, propagate the error back to each of
the nodes feeding into the output node, and update the weights for the output
node using our update rule. Repeat this error propagation followed by a weight
update for each of the nodes feeding into the output node in the same way,
compute the updates for the nodes feeding into those nodes, and so on until the
weights of the entire network are updated.

4. Repeat 2 and 3 with a new input \(\mathbf{x}\).

Unfortunately the algorithm requires a lot of computation and a large training
data set, and when it fails the reasons are not clear. Applying the same
algorithm can give different results at different times. This made people choose
other methods to work with (like Support Vector Machines) and neural networks
faded into obscurity once again.

\section{The Return of Neural Networks}
(Slide 17)  Then in 2006 in a series of papers,
Geoffrey Hinton, Yoshua Bengio and Yan LeCun revived interest in what they now
called \textit{''Deep Learning”}.

(Slide 18) Hinton, G. E., Osindero, S. \& Teh, Y.-W. A fast learning algorithm
for deep belief nets. \textit{Neural Comp.} \textbf{18}, 1527–1554

This paper introduced a novel and effective way of training very deep neural
networks by pre-training one hidden layer at a time using the unsupervised
learning procedure for restricted Boltzmann machines.

Bengio, Y., Lamblin, P., Popovici, D. \& Larochelle, H. Greedy layer-wise
training of deep networks. In \textit{Proc. Advances in Neural Information
Processing Systems 19} 153–160

This report demonstrated that the unsupervised pre-training method
introduced in the first paper significantly improves performance on test data
and generalizes the method to other unsupervised representation learning
techniques, such as autoencoders.

Ranzato, M., Poultney, C., Chopra, S. \& LeCun, Y. Efficient learning of sparse
representations with an energy-based model. In \textit{Proc. Advances in Neural
Information Processing Systems 19} 1137 – 1144

Coupled with the appearance of more generalized GPUs that allowed the
computations to be done much faster than before (some nets saw speedups of x30),
this led to, for example, Google moving all its Android phones over to use deep
learning for speech recognition.

Now I want to tell you about two of the methods being used in deep learning.

\section{Convolutional Neural Networks}
(Slide 20) Colloquially called ConvNets, Convolutional Neural Networks are
designed to process data that come in the form of multiple arrays, \textit{e.g.}
a color image composed of three 2D arrays containing pixel intensities in the
RGB channels. Data that comes in the form of multiple arrays are: 1D for
language; 2D for images and 3D for video.

There are four key ideas behind ConvNets that take advantage of the properties
of natural signals: local connections, shared weights, pooling and the use of
many layers.

(Slide 21) As we saw the simplest way to wire a neural network is to connect
every to every neuron. But if instead we take advantage of the symmetry in the
properties we look for in the data (e.g the frequency of sounds around a given
time, or the colors around a particular pixel), we can create a group of
neurons, \(A\), that look at small, local segments. That group is called a
convolutional layer.

Convolutional layers are often interweaved with pooling layers. For example the
max-pooling layer is extremely popular. From a high level perspective, we don’t
care about precisely where or when a feature is present. A max-pooling layer
takes the maximum of features over small blocks of a previous layer. The output
tells us if a feature was present in a region of the previous layer, but not
exactly where (as if we had ``zoomed out''). They allow later convolutional layers
to work on larger sections of the data, and they also make the output invariant
to some very small transformations of the data (like noise).

In traditional convolutional layers, \(A\) is a bunch of neurons in parallel,
that all get the same inputs and compute different features, but in a recent
paper \textit{Network In Network} by Min Lin, Qiang Chen and Shuicheng Yan a new
``Mlpconv'' layer is proposed. There A would have multiple layers of neurons, with
the final layer outputting higher level features for the region.

(Slide 22) What is a convolution? A convolution is a mathematical operation on
two functions f and g, producing a third function that is typically viewed as a
modified version of one of the original functions.

Imagine we drop a ball from some height onto the ground, where it only has one
dimension of motion. How likely is it that a ball will go a distance c if you
drop it and then drop it again from above the point at which it landed?

In order to find the total likelihood of the ball reaching a total distance of
c, we can’t consider only one possible way of reaching c. Instead, we consider
all the possible ways of partitioning c into two drops a and b and sum over the
probability of each way.

\((f \ast g)(c) = \sum_{a+b=c} f(a) \cdot f(b)\)

\section{Recurrent Neural Networks}
(Slide 24) Recurrent Neural Networks process an input sequence one element at a
time, maintaining in their hidden units a ‘state vector’ that implicitly
contains information about the history of all the past elements of the
sequence. We can consider the outputs of the hidden units at different discrete
time steps as if they were the outputs of different neurons in a deep multilayer
network.

(Slide 25) The artificial neurons (for example, hidden units grouped under node
\(s\) with values \(s_t\) at time \(t\)) get inputs from other neurons at
previous time steps (this is represented with the black square, representing a
delay of one time step, on the left). So the value of \(s_t\) is calculated
based on the previous hidden state and the input at the current step:

\(s_t=f(Ux_t + Ws_{t-1})\)

The function \(f\) usually is a nonlinearity such as \(\tanh\) or ReLU
(rectified linear unit), and \(s_{-1}\), which is required to calculate the
first hidden state, is typically initialized to all zeroes.

In this way, a recurrent neural network can map an input sequence with elements
\(x_t\) into an output sequence with elements \(o_t\), with each \(o_t\)
depending on all the previous \(x_{tʹ}\) (for \(tʹ \leq t\)). For example, if we
wanted to predict the next word in a sentence it would be a vector of
probabilities across our vocabulary:

\( o_t = \mathrm{softmax}(Vs_t)\)

The same parameters (matrices \(U\), \(V\), \(W\)) are used at each time
step. Many other architectures are possible, including a variant in which the
network can generate a sequence of outputs (for example, words), each of which
is used as inputs for the next time step. The backpropagation algorithm can be
directly applied to the computational graph of the unfolded network on the
right, to compute the derivative of a total error (for example, the probability
of generating the right sequence of outputs) with respect to all the states
\(s_t\) and all the parameters.

\section{The Future of Deep Learning}
(Slide 27) Currently why deep learning works so well is not very well
understood. Some recent work that hints at why and how has been done using a
wavelet scattering transform (which is more amenable to mathematical analysis)
using a deep convolution network architecture.

Mark Tygert and others have done work on complex-valued convolutional networks
where they have proved that the filter that fits a sequence of inputs is a
convolution followed by some mathematical manipulation.

Recently Pankaj Mehta and David Schwab, proved that the statistical technique
called ``renormalization''  (used by physicists to accurately describe systems
without knowing the exact state of all their component parts) also enables
neural networks to categorize data as, regardless of transformations such as
color, size or orientation.

(Slide 28) Unsupervised learning had a catalytic effect in reviving interest in
deep learning, but has since been overshadowed by the successes of purely
supervised learning. We expect unsupervised learning to become far more
important in the longer term. Human and animal learning is largely unsupervised:
we discover the structure of the world by observing it, not by being told the
name of every object.

Over the past year, several authors have made different proposals to  augment
RNNs with a memory module. Proposals include the Neural Turing Machine in which
the network is augmented by a ‘tape-like’ memory that the RNN can choose to
read from or write to, and memory networks, in which a regular network is
augmented by a kind of associative memory. Memory networks have yielded
excellent performance on standard question-answering benchmarks. The memory is
used to remember the story about which the network is later asked to answer
questions.

Ultimately, major progress in artificial intelligence will come about through
systems that combine representation learning with complex reasoning. Although
deep learning and simple reasoning have been used for speech and handwriting
recognition for a long time, new paradigms are needed to replace rule-based
manipulation of symbolic expressions by operations on large vectors.

\section{Resources}
The following are some resources I found useful while writing this talk.

While it is not an inclusive list I've tried to provide a balance between
tutorials and primary sources. Most of the code you'll find for deep learning is
in Python.

\begin{enumerate}
\item\url{http://jmozah.github.io/links/}
\item\url{http://colah.github.io/}
\item\url{http://www.thetalkingmachines.com/blog/}
\item\url{http://www.cs.toronto.edu/~hinton/deeprefs.html}
\item\url{http://www.docdroid.net/11p1b/hinton.pdf.html}
\item\url{http://deeplearning.stanford.edu/tutorial/}
\item\url{http://deeplearning.net/tutorial/contents.html}
\item\url{http://jeremykun.com/2012/12/09/neural-networks-and-backpropagation/}
\item\url{http://karpathy.github.io/2015/05/21/rnn-effectiveness/}
\item\url{http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/}
\item\url{https://docs.google.com/file/d/0BxKBnD5y2M8NVHRiVXBnOVpiYUk/edit}
\item\url{https://drive.google.com/file/d/0BxKBnD5y2M8NbWN6XzM5UXkwNDA/view?pli=1}
\item\url{https://www.youtube.com/watch?v=EK61htlw8hY}
\item\url{http://blog.shakirm.com/wp-content/uploads/2015/10/Bayes_Deep.pdf}
\item\url{http://arxiv.org/abs/1312.4400}
\item\url{http://arxiv.org/pdf/1503.03438v1.pdf}
\item\url{http://arxiv.org/pdf/1203.1513.pdf}
\item\url{http://arxiv.org/abs/1410.3831}
\item\url{https://www.quantamagazine.org/20141204-a-common-logic-to-seeing-cats-and-cosmos/}
\item\url{https://charlesmartin14.wordpress.com/2015/04/01/why-deep-learning-works-ii-the-renormalization-group/}
\end{enumerate}

\section{Closing}
(Slide 30) Does anyone have any questions? Q\&A

(Slide 31) I'd like to thank everyone of you for your time and for listening to
me ramble on.
\end{document}


\documentclass[12pt]{article}
\usepackage{amsfonts, amsmath, latexsym, epsfig}
\usepackage{epsf}
\usepackage{url}
\title{Manual of the matlab scripts of {\tt LP Bathymetry}}
\topmargin=-0.5truein
\oddsidemargin=0.25truein
\evensidemargin=0.25truein
\textwidth=6truein
\textheight=9truein
\urlstyle{sf}

%\usepackage{vmargin}
%\setpapersize{custom}{21cm}{29.7cm}
%\setmarginsrb{1.7cm}{1cm}{1.7cm}{3.5cm}{0pt}{0pt}{0pt}{0pt}
%marge gauche, marge haut, marge droite, marge bas.

\author{Mathieu Dutout Sikiri\'c}

\begin{document}
\newcommand{\RR}{\ensuremath{\mathbb{R}}}
\newcommand{\NN}{\ensuremath{\mathbb{N}}}
\newcommand{\QQ}{\ensuremath{\mathbb{Q}}}
\newcommand{\CC}{\ensuremath{\mathbb{C}}}
\newcommand{\ZZ}{\ensuremath{\mathbb{Z}}}
\newcommand{\TT}{\ensuremath{\mathbb{T}}}
\newtheorem{proposition}{Proposition}
\newtheorem{theorem}{Theorem}
\newtheorem{corollary}{Corollary}
\newtheorem{lemma}{Lemma}
\newtheorem{problem}{Problem}
\newtheorem{conjecture}{Conjecture}
\newtheorem{claim}{Claim}
\newtheorem{remark}{Remark}
\newtheorem{definition}{Definition}
\newcommand{\qed}{\hfill $\Box$ }
\newcommand{\proof}{\noindent{\bf Proof.}\ \ }

\maketitle

When one uses the ROMS model, one needs to smooth the bathymetry in
order to get realistic results.
Two roughness factors are involved: the $rx_0$ factor of
Beckman and Haidvogel:
\begin{equation*}
rx_0=\max_{e\equiv e'} \frac{|h(e) - h(e')|}{h(e)+h(e')}
\end{equation*}
which should not go above $0.2$ and the $rx_1$ factor of Haney which
should not be above $6$ \cite{haney}.
(both $rx_0$ and $rx_1$ are shown up at the beginning of a ROMS run).

The original physical bathymetry as computed by interpolation and
sampling is often too rough for the models and a smoothing operation
is needed.
The programs exposed here try given a roughness factor to find the
bathymetry that is nearest to the real one.
More details are given in \cite{lpmeth}.

The factor that matters is actually the $rx_1$ number which is required
to be small. The problem is that it is quite difficult to optimize with
respect to $rx_1$. The idea is to assume that there is a multiplicating
factor between $rx_0$ and $rx_1$, i.e. $rx_1= C rx_0$ and to optimize
$rx_0$ instead of $rx_1$.
This works quite well for {\tt Vtransform=1} but not for the other
transformations that were introduced later. Then a possible solution
is to optimize with respect to a varying factor $rx_0$. The appropriate
functions are provided.

%The program exposed here is based on linear programming and
%given a bathymetry $h^{real}$ and a fixed $r$,
%it finds the unique bathymetry $h^{filt}$ satisfying to
%\begin{equation*}
%\max_{e\equiv e'} \frac{|h^{filt}_e - h^{filt}_{e'}|}{h^{filt}_e+h^{filt}_{e'}}\leq r
%\end{equation*}
%and minimizing $\sum_e |h^{real}(e) - h^{filt}(e)|$.

\section{Availability}

The source of the program is available from \url{http://www.liga.ens.fr/~dutour/Bathymetry/index.html}

The linear programs are solved by the program {\tt lpsolve} (see \cite{lpsolve} for the installation). Note that we do not use the mex facility but the standalone program.
The scripts are {\tt matlab$^{\copyright}$} scripts and so you need to have {\tt matlab$^{\copyright}$} installed.

\section{How to use it}

First of all, you need your bathymetry in the form of an array of the
form {\tt Hobs(eta\_rho, xi\_rho)} and a mask {\tt MSK(eta\_rho, xi\_rho)}.

\subsection{Using {\tt GRID\_LinProgHeuristic}}

The command to do the filtering is then
\begin{verbatim}
>> Hfielt=GRID_LinProgHeuristic(MSK, Hobs, rx0max);
\end{verbatim}
with
\begin{enumerate}
\item {\bf MSK(eta\_rho,xi\_rho)} the mask.
\item {\bf Hobs(eta\_rho,xi\_rho)} the bathymetry.
\item {\bf rx0max} the chosen maximal $rx_0$ factor.
\end{enumerate}
The program uses a divide and conqueer strategy for reducing the time of
the run, that is it uses as subroutine {\bf GRID\_LinearProgrammingSmoothing\_rx0\_simple}, which may be used separately if desired.
If some additional constraint are needed, have a look at
{\bf GRID\_LinearProgrammingSmoothing\_rx0}.

%\subsection{Using {\tt GRID\_LinearProgrammingSmoothing\_rx0}}
%
%Suppose that you want at some point the bathymetry to only increase or
%at other point to only decrease.
%What you need to do is specify the array {\tt SignConst(eta\_rho, xi\_rho)}
%with
%
%Also, you may want to specify the maximum value of the perturbation
%at a given point. What you need to do is specify the array {\tt AmpConst(eta\_rho, xi\_rho)} with {\tt a=AmpConst(iEta, iXi)} being a constraint such that
%{\tt |Hfilt(iEta,iXi)-Hobs(iEta, iXi)| $\leq$ a $\times$ Hobs(iEta, iXi)}.
%
%
%The command to do the filtering is then
%\begin{verbatim}
%>> Hfilt=GRID_LinearProgrammingSmoothing_rx0(...
%     MSK, Hobs, rx0max, SignConst, AmpConst);
%>> Hfilt=GRID_LinearProgrammingSmoothing_rx0_simple(MSK, Hobs, rx0max);
%\end{verbatim}
%with
%\begin{enumerate}
%\item {\bf MSK(eta\_rho,xi\_rho)} the mask.
%\item {\bf Hobs(eta\_rho,xi\_rho)} the bathymetry.
%\item {\bf rx0max} the chosen maximal $rx_0$ factor.
%\item {\bf SignConst(eta\_rho,xi\_rho)} the sign matrix
%\begin{itemize}
%\item {\tt SignConst(iEta, iXi)=1} if only bathymetry increase are allowed
%at the point {\tt (iEta,iXi)}.
%\item {\tt SignConst(iEta, iXi)=-1} if only bathymetry decrease are allowed
%at the point {\tt (iEta,iXi)}.
%\item {\tt SignConst(iEta, iXi)=0} if bathymetry is allowed to decrease and
%increase at the point {\tt (iEta,iXi)}.
%\end{itemize}
%\item {\bf AmpConst(eta\_rho,xi\_rho)} for the maximal amplitude of perturbation.\\
%We will have the relation
%\begin{equation*}
%|Hfilt(iEta,iXi)-Hobs(iEta, iXi)|\leq AmpConst(iEta,iXi) \times Hobs(iEta, iXi).
%\end{equation*}
%
%\end{enumerate}

\subsection{Using {\tt GRID\_LinearProgrammingSmoothing\_rx0\_volume}}

If you want to preserve the total volume, then a variation of the
above is:
\begin{verbatim}
>> Hfilt=GRID_LinearProgrammingSmoothing_rx0_volume(MSK, Hobs, rx0max, AreaMatrix);
\end{verbatim}
with
\begin{enumerate}
\item {\bf MSK(eta\_rho,xi\_rho)} the mask of the grid
\item {\bf Hobs(eta\_rho,xi\_rho)} the observed bathymetry of the grid
\item {\bf AreaMatrix(eta\_rho,xi\_rho)} the areas of the wet and land $\rho$-points of the grid.
\item {\bf rx0max, rx1max} are roughness factors.
\end{enumerate}

\subsection{Using {\tt GRID\_LinProgSmoothVertVert\_rx0}}

Sometimes, you want to smooth the bathymetry but preserve the
total volume.
Here the method is significantly different: We increase the bathymetry
at one cell $e$ by say, $\delta_{e,e'}$ and decrease it at an adjacent
cell $e'$ by $\delta_{e,e'}$.
We minimize the quantity
\begin{equation*}
\sum_{e\equiv e'} |\delta_{e,e'}|
\end{equation*}
This method obviously preserve the volume and tend to preserve the volume
of structures like basin and seamounts.

This method is used in the following way.
\begin{verbatim}
>> Hfilt=GRID_LinProgSmoothVertVert_rx0(MSK, Hobs, r);
\end{verbatim}
with $r$ the roughness factor you want to achieve.
The problem of this method is its high computational cost since
the number variable is higher.

\subsection{Using {\tt GRID\_LinProgHeuristic\_rx0\_fixed}}

This command corrects the bathymetry (if possible) and leaves the bathymetry of a set of points invariant
\begin{verbatim}
>> Hfilt=GRID_LinProgHeuristic_rx0_fixed(MSK, Hobs, PRS, r);
Hfilt=
\end{verbatim}
with
\begin{enumerate}
\item {\bf MSK(eta\_rho,xi\_rho)} the mask of the grid.
\item {\bf Hobs(eta\_rho,xi\_rho)} the original bathymetry of the grid.
\item {\bf PRS(eta\_rho,xi\_rho)} the list of grid point for which
we want to preserve the bathymetry
(PRS(iEta, iXi) == 1 if we want to preserve it).
\item {\bf rx0max} is the maximum $rx_0$ factor.
\end{enumerate}
The program uses a divide and conqueer strategy for reducing the time of
the run, that is it uses as subroutine {\bf GRID\_LinearProgrammingSmoothing\_rx0\_fixed}, which may be used separately if desired.

%\subsection{Using {\tt GRID\_LinearProgrammingSmoothing\_rx0\_fixed}}
%This command uses linear programming for smoothing the bathymetry
%but leaves a set of cells unchanged, which can be useful if for example
%the boundary is to be preserved.
%\begin{verbatim}
%>> Hfilt=GRID_LinearProgrammingSmoothing_rx0_fixed(MSK, Hobs, PRS, rx0max);
%\end{verbatim}
%with
%\begin{enumerate}
%\item {\bf MSK(eta\_rho,xi\_rho)} the mask of the grid
%\item {\bf Hobs(eta\_rho,xi\_rho)} the observed bathymetry of the grid
%\item {\bf PRS(eta\_rho,xi\_rho)} the set of points preserved $1$ for preserved, $0$ for movable.
%\item {\bf rx0max} the maximal roughness factor.
%\end{enumerate}

%\subsection{Using {\tt GRID\_LinProgHeuristic}}
%
%This command uses a splitting procedure to dramatically speed up the
%smoothing.
%\begin{verbatim}
%>> Hfilt=GRID_LinProgHeuristic(MSK, Hobs, r);
%\end{verbatim}
%it should return the same result as {\rm GRID\_LinearProgrammingSmoothing\_rx0\_simple} but much faster.

\subsection{Using {\tt GRID\_LinearProgrammingSmoothing\_rx0\_blockconstraint}}

This command corrects the bathymetry (if possible) and returns a bathymetry satisfying a number of block condition:
\begin{verbatim}
>> Hfilt=GRID_LinearProgrammingSmoothing_rx0_blockconstraint(...
     MSK, Hobs, r, ListVal, ListBlock);
\end{verbatim}
with
\begin{enumerate}
\item {\bf MSK(eta\_rho, xi\_rho)} the mask of the grid.
\item {\bf Hobs(eta\_rho, xi\_rho)} the original bathymetry of the grid.
\item {\bf ListVal(nbBlock,1)} the list of values of constraints.
\item {\bf ListBlock(nbBlock,eta\_rho,xi\_rho)} the list of arrays of constraints. We should have for all $1\leq i\leq nbBlock$ the constraints
\begin{equation*}
\sum_{iEta, iXi} ListBlock(iEta, iXi) (h(iEta, iXi)- h^{obs}(iEta, iXi)) \leq ListVal(i,1)
\end{equation*}

\end{enumerate}

\subsection{Using {\tt GRID\_SmoothPositive\_*}}
This command makes the bathymetry correct by increasing it.
\begin{verbatim}
>> Hfilt=GRID_SmoothPositive_rx0(MSK, Hobs, rx0max);
>> Hfilt=GRID_SmoothPositive_ROMS_rx1(...
     MSK, Hobs, rx1max, ARVD);
\end{verbatim}
with
\begin{enumerate}
\item {\bf MSK(eta\_rho,xi\_rho)} the mask of the grid
\item {\bf Hobs(eta\_rho,xi\_rho)} the observed bathymetry of the grid
\item {\bf rx0max, rx1max} are roughness factors.
\item {\bf ARVD} is the record of vertical parameterization the $S$-coordinates parameters.
\begin{verbatim}
  ARVD.Vtransform=2;
  ARVD.Vstretching=1;
  ARVD.ThetaS=4; % named THETA_S in the roms.in file
  ARVD.ThetaB=0.35; % named THETA_B in the roms.in file
  ARVD.hc=10; % named TCLINE in the roms.in file
  ARVD.N=30;
\end{verbatim}

\end{enumerate}

%\subsection{Using {\tt GRID\_SmoothNegative\_*}}
%This command makes the bathymetry correct by lowering it.
%\begin{verbatim}
%>> Hfilt=GRID_SmoothNegative_rx0(MSK, Hobs, rx0max);
%>> Hfilt=GRID_SmoothNegative_rx1(...
%     MSK, Hobs, rx1max, Scoord, Ccoord, hc);
%>> Hfilt=GRID_SmoothNegative_ROMS_rx1(...
%     MSK, Hobs, rx1max, ThetaS, ThetaB, N, hc);
%\end{verbatim}
%with
%\begin{enumerate}
%\item {\bf MSK(eta\_rho,xi\_rho)} the mask of the grid
%\item {\bf Hobs(eta\_rho,xi\_rho)} the observed bathymetry of the grid
%\item {\bf rx0max, rx1max} are roughness factors.
%\item {\bf Scoord, Ccoord, hc} are the $S$-coordinates parameters.
%\item {\bf ThetaS, ThetaB, N, hc} are ROMS vertical parameters.
%\end{enumerate}

\subsection{Using {\tt GRID\_PlusMinusScheme\_rx0}}

This command makes the bathymetry correct by doing a sequence
of increase/decrease at adjacent cells (see \cite{conundrum}).
\begin{verbatim}
>> [RetBathy, HmodifVal]=GRID_PlusMinusScheme_rx0(...
       MSK, Hobs, rx0max, AreaMatrix);
\end{verbatim}
with
\begin{enumerate}
\item {\bf MSK(eta\_rho,xi\_rho)} the mask of the grid
\item {\bf Hobs(eta\_rho,xi\_rho)} the observed bathymetry of the grid
\item {\bf AreaMatrix(eta\_rho,xi\_rho)} the areas of the wet and dry $\rho$-points.
\item {\bf rx0max, rx1max} are roughness factors.
\end{enumerate}

\subsection{Using {\tt GRID\_LaplacianSelectSmooth\_rx0}}

This command makes the bathymetry correct by doing an iterated
sequence of laplacian filterings
\begin{verbatim}
>> Hfilt=GRID_LaplacianSelectSmooth_rx0(MSK, Hobs, rx0max);
\end{verbatim}
with
\begin{enumerate}
\item {\bf MSK(eta\_rho,xi\_rho)} the mask of the grid
\item {\bf Hobs(eta\_rho,xi\_rho)} the observed bathymetry of the grid
\item {\bf rx0max} the maximal roughness factor.
\end{enumerate}

\section{Notes and Recommendations}

\begin{itemize}
\item The smoothing with respect to $rx_0$ is best done with
{\bf GRID\_LinProgHeuristic} which uses a linear programming approach
and should be fast even in very large and not pathological grids.
\item The smoothing with respect to $rx_1$ is problematic since
the number of constraint is much larger. Also for {\tt Vtransform=2}
those constraints are nonlinear. A variant of the Martinho Batteen
\cite{martinho} is
implemented in {\tt GRID\_SmoothPositive\_ROMS\_rx1} and deals with
all the vertical parametrization available in ROMS.
\item The function {\tt GRID\_LaplacianSelectSmooth} has several
advantages over the function {\tt smth\_bath.m} of the ROMS matlab
package:
\begin{itemize}
\item It respects the mask
\item It is guaranteed to terminate
\item It creates a perturbation to the bathymetry of smaller amplitude.
\end{itemize}
Still our recommendation is not to use Laplacian/Shapiro filtering
as they produce worse solution than other methods and have a very
tenuous justification as an adequate method.
\item If preserving the volume is important, you can use
the function {\tt GRID\_PlusMinusScheme\_rx0}. It will always produce
a larger perturbation than {\tt GRID\_LinearProgrammingSmoothing\_rx0\_volume}
or {\tt GRID\_LinProgSmoothVertVert} but it is much faster.
%Its advantage however is in speed.
%\item If preserving volume is important, then you have two options:
%\begin{enumerate}
%\item Use {\tt GRID\_LinearProgrammingSmoothing\_volume}, which has
%a global preservation of the volume.
%\item Use {\tt GRID\_LinProgSmoothVertVert} or its variant
%{\tt GRID\_PlusMinusScheme}, which preserve the volume locally.
%\end{enumerate}
%\item If there is some specific feature you want to preserve or some
%more specific request, then you may consider using
%{\tt GRID\_LinearProgrammingSmoothing}, but you have to be careful
%with your requirement as there may be no solution at all.
\end{itemize}

%\section{How does it work}
%
%Given $m$ vectors $v_i$ in $\RR^n$, $m$ reals $b_i$ and a vector $w$,
%a linear program is the following problem:
%\begin{equation*}
%\begin{array}{rcl}
%\mbox{{\bf maximize}} &\mbox{~~}& w^T x\\
%\mbox{{\bf subject to}} &\mbox{~~}& v_i^T x\geq b_i
%\end{array}
%\end{equation*}
%This kind of problems (\cite{chvatal}) have an extended history and theory.
%They have a unique solution and methods exist for solving them
%when $n$ and $m$ are of the order of million of variables.
%The game is then to transform the minimization problem,
%which is apparently not linear into one which is linear by adding some
%new variables. We used the {\tt lpsolve} program because it has a
%support for sparse matrices.

%\section{Last remarks}
%The computational time is roughly 5 minutes with a $60\times 160$ grid
%and it can reach several hours for a $160\times 360$ grid.
%This is an important cost but only a one time cost, since the grid is
%correct once and for all.
%If you have any questions, please drop me a line
%to {\tt mdsikir@irb.hr} and I will reply
%as much as I can.

\begin{thebibliography}{WWw99}
\bibitem{haney}
R.L. Haney, {\em On the pressure gradient force over steep bathymetry in sigma coordinates ocean models}, Journal of Physical Oceanography {\bf 21} (1991) 610--619.

%\bibitem[MeBl85]{mellor}
%G.L. Mellor and A.F. Blumberg, {\em Modeling vertical horizontal diffusivities with the sigma coordinate system}, Monthly Weather Review {\bf 113} (1985) 1379--1383.

\bibitem{lpmeth}
M. Dutour Sikiric, I. Janekovic, M. Kuzmic, {\em A new approach to bathymetry smoothing in sigma-coordinate ocean models},
Ocean Modelling, Volume 29, Issue 2, 2009, Pages 128-136

\bibitem{chvatal}
V.~Chv{\'a}tal, {\em Linear Programming}, W.H.~Freeman and Company, 1983.

\bibitem{conundrum}
G.L. Mellor, T. Ezer and L.-Y. Oey, {\em The pressure gradient conundrum of Sigma coordinate Ocean models}, Journal of atmospheric and oceanic technology {\bf 11} (1994) 1126--1134.

\bibitem{martinho}
A.S. Martinho and M.L. Batteen, {\em On reducing the slope parameter in terrain following numerical ocean models}, Ocean Modelling {\bf 13} (2006) 166--175.

\bibitem{lpsolve}
P. Notebaert and K. Eikland, \url{http://lpsolve.sourceforge.net/5.5/}

\end{thebibliography}
\end{document}


\chapter{Hypothesis Tests and Confidence Intervals}

\section{The distribution of mean and the central limit theorem}
\section{Unbiased estimates of population mean and variance}
\section{Using the normal distribution in hypothesis tests}
\section{Confidence intervals}


\chapter{Related Work}
\label{chap:relatedwork}
\section{Basics of Genetic Programming}
\subsection{Program Representations}
As is stated in Chapter \ref{chap:intro}, Genetic programming is a general technique that can be applied to almost any task domain.  However, this work predominantly focuses on the task of symbolic regression.  Symbolic regression aims to find a mathematical expression that fits a given data set's training and out of sample test examples.  Therefore the programs in which we are interested define mathematical expressions by combining a set of primitive operations.

Much of the time in genetic programming random segments of code are being removed, inserted, and swapped between different programs.  Therefore, it is important to have a program representation that is resilient to these types of operations.  As a result, it can be useful to avoid loops, and other types of statements that could result in programs that do not terminate.  For this reason, often a tree structure is employed, where the internal nodes of the trees are functions, with the children as the arguments, and the leaves of the trees are terminals, such as variables and constants.

For the task of symbolic regression, the functions can be any set of primitive mathematical functions, while the terminals will typically be the features of the data set.

The tree in Figure \ref{figure:gp_tree} represents a simple program, which computes the mathematical function $f(x) = \ln(x1) + (x1 - x2)$.  Given a larger set of mathematical operations, it is possible to create arbitrarily complex mathematical expressions.

\noindent
\begin{figure}[h]
\centering
\includegraphics[width=110mm]{gp_tree_colorless}
\caption{Sample tree representing a program.}
\label{figure:gp_tree}
\end{figure}

\subsection{Generating New Programs}
\label{section:gp_operators}
One of the core aspects of genetic programming is how to generate a new population of programs from an old population.  There are two primary operations used to generate new programs: the mutation operation and the crossover operation.

\subsubsection{Mutation}
The mutation operation generates a single new program from a single old program.  First a mutation point is selected in the old program.  This can be selected uniformly at random, or biased towards certain parts of the tree.  Then the subtree located at the mutation point is removed, and a new subtree is generated in a similar way to how the first population of programs was generated.

\subsubsection{Crossover}
The crossover operation generates two new programs from two old programs.  First a crossover point is selected in each old program.  This can be done in any of the same ways that a mutation point is selected.  Then the subtrees located at the two crossover points in the two old programs are swapped, creating two new program trees.

\subsection{Choosing Programs to Survive}
After a new population of programs is generated, only half of the programs from the combined new and old populations are kept for the next generation of the evolutionary process.  If there is a single fitness function that is being optimized, this process is very straightforward.  Simply keep the programs that perform best on the fitness function.  However, often there are multiple fitness functions that are used to drive the evolutionary process.  For example, in the case of symbolic regression, one fitness function could be the absolute error for predicting the dependent variable, while the other fitness function could be the program size.  When there are multiple fitness functions, often one program will have a better fitness score than another program for one fitness function, but not the remaining fitness functions.  In this case a different method must be used to determine which programs should be kept.  The method used for the purpose of this work is discussed in section \ref{section:survival}.

\subsection{Termination}
The run of a genetic program terminates after one of several conditions is met.  The following are common termination conditions:
\begin{enumerate}[noitemsep]
\item An optimal program is found.
\item A maximum number of generations is reached.
\item The run has exceeded the maximum allocated amount of time.
\end{enumerate}

\section{Behavioral Genetic Programming}
One extension to the genetic programming paradigm is behavioral genetic programming (BGP) \cite{krawiec}.  As was mentioned in Chapter \ref{chap:intro}, BGP attempts to identify useful subprograms that can then be used to enhance the evolutionary process.

\subsection{Trace}
In most genetic programming algorithms, the search is driven by the output of the fitness functions alone.  Only programs that perform well on the fitness functions are kept, while the rest are rejected.  However, every subtree that makes up an individual program has a distinct numerical output for each fitness case in any given data set.

Let us consider the program in Figure \ref{figure:gp_tree}, and the sample data set in Table \ref{table:sample_data}.  Conventional genetic programming will only consider the outputs for each data point: $\ln(3) + 1$, $\ln(5) + 2$, and compare those values to the desired output.  However, each subtree has its own output, which is ignored by the genetic programming process.

\begin{table*}[ht]
\centering
\begin{tabular}{ c c | c }
\hline\hline
$x_{1}$ & $x_{2}$ & $y$ \\ [0.5ex]
\hline
3 & 2 & 3 \\
5 & 3 & 4 \\[1ex]
\hline
\end{tabular}
\caption{Sample data set.}
\label{table:sample_data}
\end{table*}

The collection of the outputs on each subtree for all of the data points is called the trace.  For the program in Figure \ref{figure:gp_tree} and the fitness cases in Table \ref{table:sample_data}, the trace is shown in Table \ref{table:sample_trace}.

\begin{table*}[ht]
\centering
\begin{tabular}{ c c c c c c }
\hline\hline
$s_{1}$ & $s_{2}$ & $s_{3}$ & $s_{4}$ & $s_{5}$ & $s_{6}$ \\ [0.5ex]
\hline
3 & $\ln(3)$ & 3 & 2 & 1 & $\ln(3)$ + 1 \\
5 & $\ln(5)$ & 5 & 3 & 2 & $\ln(5)$ + 2 \\[1ex]
\hline
\end{tabular}
\caption{The trace of the program from Figure \ref{figure:gp_tree} for the data set in Table \ref{table:sample_data}.}
\label{table:sample_trace}
\end{table*}

The trace is a matrix where the number of rows is equal to the number of data points, and the number of columns is equal to the number of subtrees in a given program.  There is no set order of the columns, although here they are presented in the depth first traversal order of the subtrees that produce the values.  The trace captures a full snapshot of all of the intermediate states of the program evaluation.

\subsection{Model}
The key idea behind Behavioral Programming is to exploit the trace to identify useful subtrees.  Ordinary crossover does not incorporate any information about the quality of the subtrees that are being swapped.  However, the trace opens up many possibilities to explore the quality of subtrees.  In Behavioral Programming, the trace is used to train a machine learning model to predict the desired output for each fitness case.  The model is then used for two purposes.  The first is to create additional fitness measures by which to evaluate a given program.  The second is to identify useful subtrees based on the composition of this model, which are then placed in an archive.  The content of the archive is then used for the supply the subtrees that are used in the crossover operation, instead of taking arbitrary subtrees from other programs in the population.