\input{../../_common/preamble}

\title{Ensemble methods}

\begin{document}

\maketitle

\begin{frame}{Ensemble methods}
    \begin{block}{Idea}
        \begin{itemize}
            \item Train a \alert{set} of learners
            \item Combine their predictions
        \end{itemize}
    \end{block}
    \vfill
    \begin{block}{Motivation}
        \begin{itemize}
            \item More flexible $\rightarrow$ reduce bias
            \item Less prone to overfitting $\rightarrow$ reduce variance
        \end{itemize}
    \end{block}
\end{frame}

\begin{frame}{Why do ensembles work?}
    Assume 10 classifiers\ldots
    \begin{itemize}
        \item Each with error rate $\epsilon$ = 30\%
        \item Independent (not feasible in practice!)
    \end{itemize}
    \vfill
    What's the probability that the ensemble makes a wrong prediction?
    \pause
    \begin{itemize}
        \item Wrong prediction $\rightarrow$ majority ($\geq$ 6) makes a wrong
              prediction
        \item The probability is given by the binomial distribution,
              \[
                  \sum_{k = 6}^{10} \binom{10}{k} \epsilon^{k} (1 - \epsilon)^{10 - k} \approx 4.73\% \ll \epsilon
              \]
    \end{itemize}
\end{frame}

\begin{frame}{Contents}
    \tableofcontents[hideallsubsections]
\end{frame}

\section{Bagging}

\begin{frame}{Bagging (bootstrap aggregating)}
    \only<1>{%
        \begin{itemize}
            \item Multiple datasets generated by \alert{sampling with replacement}
            \item The probability that a given observation is \emph{not} selected is
                  \[
                      \left( 1 - \frac{1}{n} \right)^{n}
                  \]
            \item For large $n$, we have
                  \[
                      \lim_{n \to +\infty} \left( 1 - \frac{1}{n} \right)^{n} = \frac{1}{e} \approx 36.79\%
                  \]
            \item[$\rightarrow$] Each dataset contains slightly less than 2/3 of the observations
        \end{itemize}}
    \only<2>{%
        \begin{block}{Variations}
            \begin{itemize}
                \item Different types of learners
                \item Sampling of variables
                \item Sampling without replacement
                \item Size of subset
            \end{itemize}
        \end{block}}
\end{frame}

\section{Boosting}

\begin{frame}{Boosting}
    \only<1>{%
        \begin{block}{Idea}
            \begin{itemize}
                \item `Later' learners focus on observations that were predicted
                      incorrectly by `earlier' learners
                \item Aggregation is weighted by errors
            \end{itemize}
        \end{block}
        \vfill
        \begin{block}{Implementation}
            \begin{itemize}
                \item Iterative procedure with weight updates
                \item \alert{Increase} the weight of
                      \alert{incorrectly classified} observations
            \end{itemize}
        \end{block}}
    \only<2>{%
        Compared to bagging, boosting is\ldots
        \begin{itemize}
            \item Much less noise\hyp{}tolerant
            \item More accurate (if it works)
            \item Less well\hyp{}calibrated
        \end{itemize}}
\end{frame}

\section{Stacking}

\begin{frame}{Stacking}
    \begin{block}{Idea}
        \begin{itemize}
            \item Use a learner to combine predictions from multiple learners
            \item Trained using:
                  \begin{description}
                      \item[$\mat{X}$] Cross\hyp{}validated predictions
                      \item[$\vec{y}$] Class labels of training data
                  \end{description}
        \end{itemize}
    \end{block}
    \vfill
    \begin{block}{Meta\hyp{}learners}
        \begin{itemize}
            \item Not restricted to being linear (as in voting)
            \item Should learn how the (base) learners make mistakes
        \end{itemize}
    \end{block}
\end{frame}

\end{document}


\documentclass{article}

\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{siunitx}
\usepackage{placeins}

\usepackage[margin=1in]{geometry}
\usepackage{float}

\def\hwtitle{Computational Physics HW6}
\def\hwauthor{Ethan Rooney}
\def\hwdate{2020-03-25}

\usepackage{fancyhdr}
\lhead{\hwauthor}
\chead{\hwtitle}
\rhead{\hwdate}
\lfoot{\hwauthor}
\cfoot{}
\rfoot{\thepage}
\renewcommand{\footrulewidth}{0.4pt}
\pagestyle{fancy}

\author{\hwauthor}
\title{\hwtitle}
\date{\hwdate}

\begin{document}

\maketitle
\thispagestyle{fancy}

\section{Introduction}

Using Linear Algebra to develop Polynomials of best fit for given experimental data points.

This week we use some standard statistical tools to fit models to data and to quantize how well our model fits the data.

\section{Results}

\subsection{Question 1}

With the first 6 seconds, or first 13 data points, of \texttt{cannonball.dat} input into \texttt{hw6} we get \texttt{16380.416133	4.573787	-5.526405	6.906034} when put into the form:

\[
	y = y_0 + v_0 t + \frac{g}{2}t^2
.\]

We find:
\[
g=-11.05281 \frac{\text{m}}{\text{s}^2}
.\]

With an \[
\chi^2 = 6.906034
.\]

This gives us a confidence interval of 0.734285

This is a reasonable fit to the data.

\subsection{Question 2}

For the entire data set of \texttt{cannonball.dat} we find \texttt{16395.644750	-3.424541	-4.692395	140.237263}.

We find:
\[
g=-9.38479 \frac{\text{m}}{\text{s}^2}
.\]

With an \[
\chi^2 = 140.237263
.\]

This gives us a confidence interval of 0.0548956

140.23 is at extreme end of a $\chi^2$ distribution, with only 5\% of fits more extreme, but leaves us inside of the magic number of 95\% confidence interval to be able to accept the underlying model.

\subsection{Question 3 Bonus}
For the Bonus problem we are trying to fit the following model:
\[
	A = A_00.5^\frac{t}{\tau}
.\]

To do this we can linearise the model by taking the $\ln()$ of both sides.

\[
	\ln(A) = \ln(A_0) + t\frac{1}{\tau}\ln(0.5)
.\]

As for the sigma, when working with radiation, we get a Poisson distribution, and the sigma of a Poisson distribution in a high count regime is $ \sqrt{N} $ and after the linearisation of the model by taking the log,$ \sigma = \frac{\sqrt{N}}{N}  $

After adjusting the program to handle the two above accommodations we get the following output \texttt{11.802353	-0.086268	12.190726}.

Where
\[11.802353 = \ln(A_0).\]

\[-0.086268=\frac{1}{\tau}\ln(0.5)\]

\[12.190726 = \chi^2\]

Solving for the parameters we were trying to find we get $A_0 = 133519$. Which is very close to the known value of $133611$ and $\tau= 8.034 \text{days}$

The radioisotope responsible for the observed activity is most likely Iodine-131. A common daughter isotope of U-235 fission with a half-life of 8.02 days\footnote{http://hpschapters.org/northcarolina/NSDS/131IPDF.pdf}.

The biggest approximation that I made was the assumption that the $\sigma$ of a Poisson is $\approx \sqrt{N}$ in a high counts regime. For the later data points it is debatable if 200 counts is "high count" or not.

This approximation is unlikely to affect my model significantly.

\section{Conclusion}

Well, we did it. The bonus problem was a fun brain teaser to think through.

\end{document}


\subsection{Muon}
\label{sec:muon}

Muons are important final state signatures of many physics analyses at the LHC, such as the Higgs analyses, studies of SM, BSM searches and so on.
Thus, high performance of muon reconstruction and identifications are crucial.
This section briefly describes some more details of the \textit{reconstruction}, \textit{identification} and \textit{isolation} of muon.

\textbf{Muon reconstruction}

Muon reconstruction is firstly performed in inner detector (ID) and muon spectrometer (MS) independently as given in section~\ref{sec:track}.
The information from each individual detector is then combined together to form the muon tracks for physics analyses.
The combined ID-MS reconstruction is developed according to several algorithm based on the information from ID, MS and calorimeters.
Four different muon types are defined~\cite{Aad:2016jkr}:
\begin{itemize}
	\item \textbf{Combined (CB) muons:} a combined track is formed by using the reconstructed tracks performed independently in ID and MS with a global refit. To improve the fit quality, the hits from MS may be added to or removed from the track. The outside-in pattern recognition is utilized for the reconstruction of most muons, in which the muons are first reconstructed in MS and then extrapolated inward to match the ID track. In the meantime, the inside-out pattern is also used as a complementary method.
	\item \textbf{Segment-tagged (ST) muons:} a reconstructed track in ID is regarded as muon, if it can be associated with at least one track segment in MDT or CSC chambers. These ST muons are used when muons can only pass across one layer of MS chambers due to their low $p_{T}$ or falling into regions with less acceptance.
	\item \textbf{Calorimeter-tagged (CT) muons:} a reconstructed track in ID is categorized as CT muon if it's matched to the energy deposit in calorimeter which is recognized with a minimum-ionizing particle. This CT muons have lowest purity amount all types of muons, but it covers the region where ATLAS muon spectrometer is only partially constructed. For the region of $|\eta| < 0.1$ and $15 GeV < p_{T} < 100 GeV$, the identification of CT muons is optimal.
	\item \textbf{Extrapolated (ME) muons:} the ME muons are reconstructed based only on the MS track and a relative loose requirement of originating from the interaction point. In general, this type of muon needs to pass at least two (three) layers of MS chambers to provide a track measurement in barrel (forward) region. ME muons are designed to extend the reconstruction acceptance to the region of $2.5 < |\eta| < 2.7$ where ID measurement doesn't cover.
\end{itemize}

Before collecting those muons for physics analyses, overlap removals are performed between different muon types with the priority of CB > ST > CT, when two types of muons share the same ID track.
Besides, the overlaps with ME muons are resolved by analyzing the track hit content, and selecting the track with better fit quality and larger number of hits.

\textbf{Muon identification}

After reconstruction, the muon identification is then performed to further separate the signal and background, especially to suppress backgrounds from pion and kaon decays by requiring prompt muons with high efficiency and guaranteeing a robust momentum measurement.
The muon identification is defined by using the fit quality of combined track.
The variables utilized in judgement for CB tracks include:
\begin{itemize}
	\item \textit{q/p significance}, the absolute difference between q/g (charge over momentum) of muons measured in ID and MS divided quadratic sum of their corresponding uncertainties;
	\item \textit{$\rho'$}, the absolute value of difference between the $p_{T}$ measured in ID and MS, divided by the $p_{T}$ of combined track;
	\item \textit{Nomalized $\chi^{2}$} of the combined track fit;
	\item \textit{Hit numbers in ID and MS}
\end{itemize}
In addition, some new variables used for \textit{LowPt} muon working point what will be described later~\cite{Zheng:2649299}:
\begin{itemize}
	\item \textit{Momentum balance significance (MBS) } is computed as momentum difference between the ID and MS standalone measurements with respect to the uncertainty $\sigma$ on energy lost in the calorimeter system.
	\item \textit{Scattering neighbor significance (SNS)} is defined to estimated the significance of a change in trajectory along the track, expected in the presence of a hadron decaying to a muon.
	\item \textit{Scattering curvature significance (SCS)} is defined as the normalized integral of the scattering angle significances, corrected for large kinks along the trajectory.
\end{itemize}

Five selection working points are developed to satisfy the different needs for different physics goals: \textit{LowPt}, \textit{Loose}, \textit{Medium}, \textit{Tight} and \textit{HighPt}.
The \textit{Tight}, \textit{Medium}, \textit{Loose} are subsets from the tighter one to looser one.
More detailed definition of each working point is given as follow:
\begin{itemize}
	\item \textit{Loose:} this working point is designed to maximize the reconstruction efficiency while keeping good-quality of muon tracks. And they are specifically developed for reconstructing the Higgs boson candidates from four-lepton final states. All four muon types are used for this selection level. The CB and ME muons passing Medium WP that will mentioned below are all included into Loose category. In addition, the CT and ST muons are restricted to $|\eta| < 0.1$ region.
In the range of $|\eta| < 2.5$, around 97.5\% Loose muons are CB muons, and about 1.5\% are CT while remaining 1\% are ST muons.
	\item \textit{Medium:} this working point is the default criteria of muon identification in ATLAS. This selection minimizes the systematic uncertainties of muon reconstruction and calibration. In this category, we only use CB and ME muons. For CB muons, at least 3 hits in at least two layers of MDT are required, except $|\eta| < 0.1$ region, in which tracks with $\geq 1$ MDT layer but $\leq 1$ MDT hole layer are allowed. For ME muons, at least 3 MDT/CSC layers are required.
Furthermore, a loose cut on the compatibility between measured momentum in ID and MS is applied to reduce the fake muons from hadrons misidentification. Besides, the q/p-significance is required to be less than 7.
	\item \textit{Tight:} this working point is used to maximize the purity of muons but with sacrifice of some selection efficiency. Only CB muons with hits in $\geq 2$ stations of MS and passing Medium criteria are selected.
In addition, the normalized $\chi^{2}$ of combined track fit should be smaller than 8. Then, a two-dimensional cut of q/p-significance and $\rho'$ is adopted as a function of muon $p_{T}$ to ensure tighter background rejection for momentum below 20 GeV, in which the fake rate is usually higher.
	\item \textit{High-$p_{T}$:} this set of selections aims to maximize the momentum resolution for tracks with $p_{T} > 100 GeV$ region. The selection is especially optimized for searching high-mass $Z'$ and $W'$ resonances. The CB muons satisfying Medium selection and with $\geq 3$ hits in 3 MS stations are chosen. The specific region in MS where alignment is suboptimal are removed as a precaution.
	\item \textit{Low-$p_{T}$:} this type of muon is newly designed for physics analyses with ATLAS software release version 21. It's designed to obtain a optimal muon identification with very low transverse momentum of $3 GeV < p_{T} < 5 GeV$, which is crucial for B-physics measurement in ATLAS.
	In this muon requirement, only CB muons are used. In the range of $|\eta| < 1.3$, it requests muons hit at least one MS station; in $1.3 < |\eta| < 1.55$, a least two MS stations are required; while in region of $|\eta| > 1.55$, \textit{Medium WP} is required. In addition, cuts are applied to suppress fakes as: |MBS| < 3.0, |SNS| < 3.0 and |SCS| < 3.0.
\end{itemize}

Figure~\ref{fig:muon_id_eff} and \ref{fig:muon_id_lowpt} show the selection efficiency of different muon identification working points.
For \textit{Medium} (\textit{Loose}), \textit{Tight} and \textit{High-$p_{T}$:}, $Z \rightarrow \mu\mu$ events with $p_{T} > 10 GeV$ are used for measurement.
In the top figure of figure~\ref{fig:muon_id_eff}, the efficiency of the Loose selection (squares) is shown comparing to Medium one, where significant difference can be observed in region of $|\eta| < 0.1$.
For \textit{LowPt}, $J/\Psi \rightarrow \mu\mu$ events with $3 GeV < p_{T} < 10 GeV$ are used for measurement.
\begin{figure}[!htb]
  \centering
  \includegraphics[width=0.42\textwidth]{figures/Simulation/muon_id_med.png} \\
  \includegraphics[width=0.42\textwidth]{figures/Simulation/muon_id_tight.png}
  \includegraphics[width=0.42\textwidth]{figures/Simulation/muon_id_highpT.png}
  \caption{Muon reconstruction efficiency as functions of $\eta$ for: Medium (and Loose), Tight and High-$p_{T}$ working points.}
  \label{fig:muon_id_eff}
\end{figure}

\begin{figure}[!htb]
  \centering
  \includegraphics[width=0.5\textwidth]{figures/Simulation/muon_id_lowpT.png}
  \caption{Muon reconstruction efficiency for Low-$p_{T}$ working point as a function of $\eta$.}
  \label{fig:muon_id_lowpt}
\end{figure}

\textbf{Muon isolation}

Similar to electron, the muon isolation is used to further distinguish the prompt muon from non-prompt backgrounds.
There are also two types of isolation variables for muon:
\begin{itemize}
	\item \textbf{Calorimeter-based variable:} $E_{T}^{topocone20}$. It's defined as the sum of the transverse energy of topological clusters within a cone of $\Delta R = 0.2$ around the candidate, after the correction of pile-up effects and the subtraction of the energy deposit contribution from muon itself.
        The contributions from pile-up and underlying events are computed using the ambient energy-density technique~\cite{CACCIARI2008119} and are corrected on an event-by-event basis.
	\item \textbf{Tracked-based variable:} $p_{T}^{varcone30}$. It's computed from the scalar sum of the transverse momenta of the tracks with $p_{T} > 1 GeV$ in a cone with size of $\Delta R = min (10 GeV/p_{T}^{\mu}, 0.3)$ around the candidate muon whose transverse momenta is $p_{T}^{\mu}$ and exclude the muon track candidate itself. This $p_{T}$-dependent cone size can help to improve the performance for muons produced in the decay of particles with a large transverse momentum.
\end{itemize}

Then the isolation selections are applied based on \textit{relative isolation variables},
which are computed as the ratio of the track- or calorimeter-based isolation variables to the transverse momentum of the muon.
Figure~\ref{fig:muon_iso} shows the distribution of those relative isolation variables by using $Z \rightarrow \mu\mu$ events for muons passing \textit{Medium} identification criteria.
\begin{figure}[!htb]
  \centering
  \includegraphics[width=0.42\textwidth]{figures/Simulation/muon_iso_calo.png}
  \includegraphics[width=0.42\textwidth]{figures/Simulation/muon_iso_track.png}
  \caption{Distributions of the calorimeter-based (right) and the track-based (left) relative isolation variables measured in $Z \rightarrow \mu\mu$ events. }
  \label{fig:muon_iso}
\end{figure}


\section{Worked Test Questions}
\noindent
These questions are modeled after actual exam questions a student might face in an undergraduate multivariable calculus course. The progression of the tests roughly follows the progression of topics covered.

\input{./additionalMaterials/test1}
\input{./additionalMaterials/test2}
\input{./additionalMaterials/test3}
\input{./additionalMaterials/test4}


\section{Terminology and first examples}
\subsection{}

\begin{frame}
\frametitleTC{Terminology}
\framesubtitleTC{(1/2)}
\myPause
 \begin{itemize}[<+-| alert@+>]
 \item \TC{System} -- when required to avoid ambiguity, \TC{\underline{controlled} system:}
       \begin{itemize}[<+-| alert@+>]
       \item[] the object or phenomenon to be governed
       \item[] (a vehicle, the thermal behaviour of a building,...).
       \end{itemize}
 \item \TC{Requirements} or \TC{objectives:}
       \begin{itemize}[<+-| alert@+>]
       \item[] what you want the system to do\\
       \item[] (cruise as close as possible to 50 km/h, keep temperature between 18$^{\circ}$C\\
                and 21$^{\circ}$C in all rooms,...)
       \end{itemize}
 \item \TC{Controls} or \TC{control actions} or simply \TC{actions:}
       \begin{itemize}[<+-| alert@+>]
       \item[] what is done to the system with the purpose of fulfilling the requirements
       \item[] (act on the gas/brakes, modulate fuel flowrate to heater,\\
                operate fan coils,...).
       \end{itemize}
 \item \TC{Outcomes:}
       \begin{itemize}[<+-| alert@+>]
       \item[] the actual behaviour of the system
       \item[] (real speed and temperature behaviour over time,...).
       \end{itemize}
 \end{itemize}
\end{frame}

\begin{frame}
\frametitleTC{Terminology}
\framesubtitleTC{(2/2)}
\myPause
 \begin{itemize}[<+-| alert@+>]
 \item \TC{Disturbances:}
       \begin{itemize}[<+-| alert@+>]
       \item[] anything that affects the outcomes and cannot be manipulated, but possibly sensed
       \item[] (road slope, wind, vehicle load, weather, room occupancy,...);
       \item[] in other words, actions from the environment to which the system must be resilient.
       \end{itemize}
 \item \TC{Sensors:}
       \begin{itemize}[<+-| alert@+>]
       \item[] the entities that gather information from the system
       \item[] (tacho/accelerometer, GPS, inside/outside temperature sensors,...).
       \end{itemize}
 \item \TC{Actuators:}
       \begin{itemize}[<+-| alert@+>]
       \item[] the entities that act on the system to exert the actions
       \item[] (engine, brakes, burners, pumps, valves,...).
       \end{itemize}
 \item \TC{Controllers:}
       \begin{itemize}[<+-| alert@+>]
       \item[] the entities that determine the actions given the information\\
               deemed relevant\\
       \item[] (the object of your design).
       \end{itemize}
 \end{itemize}
\end{frame}

\begin{frame}
\frametitleTC{Remarks}
\framesubtitleTC{}
\myPause
 \begin{itemize}[<+-| alert@+>]
 \item Quite often we can talk about a control \TC{objective} in terms of a \TC{signal}\\
       following another one.
 \item In the case we use to say that
       \begin{itemize}[<+-| alert@+>]
       \item we have a \TC{controlled variable} (e.g., the speed of a vehicle)
       \item and want it to follow a \TC{set point} or \TC{reference} signal (e.g., go from 0 to 100 km/h
             linearly in 10 seconds).
       \end{itemize}
 \item Also, quite often our action can be viewed as setting a variable (e.g., the gas\\
       valve opening in the 0--100\% range); we call such a variable\\
       the \TC{control signal}.
 \item Consistently, we can often represent exogenous actions (e.g., wind\\
       speed and directions) that we called \TC{disturbances}, as signals.
 \end{itemize}
\end{frame}

\begin{frame}
\frametitleTC{Remarks}
\framesubtitleTC{}
\myPause
 \begin{itemize}[<+-| alert@+>]
 \item You are probably thinking that in the computer world the situation just sketched is hardly ever encountered.
 \item The objection seems in fact reasonable, as the entities one manages in computers are far more
       heterogeneous than mere numbers varying over time.
 \item \vfill Nonetheless hold on; you will discover that on the contrary, much more computer-related problems
       than one expects, can be formulated in terms\\ of \TC{set point following} and/or \TC{disturbance rejection}.
 \item In the end, learning systems and control is learning a new kind\\
       of abstraction.
 \end{itemize}
\end{frame}


\documentclass[11pt]{beamer}
\usetheme{Warsaw}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{stmaryrd}
\usepackage{listings}
\author[Cabalar et al.]{Pedro Cabalar and Jorge Fandinno and Torsten Schaub and Philipp Wanko}
\title[Implementing $\mathit{HT}_C$ using CASP]{Implementing (part of) $\mathit{HT}_C$ using Constraint Answer Set Programming: A prototype}
%\setbeamercovered{transparent}
%\setbeamertemplate{navigation symbols}{}
%\logo{}
%\institute{}
\date{}
%\subject{}
\begin{document}

\input{macro}
\input{listings}

\begin{frame}
\titlepage
\end{frame}

\begin{frame}{Recap}
\begin{itemize}
\item \emph{Logic of Here-and-There with Constraints (\HTC)} allows for capturing non-monotonic reasoning over arbitrary constraints involving \emph{constraint variables} and \emph{conditional expressions}
\pause
\item Flexible syntax, \emph{constraint atoms} are (infinite) strings of symbols that may refer to variables and domain elements
\begin{align*}
"x-y\leq d"\quad"a"\quad"\mathit{sum}\agg{ \, s_1, s_2, \dots \,  } = s_0"
\end{align*}
\vspace*{-.5cm}
\pause
\item Flexible semantics, \emph{denotations} assign constraint atoms sets of satisfying \emph{valuations}
\begin{align*}
  \den{x-y\leq d}
  =
  \{v\in\mathcal{V}\mid v(x),v(y), d\in \mathbb{Z}, v(x)-v(y)\leq d\}
\end{align*}
\end{itemize}
\end{frame}

\begin{frame}{Recap}
\begin{itemize}
\item \emph{Conditional constraint atoms} may contain \emph{conditional expressions} $(\ctermm{s}{s'}{\varphi})$
\pause
\item \emph{Evaluation function} replaces conditional expressions $\tau=(\ctermm{s}{s'}{\varphi})$:
  \begin{align*}
    \eval{h}{t}(\tau)
    &=
      \left\{
      \begin{array}{ll}
        s & \text{if } \langle h,t\rangle\models\varphi\\
        s' & \text{if } \langle t,t\rangle \hspace{2pt}\not\models\varphi\\
        \undefined & \text{otherwise}
      \end{array}
      \right.
  \end{align*}
\vspace*{-.5cm}
\pause
\item \emph{Interpretations} $\tuple{h,t}$ are pairs of valuations with $h\subseteq t$
\pause
\item Syntax, semantics of \emph{formulas} are standard \HT\ but for condition:
$\langle h,t \rangle \models c \text{ if } h\in \den{\eval{h}{t}(c)}$
\end{itemize}
\end{frame}

\begin{frame}{Motivation}
\begin{align*}
  \mathit{total}(R) := \mathit{sum}\agg{ \, \mathit{tax}(P) : \mathit{lives}(P,R) \,  } \ \leftarrow \ \mathit{region}(R)
\end{align*}
\pause
\small\lstinputlisting[linerange={21-21},numbers=none,mathescape=true,basicstyle={\ttfamily},language=clingo]{examples/taxes.lp}
\quad
\pause
\begin{itemize}
  \item Linear, modular translation to CASP using the \clingo\ Python API
  \pause
  \item Python version of \clingcon\ as solver
  \pause
  \item Bonus goal: implementation only by adding atoms and rules for each theory atom via the backend
\end{itemize}
\end{frame}

\begin{frame}
\tableofcontents
\end{frame}

\section{\HTC\ to \emph{CASP}: Syntactic and semantic differences}

\begin{frame}{Input language}
\scriptsize\lstinputlisting[mathescape=true,basicstyle={\ttfamily},language=clingo]{examples/htc_lang.lp}
\end{frame}

\begin{frame}{Example}
\scriptsize\lstinputlisting[linerange={13-29},mathescape=true,basicstyle={\ttfamily},language=clingo]{examples/taxes.lp}
\end{frame}

\begin{frame}{Target language}
\scriptsize\lstinputlisting[mathescape=true,escapeinside={\#(}{\#)},basicstyle={\ttfamily},language=clingo]{examples/cp_lang.lp}
\end{frame}

\begin{frame}{Semantic differences}
  \begin{itemize}
    \item \emph{CASP} does not allow variables to be undefined
    \pause
    \item \clingcon\ treats theory atoms in the body as \emph{strict, external} and theory atoms in the head as \emph{non-strict, defined}
    \pause
    \item \HTC\ stable models require values of variables to be \emph{founded}, variables may be undefined, no implicit choice for body atoms
    \pause
    \item
    \lstinputlisting[linerange={1-1},mathescape=true,escapeinside={\#(}{\#)},basicstyle={\ttfamily},language=clingo]{examples/semantics.lp}
    \pause
    \begin{tabular}{ll}
    \clingcon : & $\{\{(\mathtt{x},v)\} \mid v \ne 1 \} \cup \{\{\mathtt{a}, (\mathtt{x},1)\}\}$\\
    \pause
    \HTC      : & $\{\emptyset\}$
    \end{tabular}
    \pause
    \item Add rules, sum constraints and auxiliary variables to implement constructs foreign to language of \clingcon
  \end{itemize}
\end{frame}

\begin{frame}{Preprocessing: Seperate head from body}
  \begin{itemize}
    \item Use the abstract syntax tree to mark body and head theory atoms while parsing
    \pause
    \[
\mathtt{\&}t\mathtt{\{}e_1:c_1,\dots,e_n:c_n\mathtt{\}} \prec k\pause \Rightarrow \mathtt{\&}t\mathtt{(}\mathit{l}\mathtt{)}\mathtt{\{}e_1:c_1,\dots,e_n:c_n\mathtt{\}}\prec k
    \]
    where $t$ is the name of the theory atom, $e_i$ are tuples of symbols, $c_i$ are conditions (both non-ground), $k$ is a theory term, $\prec$ is a relation symbol and $l\in\{\mathtt{head},\mathtt{body}\}$
    \pause
    \item
    \lstinputlisting[linerange={1-1},mathescape=true,escapeinside={\#(}{\#)},basicstyle={\ttfamily},language=clingo]{examples/semantics.lp}
    $\Rightarrow$
    \lstinputlisting[linerange={2-2},mathescape=true,escapeinside={\#(}{\#)},basicstyle={\ttfamily},language=clingo]{examples/semantics.lp}
  \end{itemize}
\end{frame}

\section{Founding variable values}

\begin{frame}{Define variables}
  \begin{itemize}
    \item For every variable \texttt{x} occurring in the program, capture definedness with atom \texttt{def(x)} and add the rule
\lstinputlisting[linerange={7-7},mathescape=true,escapeinside={\#(}{\#)},basicstyle={\ttfamily},language=clingo]{examples/semantics.lp}
    \pause
    \item For every condition-free linear constraint atom
    \lstinputlisting[linerange={3-3},mathescape=true,escapeinside={\#(}{\#)},basicstyle={\ttfamily},language=clingo]{examples/semantics.lp}
       add rule
       \lstinputlisting[linerange={4-4},mathescape=true,escapeinside={\#(}{\#)},basicstyle={\ttfamily},language=clingo]{examples/semantics.lp}
       for every $\mathtt{x} \in \vars{e_1}\cup\dots\cup\vars{e_n}\cup\vars{k}$
   \end{itemize}
\end{frame}

\begin{frame}{Translating linear constraint atoms in the body}
  For every condition-free linear constraint atom
    \lstinputlisting[linerange={8-8},mathescape=true,escapeinside={\#(}{\#)},basicstyle={\ttfamily},language=clingo]{examples/semantics.lp}
    \pause
    \begin{itemize}
      \item Copy the linear constraint and assign it to fresh atom
           \pause
      \item Do not communicate original linear constraint to \clingcon\ \pause
      \item Add rules
      \only<4>{\lstinputlisting[linerange={9-12},mathescape=true,escapeinside={\#(}{\#)},basicstyle={\ttfamily},language=clingo]{examples/semantics.lp}}
      \only<5>{\lstinputlisting[linerange={13-15},mathescape=true,escapeinside={\#(}{\#)},basicstyle={\ttfamily},language=clingo]{examples/semantics.lp}}
      where $\{\mathtt{x}_1,\dots,\mathtt{x}_n\}=\vars{e_1}\cup\dots\cup\vars{e_n}\cup\vars{k}$
    \end{itemize}
\end{frame}

\begin{frame}{Example}
  We translate
    \lstinputlisting[linerange={1-1},mathescape=true,escapeinside={\#(}{\#)},basicstyle={\ttfamily},language=clingo]{examples/semantics.lp}
    \pause
      to
    \only<2>{\lstinputlisting[linerange={16-19},mathescape=true,escapeinside={\#(}{\#)},basicstyle={\ttfamily},language=clingo]{examples/semantics.lp}}
    \only<3->{\lstinputlisting[linerange={20-23},mathescape=true,escapeinside={\#(}{\#)},basicstyle={\ttfamily},language=clingo]{examples/semantics.lp}}
    \pause
    \pause
    \begin{tabular}{ll}
    \clingcon : & $\{\{(x,0)\}\}$\\
    \pause
    \HTC      : & $\{\emptyset\}$
    \end{tabular}
\end{frame}

\section{Translating conditional terms}
\begin{frame}{Translating conditional terms}
    \[
\mathtt{\&}t(l)\mathtt{\{}\dots,e_1:c_1,\dots,e_n:c_n,\dots\mathtt{\}} \prec k
    \]
    \vspace*{-.3cm}
    \pause
    \begin{itemize}
  \item For every $e_i:c_i$, introduce auxiliary variable $\mathit{aux}_i$\pause\ and add following rules where $\{\mathtt{x}_1,\dots,\mathtt{x}_n\}=\vars{e_i}$
  \small\lstinputlisting[linerange={1-5},mathescape=true,escapeinside={\#(}{\#)},basicstyle={\ttfamily},language=clingo]{examples/conditional.lp}
  \pause
  \item Replace conditional terms with respective auxiliary variables
  \small\lstinputlisting[linerange={6-9},mathescape=true,escapeinside={\#(}{\#)},basicstyle={\ttfamily},language=clingo]{examples/conditional.lp}
    \end{itemize}
\end{frame}

\begin{frame}{Example: Vicious cycle}
  \only<1>{\lstinputlisting[linerange={1-2},mathescape=true,escapeinside={\#(}{\#)},basicstyle={\ttfamily},language=clingo]{examples/vicious.lp}}
  \only<2>{\scriptsize\lstinputlisting[linerange={3-21},mathescape=true,escapeinside={\#(}{\#)},basicstyle={\ttfamily},language=clingo]{examples/vicious.lp}}
\end{frame}

\begin{frame}{Example: Vicious cycle}
  Value for \texttt{x} is unfounded due to loop through:
  \scriptsize\lstinputlisting[linerange={23-30},mathescape=true,escapeinside={\#(}{\#)},basicstyle={\ttfamily},language=clingo]{examples/vicious.lp}
\end{frame}

\begin{frame}{Example: Vicious cycle}
  Value for \texttt{aux(0)} is unfounded due to loop through:
  \small\lstinputlisting[linerange={31-34},mathescape=true,escapeinside={\#(}{\#)},basicstyle={\ttfamily},language=clingo]{examples/vicious.lp}
\end{frame}

\section{Translating assignments, range, minimum/maximum constraint}
\begin{frame}{Translating assignments and range constraints}
 \begin{itemize}
    \item For assignment $\mathtt{\&}t\mathtt{\{}e_1,\dots,e_n\mathtt{\}} =: x$, we add
    \pause
    \lstinputlisting[linerange={1-2},mathescape=true,escapeinside={\#(}{\#)},basicstyle={\ttfamily},language=clingo]{examples/assignments.lp}
    where $\{\mathtt{x}_1,\dots,\mathtt{x}_n\}=\vars{e_1}\cup\dots\cup\vars{e_n}$
    \pause
    \item For range constraint $\mathtt{\&in}\mathtt{\{}l\mathtt{..}u{\}} = x$, we add
    \pause
    \lstinputlisting[linerange={3-4},mathescape=true,escapeinside={\#(}{\#)},basicstyle={\ttfamily},language=clingo]{examples/assignments.lp}
 \end{itemize}
\end{frame}

\begin{frame}{Translating minimum constraints}
\[\mathtt{\&min}(l)\mathtt{\{}e_1,\dots,e_n\mathtt{\}} \prec k\]
\pause
 \begin{itemize}
    \item Add auxiliary variable $\mathit{min}$ and for every $e_i$, add following rules where $\{\mathtt{x}_1,\dots,\mathtt{x}_n\}=\vars{e_i}$
    \lstinputlisting[linerange={1-3},mathescape=true,escapeinside={\#(}{\#)},basicstyle={\ttfamily},language=clingo]{examples/minmax.lp}
    \pause
    \item Check whether a correct value was selected
    \lstinputlisting[linerange={4-4},mathescape=true,escapeinside={\#(}{\#)},basicstyle={\ttfamily},language=clingo]{examples/minmax.lp}
   \end{itemize}
\end{frame}

\begin{frame}{Translating minimum constraints}
\[\mathtt{\&min}(l)\mathtt{\{}e_1,\dots,e_n\mathtt{\}} \prec k\]
\pause
 \begin{itemize}
    \item If $l=\mathtt{body}$
    \lstinputlisting[linerange={5-5},mathescape=true,escapeinside={\#(}{\#)},basicstyle={\ttfamily},language=clingo]{examples/minmax.lp}
    \pause
    \item If $l=\mathtt{head}$
    \lstinputlisting[linerange={6-7},mathescape=true,escapeinside={\#(}{\#)},basicstyle={\ttfamily},language=clingo]{examples/minmax.lp}
 \end{itemize}
\end{frame}

\begin{frame}{Translating maximum constraints}
\[\mathtt{\&max}(l)\mathtt{\{}e_1,\dots,e_n\mathtt{\}} \prec k\]
translates to
\[\mathtt{\&min}(l)\mathtt{\{}-e_1,\dots,-e_n\mathtt{\}} \prec' -k\]
 \begin{itemize}
    \item If $\prec=$ \texttt{<=}, $\prec'=$ \texttt{>=}
    \item If $\prec=$ \texttt{>=}, $\prec'=$ \texttt{<=}
    \item If $\prec=$ \texttt{<}, $\prec'=$ \texttt{>}
    \item If $\prec=$ \texttt{>}, $\prec'=$ \texttt{<}
    \item If $\prec=$ \texttt{=}, $\prec'=$ \texttt{=}
 \end{itemize}
\end{frame}

\section{Demonstration}
\begin{frame}{Let's do our taxes!}
  \huge\centering\textcolor{blue}{Demo}
\end{frame}

\end{document}


\section{Introduction}
\label{sec:introduction}

Gradient boosting is an ensemble machine learning algorithm that results in a strong prediction model based on several weak estimators e.g. decision trees. Gradient boosted decision trees (GBDT) are built sequentially so as each new tree attempts to correct the errors made by the predecessors and cover the discrepancy between the target function and the current prediction. This ensemble technique has been both theoretically and empirically proven to be advantageous in various machine learning problems and is widely employed in various machine learning competitions. There are several widely used implementations of this algorithm such as XGBoost and GradientBoosting from sklearn. XGBoost in particular has been a key component to a lot of winning models in Kaggle competitions due to its speed and performance. A recently appeared implementation of this algorithm is created by the Russian company Yandex and is called CatBoost due to its special method of categorical features utilization.

CatBoost is an open-source gradient boosting library which, according to Yandex announcements, offers fast inference, categorical feature support, and thus, improved prediction effectiveness. It is claimed that CatBoost outperforms existing implementations of GDBT in classification tasks on multiple benchmarks \cite{catboost2017yandex} whereas there is no information about its results in regression problems. The main goal of this project is to assess the performance of CatBoost in the regression task of housing price prediction on multiple datasets and compare it with the performance of two other state-of-the art GBDT libraries, namely XGBoost and sklearn.

Apart from being a real-world application of GDBT, housing price prediction is a good example of regression problem in which the model has to output a continuous variable. In the scope of this project we make use of three datasets: Boston Housing \cite{boston1978housing}, the Ames Housing \cite{de2011ames} and Sberbank Russian Housing Market \cite{sberbank2017housing} datasets, each with vastly different characteristics.

In this paper we discuss the key improvements for GDBT implemented in CatBoost and train CatBoostRegressor together with XGBoostRegressor from XGBoost and GradientBoostingRegressor from sklearn. We evaluate their performance on three datasets for housing price prediction with different characteristics and discover that there is no clear winner among these three models. Even though CatBoost succeeds to outperform others on Boston dataset, there is no clear tendency overall. Thereafter, we perform some analysis of categorical feature importance and discuss several options for future work.

The remainder of the paper is structured as follows. We discuss the related work in \cref{sec:related-work}. \cref{sec:data} describes the data used for training and evaluation. \cref{sec:methodology} addresses the methodology of our research. In \cref{sec:results}, we present experimental results whereas in \cref{sec:discussion} we reflect on the performed investigation. We conclude this research and present some ideas for future work in \cref{sec:results}.


% declare document class and geometry
\documentclass[12pt]{article} % use larger type; default would be 10pt
\usepackage[margin=1in]{geometry} % handle page geometry

% import packages and commands
\input{../header2.tex}

\title{Phys 221A -- Quantum Mechanics -- HW04}
\author{UCLA, Fall 2014}
\date{\formatdate{12}{11}{2014}} % Activate to display a given date or no date (if empty),
         % otherwise the current date is printed

\begin{document}
\maketitle

\textit{
Problems 2.24 (p. 153), 2.25 (p. 153), 2.27 (p. 154), 2.29 (p. 154; no need to answer the last question), 2.31 (p. 154), and 2.33 (p. 155) from the textbook (Sakurai and Napolitano, Modern Quantum Mechanics, 2nd edition).
}

\section*{Problem 2.24}
\textit{
Consider a particle in one dimension bound to a fixed center by a $\delta$-function potential of the form
\begin{eqn}
V(x) = -\lambda \delta(x), \qquad
\lambda > 0.
\end{eqn}
Find the wave function and the binding energy of the ground state. Are there excited bound states?
}

\section*{Problem 2.25}
\textit{
Continuing from the previous problem, at $t=0$ the potential is suddenly switched off (that is, $V=0$ for $t>0$). Find the wave function for $t>0$. (Be quantitative! But you need not attempt to evaluate an integral that may appear.)
}

\section*{Problem 2.27}
\textit{
Derive an expression for the density of free-particle states in \textbf{two} dimensions, normalized with periodic boundary conditions inside a box of side length $L$. [removed: Your answer should be written as a function of $k$ (or $E$) times $\dif E \dif \phi$, where $\phi$ is the polar angle that characterizes the momentum direction in two dimensions.]
}

\section*{Problem 2.29}
\textit{
Consider a particle moving in one dimension under the influence of a potential $V(x)$. Suppose its wave function can be written as $\exp[i S(x,t) / \hbar]$. Prove that $S(x,t)$ satisfies the classical Hamilton-Jacobi equation to the extent that $\hbar$ can be regarded as small in some sense. Show how one may obtain the correct wave function for a plane wave by starting with the solution of the classical Hamilton-Jacobi equation with $V(x)$ set equal to zero. Why do we get the exact wave function in this particular case?
}

\section*{Problem 2.31}
\textit{
Derive (2.6.16),
\begin{eqn}
K(x'', t; x', t_0) = \sqrt{\frac{m}{2\pi i \hbar (t-t_0)}} \exp \left[ \frac{im(x'' - x')^2}{2 \hbar (t-t_0)} \right],
\end{eqn}
and derive the three-dimensional generalization.
}

\section*{Problem 2.33}
\textit{
The propagator in momentum space analogous to (2.6.26),
\begin{align}
K(\v x'', t, \v x', t_0) &= \sum_{a'} \braket{\v x''}{a'} \braket{a'}{\v x'} \exp \left[ \frac{-i E_{a'} (t-t_0)}{\hbar} \right] \\
	&= \sum_{a'} \bra{\v x''}{\exp \left( \frac{-i H t}{\hbar} \right)}\ket{a'} \bra{a'}{\exp \left( \frac{i H t_0}{\hbar} \right)} \ket{\v x'} \\
	&= \braket{\v x'', t}{\v x', t_0},
\end{align}
is given by $\braket{\v p'', t}{\v p', t_0}$. Derive an explicit expression for $\braket{\v p'', t}{\v p', t_0}$ for the free-particle case.
}

\end{document}


%
\documentclass[twoside]{article}
\setlength{\oddsidemargin}{0.25 in}
\setlength{\evensidemargin}{-0.25 in}
\setlength{\topmargin}{-0.6 in}
\setlength{\textwidth}{6.5 in}
\setlength{\textheight}{8.5 in}
\setlength{\headsep}{0.75 in}
\setlength{\parindent}{0 in}
\setlength{\parskip}{0.1 in}

%
% ADD PACKAGES here:
%

\usepackage{amsmath,amsfonts,graphicx}

%
% The following commands set up the lecnum (lecture number)
% counter and make various numbering schemes work relative
% to the lecture number.
%
\newcounter{lecnum}
\renewcommand{\thepage}{\thelecnum-\arabic{page}}
\renewcommand{\thesection}{\thelecnum.\arabic{section}}
\renewcommand{\theequation}{\thelecnum.\arabic{equation}}
\renewcommand{\thefigure}{\thelecnum.\arabic{figure}}
\renewcommand{\thetable}{\thelecnum.\arabic{table}}

%
% The following macro is used to generate the header.
%
\newcommand{\lecture}[4]{
   \pagestyle{myheadings}
   \thispagestyle{plain}
   \newpage
   \setcounter{lecnum}{#1}
   \setcounter{page}{1}
   \noindent
   \begin{center}
   \framebox{
      \vbox{\vspace{2mm}
    \hbox to 6.28in { {\bf EE302 - Feedback Systems
	\hfill Spring 2019} }
       \vspace{4mm}
       \hbox to 6.28in { {\Large \hfill Lecture #1 \hfill} }
       \vspace{2mm}
       \hbox to 6.28in { {\it Lecturer: #2 \hfill } }
      \vspace{2mm}}
   }
   \end{center}
   \markboth{Lecture #1}{Lecture #1}

   \vspace*{4mm}
}
%
% Convention for citations is authors' initials followed by the year.
% For example, to cite a paper by Leighton and Maggs you would type
% \cite{LM89}, and to cite a paper by Strassen you would type \cite{S69}.
% (To avoid bibliography problems, for now we redefine the \cite command.)
% Also commands that create a suitable format for the reference list.
\renewcommand{\cite}[1]{[#1]}
\def\beginrefs{\begin{list}%
        {[\arabic{equation}]}{\usecounter{equation}
         \setlength{\leftmargin}{2.0truecm}\setlength{\labelsep}{0.4truecm}%
u         \setlength{\labelwidth}{1.6truecm}}}
\def\endrefs{\end{list}}
\def\bibentry#1{\item[\hbox{[#1]}]}

%Use this command for a figure; it puts a figure in wherever you want it.
%usage: \fig{NUMBER}{SPACE-IN-INCHES}{CAPTION}
\newcommand{\fig}[3]{
			\vspace{#2}
			\begin{center}
			Figure \thelecnum.#1:~#3
			\end{center}
	}
% Use these for theorems, lemmas, proofs, etc.
\newtheorem{theorem}{Theorem}[lecnum]
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{definition}[theorem]{Definition}
\newenvironment{proof}{{\bf Proof:}}{\hfill\rule{2mm}{2mm}}

% **** IF YOU WANT TO DEFINE ADDITIONAL MACROS FOR YOURSELF, PUT THEM HERE:

\begin{document}

% Lecture Details
\lecture{20}{Asst. Prof. M. Mert Ankarali}

\par

\section{State-Space Analysis of LTI Systems}

\subsection{State-Space to TF}

Let's first re-visit the conversion from a state-space representation to
the transfer function representations for LTI systems.

Note that a SS representation of an $n^{th}$ order LTI system has the from below.
%
\begin{align*}
  \mathrm{Let} \ x(t) &\in \mathbb{R}^n \ , \ y(t) \in \mathbb{R} \ ,\  u(t) \in
  \mathbb{R} , \\
  \dot{x}(t) &= A x(t) + B u(t) , \\
  y(t) &= C x(t) + D u(t) , \\
  \mathrm{where} \ A &\in \mathbb{R}^{n \times n} \ , \
    B \in \mathbb{R}^{n \times 1} \ ,\  C \in \mathbb{R}^{1 \times n} \ , \ D \in \mathbb{R}
\end{align*}
%
In order to convert state-space to transfer function, we start with taking the Laplace transform of the
both sides of the state-equation
%
\begin{align*}
\dot{x}(t) &= A x(t) + B u(t)
\\
s X(s) &= A X(s) + B U(s)
\\
s X(s) - A X(s) &=  B U(s)
\\
\left( s I - A) X(s) \right) &=  B U(s)
\\
X(s) &= \left( s I - A \right)^{-1} B U(s)
\end{align*}
%
Now let's concentrate on the output equation
%
\begin{align*}
y(t) &= C x(t) + D u(t)
\\
Y(s) &= \left[ C \left( s I - A \right)^{-1} B + D \right] U(s)
\\
G(s) &= C \left( s I - A \right)^{-1} B + D
\end{align*}

\vspace{12pt}

\textbf{Example:} Let $p$ be a pole of $G(s)$, then show that
$p$ is also an eigenvalue of $A$.

\textbf{Solution:} Let
%
\begin{align*}
  G(s) = \frac{n(s)}{d(s)}
\end{align*}
%
If $p$ is a pole of $G(s)$, then $d(s)|_{p} = 0$. Now let's analyze
the dependence of $G(s)$ to the state-space form.
%
\begin{align*}
  G(s) &= \left[ C \left(s I - A \right)^{-1} B + D \right]
\\
  \left(s I - G \right)^{-1} &=
\frac{\mathrm{Adj} \left(s I - A \right) }{\mathrm{det} \left(s I - A
                               \right) }
\\
 G(s) &= \frac{C \mathrm{Adj} \left(s I - G \right) B + D \mathrm{det}
        \left(s I - A
                               \right)}{\mathrm{det} \left(s I - A
                               \right) }
\end{align*}
%
If $p$ is a pole of $G(s)$, then
%
\begin{align*}
\mathrm{det} \left(s I - A \right)|_{s = p} = 0
\end{align*}
%
Obviously $p$ is an eigenvalue of $A$.

\subsection{Similarity Transformations}

Now let's define a new ``state-vector'' $\hat{x}$ such that
%
\begin{align*}
  P x(t) = \hat{x}(t) \quad \mathrm{where}
\\
  P \in \mathbb{R}^{n \times n} \quad , \mathrm{det}(P) \neq 0
\end{align*}
%
Then we can transform the state-space equations using $P$ as
%
\begin{align*}
  P^{-1} \dot{\hat{x}}(t) = A P^{-1} \hat{x}(t) + B u(t)
\quad &, \quad y(t) = C P^{-1} \hat{x}(t) + D u(t)
\\
  \dot{\hat{x}} = P A P^{-1} \hat{x}(t)+ P B u(t)
\quad &, \quad y(t) = C P^{-1} \hat{x}(t) + D u(t)
\end{align*}
%
The ``new'' state-space representation is obtained as
%
\begin{align*}
  \dot{\hat{x}}(t)  &= \hat{A} \hat{x}(t) + \hat{B} u(t)
    \\
  y(t) &= \hat{C} x(t) + \hat{D} u(t)
\\
\hat{A} &= P  A P^{-1} \ , \ \hat{B} = P  B \ , \ \hat{C} = C P^{-1} \ ,
  \ \hat{D} = D
\end{align*}
%
Since there exist infinitely many non-singular $n \times n$
matrices, for a given LTI system, there exist infinitely
many different but equivalent state-space representations.

\vspace{6pt}

\textbf{Example:} Show that $A \in \mathbb{R}^{n \times n}$ and $P^{-1} A
P$, where $P \in \mathbb{R}^{n \times n}$ and $\mathrm{det}(P) \neq 0$, have the
same characteristic equation

\textbf{Solution:}
\begin{align*}
  \mathrm{det}\left(\lambda I - P^{-1} A P \right) &=
\mathrm{det}\left( \lambda P^{-1} I P - P^{-1} A P \right)
\\
&= \mathrm{det} \left( P^{-1} \left( \lambda I - A \right) P \right)
\\
&= \mathrm{det} \left( P^{-1} \right)
\mathrm{det} \left( \lambda I - A \right)
\mathrm{det} \left( P \right)
\\
&= \mathrm{det} \left( P^{-1} \right) \mathrm{det} \left( P \right)
\mathrm{det} \left( \lambda I - A \right)
\\
\mathrm{det}\left(\lambda I - P^{-1} A P \right) &= \mathrm{det} \left( \lambda I - A \right)
\end{align*}

\subsubsection{Invariance of Transfer Functions Under Similarity
  Transformation}

Consider the two different state-space representations
%
\begin{align*}
  \dot{x}(t) = A x(t) + B u(t)  \quad & \quad \quad \dot{\hat{x}}(t) = \hat{A} \hat{x}(t) + \hat{B} u(t)
\\
  y(t) = C x(t) + D u(t) \quad & \quad \quad  y(t) = \hat{C} \hat{x}(t) + \hat{D} u(t)
\end{align*}
%
where they are related with the following similarity transformation
%
\begin{align*}
P x(t) = \hat{x}(t) \ , \
\hat{G} &= P  A P^{-1} \ , \ \hat{B} = P  B \ , \ \hat{C} = C P^{-1} \ ,
  \ \hat{D} = D
\end{align*}
%
Let's compute the transfer function for the second representation
%
\begin{align*}
  \hat{G}(s) &= \left[ \hat{C} \left(s I - \hat{A} \right)^{-1} \hat{B}
  + \hat{D} \right]
\\
&= \left[ C P^{-1} \left(s I - P  A P^{-1} \right)^{-1} P B
  + D \right]
\\
&= \left[ C P^{-1} \left( P \left( s I -   A \right) P^{-1} \right)^{-1} P B
  + D \right]
\\
&= \left[ C P^{-1} P \left( s I -   A \right)^{-1} P^{-1} P B
  + D \right]
\\
&= \left[ C \left( s I -   A \right)^{-1} B + D \right]
\\
\hat{G}(s) &= G(s)
\end{align*}

\subsection{Canonical State-Space Realizations}

We know  that for a given LTI system, there exist infinitely many
different SS representations. We previously learnt some methods
to convert  a TF/ODE into State-Space form. We will now re-visit
them and talk about the canonical state-space forms.

For the sake of clarity, derivations are given for a general $3^{rd}$ order LTI system.

\subsection{Controllable Canonical Form}

In this method of realization, we use the fact the system is
LTI. Let's consider the transfer function of the system and let's
perform some LTI operations.
%
\begin{align*}
Y(s) &= \frac{ b_3 s^3 + b_2 s^2  + b_1 s + b_0 }{ s^3 + a_2 s^2 + a_1 s + a_0} U(s)
\\
&= \left( b_3 s^3 + b_2 s^2  + b_1 s + b_0 \right)
\frac{1}{ s^3 + a_2 s^2 + a_1 s + a_0 } U(s)
\\
&= G_2(s) G_1(s) U(s) \ \mathrm{where}
\\
G_1(s) &= \frac{H(s)}{U(s)} = \frac{1}{ s^3 + a_2 s^2 + a_1 s + a_0 }
\\
G_2(s) &= \frac{Y(z)}{H(z)} = b_3 s^3 + b_2 s^2  + b_1 s + b_0
\end{align*}
%
As you can see we introduced an intermediate variable $h(t)$ or with a
Laplace transform of $H(s)$. First transfer function has static input
dynamics, operates on $u(t)$, and produces an output, i.e. $h(t)$.
Second transfer function is a ``non-causal'' system and operates on $h(t)$ and produces output
$y(t)$. If we write the ODEs of both systems we obtain
%
\begin{align*}
\dddot{h} &= -a_2 \ddot{h} - a_1 \dot{h} - a_0 h + u
\\
y &= b_3 \dddot{h} + b_2 \ddot{h} + b_1 \dot{h} + b_0 h
\end{align*}
%
Now let the state-variables be
$x = \left[ \begin{array}{c} x_1 \\ x_2 \\ x_3 \end{array} \right]
= \left[ \begin{array}{c} h \\ \dot{h} \\ \ddot{h} \end{array} \right]$. Then,
individual state equations take the form
%
\begin{align*}
\dot{x_1} &= x_2
\\
\dot{x_2} &= x_3
\\
\dot{x_3} &= -a_2 x_3 - a_1 x_2 - a_0 x_1 + u
\end{align*}
%
and the output equation take the form
%
\begin{align*}
y &= b_3\left( -a_2 x_3 - a_1 x_2 - a_0 x_1 + u \right) + b_2 x_3 +
    b_1 x_2 + b_0 x_1
\\
&= ( b_0 - b_3 a_0 ) x_1 + ( b_1 - b_3 a_1 ) x_2 + ( b_2 - b_3 a_2 ) x_3 + b_3 u
\end{align*}
%
If we re-write the equations in matrix form we obtain the state-space representation as
%
\begin{align*}
	\dot{x} &= \left[ \begin{array}{ccc} 0 & 1 & 0  \\  0 & 0 & 1 \\  -a_0 & -a_1 & -a_2 \end{array} \right] x
	+ \left[ \begin{array}{c} 0 \\ 0 \\ 1 \end{array} \right] u
	\\
	y &= \left[ \begin{array}{ccc} ( b_0 - b_3 a_0 ) & ( b_1 - b_3 a_1 )  & ( b_2 - b_3 a_2 ) \end{array} \right] x
	+ \left[ b_3 \right] u
\end{align*}

If we obtain a state-space model from this approach, the form
will be in \textit{controllable canonical form}.

For a general $n^{th}$ order transfer function controllable
canonical form has the following $A \ ,  B \ ,  C \ , \& \ D$
matrices

\begin{align*}
A &= \left[ \begin{array}{ccccc} 0 & 1 & 0 & \cdots & 0 \\ 0 & 0 & 1 &
                                                                      \cdots & 0
\\ \vdots & \vdots & \vdots & & \vdots
\\ 0 & 0 & 0 & \cdots & 1
    \\ -a_0 & -a_1 & -a_2 & \cdots & -a_{n-1} \end{array} \right]
\quad , \quad
B = \left[ \begin{array}{c} 0\\ 0 \\ \vdots \\ 0
    \\ 1 \end{array} \right]
\\ C &= \left[ \begin{array}{cccc} (b_0 - b_n a_0)
  &  (b_1 - b_n a_1) & \cdots & (b_{n-1} - b_n a_{n-1}) \end{array} \right]
\quad , \quad
D = b_n
\end{align*}

\subsection{Observable Canonical Form}

In this method will obtain a different minimal state-space realization,
the form is called observable canonical form.
The process is different and state-space structure will have a
different topology. Let's start with a $3^{rd}$ transfer function
and perform some grouping based on the $s$ elements.
%
\begin{align*}
&Y(s) = \frac{ b_3 s^3 + b_2 s^2  + b_1 s + b_0 }{ s^3 + a_2 s^2 + a_1 s + a_0} U(s)
\\
&Y(s) \left( s^3 + a_2 s^2 + a_1 s + a_0 \right) = \left( b_3 s^3 + b_2 s^2  + b_1 s + b_0 \right) U(s)
\\
&s^3 Y(s) = b_3 s^3 U(s) + s^2 \left( -a_2 Y(s) + b_2 U(s) \right) + s \left( -a_1 Y(s) + b_1 U(s) \right) +
\left( -a_0 Y(s) + b_0 U(s) \right)
\end{align*}
%
Let's multiply both sides with $\frac{1}{s^3}$ and perform further grouping
%
\begin{align*}
&Y(s) = b_3 U(s) + \frac{1}{s} \left( -a_2 Y(s) + b_2 U(s) \right) + \frac{1}{s^2}  \left( -a_1 Y(s) + b_1 U(s) \right) +
\frac{1}{s^3} \left( -a_0 Y(s) + b_0 U(s) \right)
\\
&Y(s) = b_3 U(s) + \frac{1}{s} \left[ \left( -a_2 Y(s) + b_2 U(s) \right) + \frac{1}{s}  \left\lbrace \left( -a_1 Y(s) + b_1 U(s) \right) +
\frac{1}{s} \left( -a_0 Y(s) + b_0 U(s) \right) \right\rbrace \right]
\end{align*}
%
Let the Laplace domain representations of state variables $X(s) = \left[ \begin{array}{c} X_1(s) \\ X_2(s) \\ X_3(s) \end{array} \right]$ defined as
%
\begin{align*}
X_1(s) &= \frac{1}{s} \left( -a_0 Y(s) + b_0 U(s) \right)
\\
X_2(s) &= \frac{1}{s}  \left\lbrace \left( -a_1 Y(s) + b_1 U(s) \right) +
\frac{1}{s} \left( -a_0 Y(s) + b_0 U(s) \right) \right\rbrace
\\
X_3(s) &= \frac{1}{s} \left[ \left( -a_2 Y(s) + b_2 U(s) \right) + \frac{1}{s}  \left\lbrace \left( -a_1 Y(s) + b_1 U(s) \right) +
\frac{1}{s} \left( -a_0 Y(s) + b_0 U(s) \right) \right\rbrace \right]
\end{align*}
%
In this context output equation in $s$ and time domains simply takes the form
%
\begin{align*}
	Y(s) = X_3(s) + b_3 U(S) \quad \rightarrow \quad y(t) = x_3(t) + b_3 u(t)
\end{align*}
%
Dependently the state equations (in $s$ and time domains) take the form
%
%
\begin{align*}
s X_1(s) = -a_0 X_3(s)  + ( b_0 - a_0 b_3) U(s) \quad &\rightarrow \quad \dot{x}_1 = -a_0 x_3  + ( b_0 - a_0 b_3) u
\\
s X_2(s) = X_1(s)  -a_1 X_3(s) + ( b_1 - a_1 b_3 ) U(s)  \quad &\rightarrow \quad \dot{x}_2 = x_1  - a_1 x_3 + ( b_1 - a_1 b_3 ) u
\\
s X_3(s) = X_2(s)  -a_2 X_3(s) + ( b_2 - a_2 b_3 ) U(s)  \quad &\rightarrow \quad \dot{x}_3 = x_2  - a_2 x_3 + ( b_2 - a_2 b_3 ) u
\end{align*}
%
If we re-write all equations in matrix form, we obtain the state-space representation as
%
\begin{align*}
	\dot{x} &= \left[ \begin{array}{ccc} 0 & 0 & -a_0  \\  1 & 0 & -a_1 \\  0 & 1 & -a_2 \end{array} \right] x
	+ \left[ \begin{array}{c} ( b_0 - b_3 a_0 ) \\ ( b_1 - b_3 a_1 ) \\ ( b_2 - b_3 a_2 ) \end{array} \right] u
	\\
	y &= \left[ \begin{array}{ccc} 0 & 0  & 1 \end{array} \right] x
	+ \left[ b_3 \right] u
\end{align*}

If we obtain a state-space model from this method, the form
will be in \textit{observable canonical form}. Thus we can call this representation also as
\textit{observable canonical realization}. This form and
representation is the dual of the previous representation.

For a general $n^{th}$ order system controllable
canonical form has the following $A \ ,  B \ ,  C \ , \& \ D$
matrices

\begin{align*}
A &= \left[ \begin{array}{ccccc} 0 & 0 & \cdots & 0 & -a_0
              \\ 1 & 0 & \cdots & 0 & -a_1
\\ \vdots & \vdots & \vdots & \vdots & \vdots
\\ 0 & 0 & \cdots & 0 & -a_{n-2}
    \\ 0 & 0 & \cdots & 1 & -a_{n-1} \end{array} \right]
\quad , \quad
B = \left[ \begin{array}{c} (b_0 - b_n a_0)  \\ (b_1 - b_n
             a_1 ) \\ \vdots \\ (b_{n-2} - b_n a_{n-2} ) \\   (b_{n-1} - b_n
             a_{n-1})
\end{array} \right]
\\ C &= \left[ \begin{array}{ccccc} 0 & 0 & \cdots &  0 & 1 \end{array} \right]
\quad , \quad
D = b_n
\end{align*}

\subsubsection*{Diagonal Canonical Form}

If the transfer function of the LTI system
has distinct poles, we can expand it
using partial fraction expansion
%
\begin{align*}
Y(s) &= \left[ b_0 + \frac{c_1}{z - p_1} + \frac{c_2}{z - p_2}
+ \frac{c_3}{z - p_3} \right] X(s)
\end{align*}
%
Now let's concentrate on the candidate ``state variables''
and try to write state evaluation equations
%
\begin{align*}
X_1(s) &= \frac{1}{s - p_1} U(s) \quad \rightarrow \quad \dot{x}_1 =  p_1 x_1+ u
\\
X_2(s) &= \frac{1}{s - p_2} U(s) \quad \rightarrow \quad \dot{x}_2 = p_2 x_2+ u
\\
X_3(s) &= \frac{1}{s - p_3} U(s) \quad \rightarrow \quad \dot{x}_3 = p_3 x_3 + u
\end{align*}
%
where as output equation can be derived as
%
\begin{align*}
y(t) &= b_3 u(t) + c_1 x_1(t) + c_2 x_2(t) + c_3 x_3(t)
\end{align*}
%
If we combine the state and output equations, we
can obtain the state space form as
%
%
\begin{align*}
  \dot{\mathbf{x}}(t) &= \left[ \begin{array}{ccc} p_1 & 0 & 0\\ 0 & p_2 & 0
    \\ 0 & 0 & p_3 \end{array} \right] \mathbf{x}(t)
   +
  \left[ \begin{array}{c} 1 \\ 1
    \\ 1 \end{array} \right] u(t)
\\
y(t) &= \left[ \begin{array}{ccc} c_1 & c_2 & c_3 \end{array} \right] \mathbf{x}(t)
+ b_3 u(t)
\end{align*}
%
where
%
\begin{align*}
\mathbf{x} = \left[ \begin{array}{c} x_1(t) \\ x_2(t) \\
x_3(t) \end{array} \right] \quad , \quad
A = \left[ \begin{array}{ccc} p_1 & 0 & 0 \\ 0 & p_2 & 0
    \\ 0 & 0 & p_3 \end{array} \right]
\quad , \quad
B = \left[ \begin{array}{c} 1 \\ 1
    \\ 1 \end{array} \right]
\quad , \quad
C = \left[ \begin{array}{ccc} c_1 & c_2 & c_3 \end{array} \right]
\quad , \quad
D = b_3
\end{align*}
%

The form obtained with this approach is called
diagonal canonical form. Obviously, this form is
not applicable for ``some'' systems that has repeated roots.

For a general $n^{th}$ order system with distinct
roots diagonal canonical form has the following
$A \ ,  B \ ,  C \ , \& \ D$ matrices
%
\begin{align*}
A &= \left[ \begin{array}{ccccc} p_1 & 0 & \cdots & 0 & 0
              \\ 0 & p_2 & \cdots & 0 & 0
\\ \vdots & \vdots & \vdots & \vdots & \vdots
\\ 0 & 0 & \cdots & p_{n-1} & 0
    \\ 0 & 0 & \cdots & 0 & p_n \end{array} \right]
\quad , \quad
B = \left[ \begin{array}{c} 1 \\ 1 \\ \vdots \\ 1 \\  1
\end{array} \right]
\\ C &= \left[ \begin{array}{ccccc} c_1 & c_2 & \cdots &  c_{n-1} & c_n \end{array} \right]
\quad , \quad
D = b_n
\end{align*}

\newpage

\textbf{Example:} Given that
%
\begin{align*}
G(s) &= \frac{s^2 + 8 s + 10}{s^2 + 3 s + 2}
\end{align*}
%
find a controllable, observable, and  diagonal canonical state-space representation of the given TF.

\vspace{6pt}

\textbf{Solution:}

If we follow the derivation of controllable canonical form for a
second order system we obtain the following structure
%
\begin{align*}
	\dot{x} &= \left[ \begin{array}{cc} 0 & 1  \\  -a_0 & -a_1 \end{array} \right] x
	+ \left[ \begin{array}{c} 0 \\ 1 \end{array} \right] u
	\\
	y &= \left[ \begin{array}{ccc} ( b_0 - b_2 a_0 ) & ( b_1 - b_2
                                                           a_1 )  \end{array} \right] x
	+ \left[ b_2 \right] u
\end{align*}
%
where
%
\begin{align*}
a_0= 2 \ , \ a_1 = 3 \ , \ b_0 = 10 \ , \ b_1 = 8 \ ,\& \ b_2 = 1
\end{align*}
%
Thus, the state-space representation takes the form
%
\begin{align*}
	\dot{x} &= \left[ \begin{array}{cc} 0 & 1  \\  -2 & -3 \end{array} \right] x
	+ \left[ \begin{array}{c} 0 \\ 1 \end{array} \right] u
	\\
	y &= \left[ \begin{array}{ccc} 8 & 5  \end{array} \right] x + [1] u
\end{align*}
%
Observable canonical form is the dual of the controllable canonical form
thus for the given system, we know that
%
\begin{align*}
	A_{OCF} &= A_{CCF}^T = \left[ \begin{array}{cc} 0 & -2  \\  1 & -3 \end{array} \right]
		\\
	B_{OCF} &= C_{CCF}^T = \left[ \begin{array}{c} 8  \\  5 \end{array} \right]
		\\
	C_{OCF} &= B_{CCF}^T = \left[ \begin{array}{cc} 0  &  1 \end{array} \right]
	\\
	 D_{OCF} &= D_{CCF} = [1]
\end{align*}
%
In order to find the diagonal canonical form, we need to perform partial fraction
expansion
%
\begin{align*}
G(s) &= \frac{s^2 + 8 s + 10}{s^2 + 3 s + 2}
	= 1 + \frac{3}{s+1} + \frac{2}{s+2}
\end{align*}
%
then SS matrices for the diagonal canonical form can be simply derived as
%
\begin{align*}
	A_{DCF} &=  \left[ \begin{array}{cc} -1 & 0  \\  0 & -2 \end{array} \right]
		\\
	B_{DCF} &=  \left[ \begin{array}{c} 1  \\  1 \end{array} \right]
		\\
	C_{DCF} &= \left[ \begin{array}{cc} 3  &  2 \end{array} \right]
	\\
	 D_{DCF} &=  [1]
\end{align*}

\newpage

\textbf{Example:} Consider the following general
state-space representation
%
\begin{align*}
  \dot{x}(t) &= A x(t) + B u(t) , \\
  y(t) &= C x(t) + D u(t)
\end{align*}
%
Now let's consider the following
state-space representation
%
\begin{align*}
  \dot{\bar{x}}(t) &= A^T \bar{x}(t) + C^T u(t) , \\
  y(t) &= B^T \bar{x}(t) + D u(t)
\end{align*}
%
Show that these two state-space representations
results in same transfer function form

\vspace{6pt}

\textbf{Solution:}
%
For the second representation we have
%
\begin{align*}
\bar{G}(s) &= \bar{C} \left( s I - \bar{A} \right)^{-1} \bar{B} + D
	\\
	&= B^T \left( s I - A^T \right)^{-1} C^T + D
\end{align*}
%
Since $\bar{G}(s)$ is a scalar quantity we can take its
transpose
%
\begin{align*}
	\bar{G}(s) &= [\bar{G}(s)]^T =
	[ B^T \left( s I - A^T \right)^{-1} C^T + D ]^T
	\\
	&= (C^T)^T \left( \left( s I - A^T \right)^{-1} \right)^T (B^T)^T + D
	\\
	&= C \left( \left( s I - A^T \right)^T \right)^{-1} B + D
	\\
	&= C \left( s I - A \right)^{-1} B + D
	\\
	\bar{G}(s) &= G(s)
\end{align*}
%
This result also shows that controllable and observable
canonical representations are similar.

\newpage

\section{Stability \& State-Space Representations}

Let's consider the state-representation of an LTI system
%
\begin{align*}
  \dot{x}(t) &= A x(t) + B u(t)
\\
  y(t) &= C x(t) + D u(t)
\end{align*}
%
Given LTI system is called \textit{asymptotically stable} if,
with $u(t) = 0$ and $\forall x(0) \in \mathbb{R}^n$, we have
%
\begin{align*}
  \lim_{t \to \infty} || x(t) || = 0
\end{align*}

\textit{Theorem:} A state-space representation is
asymptotically stable if and only if all of the eigenvalues
of the system matrix, $A$, have negative real parts, i.e.
%
\begin{align*}
    \forall x_0 \in \mathbb{R}^n, \lim_{t \to \infty} || x(t) || = 0
    \iff \forall i \in \lbrace 1 , \ \cdots ,  \ n \rbrace, \mathrm{Re} \lbrace \lambda_i  \rbrace < 0
\end{align*}
%

\vspace{6pt}

\textbf{Ex:} Show that if a state-space representation is asymptotically stable
then its transfer function representation is BIBO stable.

\vspace{6pt}

\textbf{Solution:} Previously we showed that
if  $p$ is a pole of $G(s)$, then it is also an
eigenvalue of $A$, since we can write $G(s)$ as
%
\begin{align*}
 G(s) &= \frac{C \mathrm{Adj} \left(s I - G \right) B + D \mathrm{det}
        \left(s I - A
                               \right)}{\mathrm{det} \left(s I - A
                               \right) }
\end{align*}
%
If the state-space representation is asymptotically stable then
we know that for each pole, $p_i$ of $G(s)$ we have $\mathrm{Re} \lbrace p_i  \rbrace < 0$
which makes the input--output dynamics BIBO stable. In conclusion,

Asymptoticly stable $\Rightarrow$ BIBO stable
%

\textbf{Example:} Consider the following state-space form of a
CT system
%
\begin{align*}
\dot{x}(t) &=
\left[ \begin{array}{cc} 0 & 1 \\ 1 & 0 \end{array} \right]
x(t)
+
\left[ \begin{array}{c} 0 \\ 1 \end{array} \right] u(t)
\\
y(t) &= \left[ \begin{array}{cc} 1 & -1 \end{array} \right] x(t)
\end{align*}

\begin{itemize}
	\item Is this system asymptotically stable?
	\item Is this system BIBO stable?
\end{itemize}

\textbf{Solution:} Let's compute the eigenvalues of $A$

\begin{align*}
\mathrm{det} \left( \left[ \begin{array}{cc} \lambda & -1 \\ -1 &
                                                                  \lambda \end{array}
                                                                  \right]
                                                                  \right)
                                                                  =
                                                                  \lambda^2
                                                                  - 1
\\
\lambda_{1,2} = \pm 1
\end{align*}

Thus the system is NOT Asymptotically Stable.
Now let's check  BIBO stability condition.
First, compute the $G(s)$

\begin{align*}
  G(s) &= \left[ \begin{array}{cc} 1 & -1 \end{array} \right]
\left[ \begin{array}{cc} s & -1 \\ -1 & s\end{array}
                                                                  \right]^{-1}
\left[ \begin{array}{c} 0 \\ 1 \end{array} \right]
\\
&=
\left[ \begin{array}{cc} 1 & -1 \end{array} \right]
\left[ \begin{array}{cc} s & 1 \\ 1 & s\end{array}
                                                                  \right]
\left[ \begin{array}{c} 0 \\ 1 \end{array} \right] \frac{1}{s^2 - 1}
\\
&=
\left[ \begin{array}{cc} 1 & -1 \end{array} \right]
\left[ \begin{array}{cc} s & 1 \\ 1 & s\end{array}
                                                                  \right]
\left[ \begin{array}{c} 0 \\ 1 \end{array} \right] \frac{1}{s^2 - 1}
\\
&=
\left[ \begin{array}{cc} 1 & -1 \end{array} \right]
\left[ \begin{array}{c} 1 \\ s \end{array} \right] \frac{1}{s^2 - 1}
\\
&= \frac{-(s-1)}{s^2 - 1}
\\
&= \frac{-1}{s + 1}
\end{align*}

Indeed, the system is BIBO Stable.

In conclusion

\begin{itemize}
  \item Asymptotically Stable $\Rightarrow$ BIBO Stable
  \item BIBO Stable $\not\Rightarrow$ Asymptotically Stable
\end{itemize}

\end{document}


%%
%% Larson Hogstrom
%% MIT 18.335
%% Final Project
%% 5/12/2015

\documentclass[final,leqno,onefignum,onetabnum]{siamltex1213}

\usepackage{algorithm2e}
\usepackage{amsmath}
\usepackage{cite}

\usepackage{graphicx}
%% \graphicspath{ {"/Users/hogstrom/Dropbox (Personal)/cources_spring_2015_MIT/18_335_Numeric_Methods/NMF/figures/"} }
\graphicspath{ {/Users/hogstrom/Documents/code/ThornsInRoses/18_335} }
\DeclareGraphicsExtensions{.pdf,.png,.jpg}

\title{Numeric methods for Nonnegative matrix factorization \thanks{A final project for 18.335, Spring 2015}}

\author{Larson Hogstrom\thanks{
(\email{hogstrom@mit.edu}). }}

\begin{document}

\maketitle
\slugger{mms}{xxxx}{xx}{x}{x--x}%slugger should be set to mms, siap, sicomp, sicon, sidma, sima, simax, sinum, siopt, sisc, or sirev

\begin{abstract} This report presents a framework and analysis to compare three common methods for computing nonnegative matrix factorization (NMF). Parts-based representations obtained through NMF have become a popular approach for dimensionality reduction in a variety of scientific and technical domain. This study focuses on the algorithms and numerical aspects of NMF computation. We review the derivation and continual descent properties of the multiplicative update rules which helped to first popularized the NMF approach. Alternating least squares and projection gradient methods are also presented as faster, more recent approaches. The accuracy and convergence of these algorithms is compared using simulated data and two of the methods were applied to a database of gene expression profiles measuring drug responses in the MCF7 cancer cell line. Lastly, the objective function of alternating least squares was directly modified to impose sparsity during matrix factorization. The fraction of near-zero entries was compared before and after modifying the objective function, as sparsity results are of known interest to many researchers using NMF.
\end{abstract}

\begin{keywords} dimensionality reduction, matrix factorization, least squares, nonnegativity, data representation \end{keywords}

\pagestyle{myheadings}
\thispagestyle{plain}
\markboth{18.335 FINAL REPORT}{Numeric methods for nonnegative matrix factorization}

\section{Introduction}

Non-negative data is ubiquitous in many domains. Chemical concentrations, word counts from written text, and pixels in digital images all take strictly non-negative values. Researchers working with these and related datatypes often seek to manipulate and interpret large matrices of input data. Low-rank approximations can aid in identifying the most important components of such data by ignoring nonessential components related to noise or known artifacts. Common goals in performing such low-rank approximations include clustering similar items into groups, retrieving related samples based on an user's input query, or isolating representative components of data that can be directly interpreted \cite{Lan}.

Nonnegative matrix factorization (NMF) is a dimensionality reduction technique that  has been employed to address a variety of scientific and engineering challenges including text data mining \cite{Pau,Lan,LeSe1} chemometrics \cite{Paa1,Paa2}, and microarray analysis \cite{Bur}. Like principal component analysis (PCA), NMF seeks to describe a large matrix of data using a small set of information dense components. Unlike PCA, which imposes orthogonality constraints on components, NMF requires nonnegativity after factorization. This procedure allows for only additive combinations of positive components to estimate the input matrix. The endpoint is a parts-based representation of the starting data where each component is more easily interpreted than PCA.

A number of algorithms have been proposed in order to construct NMF factorization with non-negative constraints. This approach was largely popularized by a 1999 article by Lee and Seung \cite{LeSe1}. Their study, written for a general audience, examined the benefits of parts-based representations and presented applications of NMF to domains including text analysis and the decomposition of facial images. The report was followed by an additional study outlining more mathematical details of NMF as well as a simple multiplicative update algorithm \cite{LeSe2}. The publications of Lee and Seung, however, were predated by lesser known work by Pentti Paatero. The method he developed and named `positive matrix factorization' was based on a constrained alternating least squares algorithm and represents the first presentation of what is today known as NMF \cite{Paa1,Paa2}.

For the purposes of this study, three algorithms were implemented and evaluated in the python language using the `numpy' and `scipy' packages for scientific computation. In simulations, the algorithms were compared by their accuracy after a fixed number of iterations. To illustrate a real-world application of NMF, non-negative, low-rank approximations were used to describe gene expression profiles of known pharmacological groups. This proceeding report is organized in the following structure. Section 2 defines the NMF approach and reviews commonly used objective functions. Section 3 outlines the multiplicative update rule which is one of the oldest and most widely used procedures to perform matrix decomposition with NMF. Section 4 covers the alternating least squares approach which has variations to better characterized convergence properties. This is followed by a discussion of NMF sparsity and applications to gene expression data.

\begin{figure}[t!]
  \centering
    \includegraphics[width=.5\linewidth]{NMF_faces_Lee_Seung_1999}
  \caption{A comparison of NMF and SVD in creating low-rank approximations of facial data. The parts-based representation of NMF yields comments that can be directly interpreted as facial features such as noses, mouths and eyes. The interpretation of SVD components is more challenging. Reproduced from Lee and Seung, 1999.}
\end{figure}

\section{Problem Definition and Objective Functions}

\begin{definition} For a nonnegative matrix A \in \Re^{m x n} $, select a low-rank approximation of size k such that there are two nonnegative matrices W $ \in \Re^{m x k}$ and H $ \in \Re^{k x n}$ which minimizes a function such as
\begin{equation}\label{ObjEuc}
 f( W,H) = \frac{1}{2} || A - WH || ^{2}_{F}
\end{equation}

\end{definition}
Other commonly used objective functions include Euclidian distance and Kullback-Leibler (KL) divergence. KL can be extended to a more general information-based framework using Renyi's divergence \cite{Dev1}. Here, a single parameter $\alpha$ is used to represent a continuum of distance measures and KL airises as a special case where $\alpha \to 1$.
\begin{equation}\label{ObjKL}
 KL(A || WH) = \sum_{ij}{[A_{ij} \log{ \frac{A_{ij}}{(WH)_{ij}} - A_{ij} + (WH)_{ij}} ]}
\end{equation}
For its simplicity and widespread use, this report will focus on the objective function defined in (2.1).

Regardless of the objective used to obtain them, the resulting W and H matrices are often interpreted directly. The matrix W $ \in \Re^{m x k}$ typically represents k different components of signal extracted from the original matrix. The H $ \in \Re^{k x n}$ matrix can be interpreted as weighting factors defining the linear combination of k components used to reconstruct a sample in the original data matrix. In the application of NMF to gene expression data for example, each of the k columns of W represent a `metagene' response. A small number of these metagene columns can be combined to estimate an original experimental column found in the original data matrix. The H matrix defines the weighting of each metagene.

\section{Multiplicative Update Rule} In their seminal paper, Lee and Seung began their analysis with a simple additive update rule for H that reduces the squared distance:
\begin{equation}\label{au1}
 H_{aj} \leftarrow  H_{aj} + \eta_{aj} [ (W^T A)_{aj} - (W^T WH)_{aj}]
\end{equation}
The authors noted that this update represents a typical gradient descent rule if each of the $\eta_{au} $ terms are all equally set to a small positive number. Alternatively, the variables can be diagonally rescaled by setting
\begin{equation}\label{et1}
 \eta_{aj} = \frac{ H_{aj} }{(W^T WH)_{aj}}
\end{equation}
and when placed into equation (3.1), this motivates the multiplicative update rules that seek to minimize the euclidian distance $||A - WH ||$:
\begin{equation}\label{au1}
 H_{aj} \leftarrow  H_{aj} \frac{(W^T A)_{aj} }{ (W^T WH)_{aj} } \; \; \; \; \; W_{ia} \leftarrow  W_{ia} \frac{(W^T A)_{ia} }{ (W^T WH)_{ia} }
\end{equation}
Where the same logic for applies for both the update of H and W. Making use of auxiliary functions, Lee and Seung went on to demonstrate the non-increasing properties of NMF under the multiplicative update rules. It has been noted that they incorrectly claimed that the algorithm converges to a local minimum \cite{Chu, Gon, Lin}. It is possible, therefore, that continual descent property of multiplicative updates could descend to a saddle point \cite{Ber}. Furthermore, if $W_{ia} = 0$ after the $k$th iteration, then this entry remains $W_{ia} = 0$ for all subsequent iterations. This property is undesirable because the algorithm could theoretically remain fixed in a non-optimal descent path. \\

\begin{algorithm}[H]
 \KwData{Input data matrix:  $\bf{A} \in \Re^{m x n}$}
 \KwResult{nonnegative factorization of \textbf{A} using k components, creating matrices \textbf{W} $ \in \Re^{m x k}$ and \textbf{H} $ \in \Re^{k x n}$ }

 initialization\;
$\bf{W} \gets $ random dense (m x k) matrix\\
$\bf{H} \gets $ random dense (k x n) matrix\\
 \For{i = 1 to \text{maxiter} }{
 	$\bf{H = H .* (W^T A) ./ (W^T WH)} $

 	$\bf{W = W .* (A H^T) ./ (WHH^T)} $	}
 \caption{Multiplicative update}
\end{algorithm}

\begin{figure}[t!]
  \centering
    \includegraphics[width=.5\linewidth]{NMF_MU_vs_PG_error}
  \caption{Error comparison of multiplicative update and projection gradient methods for performing NMF calculation. Both algorithms were applied to a matrix of fixed size and the number of iterations were systematically increased.}
\end{figure}

The multiplicative update algorithm (Alg. 1) was compared to the NMF projection gradient method from \cite{Lin} which is built into the python scientific computing package. The projection gradient method is a more complex and refined algorithm that approaches NMF as a minimization problem with bound constraints (see KKT conditions listed in equation 4.3). Figure 2 shows the normed error of NMF estimation after a fixed number of iterations with Alg.1 and the projection gradient method. The relative error obtained with multiplicative update increases with the size of the input matrix (see Fig. 3), though this is consistent with results from the projection gradient algorithm. It is important to note that the multiplicative approach will breakdown if a zero appears in the denominator. In practice, it is common to add a very small floating point value to the denominator during each iteration. The required work of O(mnk) per iteration is quite high compared to other exiting algorithms and multiplicative update tends to converge slowly for real world problems. Nevertheless, with its early establishment and its straightforward implementation this algorithm has maintained its popularity for NMF computation.

\begin{figure}[t!]
  \centering
    \includegraphics[width=.5\linewidth]{NMF_projection_gradient_vs_multiplicative_update_error}
  \caption{Estimation error results for multiplicative update and projection gradient methods increase at approximately the same rate with increasing matrix size.}
\end{figure}

\section{Alternating Least Squares} Alternating least squares can be viewed as a special case of block coordinate descent method. For this class of algorithms, the objective function is minimized with respect to a coordinate vectors $x^{k}_{i}$. Multidimensional vectors are minimized in cyclical order, with one block coordinate vector minimized during each iteration \cite{Ber}. A single iteration takes the form:
\begin{equation}\label{bcv1}
 x^{k+1}_{i} \in {\mathrm{argmin}_{\varepsilon \in X_i}} = f( x^{k+1}_{1}, ..., x^{k+1}_{i-1}, \varepsilon, x^{k}_{i+1}, ..., x^{k}_{n})
\end{equation}
For the case of NMF, this scenario is especially simple as there are only two block variables, W and H. Sequentially one matrix is fixed and the other is improved.
\begin{equation}\label{bcv2}
\begin{align}
W^{k+1} = {\mathrm{argmin}_{W \ge 0} f(W,H^k}) \\
H^{k+1} = {\mathrm{argmin}_{H \ge 0} f(W^{k+1},H})
\end{align}
\end{equation}
This takes advantage of the fact that while objective (2.1) is not convex in both W and H, it is convex in either W or H individually. One basic ALS algorithm that excites this cycle is outlined by Berry et al. \cite{Ber}. This strategy imposes nonnegativity by setting all nonzero entries of W and H to zero during each iteration. Unlike multiplicative update rules, however these zero entries are not forced to remain at 0 for all proceeding iterations. \\

\begin{algorithm}[H]
 \KwData{Input data matrix:  $\bf{A} \in \Re^{m x n}$}
 \KwResult{nonnegative factorization of \textbf{A} using k components, creating matrices \textbf{W} $ \in \Re^{m x k}$ and \textbf{H} $ \in \Re^{k x n}$ }

 initialization\;
$\bf{W} \gets $ random dense (m x k) matrix\\
$\bf{H} \gets $ random dense (k x n) matrix\\
 \For{i = 1 to \text{maxiter} }{
 	Solve for $\bf{H}$ in $\bf{W^T WH = W^T A} $

	set negative elements in $\bf{H}$ to 0

 	Solve for $\bf{W}$ in $\bf{HH^T W^T = HA^T}$

	set negative elements in $\bf{W}$ to 0	}
 \caption{Alternating least squares}
\end{algorithm}

In basic form of ALS (Alg. 2), each subproblem is solved in an unconstrained manner and the resulting W and H solutions are modified during each iteration to satisfy the desired non-negativty constraints. The simplicity of setting negative values in W and H to zero provides for an especially fast approach as demonstrated in the comparisons of this study. Theoretical evaluation of this algorithm's convergence properties, however, are difficult because each subproblem is formulated as an unconstrained least squares problem, but the solutions are directly modified and therefore do not map onto the original formulations. Other problem definitions allow for a framework to better understand the convergence properties of the two block coordinate descent method. Alternating non-negativity constrained least squares (ANLS), for instance, provides a formulation that directly incorporates non-negative constraints in each block coordinate subproblem \cite{Kim1}. Adding the $W \ge 0$ and  $ H \ge 0 $ constraints onto equation (4.2), the Karush-Kuh-Tucker (KKT) optimality conditions define stationary points of the objective function (2.1) if and only if

\begin{figure}[t!]
  \centering
    \includegraphics[width=.5\linewidth]{NMF_ALS_vs_multiplicative_update_iteration_error}
  \caption{Estimation error resulting from the alternating least squares algorithm compared to multiplicative update.}
\end{figure}

\begin{equation}\label{EKx}
 \begin{align}
  W \ge 0,\\
  \bigtriangledown_W f(W,H)  = WHH^T - AH^T \ge 0,\\
  W.* \bigtriangledown_Wf(W,H) = 0\\ \\
    H \ge 0,\\
  \bigtriangledown_H f(W,H)  = W^T WH - W^T A \ge 0,\\
  H.* \bigtriangledown_Hf(W,H) = 0
  \end{align}
\end{equation}
For the block coordinate descent algorithms, it has been shown that the limit point of a sequence of sub-blocks is a stationary point if each subproblem has a unique solution \cite{Ber2}. Unfortunately, ANLS subproblems are not unique as there exists matrices in the form $ \textbf{X} \in \Re^{kxk} $ which represent scaling and permutation that satisfy $ || \textbf{A - WH} ||_{F} = || \textbf{A - WXX}^{-1} \textbf{H} ||_{F} $ \cite{Kim1}.  In the case of two bock problems, however, stationary points can be found as the limit point of the sequence of two sub-block solutions \cite{Gri}. One classical approach to solving the two block subploblems is by way of an $\textit{active set algorithm}$ \cite{Law}. Here, the difference between a input matrix $ Y \in \Re^{lxp}$  and its resultant factorization can be split into vectors:
\begin{equation}
    \min_{G \ge 0} || BG-Y ||^{2}_{F} \to  \min_{G \ge 0} || B\textbf{g}_1-\textbf{y}_1 ||^{2}_{2}, ...,   \min_{G \ge 0} || B\textbf{g}_l-\textbf{y}_m ||^{2}_{2}
\end{equation}
where $ B \in \Re^{lxm}$ and $y_i \in \Re^{lx1}$. Since there are m inequity constraints after the split, the ith constraint will be active if the ith regression coefficient is set to zero. Thus if the active set is known, active constraints can be treated as equality constraints instead of inequality constraints. The overall least square problem is solved by examining only the variables in the passive set and calculating their unconstrained least squares solution \cite{Bro}. The ANLS algorithm begins with an initial feasible set of regression coefficients. The zero vector of coefficients is often used as an initial feasible set as no constraints are violated. ANLS proceeds by iteratively removing variables from the active set until the true active set is found. The variables removed from the active set are used in an unconstrained linear regression to calculate the block coordinate descent solution. While the convergence properties of ANLS are better understood, this report omits this strategy in comparative analysis due to the complexity of the algorithm and its slow computation. Recent work has succeeded in improving ANLS speed by precomputing cross-product terms during the unconstrained linear regression calculations \cite{Bro}.

\section{Sparsity in Alternating Least Squares} Sparse basis vectors often naturally arise during nonnegative matrix factorization. The are two potential benefits of sparsity in NMF representations. First, there is potential for savings in storage and future computation relative to alternative low-rank approximations like SVD that yield dense factorization results. Second and more importantly, sparsity may aid in interpreting the NMF results. It is often desirable to understand how an input sample data can be constructed using a small number of parts-based representations, rather than being created from a linear combination of many diffuse components. The property of spareness, therefore, can aid in giving physical meaning to NMF results. A number of approaches have been created in order to manipulate sparsity in nonnegative matrix factorization \cite{Li, Kim1, Kim2}. To better control the fraction of zero or near zero entries, for example, Kim and Park propose directly modifying The objective function of the ALS method \cite{Kim2}.
\begin{equation}\label{ObjSpar}
 f( \W,H) = \frac{1}{2} [|| A - WH ||^{2}_{F} + \lambda_W || W ||^{2}_{F} + \lambda_H || H ||^{2}_{F} ]
\end{equation}
Here additional terms are used to penalize large positive values in the W and H factorization. The new penalties to objective function, therefore, encourage sparse solutions. The modified objective function can be easily incorporated into the two block coordinate framework providing new update rules for the W and H matrices:
\begin{equation}
    \min_{W \ge 0} || \left( \begin{array}{c} H^T   \\  \sqrt{\lambda_W}I_k \\ \end{array} \right) W^T- \left( \begin{array}{c} A^T   \\  0_{kxm} \\ \end{array} \right) ||^{2}_{F}
\end{equation}
where $I_k$ is the $k\text{ x }k$ identity matrix and  $0_k$ is a zero matrix with the size $k\text{ x }m$. Similarly the update rule for the H matrix can be modified to:
\begin{equation}
    \min_{H \ge 0} || \left( \begin{array}{c} W   \\  \sqrt{\lambda_H}I_k \\ \end{array} \right) H- \left( \begin{array}{c} A   \\  0_{kxm} \\ \end{array} \right) ||^{2}_{F}
\end{equation}
These new update rules, called alternating constrained least squares (ACLS), were compared against the original alternating least squares approach outlined in section 4. The analysis presented in Fig.1 shows the fraction of non-zero entries in the resulting W matrix for ALS and ACLS factorization. Similar results can be obtained by manipulating the $\lambda_H$ parameter for the H factorization matrix. These results show a powerful approach for researchers to control NMF sparsity according to their research question.
\begin{figure}[t!]
  \centering
    \includegraphics[width=.5\linewidth]{ALS_vs_ACLS_sparsity}
  \caption{NMF sparsity manipulation using the ACLS method. Increasing the value of $\lambda$ allows users to increase the fraction of near-zero entries in W and H.}
\end{figure}

\section{Applications to gene expression analysis}
A classical example of NMF is the factorization of a large database of human face images. Factorization using PCA will yield a series of `eigenfaces' which can be added or subtracted, but individual components are not easy to interpret on their own. The NMF parts-based representation however, yields components like noses, eyes, ears, and mouths which can be added linearly to construct a face. Parts-based representations have been helpful in identifying and interpreting patterns in a number of biological contexts. For example, NMF has shown to be less sensitive to \textit{a priori} gene selection or initial conditions when identifying context-dependent patterns of expression profiles \cite{Bur}.

To examine the ability of NMF to identify unique parts-based representations in gene expression profiles, experimental data was obtained from the Connectivity Map Projects (lincscloud.org). This database represents the world's largest systematic collection of gene expression profiles performed in cancer cell lines. A small fraction of drug profiles were obtained from the database. Specifically, those drugs which were tested in MCF7 cells and fit into one of four pharmacological classes; Topoisomerase II inhibitors, RAR agonists, PKC activators, and JNK inhibitors. These represent important chemical tools for research and clinical practice. The advantage to examining this selection of drugs is that chemicals grouped in the same pharmacological classes are expected to have similar gene expression responses. The NMF was tested for its ability to recover these known relationships.

\begin{figure}[t!]
  \centering
    \includegraphics[width=.5\linewidth]{NMF_pharmacological_classification}
  \caption{Nonnegative matrix factorization used to identify parts-based representations in gene expression profiles. The weighting factors from the H matrix are displayed for drugs from four pharmacological class tested in MCF7 cells. Drug profiles from the same pharmacological class can be constructed using similar NMF components.}
\end{figure}

Both ALS and multiplicative update approaches were applied to this data. The number of components was selected \textit{a priori} at k=20. Fig.6 shows H weighting factors for the 20 NMF components obtained with multiplicative update rule for each of the four pharmacological classes. Similar patterns were also found for ALS (though with different component ordering.) The similarity of both these methods suggests researchers might opt to spend their time examining other numerical factors beside convergence that may influence biological interpretation of NMF. The effects of sparsity were not examined with this particular dataset, but could have a significantly influence on the interpretation of the NMF results. Other factors, including smoothness, and the uniqueness of factorization could also impact the biological findings. New numerical approaches that give better control of the component constraints could improve the utility of NMF in biomedical research.

\section{Conclusion} Despite the simplicity of defining nonnegative matrix factorization, the landscape of numeric methods used to address the problem remains complex. Recent efforts have sought to create new NMF algorithms with well defined convergence properties. Methods such as alternating non-negativity constrained least squares (ANLS) have succeeded in improving our analytical understanding of NMF convergence, but remain slow and cumbersome when compared to simpler methods that converge in most real-world scenarios. When evaluating NMF tools, researchers should be guided by the parameters of their data and the research question of interest. For especially large datasets the speed and simplicity ALS might serve as a reasonable starting point for NMF calculation. Researchers with modest datasets, might prefer ANLS or the multiplicative update approach. Finally, the results presented in this report also show that researchers can manipulate the sparsity of factorization by adopting specialized objective functions.

\begin{thebibliography}{1}

\bibitem{Lan} {\sc A. Langville, C. Meyer, R. Albright},
{\em Initializations for the nonnegative matrix factorization}, Preprint (2006)

\bibitem{Pau} {\sc V Pauca et al},
{\em Text mining using non-negative matrix factorizations}, Proceedings SIAM International Conference on Data Mining (2004) SDM'04.

\bibitem{Paa1} {\sc P. Paatero},
{\em Positive matrix factorization; a non-negative factor model with optimal utilization of error estimates of data values}, Environmetrics (1994) 5, 111-126.

\bibitem{Paa2} {\sc P. Paatero},
{\em Least squares formulation of robust non-negative factor analysis}, Chemometrics and Intell. Laboratory Syst. (1997) 37, 23-35.

\bibitem{Paa3} {\sc P. Paatero},
{\em The multilinear engine - a table-driven least squares program for solving multilinear problems, including the n-way parallel factor analysis model}, J. Comput. Graphical Statist. (1999) 8, 1-35.

\bibitem{LeSe1} {\sc D. Lee, H. Seung},
{\em Learning the parts of objects by non-negative matrix factorization}, Nature (1999) 401, 788-791.

\bibitem{LeSe2} {\sc D. Lee, H. Seung},
{\em Algorithms for Nonnegative Matrix Factorization}, Advances in Neural Information Processing Systems (2001) 13, 556-562.

\bibitem{Bur} {\sc J. Brunet et al. },
{\em Metagenes and molecular pattern discovery using matrix factorization}, Proceedings of the Nat. Acad. Sci (2004) 101, 4164-4169.

\bibitem{Dev1} {\sc K. Devarajan, K. Ebrahimi },
{\em Molecular pattern discovery using non-negative matrix factorization based on Renyi's information measure}, Preprint (2005)

\bibitem{Chu} {\sc M. Chu, D. Plemmons, S. Ragni},
{\em Optimality, computation, and interpretation of nonnegative matrix factorizations},  SIAM Journal on Matrix Analysis (2004)

\bibitem{Gon} {\sc E. Gonzalez, Y. Zhang},
{\em Accelerating the lee-swung algorithm for nonnegative matrix factorization}, Technical Report TR-05-02, Rice University (2005)

\bibitem{Lin} {\sc C. Lin},
{\em Projected gradient methods for non-negative matrix factorization}, Information and Support Services Technical Report (2005)

\bibitem{Ber} {\sc M. Berry, M. Browne, A. Langville, P. Pauca, R. Plemenos},
{\em Algorithms and applications for approximate nonnegative matrix factorization}, Computational Statistics and Data Analysis 52 (2007) pp. 155 - 173.

\bibitem{Ber2} {\sc D. Bertsekas},
{\em Nonlinear Programing}, Athena Scientific, Belmont, MA (1999)

\bibitem{Gri} {\sc L. Grippo et al.},
{\em On the convergence of the block nonlinear Gauss-Seidel method under convex constraints}, Oper. Res. Lett (2000) 26(3), p. 127-136.

\bibitem{Law} {\sc C. Lawson, R. Hanson},
{\em Solving Least Squares Problems}, SIAM, Philadelphia, PA (1995)

\bibitem{Bro} {\sc R. Bro, S. Jong},
{\em A fast non-negativity constrained linear least squares algorithm}, J. Chemometrics (1997) 11), p. 393-401.

\bibitem{Cic} {\sc A. Cichocki, R. Zdunek, A. Huy Phan, S. Amari}, {\em Nonnegative Matrix and Tensor Factorizations}, Wiley, Natick, MA, 2009.

\bibitem{Kim1} {\sc H. Kim, H. Park}, {\em Nonnegative matrix factorization based on alternating nonnegativity constrained least squares and active set method}, SIAM Journal on Matrix Analysis and Applications (2008) 30(2), 713?730.

\bibitem{Kim2} {\sc H. Kim, H. Park}, {\em Sparse non-negative matrix factorizations via alternating non-negativity-constrained least squares for microarray data analysis}, Bioinformatics (2007) 23, 1495-1502.

\bibitem{Alb} {\sc R. Albright, J. Cox, et al.},
{\em Algorithms, initializations, and convergence for the nonnegative matrix factorization}, Preprint (2014)

\bibitem{Lin} {\sc C. Lin},
{\em Projected Gradient Methods for Nonnegative Matrix Factorization}, Neural Computation (2007) 19, 2757-2779.

\bibitem{Li} {\sc S. Li},
{\em Learning spatially localized parts-based representations}, Pro. IEEE Conference on Computer Vis. and Patt. Rec. (2001) pp. 207-212.

\end{thebibliography}

\end{document}
%% end of file `docultex.tex'


\documentclass[11pt]{article}
\usepackage[utf8]{inputenc}
\usepackage[top=2.0cm, bottom=1.5cm, inner=5.9cm, outer=1.5cm, marginparwidth=3.9cm, marginparsep=0.5cm,footskip=0.5cm]{geometry}
\usepackage{fancyhdr}
\usepackage{titlesec}
\usepackage{titling}

\reversemarginpar

\fancyhf{}
\pagestyle{fancyplain}
\fancyfoot[R]{\footnotesize vXXX}

\setlength{\parindent}{0em}
\setlength{\parskip}{0.2em}

\titlespacing*{\section}{0pt}{1em}{0.5em}

\pretitle{\begin{center}\LARGE}
\posttitle{\par\end{center}}
\predate{}
\date{\vspace{-1em}}
\postdate{}

\title{How to Solve It}
\author{George P\'{o}lya}

\begin{document}

\maketitle

\section*{Understanding the Problem}

\emph{What is the unknown?}
\marginpar{
  \textbf{First}.

  You have to \emph{understand} the problem.
}
\emph{What are the data?}
\emph{What is the condition?}

Is it possible to satisfy the condition? Is the condition sufficient to determine the unknown? Or is it insufficient? Or redundant? Or Contradictory?

Draw a figure.
Introduce suitable notation.

Separate the various parts of the condition.
Can you write them down?

\section*{Devising a plan}

Have you seen it before?
\marginpar{
  \textbf{Second}.

  Find the connection between the data and the unknown.
  You may be obliged to consider auxiliary problems if an immediate connection cannot be found.
  You should obtain eventually a \emph{plan} of the solution.
}
Or have you seen the same problem in a slightly different form?

\emph{Do you know a related problem?}
Do you know a theorem that could be useful?

\emph{Look at the unknown!} And try to think of a familiar problem having the same or a similar unknown.

\emph{Here is a problem related to yours and solved before. Could you use it?}
Could you use its result?
Could you use its method?
Should you introduce some auxiliary element in order to make its use possible?

Could you restate the problem?
Could you restate it still differently?
Go back to definitions.

If you cannot solve the proposed problem try to solve first some related problem.
Could you imagine a more accessible related problem?
A more general problem?
A more special problem?
An analogous problem?
Could you solve a part of the problem?
Keep only a part of the condition, drop the other part; how far is the unknown then determined, how can it vary?
Could you derive something useful from the data?
Could you think of other data appropriate to determine the unknown?
Could you change the unknown or the data, or both if necessary, so that the new unknown and the new data are nearer to each other?

Did you use all the data?
Did you use the whole condition?
Have you taken into account all essential notions involved in the problem?

\section*{Carrying out the Plan}
Carrying out your plan of the solution, \emph{check each step}.
\marginpar{
  \textbf{Third}.

  \emph{Carry out} yout plan.
}
Can you see clearly that the step is correct?
Can you prove that it is correct?

\section*{Looking Back}
Can you \emph{Check the result}?
\marginpar{
  \textbf{Fourth}.

  \emph{Examine} the solution obtained.
}
Can you check the argument?

Can you derive the result differently?
Can you see it at a glance?

Can you use the result, or the method, for some other problem?
\end{document}