Optimal Reduced Isotonic Regression

Abstract: Isotonic regression is a shape-constrained nonparametric regression in which the regression is an increasing step function. For $n$ data points, the number of steps in the isotonic regression may be as large as $n$. As a result, standard isotonic regression has been criticized as overfitting the data or making the representation too complicated. So-called "reduced" isotonic regression constrains the outcome to be a specified number of steps $b$, $b \leq n$. However, because the previous algorithms for finding the reduced $L_2$ regression took $\Theta(n+bm^2)$ time, where $m$ is the number of steps of the unconstrained isotonic regression, researchers felt that the algorithms were too slow and instead used approximations. Other researchers had results that were approximations because they used a greedy top-down approach. Here we give an algorithm to find an exact solution in $\Theta(n+bm)$ time, and a simpler algorithm taking $\Theta(n+b m \log m)$ time. These algorithms also determine optimal $k$-means clustering of weighted 1-dimensional data.

\subsubsection*{Abstract}
Isotonic regression is a shape-constrained nonparametric regression in
which the regression is an increasing step function.
For $n$ data points, the number of steps in the isotonic regression
may be as large as $n$.
As a result, standard isotonic regression has been
criticized as overfitting the data or making the representation too complicated.
So-called ``reduced'' isotonic regression constrains the outcome to be a specified number of steps $b$, $b \leq n$.
However, because the previous algorithms for finding the reduced $L_2$ regression took $\Theta(n+bm^2)$ time, where $m$ is the number of steps of the unconstrained isotonic regression, researchers felt that the algorithms were too slow and instead used approximations.
Other researchers had results that were approximations because they used a greedy top-down approach.
Here we give an algorithm to find an exact solution in $\Theta(n+bm)$ time,
and a simpler algorithm taking $\Theta(n+b m \log m)$ time.
These algorithms also determine optimal $k$-means clustering of weighted 1-dimensional data.
\medskip

\noindent
\textbf{Keywords}: reduced isotonic regression, step function, v-optimal histogram,
piecewise constant approximation, k-means clustering, nonparametric regression

\section{Introduction}

Isotonic regression is an important form of nonparametric regression
that allows researchers to relax parametric assumptions
and replace them with a weaker shape constraint.
A real-valued function $f$ is \textit{isotonic} iff for all $x_1, x_2$ in its domain,
if $x_1 < x_2$ then $f(x_1) \leq f(x_2)$.
In some settings isotonic functions are called monotonic, while in others monotonic is used to indicate either nondecreasing or nonincreasing.
Myriad uses of isotonic regression can be found in citations to the fundamental books of
Barlow et al.~\cite{BarlowetalBook}
and Robertson et al.~\cite{RobertsonWrightDykstra}.
Nonparametric approaches are increasingly important as researchers encounter situations where parametric assumptions are dubious, and as algorithmic improvements make the calculations practical.

Isotonic regression is useful for situations in which the independent variable
has an ordering but no natural metric, such as S $<$ M $<$ L $<$ XL clothing sizes.
Since the only important property of the domain is its ordering,
we assume that it is the integers $1 \ldots n$ for some $n$, and use
\inter{i}{j}, $1 \leq i \leq j \leq n$ to denote the range $i \ldots j$.
By \textit{weighted values} \ensuremath{(\vec{y},\vec{w})}\ on \inter{1}{n}, we mean values $(y_i,w_i)$,
$i \in \inter{1}{n}$, where the $y$ values are arbitrary real numbers
and the $w$ values (the weights)
are nonnegative real numbers.
Given weighted values \ensuremath{(\vec{y},\vec{w})}\ and a real-valued function $f$ on \inter{1}{n},
the $L_p$ regression or approximation error of $f$ is
$$
\begin{array}{ll}
  \left(\sum_{i=1}^n w_i |y_i -f(i)|^p\right)^{1/p} & 1 \leq p < \infty \medskip \\
  \max_{i=1}^n w_i |y_i - f(i)|                     & p = \infty
\end{array}
$$
An $L_p$ \textit{isotonic regression} is an isotonic function that minimizes
the $L_p$ error among all isotonic functions.
Figure~\ref{fig:Regressions}~a) gives an example of an isotonic regression.
Because researchers from varying fields often use different expressions for a single concept,
we use the terms {\it regression} and {\it approximation} interchangeably. We identify approximations that are not optimal regressions as {\it sub-optimal} approximations.
\begin{figure}

\begin{minipage}[b]{2.05in}
\begin{center}
\resizebox{1.85in}{!}{\includegraphics{Regression_Iso.eps}}
\smallskip

a) isotonic regression
\end{center}
\end{minipage}
\hspace{0.15in}
\begin{minipage}[b]{2.05in}
\begin{center}
\resizebox{1.85in}{!}{\includegraphics{Regression_4Piece.eps}}
\smallskip

b) 4-step regression
\end{center}
\end{minipage}
\hspace{0.15in}
\begin{minipage}[b]{2.05in}
\begin{center}
\resizebox{1.85in}{!}{\includegraphics{Regression_4Iso.eps}}
\smallskip

c) 4-step reduced isotonic
\end{center}
\end{minipage}

\caption{Stepwise regressions, size indicates weight}\label{fig:Regressions}
\hrulefill
\end{figure}

Isotonic regressions are  step functions for which
the number of steps is determined by the data.
In certain cases there is criticism that such functions can overfit the data~\cite{NiculescuMizilCaruana,SalantiUlmReduced,SchellSingh}
or produce a result with too many steps~\cite{HaiminenetalReducedUnimodal}.
Consequently, some researchers utilize isotonic regressions that
restrict the number of steps.
Schell and Singh~ \cite{SchellSingh} have referred to such functions as
\textit{reduced} isotonic regressions.

Restricting the number of steps is a central issue in approximation by step functions. It arises in settings such as databases and variable width
histogramming~\cite{HalimetalFastHistogram,IoannidisIncreasingHist,PoosalaetalHistogramSurvey}, segmentation of time series and genomic data~\cite{HimbergetalTimeSeries,JacobetalFuzzySegmentation,TerziSegmentation},
homogenization~\cite{FisherHomogeneity}
and piecewise constant approximations~\cite{MaysterLopezStep}.

A function $f$ is an \textit{optimal $L_p$ $b$-step approximation}, $1 \leq b \leq n$,
iff it minimizes the $L_p$ error over all functions with $b$ steps.
Here we are primarily concerned with computing $L_2$ $b$-step reduced isotonic regressions, where
a function
 $f$ is an \textit{optimal $L_p$ $b$-step reduced isotonic regression},
$b=1,\ldots,m \leq n$,
iff it minimizes the $L_p$ error over all isotonic functions having $b$ steps.
Figure~\ref{fig:Regressions} gives examples of $b$-step regression and $b$-step reduced isotonic regression.
Optimal $b$-step approximations and $b$-step reduced isotonic regressions are not always unique.
For example, with unweighted values 1, 2, 3 on \inter{1}{3} and $b=2$,
for any $p$ the function
which is 1.5 on \inter{1}{2} and 3 at 3 is optimal, as is the function which is 1 at
1 and 2.5 on \inter{2}{3}.

In 1958 Fisher~\cite{FisherHomogeneity} gave a simple algorithm
for determining an optimal $b$-step $L_2$ regression in $\Theta(b n^2)$ time
(this is shown in Algorithm~\ref{alg:FisherDP}).
His algorithm can be easily modified to determine an optimal $b$-step $L_2$ reduced isotonic regression in the same time bounds.
His algorithm has been widely used and rediscovered, and often falsely attributed to Bellman.
However, for many researchers the quadratic time in $n$ makes it too slow for their
applications~\cite{HaiminenetalReducedUnimodal,HalimetalFastHistogram,HimbergetalTimeSeries,JacobetalFuzzySegmentation,TerziSegmentation}.
Thus most previous work utilizing reduced isotonic regression used sub-optimal approximations, with the exception of an algorithm due
to Haiminen, Gionis and Laasonen~\cite{HaiminenetalReducedUnimodal}.
Their algorithm for the $L_2$ metric takes $\Theta(n + bm^2)$ time, where $m$ is the number of pieces of the unrestricted isotonic regression.
(To lessen confusion, we use ``pieces'' to refer to the steps of the unrestricted isotonic regression.)
However, even with this reduction in time they then developed an approximation algorithm based on a greedy heuristic.

In Section~\ref{sec:Reduced} we decrease the time to find the optimal $b$-step $L_2$ reduced isotonic regression to $\Theta(n + b m)$, using an
algorithm in Section~\ref{sec:Isotonic} for the special case in which the values are themselves isotonic.
A simpler algorithm, taking $\Theta(n + b m \log m)$ time, is also given.
These algorithms should be fast enough to eliminate the need for approximations, even for very large data sets.

Since we are only looking for optimal approximations, we often omit ``optimal''.

\section{Approximation by Step Functions}\label{sec:ArbitraryStep}

A real-valued function $f$ on \inter{1}{n} is a
\textit{$b$-step function}, $1 \leq b \leq n$,
iff there are indices
$j_0 = 0 < j_1  \ldots < j_b=n$
and real values
$C_k$, $k \in \inter{1}{b}$,
such that $f(x_i) = C_k$ for $i \in \inter{j_{k\!-\!1}+\!1}{j_k}$.
If $f$ is isotonic then $C_1 \leq C_2 \ldots \leq C_b$.
An approximation with fewer than $b$ steps can be converted to a $b$-step
approximation by merely subdividing steps, and thus we do not differentiate
between ``$b$ steps'' and ``no more than $b$ steps''.

Let \mean{p}{i,j} denote an $L_p$ mean of the weighted values on \inter{i}{j}.
For $1 \leq p < \infty$, an optimal $L_p$ step function
has the property that $C_k = \mean{p}{j_{k\!-\!1}+\!1,\,j_{k}}$.
Since we are only concerned with optimal approximations, whenever
a function has a step \inter{i}{j}, then its value on that step is \mean{p}{i,j}.
Let \err{p}{i,j} denote the $p^\mathrm{th}$ power of the $L_p$ error of the step \inter{i}{j}.
Minimizing the sum of the \err{p}{} values is the same as minimizing the
$L_p$ approximation error and thus from now on only the \err{p}{} values will be used.

\subsection{Arbitrary Data}\label{sec:ArbitraryData}

Fisher's~\cite{FisherHomogeneity} dynamic programming approach to determining
an optimal $L_p$ $b$-step approximation for $1 \leq p < \infty$
is based on the observation that
if $f$ is an optimal $b$-step approximation of the data, with a first step of \inter{1}{j},
then $f$ is an optimal $(b\!-\!1)$-step approximation of
the data on \inter{j\!+\!1}{n}.
This is obvious since if it were not optimal then replacing it with
an optimal $(b\!-\!1)$-step approximation would reduce the error.
Let \Err{i,c} denote the sum of the \err{p}{} values of the steps of an
optimal $c$-step approximation on \inter{i}{n}, and let
\Errj{i,j,c} denote the sums of the \err{p}{} values of the steps of a
$c$-step approximation on \inter{i}{n} which is optimal among $c$-step approximations
where the first step is \inter{i}{j}.
Fisher's observation yields the equations:
\begin{eqnarray}
 \Errj{i,j,c} & = & \err{p}{i,j} + \Err{j\!+\!1,c\!-\!1}   \label{eqn:Errj}\\
 \Err{i,c} & = & \min\{\Errj{i,j,c}: i \leq j \leq n-c+1\}
\end{eqnarray}
By storing the $j$ that minimizes \Err{i,c} in \jopt{i,c},
in $\Theta(n)$ time
one can generate the optimal approximation after the dynamic programming has completed.
This leads to Algorithm~\ref{alg:FisherDP}.
The time is $\Theta(bn^2)$ plus the time to compute the
$\Theta(n^2)$ \err{p}{} values.
For $L_\infty$, $\Errj{i,j,c} = \max\{\err{\infty}{i,j},~ \Err{j\!+\!1,c\!-\!1}\}$.

Fisher's algorithm can be modified to determine the $b$-step reduced isotonic regression in the same time bounds.
The lines
\medskip

\noindent
\setlength{\Ainindent}{0.5in}
\Ain{1}    $\mathsf{for~ i=1 ~to~ n-c+1}$\\
\Ain{2}       $\mathsf{\Err{i,c} = min\{\Errj{i,j,c}: i \leq j \leq n-c+1\}}$
\medskip

\noindent
should be replaced by
\medskip

\noindent
\setlength{\Ainindent}{0.0in}
\Ain{0} \hspace*{-0.05in}$\mathsf{for~ i=1 ~to~ n-1}$\\
\Ain{0} \hspace*{0.11in} $\mathsf{\Err{i,c} = min \bigl\{\err{p}{i,n},~\min\{\Errj{i,j,c}: i \leq j \leq n\!-\!1,~  mean_p(i,j) \leq mean_p(j\!+\!1,\,\jopt{j\!+\!1,c\!-\!1})\,\}\, \bigr\}}$
\medskip

\noindent
Including the $\err{p}{i,n}$ term, and changing the upper bound on $\mathsf{i}$, is necessary so that, say, for unweighted data 3, 2, 1, the $L_2$ 2-step reduced isotonic regression is correctly determined to be 2, 2, 2.
Using either 3, or 3, 2, as the initial step would involve a second step that was lower, and hence the solution has only 1 step.

Throughout, the values of \Err{} and \jopt{} are stored in 2-dimensional arrays, while \Errj{} is evaluated as a function, not stored as a 3-dimensional array.
To evaluate \err{2}{}, once the scan values $\sum_{j=1}^i w_j y_j$,\,
$\sum_{j=1}^i w_j y_j^2$,\, and $\sum_{j=1}^i w_i$ have been determined for all $i \in \inter{1}{n}$,
each \err{2}{} value can then be computed in unit time.

\begin{algorithm}
\setlength{\Ainindent}{1.0in}

\Ain{0} $\mathsf{for~ i = 1~ to~ n}$\\
\Ain{1}    $\mathsf{e(i,1) = \err{p}{i,n};~~~ \jopt{\mathsf{i,1}} = i}$\\
\Ain{0} $\mathsf{for~ c= 2~ to~ b}$\\
\Ain{1}    $\mathsf{for~ i=1 ~to~ n-c+1}$\\
\Ain{2}       $\mathsf{\Err{i,c} = min\{\Errj{i,j,c}: i \leq j \leq n-c+1\}~~~~\{\Errj{}~ is~ defined~ in~ (\ref{eqn:Errj})\}}$\\
\Ain{3}           $\mathsf{\{record~ minimizing~ j~ in~ \jopt{\mathsf{i,c}}\}}$\\
\Ain{1} $\mathsf{end~ for~ i}$\\
\Ain{0} $\mathsf{end~ for~ c}$\\
\Ain{0} $\mathsf{generate~ the~ approximation~ using~ \jopt{}~ and ~\mean{p}{}}$

\vspace*{0.3in}
\hrulefill
\vspace*{-0.4in}

\caption{Fisher's algorithm for optimal $L_p$ $b$-step approximation of arbitrary data, $1 \leq p \leq \infty$} \label{alg:FisherDP}

\end{algorithm}

\subsection{Isotonic Data}\label{sec:Isotonic}

Reducing the time of Algorithm~\ref{alg:FisherDP}
requires reducing the number of \err{p}{} values referenced.
It is not known how to do this for arbitrary data, but isotonic data has some special properties.
We give two algorithms: Algorithm~\ref{alg:monotonic} is simpler than Algorithm~\ref{alg:totallymonotonic},
but, in O-notation, slower by a logarithmic factor.
It is likely that many will prefer Algorithm~\ref{alg:monotonic} over Algorithm~\ref{alg:totallymonotonic}.
Algorithm~\ref{alg:monotonic} is given in Section~\ref{sec:monotonic}, and Algorithm~\ref{alg:totallymonotonic} is in Section~\ref{sec:totallymonotonic}.

For isotonic data, the fact that values are nondecreasing allows one to make
inferences concerning the means of intervals.
For example, the $L_p$ mean of the weighted values on \inter{i}{j} is no larger than
that of the values on \inter{i\!+\!1}{j}.
Further, for any $1 < i \leq j < n$,
$\err{p}{i,j+1} - \err{p}{i,j} \geq \err{p}{i+1,j+1} - \err{p}{i+1,j}$.
That is, if we consider the increase in error of adding $(x_{j+1},w_{j+1})$ to the step \inter{i}{j},
this is greater than the increase when adding it to the step \inter{i+1}{j}.
This is true because the monotonicity insures that $x_{j+1}$ is at least as large as the mean on \inter{i+1}{j}, which has a mean not more than that of \inter{i}{j}, and the total weight of \inter{i}{j} is greater than the total weight of \inter{i+1}{j}.
When the values are not isotonic then this inequality may not hold.

Letting $M(i,j) = \err{p}{i,j}$, this can be rewritten as
\begin{equation}
M(i,j\!+\!1) + M(i\!+\!1,j) \geq M(i,j) + M(i\!+\!1,j\!+\!1) \label{eqn:Monge1}
\end{equation}
for all $1 \leq i < j < n$ and $1 \leq p \leq \infty$.
This is known as the \textit{Monge property}, and $M$ is known as a Monge matrix
(typically the Monge property has the inequality in the opposite order and is applied to maximization, not minimizing).

If \jopt{i} denotes the smallest $j$ such that $M(i,j)$ is a minimal value in row $i$ of $M$,
then the Monge property implies that for any $i < i^\prime$, ~$\jopt{i} \leq \jopt{i^\prime}$,
i.e., \jopt{} is isotonic.
This property is typically called \textit{monotonicity}.
If we define $M(i,j)=\infty$ when $j<i$ then $M$ satisfies~(\ref{eqn:Monge1}) for all $i$ and $j$.
Iteratively combining this inequality over adjacent elements shows that it holds much more widely,
in that for all $1 \leq i_1 < i_2 \leq n$ and $1 \leq j_1 < j_2 \leq n$,
\begin{equation}
M(i_1,j_2) + M(i_2,j_1) \geq M(i_1,j_1) + M(i_2,j_2)  \label{eqn:Monge2}
\end{equation}
Thus all submatricies of a Monge matrix are Monge, where a submatrix can be formed from an arbitrary set of rows and an arbitrary set of columns and the number of rows need not equal the number of columns.
Since all submatricies are Monge, all are monotonic.
This property is called \textit{total monotonicity}.
There are monotonic matrices that are not totally monotonic and
totally monotonic matrices that aren't Monge.

The fact that $M$ is a Monge matrix implies that $M^c$ is a Monge matrix, for $c > 1$,
where $M^c(i,j) = \Errj{i,j,c}$.
This is because
\begin{eqnarray*}
M^c(i,j\!+\!1)+M^c(\!i+\!1,j) & = & M(i,j\!+\!1) + \Err{j\!+\!1,c\!-\!1} + M(i\!+\!1,j) + \Err{j\!+\!2,c\!-\!1} \\
M^c(i\!+\!1,j\!+\!1)+M^c(i,j) & = & M(i\!+\!1,j\!+\!1) + \Err{j\!+\!2,c\!-\!1} + M(i,j) + \Err{j\!+\!1,c\!-\!1}
\end{eqnarray*}
Algorithm~\ref{alg:monotonic}, in Section~\ref{sec:monotonic}, exploits the monotonicity of $M^c$ and Algorithm~\ref{alg:totallymonotonic}, in Section~\ref{sec:totallymonotonic}, exploits its total monotonicity.
We will show

\begin{theorem} \label{thm:isotonic}
Given $n$ isotonic weighted values \ensuremath{(\vec{y},\vec{w})}\ and number of steps $b \leq n$,
Algorithm~\ref{alg:monotonic} finds an optimal $L_2$ $b$-step
approximation (hence an optimal $L_2$ $b$-step reduced isotonic regression),
in $\Theta(b n \log n)$ time, and Algorithm~\ref{alg:totallymonotonic} finds one in $\Theta(bn)$ time.
$\Box$
\end{theorem}

\subsection{Using Monotonicity} \label{sec:monotonic}

Let \jopt{i,b} denote the smallest $j$ such that $\Errj{i,j,b} = \Err{i,b}$.
As noted, \jopt{\cdot,b} is an isotonic function.
This fact can be used to efficiently compute \Err{\cdot,b} and \jopt{\cdot,b}
from the values of \Err{\cdot,b-1} and \jopt{\cdot,b-1}.
Figure~\ref{fig:linearscan} shows an intermediate stage of the calculations for
a single stage.
The optimal first step for each multiple of 1/4 has been computed and now
the first step for each odd multiple of 1/8 needs to be determined.
For each of these, the possible values of the endpoint of the optimal first step
are the range indicated by the
dashed lines with the solid line indicating the part that any optimal first
step must include.

\begin{figure}
\centerline{\resizebox{3.5in}{!}{\includegraphics{LinearScan.eps}}}
\caption{Possible endpoints of odd multiples of 1/8}  \label{fig:linearscan}
\hrulefill
\end{figure}

This observation forms the basis of Algorithm~\ref{alg:monotonic}.
Compared to Fisher's algorithm, for fixed $c$, the order in which \Err{i,c} values are determined is changed, as is the range of $j$ values used to compute each value.

\begin{algorithm}
\setlength{\Ainindent}{1.5in}
\Ain{0} $\mathsf{j\_start \ldots j\_end:~ range~ of~ possible~ endpoints}$\\

\vspace*{-0.08in}
\Ain{0} $\mathsf{for~ i = 1~to~n~ do}$\\
\Ain{1} $\mathsf{   \Err{i,1} = \err{p}{i,n};~~~ \jopt{i,1} = n}$\\
\Ain{0} $\mathsf{for~ c= 2~ to~ b~ do}$\\
\Ain{1} $\mathsf{   for~ level = \lfloor \log_2 (n\!-\!c\!+\!1) \rfloor ~downto~ 0 ~do}$\\
\Ain{2} $\mathsf{      for~ i = 2^{level} ~to~ n-c+1 ~by~ 2^{level\!+\!1}~ do}$\\
\Ain{3} $\mathsf{         if~ i = 2^{level} ~then~ j\_start = j}$\\
\Ain{4} $\mathsf{            else~ j\_start = max\{i, \jopt{i-2^k,c}\}}$\\
\Ain{3} $\mathsf{         if~ i+2^{level} > n-c+1 ~then~ j\_end = n-c+1}$\\
\Ain{4} $\mathsf{            else~ j\_end = \jopt{i+2^{level},c}}$\\
\Ain{3} $\mathsf{         \Err{i,c} = min\{ \Errj{i,j,c}: j\_start \leq j \leq j\_end \}}$\\
\Ain{4} $\mathsf{         \{store~ largest~ minimizing~ j~ in~ \jopt{i,c}\}}$\\
\Ain{2} $\mathsf{      end~ for~ i}$\\
\Ain{1} $\mathsf{   end~ for~ level}$\\
\Ain{0} $\mathsf{end~ for~ c}$\\
\Ain{0} $\mathsf{generate~ the~ approximation~ using~ \jopt{}~ and ~\mean{p}{}}$

\vspace*{0.3in}
\hrulefill
\vspace*{-0.4in}

\caption{$b$-step $L_p$ approximation of isotonic data, using monotonicity}  \label{alg:monotonic}

\end{algorithm}

\begin{proposition} \label{prop:monotonic}
Given $n$ isotonic weighted values \ensuremath{(\vec{y},\vec{w})}\ and number of steps $b \leq n$,
Algorithm~\ref{alg:monotonic} finds an optimal $b$-step
$L_2$ approximation in $\Theta(b n \log n)$ time.
\end{proposition}
\noindent
\textit{Proof:\,}
Suppose that \Err{\cdot,c} and \jopt{\cdot,c} have been determined for
$i_1 < i_2 \ldots < i_k$.
Let $\ell_o \ldots \ell_k$ be such that $\ell_0 < i_1 < \ell_1 < i_2 \ldots < i_k <\ell_k$.
To determine \Err{\cdot,c} and \jopt{\cdot,c} for the $\ell$ values, note that since \jopt{\cdot,c} is isotonic
then $\jopt{\ell_0,c} \in \inter{\ell_0}{\jopt{i_1,c}}$,
$\jopt{\ell_1,c} \in \inter{\max\{\ell_1,\jopt{i_1,c}\}}{\jopt{i_2,c}}$, \ldots, and
$\jopt{\ell_k,c} \in \inter{\max\{\ell_k, \jopt{i_k,c}\}}{n\!-\!c\!+\!1}$.
Thus, to determine \Err{\ell_0,c} and \jopt{\ell_0,c} we only need to evaluate
\Errj{\ell_0,j,c} for $j \in \inter{\ell_0}{\jopt{i_1,c}}$;
to determine \Err{\ell_1,c} and \jopt{\ell_1,c} we only need to evaluate
\Errj{\ell_1,j,c} for $j \in \inter{\max\{\ell_1,\jopt{i_1,c}\}}{\jopt{i_2,b}}$;
and so forth; i.e., we need at most $n + k$ total evaluations.
In Figure~\ref{fig:linearscan}, this corresponds to the fact that the dashed lines
can overlap only at endpoints.
In $1 +\lfloor \log_2 n \rfloor$ iterations
all values of \Err{\cdot,c} and \jopt{\cdot,c} can be determined.
This gives Algorithm~\ref{alg:monotonic}.

To complete the proof we need to show that each iteration
of the ``$\mathsf{for~level}$'' loop
can be completed in $\Theta(n)$ time.
The $\mathsf{j\_start}$ and $\mathsf{j\_end}$ values that control the number of $j$ values examined guarantee that, over all $i$ values in in ``$\mathsf{for~level}$'' loop, a given $j$ value is used at most twice.
$\Box$

\subsection{Using Total Monotonicity} \label{sec:totallymonotonic}

The fact that $M^c$ is totally mononotonic can be used to further reduce the total number of $j$ values examined.
Algorithm~\ref{alg:totallymonotonic} replaces
\smallskip

\centerline{$\mathsf{\Err{i,c} = min\{ \Errj{i,j,c}: j\_start \leq j \leq j\_end \}}$}
\medskip

\noindent in Algorithm~\ref{alg:monotonic} with a while loop over a smaller set of $j$ values, reducing the worst-case total number used at level $k$ from $n-2^k+1$ to $\lfloor n/2^k \rfloor$.
These $j$ values are determined in Algorithm~\ref{alg:SMAWK}.
The approach used is known as the SMAWK algorithm, an anagram of the initials of the authors of~\cite{SMAWK}.
It is likely that most readers are unfamiliar with SMAWK, and some might prefer to just view Algorithm~\ref{alg:SMAWK} as a black box having the properties that for every $c$:
\begin{itemize}
\item for any level $k$ and any $i$ for which \jopt{i} is determined at level $k$, $\mathsf{jvalues}(k,\cdot)$ contains \jopt{i},
\item the total number of $j$ values returned over all levels is $O(n)$,
\item $\mathsf{determine\_jvalues}$ takes $\Theta(n)$ time.
\end{itemize}
The pseudo-code given in Algorithm~\ref{alg:SMAWK} is quite explicit, suitable for efficient implementation in any language.
It converts the recursive list-based description in~\cite{SMAWK} to an iterative array-based one.
Mention of eliminating columns, creating submatrices, etc., is merely symbolic since there aren't any real matrices:
they are just conceptual representations of calculating \Errj{i,j,c} values.
The only arrays being used are to store $j$ values.

To see how the SMAWK algorithm works, let $M$ denote an arbitrary totally monotonic matrix.
The algorithm starts with a list of columns $J$ ($\mathsf{jvalues}$), and a subset of them are moved to $K$ and kept, with the remaining ones deleted.
The final set of values in $K$ will be the ones returned by $\mathsf{determine\_jvalues}$.
When a column $m$ is deleted from $J$ and not put into $K$
it is guaranteed that for all rows $i$, $m \neq \jopt{i}$.
The guarantees come about by exploiting two facts implied by the general Monge property~(\ref{eqn:Monge2}):
for the $2 \times 2$ submatrix with columns $\alpha < \beta$ and rows $\delta < \epsilon$,
\begin{enumerate}
\item[a)] if $\beta$ is the minimal location in row $\delta$, i.e., $M(\delta,\alpha) > M(\delta,\beta)$, then it is the minimal location in row $\epsilon$, and hence in $M$ $\alpha$ is not the minimal location in any row $\geq \delta$
\item[b)] if $\alpha$ is the minimal location in row $\epsilon$, i.e., $M(\epsilon,\alpha) \leq M(\epsilon,\beta)$, then it is the minimal location in row $\delta$, and hence in $M$ $\beta$ is not the minimal location in any row $\leq \epsilon$
\end{enumerate}

\begin{figure}
\begin{center}
\resizebox{5.0in}{!}{\includegraphics{SMAWKflowchart.eps}}
\vspace{0.25in}

Currently at row $\delta$, comparing A and B.\\
$\alpha$, $\beta$, $\gamma$ are the first 3 columns remaining in $J$;~~ $\delta$, $\epsilon$ are consecutive rows in the submatrix
\end{center}
\caption{An intermediate step of the SMAWK algorithm}  \label{fig:SMAWKflowchart}
\hrulefill
\end{figure}

At any step in the algorithm two adjacent entries of $M$ are being compared, where they are in the same row and the first two columns ($j$ values) remaining in $J$.
For every row above the current row, one column has been moved into $K$.
Suppose the algorithm is comparing $A$ and $B$ in Figure~\ref{fig:SMAWKflowchart}.
If $A\leq B$ then it might be that $\alpha = \jopt{\delta}$,
and hence $\alpha$ is moved from $J$ to $K$.
Note that $\alpha$ might also be \jopt{} for some rows above and below $\delta$.
Relative to row $\delta$, column $\beta$ does not need to be kept.
Further, for any row above $\delta$, Monge property~b) shows that $\beta$ is not needed there either.
However, it might be needed for lower rows, so the algorithm proceeds to the next row, $\epsilon$, and compares E and F.
If $E\leq F$ then $\beta$ is moved to $K$ and the algorithm proceeds to the next row.
However, if $E > F$ then $\beta$ is not needed for row $\epsilon$, and Monge property~a) shows that it is not needed for any row below.
Therefore $\beta$ can be deleted from $J$, which in the implementation is done by merely incrementing $\mathsf{next\_j\_index}$.
Deleting $\beta$ condenses the submatrix in Figure~\ref{fig:SMAWKflowchart} to the entries A, C, D, and F.
It might be that $A > C$, so the algorithm moves $\alpha$ from $K$ back to $J$ and goes back to row $\delta$, comparing $A$ and $C$.
If A $\leq$ C then $\alpha$ is put back in $K$ and the algorithm goes to the next row ($\epsilon$),
otherwise it is removed from $J$ and the algorithm backs up another row, etc.
If E\,$=\infty$, i.e., $\beta < \epsilon$, then we treat it as E\,$>$\,F even if F\,$=\infty$.

If $\epsilon$ is the last row, if $E \leq F$ then $\gamma$ can be deleted from $J$ since there are no lower rows for which $\gamma$ might need to be kept.
Combining this with the rule that if $E > F$ then $\beta$ is deleted and the algorithm goes back a row shows that if the last row is reached then all of the remaining columns are examined.
Whether it occurs in the last row or earlier, eventually there is only 1 column left,
which should be kept.
Any row results in one column being moved to $K$, or is a row after the row in which the last column is reached, and hence $|K|$ is no more than the number of rows.
Further, the time required is $\Theta(|J|)$.

To initialize, for level 0, which corresponds to all rows, all columns are kept, i.e., $\mathsf{jvalues(0,k)=k}$ for $1 \leq \mathsf{k} \leq n$.
One could apply the above reduction for level 0, but it isn't required for the time analysis nor correctness, and it slightly simplifies the implementation.
At any level $m$ above 0, the process is applied to the submatrix consisting of every second row of the submatrix used for level $m-1$, i.e., to rows that are multiples of $2^m$.
The initial $J$ for level $m$ is $\mathsf{jvalues(m\!-\!1,\,1:num\_jvalues(m\!-\!1))}$.

\begin{algorithm}
\setlength{\Ainindent}{0.5in}
\Ain{0} $\mathsf{integer~ array~ jvalues(0\!:\!\lfloor \log_2 \rfloor, 1\!:\!n),~num\_jvalues(0\!:\!\lfloor \log_2 n \rfloor)}$\\

\vspace*{-0.08in}
\Ain{0} $\mathsf{for~ i = 1~to~n~ do}$\\
\Ain{1} $\mathsf{   \Err{i,1} = \err{p}{i,n};~~~ \jopt{i,1} = n}$\\
\Ain{0} $\mathsf{for~ c= 2~ to~ b~ do}$\\
\Ain{1} $\mathsf{   determine\_jvalues(jvalues,num\_jvalues,c)~~~ \{see~ Algorithm~\ref{alg:SMAWK}\}}$\\
\Ain{1} $\mathsf{   for~ level = \lfloor \log_2 (n\!-\!c\!+\!1) \rfloor ~downto~ 0 ~do}$\\
\Ain{2} $\mathsf{      for~ i = 2^{level} ~to~ n-c+1 ~by~ 2^{level\!+\!1}~ do}$\\
\Ain{3} $\mathsf{         if~ i = 2^{level} ~then~ j\_start = i;~~ j\_index = 1}$\\
\Ain{4} $\mathsf{            else~ j\_start = max\{i, \jopt{i-2^{level},c}\}}$\\
\Ain{3} $\mathsf{         if~ i+2^{level} > n-c+1 ~then~ j\_end = n-c+1}$\\
\Ain{4} $\mathsf{            else~ j\_end = \jopt{i+2^{level},c}}$\\
\Ain{3} $\mathsf{         \Err{i,c} = \infty}$\\
\Ain{3} $\mathsf{         while~ (j\_index \leq num\_jvalues(level)) \wedge (jvalues(level,j\_index) \leq j\_end) ~do}$\\
\Ain{4} $\mathsf{            j=jvalues(level,j\_index)}$\\
\Ain{4} $\mathsf{            if~ (j \geq j\_start) \wedge (\Errj{i,j,c} < \Err{i,c}) ~then}$\\
\Ain{5} $\mathsf{               \Err{i,c} = \Errj{i,j,c};~~ \jopt{i,c}=j}$\\
\Ain{4} $\mathsf{            j\_index = j\_index + 1}$\\
\Ain{3} $\mathsf{         end~ while}$\\
\Ain{3} $\mathsf{         j\_index=j\_index-1}$\\
\Ain{2} $\mathsf{      end~ for~ i}$\\
\Ain{1} $\mathsf{   end~ for~ level}$\\
\Ain{0} $\mathsf{end~ for~ c}$\\
\Ain{0} $\mathsf{generate~ the~ approximation~ using~ \jopt{}~ and ~\mean{p}{}}$

\vspace*{0.3in}
\hrulefill
\vspace*{-0.4in}

\caption{$b$-step $L_p$ approximation of isotonic data, using total monotonicity for $\mathsf{determine\_jvalues}$}
\label{alg:totallymonotonic}

\end{algorithm}

\begin{algorithm}
\setlength{\Ainindent}{0.1in}
\Ain{0} $\mathsf{procedure~ determine\_jvalues(jvalues,num\_jvalues,c)}$\\

\Ain{0} $\mathsf{num\_jvalues(0)=n}$\\
\Ain{0} $\mathsf{for~ k=1 ~to~ n ~do~ jvalues(0,k)=k}$\\
\Ain{0} $\mathsf{for~ level = 1 ~to~ \lfloor \log_2 (n\!-\!c\!+\!1) \rfloor}$\\
\Ain{1} $\mathsf{   j=jvalues(level\!-\!1,1);~~ next\_j\_index=2}$;~~ k\_index=0\\
\Ain{1} $\mathsf{   i=2^{level}}$\\
\Ain{1} $\mathsf{   while~  next\_j\_index \leq num\_jvalues(level\!-\!1) ~do~}$\\
\Ain{2} $\mathsf{      next\_j = jvalues(level\!-\!1,next\_j\_index)}$\\
\Ain{2} $\mathsf{      if~ (j \geq i) \wedge (\Errj{i,j,c} \leq \Errj{i,next\_j,c}) ~then~}$\\
\Ain{3} $\mathsf{         if~ i+2^{level}>n-c+1 ~then~ \{at~ last~ row,~ eliminate~ next\_j\}}$\\
\Ain{4} $\mathsf{            next\_j\_index = next\_j\_index+1}$\\
\Ain{3} $\mathsf{         else~ \{keep~ this~ j, ~increment~ i,~ j\}}$\\
\Ain{4} $\mathsf{            k\_index=k\_index +1;~~ jvalues(level,k\_index)=j}$\\
\Ain{4} $\mathsf{            j= next\_j;~~ next\_j\_index=next\_j\_index + 1}$\\
\Ain{4} $\mathsf{            i=i+2^{level}}$\\
\Ain{3} $\mathsf{         end~ if}$\\
\Ain{2} $\mathsf{      else~ \{\Errj{i,j,c} > \Errj{i,next\_j,c}, eliminate~ current~ j,~ go~ back~ to~ previous~ i,~ j\}}$\\
\Ain{3} $\mathsf{         if~ i > 2^{level} ~then}$\\
\Ain{4} $\mathsf{             i = i-2^{level};~~ j=jvalues(level,k\_index);~~ k\_index = k\_index-1}$\\
\Ain{3} $\mathsf{         else~ \{at~ first~ row\}}$\\
\Ain{4} $\mathsf{             j=next\_j;~~ next\_j\_index=next\_j\_index+1}$\\
\Ain{3} $\mathsf{         end if}$\\
\Ain{2} $\mathsf{      end~ if}$\\
\Ain{1} $\mathsf{   end~ while}$\\
\Ain{1} $\mathsf{   k\_index = k\_index+1;~~ jvalues(level,k\_index) = j}$\\
\Ain{1} $\mathsf{   num\_jvalues(level)=k\_index}$\\
\Ain{0} $\mathsf{end~ for~ level}$\\
\Ain{0} $\mathsf{end~ determine\_jvalues}$\\

\caption{Reducing the number of relevant $j$ values using SMAWK} \label{alg:SMAWK}

\end{algorithm}

\begin{proposition} \label{prop:SMAWK}
Given $n$ isotonic weighted values \ensuremath{(\vec{y},\vec{w})}\ and number of steps $b \leq n$,
Algorithm~\ref{alg:totallymonotonic} finds an optimal $b$-step
$L_2$ approximation in $\Theta(b n)$ time.
\end{proposition}
\textit{Proof:}\,
Since each level halves the number of rows and the number of kept $j$ values is no more than the number of rows, the total number of $j$ values kept over all levels is $O(n)$ and the total time of $\mathsf{determine\_jvalues}$ is $\Theta(n)$.
The time for Algorithm~\ref{alg:totallymonotonic} is linear in the total number of $j$ values considered, so it too is $\Theta(n)$.
$\Box$.

\section{Reduced Isotonic Regression} \label{sec:Reduced}

For arbitrary data, isotonic regressions are somewhat easier to compute than are general
approximations by step functions.
One can use a simple left-right scan where each location is
initially a step and then adjacent steps are merged
whenever they violate the isotonic condition.
This is known as ``pool adjacent violators'', PAV,
and first appeared in 1955 in Ayer et al.~\cite{Ayeretal1955}.
For $L_2$ it can easily be computed in only $\Theta(n)$ time.

Isotonic regression is a very flexible nonparametric approach to
many problems.
However it does have its detractors
due to results with impractically many steps or overfitting.
Some researchers have instead used approximations with a
specified number of steps~\cite{HaiminenetalReducedUnimodal,TerziSegmentation}.
To reduce overfitting, Schell and Singh~\cite{SchellSingh} used the approach of
repeatedly merging pairs of adjacent steps whose difference
had the least statistical significance.
Haiminen et al.~\cite{HaiminenetalReducedUnimodal} used an approach that
repeatedly combines the adjacent steps that cause a minimum increase in the error.
These greedy (aka myopic) approaches repeatedly make
the choice that seems to be the best at the moment, but
may not produce an optimal reduced isotonic regression.
For example, for all $L_p$, $1 < p \leq \infty$,
given the unweighted values 0, 2, 4, 6, 8, 10, the unique optimal 3-step isotonic regression is 1, 1, 5, 5, 9, 9,
and the unique optimal 2-step isotonic regression is 2, 2, 2, 8, 8, 8.
Thus the 2-step isotonic regression cannot be obtained by merging steps of the
3-step isotonic regression.

The fastest previous algorithm for optimal $L_2$ reduced isotonic regression is due to Haiminen et al.\ \cite{HaiminenetalReducedUnimodal}, taking $\Theta(n + b m^2)$ time,
where $m$ is the number of pieces in the unconstrained isotonic regression.
As a reminder, we use ``pieces'' to refer to the steps of an unrestricted
isotonic regression and ``steps'' to refer to the steps of a reduced isotonic regression.
Even though often $m \ll n$,  Haiminen et al.\ felt that this may be too slow so they developed the greedy heuristic mentioned above.
Our exact algorithms should be sufficiently fast even for very large problems.

One cannot directly find $b$-step reduced isotonic regression of arbitrary data by using the approaches in Algorithms~\ref{alg:monotonic} and~\ref{alg:totallymonotonic} since it does not have the required monotonic properties.
For example, for unweighted values 7, 8, 0, 6, 9, 10, the optimal $L_2$ 2-step reduced isotonic regression has its first step on the interval \inter{1}{4},
while the optimal first step for the data starting at position 3 is the interval \inter{3}{3}, i.e.,
$4=\jopt{1,2} \not\leq \jopt{3,2}=3$.
Howevever, a critical observation in Haiminen et al.\ \cite{HaiminenetalReducedUnimodal}
is that, given the pieces of an $L_2$ unrestricted isotonic regression,
the steps of an optimal
$L_2$ reduced isotonic regression can be formed by merging the pieces.
Each piece becomes a weighted point, where the value of the point is the mean of the piece and the weight of the point is the total weight of the piece.
In the above example, the data would be represented by the 4 weighted points (5,3), (6,1), (9,1), (10,1),
and the first step of a 2-step reduced isotonic regression uses the first two pieces.

Their observation gives a simple algorithm:
find the unrestricted isotonic regression, convert the pieces to weighted points, and then find a $b$-step approximation of these isotonic points.
Haiminen et al.\ used Fisher's algorithm to determine the optimal $b$-step reduced isotonic regression in $\Theta(n+bm^2)$ time,
but Algorithms~\ref{alg:monotonic} and~\ref{alg:totallymonotonic} provide faster solutions.

\begin{theorem}   \label{thm:L2}
Given $n$ weighted values \ensuremath{(\vec{y},\vec{w})}\ and number of steps $b$,
an optimal $L_2$ $b$-step reduced isotonic regression can be found in
$\Theta(n + b m \log m)$ time via Algorithm~\ref{alg:monotonic}, and in $\Theta(n + bm)$ time via
Algorithm~\ref{alg:totallymonotonic},
where $m$ is the number of pieces in the unconstrained $L_2$ isotonic regression.
$\Box$
\end{theorem}

Unfortunately, for $p \neq 2$ the optimal reduced isotonic regression might not be formed from pieces of the unrestricted isotonic regression.
For example, for unweighted values -10, -10, -10, 0, 0, 0, -10, -1, 7, 7, 7, 7, the unique $L_1$
unrestricted isotonic regression has pieces
\inter{1}{3}, \inter{4}{8}, and \inter{9}{12}, with values -10, 0, 7, respectively.
The unique optimal 2-step reduced isotonic regression has
steps \inter{1}{7} and \inter{8}{12},
with values -10 and 7, which requires cleaving the middle piece.
However, one can determine an approximation by constructing an optimal $b$-step isotonic regression among those restricted to use unbroken pieces of the unrestricted isotonic regression.
By doing so, the problem is now similar to isotonic regression on isotonic data.
An algorithm using this approach to approximate $L_1$ reduced isotonic regression
appears in~\cite{JQReducedIso_IF2012}.
It is more complicated than the $L_2$ case since to determine medians one needs to retain the values in the original pieces, rather than combining them into a single weighted value as can be done for $L_2$.

For $L_\infty$ an optimal $b$-step reduced isotonic regression, and an optimal $b$-step approximation with no isotonic restrictions, can be found in $\Theta(n + \log n \cdot b(1 + \log n/b))$ time~\cite{QLinftyReduced}.
The approaches used there are quite different, unrelated to dynamic programming.

\section{Final Comments}\label{sec:Final}

The thousands of citations to the books by Barlow et al.~\cite{BarlowetalBook}
and Robertson et al.~\cite{RobertsonWrightDykstra} shows a significant interest in isotonic regression.
Further, this interest is growing as researchers seek to remove parametric assumptions from their modeling.
Similarly, step functions with a constraint on the number of steps arise in a wide range of applications and guises~\cite{FisherHomogeneity,HalimetalFastHistogram,HimbergetalTimeSeries,IoannidisIncreasingHist,JacobetalFuzzySegmentation,MaysterLopezStep,PoosalaetalHistogramSurvey,TerziSegmentation}.
For reduced isotonic regression both aspects are important~\cite{HaiminenetalReducedUnimodal,SalantiUlmReduced,SchellSingh}, using a reduced number of steps to simplify the regression and/or prevent overfitting.

However, researchers used approximations, rather than the optimal answer, due to the slowness of the available algorithms.
The fastest previous algorithm for optimal $L_2$ $b$-step reduced isotonic regression takes $\Theta(n + bm^2)$ time~\cite{HaiminenetalReducedUnimodal}, where $m$ is the number of pieces in the unconstrained isotonic regression.
Algorithm~\ref{alg:monotonic} reduces this to $\Theta(n + b m \log m)$ time,
and the somewhat more complicated Algorithm~\ref{alg:totallymonotonic}
further reduces this to $\Theta(n + b m)$.
Note that the minimal time for optimal $b$-step approximation, with no isotonic restrictions, is a long-standing open question.

Fisher~\cite{FisherHomogeneity} called the $b$-step approximations ``restricted homogenization'', and defined another form of approximation that he
called ``unrestricted homogenization'': given $n$ weighted values \ensuremath{(\vec{y},\vec{w})}\ and $b \in \inter{1}{n}$,
partition the values into $b$ subsets $P_i, i \in \inter{1}{b}$ and assign a value $C_i$
to each $P_i$ so as to minimize
$$
\sum_{i=1}^b \sum_{j \in P_i} w_j |y_j - C_i|^2
$$
among all such partitions.
This is now known as \textit{$k$-means clustering} of 1-dimensional data, for $k = b$.
He noted it could be solved by sorting the values and then finding
the optimal $b$-step approximation, i.e., the optimal $b$-step isotonic
regression of the sorted data.
Thus for 1-dimensional data Algorithm~\ref{alg:monotonic} solves the $k$-means clustering problem in $\Theta(kn \log n)$ time, and
for sorted data Algorithm~\ref{alg:totallymonotonic} reduces this to $\Theta(kn)$.

Finally, an interesting problem is that of selecting the most desirable number of steps.
For reduced isotonic regression,
Schell and Singh~\cite{SchellSingh}, Strobl et al.~\cite{StrobletalCARTreduced} and Haiminen et al.~\cite{HaiminenetalReducedUnimodal} start with an unconstrained isotonic
regression and then repeatedly merge pieces until their criteria are met.
However, Haiminen et al.\ showed that the regression error of their greedy approximation can be nearly twice that of the optimal reduced isotonic regression with the same number of steps.
They believe that 2 is an upper bound on the relative error of their approximation, but that has not been proven, nor have bounds been proven for other approximation schemes.
For $b$-step approximation,
many researchers choose $b$ \textit{a priori} based on considerations
such as storage or access time requirements.
This seems to be especially true in the database community, where $L_2$ $b$-step approximations
are known as ``v-optimal histograms''.

In contrast, the dynamic programming approach generates optimal
$b$-step reduced isotonic regressions for each value of $b$ as $b$ increases.
One can stop when a criterion is met and always have an optimal result.
However, appropriate stopping criteria for a given application may be somewhat subtle
since they would be applied repeatedly.

\subsubsection*{Acknowledgements}
Research partially supported by NSF grant CDI-1027192 and DOE grant
DE-FC52-08NA28616.
Some of these results were announced in~\cite{JQReducedIso_IF2012}.

Computation (stat.CO); Data Structures and Algorithms (cs.DS)

Optimal Reduced Isotonic Regression


Iteration Complexity of Fixed-Step Methods by Nesterov and Polyak for Convex Quadratic Functions

Abstract: This note considers the momentum method by Polyak and the accelerated gradient method by Nesterov, both without line search but with fixed step length applied to strictly convex quadratic functions assuming that exact gradients are used and appropriate upper and lower bounds for the extreme eigenvalues of the Hessian matrix are known. Simple 2-d-examples show that the Euclidean distance of the iterates to the optimal solution is non-monotone. In this context an explicit bound is derived on the number of iterations needed to guarantee a reduction of the Euclidean distance to the optimal solution by a factor $\varepsilon$. For both methods the bound is optimal up to a constant factor, it complements earlier asymptotically optimal results for the momentum method, and it establishes another link of the momentum method and Nesterov's accelerated gradient method.

\section{Introduction}
\label{sec1}

Let $f:{\mathbb{R}}^n\to{\mathbb{R}}$ be a strictly convex quadratic function.
For the minimization of $f$ starting at an initial point $x^0\in{\mathbb{R}}^n$
with negative gradient $m^0:=-\nabla f(x^0)$
the following ``Momentum Method'' is considered for $k \ge 0$:
$$
x^{k+1} = x^k+\alpha m^k, \qquad m^{k+1} = \beta m^k - \nabla f(x^{k+1}).
\leqno(MM)
$$
Here $\alpha > 0$ is a sufficiently small step length that will be analyzed
below and $\beta\in[0,1)$ is a parameter that determines how much of the
previous search direction will be added to the new search direction. This
parameter is also discussed below.

As pointed out for example in \cite{gower1}, with the initialization
$x^{-1}:=x^0$ the Momentum Method $(MM)$
is equivalent to the ``Heavy Ball Method'' (\cite{polyak}):
$$
x^{k+1} = x^k - \alpha\nabla f(x^{k}) +\beta(x^k-x^{k-1}).
\leqno(HBM)
$$

In spite of the proof of asymptotic optimality of the momentum method
$(MM)$ established almost 60 years ago in \cite{polyak} this method has
not received due attention until recently in the context of
machine learning and neural networks.
It has proved to be very efficient in
numerical implementations, also when the gradients are replaced
with approximate stochastic gradients, see e.g. \cite{defazio_mm,gower1}.
Today modifications of the $(MM)$ are widely
used, in particular also in machine learning libraries.

While being efficient in practice, the convergence behavior of momentum
methods is ``somewhat irregular''.
{\bf Figure \ref{pic.tp0}}
shows a typical non-monotone convergence behavior of the $(MM)$
for the simple 2-d-example  $f(x)\equiv\tfrac 12 (x_1^2+100x_2^2)$ with
$\beta = 0.85$ and \blue{$\alpha = \tfrac{1.9}{100}$}
during the first 100 iterations for the initial values $x^0=(\tfrac 1{100},1)^\text{T}$,
$x^0=(1,1)^\text{T}$, and $x^0=(100,1)^\text{T}$.

\begin{figure}[h]
\begin{center}
  \caption{Convergence of the $(MM)$ for a 2-d-example with $\beta = 0.85$, and $\alpha = 1.9$.}
\label{pic.tp0}
\includegraphics[width=170truemm]{2dexample.eps}
\end{center}
\end{figure}

For convex quadratic functions the iterative process $(MM)$ or $(HBM)$
can be written as a recursion with a fixed matrix $M$, and the
spectral radius of $M$ is the main factor determining the
convergence behavior. A secondary factor is the approximation of
the spectral radius by a matrix norm.
The connection of both factors and the implication on the
choice of parameters is addressed in this paper.

\bigskip

\blue{
Closely related to the $(MM)$ is Nesterov's accelerated gradient method
\cite{nestg}. Following the presentation in \cite{nestg2} (Theorem 2.2.3),
it can be written as follows: Given $x^{0}\in{\mathbb{R}}^n$ set
$y^{0}:=x^{0}$ and for $k\ge 0$:
$$
x^{k+1}:=y^{k+1}+\beta (y^{k+1}-y^{k}) \qquad \hbox{where} \qquad
y^{k+1}:=x^{k}-\alpha \nabla f(x^{k})
\leqno(NAG)
$$
for suitable parameters $\alpha,\beta >0$.
As observed in \cite{yang}, for example, this method falls in the same
general framework as $(MM)$: Indeed, by eliminating the variable $y^k$
and setting $x^{-1}:=x^0$, the iteration
$(NAG)$ can also be written in the following compact form,
\begin{equation}\label{nag}
x^{k+1}:=x^{k}-\alpha \nabla f(x^{k}) +
\beta\left(x^{k}-x^{k-1} - \alpha(\nabla f(x^{k})-\nabla f(x^{k-1}))
\right),
\end{equation}
where in contrast to $(HBM)$ the momentum term not only includes the
previous iterates $x^k$ but also
the previous descent steps ``$-\alpha\nabla f(x^{k})$''.
(To obtain the identical initialization as in $(NAG)$, in the very first step
$\nabla f(x^{-1})$ needs to be replaced with $0$.)
}

\subsection{Main Results}
The main theoretical
result of this paper is summarized in the following theorem.

\begin{theorem}\label{thm1}
  Let $f:{\mathbb{R}}^n\to{\mathbb{R}}$ be a strictly convex quadratic function
  with minimizer $x^*$ and
  denote $H:=\nabla^2f(x^*)$. Let
  $\overline{m}$ be an upper bound for the eigenvalues of $H$
  and let $0<\underline{m}$ be a lower bound for the eigenvalues
  of $H$. Then $\overline{\hbox{\rm con}}\hbox{\rm d}(H):=\overline{m}/\underline{m}$
  is an upper bound for the condition number of $H$. Assume
  that $\overline{\hbox{\rm con}}\hbox{\rm d}(H)\ge 28$.

  In $(HBM)$ define $\alpha=2/\overline{m}$, \
  \blue{$\beta = \left(1-\sqrt{2\ /\  \overline{\hbox{\rm con}}\hbox{\rm d}(H)}\right)^2$
  and let

  $\epsilon\le 1\, / \, \overline{\hbox{\rm con}}\hbox{\rm d}(H)$ be given.
  Then, given $x^0\in{\mathbb{R}}^n$, after
  $$
  1+\lceil \sqrt{2\ \overline{\hbox{\rm con}}\hbox{\rm d}(H)}\
  \ln(\tfrac {2}\epsilon)\rceil
  $$
  steps of the $(HBM)$, an approximate solution
  $\bar x^k:=\tfrac 12 (x^{k-1}+x^k)$ is
  generated with
  $$
  \|\bar x^k-x^*\|_2\le

  \epsilon \|x^0-x^*\|_2.
  $$}
\end{theorem}

A corollary of the analysis of the $(MM)$ is the following
well known observation (see \cite{goh}, for example):
It may seem intuitive that with the use of a momentum $\beta >0$,
the step length $\alpha$ in $(MM)$ needs to be chosen more carefully.
However, at least for the minimization of
convex quadratic functions and in the absence of noise, the opposite is true:
 “If the step itself is made longer by adding more of the previous search
step to the new step (i.e.\ by increasing $\beta\in[0,1)$)
  then also the step length can be made longer
while maintaining global convergence.”

\bigskip

\blue{When $\overline{m}$ and $\underline{m}$ are known, then
  -- as detailed in Section~\ref{sec.nest} -- the analysis for $(MM)$
  also applies to $(NAG)$, and for an appropriate choice of
  $\alpha,\beta$,
  an iteration complexity can be established for $(NAG)$
  that is a factor $\sqrt 2$ higher than the bound for $(MM)$
  in Theorem~\ref{thm1}. The factor $\sqrt 2$ arises since the step
  length is reduced by a factor 2 -- which essentially amounts to an
  increase of the estimate $\overline{\hbox{\rm con}}\hbox{\rm d}(H)$ by a factor 2.}

\subsection{Related work}
The paper \cite{polyak} establishes for $(HBM)$ applied to
(non-quadratic smooth) strictly
convex functions local convergence of the form
$$
\|x-x^*\|_2\le c(\epsilon) (1-\blue{\tfrac 2{1+\sqrt{\overline{\hbox{\rm con}}\hbox{\rm d}(H)}}}+\epsilon)^k
\sqrt{\|x^1-x^*\|_2^2+ \|x^0-x^*\|_2^2}
$$
where $c(\epsilon)$ is a constant that only depends on $\epsilon$
(Theorem 9, Statement (3)). This result is obtained by analyzing the
spectral condition of a recursion associated with $(HBM)$
and using the fact that for a given matrix there
always is a vector norm and an associated matrix norm that
approximates the spectral radius to a given precision $\epsilon >0$.

For the case of convex quadratic functions the constant $c(\epsilon)$
is worked out explicitly in a slightly different setting in \cite{goh}.

The optimal step length $\alpha$ in \cite{polyak,goh}
turns out to be increasing with
increasing values of $\beta$. On the other hand,
in practical applications,
the use of long steps leads to an amplification of the
stochastic error associated with an inexact evaluation of the gradient.
Subsequently, only step lengths at most $2/\overline m$ are considered
in this paper.
This imposes a restriction on $\alpha$ cutting off the
optimal step lengths identified in \cite{polyak,goh} at the
expense of a small constant factor in the theoretical efficiency
of the method.

Further studies of $(MM)$ for smooth strongly convex optimization
where the strong convexity parameter is to be estimated are
presented in \cite{barre,donoghue}.

Momentum methods and modifications thereof are in the focus of numerous
further recent research projects, in particular also for the non-convex case
-- which is much more difficult to analyze than the convex quadratic
case considered in this paper.
The articles quoted below form a rather brief and thus incomplete glimpse
on recent lines of research on momentum methods.

An analysis comparing the $(MM)$ for the choice $\beta =0$ (steepest descent)
and $\beta>0$ in the non-convex case
and in the presence of noise is presented, for example, in \cite{defazio_mm}.
This paper also gives a theoretical explanation for the
observed efficiency of $(MM)$.
The $(MM)$ is closely related to ADAM or ADAGRAD for which a recent analysis
covering the non-convex case is given in \cite{defossez}.
In particular, a new tight dependency on a certain ``heavy ball
momentum decay rate'' (which is zero in the context considered in this paper)
is established in \cite{defossez}.
Another recent approach that also considers the non-convex situation
without line search is analyzed in \cite{gratton,gratton2,gratton3}.
In this situation convergence to a second-order minimizer
-- along with estimates on the rate of convergence --
is established under mild conditions.
An analysis generalizing $(HBM)$ and Nesterov's method
to a broad class of momentum methods and giving a unified
convergence analysis is given in \cite{diakonikolas}.\\
A unified analysis of stochastic momentum methods
in the weakly convex case -- and in the non-convex case --
is given in \cite{yang}, and a further unified analysis
that considers ``Quasi-Hyperbolic Momentum Methods''can be found
in \cite{gitman}.
Limitations that arise when generalizing $(MM)$ to
stochastic optimization are addressed in the recent paper \cite{ganesh}.

\section{Analysis of $(HBM)$}
The analysis of $(HBM)$ and thus also of $(MM)$ is
carried out with the following steps:
\begin{enumerate}
\item
  Section \ref{sec.spec}
  follows the approach in \cite{polyak,donoghue} and derives
  a linear recursion for the iterates $x^k$ and analyzes
  the spectral radius of the underlying system matrix.
\item
  Based on this analysis suitable parameters $\alpha,\beta$
  for $(HBM)$ are identified in Section~\ref{sec.par}.
\item
  Then the condition number of the similarity transformations
  to diagonalize the system matrix is analyzed.
\item
  It is shown that for the above parameters $\alpha,\beta$
  this condition number is unbounded in general.
\item
  A transformation to Schur canonical form is analyzed ``instead''.
\item
  Based on this transformation the main theoretical result is proved.
\end{enumerate}
The derivations in
Steps 3) to Step 5) appear to be new; the results of
these steps, however, are essentially the same as in \cite{goh}
while the bound on the number of iterations derived in Step 6) may
not have been derived explicitly before.

As the $(HBM)$ of Section \ref{sec1} is invariant with respect to a
shift of the variable
and with respect to orthogonal transformations, for the analysis
it is assumed without loss of generality
that
\begin{equation}\label{assu}
f(x)\equiv \tfrac 12\, x^TDx
\end{equation}
with a positive definite diagonal matrix $D$.
  (The eigenvalue decomposition of the Hessian of $f$ that leads to
  the simple reformulation \mr{assu} is used only for the analysis
  of the algorithm, but not for the algorithm itself.)
In this case, the plain steepest descent method with $\beta = 0$ (no momentum)
converges if, and only if, $\alpha \in (0,\ 2/\max_{1\le i\le n} \{D_{i,i}\})$.

The following slightly weaker assumption will be made:
\begin{assumption}
It is assumed throughout that $f$ is given by \mr{assu} and
$$
\beta\in[0,1)\qquad\hbox{and}\qquad \alpha \in (0,\bar \alpha]
  $$
where $\bar \alpha := 2/\max_{1\le i\le n} \{D_{i,i}\}$
\end{assumption}

\bigskip

The $(HBM)$
can then be written with the following recursion
\begin{equation}\label{recursion}
\left(\begin{matrix} x^k \\ x^{k+1} \end{matrix} \right) =
\left(\begin{matrix} 0 & I \\ -\beta I & (1+\beta) I-\alpha D \end{matrix} \right)
\left(\begin{matrix} x^{k-1} \\ x^{k} \end{matrix} \right).
\end{equation}
This is a discrete linear dynamical system
$\hat z^{k+1}=\hat M\hat z^k$ with the variable
$$
\hat z^k := \left(\begin{matrix} x^{k-1} \\ x^{k} \end{matrix} \right)
\qquad \hbox{and} \qquad
\hat M:= \left(\begin{matrix} 0 & I \\ -\beta I & (1+\beta) I-\alpha D \end{matrix} \right).
$$
Recursion \mr{recursion}  is block-separable, i.e.\
for $i\not=j$ the variables $x^k_i, \ x^{k+1}_i$
do not depend on $x^\ell_j$ for any $\ell \le k+1$.
Thus, when setting
$$
z^k_{(i)}:=\left(\begin{matrix} x^{k-1}_i \\ x^{k}_i \end{matrix} \right)
$$
for $k\ge 0$ and $1\le i\le n$, then Recursion \mr{recursion} can be written as
$$
z^{k+1}_{(i)} = M^{(i)}z^k_{(i)}\qquad \hbox{for} \ 1\le i\le n
$$
with
\begin{equation}\label{defmi}
M^{(i)}:=\left(\begin{matrix} 0 && 1 \\ -\beta  && 1+\beta -\alpha D_{i,i} \end{matrix} \right).
\end{equation}

(This means that
rows and columns of $\hat M$ can be permuted so that a block-diagonal matrix $M$
is obtained with $2\times 2$ diagonal blocks $M^{(i)}$ on the diagonal.)

\subsection{The spectral radius of $M^{(i)}$ in
  dependence of $\alpha$ and $\beta$}\label{sec.spec}

Possible convergence of the iterates $\hat z^k$ depends on
the norm $\|(M^{(i)})^k\|_2$ for large $k$, i.e.\ on
the spectral radius of $\hat M$
which coincides with the maximum spectral radius $\rho(M^{(i)})$
of $M^{(i)}$ for all $i$.

In the following the index $i$ is kept fixed. To simplify the notation, let
$$
\alpha_i:=\alpha D_{i,i}\in\,(0,2],
\quad \beta_i:=1+\beta-\alpha_i\in [-1,2),
\quad  \hbox{and} \ \gamma_i:=\sqrt{\beta_i^2-4\beta}.
$$
Here, $\gamma_i$ is either a non-negative real number
or a (purely imaginary) number with positive imaginary part.
In both cases, $\gamma_i^2=\beta_i^2-4\beta$ will be used below.
With these abbreviations $M^{(i)}$ is given by
$M^{(i)}=\begin{pmatrix}0&1\\-\beta&\beta_i\end{pmatrix}$.
Let further
\begin{equation}\label{eigenv}
 \lambda_+
 :=\tfrac12(\beta_i+\gamma_i), \quad
 \lambda_-
 :=\tfrac12(\beta_i-\gamma_i), \quad
 v_+:=\begin{pmatrix}1\\ \lambda_+\end{pmatrix}, \
 \text{ and }\  v_-:=\begin{pmatrix}1\\\lambda_-\end{pmatrix}.
\end{equation}
Observing that
$$
\lambda_\pm^2 \ = \
\tfrac 14 \beta_i^2 \pm \tfrac 12 \beta_i\gamma_i +\tfrac 14 \gamma_i^2 \ = \
\tfrac 14 \beta_i^2 \pm \tfrac 12 \beta_i\gamma_i +\tfrac 14 \beta_i^2-\beta
\ = \ -\beta + \tfrac 12 \beta_i^2 \pm \tfrac 12 \beta_i\gamma_i
\ = \ -\beta + \beta_i\lambda_\pm
$$
it follows that $M^{(i)}v_\pm=\lambda_\pm v_\pm$.
Thus, the -- possibly complex --
eigenvalues of $M^{(i)}$
are given by $\lambda_+$ and $\lambda_-$ and
the spectral radius $\rho(M^{(i)})$ of $M^{(i)}$ is given by the larger
one of the two values
$$
|\lambda_\pm|\ =\ \tfrac12|\beta_i\pm \gamma_i| \ = \
\frac 12 \left| \beta_i
\pm\sqrt{\beta_i^2-4\beta} \right|.
$$
If the square root
is purely imaginary, i.e., if $\beta_i^2<4\beta$ then
the square of the absolute value is given by the sum of the squares of real and imaginary part,
i.e.
$$
\rho(M^{(i)})^2=\frac 14 \left(\beta_i^2 + |\gamma_i|^2)\right)=
\frac 14 \left(\beta_i^2 + (4\beta-\beta_i^2)\right)=\beta.
$$
Thus, the expression for $\rho(M^{(i)})$ simplifies to
\begin{equation}\label{defrho}
\rho:=\rho(M^{(i)}) = \left\{\begin{array}{ll}
\sqrt{\beta} & \hbox{if} \ \ \beta_i^2<4\beta\\
\frac 12\left( | \beta_i |+\gamma_i \right)
& \hbox{else}.
\end{array}\right.
\end{equation}

Note that there is a double eigenvalue $\lambda_+=\lambda_-$
if $0=\gamma_i=\beta_i^2-4\beta=(1+\beta -\alpha D_{i,i})^2-4\beta$
with $\beta\in[0,1)$, i.e.\ if,
and only if, $\beta = (1-\sqrt{\alpha D_{i,i}})^2$ where
the definition of $\bar \alpha$ in Assumption~\ref{assu}
implies that $\alpha D_{i,i}\in(0,2]$ for all $i$.
(This is sketched on the right of Figure~\ref{specs} below.)

\begin{figure}[h]
\begin{center}
  \caption{Spectral radius of $M^{(i)}$ as a function of $\alpha D_{i,i}\in (0,2]$
    and $\beta\in[0,1)$.}
\label{pic.tp1}
\includegraphics[width=130truemm]{spectral.eps}
\end{center}
\end{figure}

\bigskip

In {\bf Figure \ref{pic.tp1}},
the value of $\alpha D_{i,i}\in(0,2]$ is plotted from the middle to the right
and the value of $\beta$ from the middle to the rear-left.
The associated values of $\rho$ are plotted on the vertical axis.

It is assumed that an upper bound $\overline{m}$
for the eigenvalues $D_{i,i}$ is known
and that $\alpha$ is chosen less than or equal to $2/\overline{m}$ so that
the possible values of $\alpha D_{i,i}$  vary over a (typically wide)
range in the interval (0,2] and depend on the distribution of the eigenvalues.
This range is problem dependent and not subject to the design of the method.

A fast convergence of the HBM (or of the momentum method MM) is obtained if the value of
$\beta$ is chosen such that $\rho$ is small for a wide range of values
$\alpha D_{i,i}$. {\bf Figure \ref{pic.tp1}} gives the impression that this is achieved
when choosing $\beta = 0$. And indeed, when the values of
$\alpha D_{i,i}$ all cluster about the value ``1'', then the problem of
minimizing $f$ not only is quite easy (since the condition number of $D$
is close to 1) but also choosing $\beta = 0$ is nearly optimal.

\bigskip

However, when the condition number of $D$ is poor, then very small
values of $\alpha D_{i,i}>0$ will occur. In this case, {Figure \ref{pic.tp1}}
is not suitable for understanding the best possible choice of
$\beta$. Instead, {\bf Figure \ref{pic.tp2}} displays
the plots of $\rho(M^{(i)})$ as a function of $\alpha D_{i,i}\in (0,2]$
for $\beta \equiv 0.9$ in red, $\beta \equiv  0.5$ in green,
and $\beta \equiv  0.1$ in blue.

\begin{figure}[h]
\begin{center}
  \caption{$\rho(M^{(i)})$ as a function of $\alpha D_{i,i}\in (0,2]$
    for fixed values of $\beta$.}
\label{pic.tp2}
\includegraphics[width=130truemm]{threebeta.eps}
\end{center}
\end{figure}

In {\bf Figure \ref{pic.tp2}} the choice $\beta=0.1$ in blue leads to
small values of $\rho(M^{(i)})$ when $\alpha D_{i,i}\in [0.5,1.5]$
(and a little beyond this interval).
For values $\alpha D_{i,i}\approx 0.2$, the choice $\beta=0.5$ in green
results in a much smaller value of  $\rho(M^{(i)})$ than $\beta=0.1$,
and as displayed in {\bf Figure~\ref{pic.tp3}}, when considering poorly
conditioned problems with some values of $\alpha D_{i,i}\approx 0.004$
the choice of $\beta=0.9$ in red results in a smaller value of
$\rho(M^{(i)})$ than $\beta=0.5$ or $\beta=0.1$.
({Figure \ref{pic.tp3}} is a zoom of {Figure \ref{pic.tp2}};
of course, small values of $\alpha D_{i,i}$ occurring in
ill-conditioned problems result in an increase of $\rho(M^{(i)})$
but for small $\alpha D_{i,i}>0$
the increase is much less for larger values of $\beta < 1$.)

  An interesting observation that can be deduced from
  {\bf Figure \ref{pic.tp2}} is the following:
  Recall that a step length $\alpha D_{i,i}>2$  is
  ``too long'' in the sense that it results in a divergent algorithm
  for the plain steepest descent method.
  For larger values of $\beta\ge 0.5$, however, the green line and the
  red line seem to (and do) continue ``for a little while'' with constant
  values to the right of $\alpha D_{i,i}=2$ which implies
  that for larger values of $\beta$ a step length that is
  a little bit ``too long'' still results in a convergent algorithm.
  In contrast to the intuition that with the use of momentum the step length
  needs to be chosen more carefully, actually the opposite is true.

\begin{figure}[h]
\begin{center}
  \caption{$\rho(M^{(i)})$ as a function of $\alpha D_{i,i}\in (0,0.1)$
    for fixed values of $\beta$.}
\label{pic.tp3}
\includegraphics[width=130truemm]{threebetas.eps}
\end{center}
\end{figure}

\subsection{Selecting $\alpha$ and $\beta$}\label{sec.par}
In the following it is assumed that a lower bound
$$
0< \underline{m} \le \min_{1\le i\le n} D_{i,i} \qquad
\hbox{and an upper bound} \qquad \overline{m} \ge \max_{1\le i\le n} D_{i,i}
$$
are known.
Given $\overline{m}$, the step length $\alpha$ is fixed
to
\begin{equation}\label{alphapol}
\alpha := 2/\overline{m}.
\end{equation}
In the following let
$$
\overline{\hbox{\rm con}}\hbox{\rm d}(D)\ \ :=\ \ \overline{m}/\underline{m}\ \ \ge \ \ \hbox{\rm cond}(D)
$$
be an upper bound for $\hbox{\rm cond}(D)$.
Often, an upper bound for $\overline{m}$ can be obtained, for example by
Gershgorin's theorem, while it may be difficult to determine
a positive lower bound for $\underline{m}$.
(The numerical effects of overestimating or underestimating
the condition number of $H$ are analyzed in \cite{donoghue}
along with an approach to adaptively adjust the estimate of the
condition number, provided that function evaluations are
available -- an assumption that unfortunately is not satisfied
in many stochastic settings.)

To simplify the discussion below, it is further assumed that
\begin{equation}\label{condd}
\overline{\hbox{\rm con}}\hbox{\rm d}(D) > 2.
\end{equation}
(In Theorem \ref{thm1} a stronger restriction on $\overline{\hbox{\rm con}}\hbox{\rm d}(D)$
is made; for now \mr{condd} is sufficient.
For well-conditioned problems with $\overline{\hbox{\rm con}}\hbox{\rm d}(D) \le 2$ the
plain steepest descent method with $\beta = 0$ is
optimal up to a factor of at most 2.)

When $\beta < 3-2\sqrt 2\approx 0.1716$ then there are
two values of $\alpha D_{i,i}$ for which $(1+\beta-\alpha D_{i,i})^2=4\beta$
(corresponding, for example, to the end points of the blue horizontal
line in Figure~\ref{pic.tp2}).
The considerations below apply to values
$$
\beta \ge 3-2\sqrt 2
$$
which have proved to be efficient in numerical experiments,
see, for example, \cite{gower1}, and
where only the left end point of the interval with constant values $\rho$
(i.e.\ of the horizontal lines in Figure~\ref{pic.tp2}) is relevant.

Let $\underline{\alpha} = \alpha\underline{m}$ be a lower bound for
possible values of $\alpha D_{i,i}$.
Assumption \mr{condd} implies
$\underline{\alpha} < 1$.
Setting $\beta := (1-\sqrt{\underline{\alpha}})^2$, the spectral
radius of all $M^{(i)}$ (and thus, $\rho(M)$)) is upper
bounded by $\sqrt{\beta} = 1-\sqrt{\underline{\alpha}}$, i.e.
$$
\rho(M)\leq\sqrt{\beta} = 1-\sqrt{\underline{\alpha}} =
1-\frac {\sqrt 2}{\sqrt{\overline{\hbox{\rm con}}\hbox{\rm d}(D)}}.
$$
When $\overline{\hbox{\rm con}}\hbox{\rm d}(D)$ approximates $\hbox{\rm cond}(D)$ up to a
constant factor (this is what is meant with ``appropriate'' bound
in the abstract of this paper) then the resulting
order $1- \frac 1{O(\sqrt{\hbox{\rm cond}(D)})}$ is the same order as
the optimal rate of convergence of the conjugate gradient method,
see e.g. \cite{li}.
Moreover, the $2\times 2$ transformation matrices
transforming $M^{(i)}$ to diagonal form
or, more generally, to the Jordan canonical form
are independent
of the dimension, so that the 2-norm of the transformation matrices
that transform $\hat M$ to a canonical
form also is independent of $n$.\\
However, the optimal bound for the conjugate gradient iterations
applies to the
$D$-norm, $\|z\|_D=\sqrt{z^TDz}$, while the above rate applies to
a transformed problem, and the transformation matrix may be very
ill-conditioned. This is studied
further in Section \ref{sec.trans} below.

\subsection{On the norm of the transformation matrices}\label{sec.trans}

Consider the case in  \mr{eigenv} that  $\gamma_i=\sqrt{\beta_i^2-4\beta}$
is nonzero. Then $v_+\not=v_-$
and $M^{(i)}$ is diagonalizable, i.e.\ $M^{(i)}=S\Lambda S^{-1}$ where
$\Lambda$ is the diagonal matrix with the eigenvalues $\lambda_\pm=
\tfrac 12 (\beta_i\pm\gamma_i)$ of \mr{eigenv}
and
$S:=S^{(i)}:=\begin{pmatrix}1&1\\ \lambda_+&\lambda_-\end{pmatrix}$
with columns $v_\pm$ as defined in \mr{eigenv}.

\bigskip

There are two cases: Either $\gamma_i>0$ is real or $\gamma_i$ is purely imaginary.

\bigskip

If $\gamma_i>0$ is real then $\lambda_+\lambda_- =
\tfrac 14 (\beta_i^2-\gamma_i^2)=\tfrac 14 (\beta_i^2-(\beta_i^2-4\beta))= \beta$ and
$$
S^TS = \begin{pmatrix}1+\tfrac 14 (\beta_i+\gamma_i)^2 &1+\beta \\
 1+ \beta  &  1+\tfrac 14 (\beta_i-\gamma_i)^2 \end{pmatrix}.
$$
Denote the eigenvalues of $S^TS$ by $\mu_+ \ge \mu_- > 0$.
Then
\begin{equation}\label{mu1}
\mu_\pm=1+\tfrac 14 (\beta_i^2+\gamma_i^2)\pm
\tfrac 12 \sqrt{\beta_i^2\gamma_i^2+4(1+\beta)^2}.
\end{equation}

\bigskip

If $\gamma_i$ is purely imaginary and $S^H$ denotes the complex conjugate
transpose of $S$, then
$$
S^HS = \begin{pmatrix}
  1+|\lambda_+|^2 & 1 + \bar\lambda_+\lambda_-  \\[4pt]
  1 + \lambda_+\bar\lambda_-   & 1+|\lambda_-|^2
\end{pmatrix}.
$$
Using the definitions of $\lambda_\pm$ and $\gamma_i$ it follows
$\lambda_+\bar\lambda_-  =  \lambda_+^2  $ and $|\lambda_+|^2=|\lambda_-|^2=\beta$,
and $S^HS$ is given by
$S^HS=  \begin{pmatrix}
  1+\beta & 1 + \lambda_-^2  \\
  1 + \lambda_+^2   & 1+\beta
\end{pmatrix}$
with the eigenvalues
\begin{equation}\label{mu2}
\mu_\pm =1+\beta\pm |1+\lambda_+^2|.
\end{equation}

\bigskip

\noindent
In both cases the condition number of $S$ is
$\hbox{\rm cond}(S)= \sqrt{\mu_+/\mu_-}$
with $\mu_\pm$ defined in \mr{mu1}~or~\mr{mu2}.

\bigskip

Let $\Lambda$ denote the diagonal matrix with
diagonal entries $\lambda_+$ and $\lambda_-$.
As $\lambda_+$ and $\lambda_-$ approach each other,
i.e.\ as $\gamma_i \to 0$,
the columns of $S=S^{(i)}$ become nearly linearly dependent and
$\hbox{\rm cond}(S)$ tends to infinity.
This observation is illustrated in {\bf Figure~\ref{specs}},
and it implies that the argument
\begin{equation}\label{argu1}
\|(M^{(i)})^k\|_2 = \|S\Lambda^kS^{-1}\|_2\le  \|S\|_2\|\Lambda^k\|_2\|S^{-1}\|_2
=\hbox{\rm cond}(S)\|\Lambda\|_2^k
\end{equation}
for the convergence of $(M^{(i)})^k\to 0$ as $k\to \infty$
needs to be refined when $D$ has (small) positive eigenvalues for which
$\gamma_i\approx 0$.

\begin{figure}[h]
  \caption{$\min\{\hbox{\rm cond}(S^{(i)}),\, 20\}$
    as a function of $\alpha D_{i,i}\in (0,2]$
    and $\beta\in[0,1)$ on the left, and zeros of the terms
      $\gamma_i$ depending on $\alpha D_{i,i}$
    and $\beta$ on the right plot.}
\label{specs}
\includegraphics[width=100truemm]{conds.eps}
\includegraphics[width=70truemm]{gammaizero.eps}
\end{figure}

\subsection{The Schur canonical form}\label{sec.schur}

It turns out that the case $\gamma_i\approx 0$
can be analyzed by using the Schur-decomposition
of $M^{(i)}$ with an upper right triangular matrix $R$
and a transformation matrix $T$ given by
\begin{align*}
 TRT^{-1}:=\begin{pmatrix}1&0\\ \lambda_+&1\end{pmatrix}\begin{pmatrix}\lambda_+&1\\ 0&\lambda_-\end{pmatrix}\begin{pmatrix}1&0\\ -\lambda_+&1\end{pmatrix}=\begin{pmatrix}0&1\\ -\lambda_+\lambda_-&\lambda_++\lambda_-\end{pmatrix}=\begin{pmatrix}0&1\\ -\beta&\beta_i\end{pmatrix}=M^{(i)}.
\end{align*}
Note that this decomposition is valid for both cases: for $\gamma_i=0$
as well as for $\gamma_i\neq 0$.
(When $\gamma_i=0$ i.e.\ when $\lambda_+=\lambda_-$
this is the Jordan canonical form.)

Now, in place of \mr{argu1}, also the argument
\begin{equation}\label{argu2}
  \|(M^{(i)})^{k}\| = \|(TRT^{-1})^{k}\|
  = \|TR^{k}T^{-1}\|
 \leq \|T\| \cdot \|T^{-1}\| \cdot \|R^{k}\|  =
\hbox{\rm cond}(T)\cdot \|R^{k}\|
\end{equation}
can be used. While both \mr{argu1} and \mr{argu2} lead to valid bounds
for $\|(M^{(i)})^{k}\|$ whenever \mr{argu1} is defined and thus,
the better of both bounds can be used, the estimate below relies on
\mr{argu2} which is always well defined.
Bounds on $\hbox{\rm cond}(T)$ and on $\|R^{k}\|$ are derived next:

By Gershgorin's theorem, given a triangular matrix
$$
\hat R =
\begin{pmatrix}a & b\\ 0 & c\end{pmatrix}
$$
with complex numbers $a,b,c$ and $|a|\ge |c|$, the 2-norm of
$\hat R$ is bounded by
\begin{equation}\label{2norm}
\|\hat R\|_2 =
\lambda_{max}\left(
\begin{pmatrix}\bar a & 0\\ \bar b & \bar c\end{pmatrix}
\begin{pmatrix}a & b\\ 0 & c\end{pmatrix}
\right)^{1/2}
\!\!=
\lambda_{max}\left(
\begin{pmatrix}|a|^2 & \bar ab\\ a\bar b & |c|^2+|b|^2\end{pmatrix}
\right)^{1/2}
\le
\sqrt{|a|^2+|ab|+|b|^2}
\end{equation}
(with $\lambda_{max}$ denoting the maximum eigenvalue).

Since $\|\hat R\|_2=\|\hat R^H\|_2$ the bound \mr{2norm} also applies to
the matrix $T$ and yields
\begin{equation}\label{est.t}
\|T\|_2^2
\le 1+|\lambda_+|+|\lambda_+|^2 \le 3
\end{equation}
since $|\lambda_+|\le \rho(M^{(i)})< 1$.
The same estimate applies to $\|T^{-1}\|_2^2$, so that
$\hbox{\rm cond}(T)\le 3$.

Inductively, it can be verified that the $k^\text{th}$ power of the
matrix $R$ is given by
\begin{align*}
 R^k = \begin{pmatrix} \lambda_+^k & \sum_{\ell=0}^{k-1}\lambda_+^\ell\lambda_-^{k-1-\ell} \\ 0 & \lambda_-^k \end{pmatrix}.
\end{align*}
Without loss of generality assume that $|\lambda_-|\le|\lambda_+|$,
else rename $\lambda_-$ and $\lambda_+$.
(If $\gamma_i=0$, then actually $|\lambda_-|=|\lambda_+|$ ---
and this is the ``worst case'' in the estimate below).
In any case, it follows with the triangle inequality
that the absolute value of the upper right entry
of $R^k$ is bounded by $|(R^k)_{1,2}|\le k|\lambda_+|^{k-1}$, and then,
by \mr{2norm}, that
\begin{equation}\label{est.r}
\|R^k\|\le |\lambda_+|^{k-1}\sqrt{ |\lambda_+|^2+ k|\lambda_+|+k^2}
\le |\lambda_+|^{k-1}\sqrt{ 1+ k+k^2}\le |\lambda_+|^{k-1}(k+1).
\end{equation}
Since the condition number of $T$ is at most 3, the norm
$\|(M^{(i)})^{k}\|$ is at least $\frac 13 \|R^k\|$ and at most $3\|R^k\|$.
The latter observation can be used to bound the
number of iterations needed to reduce the (unknown) Euclidean distance
of a given point $x^0$ to the optimal solution by a factor $\epsilon>0$.
This is done in the next section with a slightly tighter bound
of $\|(M^{(i)})^k\|$.

\subsection{On the rate of convergence}
Inserting \mr{est.t} and \mr{est.r} in \mr{argu2} yields
$
\|(M^{(i)})^k\|_2\le 3 \rho^{k-1}(k+1)
$
where $\rho=\rho(M^{(i)})\ge |\lambda_+|$. This bound is based on
the sub-multiplicativity of the 2-norm.
When computing $(M^{(i)})^k$ explicitly, it can be reduced to
\begin{equation}\label{verylast}
\|(M^{(i)})^k\|_2\le 2 \rho^{k-1}(k+1).
\end{equation}
The somewhat tedious calculations leading to \mr{verylast} are
given in the Appendix.
Let a desired accuracy $\epsilon\in(0,1)$ be given.
To obtain a sufficient condition for
\begin{equation}\label{boundk}
2 \rho^{k-1}(k+1)\le \epsilon  \ \ \Longleftrightarrow \ \
(k\!-\!1)\ln(\rho)  +\ln(k+1)\le \ln(\tfrac \epsilon 2)
\end{equation}
the bound $\ln(\rho)\le \rho-1$ is used,
\begin{eqnarray}
\mr{boundk}   \ \ &\Longleftarrow& \ \
(k\!-\!1)(\rho-1)  +\ln(k+1)\le \ln(\tfrac \epsilon 2) \nonumber \\
&\Longleftrightarrow& \ \ (1-\rho)(k\!-\!1) \ge \ln(\tfrac 2\epsilon)
+ \ln(k+1)\label{bound1}
\end{eqnarray}
Denote
$$
\delta := \tfrac1{\sqrt{2\ \overline{\hbox{\rm con}}\hbox{\rm d}(D)}} =
 \tfrac{1-\sqrt\beta}{2} \le \tfrac{1-\rho}{2}.
$$
If the following two
inequalities are satisfied then \mr{bound1} is satisfied as well,
\begin{equation}\label{bound2}
\delta (k-1) \ge \ln(\tfrac 2\epsilon) \qquad \hbox{and} \qquad
\delta (k-1) \ge \ln(k+1).
\end{equation}
Here $\delta>0$ depends on $\beta=(1-\sqrt{\underline{\alpha}})^2$,
i.e.\ on the choice of the parameters $\alpha,\beta$ that in turn
depend on the bounds $\underline{m}$ and $\overline{m}$
of the eigenvalues of $D$. Below it is assumed that $\delta \le \exp(-2)$
-- which is the case when $\overline{\hbox{\rm con}}\hbox{\rm d}(D)\ge 28$.

Then, the second relation in \mr{bound2} is satisfied whenever
$k\ge \bar k:=\frac 2\delta \ln(\frac 1\delta)-1$.

\noindent
Indeed,
\begin{eqnarray*}
\delta(\bar k-1) - \ln(\bar k + 1)&=&
2\ln(\tfrac 1\delta)-2\delta-\ln(\tfrac 2\delta \ln(\tfrac 1\delta))\\
&=& \ln(\tfrac 1\delta)-2\delta-\ln(\ln(\tfrac 1\delta))-\ln(2)\\
&>&0
\end{eqnarray*}
for $0<\delta \le \exp(-2)$.

Thus it suffices to ensure that the first inequality in \mr{bound2}
implies the second. This is the case when $\epsilon$ is sufficiently small:
Indeed, the first bound in \mr{bound2} is satisfied
for $k=\bar k$ with equality if
$$
\epsilon = \bar\epsilon :=
2\delta^2e^{2\delta} .
$$
For $\epsilon \le \bar\epsilon$ all values of $k$ satisfying
the first relation in \mr{bound2} are greater or equal to $\bar k$
so that the second relation is satisfied as well.
Since $e^{2\delta}>1$ it is sufficient to choose
\blue{$\epsilon \le 2\delta^2 = 1\, / \overline{\hbox{\rm con}}\hbox{\rm d}(H)$}. The first relation in
\mr{bound2} can be written as
\begin{equation}\label{last}
k\ge \ln\left(\tfrac 2\epsilon \right)\cdot \tfrac 1\delta +1 = \ln\left(\tfrac 2\epsilon \right)\cdot
\sqrt{ 2\ \overline{\hbox{\rm con}}\hbox{\rm d}(D)}+1.
\end{equation}
Thus, when $k$ satisfies \mr{last}
with $\epsilon \le 2\delta^2$, relation \mr{boundk} is satisfied and
\blue{
\begin{eqnarray*}
&&\sqrt 2 \| \tfrac 12 (x^{k-1}+x^k)\|_2
=\left\| \tfrac 12\left(\left(
\begin{matrix} x^{k-1} \\ x^k \end{matrix} \right)+\left(
\begin{matrix} x^k \\ x^{k-1} \end{matrix} \right)\right)
\right\|_2\le
\tfrac 12\left(\left\| \left(\begin{matrix} x^{k-1} \\ x^k \end{matrix} \right)
\right\|_2 + \left\| \left(\begin{matrix} x^{k} \\ x^{k-1} \end{matrix} \right)
\right\|_2\right) \\
&=&
\tfrac 12\left(\left\| \left(\begin{matrix} x^{k-1} \\ x^k \end{matrix} \right)
\right\|_2 + \left\| \left(\begin{matrix} x^{k-1} \\ x^k \end{matrix} \right)
\right\|_2\right) =
\left\| \left(\begin{matrix} x^{k-1} \\ x^k \end{matrix} \right)
\right\|_2 \le \epsilon
\left\| \left(\begin{matrix} x^{0} \\ x^1 \end{matrix} \right)
\right\|_2 \le \sqrt 2 \epsilon \|x^0\|_2
\end{eqnarray*}
}%
where the last inequality follows from $\|x^1\|_2\le \|x^0\|_2$,
the first step being a plain steepest descent step with step length
$2/\overline m$.
Summarizing, the claim of Theorem \ref{thm1} follows.
\hfill $\square$

\blue{
\subsection{Nesterov's accelerated gradient method}\label{sec.nest}
For the analysis of Nesterov's accelerated gradient method
again it suffices to consider the function $f$ in \mr{assu}
with $\nabla f(x) = Dx$.
Replacing $(HBM)$ with $(NAG)$ in the form \mr{nag} it follows that the matrix
$
M^{(i)}=\left(\begin{matrix} 0 && 1 \\ -\beta  && \beta_i \end{matrix} \right)
$
in \mr{defmi}
is to be replaced with
$$
M^{(i)}:=\left(\begin{matrix} 0 && 1 \\ -\beta(1-\alpha_i)  && (1+\beta)(1-\alpha_i) \end{matrix} \right)
$$
where again, $\alpha_i:=\alpha D_{i,i}$.
Thus, replacing $\beta$ and $\beta_i$ in \mr{defmi} with
$\beta(1-\alpha_i)$ and $(1+\beta)(1-\alpha_i)$ it follows that $\rho$
in \mr{defrho} changes to
\begin{equation}\label{defrho2}
\rho:=\rho(M^{(i)}) = \left\{\begin{array}{ll}
\sqrt{\beta(1-\alpha_i)} & \hbox{if} \ \ (1+\beta)^2(1-\alpha_i)^2\le 4\beta(1-\alpha_i)\\[4pt]
\frac 12\left( | (1+\beta)(1-\alpha_i) |+\bar\gamma_i \right)
& \hbox{else}
\end{array}\right.
\end{equation}
where $\bar \gamma_i:= \sqrt{(1+\beta)^2(1-\alpha_i)^2- 4\beta(1-\alpha_i)}$.
The first case of definition \mr{defrho2} can only be obtained for
$\alpha_i\le 1$.
This results in
half the step length $\alpha:=\frac 1{\overline{m}}$
compared to \mr{alphapol} for $(HBM)$ and
it then follows for $\alpha_i\in [1/\overline{\hbox{\rm con}}\hbox{\rm d}(D),1]$ and
$\beta:=\tfrac{\left(\sqrt{\overline{\hbox{\rm con}}\hbox{\rm d}(D)}-1\right)^2}{\overline{\hbox{\rm con}}\hbox{\rm d}(D)-1}$ that
$(1+\beta)^2(1-\alpha_i)\le 4\beta$
and
$$
\rho=\sqrt{\beta(1-\alpha_i)}\le 1-\tfrac 1{\sqrt{\overline{\hbox{\rm con}}\hbox{\rm d}(D)}}.
$$
This is the same bound as for the $(MM)$ when taking half the step length
$\alpha$ or -- which is the same -- when replacing $\overline{m}$
in $(MM)$ with $2\overline{m}$.
In contrast to the $(MM)$, however,
based on this analysis a theoretical justification of a longer step
length $\alpha > 1/\overline{m}$ is not possible.\\
The eigenvalues of $M^{(i)}$ are now given by
$$
\lambda_\pm = \tfrac 12 (\ (1+\beta)(1-\alpha_i)\pm
\sqrt{(1+\beta)^2(1-\alpha_i)^2-4\beta(1-\alpha_i)}\ )
$$
and with this definition
the matrix $T$ transforming $M^{(i)}$ to  the Schur canonical form
is the same as in Section \ref{sec.schur}.
The iteration complexity for $(MM)$ thus applies to $(NAG)$
as well when replacing $\overline{\hbox{\rm con}}\hbox{\rm d}(D)$ with $2\,\overline{\hbox{\rm con}}\hbox{\rm d}(D)$.
Summarizing, the following result is obtained:
\begin{theorem}\label{thm2}
  Let $f:{\mathbb{R}}^n\to{\mathbb{R}}$ be a strictly convex quadratic function
  with minimizer $x^*$ and
  denote $H:=\nabla^2f(x^*)$. Let
  $\overline{m}$ be an upper bound for the eigenvalues of $H$
  and let $0<\underline{m}$ be a lower bound for the eigenvalues
  of $H$. Then $\overline{\hbox{\rm con}}\hbox{\rm d}(H):=\overline{m}/\underline{m}$
  is an upper bound for the condition number of $H$. Assume
  that $\overline{\hbox{\rm con}}\hbox{\rm d}(H)\ge 28$.
  \\
  In $(HBM)$ define $\alpha=1/\overline{m}$, \
  $\underline{\alpha}=1/\overline{\hbox{\rm con}}\hbox{\rm d}(H)$,  \
  $\beta = \tfrac{\left(1-\sqrt{\underline{\alpha}}\right)^2}
  {1-\underline{\alpha}}$
  and let
  $\epsilon\le 1\, / \, \overline{\hbox{\rm con}}\hbox{\rm d}(H)$ be given.
  Then, given $x^0\in{\mathbb{R}}^n$, after
  $$
  1+\lceil 2\,\sqrt{\overline{\hbox{\rm con}}\hbox{\rm d}(H)}\
  \ln(\tfrac {2}\epsilon)\rceil
  $$
  steps of the $(NAG)$, an approximate solution
  $\bar x^k:=\tfrac 12 (x^{k-1}+x^k)$ is
  generated with
  $$
  \|\bar x^k-x^*\|_2\le

  \epsilon \|x^0-x^*\|_2.
  $$
\end{theorem}
}

\noindent
\subsection{\red{A note on the rate of convergence}}
Theorem~\ref{thm1} and Theorem~\ref{thm2} address the iteration
complexity with respect to the Euclidean norm while the
asymptotic rate of convergence of $(NAG)$ or $(HBM)$ (with respect to any
fixed norm)
is given by the maximum spectral radius of all matrices $M^{(i)}$ which coincides
with the value $1-\sqrt{\underline\alpha}$ under the assumptions of
Theorems~\ref{thm1} and \ref{thm2}.
Thus, the asymptotic rate of convergence is
\begin{itemize}
\item
$1- \sqrt{1/\overline{\hbox{\rm con}}\hbox{\rm d}(H)}$ for $(NAG)$ and
\item
$1- \sqrt{2/\overline{\hbox{\rm con}}\hbox{\rm d}(H)}$ for $(HBM)$.
\end{itemize}
Here, the rate for $(HBM)$ is somewhat worse than the asymptotic rate
\hbox{$1-2/\sqrt{\overline{\hbox{\rm con}}\hbox{\rm d}(H)}$}
for $(HBM)$ established in \cite{polyak}, Theorem 9 (3),
the difference being due to the limitation of the step length $\alpha$
which is nearly twice as long in \cite{polyak} as considered here
-- and four times as long as for Nesterov's accelerated gradient method.

The rates of convergence are not just upper bounds but exact rates
once the values $\underline m$ and $\overline m$ are given, and hence,
the difference in the rates of convergence for $(NAG)$ and $(HBM)$
can be explained just by the limitations of the step length: when the $(HBM)$
is restricted to steps $\alpha = 1/\overline m$ as for $(NAG)$, its asymptotic
rate of convergence is exactly the same as $(NAG)$.

The above rate for Nesterov's accelerated gradient method is the same
as established for the convergence of the function values in
\cite{nestg2}, Theorem 2.2.3.
Translated to error measure
$\|x-x^*\|_H = \sqrt{f(x)-f(x^*)}$ the rate of convergence
of \cite{nestg2}, Theorem 2.2.3 reduces to approximately
$1-\sqrt{1/2\, \overline{\hbox{\rm con}}\hbox{\rm d}(H)}$.

\section{Conclusion}
The asymptotic rate of convergence of the momentum method for
fixed parameters has been analyzed in the classical work \cite{polyak}.
Here, the constants in this work are considered and
an explicit bound for the iteration complexity is derived
\blue{that also applies in slightly modified form to
Nesterov's accelerated gradient method}.

\subsection*{Acknowledgment}
The authors would like to thank Robert Gower for helpful
e-mail-discussions.

\subsection*{Appendix}
$M^k\equiv(M^{(i)})^k$ can also be written as
\begin{align*}
(M^{(i)})^k &= TR^kT^{-1} =
\begin{pmatrix}1&0\\\lambda_+&1\end{pmatrix}
\begin{pmatrix}\lambda_+^k&\ \ \ \sum_{\ell=0}^{k-1}\lambda_+^\ell\lambda_-^{k-1-\ell}\\[4pt]
  0&\lambda_-^k\end{pmatrix}
\begin{pmatrix}1&0\\-\lambda_+&1\end{pmatrix}\\
&=
\begin{pmatrix}1&0\\\lambda_+&1\end{pmatrix}
\begin{pmatrix}\lambda_+^k-\lambda_+\sum_{\ell=0}^{k-1}\lambda_+^\ell\lambda_-^{k-1-\ell}&
  \ \ \ \ \ \sum_{\ell=0}^{k-1}\lambda_+^\ell\lambda_-^{k-1-\ell}\\[4pt]
  -\lambda_+\lambda_-^k&\lambda_-^k\end{pmatrix}\\
&=
\begin{pmatrix}\lambda_+^k-\lambda_+\sum_{\ell=0}^{k-1}\lambda_+^\ell\lambda_-^{k-1-\ell}&
  \sum_{\ell=0}^{k-1}\lambda_+^\ell\lambda_-^{k-1-\ell}\\[4pt]
  \lambda_+^{k+1}-\lambda_+^2\sum_{\ell=0}^{k-1}\lambda_+^\ell\lambda_-^{k-1-\ell}-\lambda_+\lambda_-^k&
  \ \ \ \ \ \lambda_+\sum_{\ell=0}^{k-1}\lambda_+^\ell\lambda_-^{k-1-\ell}+\lambda_-^k\end{pmatrix}\\
&=
\begin{pmatrix}\lambda_+^k-\sum_{\ell=0}^{k-1}\lambda_+^{\ell+1}\lambda_-^{k-1-\ell}&
  \sum_{\ell=0}^{k-1}\lambda_+^\ell\lambda_-^{k-1-\ell}\\[4pt]
  \lambda_+^{k+1}-\sum_{\ell=0}^{k-1}\lambda_+^{\ell+2}\lambda_-^{k-1-\ell}-\lambda_+\lambda_-^k&
  \ \ \ \ \ \sum_{\ell=0}^{k-1}\lambda_+^{\ell+1}\lambda_-^{k-1-\ell}+\lambda_-^k\end{pmatrix}\\
&=
\begin{pmatrix}-\sum_{\ell=0}^{k-2}\lambda_+^{\ell+1}\lambda_-^{k-1-\ell}&
  \sum_{\ell=0}^{k-1}\lambda_+^\ell\lambda_-^{k-1-\ell}\\[4pt]
  -\sum_{\ell=-1}^{k-2}\lambda_+^{\ell+2}\lambda_-^{k-1-\ell}&
  \ \ \ \sum_{\ell=-1}^{k-1}\lambda_+^{\ell+1}\lambda_-^{k-1-\ell}\end{pmatrix}\\
&=
\begin{pmatrix}-\sum_{t=1}^{k-1}\lambda_+^{t}\lambda_-^{k-t}&
  \ \ \sum_{\ell=0}^{k-1}\lambda_+^\ell\lambda_-^{k-1-\ell}\\[4pt]
  -\sum_{t=0}^{k-1}\lambda_+^{t+1}\lambda_-^{k-t}&
  \sum_{t=0}^{k}\lambda_+^{t}\lambda_-^{k-t}\end{pmatrix},
\end{align*}
where in the last step the index shift $t:=\ell+1$ was used.
The matrix product $M^k(M^k)^\text{H}$ is
\begin{align*}
 M^k(M^k)^\text{H}
 &= \begin{pmatrix}-\sum_{\ell=1}^{k-1}\lambda_+^{\ell}\lambda_-^{k-\ell}&\sum_{\ell=0}^{k-1}\lambda_+^\ell\lambda_-^{k-1-\ell}\\[4pt] -\sum_{\ell=0}^{k-1}\lambda_+^{\ell+1}\lambda_-^{k-\ell}&\sum_{\ell=0}^{k}\lambda_+^{\ell}\lambda_-^{k-\ell}\end{pmatrix}
 \begin{pmatrix}-\sum_{\ell=1}^{k-1}\bar{\lambda}_+^{\ell}\bar{\lambda}_-^{k-\ell}&-\sum_{\ell=0}^{k-1}\bar{\lambda}_+^{\ell+1}\bar{\lambda}_-^{k-\ell}\\[4pt]
 \sum_{\ell=0}^{k-1}\bar{\lambda}_+^\ell\bar{\lambda}_-^{k-1-\ell} &\sum_{\ell=0}^{k}\bar{\lambda}_+^{\ell}\bar{\lambda}_-^{k-\ell}\end{pmatrix}\\
 &=\begin{pmatrix}a&b\\c&d\end{pmatrix}
\end{align*}
with
\begin{align*}
 a &:= \left(\sum_{\ell=1}^{k-1}\lambda_+^{\ell}\lambda_-^{k-\ell}\right)\left(\sum_{\ell=1}^{k-1}\bar{\lambda}_+^{\ell}\bar{\lambda}_-^{k-\ell}\right)+\left(\sum_{\ell=0}^{k-1}\lambda_+^\ell\lambda_-^{k-1-\ell}\right)\left(\sum_{\ell=0}^{k-1}\bar{\lambda}_+^\ell\bar{\lambda}_-^{k-1-\ell}\right),\\
 b &:= \left(\sum_{\ell=1}^{k-1}\lambda_+^{\ell}\lambda_-^{k-\ell}\right)\left(\sum_{\ell=0}^{k-1}\bar{\lambda}_+^{\ell+1}\bar{\lambda}_-^{k-\ell}\right)+\left(\sum_{\ell=0}^{k-1}\lambda_+^\ell\lambda_-^{k-1-\ell}\right)\left(\sum_{\ell=0}^{k}\bar{\lambda}_+^{\ell}\bar{\lambda}_-^{k-\ell}\right),\\
 c &:= \left(\sum_{\ell=0}^{k-1}\lambda_+^{\ell+1}\lambda_-^{k-\ell}\right)\left(\sum_{\ell=1}^{k-1}\bar{\lambda}_+^{\ell}\bar{\lambda}_-^{k-\ell}\right)+\left(\sum_{\ell=0}^{k}\lambda_+^{\ell}\lambda_-^{k-\ell}\right)\left(\sum_{\ell=0}^{k-1}\bar{\lambda}_+^\ell\bar{\lambda}_-^{k-1-\ell}\right),\\
 d &:= \left(\sum_{\ell=0}^{k-1}\lambda_+^{\ell+1}\lambda_-^{k-\ell}\right)\left(\sum_{\ell=0}^{k-1}\bar{\lambda}_+^{\ell+1}\bar{\lambda}_-^{k-\ell}\right)+\left(\sum_{\ell=0}^{k}\lambda_+^{\ell}\lambda_-^{k-\ell}\right)\left(\sum_{\ell=0}^{k}\bar{\lambda}_+^{\ell}\bar{\lambda}_-^{k-\ell}\right).
\end{align*}
Again it can be assumed that $|\lambda_-|\leq|\lambda_+|$. With Gershgorin's theorem, triangle inequality and $|\lambda_+|<1$ one obtains the following bound for the maximum eigenvalue of $M^k(M^k)^\text{H}$
\begin{align*}
 \lambda_{\max}(M^k(M^k)^\text{H})
 &\leq \max\{ |a|+|b|, |c| + |d| \}\\
 &\leq \max\{ (k-1)^2|\lambda_+|^{2k}+k^2|\lambda_+|^{2k-2} + k(k-1)|\lambda_+|^{2k+1}+k(k+1)|\lambda_+|^{2k-1},\\
 &\ \ \ \ \ \ \ \ \ \ \ k(k-1)|\lambda_+|^{2k+1}+k(k+1)|\lambda_+|^{2k-1} + k^2|\lambda_+|^{2k+2}+(k+1)^2|\lambda_+|^{2k} \}\\
 &\leq (k+1)^2|\lambda_+|^{2k}+k^2|\lambda_+|^{2k-2} + k(k-1)|\lambda_+|^{2k+1}+k(k+1)|\lambda_+|^{2k-1}\\
 &= |\lambda_+|^{2(k-1)}\left((k+1)^2|\lambda_+|^{2}+k^2+k(k-1)|\lambda_+|^3+k(k+1)|\lambda_+|\right)\\
 &\leq |\lambda_+|^{2(k-1)} ((k+1)^2+k^2+k^2-k+k^2+k)
 = |\lambda_+|^{2(k-1)} ((k+1)^2+3k^2)\\
 &\leq |\lambda_+|^{2(k-1)} ((k+1)^2+3(k+1)^2) = 4|\lambda_+|^{2(k-1)} (k+1)^2.
\end{align*}
This way, an upper bound for the norm of $M^k$ is given by
\begin{align*}
 \|M^{(i)}\|_2 \leq 2|\lambda_+|^{k-1} (k+1).
\end{align*}