Solving linear algebraic equations with Krylov subspace methods is still interesting!

Abstract: The title of this paper is motivated by the title of the paper by Forsythe written in 1952 and published in Bull. Amer. Math. Soc., 59 (1953), pp. 299-329. Forsythe argues that solving a system of $n$ linear algebraic equations in $n$ unknowns is mathematically a lowly subject. His beautiful text graduates with what was at that time ``the newest process on the roster, the method of conjugate gradients.'' We consider it important to revisit, after 70 years, to what extent Forsythe's views, and the views presented in the related contemporary works of Hestenes, Stiefel, Lanczos, Karush, and Hayes, remain relevant today.Including, besides the conjugate gradient method (CG), also the generalized minimal residual method (GMRES), we point out building blocks that we consider central for the current mathematical and computational understanding of Krylov subspace methods. We accomplish this through a set of computed examples. We keep technical details to a minimum and provide references to the literature. This allows us to demonstrate the mathematical beauty and intricacies of the methods, and to recall some persistent misunderstandings as well as important open problems. We hope that this work can initiate further theoretical investigations of Krylov subspace methods.This paper can not cover all Krylov subspace methods. The principles discussed for CG and GMRES are, however, important for all of them. Further, practical computations always incorporate preconditioning. We will not deal with preconditioning techniques, but we will deal with the basic question of how preconditioning is motivated, and we will recall some recent analytic results.

\section{Introduction}\label{sec:intro}

Taking the 1952 landmark paper of Hestenes and Stiefel~\cite{HesSti52} as their historical starting point, Krylov subspace methods for solving linear algebraic systems $Ax=b$ have been around for exactly 70 years in 2022. The algorithmic ideas behind these methods are counted among the Top 10 algorithmic ideas of the 20th century~\cite{Cip00}. Hence it is certainly not a surprise that numerous different Krylov subspace methods have been developed over the last 70 years, and that these methods are nowadays used widely throughout the sciences. Their names typically give short-hand descriptions of their mathematical properties, and the methods then are referred to by acronyms that abbreviate these names. Examples include the conjugate gradient (CG) method~\cite{HesSti52}, the biconjugate gradient (BiCG) method~\cite{Fle76}, the stabilized biconjugate gradient (BiCG-Stab) method~\cite{Van92}, the full orthogonalization method (FOM)~\cite{Saa81}, the minimal residual (MINRES) method~\cite{PaiSau75}, the quasi-minimal resiudal (QMR) method~\cite{FreNac91}, the generalized minimal residual (GMRES) method~\cite{SaaSch86}, and related methods like the induced dimension reduction (IDR) method~\cite{WesSon80}.
In this paper we will focus on CG and GMRES, which have evolved as the standard methods for symmetric positive definite and general (nonsymmetric) matrices, respectively.

Over the last 70 years, 10.000s of research articles on derivation, analysis, or applications of Krylov subspace methods have been published by authors coming from the most diverse scientific backgrounds. Some of them have been covered in the published monographs or survey articles devoted entirely, or at least in part, to Krylov subsapce methods; see, e.g., \cite{GreBook97,LieStr13,MeuBook06,MeuDui20,Saa03,VanBook03} and~\cite{EieErn01,FreGolNac92,GolVdV97,LieTic04,MeuStr06}, respectively. Naturally, given such a plethora of publications, not all are equally important, and not everything is consistent, or even mathematically correct. Yet, the field of Krylov subspace methods is plagued by a number of misunderstandings, which are published again and again, and which obstruct the further development of the field. Our main goal in this paper is to clarify the most serious misunderstandings.

Each Krylov subspace method for $Ax=b$ starts with some initial vector $x_0$ and is based on projecting onto (some variant of) the Krylov subspaces $\mathcal{K}_k(A,r_0)={\rm span}\{r_0,Ar_0,\dots,A^{k-1}r_0\}$, $k=1,2,\dots,$, where $r_0=b-Ax_0$. Often the mathematical characterization of a method and even its algorithmic implementation are incredibly simple to write down. But describing its behavior, both mathematically and computationally, can be full of surprises and subtleties. The repeated multiplications with~$A$ and subsequent projection onto a small ($k$-dimensional) subspace yield a method that is \emph{nonlinear} in the matrix $A$ as well as the initial residual $r_0$, although the problem to be solved is \emph{linear}. Neglecting this essential point appears to be the source of the persistent misunderstandings mentioned above. In some cases the nonlinear behavior of Krylov subspaces is simply, and mathematically incorrectly, identified with widely known linear convergence bounds. In other cases, the nonlinear behavior is accepted but viewed primarily as an obstacle for the analysis. However, the \emph{nonlinearity is the main mathematical asset} of this class of methods, since it allows the method to adapt to the hidden inner structure of the problem to be solved. This can lead to a significant speedup of the convergence in comparison with (linear) iterative methods that do not adapt to the problem.

For this paper we have surveyed the published literature, and we have selected results about the CG and GMRES methods that we consider milestones of the development, particularly in the methods' understanding, in the last 70 years. Keeping technical details at a minimum, we present all results in form of \emph{computed examples} rather than with theorems and proofs. The purpose of the examples is to show the mathematical beauty of the methods and their hidden intricacies, to clarify some frequent misunderstandings, and to point out important open problems. Some of these open problems have been left untreated because of a frequent use of severely restrictive simplifications that, at the same time, claim general applicability. This hampers further investigation of difficult points, which is critically needed for both advancing theory and for practical applications.

In each example we first describe the setup as transparent as possible. We then describe the observations to be made in the computed figures, followed by an explanation which usually contains pointers to the research literature. The examples purposely use simple data (matrices and right-hand sides), so that they can be easily recomputed. The same phenomena can be observed also in problems coming from real-world applications.

Throughout this paper we consider linear algebraic systems and finite-dimensional problems. We point out that Krylov subspace methods can also be used in infinite-dimensional settings. The papers by Hestenes and Stiefel~\cite{HesSti52}, and Lanczos~\cite{Lan50,Lan52}, which introduced the first Krylov subspace methods for linear algebraic systems, were immediately followed by the description and investigation of the CG method in the infinite dimensional Hilbert space setting; see, e.g., the work of Karush~\cite{Kar52}, Hayes~\cite{Hay54}, and Vorobyev~\cite{VorBook65}. Some of the results for finite dimensions can be easily extended to an operator-based setting. This holds particularly in the context of PDEs and operator preconditioning. On the other hand, the results obtained in the infinite-dimensional setting can be stimulating for understanding the behavior of the methods applied to discretized problems; see, e.g.,~\cite{MSB15} and the references given there.

The paper is organized as follows. In Section~\ref{sec:CG} we consider the CG method, and in Section~\ref{sec:GMRES} the GMRES method. Both sections start with a brief description of the methods (mathematical properties and standard implementations), followed by the computed examples. Section~\ref{sec:Conclusions} contains our concluding remarks.

\smallskip
\paragraph{Notation and conventions} Throughout the paper we consider real linear algebraic systems for simplicity of notation. Most results can be easily extended to the complex case. We use $N$ for the matrix size, i.e., $A\in\mathbb{R}^{N\times N}$, and~$k$ denotes the iteration number. Usually the right-hand side~$b$ is a normalized vector of ones, and the initial approximate solution is $x_0=0$. We use the term ``mathematical'' to refer to cases where computations are performed exactly, i.e., in infinite precision, and the term ``computational'' to refer to finite precision computations. In some experiments we compare the infinite and finite precision behavior of algorithms. Unless otherwise specified, the infinite precision behavior is then simulated using the Advanpix Toolbox for MATLAB \cite{advanpix}.

\smallskip
\paragraph{Dedication} {We have written this paper on the occasion of the 70th anniversary of the publication of the Hestenes and Stiefel paper~\cite{HesSti52} in the Journal of Research of the National Bureau of Standards (NBS) in 1952. The official publication date on the paper is ``December~1952''. The paper ends with ``Los Angeles, May 8, 1952'', so the work was supposedly finished on that day. Interestingly, the NBS Report No.~1659 containing the paper ends with ``May 8, 1952'' (without ``Los Angeles''), but the title page states the date ``March 10, 1952''. And indeed, the NBS Report No.~1661 (Projects and Publications of the National Applied Mathematics Laboratories January 1952 through March 1952) contains: ``Method of conjugate gradients for solving linear systems,'' by E. Stiefel and M. R. Hestenes.\footnote{A delayed errata: Chris Paige pointed out that Theorem 7.5 in the published paper contains 3 typos, one in the statement and 2 in the proof. None of these typos occur in the typewritten NBS report.}}

The Hestenes and Stiefel paper as well as the similarly fundamental papers of Lanczos~\cite{Lan50,Lan52} laid the foundations for Krylov subspace methods (for the description of the early history see~\cite{GolOle89}), which are counted among the Top~10 algorithmic ideas of the 20th century~\cite{Cip00}. These papers, and another important work of Lanczos on the use of Chebyshev polynomials in the iterative solution of linear algebraic systems~\cite{Lan53}, made links to a vast variety of mathematical topics, and contain many deep insights; see~\cite[Figure~1.4]{LieStr13} for an overview. Nowadays the papers are frequently cited; the paper~\cite{HesSti52} has more than 1,000 citations in MathSciNet and more than 10,000 in Google Scholar as of late 2022. Yet, many fundamental points presented in these papers remain almost unnoticed, and the common state-of-the-art literature even contains views that contradict them.

\section{The CG Method}\label{sec:CG}

The CG method is well defined for any linear algebraic system $Ax=b$ with a symmetric positive definite matrix $A\in\mathbb{R}^{N\times N}$ and right-hand side $b\in\mathbb{R}^N$. If $x_0\in\mathbb{R}^N$ is an initial approximation, and $d=d(A,r_0)$ is the grade of the initial residual $r_0=b-Ax_0$ with respect to $A$, then at every step $k=1,2,\dots,d$ the CG method constructs a uniquely determined approximation
\begin{equation}\label{eqn:CGorth}
x_k\in x_0+\mathcal{K}_k(A,r_0)\quad\mbox{such that}\quad r_k\perp \mathcal{K}_k(A,r_0),
\end{equation}
where $\mathcal{K}_k(A,r_0):={\rm span}\{r_0,Ar_0,\dots,A^{k-1}r_0\}$ is the $k$th Krylov subspace generated by $A$ and $r_0$. In exact arithmetic the method terminates with $x_d=x$.

There are many mathematically equivalent formulations of the task that is solved by the CG method. For example, at step $k$ the CG method determines the solution of the simplified Stieltjes moment problem (see~\cite{PozStr19}) or, equivalently, it determines the $k$-point Gauss quadrature of the Riemann-Stieltjes
integral defined by $A$ and $r_0$; see, e.g.,~\cite[Section~3.5]{LieStr13} and~\cite[Section~5.2]{MSB15} for overviews.

There are also many mathematically equivalent algorithms that realize the projection process \eqref{eqn:CGorth}. The most popular variant is the original formulation of Hestenes and Stiefel~\cite{HesSti52}, shown in Algorithm \ref{alg:hscg}. This algorithm recursively updates 2-term recurrences for the approximate solution $x_{k+1}$ and residual $r_{k+1}$, as well as the auxiliary ``search direction'' vector $p_{k+1}$. As it turns out, this variant is also preferable numerically; see, e.g., \cite{Rei71, GutStr00}.

\begin{algorithm}[htbp!]
	\caption{Conjugate Gradient (2-term recurrence variant) \label{alg:hscg}}
	\begin{algorithmic}[1]
		\Require{Symmetric positive definite matrix $A\in\mathbb{R}^{N\times N}$; right-hand side $b$; initial approximation $x_0$; convergence tolerance $\tau$; maximum number of iterations $n_{\max}$.}
		\State{$r_0 = b-Ax_0$}
		\State{$p_0 = r_0$}
		\For{$k = 0, 1, 2,\ldots,n_{\max} $}
			\State{$\alpha_k = (r_k^T r_k)/(p_k^T Ap_k)$}
			\State{$x_{k+1} = x_k + \alpha_k p_k$}
			\State{$r_{k+1} = r_k - \alpha_k Ap_k$}
			\State{Test for convergence using tolerance $\tau$. If satisfied, then return $x_{k+1}$ and stop.}
			\State{$\beta_{k+1} = (r_{k+1}^T r_{k+1})/(r_k^T r_k)$}
			\State{$p_{k+1} = r_{k+1} + \beta_{k+1}p_k$}
		\EndFor
	\end{algorithmic}
\end{algorithm}

Let $A=Q\Lambda Q^T$ be an orthogonal diagonalization of $A$ with $\Lambda={\rm diag}(\lambda_1,\dots,\lambda_N)$ and $0<\lambda_1\leq\cdots\leq \lambda_N$. We can represent the initial residual $r_0$ by its components in the individual eigenvectors of $A$, stored in the columns of $Q$, as $r_0=Q[\eta_1,\dots,\eta_N]^T$. The approximation $x_k\in x_0+\mathcal{K}_k(A,r_0)$ that is uniquely determined by the orthogonality condition in \eqref{eqn:CGorth} satisfies the (equivalent) optimality property
\begin{align}
\|x-x_k\|_A &= \min_{p\in P_k(0)}\|p(A)(x-x_0)\|_A=\min_{p\in P_k(0)}\left(\sum_{i=1}^N \eta_i^2
\frac{p(\lambda_i)^2}{\lambda_i}\right)^{1/2}, \label{eqn:CGmin}
\end{align}
where $P_k(0)$ denotes the set of polynomials of degree at most $k$ with value~$1$ at the origin; see, e.g.,~\cite[Section~5.6]{LieStr13}. Thus, in every step the CG method solves a certain weighted polynomial approximation problem on the discrete set $\{\lambda_1,\dots,\lambda_N\}$. Moreover, if $\theta_1^{(k)}, \dots , \theta_k^{(k)}$ are the $k$ roots of the polynomial providing the minimum in~(\ref{eqn:CGmin}), then we can easily get
\begin{equation}
\|x - x_k\|_A^2 =  \sum_{i=1}^N\prod_{\ell=1}^k \left(1-\frac{\lambda_i}{\theta_\ell^{(k)}}\right)^2 \frac{\eta_i^2}{\lambda_i},
\label{eqn:CG_Ritz_values}
\end{equation}
which establishes the relationship of the roots of the minimizing polynomial in~(\ref{eqn:CGmin}), called also the Ritz values, with the eigenvalues of the matrix $A$.

Note that
$$\sum_{i=1}^N \frac{\eta_i^2}{\lambda_i} = r_0^T A^{-1}r_0 =(x-x_0)^T A (x-x_0)=\|x-x_0\|_A^2.$$
Therefore, maximizing over the values $p(\lambda_i)$ in the minimization problem on the right-hand side of \eqref{eqn:CGmin} and dividing by $\|x-x_0\|_A$ gives the upper bound
\begin{equation}
\frac{\|x-x_k\|_A}{\|x-x_0\|_A}\leq
\min_{p\in P_k(0)}\max_{1\leq i\leq N} |p(\lambda_i)|
\label{eqn:polynomial_Worst_case_bound}
\end{equation}
It is important to note that the polynomial min-max approximation problem on the right-hand
side of \eqref{eqn:polynomial_Worst_case_bound} only depends on $A$, but not on $r_0$.

Let $d(A)$ be the degree of the minimal polynomial of $A$, $d(A)\geq d(A,r_0)=d$. It was shown by Greenbaum~\cite{Gre79}, that for any given symmetric positive definite matrix~$A\in\mathbb{R}^{N\times N}$ the bound \eqref{eqn:polynomial_Worst_case_bound} is sharp in the sense that for every step $k\leq d(A)$ (the degree of the minimal polynomial of $A$) there exists an initial residual $r_0$
so that equality holds.
Moreover, for every $k=1,\dots,d(A)-1$, there exist $k+1$ distinct eigenvalues
$\widehat{\lambda}_1,\dots,\widehat{\lambda}_{k+1}$ of $A$, such that
\begin{equation}\label{eqn:CG_eigbd2}
\min_{p(0)=1\atop {\deg}(p)\leq k}\max_{1\leq j\leq N}\,|p(\lambda_j)|=
\left(\sum_{i=1}^{k+1}\prod_{j=1\atop j\neq i}^{k+1}
\frac{|\widehat{\lambda}_j|}{|\widehat{\lambda}_j-\widehat{\lambda}_i|}\right)^{-1}.
\end{equation}
Thus, for the given matrix $A$ the value of the polynomial min-max approximation problem is an attainable \emph{worst-case bound} on the relative $A$-norm of the error in the CG method at every step $k \leq d(A)$. (The step $k=d(A)$ is trivial.) In addition, it expresses the value of the bound in terms of the particular eigenvalues of~$A$.

Replacing the discrete set $\{\lambda_1,\dots,\lambda_N\}$ by the continuous interval $[\lambda_1,\lambda_N]$ and using Chebyshev polynomials on this interval yields (with a small additional simplification)
\begin{align}\label{eqn:rhoAbound}
\min_{p\in P_k(0)}\max_{1\leq i\leq N} |p(\lambda_i)|\leq
2\left(\frac{\sqrt{\kappa(A)}-1}{\sqrt{\kappa(A)}+1}\right)^k,\quad
\kappa(A)=\frac{\lambda_N}{\lambda_1}.
\end{align}
Combining this with \eqref{eqn:CGmin} results in the frequently stated convergence bound
\begin{align}\label{eqn:rhoAbound1}
\frac{\|x-x_k\|_A}{\|x-x_0\|_A}\leq 2\left(\frac{\sqrt{\kappa(A)}-1}{\sqrt{\kappa(A)}+1}\right)^k.
\end{align}
We will sometimes refer to \eqref{eqn:rhoAbound1} as the \emph{$\kappa(A)$-bound}. This bound implies that if the condition number $\kappa(A)$ is (very) small, then a fast reduction of the $A$-norm of the error in the CG method can be expected. This bound does \emph{not} imply, however, that a large condition number results in a slow convergence of CG.

Also note that the $\kappa(A)$-bound for CG is a \emph{linear bound for a nonlinear process}. A comparison with the value of the polynomial min-max approximation problem in~(\ref{eqn:CG_eigbd2}), which gives the worst-case CG value in step $k$ for the given matrix~$A$, shows that neglecting the eigenvalue distribution of $A$ can mean a substantial loos of information. Similarly, a comparison with the actual minimization problem~(\ref{eqn:CGmin}), which is solved by CG applied to the linear system $Ax=b$ with the initial approximation $x_0$, shows that the size of the components $\eta_j$ of $r_0$ in the invariant subspaces of $A$ can be important; see also~(\ref{eqn:CG_Ritz_values}).

In the examples that follow, we will frequently make use of a certain class of diagonal matrices which is often used in the literature to illuminate the behavior of CG; see, e.g., \cite{GreStr92}. For given $N\geq 3$, $0<\lambda_1<\lambda_N$, and $\rho>0$ we define
\begin{equation}
A={\rm diag}(\lambda_1,\lambda_2,\dots,\lambda_{N-1},\lambda_N),\quad\mbox{where}\quad
\lambda_i = \lambda_1 + \left( \frac{i-1}{N-1}\right) (\lambda_N -\lambda_1)\rho^{N-i},
\label{eq:diagmatrix}
\end{equation}
for $i=2,\dots,N-1$.
The parameter $\rho$ determines the eigenvalue distribution of $A$. When $\rho=1$, the eigenvalues are equally spaced between $\lambda_1$ and $\lambda_N$. As $\rho$ becomes smaller, the eigenvalues accumulate towards $\lambda_1$.

\subsection{Mathematical behavior of CG for different eigenvalue distributions}\label{sec:cgeigdist}
\phantom{}
\smallskip
\begin{tcolorbox}
\emph{Main point: The convergence of CG depends on the eigenvalues; CG localizes the edges of the spectrum and adapts to the eigenvalue distribution.}
\end{tcolorbox}

\emph{Setup:} We consider the behavior of CG in exact arithmetic for matrices having three different eigenvalue distributions. All matrices are diagonal with $N=30$, $\lambda_1=0.1$, and $\lambda_N=10^3$. The first matrix is a slight modification of \eqref{eq:diagmatrix} with $\rho=0.6$, so that the eigenvalues accumulate on the right side of the spectrum. The second matrix is of the form \eqref{eq:diagmatrix} with $\rho=0.6$, so that its eigenvalues accumulate to the left side of the spectrum, and the third matrix is of the form \eqref{eq:diagmatrix} with $\rho=1$, so that its eigenvalues are equally spaced. In all cases we use $b=[1,\dots,1]^T/\sqrt{N}$, and $x_0=0$.

\smallskip
\emph{Observations:} The eigenvalue distributions are shown in the left part of Figure~\ref{fig:cgeigdist}. The right part of Figure~\ref{fig:cgeigdist} shows the behavior of exact CG (i.e., no rounding errors). For the matrix with eigenvalues accumulated to the right (blue), CG  converges fastest. For the matrix with eigenvalues accumulated to the left (red), CG converges significantly slower. For the matrix with equally spaced eigenvalues (green), CG converges the slowest.  In Figure \ref{fig:specdensexact} we show cumulative spectral density (CSD) plots using the stepwise functions with points of increase at Ritz values and the size of the vertical steps equal for each Ritz value (see \cite[Appendix C]{LinSaaYan16}). We observe large differences in the CSD approximations as the iterations proceed.

\begin{figure}
\includegraphics[width=0.43\textwidth]{figures/eigdist-eps-converted-to.pdf}\hfill
\includegraphics[width=0.475\textwidth]{figures/exactcgdist-eps-converted-to.pdf}
\caption{Left: Three distributions of 30 eigenvalues in $[0.1,\,10^3]$.
Right: The relative error in the $A$-norm versus iteration number for exact CG run with $x_0=0$ on the corresponding linear systems.}\label{fig:cgeigdist}
\end{figure}

\begin{figure}
\centering
\includegraphics[trim={4cm 3cm 4cm 3cm},clip,width=0.55\textwidth]{figures/specdensexact.pdf}
\caption{Cumulative spectral density plots for Figure \ref{fig:cgeigdist}.}\label{fig:specdensexact}
\end{figure}

\smallskip
\emph{Explanation:}
As can be seen from \eqref{eqn:CGmin}, for equal components of the initial residual in the invariant subspaces, the $A$-norm of the error minimizes the sum of the squared values of the CG polynomial divided by the associated eigenvalues. For an accumulation of the eigenvalues to the right, the CG polynomial approximates by its roots (Ritz values) the small outlying eigenvalues within a few iterations, which takes care for the associated part of the sum. For the rest of the eigenvalues that are large and close to each other, the values of the polynomial do not need to be be so small, because their squares are divided by the large eigenvalues. Therefore fast convergence (as if the small outlying eigenvalues were nonexistent) will occur within a few iterations; see \cite[Theorem 5.6.9]{LieStr13} and the enlightening paper by van der Sluis and van der Vorst \cite{SluVor86}. Section 5.6.4 of \cite{LieStr13}, called ``Outlying Eigenvalues and Superlinear Convergence'', recalls further closely related results by Lanczos, Rutishauser, Jennings, and others. The arguments above also explain why in this case the convergence rate becomes fast even when the CSD is not yet closely approximated.

For the eigenvalues accumulated to the left, the large outliers are also well approximated by the Ritz values within a few iterations. However, since for the bulk of the small eigenvalues the CG polynomial must place many roots close to the left end of the spectrum in order to make up for the division of its squared values by the small eigenvalues, the acceleration of convergence appears much later. Therefore also the CSD must be closely approximated in order to significantly decrease the CG error. For the equally spaced eigenvalues the CSD seems visually well approximated. But a closer look reveals rather slow convergence of the Ritz values to the individual eigenvalues, which proceeds from both edges of the spectrum. For more on the convergence of Ritz values in this case see \cite{DruKni95,GreStr92} and \cite[Lecture 36]{TreBau97}. Further interesting points will occur when the same experiments will be performed in finite precision arithmetic; see Section \ref{sec:fpcg} below.

\subsection{Worst-case CG and the quality of convergence bounds}\label{sec:worstcg}
\phantom{}
\smallskip
\begin{tcolorbox}
\emph{Main point: The $\kappa(A)$-bound does not account for the eigenvalue distribution or the initial residual (right-hand side), and thus can be tight only in a very particular scenario. The polynomial min-max approximation problem bound captures dependence on eigenvalue distribution, but still can be tight only for particular initial residuals (right-hand sides).}
\end{tcolorbox}

\smallskip
\emph{Setup:} We test CG on four different linear systems $Ax=b$ with $N=48$:
\begin{enumerate}
    \item $A$ as in \eqref{eq:diagmatrix} with $\lambda_1=1$, $\lambda_N=5$, $\rho=1.0$, and $b=[1,\dots,1]^T/\sqrt{N}$.
    \item $A$ as in \eqref{eq:diagmatrix} with $\lambda_1=1$, $\lambda_N=100$, $\rho=1.0$, and $b=[1,\dots,1]^T/\sqrt{N}$.
    \item $A$ as in \eqref{eq:diagmatrix} with $\lambda_1=1$, $\lambda_N=5$, $\rho=0.1$, and $b=[1,\dots,1]^T/\sqrt{N}$.
    \item $A$ as in \eqref{eq:diagmatrix} with $\lambda_1=1$, $\lambda_N=5$,  $\rho=1.0$, and $b=[\eta_1,\dots,\eta_N]^T$ is a unit norm vector with $|\eta_1|,|\eta_N|\approx 1$, and $|\eta_i|\approx 10^{-13}$, $i=2,\dots,N-1$.
\end{enumerate}

\smallskip
\emph{Observations:}
The results are shown in Figure \ref{fig:worst}, where the relative errors in the $A$-norm for CG are given by the solid blue lines, the bounds given by the polynomial min-max approximation problem \eqref{eqn:polynomial_Worst_case_bound} are given by the dotted black lines, and the $\kappa(A)$-bounds \eqref{eqn:rhoAbound1} are given by the dashed black lines. First note that the $\kappa(A)$-bound gives a good description of the CG convergence only in case 1. The bound based on the polynomial min-max approximation problem describes the convergence well in cases 1, 2, and 3, where $b$ has equal components in the eigenbasis of $A$, but not in case 4, where the $b$ only has sizeable components in the eigenvectors corresponding to the largest and smallest eigenvalues.

\begin{figure}
\centering
\includegraphics[width=0.475\textwidth]{figures/worst1-eps-converted-to.pdf}
\includegraphics[width=0.475\textwidth]{figures/worst2-eps-converted-to.pdf}\\\vspace{3mm}
\includegraphics[width=0.475\textwidth]{figures/worst3-eps-converted-to.pdf}
\includegraphics[width=0.475\textwidth]{figures/worst4-eps-converted-to.pdf}
\caption{The relative error in the $A$-norm of CG (solid blue), the bound given by the polynomial min-max approximation problem \eqref{eqn:polynomial_Worst_case_bound} (dotted black), and the $\kappa(A)$-bound \eqref{eqn:rhoAbound1} (dashed black).}\label{fig:worst}
\end{figure}

\smallskip
\emph{Explanation:}
The $\kappa(A)$-bound \eqref{eqn:rhoAbound1} does not take into account the eigenvalue distribution or the initial residual (right-hand side). Therefore this bound can only be qualitatively or quantitatively descriptive of CG convergence in a very particular scenario: when
\begin{enumerate}
    \item $A$ is well-conditioned,
    \item the eigenvalues of $A$ are uniformly distributed, and
    \item the right-hand side contains components in all eigenvectors of $A$.
\end{enumerate}
If any of these conditions fail to hold, the bound \eqref{eqn:rhoAbound1} is not a good indication of CG convergence.
As any linearization of a highly nonlinear phenomenon, the bound \eqref{eqn:rhoAbound1} can in general capture CG behavior only locally (for several iterations). It can not capture the adaptivity of CG to the data, which is its main strength. The identification of the bound \eqref{eqn:rhoAbound1} with CG convergence proclaimed elsewhere, contradicts the basic principles of CG. As a side note, we note that the $\kappa(A)$-bound \eqref{eqn:rhoAbound1} also holds approximately for finite precision CG \cite{Gre89}.

The polynomial min-max approximation problem bound from \eqref{eqn:polynomial_Worst_case_bound}, unlike the $\kappa(A)$-bound \eqref{eqn:rhoAbound1}, \emph{does} capture the effect of the eigenvalue distribution of $A$ on CG convergence. With respect to the initial residual (right-hand side) it represents the worst case scenario which is in practice often not realistic. The bound \eqref{eqn:polynomial_Worst_case_bound} will only be tight for the case where the initial residual has sizeable components $\eta_i$ in all eigenvectors of $A$. As evident from \eqref{eqn:CG_eigbd2}, the bound \eqref{eqn:polynomial_Worst_case_bound} depends with an increasing iteration number~$k$ on the same increasing number of distinct eigenvalues of $A$. Apart from very particular cases, it is therefore clear that the $\kappa(A)$-bound, which contains information only on $\lambda_1$ and $\lambda_N$, also can not capture the worst-case bound \eqref{eqn:polynomial_Worst_case_bound}. For an interesting approach regarding worst case CG behavior, see \cite{BecKui01b} and \cite{BecKui01}.

\subsection{Numerical behavior of CG on standard model problems}\label{sec:cgpoisson}
\phantom{}
\smallskip
\begin{tcolorbox}
\emph{Main point: Some model problems typically used for studying the behavior of CG, including problems with random Wishart matrices and Poisson problems, are canonically easy cases for CG, and are not indicative of the behavior of CG in general.}
\end{tcolorbox}

\smallskip
\emph{Setup:}
We test two commonly-used model problems for CG. The first is the class of random Wishart matrices which are of the form $A=R^T R\in \mathbb{R}^{n\times n}$ where $R\in \mathbb{R}^{m\times n}$ with $m\geq n$ (usually $m\gg n$) is a full rank random matrix drawn from the the standard normal distribution (generated with {\tt randn} in MATLAB).

For the second class of model problems, we generate a 2D Poisson problem discretized on a $50\times 50$ grid. This results in a linear system of dimension $N=2500$. We run CG in double precision in two settings: first, with reorthogonalization, and second, without reorthogonalization (standard CG). The setting with reorthogonalization mimics the convergence of exact CG (i.e., without rounding errors).

\smallskip
\emph{Observations:}
 In Figure \ref{fig:wishart}, we plot the spectra of 100 Wishart matrices $A=R^TR\in\mathbb{R}^{100\times 100}$, where $R\in\mathbb{R}^{500\times 100}$ along with the corresponding CG convergence curves and the $\kappa(A)$-bound \eqref{eqn:rhoAbound1} corresponding to the mean value of $\kappa(A)$. We observe that CG converges very quickly (and linearly) for the Wishart matrices, and that this behavior is well-described by the bound.

In Figure \ref{fig:poissonCG} we plot two quantities for the Poisson model problem for each setting (i.e., with and without reorthogonalization): the loss of orthogonality among Lanczos basis vectors, measured as $\Vert I-V_k^T V_k\Vert_F$, and the relative $A$-norm of the error,
$\Vert x-x_k\Vert_A/\Vert x-x_0\Vert_A$. When reorthogonalization is used, the loss of orthogonality stays around $O(\epsilon)$. Without reorthogonalization, the loss of orthogonality grows, but it seems to mirror the convergence of the relative error in the $A$-norm. The relative error in the $A$-norm is almost the same whether or not reorthogonalization is used.

\begin{figure}
\includegraphics[width=0.475\textwidth]{figures/evals_SPD_rand-eps-converted-to.pdf}
\includegraphics[width=0.475\textwidth]{figures/CG_SPD_rand-eps-converted-to.pdf}
\caption{Left: Spectra of 100 Wishart matrices $A=R^TR$ with $R\in\mathbb{R}^{500\times 100}$ and
${\rm mean}(\kappa(A))\approx 6.49$. Right: CG error norms and the bound \eqref{eqn:rhoAbound1} for mean($\kappa(A)$).}\label{fig:wishart}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=0.525\textwidth]{figures/poissonCG-eps-converted-to.pdf}
\caption{
The relative error in the $A$-norm for CG with reorthogonalization (dash-dotted blue) and without reorthogonalization (solid blue) and the loss of orthogonality amongst Lanczos vectors for CG with reorthogonalization (dotted red) and without reorthogonalization (dashed red), for a 2D Poisson problem.}
\label{fig:poissonCG}
\end{figure}

\smallskip
\emph{Explanation:}
Wishart matrices have a provably low condition number (see, e.g., \cite{Gem80,Sil85}) and, as seen in Figure~\ref{fig:wishart}, they have rather evenly-spaced eigenvalues. Thus we expect that CG will converge very quickly. The Wishart matrices $A=R^T R$ are pathologically easy examples in the CG context, and the behavior of CG for these matrices gives no information about the behavior for more difficult problems.

The Poisson problem is also a particularly easy case for CG. The loss of orthogonality is gradual, and there is no loss of rank in the computed basis, hence finite precision error does not cause delay of the CG convergence; see \cite[Section 5.9.4]{LieStr13}. For the Poisson problem, the eigenvalues are almost uniformly distributed, and the basis for the Krylov space is of good quality until convergence is nearly attained. One cannot extrapolate from the numerical behavior of CG on the Poisson problem to the numerical behavior of CG in general.

\subsection{Preconditioned CG and the condition number}\label{sec:pcg}
\phantom{}
\smallskip
\begin{tcolorbox}
\emph{Main point: Smaller condition number $\neq$ faster convergence.}
\end{tcolorbox}

\smallskip
\emph{Setup:} We define the matrix $A$ to be of the form \eqref{eq:diagmatrix} with $N=40$, $\lambda_1=10^{-3}$, $\lambda_N=100$, and $\rho=0.1$.
The $2$-norm condition number of $A$ is thus $10^5$, and eigenvalues accumulate at the lower end of the spectrum.
Let the preconditioner $P$ be the diagonal matrix such that $P^{-1}A$ is a diagonal matrix with eigenvalues equally spaced between $\lambda_1 = 10$ and $\lambda_N = 100$. Thus the preconditioned coefficient matrix has $2$-norm condition number $10$, a significant reduction from that of $A$.

\smallskip
\emph{Observations:} In Figure \ref{fig:pcgvscg}, we plot the relative error in the $A$-norm for exact CG on both $Ax=b$ (blue) and $P^{-1}A=P^{-1}b$ (red), where $b_i = 1/\sqrt{40}$ for all $i=1,\ldots,40$. For the ``preconditioned'' linear system, CG requires almost $3\times$ the number of iterations to converge to an accuracy level on the order of $10^{-8}$ as the unpreconditioned system, despite that its condition number is four orders of magnitude smaller.

\begin{figure}
\centering
\includegraphics[width=0.55\textwidth]{figures/pcgvscg-eps-converted-to.pdf}
\caption{The relative error in the $A$-norm versus iteration number for unpreconditioned CG (blue) and preconditioned CG (red). }\label{fig:pcgvscg}
\end{figure}

\smallskip
\emph{Explanation:} A smaller condition number does not necessarily imply faster convergence. The oft-repeated notions that the goal of preconditioning should be to reduce the condition number of the coefficient matrix (thus reducing the $\kappa(A)$-bound) and that doing so \emph{guarantees} an improvement in CG convergence rate is false. We direct the reader to \cite{GerMarNieStr19}, which illustrates this notion for a problem coming from PDEs. As we have previously seen, the distribution of eigenvalues is what matters for CG, and the design of preconditioners must take this into account. Identification of CG convergence behavior with the $\kappa(A)$-bound is methodologically wrong.

\subsection{Mathematical behavior of CG for problems with clustered eigenvalues}\label{sec:cgclust}
\phantom{}
\smallskip
\begin{tcolorbox}
\emph{Main point: A spectrum localized in $\ell$ tight clusters does not mean reaching a good CG approximation to the solution in $\ell$ steps.}
\end{tcolorbox}

\smallskip
\emph{Setup:} We generate three auxiliary diagonal matrices via \eqref{eq:diagmatrix} with different $\rho$ parameters to control the eigenvalue distributions. All matrices have $N=10$, $\lambda_1=0.1$, and $\lambda_N=10^3$. The first matrix uses $\rho=0.6$ and a slight modification of \ref{eq:diagmatrix} so that eigenvalues accumulate to the right. The second matrix uses $\rho=0.6$ with eigenvalues accumulated to the left. The third matrix uses $\rho=1.0$, which gives equally spaced eigenvalues. For each auxiliary matrix, we then construct a new matrix of size $N=100$, which is used in the experiment, by replacing each of the eigenvalues (diagonal entries) by a tighter cluster of $10$ eigenvalues with spacing $10^{-12}$. Thus our matrices have 10 clusters, each with 10 eigenvalues, with cluster diameter $O(10^{-12})$.

\smallskip
\emph{Observations:}
In Figure \ref{fig:cgclust} we plot the convergence of exact CG for the three problems. Accompanying CSD plots are given in Figure \ref{fig:specdensclust}. The matrix has in each case 10 very tight clusters of eigenvalues. When the clusters of eigenvalues are accumulated to the right (blue) and when they are the equally spaced (green), the relative error in the A-norm reaches in 10 iterations the level below $10^{-10}$. When the clusters are accumulated to the left (red), the relative error in the A-norm makes no progress in 10 iterations. This contradicts the widespread general claims about clustering of eigenvalues and CG convergence that do not take into account the positions of clusters.

\begin{figure}
\centering
\includegraphics[width=0.5\textwidth]{figures/exactcgclust-eps-converted-to.pdf}
\caption{The relative error in the $A$-norm versus iteration number for exact CG run on three problems with matrices having different distributions of 10 eigenvalue clusters, where each cluster contains 10 eigenvalues. }\label{fig:cgclust}
\end{figure}

\begin{figure}
\centering
\includegraphics[trim={4cm 3cm 4cm 3cm},clip,width=0.60\textwidth]{figures/specdensclustexact.pdf}
\caption{Cumulative spectral density plots for Figure \ref{fig:cgclust}.}\label{fig:specdensclust}
\end{figure}

\smallskip
\emph{Explanation:}
In 10 iterations the CG polynomials for all cases place a single Ritz value to  each cluster. For the clusters accumulated to the right as well as equally spaced this is sufficient for approximating the minimal polynomial (of degree 100) of the matrix {\em in the sense of \eqref{eqn:CG_Ritz_values}}. For the clusters accumulated to the left, placing one Ritz value in each cluster is not enough to decrease the CG error and the matrix minimal polynomial is in the same sense not well approximated,
despite the seemingly analogous position of Ritz values; see the CSD plots, where for $k=10$ the solid lines and the associated dashed lines graphically coincide. To achieve the desired decrease of the error, for the case of clusters accumulated to the left end of the spectrum, CG must place additional Ritz values in the rightmost clusters, which delays convergence. At iteration 15, the five rightmost clusters contain two Ritz values each, and the dashed line representing the CSD for $k=15$ departs from the CSD representing the matrix spectrum. If the computation  proceeds, then this departure would become more and more significant because more and more Ritz values will be placed in the rightmost clusters.

This mechanism has been demonstrated in \cite{GreStr92} and it was further thoroughly explained in \cite[Section~5.6]{LieStr13}. A very detailed account of the relationship between preconditioning and the clustering argument can be found in \cite[Section~3(c)]{CarStr20}.

\newpage
\subsection{Sensitivity of CG to rounding errors}\label{sec:fpcg}
\phantom{}
\smallskip
\begin{tcolorbox}
\emph{Main point: Particular eigenvalue distributions, specifically in cases of large outlying eigenvalues, cause CG convergence to be more susceptible to delay caused by finite precision errors. Convergence behavior for finite precision CG can be equated (up to an unimportant difference) with exact CG on a larger problem, whose eigenvalues are replaced by tight clusters.}
\end{tcolorbox}

\smallskip
\emph{Setup:} We redo the same experiment as in Section \ref{sec:cgeigdist} but now in finite (double) precision. We use the same three diagonal matrices and the same right-hand sides.
We plot the resulting CG convergence behavior on the left in Figure \ref{fig:fpeigdist} as well as the CSDs at certain iterations for each problem in Figure \ref{fig:specdens}.

Then, for each diagonal matrix, we create a larger matrix by replacing each eigenvalue with a tighter cluster of 4 eigenvalues. The spacing between eigenvalues in a cluster is $10^{-13}$.

\smallskip
\emph{Observations:} We run exact CG for these problems and display the resulting convergence curves on the right in Figure \ref{fig:fpeigdist}. Although for the matrix with eigenvalues accumulated to the left of the spectrum (red) CG converges in exact arithmetic faster than for the matrix with equally spaced eigenvalues (green) (see Figure \ref{fig:cgeigdist}), this problem is much more susceptible to the effects of rounding errors in finite precision computation, resulting in significantly slower convergence.  On the other hand, comparing with the previous experiment, the convergence trajectories of CG for the matrices with eigenvalues accumulated to the right(blue) and for equally distributed eigenvalues (green) are unchanged by finite precision error. The behavior of finite precision CG (left plot in Figure \ref{fig:fpeigdist}) is remarkably similar to the behavior of exact CG where the eigenvalues are replaced by tight clusters (right plot in Figure \ref{fig:fpeigdist}).

\begin{figure}
\centering
\includegraphics[width=0.475\textwidth]{figures/dpcgdist-eps-converted-to.pdf}
\includegraphics[width=0.475\textwidth]{figures/exactcgclustlikefp-eps-converted-to.pdf}
\caption{Left: The relative error in the $A$-norm versus iteration number for CG in finite precision run on three  problems with matrices having different eigenvalues distributions, corresponding to those in Figure \ref{fig:cgeigdist}. Right: The relative error in the $A$-norm versus iteration number for exact CG run on three  problems with matrices with the same eigenvalue distributions as the left plot, but with each eigenvalue replaced by a tight cluster of 4 eigenvalues. }\label{fig:fpeigdist}
\end{figure}

\begin{figure}
\centering
\includegraphics[trim={4cm 3cm 4cm 3cm},clip,width=0.60\textwidth]{figures/specdens.pdf}
\caption{Cumulative spectral density plots for Figure \ref{fig:fpeigdist}. Compare with Figure \ref{fig:specdensexact}.}\label{fig:specdens}
\end{figure}

\smallskip
\emph{Explanation:} There are two phenomena working against each other here. Whereas large outlying eigenvalues are desirable in exact arithmetic, they cause the problem to be more sensitive to rounding errors, which can result in convergence delay in finite precision. This phenomenon was investigated in \cite{Str91}, which was inspired by the earlier discussion by Jennings \cite{Jen77}, who related the convergence of CG to a polynomial curve fitting problem. It can be nicely viewed via the CSD plots (Figure \ref{fig:specdens}). While for the eigenvalues accumulated to the right and for equally distributed eigenvalues there is no observable difference between the plots generated by the exact and finite precision CG computations (compare the top and the bottom plots in Figures \ref{fig:fpeigdist} and \ref{fig:specdensexact}), the plots in these figures  associated with the eigenvalues accumulated to the left are remarkably different. There is no chance that for this problem  the CSD determined by the eigenvalues can  be closely approximated by the CSD generated by the Ritz values resulting from the finite precision CG computation. The plot in Figure \ref{fig:fpeigdist} shows that  with increasing iteration number more and more Ritz values have to be placed close to the rightmost outlying eigenvalues.

A theoretical explanation is provided by the work of Greenbaum~\cite{Gre89} that was further thoroughly illustrated in~\cite{GreStr92} and extended by Notay~\cite{Not93}. Additional Ritz values close to the large outlying eigenvalues have to appear in order to eliminate excessively large gradients of the CG approximation polynomials, which would otherwise occur in their neighborhood; see \cite[Section~5.9]{LieStr13} and \cite[Section~5]{MeuStr06}. For the accumulation of the eigenvalues to the right and for equally distributed eigenvalues the gradient of the CG approximation polynomial near all eigenvalues is sufficiently bounded without a need for placing additional Ritz values in their neighborhoods. This explains the numerical stability of CG for these problems; see also \cite[Section 3(b)(i)]{CarStr20}. It is also worth recalling the arguments in Section~\ref{sec:cgclust} above that deal with the mathematical behavior for problems with tight clusters of eigenvalues.

\subsection{Computational behavior of different CG algorithms}\label{sec:2vs3term}
\phantom{}
\smallskip
\begin{tcolorbox}
\emph{Main point: Rounding errors cause convergence delay and loss of attainable accuracy. The magnitude of these effects, and in the case of attainable accuracy, the mechanism, depends on the particular algorithm/implementation of CG.}
\end{tcolorbox}

\smallskip
\emph{Setup:}
We use a diagonal matrix $A$ as defined in \eqref{eq:diagmatrix} with $N=48$, $\lambda_1 = 0.1$, $\lambda_N=10^3$, and $\rho=0.25$. We use a right-hand side $b$ with equal components and unit $2$-norm. Additionally, we test two different algorithmic variants of CG: the variant of Hestenes and Stiefel \cite{HesSti52}, which uses three 2-term recurrences, and a different variant which uses two 3-term recurrences.

\smallskip
\emph{Observations:} The results comparing the relative $A$-norm of the error for both exact arithmetic and double precision are shown in Figure \ref{fig:fp2v3}. In exact arithmetic the $A$-norm error curves of the 2-term and 3-term variants are identical. In finite precision the convergence in both variants is delayed. The delay is slightly worse in the 3-term variant, and in addition the final accuracy level attained of this variant is over two orders of magnitude worse than the level attained by the 2-term variant.

\begin{figure}
\centering
\includegraphics[width=0.55\textwidth]{figures/twovthree-eps-converted-to.pdf}
\caption{The relative error in the $A$-norm versus iteration number (1) the variant of CG based on three 2-term recurrences in exact arithmetic (solid red) and double precision (dotted red), and (2) the variant of CG based on two 3-term recurrences in exact arithmetic (dashed blue) and double precision (dotted blue).}\label{fig:fp2v3}
\end{figure}

\smallskip
\emph{Explanation:}  The variant which uses 3-term recurrences exhibits a greater loss of accuracy than the variant which uses 2-term recurrences. Whereas in the 2-term recurrence variant, the loss of accuracy is caused by a simple accumulation of local rounding errors, in the 3-term recurrence variant these local rounding errors can be \emph{amplified}.
This behavior is analyzed together with its dependence on the initial residual in \cite{GutStr00}.

It should be cautioned that despite maintaining mathematical equivalence to Algorithm \ref{alg:hscg}, any algorithm that reorders computations or introduces auxiliary quantities can have different behavior computationally, i.e., in finite precision. Examples of CG algorithms designed for high-performance parallel environments include $s$-step (communication-avoiding) CG and pipelined CG, both of which are subject to potential amplification of local rounding errors and thus greater loss of accuracy than Algorithm \ref{alg:hscg}; see, e.g., \cite{Car15, CarRozStrTicTum18, CooYetAguGirVan18}.

\subsection{Residual versus Error and Stopping Criteria}\label{sec:resvserror}
\phantom{}
\smallskip
\begin{tcolorbox}
\emph{Main point: The residual $2$-norm is not a reliable indicator of the error in CG.}
\end{tcolorbox}

\smallskip
\emph{Setup:} We follow the construction given by Meurant in \cite{Meu20} for constructing linear systems such that the trajectories of the residual $2$-norm and the $A$-norm of the error in CG are prescribed. We construct two linear systems of size $N=20$. For the first, the residual norms $\Vert r_k\Vert_2$ oscillate between 1 and 2, and the errors are $\Vert e_0 \Vert_A = \Vert x-x_0\Vert_A =1$, $\Vert e_k\Vert_A = 0.4 \cdot \Vert e_{k-1}\Vert_A$  for $k=1,\ldots,N-1$. For the second, the residual norms are $\Vert r_0 \Vert_2 = 1$, $\Vert r_k\Vert_2 = 0.4\cdot \Vert r_{k-1}\Vert_2$ for $k=1,\ldots, N-1$, and the errors are $\Vert e_0 \Vert_A =  =1$, $\Vert e_k\Vert_A = 0.999 \cdot \Vert e_{k-1}\Vert_A$ for $k=1,\ldots,N-1$.

To construct the linear systems, following \cite{Meu20}, we set
\begin{align*}
\nu_k &= 1/\Vert r_k \Vert_2, \quad k=1,\ldots, N-1\\
\sigma_k &= \Vert e_k \Vert_A^2 / (\Vert r_k\Vert_2 \Vert r_0\Vert_2), \quad k=0,\ldots N-1,
\end{align*}
and then set
\[
L = \begin{bmatrix}
\sigma_0 & & & \\
\sigma_1 & \sigma_1 \nu_1 & & \\
\vdots & \vdots& \ddots & \\
\sigma_{N-1} & \sigma_{N-1} \nu_1 & \cdots & \sigma_{N-1}\nu_{N-1}
\end{bmatrix}.
\]
Our linear system to solve is then $A= (L+\hat{L}^T)^{-1}$ and $b=e_1$, where $\hat{L}$ is the strictly lower triangular part of $L$ and $e_1$ is the first column of the identity.

\smallskip
\emph{Observations:}
The convergence trajectories of the residual and the error for the two linear systems are shown in Figure \ref{fig:resvserror}. As expected, for the constructed linear systems CG  follows the prescribed convergence trajectories for both the residual norm and the error norm. For the first linear system (left plot in Figure \ref{fig:resvserror}), the residual completely stagnates, oscillating between 1 and 2, whereas the error  decreases linearly. For the second linear system (right plot in Figure \ref{fig:resvserror}), the error is decreasing very slowly, whereas the residual norm is decreasing relatively quickly.

\begin{figure}
\includegraphics[width=0.475\textwidth]{figures/resvserror1-eps-converted-to.pdf}
\includegraphics[width=0.475\textwidth]{figures/resvserror2-eps-converted-to.pdf}
\caption{Comparison of the convergence trajectories of the residual $2$-norm (solid blue) and the $A$-norm of the error in CG (dashed red) for two linear systems. }\label{fig:resvserror}
\end{figure}

\smallskip
\emph{Explanation:}
Hestenes and Stiefel comment in their original paper \cite[Section 18]{HesSti52} that for \emph{any} prescribed sequence of residual $2$-norms, there exists a symmetric positive definite matrix $A$ and right-hand side $b$ such that CG exhibits the prescribed convergence behavior. Thus, in general circumstances, one cannot equate a small residual norm with a small error. The converse also does not hold: a large residual does not imply a large error. Note that because the convergence of CG is linked with the distribution of the eigenvalues of $A$, it is not possible to also  simultaneously prescribe the eigenvalues of $A$. This is in contrast to the GMRES method; see Example~\ref{sec:GMRESevery}.

This behavior opens the question: what should we use for stopping criteria in CG? Many implementations use the relative (recursively computed) residual norm. Although we have just seen that this is not a reliable stopping criterion, it is nonetheless inexpensive to compute, since we already have access to the recursively computed residual produced in each iteration of CG. In practice, we of course do not know the true solution $x$, and thus we can not compute the exact error $e_k$. This was commented on already by Hestenes and Stiefel, who gave formulas for estimating the error \cite[Section 4]{HesSti52}. Much research has focused on developing reliable error norm estimation and associated stopping criteria for CG; see, e.g., \cite{GolMeu97, GolStr94, MeuPapTic21, StrTic02, StrTic05}.

\subsection{CG with ``average'' initial residuals}\label{sec:cgavgres}
\phantom{}
\smallskip
\begin{tcolorbox}
\emph{Main point: The terms ``average'' and ``random'' have very specific meanings and should be used carefully along with a complete specification of assumptions. This is important in particular when one subsequently draws conclusions about CG convergence behavior for more general problems.}
\end{tcolorbox}

\smallskip
\emph{Setup:} We test the behavior of double precision preconditioned CG (in terms of relative $A$-norm of the error) with particular and random right-hand sides $b$, which, for the choice of zero initial approximation solution, results in random initial residuals $r_0=b$.
The matrix $A$ and the particular right-hand side $b_*$ come from the  discretization of a boundary value problem with Dirichlet boundary conditions, taken from \cite[Section 5.3]{MorNocSie02}, with a detailed analysis of the preconditioned CG convergence in \cite{GerMarNieStr19}. The linear system resulting from the standard finite element discretization using the standard uniform triangulation has $N=3969$. We test two preconditioners: Laplace operator preconditioning and an algebraic incomplete Cholesky factorization of $A$ with drop tolerance $10^{-2}$.
For each preconditioning setting, we run preconditioned CG in double precision with 100 random right-hand sides, generated via MATLAB's \texttt{randn} function and then normalized, as well as the normalized particular right-hand side $b_*$.

\smallskip
\emph{Observations:} In Figure \ref{fig:randx0}, we see that there is some slight variability in the convergence trajectories for CG with different random initial residuals. The range of convergence behavior is wider for the case of incomplete Cholesky preconditioning, where the convergence trajectories of the random initial residuals are somewhat indicative of the behavior for the particular initial residual from the PDE model problem. This is not the case for the Laplace operator preconditioning, both in early iterations and after the convergence accelerates due to CG adaptation.

\begin{figure}
\centering
\includegraphics[width=0.55\textwidth]{figures/randx0_new-eps-converted-to.pdf}
\caption{The relative error in the $A$-norm versus iteration number for preconditioned CG run on a boundary value problem, using  Laplace operator preconditioning with 100 random initial residuals (dotted red) and one particular initial residual (dash-dotted black), and using algebraic incomplete Cholesky preconditioning with 100 random initial residuals (dotted blue) and one particular initial residual (dashed black). }\label{fig:randx0}
\end{figure}

\smallskip
\emph{Explanation:}
It may seem from \eqref{eqn:CGmin} that, apart from some artificially constructed, unrealistic initial residuals (right-hand sides), an ``average'' or ``random'' initial residual will give enough information about the behavior of CG on real-world problems. But, one must be very careful, because a superficial look at \eqref{eqn:CGmin}, which contains values of the highly oscillating (orthogonal) optimal CG polynomial giving the minimum can be easily misleading. A very detailed quantitative explanation in \cite{GerMarNieStr19} addresses the matter in full clarity. Here, we did not look for an example that would show the largest discrepancy. We simply used the example that has already been mentioned in Section \ref{sec:pcg} on inadequate use of condition number bounds.

As pointed out by Greenbaum and Trefethen in the context of the Arnoldi and the GMRES methods, taking the upper bound ``disentangles the matrix essence of the process from the distracting effects of the initial vector''~\cite[p.~361]{GreTre94}. Since in the nonsymmetric case one cannot rely, in general, on the spectral decomposition based formula analogous to \eqref{eqn:CGmin}, the dependence of GMRES convergence behavior on the initial residual can be much more dramatic; see Section~\ref{sec:GMRESinitres} below.

We note that in the literature on randomized numerical linear algebra, a common approach is termed ``randomized Lanczos''; see, e.g., the recent survey \cite{MarTro20}. Here, the issue is different; randomized Lanczos refers to running the Lanczos algorithm with a random starting vector, which can be used, for example, for estimating the spectral norm of a matrix. The rationale behind using a random starting vector is that one could choose an unfavorable starting vector for obtaining a good approximation of the largest eigenvalue, namely, an initial vector which is orthogonal to the largest eigenvector. The choice of a random starting vector avoids this problem with high probability; see \cite{KucWoz92}.

\newpage
\subsection{The trajectory of finite precision CG computations}\label{sec:cgtraj}
\phantom{}
\smallskip
\begin{tcolorbox}
\emph{Main point:
The approximate solutions produced by finite precision CG can be mapped to those produced by exact CG via a mapping defined by the rank deficiency of the Krylov subspace basis. It seems that the trajectory of the approximate solutions produced by finite precision CG remain in a narrow ``tunnel'' around those produced by exact CG.}
\end{tcolorbox}

\smallskip
\emph{Setup:} We generate a diagonal matrix $A$ as defined in \eqref{eq:diagmatrix} with $N=35$, $\lambda_1 = 0.1$, $\lambda_N=10^2$, and $\rho=0.65$. We use a right-hand side $b$ with equal components and unit $2$-norm. We run CG in double precision in two settings: first, with reorthogonalization, which mimics the convergence of exact CG, and second, without reorthogonalization (standard CG).

We then shift the finite precision CG iterates as follows. We define the sequence
\begin{equation}
\ell(k) = \max\left\{i\hspace{2pt} \vert \hspace{2pt}\text{rank}(\mathcal{K}_i(A,r_0))=k \right\}, \quad \ell=1,2,\ldots,
\label{eq:mapping}
\end{equation}
for the (inexact) Krylov subspace $\mathcal{K}_i(A,r_0)$ computed in double precision. To compute the rank, we use the built-in MATLAB function \texttt{rank} with threshold set to $10^{-1}$; compare with \cite[Section 5.9.1]{LieStr13}, in particular, Figure 5.16. For ``exact'' CG iterates $x_k$ and finite precision iterates $\bar{x}_{\ell(k)}$, we measure the ratios
\begin{equation}
    \frac{\Vert x- \bar{x}_{\ell(k)}\Vert_A}{\Vert x-x_k \Vert_A}
    \label{eq:ratio1}
\end{equation}
and
\begin{equation}
   \left\vert 1- \frac{\Vert x- \bar{x}_{\ell(k)}\Vert_A}{\Vert x-x_k \Vert_A} \right\vert.
    \label{eq:ratio2}
\end{equation}

\smallskip
\emph{Observations:} In Figure \ref{fig:cgtraj}, we plot the relative error in the $A$-norm versus iteration number of the exact CG iterates $x_k$ (solid blue), the relative error in the $A$-norm versus iteration number for the shifted finite precision CG iterates $\bar{x}_{\ell(k)}$ (blue circles), the ratio \eqref{eq:ratio1} (dotted red) and the ratio \eqref{eq:ratio2} (dashed red). We see that using the mapping \eqref{eq:mapping}, the finite precision CG iterates match well with the exact CG iterates. The ratio \eqref{eq:ratio1} stays close to 1 throughout the computation. The ratio \eqref{eq:ratio2} starts close to machine precision and grows as the iteration proceeds, but stays below 1.

\begin{figure}
\centering
\includegraphics[width=0.55\textwidth]{figures/cgtraj-eps-converted-to.pdf}
\caption{The relative error in the $A$-norm versus iteration number for exact CG (solid blue) and shifted double precision CG (blue circles), the ratio \eqref{eq:ratio1} (dotted red) and the ratio \eqref{eq:ratio2} (dashed red).}\label{fig:cgtraj}
\end{figure}

\smallskip
\emph{Explanation:}
That the quantity \eqref{eq:ratio1} remains close to one means that the convergence trajectories for exact CG and the shifted finite precision CG are close to being identical. This means that finite precision CG computations follow exactly the trajectory of exact CG computations, but with progress delayed according to the rank deficiency of the computed Krylov subspaces. The ratio \eqref{eq:ratio2} tells us how far \eqref{eq:ratio1} is from one. Rewriting \eqref{eq:ratio2}, we can say that if
\begin{equation}
\left\vert \frac{\Vert x-x_k\Vert_A - \Vert x- \bar{x}_{\ell(k)}\Vert_A}{\Vert x-x_k\Vert_A} \right\vert \ll 1,
\label{eq:tunnel}
\end{equation}
then we can consider the trajectory of finite precision CG iterates to be enclosed within a narrow tunnel around the trajectory of exact CG iterates, where the diameter of the tunnel is given by the left-hand side of \eqref{eq:tunnel}. Although \eqref{eq:tunnel} holds for this particular example, it has not been shown to hold for finite precision computations in general. We strongly suspect that such a result holds; initial investigations have been carried out in, e.g., \cite{Ger13} and \cite{GerHneKub17}. The analysis is complicated the the fact that in finite precision computations, we can not easily compare the exact Krylov subspaces with the computed subspaces.

\section{The GMRES Method}\label{sec:GMRES}

The GMRES method~\cite{SaaSch86} is well defined for any linear algebraic system $Ax=b$ with a nonsingular matrix~$A\in\mathbb{R}^{N\times N}$ and right-hand side $b\in\mathbb{R}^{N}$. If $d=d(A,r_0)$ is the grade of $r_0=b-Ax_0$ with respect to $A$, then at every step $k=1,2,\dots,d$ the GMRES method constructs a uniquely determined approximation $x_k\in x_0+{\cal K}_k(A,r_0)$ such that $r_k\perp A{\cal K}_k(A,r_0)$, or equivalently
\begin{equation}\label{eqn:GMRES}
\|r_k\|_2=\min_{z\in x_0+{\cal K}_k(A,r_0)}\|b-Az\|_2=\min_{p\in P_k(0)}\|p(A)r_0\|_2.
\end{equation}
GMRES is implemented using the Arnoldi algorithm~\cite{Arn51}, usually in the modified Gram-Schmidt (MGS) variant, which generates an orthonormal basis $V_k$ of ${\cal K}_k(A,r_0)$ and a matrix decomposition of the form $AV_k=V_{k+1}H_{k+1,k}$, where $H_{k+1,k}\in\mathbb{R}^{k+1,k}$ is an unreduced upper Hessenberg matrix. Then $x_k=x_0+V_kt_k$ is determined by
\begin{equation}\label{eqn:tk}
t_k=\argmin_{t\in\mathbb{R}^k}\|b-A(x_0+V_kt)\|_2=\argmin_{t\in\mathbb{R}^k} \|\|r_0\|e_1-H_{k+1,k}t\|_2.
\end{equation}
In practical implementations of GMRES, the least squares problem on the right is solved by computing the QR decomposition of the matrix $H_{k+1,k}$. Because of the upper Hessenberg structure, this decomposition can be obtained using Givens rotations, which can be updated in every step. This process also yields an update of the value $\|r_k\|_2$ without explicitly computing $x_k$. Thus, in practical implementations of GMRES, the least squares problem is solved only when the updated value of the residual norm is below a given tolerance. While all of this is very efficient, the Arnoldi algorithm for a general nonsymmetric matrix requires full recurrences (unlike the short recurrences in CG), and hence the computational cost in terms of work and storage requirements of GMRES grows. A pseudocode implementation of GMRES is shown in Algorithm~\ref{alg:GMRES}, and more details about the implementation of GMRES can be found, for example, in~\cite[Section~2.5.5]{LieStr13} and \cite{SaaSch86}. A detailed analysis why the computation of orthogonal Krylov subspace bases for general nonsymmetric matrices in general requires full instead of short recurrences is given in~\cite[Chapter~4]{LieStr13}.

\begin{algorithm
    \caption{GMRES method (pseudocode) \label{alg:GMRES}}
    \begin{algorithmic}[1]
        \Require{Nonsingular matrix $A\in \mathbb{R}^{N\times N}$, right-hand side $b$, initial approximation $x_0$; convergence tolerance $\tau$; maximum number of iterations $n_{\max}$.}
        \State{$r_0 = b - A x_0$}
        \For{$k=1,2,\dots,n_{\max}$}
            \State{Compute step $k$ of the Arnoldi algorithm to obtain $AV_k=V_{k+1}{H}_{k+1,k}$.}
            \State{Update the QR factorisation of ${H}_{k+1,k}$ and compute the updated $\|r_k\|_2$.}
            \State{If $\|r_k\|_2\leq\tau$, then compute $t_k$ in~\eqref{eqn:tk}, return $x_k=x_0+V_kt_k$, and stop.}
        \EndFor
    \end{algorithmic}
\end{algorithm}

If $A$ is diagonalizable, $A=X\Lambda X^{-1}$ with $\Lambda={\rm diag}(\lambda_1,\dots,\lambda_N)$, then
\begin{align}
\|r_k\|_2 &= \min_{p\in P_k(0)}\|p(A)r_0\|_2 \,
= \min_{p\in P_k(0)}\|Xp(\Lambda)X^{-1}r_0\|_2
\label{eqn:GMRESnorm}\\
&\leq \kappa(X)\,\|r_0\|_2\,\min_{p\in P_k(0)}\max_{1\leq i\leq N} |p(\lambda_i)|.\label{eqn:GMRESbd}
\end{align}

If $A$ is normal we can choose an eigenvector matrix with $\kappa(X)=1$, and it can be shown that the bound \eqref{eqn:GMRESbd} is sharp in the sense that for each step~$k$ there exists an initial residual $r_0$ (depending on $A$ and $k$) so that equality holds; see the original proofs in~\cite{GreGur94,Jou94} as well as~\cite{LieTic14}. This means that for a normal (or close to normal) matrix $A$ the location of its eigenvalues determines the worst-case behavior of GMRES and, in this worst-case sense, gives an indication of the possible actual behavior. For an analysis of the mathematical properties of worst-case GMRES (for general nonsingular matrices $A$) we refer to~\cite{FabLieTic13}.

Note that the eigenvalues of a general nonsingular matrix~$A$ may be anywhere in the complex plane, so that estimating the value of the polynomial min-max approximation problem in \eqref{eqn:GMRESbd} in general can be very challenging. A quantitative bound can be given in the (simple) case that the eigenvalues of $A$ are contained in a disk centered at $c\in\mathbb{C}$ and with radius $\rho>0$, where $\rho<|c|$ is necessary so that zero is outside the disk. Taking the polynomial $(1-z/c)^k\in P_k(0)$ then shows that
$$\min_{p\in P_k(0)}\max_{1\leq i\leq N} |p(\lambda_i)|\leq \left(\frac{\rho}{|c|}\right)^k;$$
see, e.g.,~\cite[Section~6.11]{Saa03}.
Thus, we can expect that GMRES converges quickly when $\kappa(X)$ is small, and the eigenvalues of $A$ are contained in a small disk that is far away from the origin in the complex plane; see Section~\ref{sec:GMRESnormal} for examples. A survey of approaches for estimating the value of the min-max approximation problem beyond this special case is given in~\cite[Sections~5.7.2--5.7.3]{LieStr13}.

It needs to be stressed that the sharpness of the bound \eqref{eqn:GMRESbd} for normal matrices does not imply that GMRES converges faster for normal matrices, or that the (departure from) normality has an easy analyzable effect on the convergence of GMRES. In fact, it can be shown that GMRES may exhibit a complete stagnation even for unitary and hence normal matrices; see~\cite{GreStr94}. On the other hand, for a nonnormal matrix the location of the eigenvalues and hence the value of the min-max approximation problem in \eqref{eqn:GMRESbd} may not give relevant information about convergence behavior of GMRES. If $A$ is not diagonalizable, then its spectral decomposition does not exist, and an analogue of~\eqref{eqn:GMRESbd} based on the Jordan canonical form is of very limited use. As shown in~\cite{GrePtaStr96,GreStr94}, any nonincreasing convergence curve is possible for GMRES for a (in general, nonnormal) matrix $A$ having any prescribed set of eigenvalues. This result will be illustrated in Section~\ref{sec:GMRESevery}. The work in~\cite{AriPtaStr98} gives a parametrization of the set of {\em all matrices and right-hand-sides} such that GMRES provides a given convergence curve while the matrix has a prescribed spectrum. See~\cite[Section~5.7.4]{LieStr13} for a summary.

\subsection{Any nonincreasing GMRES convergence curve is possible for any eigenvalues}\label{sec:GMRESevery}
\phantom{}
\smallskip
\begin{tcolorbox}
\emph{Main point: Eigenvalues alone are in general not sufficient for describing the GMRES convergence behavior.}
\end{tcolorbox}

\smallskip
\emph{Setup:} We follow~\cite[Section 2]{GrePtaStr96} for constructing a linear algebraic system $Ax=b$, where $A\in\mathbb{R}^{N\times N}$ has a prescribed set of eigenvalues $\lambda_1,\dots,\lambda_N\in\mathbb{C}$, so that the residual norms of GMRES applied to this system with $x_0=0$ are given by a prescribed nonincreasing sequence $f_0\geq f_1\geq \cdots \geq f_{N-1} >f_N=0$.

Define $g_j=\sqrt{(f_{j-1})^2 - (f_j)^2}$ for $j=1,\ldots, N$, let $V\in\mathbb{R}^{N\times N}$ be any orthogonal matrix, and let $b=V[g_1,\ldots, g_N]^T$. We then construct the polynomial
\[
(z-\lambda_1)(z-\lambda_2)\cdots(z-\lambda_N)=z^n - \sum_{j=0}^{N-1} \alpha_j z^j,
\]
and its companion matrix
\[
A^B = \begin{bmatrix}
0 & \cdots & 0 & \alpha_0 \\
1 & & 0 & \alpha_1 \\
& \ddots & \vdots& \vdots \\
& & 1 & \alpha_{N-1}
\end{bmatrix}.
\]
With $B=[b,v_1,\ldots, v_{N-1}]$, where $v_j$ denotes the $j$th column of $V$, we set $A = BA^BB^{-1}$.

We use $N=21$ and consider two scenarios: In the first we prescribe the eigenvalues $\lambda_1=\cdots=\lambda_{N}=1$, and the convergence curve $f_1=\cdots=f_{N}=1>f_{N+1}=0$. In the second we prescribe the eigenvalues $\lambda_j=j$ for $j=1,\dots,N$, and a convergence curve that starts at $f_1=1$, and then decreases every 4 steps through $10^{-2}$, $10^{-4}$, and $10^{-6}$, to $10^{-8}$. In both cases we take $V=I$, and apply GMRES to $Ax=b$ with $x_0$.

\smallskip
\emph{Observations:} The residual norms computed by applying
MATLAB's {\tt gmres} function to the two linear algebraic systems
described above are shown by the solid blue and dashed red curves in
Figure \ref{fig:gmresconv}. As expected, they follow the prescribed
convergence curves.

\begin{figure}
\begin{center}
\includegraphics[width=0.475\textwidth]{figures/gmres_any_curve-eps-converted-to.pdf}
\caption{Residual norms of GMRES (solid blue and dashed red) applied to the two linear algebraic systems constructed with prescribed eigenvalues and convergence curves.}\label{fig:gmresconv}
\end{center}
\end{figure}

\smallskip
\emph{Explanation:} The convergence curves illustrate the proven theorems from~\cite{AriPtaStr98,GrePtaStr96,GreStr94}, and hence are certainly not surprising. It is important to realize that the two distinctly different types of GMRES convergence behavior can by no means be related to the eigenvalue distributions of the two matrices. Both are highly nonnormal, and the bound \eqref{eqn:GMRESbd} is useless for obtaining an insight into the convergence behavior. Computations with MATLAB's {\tt eig} function yield  eigenvector matrices with $\kappa(X)\approx 7.6\times 10^{10}$ and $\kappa(X)\approx 1.5\times 10^{21}$, respectively. It is worth recalling a fact that seems to remain unnoticed. The complete parametrization provided in~\cite{AriPtaStr98} contains, besides artificially constructed examples, also all matrices and right hand sides that arise in solving practical problems.\\

\subsection{GMRES convergence for normal matrices}\label{sec:GMRESnormal}
\phantom{}
\smallskip
\begin{tcolorbox}
\emph{Main point: For normal matrices, the eigenvalues can give reasonably descriptive information about the convergence behavior of GMRES with ``average'' initial residuals.}
\end{tcolorbox}

\smallskip
\emph{Setup:} We consider nonsymmetric matrices $A_N\in\mathbb{R}^{N\times N}$ with normally distributed random entries that are generated with {\tt randn} in MATLAB. We compute the eigenvalues of these matrices in MATLAB and form diagonal (and hence normal) matrices $D_N$ with these eigenvalues. We then apply GMRES with $x_0=0$ to linear algebraic systems with the three matrices $D_N/\sqrt{N}$, $I+D_N/\sqrt{N}$ and $2I+D_N/(2\sqrt{N})$. The right-hand sides of the linear algebraic systems are normalized random vectors, also generated with {\tt randn} in MATLAB. We use $N=100$, and we repeat the computation 100 times.

\smallskip
\emph{Observations:} The top left part of Figure~\ref{fig:GMRESrandn} shows the eigenvalues of the 100 matrices $D_N/\sqrt{N}$ and the boundary of the unit disk, the top right part of Figure~\ref{fig:GMRESrandn} shows the eigenvalues of the 100 matrices $I+D_N/\sqrt{N}$ and the boundary of the disk centered at $1$ and with radius~$1$, and the bottom left part of Figure~\ref{fig:GMRESrandn} shows the eigenvalues of the 100 matrices $2I+D_N/(2\sqrt{N})$ and the boundary of the disk centered at $2$ and with radius~$1/2$.

We observe three distinctly different types of GMRES convergence behavior in the bottom right part of Figure~\ref{fig:GMRESrandn}:
\begin{itemize}
\item For the matrices $D_N/\sqrt{N}$ the GMRES method makes almost no progress until the very end of the iteration.
\item For the matrices $I+D_N/\sqrt{N}$ the GMRES method initially converges slowly, followed by a phase of faster convergence. In this case the different convergence curves have the largest variations.
\item For the matrices $2I+D_N/(2\sqrt{N})$ the GMRES method converges linearly and rather quickly throughout the iteration.
\end{itemize}

\begin{figure}
\begin{center}
\includegraphics[width=0.475\textwidth]{figures/evals_rand_normal_sqr-eps-converted-to.pdf}\hfill
\includegraphics[width=0.475\textwidth]{figures/evals_rand_normal_sqrt_p1-eps-converted-to.pdf}\\
\includegraphics[width=0.475\textwidth]{figures/evals_rand_normal_2_p1-eps-converted-to.pdf}\hfill
\includegraphics[width=0.475\textwidth]{figures/gmres_rand_normal-eps-converted-to.pdf}
\caption{Eigenvalues of matrices $D_N/\sqrt{N}$, $I+D_N/\sqrt{N}$, and $2I+A_N/(2\sqrt{N})$ (top left, top right, and bottom left), and relative GMRES residual norms for linear algebraic systems with these matrices (bottom right).}\label{fig:GMRESrandn}
\end{center}
\end{figure}

\smallskip
\emph{Explanation:} According to Girko's Circular Law, the eigenvalues of the matrices $A_N/\sqrt{N}$ become uniformly distributed in the unit disk for $N\rightarrow\infty$; see the Introduction of~\cite{Ede97} for a summary of results in this context. This explains the eigenvalue distributions that are shown in Figure~\ref{fig:GMRESrandn}.

 All matrices used in this experiment are normal. Therefore the location of the eigenvalues is descriptive for the convergence behavior of GMRES in the worst case as well as with ``average" initial residuals, and the bound  \eqref{eqn:GMRESbd} can give descriptive information. Since the GMRES iteration polynomials are normalized at the origin, it is clear that GMRES can make almost no progress when the eigenvalues are uniformly distributed in the unit disk, which applies to the matrices $D_N/\sqrt{N}$. The fast convergence speed for the matrices $2I+D_N/(2\sqrt{N})$ is explained by the fact that the eigenvalues are in a rather small disk that is far away from the origin. For the matrices $I+D_N/\sqrt{N}$ the location of the eigenvalues suggests that the convergence speed of GMRES is in between the two other cases, and this is what we observe in the experiment. An argument using an inclusion disk or any other ``simple'' compact inclusion set for the eigenvalues is formally not applicable, however, since the boundary of such a set would contain the origin. The analysis of the GMRES convergence in such a case becomes intricate, because it involves a challenging polynomial min-max approximation problem in the complex plane.

We point out that similar matrices have been used in attempts to present evidence for ``universality'' in numerical computations with random data; see, e.g.,~\cite{DeiTro19,DeiMenOlvTro14}. As shown by this experiment, the matrices $A_N$ and $D_N$ are rather special in the sense that their eigenvalues become uniformly distributed in a disk for $N\rightarrow\infty$. The convergence behavior of GMRES in the worst case or with ``average'' initial residuals therefore strongly depends on how these matrices are
scaled and shifted. We stress that analogously to the behavior of CG described in Section~\ref{sec:cgavgres}, for normal matrices and particular initial residuals arising from practical problems, the behavior of GMRES can be different from the behavior with ``average" initial residuals.

\subsection{MGS-GMRES is normwise backward stable}\label{sec:GMRESorth}
\phantom{}
\smallskip
\begin{tcolorbox}
\emph{Main point: In MGS-GMRES, complete loss of orthogonality of the computed Krylov subspace basis means convergence of the normwise relative backward error to the maximal attainable accuracy, and vice versa. Therefore, the MGS-GMRES method is normwise backward stable.}
\end{tcolorbox}

\smallskip
\emph{Setup:} The experiment investigates the relation between the loss of orthogonality of the Krylov subspace basis vectors computed in finite precision arithmetic using the modified Gram-Schmidt (MGS) variant of the Arnoldi algorithm (measured by $\|I-V_k^TV_k\|_F$), and the convergence of the normwise backward error  $\|b-Ax_k\|_2/(\|b\|_2+\|A\|_2\|x_k\|_2$ in the corresponding MGS-GMRES method. We consider the example matrices {\tt fs1836} and {\tt sherman2} from Matrix Market\footnote{\url{https://math.nist.gov/MatrixMarket/}}. The matrix {\tt fs1836} is of size $N=183$ and has a condition number of approximately $1.0\times 10^7$. The matrix {\tt sherman2} is of size $N=1080$ and has a condition number of approximately $2.4\times 10^7$. Both matrices are diagonalzable, and the condition number of their eigenvector matrices computed by MATLAB are approximately $1.7\times 10^{11}$ and $9.6\times 10^{11}$, respectively. We use the right-hand side $b=Ax$, where $x=[1,\dots,1]^T/\sqrt{N}$, and the initial vector $x_0=0$.

\smallskip
\emph{Observations:} As shown in Figure~\ref{fig:bwerr_loss}, throughout the iteration the product
$$\|I-V_k^TV_k\|_F\,\times\,\frac{\|b-Ax_k\|_2}{\|b\|_2+\|A\|_2\|x_k\|_2}$$
is almost constant, and close to the machine precision (approximately $10^{-16}$). The orthogonality of the basis vectors is completely lost only when the normwise backward error of the MGS-GMRES iteration has reached its maximal attainable accuracy level. We point out that this is a significant difference to the finite precision behavior of CG and other methods based on short recurrences, where not only a loss of orthogonality, but also a loss of rank in the computed subspace may occur, which leads to a
delay of convergence; see Section~\ref{sec:fpcg} and the detailed
explanations in, e.g., \cite[Section~5.9]{LieStr13} and \cite{MeuStr06}.
We point out again that the CG convergence behavior for a specific model problem can not be generalized unless there is a convincing argument for such a generalization; see the example in Section \ref{sec:cgpoisson}.

\begin{figure}
\begin{center}
\includegraphics[width=0.475\textwidth]{figures/bwerr_loss_fs183_6-eps-converted-to.pdf} \hfill
\includegraphics[width=0.475\textwidth]{figures/bwerr_loss_sherman2-eps-converted-to.pdf}
\caption{MGS-GMRES normwise backward errors (solid blue) and loss of orthogonality (dashed red), and the product of the two (dotted) for linear algebraic systems with the matrices {\tt fs1836} (left) and {\tt sherman2} (right).}\label{fig:bwerr_loss}
\end{center}
\end{figure}

\smallskip
\emph{Explanation:} The full explanation of the phenomenon observed in Figure~\ref{fig:bwerr_loss} is based on a detailed analysis of rounding errors that occur in MGS-GMRES; see in particular the original papers~\cite{DrkGreRozStr95,PaiStr02,PaiRozStr06} or the summary in~\cite[Section~5.10]{LieStr13}. In this analysis it is shown (under some technical assumptions, e.g., that $A$ is not too close to being singular) that the loss of orthogonality in the Krylov subspace basis $V_k$ computed in finite precision arithmetic using the MGS variant of the Arnoldi algorithm is essentially controlled by the condition number $\kappa([\gamma v_1,AV_kD_k])$, where $\gamma\in\mathbb{R}$ and $D_k\in\mathbb{R}^{k\times k}$ are suitable chosen scalings. Since $r_k=r_0-AV_kt_k=\|r_0\|v_1-AV_kt_k$, the conditioning of the matrix $[\gamma v_1,AV_kD_k]$ can be related to the residual norm $\|b-Ax_k\|_2$, and in a second step also to the normwise backward error $\|b-Ax_k\|_2/(\|b\|_2+\|A\|_2\|x_k\|_2)$. This yields a rigorous mathematical proof of the numerically observed behavior of MGS-GMRES. It is worth noting that working on this challenge lead to revisiting (scaled) total least squares problems; see, for example, \cite{PaiStr02a}.

Note that the (more costly) GMRES implementation based on Householder orthogonalization in the Arnoldi algorithm is also normwise backward stable~\cite{DrkGreRozStr95}. In the classical Gram-Schmidt orthogonalization the orthogonality is lost too quickly to guarantee backward stability of the corresponding GMRES implementation; see~\cite{GirLanRozEsh05} for a rounding error analysis of classical Gram-Schmidt.

\subsection{GMRES convergence for approximately computed preconditioning}\label{sec:pGMRES}
\phantom{}
\smallskip
\begin{tcolorbox}
\emph{Main point: In practical finite precision computations, preconditioning can not be performed exactly. Therefore theoretical results, which hold in exact arithmetic, have to be used with caution when applied to practical heuristics.}
\end{tcolorbox}

\newpage
\emph{Setup:} We set up a linear algebraic system ${\cal A}x=b$, where $b$ is a normalized vector of ones, and
$${\cal A}=\begin{bmatrix}A & B^T\\B & 0\end{bmatrix}$$
comes from a discretization of a Navier-Stokes model problem in IFISS 3.6~\cite{ElmRamSil14}. We run the {\tt navier\_testproblem} with the (mostly default) parameters: cavity; regularized; 16x16 grid; uniform grid; Q1-P0; viscosity: 1/100; hybrid; Picard: 1; Newton: 1; nonlinear tolerance: 1.1*eps; uniform streamlines. The matrix $A\in\mathbb{R}^{n\times n}$ is nonsymmetric, and $B\in\mathbb{R}^{m\times n}$ has full rank $m$. For our chosen model problem parameters we have $n=578$ and $m=256$. We apply GMRES with $x_0=0$ to ${\cal A}x=b$.

We consider the block diagonal preconditioner
$${\cal P}=\begin{bmatrix}A & 0\\0 & S\end{bmatrix},$$
where $S=BA^{-1}B^T$ is the Schur complement, and we apply GMRES with $x_0=0$ to the preconditioned system ${\cal P}^{-1}{\cal A}x={\cal P}^{-1}b$. Formally, each GMRES iteration step requires one multiplication with ${\cal P}^{-1}$. This multiplication is performed by solving a linear algebraic system with ${\cal P}$, and in this experiment we study how different tolerances for these ``inner solves'' impact the convergence of GMRES. The top block of the preconditioner ${\cal P}$ is given by the explicitly known matrix $A$. We compute the bottom block $S$ using MATLAB's backslash operator for the inversion of~$A$. We perform the ``inner solves'' with GMRES starting with $x_0=0$, and we stop the iteration when the relative residual norm reaches the respective tolerances $10^{-4}$, $10^{-3}$, $10^{-2}$, $10^{-1}$.

\smallskip
\emph{Observations:} Figure~\ref{fig:gmres_navier} shows the relative residual norms of GMRES applied to ${\cal A}x=b$ (solid blue) and the relative residual norms of GMRES applied to the approximately preconditioned system with the four different tolerances for the ``inner solves'' (dashed red). We see that solving the systems with ${\cal P}$ more accurately leads to a faster convergence of GMRES for the preconditioned system (measured by the preconditioned relative residual norm).

\begin{figure}
\begin{center}
\includegraphics[width=0.475\textwidth]{figures/gmres_navier_b_ones-eps-converted-to.pdf}
\caption{GMRES convergence for a Navier-Stokes test problem: Unpreconditioned system (solid blue) and approximately preconditioned system with ``inner solve'' tolerances $10^{-4}$, $10^{-3}$, $10^{-2}$, and $10^{-1}$ (dashed red). GMRES convergences faster when the ``inner solves'' are performed more exactly.}\label{fig:gmres_navier}
\end{center}
\end{figure}

\smallskip
\emph{Explanation:} As shown in the widely cited paper~\cite{MurGolWat00}, the minimal polynomial of the (nonsingular) preconditioned matrix ${\cal P}^{-1}{\cal A}$ is given by $(z-1)(z^2-z-1)$, and hence this matrix has the three distinct eigenvalues $1$ and $(1\pm \sqrt{5})/2$. Thus, in exact arithmetic, GMRES applied to the exactly preconditioned system ${\cal P}^{-1}{\cal A}x={\cal P}^{-1}b$ converges to the exact solution in at most three steps. Note that, as clearly pointed out in~\cite{MurGolWat00}, the degree of the minimal polynomial is essential for this property. In general, for convergence of GMRES to the exact solution in at most~$k$ steps (in exact arithmetic), it is not sufficient that~$A$ has only~$k$ distinct eigenvalues. The matrix additionally must be diagonalizable; cf. the example with $\lambda_1=\cdots=\lambda_N=1$ in Section~\ref{sec:GMRESevery}.

In practical computations we usually do not form a preconditioned matrix, but instead use ``inner solves'' for the linear algebraic systems with ${\cal P}$. In addition, these ``inner solves'' are usually based only on an approximation of ${\cal P}$ which is obtained, for example, by approximating a Schur complement. Clearly, the exact mathematical properties of the preconditioned matrix \emph{do no longer hold} for the approximate preconditioning. While it is tempting to come up with handwaving arguments for the impact of some ``perturbation'' introduced by the approximate preconditioning, it is in fact very difficult to rigorously justify any perturbation argument that would link the actual GMRES convergence behavior to the theoretical result about the minimal polynomial and the eigenvalues of the (exact) matrix ${\cal P}^{-1}{\cal A}$. Even if such an argument would be available, it would hardly be applicable to nonnormal matrices with eigenvalues that are highly sensitive to perturbations. In addition, there are no guarantees that the (hypothetical) approximately preconditioned matrix still is diagonalizable, and that we can base at least a heuristic for the GMRES convergence behavior on the spectrum only; cf. Section~\ref{sec:GMRESevery}.

Finding a mathematically sound explanation of the GMRES convergence behavior under approximate preconditioning is a challenging open problem. An analysis of this issue requires a rigorous approach rather than handwaving heuristics that operate with objects which are in practical computations not well defined. In particular cases, where the theoretical results on ${\cal P}^{-1}{\cal A}$ indeed explain the results of practical computations, one should try to identify the particular properties (in the words of Lanczos ``the inner nature'') of the problem, possibly starting with the mathematical model that makes these links possible.

\subsection{GMRES and the minimal polynomial}\label{sec:GMRESminpoly}
\phantom{}
\smallskip
\begin{tcolorbox}
\emph{Main point:
The roots of the GMRES iteration polynomial do not need to match those of the minimal polynomial.}
\end{tcolorbox}

\smallskip
\emph{Setup:}
We consider the \emph{Grcar matrix} (see, e.g.,~\cite[Example~3]{Tre92})
$$A=\begin{bmatrix}
 1 & 1 & 1 & 1 & & & \\
-1 & 1 & 1 & 1 & 1 & &\\
& \ddots & \ddots & \ddots & \ddots & \ddots &\\
& & \ddots & \ddots & \ddots & \ddots & 1\\
& & & \ddots & \ddots & \ddots & 1\\
& & & & \ddots & \ddots & 1\\
& & & & & -1 & 1
\end{bmatrix}\in\mathbb{R}^{500\times 500}$$
This matrix is diagonalizable (as any unreduced upper Hessenberg matrix) and well conditioned with $\kappa(A)\approx 3.63$. However, as usual for nonsymmetric Toeplitz matrices (see, e.g.,~\cite{ReiTre92}), the eigenvectors of~$A$ are very ill conditioned, so that~$A$ is highly nonnormal. Using MATLAB's {\tt eig} function yields an eigenvector matrix~$X$ with $\kappa(X)\approx 7.2\times 10^{38}$. We use $b=[1,\dots, 1]^T/\sqrt{500}$ and apply GMRES to $Ax=b$ with $x_0=0$.

\smallskip
\emph{Observations:}
The relative GMRES residual norms are shown in the bottom right part of Figure~\ref{fig:GMRESgrcar}. In the first iteration, the relative residual norm drops from $1.0$ to approximately $0.05$, and then the relative residual norms decrease almost linearly for the following approximately 250 steps. In the other three parts of Figure~\ref{fig:GMRESgrcar} the (blue) pluses show the (approximate) eigenvalues of~$A$ computed by MATLAB's {\tt eig} function. (Note that because of the severe ill-conditioning of the eigenvalue problem, this computation is, for the eigenvalues with large imaginary parts, affected by rounding errors. A thorough discussion of the spectrum of the Grcar matrix and of its approximation can be found in \cite{TreEmbBook05}.) The (red) dots show the roots of the GMRES polynomials, also called the harmonic Ritz values (see, e.g.,~\cite[Section~5.7.1]{LieStr13} for mathematical characterizations), at iterations $50, 100$, and $200$ (top left, top right, and bottom left, respectively). During the iteration, these roots fill up more and more of the same curve that ``surrounds'' the eigenvalues of~$A$. However, they do not move any closer towards the eigenvalues, although the GMRES residual norms converge almost linearly.

\begin{figure}
\begin{center}
\includegraphics[width=0.475\textwidth]{figures/gmres_grcar_ritz_50-eps-converted-to.pdf}\hfill
\includegraphics[width=0.475\textwidth]{figures/gmres_grcar_ritz_100-eps-converted-to.pdf}\\
\includegraphics[width=0.475\textwidth]{figures/gmres_grcar_ritz_200-eps-converted-to.pdf}\hfill
\includegraphics[width=0.475\textwidth]{figures/gmres_grcar-eps-converted-to.pdf}
\caption{Eigenvalues of the $500\times 500$ Grcar matrix $A$ computed by MATLAB (pluses) and the harmonic Ritz values (dots) at GMRES steps 50, 100 and 200 (top left, top right, and bottom left), and relative GMRES residual norms for $Ax=b$ with $b=[1,\dots,1]^T/\sqrt{N}$ and $x_0=0$ (bottom right).}\label{fig:GMRESgrcar}
\end{center}
\end{figure}

\emph{Explanation:}
In the CG method, the polynomial of degree $k$ providing the minimum in \eqref{eqn:CGmin} indeed approximates the minimal polynomial of the matrix $A$ (assuming that all $\eta_i$ are nonzero) {\em in the sense of solving the simplified Stietjes moment problem} (see~\cite{PozStr19}) or, equivalently, {\em in the sense of determining the nodes of the associated $k$-point Gauss quadrature}; see, e.g., ~\cite[Section~3.5]{LieStr13} and~\cite[Section~5.2]{MSB15}. As repeatedly demonstrated in the sections above, this does not mean that there exists a simple revealing relationship between the eigenvalues of $A$ and the Ritz values $\theta_\ell^{(k)}$ in \eqref{eqn:CG_Ritz_values}, not even for tightly clustered eigenvalues. Of course, we have many beautiful properties due to the underlying orthogonal polynomials, and many results about convergence of Ritz values to the eigenvalues. But this is not the same as defining a meaningful measure in relation to approximation of the minimal polynomial.

For the GMRES polynomial we do not have, in general, analogues of \eqref{eqn:CGmin} and \eqref{eqn:CG_Ritz_values}. But we do have generalizations of the Gauss quadrature through the matrix formulation of the Vorobyev moment problem; see, e.g, \cite[Section~3.7.4]{LieStr13} and \cite{Str09}. This applies to the Arnoldi algorithm for approximating eigenvalues and to the FOM method for solving linear systems, which is closely related to GMRES; see, e.g., \cite[Section 6.4]{Saa03}.
In this sense, and {\em only in this sense}, the GMRES polynomial at step $k$ approximates the minimal polynomial of $A$. However, there is no apparent way how  these very involved relationships can give meaningful general insights into the location of the $k$ harmonic Ritz values in relation to the roots of the minimal polynomial of~$A$. This important point is illustrated in Figure~\ref{fig:GMRESgrcar}, where the harmonic Ritz values remain far from the eigenvalues of $A$, although the GMRES method appears to converge well.

In numerous publications it is claimed that the main idea behind Krylov subspace methods is approximation of the minimal polynomial of the system matrix. This somewhat resonates with the original paper of Krylov published in 1931~\cite{Kry31}, which deals with computing eigenvalues. When the same idea is applied to the context of solving large linear algebraic systems, it leads to severe misconceptions for two main reasons:

\emph{Reason 1:}  The number of iterations performed in practice is typically many orders of magnitude smaller than the degree of the minimal polynomial in question.

This  remains true also in cases where an ``ideal'' preconditioner guarantees that, in theory, the degree of the minimal polynomial of the {\em exactly preconditioned} matrix is very small. As discussed in Section~\ref{sec:pGMRES}, in practice we do not precondition exactly. It is then often argued that an inexactly preconditioned matrix has, instead of a few eigenvalues (with large multiplicity), a few clusters of eigenvalues\footnote{In Section~\ref{sec:pGMRES} we explain that such an argument requires rigour, since in practical computations we do not invert the preconditioner.}. The argument continues that instead of reaching the low degree minimal polynomial (and hence the exact solution) associated with exact preconditioning in only a few steps, we can utilize these clusters for approximating the minimal polynomial of the inexactly preconditioned matrix. However, even if such a matrix is diagonalizable (which is in general not obvious), its minimal polynomial has a very large degree comparable to the size of the problem.

Approximating clusters of eigenvalues by single roots of the iteration polynomial does not work in general, regardless of how tight the clusters are; see Section \ref{sec:cgclust} above. It may only work under some very specific circumstances and restrictions that have to be clearly stated whenever the argument is used. Moreover, as already mentioned in Section~\ref{sec:pGMRES}, for highly nonnormal matrices, even small perturbations can make the eigenvalue clusters very large.

\emph{Reason 2:} The mathematical term ``approximation" can not be used without a precise description of the measure of its accuracy.

In the context of Krylov subspace methods the flaw is not in using the term ``approximation" in relation to the minimal polynomial of the system matrix. The flaw is either in not specifying any measure at all, or in a very vague identification of such a measure with the locations of the roots of the iteration polynomials.

\smallskip
We hope that this explanation together with the illustration using the Grcar matrix in Figure \ref{fig:GMRESgrcar} will help to put the arguments about the relationship between the approximation of the minimal polynomial and Krylov subspace methods (both in the symmetric and the nonsymmetric case) back on a solid mathematical ground.

\subsection{GMRES convergence for different initial residuals}\label{sec:GMRESinitres}
\phantom{}
\smallskip
\begin{tcolorbox}
\emph{Main point: The convergence behavior of GMRES can depend strongly on the initial residual, and hence convergence analysis based based only on the matrix cannot be descriptive in general.}
\end{tcolorbox}

\smallskip
\emph{Setup:}
The first example is a variation of the \emph{Frank matrix}, which is a test matrix of upper Hessenberg form that can be generated by {\tt gallery('frank',N,N)} in MATLAB. We ``flip'' this matrix and consider
$$F_N=\begin{bmatrix}
    N & N-1  &   N-2  &   \cdots  &   1\\
    N-1 & N-1  &   N-2  &  \cdots  &   1\\
     & N-2  & N-2    &  \cdots   &   1\\
     &   &   \ddots  &   \ddots  &   \vdots\\
     &   &     &   1  &   1
\end{bmatrix}\in \mathbb{R}^{N\times N}.$$
We use $N=16$ and apply GMRES with $x_0=0$ to the systems $F_{N}x=b^{(j)}$, where $b^{(1)}=[1,\dots,1]^T/\sqrt{N}$, and $b^{(2)}$ is a normalized random vector with normally distributed entries, generated using {\tt randn} in MATLAB.

The second example is a discretized convection-diffusion problem that was studied in~\cite{LieStr05}; see also~\cite[Section~5.7.5]{LieStr13}. Here the SUPG discretization with stabilization parameter $\delta$ of the problem
$$-\nu (u_{xx}+u_{yy})+u_y=0\;\;\mbox{in}\;\;\Omega=(0,1)\times (0,1),\quad
u=g\;\;\mbox{on}\;\;\partial\Omega,$$
leads to linear algebraic systems $Ax=b$ with $A=A(h,\delta,\nu)$ and $b=b(h,\delta,g)$. We use the discretization size $h=1/25$ (leading to $A\in\mathbb{R}^{N\times N}$ with $N=h^{-2}=625$) and fixed parameters $\nu=0.01,\,\delta=0.3$, but 25 different boundary conditions $g$. These boundary conditions set $g=0$ everywhere on $\partial\Omega$ except for a certain part of the right side of $\partial\Omega$; see~\cite[Example~2.2]{LieStr05} for details. The essential point is that we have only one matrix $A$, but 25 different right-hand sides $b^{(1)},\dots,b^{(25)}$. We apply GMRES with $x_0=0$.

\smallskip
\emph{Observations:} The relative GMRES residual norms for the two different matrices and the corresponding different right-hand sides are shown in Figure~\ref{fig:frank_supg}. For the flipped Frank matrix the solid blue curve corresponds to $b^{(1)}$, and dashed red curve to $b^{(2)}$. Apparently, GMRES converges much faster for $b^{(1)}$ than for $b^{(2)}$.

In the convection-diffusion problem we have for each $j=1,\dots,25$ a right-hand side $b^{(j)}$ so that GMRES has an initial phase of slow convergence (almost stagnation) for exactly $j-1$ steps. After the initial phase, the GMRES convergence speed is almost the same for all right-hand sides.

\begin{figure}
\begin{center}
\includegraphics[width=0.46\textwidth]{figures/frank-gmres-eps-converted-to.pdf}\hfill
\includegraphics[width=0.46\textwidth]{figures/gmres-conv-diff-initial-stag-eps-converted-to.pdf}
\caption{Relative residual norms of GMRES for linear algebraic systems with the flipped Frank matrix (left) and two different right-hand sides, and a matrix from the SUPG discretization convection-diffusion model problem and 25 different right-hand sides (right).}\label{fig:frank_supg}
\end{center}
\end{figure}

\smallskip
\emph{Explanation:}
The matrices used in this example are highly nonnormal. Computations with MATLAB's {\tt eig} function yield eigenvector matrices $X$ with $\kappa(X)\approx 1.1\times 10^{13}$ for the flipped Frank matrix, and $\kappa(X)\approx 2.5\times 10^{17}$ for the SUPG discretized convection-diffusion operator. The convergence bound \eqref{eqn:GMRESbd} is of little use in this situation, since it contains the very large constant $\kappa(X)$. In the derivation of the bound we have lost all information about the relation between the particular given $A$ and $r_0$. This relation may be essential, and careful analyses on a case-by-case basis are required to understand its effect on the GMRES convergence. For the SUPG discretized convection-diffusion problem such an analysis is done in~\cite{LieStr05}. It reveals that the length of the initial stagnation phase of GMRES for different boundary conditions (see Figure~\ref{fig:frank_supg}) depends on how many steps it takes to propagate the boundary information across the discretized domain by repeated multiplication with the matrix~$A$. The analysis does not explain, however, how the convergence speed after the initial phase of stagnation depends on the parameters of the problem.

\subsection{GMRES convergence for nonzero initial approximate solutions}\label{sec:GMRESnonzero}
\phantom{}
\smallskip
\begin{tcolorbox}
\emph{Main point: Unless a good initial approximate solution is known, the initial approximate solution $x_0=0$ should be used, particularly when the matrix $A$ is ill conditioned.}
\end{tcolorbox}

\smallskip
\emph{Setup:} We consider a linear algebraic system $Ax=b$, where  $A\in\mathbb{R}^{240\times 240}$ is the matrix {\tt steam1} from Matrix Market. This matrix is nonsymmetric and has a condition number of approximately $2.8\times 10^7$. The right-hand side is the normalized vector of ones. We apply GMRES with $x_0=0$ and with a normalized random $x_0$ generated using {\tt randn} in MATLAB. We stop the iteration when the relative residual norm is less than $10^{-12}$ or after 210 iterations. We also compute $x=A^{-1}b$ using MATLAB's backslash operator.

\smallskip
\emph{Observations:} The relative residual norms $\|r_k\|_2/\|r_0\|_2$ and the relative error norms $\|x-x_k\|_2/\|x-x_0\|_2$ of GMRES are shown in the left and right plots of Figure~\ref{fig:GMRESnonzeroX0}, respectively. The black solid curves correspond to $x_0=0$, and the red dashed curves to the normalized random $x_0$.

In the left plot we see that relative residual norms for the normalized random $x_0$ decrease quickly and reach the stopping criterion $\|r_k\|_2/\|r_0\|_2< 10^{-12}$ for $k=186$. For $x_0=0$ the convergence is much slower, we do not reach the stopping criterion for the relative residual norm, and we stop after $210$ iterations. In the right plot we observe that until iteration $k=186$ the relative error norms for both initial approximate solutions are virtually the same. After that, only the iteration with $x_0=0$ continues, and in the remaining iterations until $k=210$ the relative error norms continue to decrease. Eventually, the relative error norms for $x_0=0$ reach a much lower tolerance than the ones for the normalized random $x_0$.

\begin{figure}
\begin{center}
\includegraphics[width=0.475\textwidth]{figures/resids_steam1-eps-converted-to.pdf}\hfill
\includegraphics[width=0.475\textwidth]{figures/errors_steam1-eps-converted-to.pdf}
\caption{Relative residual norms (left) and relative error norms (right) of GMRES applied to a linear algebraic system with the matrix {\tt steam1}, starting with $x_0=0$ (solid black) and a normalized random $x_0$ (dotted red).}\label{fig:GMRESnonzeroX0}
\end{center}
\end{figure}

\smallskip
\emph{Explanation:} When the matrix~$A$ is ill conditioned, a nonzero initial approximate solution $x_0$ may lead to $\|r_0\|_2=\|b-Ax_0\|_2\gg\|b\|_2$. (In our example we have $\|r_0\|_2\approx 5.1\times 10^6$ for the normalized random $x_0$.) The vector $r_0$ then contains an artificially created bias towards the dominant information in the matrix~$A$ (such as large eigenvalues or singular values) that may not be related to the solution of the original linear algebraic system. Elimination of this bias by GMRES can lead to a fast reduction of the residual norms particularly in the initial phase of the iteration. This creates an illusion of fast convergence, while the approximate solutions $x_k$ actually do not approach the exact solution. (Similar examples with different matrices can be seen in~\cite[Figure~5.13]{LieStr13} or~\cite{PaiStr02}.) In order to avoid an illusion of fast convergence while using a nonzero initial approximate solution $x_0$, one can use the rescaling $\zeta_{\min} x_0$, where $\zeta_{\min}=(b^TAx_0)/\|Ax_0\|_2^2$ solves the approximation problem $\min_{\zeta}\|b-\zeta Ax_0\|_2$; see~\cite[p.~318]{LieStr13}. In our example $|\zeta_{\min}|\approx 1.9\times 10^{-8}$ holds for the normalized random $x_0$, and the residual and error norm curves of GMRES started with $\zeta_{\min} x_0$ are indistinguishable from the black solid curves in Figure~\ref{fig:GMRESnonzeroX0}.

\section{Concluding Remarks}\label{sec:Conclusions}

As an algorithmic idea, Krylov subspace methods are in their seventies, although their mathematical roots are closely related to much older objects like continued fractions, moments, and quadrature.  It might seem that, from the mathematical point of view, and perhaps apart from some more or less theoretical academic issues, they can be ready for retirement. This paper argues just the opposite.

We suggest that on the occasion of this 70th anniversary it should be considered whether the current prevailing common practice of viewing Krylov subspace methods as a utilitarian computational tool, with often oversimplified argumentation, does not miss fundamental points. Using the words attributed to Albert Einstein, ``Everything should be made as simple as possible, but not simpler''. Some influential publications seem to separate computational practice from the investigation of possibly hard but essential mathematical questions. The questions are hard because they address highly nonlinear problems that cannot be avoided when we really want to understand the methods. We therefore argue that Krylov subspace methods are remarkable {\em mathematical} objects for important further investigations. We believe that practical applications should stimulate rather than obstruct such studies. They can eventually lead to very practical discoveries. The experiments presented above give many interesting examples. We have focused on the widely used CG and GMRES methods because their understanding is a prerequisite for getting insight into other Krylov subspace methods that are based on even more complicated mathematical principles and more involved algorithmic realizations.

We have intentionally left aside many issues, such as the relationship between the infinite dimensional formulation of Krylov subspace methods and their algebraic counterparts arising from {\em their discretization}; see, e.g., \cite{MeuBook06}. We have also not included the questions on the relationship between the eigenvalues of preconditioned matrices and the {\em continuous spectrum} of their infinite dimensional non-compact operator origins, which were recently posed in \cite{GerNieStr22}. We believe that further exploiting the relationship between the analysis of infinite dimensional formulations and the analysis of  matrix computations will lead to interesting results.

In summary, Krylov subspace methods are both efficient computational tools and exciting mathematical objects. Keeping these two sides together in the journey forward will bring great benefits.

\bibliographystyle{siam}