GMRES convergence bounds for eigenvalue problems

Abstract: The convergence of GMRES for solving linear systems can be influenced heavily by the structure of the right hand side. Within the solution of eigenvalue problems via inverse iteration or subspace iteration, the right hand side is generally related to an approximate invariant subspace of the linear system. We give detailed and new bounds on (block) GMRES that take the special behavior of the right hand side into account and explain the initial sharp decrease of the GMRES residual. The bounds give rise to adapted preconditioners applied to the eigenvalue problems, e.g. tuned and polynomial preconditioners. The numerical results show that the new (block) GMRES bounds are much sharper than conventional bounds and that preconditioned subspace iteration with either a tuned or polynomial preconditioner should be used in practice.

\section{Introduction}

We consider the convergence of GMRES~\cite{SaadSchultz86} for solving sequences of linear systems of the form
\begin{equation}
\label{eq:linsys1}
Bx_i = y_i,
\end{equation}
where $B = A-\sigma I$, $A\in\mathbb{C}^{n\times n}$ and  $\sigma$ is a fixed or variable scalar shift.
Throughout, we assume that  $A$ is diagonalizable
with eigenvalues $\gamma_j$, $j = 1,\dotsc, n$, so that  $B = A - \sigma I$ has eigenvalues $\lambda_j = \gamma_j - \sigma$, $j = 1,\dotsc, n$.
Thus, we can write $B = Z\Lambda Z^{-1}$, where
$\Lambda = \diag{\lambda_1, \dotsc, \lambda_n}\in\mathbb{C}^{n\times n}$
and $Z = [z_1, \ z_2, \ \dotsc, \ z_n]$ is a matrix of eigenvectors.
Without loss of generality, we let $y_i$ be an approximation of the eigenvector $z_1$ (with both vectors normalized to have unit norm).
Our main aim is to describe accurately, using simple bounds, the convergence of GMRES when the right-hand side is an approximate eigenvector; to clearly explain why this convergence behavior is lost for many preconditioners; and how it can be recovered by choosing or modifying the preconditioner appropriately.

Systems of the form~\eqref{eq:linsys1} arise, for example, when computing an eigenvector and corresponding eigenvalue of a matrix $A\in\mathbb{C}^{n\times n}$ using inverse
iteration (see Algorithm~\ref{algo:inverse_iter}). In this case the shift is chosen to be close to an eigenvalue of $A$ and the approximate eigenvector $y_i$ is
the $i$th iterate of inverse iteration.
If the system \gref{eq:linsys1} is solved exactly (e.g.\ using a
direct solver) one can show that, under certain generic assumptions, the right-hand side $y_i$ converges at least linearly to an eigenvector of $A$ (see, e.g.
\cite{PetW93,Ipsen97} and references therein).
However, for very large matrix sizes $n$, direct solvers for \gref{eq:linsys1} are infeasible and an iterative method is used to solve  \gref{eq:linsys1}  inexactly.
In this case, one can show that if the solve tolerance is chosen appropriately (e.g.\ it is reduced proportionally to the eigenvalue residual) we still obtain the same convergence rates for this inexact version of inverse iteration \cite{FrSp05b,FrSp06a}.

\begin{algorithm}[H]
\DontPrintSemicolon
  \KwIn{$x_0$, $i_{\text{max}}$}
\For{$i = 0,\dotsc,i_{\text{max}}-1$}{
Choose $\sigma$ and $\tau_i$\;
Find $y_i$ such that $\|(A-\sigma I) y_i - x_i\| \le \tau_i$\;
Set $x_{i+1} = y_i/\|y_i\|$ and $\lambda_{i+1} = x_{i+1}^TAx_{i+1}$\;
Evaluate $r_{i+1} = (A - \lambda_{i+1}I)x_{i+1}$ and test for convergence\;
}
\KwOut{$x_{i_{\text{max}}}$, $\lambda_{i_{\text{max}}}$\;}
\label{algo:inverse_iter}
  \caption{Inexact inverse iteration}
\end{algorithm}

Here we are concerned with the the convergence behavior of iterative methods for solving \gref{eq:linsys1}.
It is well known that when the right-hand side is an approximate
eigenvector of $B$, Krylov subspace methods converge faster
than for generic right-hand sides. This was first observed for
Hermitian positive matrices by Simoncini and Eld\'en~\cite{SimonElden02}, who considered linear solves in Rayleigh quotient iteration.
Further results in this direction include convergence
bounds in~\cite{BeGrSp04} for MINRES (see~\cite{PaiS75}) applied within inverse iteration for symmetric
 matrices, and
Theorem 2.2 in~\cite{FrSp07b}, which characterizes GMRES convergence
for non-Hermitian generalized eigenvalue problems using the
Schur factorization.
For more general systems that do not necessarily come from
eigensolvers, research into GMRES convergence has
also  highlighted the influence of  the
right-hand side~\cite{APS98,DTM14}, and
its representation in the eigenvector basis~\cite{MDT15, TPPW14}.

In this work, we show that if $A$ is diagonalizable  the  convergence theory developed in \cite{TPPW14} yields an insightful explanation for the GMRES behavior
in inverse iteration
 with different
types of preconditioned inner solves. Moreover, we explain why the GMRES residual often decreases sharply in the first iteration \cite{ElmXue11}. A more detailed description of this phenomenon is given in Section~\ref{sec:initdecrease}.
Regarding the preconditioned situation, it is well known that so
called tuned preconditioners can significantly reduce the number of required GMRES steps. We show that using polynomial preconditioners for the inner solves can
be similarly beneficial.

The rest of this paper is structured as follows. In Section~\ref{sec:gmres_bound} we present the GMRES bounds from~\cite{TPPW14} and show why the GMRES residual
norm often has a rapid initial decrease. In Section~\ref{sec:pre_gmres} we apply these bounds to  GMRES for inverse iteration  without  preconditioning, and
with standard, tuned and  polynomial preconditioners. Extensions to generalized eigenvalue problems and block GMRES are discussed in
 Section~\ref{sec:extensions}, while numerical experiments can be found in Section~\ref{sec:num_ex}.

Throughout,
 $k$ denotes the iteration count for the GMRES algorithm and $i$ is the iteration count for the outer iteration (which is inverse iteration here). In our analysis $j$ will denote the index of eigenvalues and eigenvectors, e.g. $Bz_j = \lambda_j z_j$ and is not to be mixed up with the iteration indices.

\section{Analysis of the GMRES convergence bound}\label{sec:gmres_bound}

 In this section we apply results from~\cite{TPPW14} to the system
in~\eqref{eq:linsys1} to give a lower bound on the number of GMRES
iterations required to achieve a given residual reduction
and explain why GMRES residuals rapidly
decrease in the first iteration when $y_i \approx z_1$, i.e., when
$y_i$ is a good approximation of an eigenvector of $A$.
The GMRES convergence bounds we use to achieve this are
related to those in~\cite{BeGrSp04} for Hermitian problems and to more
general results in \cite{FrSp07b}
for non-Hermitian generalized eigenvalue problems.

We begin by stating the relevant results from~\cite{TPPW14}.
The key idea of these bounds
is to express $y_i$ in the eigenvector basis.
Accordingly we introduce
\[w^{[i]} = Z^{-1}y_i/\|y_i\|_2 = \begin{bmatrix}w_1^{[i]} & w_2^{[i]} & \dotsc & w_n^{[i]}\end{bmatrix}^T = \begin{bmatrix} w_1^{[i]} & {w^{(2)}}^{[i]}\end{bmatrix}^T,\]
where ${w^{(2)}}^{[i]} \in \mathbb{C}^{n-1}$.
If $y_i\approx z_1$ it is reasonable to expect that $|w_1^{[i]}| \gg\|{w^{(2)}}^{[i]}\|_2$
and this is observed in practice (see results in corresponding columns of Tables~\ref{t:cdde:U}--\ref{t:all}).
Note that in the rest of this section, we drop the outer iteration index $i$ from $w$ and
related quantities for clarity.
For simplicity, let us assume that the initial guess for GMRES is the zero vector,
so that $r_0 = y_i$.
Since $y_i$ is normalized at every step to have unit norm,
 $\|r_0\| = \|y_i\|=1$ $\forall i$.

We are now in a position to recall Theorem~2.2 from~\cite{TPPW14}, which states that
\begin{equation}\label{eq:2.5}
\|r_k\|_2 \le \|Z\|_2  \min_{\substack{q\in\Pi_k,\\q(0)=1}}\| W q(\Lambda)e\|_2
= \|Z\|_2  \min_{\substack{q\in\Pi_k,\\q(0)=1}} \left(\sum_{j = 1}^n |w_j q(\lambda_j)|^2 \right)^{\frac{1}{2}},
\end{equation}
where $W = \diag{w_1,\dotsc,w_n}$.
This bound highlights
the role of the eigenvalues of $B$ weighted by the corresponding component of $w$
in the GMRES convergence.
In particular, whenever $\|w^{(2)}\|_2$ is small enough,
$|w_j q(\lambda_j)| \ll |w_1 q(\lambda_1)|$, $j = 2, \dotsc, n$ and the
minimizing polynomial will focus first on $|w_1 q(\lambda_1)|$.
Thus, fast convergence of GMRES is linked both with the spectrum of $B$ and with
the quality of the eigenvector approximation $y_i$.

A slightly different bound is obtained if, similarly to~\cite{BeGrSp04,FrSp07b},
we replace the minimizing polynomial
in~\eqref{eq:2.5} by $\tilde q_k(\lambda) = (1-\lambda/\lambda_1)q_{k-1}(\lambda)$,
where $q_{k-1}$ is a polynomial of degree $k-1$ for which $q_{k-1}(0) = 1$.
This is subject of Theorem 2.3 in~\cite{TPPW14}, which states that for $k>1$,
\begin{subequations}
\label{eq:2.6}
\begin{align}
\|r_k\|_2
& \le  \|Z\|_2  \min_{\substack{q\in\Pi_{k-1},\\q(0)=1}}
\left(\sum_{j = 2}^n |\tilde w_j q(\lambda_j)|^2 \right)^{\frac{1}{2}} \label{eq:2.6a}\\
& \le \|Z\|_2 \|\tilde w\|_2 \min_{\substack{q\in\Pi_{k-1},\\q(0)=1}}
\max_{\lambda\in\{\lambda_2,\dotsc,\lambda_n\}}|q(\lambda)|, \label{eq:2.6b}
\end{align}
\end{subequations}
where $\tilde w_j= w_j(1-\lambda_j/\lambda_1)$, $j=2,\ldots,n$. Starting from \eqref{eq:2.5},
a similar approach to the proof of~\eqref{eq:2.6} shows that, additionally,
\begin{equation} \label{eq:2.6k1}
\|r_1\|_2 \le \|Z\|_2 \left(\sum_{j = 2}^n |\tilde w_j |^2 \right)^{\frac{1}{2}}
= \|Z\|_2 \|\tilde w\|_2.
\end{equation}
In contrast to~\eqref{eq:2.5}, neither~\eqref{eq:2.6} nor \eqref{eq:2.6k1}
 involves $w_1$ and so they emphasize the fact
 that the relative residuals may be small when
$\|\tilde w\|_2$ is small.
(Note that we may always normalize $\|Z\|_2$, but that this affects $w$,
and hence $\|\tilde w\|_2$).

\subsection{Inner iterations vs outer iterations}
In~\cite{FrSp07b} it was shown that the term
\begin{equation}\label{eq:min_max}
\min_{\substack{q\in\Pi_{k-1},\\q(0)=1}}
\max_{\lambda\in\{\lambda_2,\dotsc,\lambda_n\}}|q(\lambda)|
\end{equation}
can often be bounded
by an expression of the form
\begin{equation}\label{eq:power}
S(1/C)^{k-1}
\end{equation} (by, e.g.\ \cite[Proposition 2.3]{FrSp07b}) where $C$ and $S$ depend on the spectrum of $B$. The authors of~\cite{FrSp07b} used this bound to show that
the number of GMRES iterations required to reach a desired tolerance,
 i.e.,  to satisfy $\|r_k\|_2/\|r_0\|_2 \le \tau$, can also be bounded.
 Combining~\eqref{eq:2.6} and~\eqref{eq:power} gives us a simple alternative lower
bound on the number of GMRES iterations:
\begin{equation}
\label{eq:iter_bound}
 k \ge 1 + \frac{1}{\log(C)}\left[\log\left(S\right) +
\log\left(\frac{\|Z\|_2\|\tilde w\|_2}{\tau}\right) \right] .
\end{equation}

Since both $C$ and $S$ depend only on the spectrum of $A$ and the shift $\sigma$,
they are independent of the inverse
iteration step $i$. Thus, if we can bound  ${\|Z\|_2\|\tilde w\|_2}/{\tau}$ independently of the
inverse iteration step then the above bound on $k$ suggests that
 the number of GMRES iterations should not increase as
 inverse iteration proceeds. We will observe that this occurs if GMRES is applied either without a preconditioner, or with specially chosen preconditioners (see Tables~\ref{t:cdde:U} and \ref{t:cdde:T}).

\subsection{Initial decrease in GMRES residuals}
\label{sec:initdecrease}
Another phenomenon that often arises when solving linear
systems with approximate eigenvectors is that $\|r_1\|_2 \ll \|r_0\|_2$
(see e.g. Figure~\ref{f:cdde1_residuals}).
The bound~\eqref{eq:2.6} shows clearly why this occurs.

First note that
\[|\tilde w_j| = \left| 1-\frac{\lambda_j}{\lambda_1}\right| |w_j| = \frac{|\gamma_1-\gamma_j|}{|\gamma_1-\sigma|}|w_j|
\le |w_j| \max_{j\in[2,\dotsc,n]} \frac{|\gamma_1-\gamma_j|}{|\gamma_1-\sigma|} = C_1|w_j|,\]
where the constant $C_1$ depends only on the spectrum of $A$ and the shift $\sigma$.
Accordingly, $\|\tilde w\|_2 \le C_1 \|w^{(2)}\|$, and~\eqref{eq:2.6k1} becomes
\begin{equation}
\label{eq:r1}
\|r_1\|_2 \le C_2 \|w^{(2)}\|_2,
\end{equation}
where $C_2 =  \|Z\|_2 \max_{j\in[2,\dotsc,n]} {|\lambda_1-\lambda_j|}/{|\lambda_1-\sigma|}$.

However, the values of $w_j$, $j=2,\ldots n$ are very small, and indeed are zero if the right-hand side is an exact eigenvector. Hence, in the latter
stages of inverse iteration there is a sharp decrease in the bound for the relative GMRES residual norms. Of course, this is only true if the right-hand side is
an approximate eigenvector of $B$, i.e.\ in the unpreconditioned case, or with specially chosen preconditioners. An example of this phenomenon is given in
Figure~\ref{f:cdde1_residuals}.
\section{Convergence bounds for preconditioned GMRES}\label{sec:pre_gmres}

Having considered a general analysis for GMRES convergence that involves the right-hand side vector, we now investigate what this analysis tells us about  solving the linear systems in inverse iteration, both with and without preconditioning.

\subsection{No preconditioner}

When GMRES is applied to \gref{eq:linsys1} without a preconditioner, the coefficient matrix is $B = A - \sigma I$, $A\in\mathbb{C}^{n\times n}$
where, assuming $A$ is diagonalizable, $B = Z\Lambda Z^{-1}$ with $\Lambda\in\mathbb{C}^{n\times n}$.
The right-hand side $y_i$ is an approximation of $z_1$ (and both vectors are normalized to have unit norm). Thus it is possible to write
\begin{equation}
\label{eq:decomp}
y_i = \alpha_i z_1 + Z_2 p_i,
\end{equation}
where $\alpha_i\in\mathbb{C}$, $Z_2 = [z_2,\ldots,z_n]\in \mathbb{C}^{n\times n-1}$ and $p_i\in\mathbb{C}^{n-1}$. We assume that $\alpha_i$ and $p_i$ are chosen such that $\|y_i\|=1$ $\forall i$. The scalar $\alpha_i$ measures the deviation of $y_i$ from $z_1$ and, under generic assumptions for exact solves, inverse iteration converges, that is $\alpha_i\rightarrow 1$ and $\|p_i\|\rightarrow 0$ as $i\rightarrow\infty$.

The bound in \gref{eq:2.5}, and in particular the vector $w$, depends on the outer iteration~$i$. Using \gref{eq:decomp} we obtain
\begin{eqnarray*}
w^{[i]} &=& Z^{-1}y_i= Z^{-1}(\alpha_i z_1 + Z_2 p_i)\\
&=& \alpha_i e_1 + E_{n-1}p_i,
\end{eqnarray*}
where $e_1\in\mathbb{R}^n$ and $E_{n-1}\in\mathbb{R}^{n,n-1}$ contain the first and the last $n-1$ columns of the identity matrix, respectively. Recall that $w^{[i]}$ denotes the vector $w$ at outer iteration $i$. With $\alpha_i\rightarrow 1$ and $\|p_i\|\rightarrow 0$ as $i\rightarrow\infty$ for a convergent outer iteration we obtain
$w^{[i]}\rightarrow e_1$. Hence in the limit $q(\lambda)$ only needs to be minimized over $\lambda_1$.

Noting that
$\|Z\|_2$ can be normalized,  $\|w^{[i]}\|_2\rightarrow 1$
and $\|\tilde{w}^{[i]}\|_2 \rightarrow 0$, we see that
the terms $\|Z\|_2\|w^{[i]}\|_2$ and $\|Z\|_2\|\tilde{w}^{[i]}\|_2$ in~\eqref{eq:2.5} and ~\eqref{eq:2.6} can be
bounded by an arbitrarily small constant for large enough $i$. Therefore, as the outer iteration progresses, small  relative GMRES residual norms are rapidly obtained.
Experimentally we see that the number of inner iterations is bounded
because $\|\tilde{w}^{[i]}\|_2\rightarrow 0$ at a rate proportional to the decrease of $\tau$.
This is reflected by the lower bound in~\eqref{eq:iter_bound}, which is constant if  $\|\tilde{w}^{[i]}\|_2 \propto \tau$  (cf. Lemma 3.11 in~\cite{frei07}).
As the outer iterations progress (i.e.\ for larger $i$) there is an initial decrease in the relative GMRES residual norms, as suggested by~\eqref{eq:r1},
because $\|\tilde{w}^{[i]}\|_2 \approx 0$.

\subsection{Standard preconditioner}

We now investigate the changes that occur when a preconditioner $P$ is introduced.
Usually GMRES is used
with a preconditioner  to cluster the eigenvalues of the system matrix. The aim of this is to  reduce the bound on the term~\eqref{eq:min_max}
and hence improve the convergence bound~\eqref{eq:2.6}. We will see that this usually comes at the expense of increasing $\|\tilde w\|_2$, so that the number of GMRES iterations actually grows as inverse iteration proceeds (although this number is generally still lower than the number of GMRES iterations needed without a preconditioner).

 Without loss of generality we consider a right
preconditioner for \gref{eq:linsys1}, that is
\begin{equation}
BP^{-1}\tilde{x}_i = y_i,
\end{equation}
where $ P^{-1}\tilde{x}_i = x_i$.
Although the eigenvalues of $BP^{-1}$ may be clustered, the eigenvectors will typically differ from those of $B$. Thus, although $y_i$ is an approximate
eigenvector of $B$, it is not usually an approximate eigenvector of the coefficient matrix $BP^{-1}$. This causes  the number of GMRES iterations to increase as
inverse iteration progresses,  as we now show.

In general we have $BP^{-1}\neq ZDZ^{-1}$ (with $D$ a diagonal matrix), that is,  $BP^{-1}$ does not have the same eigenvectors as $B$. Hence, with $BP^{-1}= \bar{Z}D\bar{Z}^{-1}$ and using the decomposition of $y_i$ we obtain
\[
w^{[i]} = \bar{Z}^{-1}y_i= \alpha_i \bar{Z}^{-1} z_1 + \bar{Z}^{-1}Z_2 p_i\rightarrow \alpha_i \bar{Z}^{-1} z_1,
\]
as $i\rightarrow\infty$ since $\|p_i\|\rightarrow 0$, but $\alpha_i \bar{Z}^{-1} z_1\neq e_1$ in general.

Hence, there is no reason for $\|{w^{(2)}}^{[i]}\|_2$ to be small. Instead, $\|{w^{(2)}}^{[i]}\|_2\rightarrow c$ for some constant $c$ as $i$ increases (see
corresponding column in Table~\ref{t:cdde:U}) and we observe that for some problems $\|{w^{(2)}}^{[i]}\|_2 > |w_1^{[i]}|$, i.e., $c>1$.
Since $\|{w^{(2)}}^{[i]}\|_2$ does not approach $0$, but $\tau$ decreases as the outer iteration progresses,
we would expect from~\eqref{eq:iter_bound} that the number of inner iterations increases with increasing~$i$, and indeed this is what our experiments in Section~\ref{sec:num_ex} show.
Moreover, we find that $\|r_1\|_2$ is not significantly smaller than $\|r_0\|_2$ at later outer iterations.
However, for large enough $i$ the terms $\|Z\|_2\|w^{[i]}\|_2$ or $\|Z\|_2\|\tilde{w}^{[i]}\|_2$ can be bounded by constants that are are still small
relative to $\kappa_2(Z) = \|Z\|_2 \|Z^{-1}\|_2$, and the bounds~\eqref{eq:2.5} and \eqref{eq:2.6} can still capture GMRES convergence behavior, especially at later outer iterations (see Figure~\ref{f:cdde1_residuals}).

To recover bounded numbers of inner iterations when preconditioning we must ensure that the preconditioner preserves the relevant direction given by the right-hand side. For this, we may construct a preconditioner such that
\begin{enumerate}[(a)]
\item $BP_i^{-1}= \bar{Z}_iD_i\bar{Z}_i^{-1}$ such that (in the limit for large $i$) $\bar{Z}_i^{-1} z_1 \rightarrow e_1$, or
\item $BP^{-1}$ has the same eigenvectors as $B$.
\end{enumerate}

The next two subsections show how these preconditioners can be constructed.

\subsection{Tuned preconditioner}

Assume we have an a preconditioner $P$ such that $BP^{-1}$ is diagonalizable. Then $BP^{-1} = Z\Lambda Z^{-1}P^{-1} = \bar{Z}D\bar{Z}^{-1}$, where $D$ is a diagonal matrix. Since we want to recover bounded GMRES iteration counts, we want $y_i$ to be an approximate eigenvector of $BP^{-1}$. To achieve this, it is not necessary that $BP^{-1}$ has all the same eigenvectors as $B$, but it should have the same eigenvector in the direction of the sought eigenvector $z_1$, that is, we want to enforce $\bar{Z}^{-1} z_1 = e_1$.

If we enforce this condition,  we obtain
\[
Z\Lambda Z^{-1}P^{-1}z_1 = \bar{Z}D\bar{Z}^{-1}z_1 = \bar{Z}De_1 = d_{11} z_1.
\]
Since $\Lambda = \text{diag}(\lambda_1,\ldots,\lambda_n)=\text{diag}(\gamma_1-\sigma,\ldots,\gamma_n-\sigma)$, and assuming that $\gamma_j\neq\sigma$, $\forall j$, the above requirement shows that $P^{-1}$ needs to satisfy
\begin{equation}
\label{eq:tuning}
P^{-1}z_1 = \frac{d_{11}}{\lambda_1} z_1.
\end{equation}
Hence, choosing a preconditioner $P$ such that \gref{eq:tuning} holds, or equivalently
\[
Pz_1 = \frac{\lambda_1}{d_{11}} z_1,
\]
yields $\bar{Z}^{-1} z_1 = e_1$. Note that $d_{11}\neq 0$ (otherwise $\lambda_j=0$ and $\gamma_j=\sigma$ for some $j$, a case we excluded).

Clearly, \gref{eq:tuning} is infeasible as a preconditioner, since $z_1$, $\lambda_1$ and $d_{11}$ are unknown during the iteration. Hence, we propose to use a preconditioner $P_i$ at each iteration $i$ that satisfies
\begin{equation}
\label{eq:tunedpract}
P_i y_i = \lambda^{(i)}y_i =  (\gamma^{(i)}-\sigma) y_i,
\end{equation}
where $\gamma^{(i)}$ is an eigenvalue approximation for $\gamma_1$; it can be obtained as part of the the inverse iteration algorithm.
\begin{remark} Instead of \gref{eq:tunedpract} one could use
\begin{itemize}
\item $P_i y_i = y_i$, since the direction is important for the preconditioner;
\item $P_i y_i = B y_i$, since $B y_i = (\gamma^{(i)}-\sigma) y_i + r_i = \lambda^{(i)} y_i + r_i$ with $r_i = (A-\gamma^{(i)}I)y_i$ the eigenvalue residual, and in the limit \gref{eq:tunedpract} holds;
\item $P_i y_i = A y_i$, since $A y_i = \gamma^{(i)} y_i + r_i$, which, in the limit, lies in the same direction.
\end{itemize}
The action of $P_i^{-1}$ on a vector $v$ can easily be obtained as an update of  $P^{-1}v$ using the Sherman-Morrison formula, that is
\[
P_i^{-1}v = P^{-1}v -\frac{P^{-1}y_i -(\lambda^{(i)})^{-1} y_i}{y_i^T P^{-1}y_i} y_i^T P^{-1}v.
\]
This can be implemented using only one extra linear solve with $P$ (to compute $P^{-1}y_i$) per outer iteration. Note that we assume that $\lambda^{(i)}\neq 0$.
\end{remark}

\begin{theorem}
Consider inverse iteration, where at each inner iteration a preconditioned system of the form
$BP_i^{-1}\tilde{x}_i = y_i$ is solved with $P_i^{-1}\tilde{x}_i = x_i$, and $P_i$ satisfies  \gref{eq:tunedpract}. Assume that $BP_i^{-1}$ is diagonalizable $\forall i$, that is $BP_i^{-1} = Z_iD_iZ_i^{-1}$ with $D_i$ diagonal. Further, assume that $B$ is nonsingular, that is $\lambda^{(i)}=\gamma^{(i)}-\sigma\neq 0$ $\forall i$. Then
\begin{enumerate}[(i)]
\item $BP_i^{-1}y_i = y_i + \frac{r_i}{\gamma^{(i)}-\sigma}$, where $r_i = (A-\gamma^{(i)}I)y_i$ is the eigenvalue residual (that is, in the limit $BP_i^{-1}$ has an eigenvalue at $1$);
\item $w^{[i]}\rightarrow e_1$ as $i\rightarrow\infty$.
\end{enumerate}
\end{theorem}
\begin{proof}
\begin{enumerate}[(i)]
\item Using the definition of $P_i^{-1}$ we obtain
\[
BP_i^{-1}y_i = (\gamma^{(i)}-\sigma)^{-1} B y_i = y_i + \frac{r_i}{\gamma^{(i)}-\sigma},
\]
where we have used the fact that $By_i=(A-\sigma I)y_i = (\gamma^{(i)}-\sigma)y_i+r_i$.
\item We need to compute $w^{[i]} = Z_i^{-1}y_i$ as $i\rightarrow\infty$. Consider the eigendecomposition $BP_i^{-1} = Z_iD_iZ_i^{-1}$ as $i\rightarrow\infty$. Then $y_i\rightarrow z_1$ and, using part (i) we have
\[
BP_i^{-1}y_i\rightarrow z_1,
\]
since $r_i\rightarrow 0$.
Therefore, in the limit $Z_i \rightarrow [z_1, \tilde{Z}_2]$, as $i\rightarrow\infty$, for some $\tilde{Z}_2$ and hence,
\[
w^{[i]} = Z_i^{-1}y_i \rightarrow  [z_1, \tilde{Z}_2]^{-1} z_1 = e_1.
\]
\end{enumerate}
\end{proof}

We obtain $w^{[i]} \rightarrow e_1$. Since
 $\|Z\|_2$ can be normalized, $\|w^{[i]}\|_2\rightarrow 1$,
and $\|\tilde{w}^{[i]}\|_2\rightarrow 0$, the terms $\|Z\|_2\|w^{[i]}\|_2$ and $\|Z\|_2\|\tilde{w}^{[i]}\|_2$
in~\eqref{eq:2.5} and \eqref{eq:2.6} can be bounded by small constants for large enough $i$.
Experimentally we see that the number of inner iterations is bounded
because $\|w^{[i]}\|_2\rightarrow 0$ at a rate proportional to $\tau$
(cf.\ Theorem~6.22 in~\cite{frei07}). Additionally, as the outer iterations progress there is an initial decrease
in the relative GMRES residual norms, that is, $\|r_1\|_2 \ll \|r_0\|_2$,  because
$\|\tilde{w}^{[i]}\|_2 \approx 0$.

\subsection{Polynomial preconditioner}\label{ssec:polyprec}
As an alternative to tuning we can construct a preconditioner such that $BP^{-1}$ has the same eigenvectors as $B$.
If we choose
\[
P^{-1} = p(B),
\]
where $p(B)$ is a polynomial of $B$, then $p(B) = Zp(\Lambda) Z^{-1}$ and $BP^{-1} = Z\Lambda p(\Lambda) Z^{-1}$ has the same eigenvectors as $B$. In this case  the theory
for the unpreconditioned case applies.

We consider the (right) polynomially preconditioned system $ Bp(B)\ensuremath{\tilde{x}}_i=y_i$, where $x_i=p(B)\ensuremath{\tilde{x}}_i$ and
\begin{align*}
p(z)=\sum\limits_{h=0}^{d}\mu_hz^h\in\Pi_d.
\end{align*}
Since $p(B)=Zp(\Lambda)Z^{-1}$, $y_i$ is still an approximate eigenvector of the preconditioned coefficient matrix $Bp(B)$ and it holds that
\begin{equation*}
 \|r_k\|_2\leq \|Z\|_2\min\limits_{\stackrel{q_k\in\Pi_k}{q_k(0)=1}}
 \left(\sum_{j = 1}^n\vert w_jq_k(\lambda_jp(\lambda_j))\vert^2\right)^{\ensuremath{\frac{1}{2}}}.
 \end{equation*}
Hence, all the weights $w$ of the unpreconditioned version are preserved.
Typically, $p(B)$ is chosen such that
\begin{align*}
 B(p(B))\approx I\quad\text{or}\quad g(B):=I-Bp(B)\approx 0.
\end{align*}
In the latter choice $g\in\Pi_{d+1}$ is the residual polynomial, which can be written as
\begin{align*}
 g(z)=\prod\limits_{h=1}^{d+1}(1-\nu_h z)=1-\sum\limits_{h=1}^{d+1}\mu_{h-1}z^{h}.
\end{align*}
There are different strategies to choose the polynomial coefficients $\mu_h$, which
can be determined from the $\nu_h$ recursively via
\begin{align*}
 g_{h}(z)=g_{h-1}(z)-\nu_hzg_{h-1}(z),\quad h=1,\ldots,d+1,~g_0(z)=1,
\end{align*}
see, e.g.\ \cite{AhmSG12} for more details.
A thorough discussion of various choices for the polynomial coefficients is beyond the scope of this study and we restrict ourselves to a few selected, often
used approaches. More information on the choice of the polynomial can be found, e.g., in \cite{Freund1990,AshMO92,Gij95,AhmSG12,templatesLS,LiuMW15} and the
references therein.

A basic common choice \cite{AhmSG12} is to take the $\nu_h$ as reciprocal Chebychev nodes with respect to an interval $[a,b]$:
\begin{align}\label{polycoeffcheb}
 \phi_h=\frac{2h-1}{2(d+1)},\quad \nu_h=2\left(b+a-(b-a)\cos(\pi\phi_h)\right)^{-1},\quad h=1,\ldots,d+1.
\end{align}
Here, $a,b$ denote the smallest and largest eigenvalues (or approximations thereof) of $\Lambda(B)\subset\mathbb{R}$. For complex spectra, $a,b$ are the foci of the
ellipse enclosing $\Lambda(B)$. The values $a,b$ can be obtained from approximate eigenvalues of $A$ or $B$, e.g.\ by employing
\texttt{eigs} with the 'SM' and 'LM' flags, or using the Ritz values generated by the Arnoldi process. Alternatively one can compute a
very small number of
(harmonic)  Ritz values \cite{Gij95,AhmSG12}, which can be generated by a few steps of an (harmonic) Arnoldi process.
When using this approach, it is important to ensure that $a,b$ have the same sign but in some of our examples this not the case and either $a$ or $b$ lie a little bit on the wrong side
of
the origin. In these cases we slightly shifted $a,b$, e.g.\ if $b>0$ but $a<0$ is small we set $b\leftarrow b-2a$, $a\leftarrow -a$.

For complex spectra  a more sophisticated approach is proposed in \cite{Gij95}. The key idea is to consider the norm
\begin{align}\label{polycoeffmin}
 \|g\|^2=\frac{1}{L}\intab{\Gamma}{}g(z)\overline{g(z)}\vert\mathrm{d}z\vert
\end{align}
induced by the scalar product $\langle f,g\rangle=\frac{1}{L}\intab{\Gamma}{}f(z)\overline{g(z)}\vert\mathrm{d}z\vert$.
Here $\Gamma$ is a piecewise linear contour approximating the shape of the spectrum of $B$ and $L$ is its arc length.
The coefficients $\mu_i$ in the polynomial are determined such that the above norm is minimized in a least-squares sense.
Using (harmonic) Ritz values allows us to easily generate the piecewise linear contour, e.g.  using the convex hull of the Ritz values.
However, this approach also allows us to approximate the spectrum by non-convex objects such as a boomerang-shaped region, which can be beneficial in some cases.
Note that if $B$ is real, it is sufficient to incorporate only the part of $\Gamma$ with nonnegative imaginary parts. More details on the implementation of this
strategy are given in \cite{Gij95}.

A polynomial preconditioner preserves the eigenvector basis and hence appears to be an ideal preconditioner to use within GMRES for linear systems arising
within inverse iteration (using the theory in this paper). However, we would like to note that
a limitation of good polynomial preconditioners is that we require knowledge about the location of the spectrum of $B$.
Since we are actually seeking an
eigenvalue, the necessary information to construct a good polynomial preconditioner might be hard to obtain. We already mentioned the use of spectral estimates, which can be obtained by, e.g., (harmonic) Arnoldi processes.
Since we are using inexact inverse iteration with GMRES as inner solver, one
 strategy deserving further study is to use the  upper Hessenberg matrix generated at
outer iteration step $i-1$ to acquire spectral estimates for outer iteration step $i$. In a similar way, the generated basis matrices of the Krylov subspaces can be used to acquire the coefficients
by solving a least square problem along the lines of \cite{LiuMW15}.

For completeness we mention a third polynomial preconditioning strategy. Since iterative Krylov methods for linear systems work implicitly with polynomials
of $B$, we could in principle use any of these  methods as a preconditioner for the inner iteration. In other words, we could use (a small number of steps of) a Krylov
method within a Krylov method, which is GMRES here. However, since Krylov methods also depend on the right-hand side,  which determines the first basis vector in
the Krylov subspace,
the preconditioner will change with every iteration of GMRES in our consideration. Therefore, a flexible preconditioned Krylov method, such as flexible
GMRES~\cite{Saa93} must be applied, which is beyond the scope of this study. Similar approaches can be found in, e.g.  \cite{GijSZ15,DuSZ15,BG15}. The
advantage of this strategy is that one does not have to worry about the generation of the polynomial coefficients.
\section{Extensions and related issues} \label{sec:extensions}

In this section several extensions to inverse iteration for the standard eigenvalue problems are considered.
First we extend our analysis of GMRES within inverse iteration to generalized eigenvalue problems. Then,  we obtain bounds for block GMRES arising in inverse subspace iteration for the standard eigenvalue problem.

\subsection{Generalized eigenvalue problems} \label{subsec:gen_eig}

Consider inverse iteration for the generalized eigenvalue
problem $Ax = \lambda Mx$, where at least one of the matrices $A$ and $M$ is nonsymmetric. We shall also assume that the desired eigenvalue is finite in the case that
$M$ is singular.
For generalized eigenvalue problems, the linear systems we wish to solve in inverse iteration are of the form
$(A - \sigma M)x_i = My_i$, where
 we normalize $y_i$ so that
$\|My_i\|_2  = 1$. The eigenvalue approximation can then be obtained by a generalization of the Rayleigh quotient, e.g.\
$(x_i^TM^TAx_i)/\|Mx_i\|^2$.
Let us now assume that $A - \sigma M$ is diagonalizable, i.e.\
$A - \sigma M = Z \Lambda Z^{-1}$ where $Z$ is a matrix
of eigenvectors and $\Lambda$ is a diagonal matrix of eigenvalues.
We also assume without loss of generality that $y_i$ approximates
$z_1$, the first column of $Z$.

Setting
\[w^{[i]} = Z^{-1}y_i = \begin{bmatrix} w_1^{[i]} & {w^{(2)}}^{[i]}\end{bmatrix}^T\]
 we would again expect that
when $y_i$ is a good approximation of $z_1$ that $|w_1^{[i]}|\gg \|{w^{(2)}}^{[i]}\|_2$. Again, for clarity, in the rest of this section, we drop the outer
iteration index $i$ from $w$ and related quantities where appropriate. The GMRES bounds in this case becomes

\begin{equation}\label{eq:gep_gmres}
\begin{aligned}
\|r_k\|_2 &=  \min_{\substack{q\in\Pi_k,\\q(0)=1}}\|q(A-\sigma M)My_i\|_2 \\
&= \min_{\substack{q\in\Pi_k,\\q(0)=1}} \|Z q(\Lambda) (Z^{-1}MZ) w\|_2 =  \min_{\substack{q\in\Pi_k,\\q(0)=1}} \|Z q(\Lambda) f\|_2,
\end{aligned}
\end{equation}
where $f:= Z^{-1}MZw$. Note that if $w$ approximates the first
unit vector, then $f$ approximates the first column of $Z^{-1}MZ$.
In this case an analogous bound to~\eqref{eq:2.5} is
\begin{equation}\label{eq:2.5gep}
\|r_k\|_2 \le \|Z\|_2  \min_{\substack{q\in\Pi_k,\\q(0)=1}}\| F q(\Lambda)e\|_2
= \|Z\|_2  \min_{\substack{q\in\Pi_k,\\q(0)=1}} \left(\sum_{j = 1}^n |f_j q(\lambda_j)|^2 \right)^{\frac{1}{2}},
\end{equation}
where $F:= \diag{f_1, \dotsc, f_n}$.
Thus, as in Section~\ref{sec:gmres_bound}, we  have a weighted polynomial approximation.
However,  the weights
may all be large because $My_i$ is not close to an eigenvector of $A - \sigma M$ in general.

Considering~\eqref{eq:2.6}, we find that an analogous bound is
\begin{equation}\label{eq:2.6gep}
\begin{aligned}
\|r_k\|_2
&\le  \|Z\|_2  \min_{\substack{q\in\Pi_{k-1},\\q(0)=1}}
\Big(\sum_{j = 2}^n |\tilde f_j q(\lambda_j)|^2 \Big)^{\frac{1}{2}}\\
&\le \|Z\|_2 \|\tilde f\|_2 \min_{\substack{q\in\Pi_{k-1},\\q(0)=1}}
\max_{\lambda\in\{\lambda_2,\dotsc,\lambda_n\}}|q(\lambda)|,
\end{aligned}
\end{equation}
where $\tilde f_j = (1-\lambda_j/\lambda_1)f_j$.
Again, there is no reason for $\|\tilde f\|_2$ to be small.
We conclude that for generalized eigenproblems, unpreconditioned GMRES may not show a residual reduction similar to the case $M=I$.

One way to reintroduce this behavior is by using the tuned preconditioner $P_i = P(I - y_iy_i^H) + My_iy_i^H$, which satisfies $P_iy_i = My_i$ \cite{FrSp07b}. We stress again that application of the tuned preconditioner requires  only one extra matrix-vector
product $P^{-1}My_i$ per outer iteration \cite{FrSp07b}.
We also note that in the absence of a good preconditioner, the choice $P=I$ should
at least recover the behavior observed for  inverse iteration with unpreconditioned GMRES applied to  the standard eigenvalue problem.
It is also obvious that the nice properties of polynomial preconditioners do not hold for the generalized eigenproblem, since the weight vectors $f_j$
will be the same as in the unpreconditioned case.
One could  add tuning to a polynomial preconditioner, but from numerical experience this strategy was not competitive.

\subsection{Block GMRES}
\label{sec:bgmres}
The linear system in \gref{eq:linsys1} can be generalized to a block linear system,
that is, a linear system with multiple right-hand sides of the form
\begin{equation}
\label{eq:blocklinsys1}
BX_i = Y_i,
\end{equation}
where $B=A-\sigma I$ as before, and $Y_i$, $X_i\in\mathbb{C}^{n\times u}$, $u\lll n$. Systems of this form arise when seeking an invariant subspace using
inverse subspace iteration~\cite{RSS06}, a block version of the inverse iteration. If these systems are solved by block-GMRES, we obtain similar bounds to  \gref{eq:2.5} (see also Theorem 2.2
in~\cite{TPPW14}), and can gain insight into the convergence behavior of block-GMRES as the inverse subspace iterations progress.

\begin{lemma}\label{lem:gen_eig}
Suppose that $B$ is diagonalizable, that is $B=Z\Lambda Z^{-1}$, and block-GMRES is used to solve the linear system with multiple right-hand sides of the form $BX=Y$,
$Y\in\mathbb{C}^{n\times u}$. The residual $Y-BX_k$ associated with the approximate solution $X_k$, obtained with $k$ iterations of block-GMRES starting with
$X_0=0$ is such that
\begin{align*}
\|Y-BX_k\|_F& \le \|Z\|_2\min_{G_i\in\mathbb{C}^{u\times u}} \left\|  W+ \sum_{i=1}^k \Lambda^i WG_i\right\|_F\\
& = \|Z\|_2\min_{\substack{q_{\ell,j} \in \Pi_k,\\q_{\ell,j}(0) = \delta_{\ell,j}}}   \left\|\begin{bmatrix}  \sum_{\ell=1}^u q_{\ell,1}(\Lambda)w_\ell\\
 \vdots\\
  \sum_{\ell=1}^u q_{\ell,u}(\Lambda)w_\ell\\
 \end{bmatrix}
 \right\|_2,
\end{align*}
where $ W = Z^{-1}Y\in\mathbb{C}^{n\times u}$, $w_\ell$ is the $\ell$th column of $W$, $\Pi_k$ is the space of polynomials of at most degree $k$ and
$\delta_{\ell,j}$ is the Kronecker delta.
\end{lemma}
\begin{proof}
Since $X_k\in\text{span}\{Y,BY,\ldots, B^{k-1}Y\}$ and block-GMRES minimizes the residual we have
\begin{align*}
\|Y-BX_k\|_F
& \le \min_{G_i\in\mathbb{C}^{u\times u}}\|Y+\sum_{i=1}^k B^i Y G_i\|_F \\
&=\min_{G_i\in\mathbb{C}^{u\times u}} \left\|  (I\otimes Z) \vecop{W+ \sum_{i=1}^k \Lambda^i WG_i}\right\|_2\\
& \le \|Z\|_2\min_{G_i\in\mathbb{C}^{u\times u}} \left\|  W+ \sum_{i=1}^k \Lambda^i WG_i\right\|_F,
\end{align*}
where we have used vectorization and the properties of the Kronecker product in the second step. If we introduce the additional matrix $G_0 = I_u$ then
\[\left(W+ \sum_{i=1}^k \Lambda^i WG_i\right)e_j = \sum_{i=0}^k \Lambda^i WG_i e_j = \sum_{i=0}^k \Lambda^i \sum_{\ell=1}^u g_{\ell j}^{({i})}w_\ell =
\sum_{\ell=1}^u q_{\ell,j}(\Lambda)w_\ell  ,\]
where $e_j$ is the $j$th unit vector, $w_\ell$ is the $\ell$th column of $W$, $g_{\ell j}^{(i)}$ is the $(\ell,j)$th element of $G_i$ and  $q_{\ell,j}(\lambda)
= \sum_{i=0}^k \lambda^i g_{\ell j}^{(i)}$, $j,\ell = 1,\dotsc,u$.  Clearly $q_{\ell,j} \in \Pi_k$. Additionally, the condition $G_0=I_u$ means that
$q_{\ell,j}(0) = \delta_{\ell,j}$.

Thus,
\begin{align*}
 \left\|  W+ \sum_{i=1}^k \Lambda^i WG_i\right\|_F
 &= \left\|\begin{bmatrix} \sum_{\ell=1}^u q_{\ell,1}(\Lambda)w_\ell, & \dotsc, & \sum_{\ell=1}^u q_{\ell,u}(\Lambda)w_\ell \end{bmatrix}  \right\|_F\\
 &= \left\|\begin{bmatrix}  \sum_{\ell=1}^u q_{\ell,1}(\Lambda)w_\ell\\
 \vdots\\
  \sum_{\ell=1}^u q_{\ell,u}(\Lambda)w_\ell\\
 \end{bmatrix}
 \right\|_2,
 \end{align*}
 from which the result follows.
\end{proof}

A consequence of Lemma~\ref{lem:gen_eig} is the following block-GMRES bound.
\begin{theorem}
\label{th:gen_eig}
Suppose that $B$ is diagonalizable, that is $B=Z\Lambda Z^{-1}$, and block-GMRES is used to solve the linear system with multiple right-hand sides of the form $BX=Y$,
$Y\in\mathbb{C}^{n\times u}$. The residual $Y-BX_k$ associated with the approximate solution $X_k$, obtained with $k$ iterations of block-GMRES starting with
$X_0=0$, is such that
\begin{equation}
\label{eq:boundgen_eig}
\|Y-BX_k\|_2\le \|Z\|_2  \min_{\substack{q\in\Pi_k,\\q(0)=1}} \left(\sum_{\ell=1}^u\sum_{j = 1}^n |w_{j\ell} q(\lambda_j)|^2 \right)^{\frac{1}{2}}
\end{equation}
where $Z^{-1}Y = W$, and $w_{j\ell}$ is the $(j,\ell)$th entry of $W$.
\end{theorem}
\begin{proof}
In Lemma~\ref{lem:gen_eig}, replace the minimizing polynomial $q$ by $\hat q$, where $\hat q_{\ell,j}(\lambda) \equiv 0$, $j\ne \ell$ and
\[\hat q_{11}(\lambda) = \hat q_{22}(\lambda) = \dots = \hat q_{uu}(\lambda) \equiv \hat q(\lambda).\]
(Note that this is equivalent to choosing $G_i = g_i I$, $g_i$ constant, $i = 1,\dotsc, k$.)
Then,
\[\|Y-BX_k\|_F \le \|Z\|_2 \min_{\substack{q\in\Pi_k,\\q(0)=1}} \left\|\begin{bmatrix}  q(\Lambda)w_1\\
 \vdots\\
q(\Lambda)w_u\\
 \end{bmatrix}
 \right\|_2\!
=\!\|Z\|_2 \min_{\substack{q\in\Pi_k,\\q(0)=1}}\Big(\sum_{\ell=1}^u\sum_{j = 1}^n |w_{j\ell} q(\lambda_j)|^2\Big)^{\ensuremath{\frac{1}{2}}}\!\!.
\]
The result follows from $\|Y-BX_k\|_2\le\|Y-BX_k\|_F$.
\end{proof}

The key idea in Lemma~\ref{lem:gen_eig} and Theorem~\ref{th:gen_eig} is, again, to express the right-hand side $Y$ in the eigenvector basis of $B$,
i.e.,
\[
W = Z^{-1}Y = \begin{bmatrix}W_1\\W_2\end{bmatrix},
\]
where $W_1\in\mathbb{C}^{u\times u}$ and $W_2\in\mathbb{C}^{(n-u)\times u}$.
We write the right-hand side $Y_i\in\mathbb{C}^{n\times u}$ in the form
\[
Y_i = Z_1 Y_i^1 + Z_2 Y_i^2,
\]
where $Z_1\in\mathbb{C}^{n\times u}$ and $Z_2\in\mathbb{C}^{n\times (n-u)}$ are the first $u$ and the last $n-u$ columns of $Z$, that is $B = [Z_1\,Z_2]\Lambda
[Z_1\,Z_2]^{-1}$, and $Y_i^1 \in\mathbb{C}^{u\times u}$ is nonsingular and $Y_i^2\in\mathbb{C}^{(n-u)\times u}$.
In the generic situation, we have that inverse
subspace iteration converges, e.g. $\text{ran}(Y_i)$ converges to  $\text{ran}(Z_1)$ and hence $\|Y_i^2\|\rightarrow 0$. Hence for inverse subspace iteration
\[
W^{[i]} = Z^{-1}Y_i = Z^{-1} ( Z_1 Y_i^1 + Z_2 Y_i^2)= E_1 Y_i^1 + E_2 Y_i^2 = \begin{bmatrix}W_1^{[i]}\\W_2^{[i]}\end{bmatrix},
\]
where $E_1$ and $E_2$ are the first $u$ and the last $n-u$ columns of the identity matrix respectively. As $\|Y_i^2\|\rightarrow 0$ we have, in the limit,
\[
W^{[i]}\approx \begin{bmatrix}W_1^{[i]}\\0\end{bmatrix},
\]
where $W_1^{[i]}\in\mathbb{C}^{u\times u}$ and otherwise $\|W_1^{[i]}\|\gg \|W_2^{[i]}\|$, similar to the case where $u=1$ in the main part of this paper. In  light of
Lemma~\ref{lem:gen_eig} and Theorem~\ref{th:gen_eig} this means that a lot of entries of $w_\ell^{[i]}$, the $\ell$th column of $W^{[i]}$ are small or zero. In
the limit the bound in \eqref{eq:boundgen_eig} becomes
\[
\label{eq:boundgen_eig2}
\|Y_i-BX_k\|_2\le \|Z\|_2  \min_{\substack{q\in\Pi_k,\\q(0)=1}} \left(\sum_{\ell=1}^u\sum_{j = 1}^u |w_{j\ell}^{[i]} q(\lambda_j)|^2 \right)^{\frac{1}{2}}
\]
and hence the minimizing polynomial will focus on minimizing over the relevant sought spectrum, e.g. $q(\lambda_1)\,\ldots,q(\lambda_u)$.

This property is violated when a preconditioner is applied in block GMRES, but can be overcome, in a similar way to the case $u=1$, by a tuned preconditioner,
e.g. by using a preconditioner $P_i$ which satisfies $P_iY_i = BY_i$, see~\cite{RSS06} for details.

It is also possible to use block GMRES within inverse subspace iteration for generalized eigenvalue problems. In this case the linear system that must be solved is of the form $(A-\sigma M)X_i = MY_i$, where again $X_i,Y_i\in \mathbb{C}^{n\times u}$. Assuming that $A-\sigma M = Z\Lambda Z^{-1}$ is diagonalizable, our results carry over to this case, and we obtain the following theorem.

\begin{corollary}
\label{th:gen_eig_blk}
Suppose that $A-\sigma M$ is diagonalizable, that is $A-\sigma M=Z\Lambda Z^{-1}$, and block-GMRES is used to solve the linear system with multiple right-hand sides of the form $(A-\sigma M)X=MY$,
$Y\in\mathbb{C}^{n\times u}$. The residual $MY-(A-\sigma M)X_k$ associated with the approximate solution $X_k$, obtained with $k$ iterations of block-GMRES starting with
$X_0=0$, is such that
\begin{equation}
\label{eq:boundgen_eig_gen}
\|MY-(A-\sigma M)X_k\|_2\le \|Z\|_2  \min_{\substack{q\in\Pi_k,\\q(0)=1}} \left(\sum_{\ell=1}^u\sum_{j = 1}^n |h_{j\ell} q(\lambda_j)|^2 \right)^{\frac{1}{2}}
\end{equation}
where $Z^{-1}MY = H$, and $h_{j\ell}$ is the $(j,\ell)$th entry of $H$.
\end{corollary}
\begin{proof}
The proof is the same as the proofs of Lemma~\ref{lem:gen_eig} and Theorem~\ref{lem:gen_eig} with $W$ replaced by $H$ throughout.
\end{proof}

As in Section~\ref{subsec:gen_eig}, we see that although the block GMRES residual can be expressed in terms of a weighted polynomial approximation, there is no
reason for any of the weights to be small. However, tuning can also be applied within block GMRES to accelerate convergence, as in~\cite{ElmXue11}.

\section{Numerical experiments} \label{sec:num_ex}
In this section we consider GMRES convergence within inverse iteration and the bounds previously
discussed for nonsymmetric matrices whose properties are summarized in Table~\ref{t:ex1}.
The matrix \texttt{cd\_fd}~is taken from~\cite{GolubYe00, frei07}, whereas \texttt{cdde1}{} and \texttt{olm2000}{}
~are from the matrix market.
In Table~\ref{t:ex1}, the spectral norms and condition numbers of the eigenvector matrices of the unpreconditioned $B$ as well as $BP^{-1}$ for different
standard preconditioners
are listed. The standard preconditioners $P$ are incomplete LU factorizations with three different drop tolerances $\theta$.
To mimic a similar increasing quality of the polynomial preconditioners, the polynomial degrees are set to $d=5,10,15$. (Note that the polynomial
preconditioned matrices have the same eigenvector matrix as $B$.)
For the coefficients of $p(B)$, we first generate a small number of Ritz and harmonic Ritz values of $B$.
If these Ritz values are all real or only have small imaginary parts, the coefficient generation \eqref{polycoeffcheb} based on the reciprocal Chebychev nodes
is used, whereas the approach \eqref{polycoeffmin} by \cite{Gij95} is used in the case of complex Ritz values. The standard and polynomial preconditioners are kept
unchanged during the outer iteration, i.e.\ they are computed only once at the start.
We employ two variants of the tuned preconditioners, which satisfy $P_iy_i=y_i$ and $P_iy_i=Ay_i$.
We also investigated tuning with $P_iy_i = (\gamma^{(i)} - \sigma) y_i$ as in~\eqref{eq:tunedpract} but the results were similar to the simpler $P_iy_i=y_i$ and
so have not been included.

Note that we selected test examples of comparably small sizes in order to be able to compute the eigendecompositions needed for the weight
vectors $w$. The effects and performance gains resulting from the application of tuned preconditioners have been demonstrated with large matrices, e.g.\ in
\cite{SzyX11,Mar16}.

For each problem we run inverse iteration with an
initial eigenvector approximation of $y_0 = \frac{1}{n}[ 1, \ 1, \ \dotsc, \ 1]^T$.
We let $\rho_i$ be the inverse iteration residual, so that
\begin{equation}\label{eq:iires}
\rho_i = Ax_{i} - \lambda_{i} x_{i}.
\end{equation}
The shifts $\sigma$ are as in Chapter 6 of~\cite{frei07}.
We solve the linear system using GMRES with a zero initial guess and a decreasing tolerance of
$\tau = \min\{\delta,\delta\|\rho_{i-1}\|_2\}$.
We choose $\delta = 0.1$ except for \texttt{cdde1}{} ($\delta = 0.001$);
these are the same values as in Chapter~6 of~\cite{frei07}.

\begin{table}
\footnotesize
\setlength{\tabcolsep}{0.5em}
\centering
\caption{Matrices used in examples.}
\label{t:ex1}
\begin{tabular}{l r r r |r r | rr | rr }
& & \multicolumn{2}{c}{Unpreconditioned} & \multicolumn{2}{|c}{$P$ ($10^{-1}$)} & \multicolumn{2}{|c}{$P$ ($10^{-2}$)} & \multicolumn{2}{|c}{$P$ ($10^{-3}$)}\\
Matrix & $n$ & $\|Z\|_2$ & $\kappa_2(Z)$ & $\|Z\|_2$ & $\kappa_2(Z)$ & $\|Z\|_2$ & $\kappa_2(Z)$ & $\|Z\|_2$ & $\kappa_2(Z)$\\
\hline
\input{Tab/IICond.tex}
\end{tabular}
\end{table}
At first we have a detailed look at the progress of the outer iteration for the matrix \texttt{cdde1}{}~using no preconditioner,
ILU with $\theta=10^{-2}$ and the corresponding two tuned variants, and a polynomial preconditioner with deg$(p)=d=10$.
 Tables~\ref{t:cdde:U}--\ref{t:cdde:T} summarize the changes in the relevant quantities as the outer iteration proceeds, including the quality of the
eigenpair approximations, the components in $w^{[i]}$, the constants in the GMRES bounds \eqref{eq:2.5} and~\eqref{eq:2.6}, and the number of executed GMRES steps. For the tuned preconditioners, also the progress of the spectral norms and condition numbers of the eigenvector matrices is given. Note that for clarity we drop the outer iteration index $i$ from $w$ for the remainder of this section.

\begin{table}
\footnotesize
\setlength{\tabcolsep}{0.5em}
\centering
\caption{\texttt{cdde1} without preconditioning, with untuned ILU preconditioning (drop tolerance $10^{-2}$) and with polynomial preconditioning
(polynomial degree 10): inverse iteration residual $\|\rho_{i-1}\|_2$, sizes of components of $w$, constants in~\eqref{eq:2.5} and~\eqref{eq:2.6} and
GMRES iteration numbers at each outer iteration $i$.}
\label{t:cdde:U}
\begin{tabular}{*{10}{l}}
 & $i$ & $\lambda^{(i)}$ & $\|\rho_{i-1}\|_2$ & $|w_1|$ & $\|{w^{(2)}}\|_2$ & $\|Z\|_2\|w\|_2$ &  $\|Z\|_2 \|\tilde w\|_2$ & $\tfrac{\|Z\|_2\|\tilde w\|_2}{\tau}$
&it\\
\hline
\input{./Tab/cdde11e-2Comb1.tex}
\end{tabular}
\end{table}

\begin{table}
\scriptsize
\setlength{\tabcolsep}{0.25em}
\centering
\caption{\texttt{cdde1} with tuned preconditioning ($Py_i = Iy_i$ and $Py_i = Ay_i$): inverse iteration residual $\|\rho_{i-1}\|_2$, eigenvector matrix norm
$\|Z\|_2$ and condition number $\kappa_2(Z)$, sizes of components of $w$, constants in~\eqref{eq:2.5} and~\eqref{eq:2.6} and
GMRES iteration numbers at each outer iteration $i$. The ILU drop tolerance is $10^{-2}$.}
\label{t:cdde:T}
\begin{tabular}{*{12}{l}}
 &$i$&$\lambda^{(i)}$&$\|\rho_{i-1}\|_2$&$\|Z\|_2$&$\kappa_2(Z)$&$|w_1|$&$\|{w^{(2)}}\|_2$&
 {\smaller $\|Z\|_2\|w\|_2$}
 &
 {\smaller $\|Z\|_2\|\tilde w\|_2$}
 &$\tfrac{\|Z\|_2\|\tilde w\|_2}{\tau}$&it\\
\hline
\input{./Tab/cdde11e-2Comb2.tex}
\end{tabular}
\end{table}

\begin{figure}
\centering
\begin{subfigure}[b]{0.45\textwidth}
\includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/cdde1P0I1.pdf}
\caption{No preconditioner, $k=1$}
\end{subfigure}
\begin{subfigure}[b]{0.45\textwidth}
\includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/cdde1P0I8.pdf}
\caption{No preconditioner, $k=8$}
\end{subfigure}

\begin{subfigure}[b]{0.45\textwidth}
\includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/cdde11e-2P2aI1.pdf}
\caption{Tuned ILU ($Py_i = y_i$), $k=1$}
\end{subfigure}
\begin{subfigure}[b]{0.45\textwidth}
\includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/cdde11e-2P2aI8.pdf}
\caption{Tuned ILU ($Py_i = y_i$), $k=8$}
\end{subfigure}

\begin{subfigure}[b]{0.45\textwidth}
\includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/cdde11e-2P2dI1.pdf}
\caption{Tuned ILU ($Py_i = Ay_i$), $k=1$}
\end{subfigure}
\begin{subfigure}[b]{0.45\textwidth}
\includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/cdde11e-2P2dI8.pdf}
\caption{Tuned ILU ($Py_i = Ay_i$), $k=8$}
\end{subfigure}

\begin{subfigure}[b]{0.45\textwidth}
\includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/cdde11e-2P3I1.pdf}
\caption{Polynomial, $k=1$}
\end{subfigure}
\begin{subfigure}[b]{0.45\textwidth}
\includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/cdde11e-2P3I8.pdf}
\caption{Polynomial, $k=8$}
\end{subfigure}

\caption{Relative residual norms, bounds~\eqref{eq:2.5} and \eqref{eq:2.6a}, and the constants $\|Z\|_2\|w\|_2$ and $\|Z\|_2\|\tilde w\|_2$ for \texttt{cdde1}
for different preconditioners. The ILU drop tolerance is $10^{-2}$ and deg$(p)=10$.}
\label{f:cdde1_residuals}
\end{figure}

Obviously, the values of $\lambda^{(i)}$, $\|\rho_{i}\|_2$ at each outer iteration reveal that using different  preconditioners does not hamper  the convergence
of
the eigenpairs. It is also apparent that $w\rightarrow e_1$, as the outer iteration $i$ proceeds, in the unpreconditioned case (Table~\ref{t:cdde:U}). This
behavior is somehow destroyed
by the
standard preconditioner (Table~\ref{t:cdde:U}) but mimicked by the two tuned preconditioners (Table~\ref{t:cdde:T}) and exactly
matched for the polynomial one (Table~\ref{t:cdde:U}). Using either tuned or polynomial preconditioners also has a positive effect on the constants in
\eqref{eq:2.5} and~\eqref{eq:2.6}, which are larger for the standard preconditioner. Most importantly, the number of GMRES iteration steps is notably
reduced and remains at an approximately constant level for tuned and polynomial preconditioners. The increasing trend for the standard preconditioner is also
evident.

A visual illustration of these observations is given in Figures~\ref{f:cdde1_residuals} and \ref{f:cdde:1e-2}. Figure~\ref{f:cdde1_residuals} shows the
behavior of the GMRES residuals for no preconditioner,  standard, tuned, and polynomial preconditioners. As discussed in Section~\ref{sec:initdecrease} we
observe the initial decrease of both the GMRES residual and the bound in \eqref{eq:2.6a}  (when no preconditioner or the tuned preconditioner is used), which is
particularly prominent in the later stages of the iteration.
Figure~\ref{f:cdde:1e-2} shows the history of components of $w$ as the outer iteration proceeds in
the two top plots.
The bottom plots show the required number of GMRES steps against the outer iteration (left plot) as well as the eigenvalue residual norm against the
cumulative sum of inner GMRES steps (right plot). The significant reduction of inner iterations by tuned and polynomial preconditioners is apparent.

\begin{figure}
    \centering
    \begin{subfigure}[b]{0.45\textwidth}
        \includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/cdde11e-2w1.pdf}
    \end{subfigure}
    \begin{subfigure}[b]{0.45\textwidth}
        \includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/cdde11e-2w2.pdf}
    \end{subfigure}
    \begin{subfigure}[b]{0.45\textwidth}
        \includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/cdde11e-2GIts.pdf}
    \end{subfigure}
        \begin{subfigure}[b]{0.45\textwidth}
        \includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/cdde11e-2OuterRes.pdf}
    \end{subfigure}
    \caption{\texttt{cdde1}: $|w_1|$, $\|w_2\|$, number of GMRES iterations and outer residual. The ILU drop tolerance is $10^{-2}$ and deg$(p)=10$.}
    \label{f:cdde:1e-2}
\end{figure}

Figures~\ref{f:cdfd:1e-2} and \ref{f:olm:1e-2} show similar plots for the matrices \texttt{cd\_fd}{}~and \texttt{olm2000}{} using the same preconditioning settings.
For \texttt{cd\_fd}{}, using a standard preconditioner results in a drastic increase in the magnitude of the components of the weight vector $w$ (top plots of
Figure~\ref{f:cdfd:1e-2}). The beneficial effects of tuned and polynomial preconditioners are similar to the previous examples.
For the matrix \texttt{olm2000}{}, the tuned preconditioners lead to a slight increase of the outer iteration steps (14 compared to 11 for the other choices).
However, the amount of work in terms of the number of required inner iteration steps is still smaller than for the other variants (bottom plots of
Figure~\ref{f:olm:1e-2}). The polynomial preconditioner seems to be of lesser quality compared to other preconditioners for the matrix \texttt{olm2000}{}, leading to more
inner iterations. Increasing the polynomial degree did not lead to improvements. It seems that for this example, the basic strategies mentioned in
Section~\ref{ssec:polyprec} to select the coefficients of the preconditioning polynomial are not sufficient. For these cases, this highlights an advantage of
tuned preconditioners over polynomial preconditioning, especially for tuned preconditioners that are built from standard preconditioning approaches, e.g.
incomplete factorizations, which can be constructed in a much more automatic and
straightforward manner.

\begin{figure}
    \centering
    \begin{subfigure}[b]{0.45\textwidth}
        \includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/cd_fd1e-2w1.pdf}
    \end{subfigure}
    \begin{subfigure}[b]{0.45\textwidth}
        \includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/cd_fd1e-2w2.pdf}
    \end{subfigure}
    \begin{subfigure}[b]{0.45\textwidth}
        \includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/cd_fd1e-2GIts.pdf}
    \end{subfigure}
        \begin{subfigure}[b]{0.45\textwidth}
        \includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/cd_fd1e-2OuterRes.pdf}
    \end{subfigure}
    \caption{\texttt{cd\_fd}: $|w_1|$, $\|w_2\|$, number of GMRES iterations and eigenvalue residual. The ILU drop tolerance is $10^{-2}$ and deg$(p)=10$.}
     \label{f:cdfd:1e-2}
\end{figure}

\begin{figure}
    \centering
    \begin{subfigure}[b]{0.45\textwidth}
        \includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/olm20001e-2w1.pdf}
    \end{subfigure}
    \begin{subfigure}[b]{0.45\textwidth}
        \includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/olm20001e-2w2.pdf}
    \end{subfigure}
    \begin{subfigure}[b]{0.45\textwidth}
        \includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/olm20001e-2GIts.pdf}
    \end{subfigure}
    \begin{subfigure}[b]{0.45\textwidth}
            \includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/olm20001e-2OuterRes.pdf}
    \end{subfigure}
    \caption{\texttt{olm2000}: $|w_1|$, $\|w_2\|$, number of GMRES iterations and eigenvalue residual. The ILU drop tolerance is $10^{-2}$ and deg$(p)=10$.}
     \label{f:olm:1e-2}
\end{figure}

\begin{table}[p]
\footnotesize
\centering
\caption{Results for all examples: final weight components $|w_1|$, $\|w^{(2)}\|_2$, and total numbers of outer and inner (GMRES)
iterations.}
\label{t:all}
\begin{tabular}{*{7}{l}}
ex.& prec. & $\theta, d$& final $|w_1|$ &final  $\|w^{(2)}\|_2$ & outer & $\sum$ inner\\
\hline
\input{./Tab/results_all.tex}
\end{tabular}
\end{table}

For all examples and all used preconditioners, Table~\ref{t:all} gives a condensed overview of the obtained results, listing
the final magnitudes of the weight components in $w$, and the total numbers of outer and inner (GMRES) iterations.
While the majority of the results are in line with the previous observations, there are some outliers.
For the matrices \texttt{cd\_fd}~and \texttt{cdde1}, the tuned preconditioners built from the ILU with drop tolerance $\theta=10^{-1}$ lead to much smaller magnitudes of the
weights
compared to standard preconditioner, but the components of $w$ do not have the property $|w_1|>\|w^{(2)}\|_2$.
Decreasing the drop tolerance to $\theta=10^{-2},~10^{-3}$ leads to the desired situation $w\rightarrow e_1$.
Moreover, increasing the degree of the polynomial preconditioner is not always beneficial as we see in all three examples for $d=15$.
With this setting the polynomial preconditioner leads to a worse performance compared to lower polynomial degrees.
As we mentioned above, the origin of this behavior can is the basic coefficient generation approaches, which appear to return more and more very tiny
coefficients if the degree is increased. Given the difficulties with the polynomial preconditioner in these cases, we recommend to either
look for better coefficient selection strategies or use a tuned
preconditioner.

We now transform \texttt{cd\_fd}{} into a generalized problem by adding an artificial, tridiagonal matrix $M$ having the values 2.5, 5, 2.5 on the lower, main, and upper
diagonal, respectively. The shift is $\sigma=30$ but the remaining settings to execute this test are  unchanged. The result are illustrated in
Figure~\ref{f:Gcdfd}. As predicted, the two upper plots show that, in contrast to the situation $M=I$, the weight vector
in the unpreconditioned and polynomial preconditioned case does not converge to $e_1$. This weight behavior is only achieved by employing tuned preconditioners.
The bottom left plot also shows a increasing number of inner iterations when no or a polynomial preconditioner is used.

\begin{figure}
    \centering
    \begin{subfigure}[b]{0.45\textwidth}
        \includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/G_cd_fd1e-1w1.pdf}
    \end{subfigure}
    \begin{subfigure}[b]{0.45\textwidth}
        \includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/G_cd_fd1e-1w2.pdf}
    \end{subfigure}
    \begin{subfigure}[b]{0.45\textwidth}
        \includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/G_cd_fd1e-1GIts.pdf}
    \end{subfigure}
        \begin{subfigure}[b]{0.45\textwidth}
        \includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/G_cd_fd1e-1OuterRes.pdf}
    \end{subfigure}
    \caption{Results for the generalized problem $A=$\texttt{cd\_fd}, $M=\mathrm{tridiag}(2.5,5,2.5)$: weight components $|f_1|$, $\|f_2\|$, number of GMRES
iterations and eigenvalue residual. The ILU drop tolerance is $10^{-1}$ and deg$(p)=5$.}
     \label{f:Gcdfd}
\end{figure}

Even though this paper concentrates on GMRES bounds for inverse iteration, we finally show results of block-GMRES convergence for inverse subspace iteration. We
consider the matrix \texttt{cdde1}{} with the settings from above~and we seek an invariant subspace of dimension $u=6$. The drop tolerance of the incomplete LU
factorization used for the preconditioner is $10^{-2}$ and we apply a tuned preconditioner which satisfies $P_iY_i = AY_i$, e.g.\ $P_i = P+(A-P)Y_i(Y_i^T
Y_i)^{-1}Y_i^T$. For the implementation of block GMRES we used an algorithm discussed in \cite{Sood15,Sood16}.

Figure~\ref{f:block} shows the history of the norms of $W_1$ and $W_2$, the splittings of the matrix $W\in\mathbb{C}^{n\times u}$ discussed in
Section~\ref{sec:bgmres}, as the outer iteration proceeds. The bottom plots show the required number of block GMRES steps against the outer iteration (left
plot)
and the residual norm against the cumulative sum of inner block GMRES steps (right plot). As expected the reduction of inner iteration numbers by the tuned
preconditioner is apparent. The reduction of $\|W_2\|$ can be seen for the unpreconditioned as well as tuned case, the magnitude of $\|W_1\|$ is close to
2.449, but the overall behavior of the weights is similar to the single vector inverse iteration.
\begin{figure}[t]
    \centering
    \begin{subfigure}[b]{0.45\textwidth}
        \includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/bg_cdde1_W1.pdf}
    \end{subfigure}
    \begin{subfigure}[b]{0.45\textwidth}
       \includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/bg_cdde1_W2.pdf}
    \end{subfigure}
    \begin{subfigure}[b]{0.45\textwidth}
        \includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/bg_cdde1_innout.pdf}
    \end{subfigure}
        \begin{subfigure}[b]{0.45\textwidth}
        \includegraphics[width=\textwidth,trim = 1cm 6.5cm 1cm 6.5cm,clip=true]{./Pic/bg_cdde1_resinn.pdf}
    \end{subfigure}
    \caption{Inverse subspace iteration for \texttt{cdde1}, $u=6$: $\|W_1\|$, $\|W_2\|$, number of block-GMRES iterations and eigenvalue residual. The ILU drop
tolerance is $10^{-2}$.}
     \label{f:block}
\end{figure}

\section{Conclusion}

In this paper we discussed the convergence behavior of GMRES (as a prominent iterative method) for solving linear systems that arise during the solution of
eigenvalue problems via inverse iteration. We gave detailed bounds on GMRES that take the special behavior of the right hand side into account and explained the
initial sharp decrease of the GMRES residual. The bounds gave rise to adapted preconditioners for GMRES when applied to eigenvalue problems, e.g. tuned and
polynomial preconditioners. The analysis was extended to inverse iteration for the generalized eigenvalue problem and subspace iteration, where block GMRES
bounds were given. The numerical results showed that the new GMRES (block GMRES) bounds are much sharper than  conventional bounds and that preconditioned
subspace iteration with either a tuned or polynomial preconditioner should be used, where the tuned preconditioner is generally easier
to construct. Possible future research perspectives should therefore, e.g., include generation strategies of high quality polynomial coefficients adapted to the
outer eigenvalue iteration.

\paragraph*{Acknowledgement.}
 The authors thank Kirk Soodhalter (Radon Institute for
Computational and Applied Mathematics (RICAM)) for kindly providing his Matlab implementation of block-GMRES.
\input{gmres_eval_fin.bbl}
\end{document}