The Stability of the RLS Algorithm for Equalization
Short Answer:
Basic RLS (no forgetting, no weird weighting, etc.) is ALWAYS Lyapunov stable. If the regressor sequence for the LS problem is persistently exciting--which is data and problem dependent, not algorithm dependent--then RLS is exponentially stable. So I don't know what you mean by &quot;LMS is more stable than RLS&quot;--more stable in what sense?
More Details:
Basic RLS has the discrete-time update equations
\begin{align}
%
\theta_{k+1} &amp;= \theta_k + P_{k+1}\phi_k^T(y_k-\phi_k\theta_k), \\
%
P_{k+1} &amp;= P_k - P_k\phi_k^T(I+\phi_kP_k\phi_k^T)^{-1}\phi_kP_k. 
\end{align}
Note that the covariance update is decoupled from the estimate update and is also monotonically decreasing. If the regressor sequence is persistently exciting, then the origin of the covariance update is exponentially stable. If nothing can be said about the regressor, then the origin of the covariance update is only guaranteed to be Lyapunov stable.
Next, define the error $\tilde \theta_k = \theta_k-\theta$, where $\theta$ is the true value of the parameters. Then you can show that
$$\tilde \theta_{k+1} = P_{k+1}P_0^{-1}\theta_0.$$
In other words, the error is entirely dictated by the covariance, and hence is also exponentially stable or Lyapunov stable as the covariance is.