title: Approximate Message Passing algorithms for rotationally invariant matrices
authors: Zhou Fan
text: 
  
  
  Approximate Message Passing (AMP) algorithms have seen widespread use across a
  variety of applications. However, the precise forms for their Onsager
  corrections and state evolutions depend on properties of the underlying random
  matrix ensemble, limiting the extent to which AMP algorithms derived for white
  noise may be applicable to data matrices that arise in practice.
  
  In this work, we study more general AMP algorithms for random matrices 
  that satisfy orthogonal rotational invariance in law, where  may have a
  spectral distribution that is different from the semicircle and Marcenko-Pastur laws characteristic of white noise. The Onsager corrections and state
  evolutions in these algorithms are defined by the free cumulants or
  rectangular free cumulants of the spectral distribution of . Their forms
  were derived previously by Opper, Çakmak, and Winther using non-rigorous
  dynamic functional theory techniques, and we provide rigorous proofs.
  
  Our motivating application is a Bayes-AMP algorithm for Principal Components
  Analysis, when there is prior structure for the principal components (PCs)
  and possibly non-white noise. For sufficiently large signal strengths and any
  non-Gaussian prior distributions for the PCs, we show that
  this algorithm provably achieves higher estimation accuracy than the sample PCs.
  
  
  Statistical inference for principal components of spiked covariance matrices
      Ke Wangt2[label=e2]kewang@ust.hk
      Aug. 2020
  ============================================================================
  
  
  
  
  
  
  
  § INTRODUCTION
  
  
  Approximate Message Passing (AMP) algorithms are a general family of iterative
  algorithms that have seen widespread use in a variety of applications. 
  First developed for Bayesian linear regression and compressed sensing in
  <cit.>, they have since
  been applied to many high-dimensional problems arising in statistics and
  machine learning, including Lasso estimation and sparse linear regression
  <cit.>, generalized linear models and phase
  retrieval <cit.>,
  robust linear regression <cit.>, sparse or structured principal
  components analysis (PCA)
  <cit.>, group synchronization problems <cit.>,
  deep learning <cit.>,
  and optimization in spin glass models
  <cit.>. We
  refer to <cit.> for a recent review.
  
  In their basic form as described in <cit.>, given a data
  matrix ∈^m × n and an initialization _̆1 ∈^m, an AMP
  algorithm consists of the iterative updates
  
      _t   =^⊤_̆t-b_t_̌t-1
      _̌t   =v_t(_t)
      _t   =_̌t-a_t_̆t
      _̆t+1   =u_t+1(_t).
  
  Here, a_t,b_t ∈ are two sequences of debiasing coefficients, and
  v_t:→ and u_t+1:→ are two sequences of 
  functions applied entrywise to _t ∈^n and _t ∈^m.
  By appropriately designing these functions v_t and u_t+1, possibly
  to also depend on additional “side information” such as response variables in
  regression problems, this basic iteration may be
  applied to perform optimization or Bayes posterior-mean estimation in the
  above applications.
  
  A defining characteristic of the AMP algorithm is the subtraction of the
  two “memory” terms b_t_̌t-1 and a_t_̆t in the definitions of
  _t and _t, known as the Onsager corrections.
  This achieves the effect of removing a bias of 
  ^⊤_̆t and _̌t in the directions of the preceding iterates, so
  that as m,n →∞, the empirical distributions of
  _t and _t converge to certain Gaussian limits
  
      _t →(0,σ_t^2)    and   _t →(0,ω_t^2).
  
  This was proven rigorously in the Sherrington-Kirkpatrick model by
  Bolthausen in <cit.> and for general AMP algorithms of
  the above form by Bayati and Montanari in <cit.>,
  and various extensions have been established in
  <cit.>. The description of the variances
  σ_t^2 and ω_t^2 across iterations is known as the algorithm's
  state evolution. This ability to characterize the distributions of the
  iterates is a major appeal of the AMP approach, and has enabled a more
  precise theoretical understanding of many high-dimensional statistical
  estimators and the development of associated inference procedures that
  quantify statistical uncertainty
  <cit.>.
  
  A drawback of AMP algorithms, however, is that the correct forms of the
  debiasing coefficients a_t,b_t and resulting variances
  σ_t^2,ω_t^2 depend on the properties of the
  data matrix . When  has i.i.d. (0,1/n) entries,
  these quantities are given explicitly by
  
      a_t=⟨ v_t'(_t) ⟩,     
      b_t=γ⟨ u_t'(_t-1) ⟩,     σ_t^2=⟨ v_t(_t)^2 ⟩,     ω_t^2=γ⟨ u_t(_t-1)^2 ⟩
  
  where γ=m/n, u_t'(·),v_t'(·),u_t(·)^2,v_t(·)^2
  denote the derivatives and squares of u_t,v_t applied entrywise, and
  ⟨·⟩ denotes the empirical average of coordinates. It has
  been shown in <cit.> that
  these forms enjoy a certain amount of universality, being valid also for 
  having i.i.d. non-Gaussian entries. Extensions to  having independent
  entries with several blocks of differing variances were derived in
  <cit.>. Unfortunately, these results
  do not apply to  with more complex correlation structure, which is
  common in data applications. A sizeable body of work has developed alternative
  algorithms or damping procedures to address this shortcoming
  <cit.>,
  and the connections between several of these algorithms were discussed recently in
  <cit.>. However, many such algorithms are no longer
  characterized by a rigorous state evolution, and some have been empirically
  observed to exhibit slow convergence or divergent behavior.
  
  
  
   §.§ Contributions
  
  
  We develop a rigorous extension of general AMP procedures of the above form to
  rotationally invariant matrices. We then apply these general
  algorithms to a prototypical “structured PCA”
  problem of estimating a rank-one matrix in (possibly non-white) noise.
  In this PCA application, we develop a Bayes-AMP algorithm that provably
  achieves lower mean-squared-error than the rank-one estimate constructed from
  the sample principal components (PCs), for any sufficiently large signal
  strength and any prior distributions of the PCs that are not mean-zero Gaussian
  laws.
  
  Let us first describe the general AMP algorithm in the
  simpler setting of symmetric square matrices. We study matrices
  ∈^n × n that satisfy the equality in law
  
      L=Ø̃^⊤Ø̃
  
  for any deterministic orthogonal matrix Ø̃∈^n × n.
  Equivalently, such matrices admit the eigendecomposition =Ø^⊤Ø
  where the eigenvectors Ø∈^n × n are independent of the
  eigenvalues  and are uniformly distributed on the orthogonal group.
  The AMP algorithm will take the form
  
      _t   =_̆t-b_t1_̆1-b_t2_̆2-…-b_tt_̆t
      _̆t+1   =u_t+1(_1,_2,…,_t)
  
  where the coefficients b_ts are defined so that each _t has an
  empirical Gaussian limit as in (<ref>).
  For greater generality and applicability,
  we will allow u_t+1:^t → to be
  a function of all previous iterates _1,…,_t, rather than only the
  preceding iterate _t. (Outside of the 
  i.i.d. Gaussian setting, the full debiasing of _̆t by
  _̆1,…,_̆t is necessary even if u_t+1(·) 
  depends only on _t.) The correct forms for b_t1,…,b_tt and the
  corresponding state evolution
  
      (_1,_2,…,_t) →(0,_t)
  
  were derived previously by Opper, Çakmak, and Winther using non-rigorous
  dynamic functional theory techniques <cit.>. These forms
  depend on the free cumulants of the eigenvalue distribution of , and we
  describe them in Section <ref>. Our work provides a rigorous
  proof of the validity of this state evolution.
  
  In the rectangular setting, we study bi-rotationally invariant matrices
  ∈^m × n satisfying the equality in law
  
      L=Ø̃^⊤
  
  for any deterministic orthogonal matrices Ø̃∈^m × m and
  ∈^n × n. Equivalently, such matrices
  admit the singular value decomposition =Ø^⊤ where the singular
  vectors Ø∈^m × m and ∈^n × n are independent
  of the singular values  and are both uniformly distributed over the
  orthogonal groups. The analogous AMP algorithm takes the form
  
      _t   =^⊤_̆t-b_t1_̌1-b_t2_̌2-…-b_t,t-1_̌t-1
      _̌t   =v_t(_1,_2,…,_t)
      _t   =_̌t-a_t1_̆1-a_t2_̆2-…-a_tt_̆t
      _̆t+1   =u_t+1(_1,_2,…,_t)
  
  We describe in Section <ref> the forms of the
  debiasing coefficients a_ts,b_ts and the corresponding state evolutions
  
      (_1,…,_t) →(0,_t),      (_1,…,_t) →(0,_t),
  
  which are related to the rectangular free cumulants of the singular value
  distribution of  as introduced in
  <cit.>.
  This algorithm has also been derived recently and independently
  in <cit.>, using a dynamic functional theory approach
  similar to <cit.>.
  
  These classes of rotationally invariant matrices include, but are not
  restricted to,  having i.i.d. Gaussian entries. Importantly,
  the spectral distribution of  can be arbitrary, rather than following the
  behavior prescribed by the semicircle or Marcenko-Pastur law. Our primary
  motivation for studying such rotationally invariant models is that we expect
  the resulting AMP algorithms to be valid under a much larger universality class 
  of matrices  than AMP algorithms derived in the i.i.d. Gaussian setting,
  and that this class may provide a more flexible model for data matrices arising
  in practice.
  
  In the contexts of compressed sensing and generalized linear models,
  alternative “vector AMP” or “orthogonal AMP” approaches for
  rotationally-invariant matrices have been developed in
  <cit.>, and rigorous state evolutions for these algorithms were also derived.
  These derivations are based on analyses of denoising functions that satisfy
  the divergence-free conditions
  
      ⟨∂_s v_t(_1,…,_t) ⟩=0,     ⟨∂_s u_t+1(_1,…,_t) ⟩=0      for all  s ≤ t.
  
  A similar idea was used
  in <cit.> to develop an algorithm for solving the
  TAP equations for Ising models with rotationally-invariant couplings.
  Analyses of certain “long-memory” Convolutional AMP algorithms for
  compressed sensing, related to our work, were recently carried out in
  <cit.> by
  mapping these algorithms to a divergence-free form. Our proofs build on the
  insight in <cit.> that Bolthausen's
  conditioning technique may be applied to rotationally-invariant models. However,
  we derive directly the forms of the Onsager corrections and state evolutions
  for AMP algorithms that do not restrict v_t(·) and u_t+1(·) to be
  divergence-free, in a general setting that extends beyond compressed sensing
  applications. We clarify the relation between certain long-memory algorithms and
  the AMP algorithms of <cit.> for Gaussian matrices, by
  relating their Onsager corrections and state evolutions to the free cumulants 
  of the spectral distribution of .
  
  
  
   §.§ Organization of paper
  
  Section <ref> establishes preliminary background and notation
  on Wasserstein convergence of empirical measures and free cumulants. Section
  <ref> first discusses the specific application of structured PCA and
  the Bayes-AMP algorithms for this application that specialize the more general
  AMP algorithms to follow. Section <ref> describes the general AMP
  algorithm and state evolution for symmetric square matrices, and Section
  <ref> describes the analogous general algorithm for rectangular
  matrices. Section <ref> provides a high-level overview of the proofs,
  which are contained in the Supplementary Appendices.
  
  
  
  § PRELIMINARIES ON WASSERSTEIN CONVERGENCE AND FREE
  PROBABILITY
  
  
  
  
   §.§ Notation and conventions
  
  
  For vectors ∈̌^n and ∈^m, we denote
  
      ⟨⟩̌=1/n∑_i=1^n v_i,
          ⟨⟩=1/m∑_i=1^m w_i.
  
  For a matrix (_̌1,…,_̌k) ∈^n × k and a function f:^k
  →, we write f(_̌1,…,_̌k) ∈^n
  as its row-wise evaluation.
  
  For a weakly differentiable function u:^k →, we denote by
  ∂_s u (any version of) its s^th partial derivative. 
  For a matrix (_̌1,…,_̌k) ∈^n × k, we write
  _(_̌1,…,_̌k)∈^n × n for the orthogonal projection
  onto the linear span of (_̌1,…,_̌k), and
  _(_̌1,…,_̌k)^⊥=-_(_̌1,…,_̌k) for the
  projection onto its orthogonal complement.  is the identity matrix, and we
  write _k × k to specify the dimension k. We will use the
  convention
  
      ^0=
  
  for the zero-th power of any square matrix , even if some eigenvalues of
   may be 0.
  
  Products over the empty set are equal to 1, and sums over the
  empty set are equal to 0.
  · is the ℓ_2 norm for vectors and ℓ_2 →ℓ_2
  operator norm for matrices. _∞=max_i |v_i| is the vector
  ℓ_∞ norm, and _F=(∑_i,j m_ij^2)^1/2 is the matrix
  Frobenius norm.
  
  
  
   §.§ Wasserstein convergence of empirical
  distributions
  
  
  
  For p ≥ 1, a matrix
  (_̌1,…,_̌k)=(v_i,1,…,v_i,k)_i=1^n
  ∈^n × k, and a probability distribution ℒ
  over ^k or a random vector (V_1,…,V_k) ∼ℒ, we write
  
      (_̌1,…,_̌k) W_p→ℒ     or      (_̌1,…,_̌k) W_p→ (V_1,…,V_k)
  
  for the convergence of the empirical distribution of rows of
  (_̌1,…,_̌k) to ℒ in the Wasserstein space of order p.
  This means, for any C>0 and continuous function f:^k →
  satisfying
  
      |f(v_1,…,v_k)| ≤ C(1+(v_1,…,v_k)^p),
  
  as n →∞,
  
      1/n∑_i=1^n f(v_i,1,…,v_i,k) →[f(V_1,…,V_k)].
  
  Implicit in this notation is the finite moment condition
  _(V_1,…,V_k) ∼ℒ[(V_1,…,V_k)^p]<∞. 
  
  We write
  
      (_̌1,…,_̌k) ℒ     or     
      (_̌1,…,_̌k)  (V_1,…,V_k)
  
  to mean that this convergence holds for every fixed p ≥ 1,
  where ℒ has finite moments of all orders.
  
  
  We will use a certain calculus associated to these notations
  W_p→ and , which we review in Appendix
  <ref>. By <cit.>, to
  show that (<ref>) holds for all continuous
  functions f satisfying (<ref>),
  it suffices to check that it holds for all bounded Lipschitz functions f
  together with the function f(v_1,…,v_k)=(v_1,…,v_k)^p. See
  Chapter 6 of <cit.> for further background.
  
  
  
   §.§ Free cumulants
  
  
  We briefly review the notion of free cumulants, and refer readers to
  <cit.> for a more thorough and motivated introduction.
  
  Let X be a random variable with finite moments of all orders, and denote
  m_k=[X^k]. In what follows, the law of X will be the empirical
  eigenvalue distribution of a symmetric matrix ∈^n × n. Let
  (k) be the set of all non-crossing partitions of {1,…,k}. 
  The free cumulants κ_1,κ_2,κ_3,… of X
  are defined recursively by the moment-cumulant relations
  
      m_k=∑_π∈(k)∏_S ∈πκ_|S|
  
  where |S| is the cardinality of the set S ∈π.
  The first four free cumulants may be computed to be
  
      κ_1   =m_1=[X]
      κ_2   =m_2-m_1^2=[X]
      κ_3   =m_3-3m_2m_1+2m_1^3
      κ_4   =m_4-4m_3m_1-2m_2^2+10m_2m_1^2-5m_1^4,
  
  where κ_4 is the first free cumulant that differs from the classical
  cumulants. The free cumulants linearize free additive convolution, describing
  the eigenvalue distribution of sums of freely independent symmetric square
  matrices. If X has the Wigner semicircle law supported on [-2,2], then
  
      κ_1=0,     κ_2=1,     κ_j=0    for all  j ≥ 3.
  
  
  Defining the formal generating functions
  
      M(z)=1+∑_k=1^∞ m_k z^k,
           R(z)=∑_k=1^∞κ_k z^k-1,
  
  the relations (<ref>) are equivalent to an identity
  of formal series (see <cit.>)
  
      M(z)=1+zM(z) · R(zM(z)).
  
  Here, R(z) is the R-transform of X.
  Comparing the coefficients of z^k on both sides,
  each free cumulant κ_k may be computed from m_1,…,m_k and
  κ_1,…,κ_k-1 as
  
      κ_k=m_k-[z^k]∑_j=1^k-1κ_j (z+m_1z^2+m_2z^3+…+m_k-1z^k)^j
  
  where [z^k](q(z)) denotes the coefficient of z^k in the polynomial q(z).
  
  
  
   §.§ Rectangular free cumulants
  
  
  For rectangular matrices ∈^m × n, we review the notion
  of rectangular free cumulants developed in <cit.>.
  This is an example of the operator-valued free cumulants described in
  <cit.>, where freeness is with amalgamation over a
  2-dimensional subalgebra corresponding to the 2 × 2 block structure of
  ^(m+n) × (m+n). 
  
  We fix an aspect ratio parameter
  
      γ=m/n>0.
  
  Let X be a random variable with finite moments of all orders, and denote the
  even moments by m_2k=[X^2k]. The law of X^2
  will be the empirical eigenvalue distribution of
  ^⊤∈^m × m, so that m_2k is the k^th moment
  of this distribution. Define also an auxiliary sequence of even moments
  
      m̅_2k= 1     if  k=0 
      γ· m_2k    if  k ≥ 1.
  
  Since the eigenvalues of ^⊤ and ^⊤ coincide up to the addition
  or removal of |m-n| zeros,
  the value m̅_2k is the k^th moment of the empirical
  eigenvalue distribution of ^⊤∈^n × n.
  
  Let '(2k) be the non-crossing partitions π of
  {1,…,2k} where each set S ∈π has even cardinality. Then we may
  define two sequences of rectangular free cumulants
  κ_2,κ_4,κ_6,… and
  κ̅_2,κ̅_4,κ̅_6,… by the moment-cumulant
  relations
  
      m_2k   =∑_π∈'(2k)∏_S ∈π_min S  is
      oddκ_|S|·∏_S ∈π_min S  is evenκ̅_|S|
      m̅_2k   =∑_π∈'(2k)∏_S ∈π_min S  is oddκ̅_|S|·∏_S ∈π_min S  is evenκ_|S|
  
  See <cit.>. These cumulants have a simple
  relation given by
  
      κ̅_2k=γ·κ_2k   for all  k ≥ 1,
  
  so outside of the proofs, we will always refer to the first sequence
  {κ_2k}_k ≥ 1 for simplicity.
  
  Letting e(π) be the number of sets S ∈π where the smallest element
  of S is even, and letting o(π) be the number where the smallest element
  is odd, applying (<ref>) above implies
  
      m_2k=∑_π∈'(2k)γ^e(π)∏_S ∈πκ_|S|,
          m̅_2k=∑_π∈'(2k)γ^o(π)∏_S ∈πκ_|S|.
  
  See also <cit.>.
  The first four rectangular free cumulants may be computed as
  
      κ_2   =m_2=[X^2]
      κ_4   =m_4-(1+γ)m_2^2
      κ_6   =m_6-(3+3γ)m_4m_2
      +(2+3γ+2γ^2)m_2^3
      κ_8   =m_8-(4+4γ)m_6m_2
      -(2+2γ)m_4^2+(10+16γ+10γ^2)m_4m_2^2
         -(5+10γ+10γ^2+5γ^3)m_2^4.
  
  The rectangular free cumulants linearize rectangular free additive
  convolution, describing the singular value distribution of sums of freely
  independent rectangular matrices. If X^2 has the Marcenko-Pastur law with
  aspect ratio γ, then
  
      κ_2=1,     κ_2j=0    for all  j ≥ 2.
  
  
  The rectangular free cumulants may be computed
  from the following relation of generating functions: Let
  
      M(z)=∑_k=1^∞ m_2kz^k,
           R(z)=∑_k=1^∞κ_2k z^k.
  
  Here, R(z) is the rectangular R-transform of X. Then
  
      M(z)=R(z(γ M(z)+1)(M(z)+1)),
  
  see <cit.>.
  Thus, comparing the coefficients of z^k on both sides, each value
  κ_2k may be computed from m_2,…,m_2k and
  κ_2,…,κ_2k-2 as
  
      κ_2k=m_2k-[z^k]∑_j=1^k-1κ_2j(z(γ M(z)+1)(M(z)+1))^2j
  
  where [z^k](q(z)) again denotes the coefficient of z^k in the
  polynomial q(z).
  
  
  The reasons for the appearance of the square/rectangular free cumulants 
  in the forms of the Onsager corrections and state evolution for AMP are somewhat
  opaque in our work, as they will arise from a certain combinatorial unfolding
  of the moment-cumulant relations on the non-crossing partition lattice; we
  discuss this further in Section <ref>. Their emergence is
  conceptually clearer in the (non-rigorous, but illuminating)
  analysis of the limit characteristic function of the AMP iterates in
  <cit.>, where they arise instead from the evaluation of a
  low-rank HCIZ integral over the Haar-orthogonal randomness in , and from the
  coefficients of the series expansion of the R-transform that describes this
  integral. See <cit.> and
  <cit.> for this connection in the square and rectangular
  settings, respectively.
  
  
  
  
  § STRUCTURED PRINCIPAL COMPONENTS ANALYSIS
  
  
  We study the problem of estimating a rank-one signal matrix in possibly
  non-white noise, where the singular vectors of the rank-one signal have some
  “prior” structure. For sufficiently large signal strengths, we describe a 
  Bayes-AMP algorithm that provably achieves lower mean-squared-error than the
  rank-one estimate constructed from the sample principal components. This
  extends the types of AMP algorithms that were studied for i.i.d. Gaussian
  noise in <cit.>.
  
  
  
   §.§ Symmetric square matrices
  
  
  Suppose first that we observe a symmetric data matrix
  
      =α/n_̆*_̆*^⊤+∈^n × n
  
  and seek to estimate _̆* ∈^n. Writing the eigendecomposition
  =Ø^⊤Ø where =(), we assume that  is
  rotationally-invariant in law and that as n →∞,
  the empirical distributions of  and _̆* satisfy
  
      Λ,     _̆*  U_*
  
  for two limit laws Λ and U_*. This notation  denotes
  Wasserstein convergence at all orders, as discussed in
  Section <ref>. To fix the scaling, we take _̆*=√(n),
  so that
  
      [U_*^2]=lim_n →∞1/n_̆*^2=1.
  
  Here, the law of Λ is the limit spectral distribution of . The law
  of U_* represents a prior distribution for the entries of _̆*, which
  may reflect assumptions of sparsity <cit.>,
  non-negativity <cit.>, or a discrete support that encodes
  cluster or community membership <cit.>.
  
  We assume for simplicity that we have an initialization
  _̆1 ∈^n independent of ,
  satisfying the joint empirical convergence
  
      (_̆1,_̆*)  (U_1,U_*),     [U_1U_*]>0.
  
  We then estimate _̆* by the iterates _̆t of an AMP algorithm
  
      _t   =_̆t-b_t1_̆1-…-b_tt_̆t
      _̆t+1   =u_t+1(_t).
  
  It will be shown that each iterate _t behaves like _̆* corrupted by
  entrywise Gaussian noise, so we take each function u_t+1(·) to be a
  scalar denoiser that estimates _̆* from _t.
  
  To describe the forms of the debiasing coefficients
  b_t1,…,b_tt, let us write
  λ_1() ≥…≥λ_n() as the eigenvalues of
  . For each k ≥ 1, let
  
      m_k=1/n∑_i=2^n λ_i()^k
  
  be the k^th moment of the empirical eigenvalue distribution of 
  excluding its largest eigenvalue. Let {κ_k}_k ≥ 1 be the free
  cumulants corresponding to this sequence of moments {m_k}_k ≥ 1, as
  defined in Section <ref>. It is easy to check that under the
  assumption (<ref>), as n →∞,
  
      m_k → m_k^∞=[Λ^k],
          κ_k →κ_k^∞
  
  for each fixed k ≥ 1, where these limits are the moments and free
  cumulants of the limit spectral distribution Λ of the noise . The
  debiasing coefficients in (<ref>) are set as
  
      b_tt=κ_1,      b_t,t-j=κ_j+1∏_i=t-j+1^t ⟨
      u_i'(_i-1) ⟩ for  j=1,…,t-1.
  
  
  The state evolution that describes the AMP iterations
  (<ref>–<ref>) is
  expressed in terms of a sequence of mean vectors
  _T^∞=(μ_t^∞)_1 ≤ t ≤ T and covariance matrices
  _T^∞=(σ_st^∞)_1 ≤ s,t ≤ T, defined
  recursively as follows: For T=1, we set
  
      μ_1^∞=α·[U_1U_*],
          σ_11^∞=κ_2^∞[U_1^2].
  
  Having defined _T^∞ and _T^∞, we denote
  
      U_t=u_t(F_t-1)  for  t=2,…,T+1,     
      (F_1,…,F_T)=_T^∞· U_*+(Z_1,…,Z_T),
       and
  
  
      (Z_1,…,Z_T) ∼(0,_T^∞)  independent of  (U_1,U_*).
  
  We then define _T+1^∞ and _T+1^∞ to have the
  entries, for 1 ≤ s,t ≤ T+1,
  
      μ_t^∞   =α·[U_tU_*]
      σ_st^∞   =∑_j=0^s-1∑_k=0^t-1κ_j+k+2^∞(∏_i=s-j+1^s [u_i'(F_i-1)] )
      (∏_i=t-k+1^t [u_i'(F_i-1)])[U_s-jU_t-k].
  
  In the limit n →∞, the iterates of (<ref>) will satisfy the
  second-order Wasserstein convergence
  
      (_1,…,_T,_̆*)  (F_1,…,F_T,U_*).
  
  Thus, the rows of (_1,…,_T) behave like Gaussian vectors with mean
  _T^∞· U_* and covariance _T^∞.
  
  As one example of choosing the functions u_t+1(·),
  let us analyze this state evolution for the following “single-iterate
  posterior mean” denoisers: In the scalar Gaussian observation model
  
      F=μ· U_*+Z,      Z ∼(0,σ^2)  independent of  U_*,
  
  we denote the Bayes posterior-mean estimate of U_* as
  
      η(f |μ,σ^2)=[U_* | F=f]
      =[U_* exp(-(f-μ· U_*)^2/2σ^2)]/[exp(-(f-μ· U_*)^2/2σ^2)].
  
  We denote the Bayes-optimal mean-squared-error of this estimate as
  
      (μ^2/σ^2)
      =[(U_*-η(F |μ,σ^2))^2].
  
  The single-iterate posterior mean denoiser is the choice
  
      u_t+1(f_t)=η(f_t |μ_t^∞,σ_tt^∞)
  
  where μ_t^∞ and σ_tt^∞ are the above state evolution
  parameters that describe the univariate Gaussian law of F_t. These
  parameters may be replaced by consistent estimates in practice.
  
  Let R(x) be the R-transform of the limit spectral distribution Λ,
  as discussed in Section <ref>,
  and let R'(x) be its derivative. For small |x|, these may be defined by the
  convergent series (see Proposition <ref>)
  
      R(x)=∑_k=1^∞κ_k^∞ x^k-1.
  
  
  
  Suppose =Ø^⊤Ø∈^n × n where Ø is a
  Haar-uniform orthogonal matrix. Let =(), where
  (,_̆1,_̆*) are independent of Ø,
  _̆*=√(n), and
  
      Λ,      (_̆1,_̆*)  (U_1,U_*)
  
  almost surely as n →∞. Suppose [U_1^2] ≤ 1,
  [U_1U_*]=>0, and
  _∞≤ C_0 almost surely for all large n and some
  constants C_0,>0.
  
  
    * Let α≥ 0, and let each function u_t+1(·) be
  continuously differentiable and Lipschitz on . Then for each fixed
  T ≥ 1, almost surely as n →∞,
  
      (_̆1,…,_̆T+1,_1,…,_T,_̆*) 
      (U_1,…,U_T+1,F_1,…,F_T,U_*)
  
  where the joint law of this limit is described by (<ref>).
  
    * Suppose each function u_t+1(·) is the posterior-mean denoiser in
  (<ref>), and suppose this is Lipschitz on . Then
  there exist constants C,α_0>0 depending only
  on C_0, such that for all α>α_0,
  defining I_Δ=[1-C/α^2,1] and
  I_Σ=[κ_2^∞/2,3κ_2^∞/2], there is a unique fixed point
  (Δ_*,Σ_*) ∈ I_Δ× I_Σ to the equations
  
      1-Δ_*=(α^2 Δ_*^2/Σ_*),
          Σ_*=Δ_* R'(αΔ_*(1-Δ_*)/Σ_*).
  
  Furthermore,
  
      lim_T →∞(lim_n →∞1/n_̆T^⊤_̆*)
      =lim_T →∞(lim_n →∞1/n_̆T^2)=Δ_*.
  
  
  
  
  The proof of this result is provided in Appendix <ref>. As
  discussed in Section <ref>, the notation  in part (a)
  guarantees that for any continuous function f:^2T+2→ satisfying
  [f(U_1,…,U_T+1,Z_1,…,Z_T,U_*)^2]<∞,
  
      ⟨ f(_̆1,…,_̆T+1,_1,…,_T,_̆*)⟩→[f(U_1,…,U_T+1,Z_1,…,Z_T,U_*)]
  
  where the left side is the empirical average of this function f evaluated
  across the n rows.
  
  
  Theorem <ref>(b) implies that the asymptotic matrix mean-squared-error
  of the rank-one estimate _̆T_̆T^⊤ for _̆*_̆*^⊤,
  in the limit T →∞, is given by
  
      MSE   ≡lim_T →∞(
      lim_n →∞1/n^2_̆T_̆T^⊤-_̆*_̆*^⊤_F^2)
         =lim_T →∞(lim_n →∞1/n^2(_̆T^2)^2
      -2/n^2(_̆T^⊤_̆*)^2+1/n^2(_̆*^2)^2)
      =1-Δ_*^2.
  
  Let us compare this with the matrix mean-squared-error of the best PCA estimate
  c ·__^⊤ optimized over c>0,
  where _ is the leading sample eigenvector of . Normalizing
  _ such that _=_̆*=√(n), 
  <cit.> shows for sufficiently large
  α that
  
      lim_n →∞(1/n_^⊤_̆*)^2
      =Δ_≡-1/α^2 G'(G^-1(1/α)),
  
  where G(z)=[(z-Λ)^-1] is the Cauchy transform of Λ, and
  G^-1(z) is the functional inverse of G (which is
  well-defined for small |z|). Then
  
      MSE_   ≡min_c>0(lim_n →∞1/n^2c ·__^⊤-_̆*_̆*^⊤_F^2
      )
         =min_c>0 c^2-2cΔ_+1=1-Δ_^2,
  
  with the minimum attained at the rescaling c=Δ_<1.
  
  To see that 1-Δ_*^2 ≤ 1-Δ_^2,
  observe that for any prior distribution U_* satisfying our normalization
  [U_*^2]=1, we have
  
      (μ^2/σ^2) ≤1/1+μ^2/σ^2.
  
  This is because under the scalar observation model (<ref>),
  the right side of (<ref>) is the risk [(Û-U_*)^2]
  of the linear estimator Û=(μ/(σ^2+μ^2))F,
  which upper bounds the Bayes risk on the left side of (<ref>).
  Equality holds in (<ref>) if and only if Û is the Bayes
  estimator in this model, i.e. if and only if the prior distribution
  is U_* ∼(0,1). Applying (<ref>) to the
  first equation of (<ref>) and rearranging, we obtain
  
      αΔ_*(1-Δ_*)/Σ_*≤1/α.
  
  Now applying this to the second equation of (<ref>), and
  using that κ_2^∞=[Λ]>0 so that the function
  xR'(x)=κ_2^∞ x+2κ_3^∞ x^2+3κ_4^∞ x^3+… is
  increasing in a neighborhood of 0, we have for α>α_0 sufficiently
  large that
  
      1-Δ_*=1/α·αΔ_*(1-Δ_*)/Σ_*R'(
      αΔ_*(1-Δ_*)/Σ_*)
      ≤1/α^2R'(1/α).
  
  Differentiating the R-transform identity R(x)=G^-1(x)-1/x, this
  is equivalently written as
  
      Δ_* ≥ 1-1/α^2R'(1/α)
      =-1/α^2 G'(G^-1(1/α))=Δ_,
  
  so that
  
      MSE=1-Δ_*^2 ≤ 1-Δ_^2=MSE_
  
  as desired. Equality holds here if and only if equality holds in
  (<ref>), i.e. when U_* ∼(0,1). Thus, for any
  signal strength α>α_0 sufficiently large and any distribution of
  U_* other than (0,1), the above AMP algorithm achieves strictly better
  estimation accuracy than PCA.
  
  
  An illustration of the algorithm and state evolution is presented in the left
  panel of Figure <ref>, with noise eigenvalues drawn from a centered and
  rescaled Beta(1,2) distribution. We observe a close agreement
  with the state evolution predictions at sample size n=2000, and a significant
  improvement in estimation accuracy over the naive principal components for this
  prior distribution U_* ∼Uniform{+1,-1}. Let us remark
  that although carrying out many iterations of this AMP algorithm would require
  estimating successively higher-order free cumulants of the spectral
  distribution of , for large signal strengths α the algorithm only
  needs a very small number of iterations to converge.  
  
  
  In this algorithm, the Onsager corrections involving the free cumulants 
  may be understood as iteratively constructing the series (<ref>)
  for R(αΔ_*(1-Δ_*)/Σ_*), whose derivative appears
  in the characterization of the fixed-point in Theorem <ref>. This is
  somewhat analogous to the single-step-memory algorithm in
  <cit.> for solving the TAP equations in a related Ising model,
  which alternatively constructs a series for the inverse R-transform.
  
  The convergence condition and final mean-squared-error of this algorithm
  are likely not Bayes-optimal. For example, we believe that the convergence of
  (<ref>–<ref>) requires convergence of the series
  (<ref>) at x=αΔ_*(1-Δ_*)/Σ_*, which (depending
  on the spectral law of ) may impose a stronger condition for the signal
  strength α than the spectral phase transition. One
  natural way to improve upon the algorithm is to consider more generally
  
      u_t+1(_1,…,_t)=η(c_t1_1+…+c_tt_t |_t^⊤_t^∞, _t^⊤_t^∞_t)
  
  for a vector _t=(c_t1,…,c_tt) in each iteration, or specialize
  this to _t=(_t^∞)^-1_t^∞ to obtain
  the posterior mean estimate of U_* given all previous observations
  (F_1,…,F_t). Our general results describe also the state evolution for
  these extensions, but analyses of their fixed points are more
  involved, and we will not pursue this in the current work.
  
  These procedures differ from the “Vector AMP” or “memory-free” algorithms
  of <cit.>, whose forms may be derived from
  the Expectation Propagation framework of <cit.>. These latter
  algorithms operate directly on a resolvent of  and use divergence-free
  nonlinearities, corresponding to _t=_t=0 in our notations to follow.
  Thus their state evolutions have simpler forms that depend on the first two
  moments of the resolvent but not (explicitly) on the free cumulants
  of . PCA differs from the applications in
  <cit.> in two important
  ways: First, the log-likelihood of  given $̆ is not quadratic in$̆
  under general spectral laws of . Second, the noise matrix  is not
  directly observed in PCA, and its resolvent cannot be directly computed.
  Due to these differences, we believe that extending the algorithmic ideas
  of <cit.> to PCA may be an interesting
  open question to study in future work.
  
  
  
  
   §.§ Rectangular matrices
  
  
  
  
  Consider now a rectangular data matrix
  
      =α/m_̆*_̌*^⊤+∈^m × n,
  
  and the task of estimating _̆* ∈^m and _̌* ∈^n.
  Writing the singular value decomposition =Ø^⊤ where
  =() and ∈^min(m,n), we assume that
   is bi-rotationally invariant in law and that
  
      m/n=γ,
          Λ,     _̆*  U_*,     _̌*  V_*
  
  as m,n →∞,
  for some constant γ∈ (0,∞) and some limit laws
  Λ,U_*,V_*.
  We fix the scalings _̆*=√(m) and _̌*=√(n), so that
  
      [U_*^2]=[V_*^2]=1.
  
  Note that the rank-one signal component (α/m)_̆*_̌*^⊤ has singular
  value α/√(γ).
  
  We again assume that we have an initialization _̆1 ∈^m independent of
  , for which
  
      (_̆1,_̆*)  (U_1,U_*),     [U_1U_*]>0.
  
  We then estimate _̆* and _̌* by the iterates _̆t and _̌t of an
  AMP algorithm
  
      _t   =^⊤_̆t-b_t1_̌1-…-b_t,t-1_̌t-1
      _̌t   =v_t(_t)
      _t   =_̌t-a_t1_̆1-…-a_tt_̆t
      _̆t+1   =u_t+1(_t),
  
  where u_t+1(·) and v_t(·) are scalar denoisers that estimate
  _̆* and _̌* from _t and _t.
  
  To describe the forms of the debiasing coefficients a_ts and b_ts, let 
  us define _m ∈^m to be  if m ≤ n or 
  extended by m-n additional 0's if m>n. We will work instead with the limit
  
      _m Λ_m,
  
  which is a mixture of Λ and a point mass at 0 if γ=m/n>1.
  Denoting the singular values of  by λ_1() ≥…≥λ_min(m,n)(), for each k ≥ 1 we set
  
      m_2k=1/m∑_i=2^min(m,n)λ_i()^2k.
  
  We then define {κ_2k}_k ≥ 1 as the rectangular free cumulants
  associated to these even moments {m_2k}_k ≥ 1
  and aspect ratio γ, as defined in
  Section <ref>. It is easily checked that as m,n →∞,
  
      m_2k→ m_2k^∞=[Λ_m^2k],
          κ_2k→κ_2k^∞,
  
  where these limits are the even moments and rectangular free cumulants of
  Λ_m. Then the debiasing coefficients in
  (<ref>–<ref>) are set as
  
      a_t,t-j   =κ_2(j+1)⟨ v_t'(_t) ⟩∏_i=t-j+1^t ⟨ u_i'(_i-1)⟩⟨ v_i-1'(_i-1)
      ⟩   for    j=0,…,t-1,
      
      b_t,t-j   =γκ_2j⟨ u_t'(_t-1) ⟩∏_i=t-j+1^t-1⟨ v_i'(_i) ⟩⟨ u_i'(_i-1) ⟩   for    j=1,…,t-1.
  
  We use the convention that empty products equal 1, so the first coefficients
  here are simply
  
      a_tt=κ_2⟨ v_t'(_t) ⟩,     
      b_t,t-1=γκ_2 ⟨ u_t'(_t-1) ⟩.
  
  
  The state evolution for this algorithm may be expressed in terms of two
  sequences of mean vectors _T^∞=(μ_t^∞)_1 ≤ t ≤ T and
  _T^∞=(ν_t^∞)_1 ≤ t ≤ T and 
  covariance matrices
  _T^∞=(σ_st^∞)_1 ≤ s,t ≤ T and
  _T^∞=(ω_st^∞)_1 ≤ s,t ≤ T, defined as follows:
  For T=1 we set
  
      ν_1^∞=α·[U_1U_*],
          ω_11^∞=γκ_2^∞·[U_1^2].
  
  Having defined _T-1^∞, _T-1^∞,
  _T^∞, and _T^∞, we denote
  
      U_t=u_t(F_t-1)  for  t=2,…,T,     
      (F_1,…,F_T-1)=_T-1^∞· U_*+(Y_1,…,Y_T-1),
  
  
      (Y_1,…,Y_T-1) ∼(0,_T-1^∞)  independent of 
      (U_1,U_*),
  
  
      V_t=v_t(G_t)  for  t=1,…,T,     
      (G_1,…,G_T)=_T^∞· V_*+(Z_1,…,Z_T),
  
  
      (Z_1,…,Z_T) ∼(0,_T^∞)  independent of  V_*.
  
  We then define _T^∞ and _T^∞ with the entries, for
  1 ≤ s,t ≤ T,
  
      μ_t^∞   =(α/γ) ·[V_tV_*]
      σ_st^∞   =∑_j=0^s-1∑_k=0^t-1(∏_i=s-j+1^s [v_i'(G_i)][u_i'(F_i-1)])
      (∏_i=t-k+1^t [v_i'(G_i)][u_i'(F_i-1)])
         ·(κ_2(j+k+1)^∞[V_s-jV_t-k]
      +κ_2(j+k+2)^∞[v_s-j'(G_s-j)]
      [v_t-k'(G_t-k)][U_s-jU_t-k]).
  
  Now having defined _T^∞ and _T^∞, we extend
  (<ref>) to 
  
      U_t=u_t(F_t-1)  for  t=2,…,T+1,     
      (F_1,…,F_T)=_T^∞· U_*+(Y_1,…,Y_T),
  
  
      (Y_1,…,Y_T) ∼(0,_T^∞)  independent of 
      (U_1,U_*)
  
  and define _T+1^∞ and _T+1^∞ with the entries, for
  1 ≤ s,t ≤ T+1,
  
      ν_t^∞   =α·[U_tU_*]
      ω_st^∞   =γ∑_j=0^s-1∑_k=0^t-1(∏_i=s-j+1^s [u_i'(F_i-1)][v_i-1'(G_i-1)])
      (∏_i=t-k+1^t [u_i'(F_i-1)][v_i-1'(G_i-1)])
      
         ·(κ_2(j+k+1)^∞[U_s-jU_t-k]+κ_2(j+k+2)^∞[u_s-j'(F_s-j-1)]
      [u_t-k'(F_t-k-1)][V_s-j-1V_t-k-1]).
  
  We use the convention V_0=0, so that the second term of
  (<ref>) is 0 for j=s-1 or k=t-1. 
  In the limit m,n →∞, the iterates of
  (<ref>–<ref>) will satisfy
  
      (_1,…,_T,_̆*)  (F_1,…,F_T,U_*),
           (_1,…,_T,_̌*)  (G_1,…,G_T,V_*).
  
  
  As an example of choices for v_t(·) and u_t+1(·), let us again
  analyze the single-iterate posterior mean denoisers given by
  
      v_t(g_t)=η(g_t |ν_t,ω_tt),
           u_t+1(f_t)=η(f_t |μ_t,σ_tt),
  
  where η(·) is as defined in (<ref>), and
  (ν_t,ω_tt) and (μ_t,σ_tt) are the state evolution
  parameters describing the univariate Gaussian laws of G_t and F_t.
  We denote by (·) the scalar mean-squared-error function from
  (<ref>), and by R(x) the rectangular R-transform of
  Λ_m with aspect ratio γ, as discussed in Section
  <ref>. This may be defined for small
  |x| by the convergent series (see Proposition <ref>)
  
      R(x)=∑_k=1^∞κ_2k^∞ x^k,
  
  where κ_2k^∞ are the rectangular free cumulants of Λ_m
  above. We denote R'(x) as its derivative, and
  
      S(x)=(R(x)/x)'=xR'(x)-R(x)/x^2.
  
  
  
  Suppose =Ø^⊤∈^m × n where  and Ø are
  Haar-uniform orthogonal matrices. Let =(), where
  (,_̆1,_̆*,_̌*) are independent of (Ø,),
  _̆*=√(n), _̌*=√(m), and
  
      Λ,      (_̆1,_̆*)  (U_1,U_*),
          _̌*  V_*,      m/n=γ∈ (0,∞)
  
  as m,n →∞. Suppose [U_1^2] ≤ 1, [U_1U_*]=>0,
  and _∞≤
  C_0, almost surely for all large n and some constants C_0,>0.
  
  
    * Let α≥ 0, and let each function v_t(·) and
  u_t+1(·) be continuously differentiable and Lipschitz on . Then
  for each fixed T ≥ 1, almost surely as m,n →∞,
  
      (_̆1,…,_̆T+1,_1,…,_T,_̆*)    
      (U_1,…,U_T+1,F_1,…,F_T,U_*),
      
      (_̌1,…,_̌T,_1,…,_T,_̌*)    
      (V_1,…,V_T,G_1,…,G_T,V_*),
  
  where these limits are as defined in (<ref>) and
  (<ref>).
  
    * Suppose v_t(·),u_t+1(·) are the posterior-mean denoisers
  in (<ref>) and are Lipschitz on .
  There exist constants C,α_0>0 depending only on
  C_0,,γ such that for all α>α_0, setting
  
      I_Δ=I_Γ=[1-C/α^2,1],   
      I_Σ=[κ_2^∞/2,3κ_2^∞/2],
         I_Ω=γ· I_Σ,
  
  there is a unique fixed point
  (Δ_*,Σ_*,Γ_*,Ω_*,X_*) ∈ I_Δ× I_Σ×
  I_Γ× I_Ω× to the equations
  
      X_*   =α^2Δ_*Γ_*(1-Δ_*)(1-Γ_*)/γΣ_*Ω_*,  
      1-Δ_*=(α^2Γ_*^2/γ^2Σ_*),  
      1-Γ_*=(α^2Δ_*^2/Ω_*),
      Σ_*   =Γ_*R'(X_*)+α^2Δ_*^3(1-Γ_*)^2/Ω_*^2S(X_*),  Ω_*=γΔ_*R'(X_*)
      +α^2Γ_*^3(1-Δ_*)^2/γΣ_*^2S(X_*).
  
  Furthermore,
  
      lim_T →∞(lim_m,n →∞1/m_̆T^⊤_̆*)
         =lim_T →∞(lim_m,n →∞1/m_̆T^2)
      =Δ_*
      lim_T →∞(lim_m,n →∞1/n_̌T^⊤_̌*)
         =lim_T →∞(lim_m,n →∞1/n_̌T^2)
      =Γ_*.
  
  
  
  The proof of this result is provided in Appendix <ref>.
  
  
  As in the symmetric square setting of Remark <ref>,
  the above fixed points imply that the asymptotic matrix
  mean-squared-error is given by
  
      MSE≡lim_T →∞(lim_m,n →∞1/mn_̆T_̌T^⊤-_̆*_̌*^⊤_F^2)=1-Δ_*Γ_*.
  
  We may compare this with the asymptotic error of the PCA estimate: Assume
  without loss of generality that γ=m/n ≤ 1. Let _ and
  _ be the leading left and right singular vectors of , with the
  scalings _=_̆*=√(m) and
  _=_̌*=√(n). Recall that the singular value of the
  rank-one signal (α/m)_̆*_̌*^⊤ is α/√(γ), and set
  
      x=γ/α^2.
  
  Then <cit.> shows
  
      lim_m,n →∞(1/m_^⊤_̆*)^2
         =Δ_≡-2xφ(D^-1(x))/D'(D^-1(x))
      lim_m,n →∞(1/n_^⊤_̌*)^2
         =Γ_≡-2xφ̅(D^-1(x))/D'(D^-1(x))
  
  where
  
      φ(z)=[z/z^2-Λ_m^2],
        φ̅(z)=γφ(z)+1-γ/z,
         D(z)=φ(z)φ̅(z),
  
  and D^-1(z) is the functional inverse of D for small |z|. Then the
  matrix mean-squared-error for the best rescaling of the PCA estimate is
  
      MSE_   ≡min_c>0(lim_m,n →∞1/mnc·__^⊤
      -_̆*_̌*^⊤_F^2)
         =min_c>0(c^2-2c√(Δ_Γ_)+1)
      =1-Δ_Γ_
  
  with the minimum attained at c=√(Δ_Γ_).
  We verify in Appendix <ref> that for all
  α>α_0 sufficiently large, the fixed points of
  Theorem <ref>(b) satisfy
  
      MSE=1-Δ_*Γ_* ≤ 1-Δ_Γ_=MSE_,
  
  and that equality holds if and only if both U_* ∼(0,1) and V_* ∼(0,1). Thus, for sufficiently large signal strength and any
  non-Gaussian prior for either U_* or V_*, the above AMP algorithm achieves
  strictly better estimation accuracy than PCA.
  
  
  An illustration of this AMP algorithm and its state evolution is presented in
  the middle and right panels of Figure <ref>, with noise singular values
  drawn from a rescaled Beta(1,2) distribution. Again, close
  agreement with the state evolution predictions is observed at these sample sizes
  (m,n)=(2000,4000) and γ=1/2.
  
  
  
  § GENERAL AMP ALGORITHM FOR SYMMETRIC SQUARE MATRICES
  
  
  We now describe the general AMP algorithm for symmetric square matrices
  
      =Ø^⊤Ø∈^n × n,     =()
  
  and we state a formal theorem for its state evolution.
  
  We consider an initialization _̆1 ∈^n, and also a possible matrix of
  side information
  
      ∈^n × k
  
  for a fixed dimension k ≥ 0, both independent of .
  (We may take k=0 if there is no such
  side information.) Starting from this initialization _̆1,
  the AMP algorithm takes the form
  
      _t   =_̆t-b_t1_̆1-b_t2_̆2-…-b_tt_̆t
      _̆t+1   =u_t+1(_1,…,_t,)
  
  Each function u_t+1:^t+k→ is applied row-wise
  to (_1,…,_t,) ∈^n × (t+k).
  The debiasing coefficients b_t1,…,b_tt∈ are defined
  to ensure the empirical convergence
  
      (_1,…,_t) (0,_t^∞)
  
  as n →∞. The forms of b_t1,…,b_tt and _t^∞
  were first described in <cit.>, and we review this
  in the next section.
  
  
  
   §.§ Debiasing coefficients and limit covariance
  
  
  Define the t × t matrices
  
      _t=[   ⟨_̆1^2 ⟩ ⟨_̆1 _̆2 ⟩          ⋯ ⟨_̆1 _̆t ⟩;  ⟨_̆2_̆1 ⟩   ⟨_̆2^2 ⟩          ⋯ ⟨_̆2 _̆t
      ⟩;          ⋮          ⋮          ⋱          ⋮;  ⟨_̆t_̆1 ⟩  ⟨_̆t_̆2
      ⟩          ⋯   ⟨_̆t^2 ⟩ ],
        _t=[           0           0           ⋯           0           0;  ⟨∂_1 _̆2 ⟩           0           ⋯           0           0;  ⟨∂_1 _̆3 ⟩  ⟨∂_2 _̆3 ⟩           ⋯           0           0;           ⋮           ⋮           ⋱           ⋮           ⋮;  ⟨∂_1 _̆t ⟩  ⟨∂_2 _̆t ⟩           ⋯ ⟨∂_t-1_̆t ⟩           0 ]
  
  where _̆s_̆s'∈^n, _̆s^2 ∈^n, and ∂_s'_̆s
  ∈^n denote the entrywise product, square, and partial derivative with
  respect to z_s'. For each j ≥ 0, define
  
      ^(j)_t=∑_i=0^j _t^i _t (_t^j-i)^⊤.
  
  For example,
  
      _t^(0)=_t,   _t^(1)=_t_t
      +_t_t^⊤,   _t^(2)=_t^2_t
      +_t_t_t^⊤+_t(_t^2)^⊤.
  
  
  Let {κ_k}_k ≥ 1 be the free cumulants of the empirical
  eigenvalue distribution of . These are the free cumulants
  as defined in Section <ref>
  corresponding to the empirical moments
  
      m_k=1/n∑_i=1^n λ_i^k,
  
  where (λ_1,…,λ_n)= are the eigenvalues of .
  Then define two matrices _t and _t by
  
      _t=(∑_j=0^∞κ_j+1_t^j)^⊤,
          _t=∑_j=0^∞κ_j+2_t^(j).
  
  Here, _t may be interpreted as the R-transform applied to _t^⊤.
  Note that we write these as infinite series for
  convenience, but in fact the series are finite because _t^j=0 for
  all j ≥ t, and hence also 
  _t^(j)=0 for all j ≥ 2t-1. So for example,
  
      _1=κ_1_1 × 1,     _2=κ_1 _2 × 2
      +κ_2_2^⊤,
  
  
      _1=κ_2_1^(0),     _2=κ_2_2^(0)+κ_3_2^(1)
      +κ_4_2^(2).
  
  
  Each matrix _t is upper-triangular, which we may write entrywise as
  
      _t=[ b_11 b_21    ⋯ b_t1;      b_22    ⋯ b_t2;              ⋱    ⋮;                b_tt ].
  
  The debiasing coefficients in (<ref>) are defined to be
  the last column of _t. Note that
  the diagonal entries b_11,b_22,…,b_tt are all equal to κ_1,
  corresponding to the subtraction of κ_1_̆t in (<ref>) when the
  eigenvalue distribution of  has mean κ_1. If κ_1=0, then the
  debiasing for _̆t depends only on the previous iterates
  _̆1,…,_̆t-1.
  
  Under the conditions to be imposed in Assumption <ref>, all of the
  matrices _t, _t, _t, and
  _t will converge to deterministic t × t matrices in the
  n →∞ limit, which we denote as
  
      (_t^∞,_t^∞,_t^∞,_t^∞)
      =lim_n →∞ (_t,_t,_t,_t).
  
  This matrix _t^∞ is the covariance defining the state
  evolution of the iterates (_1,…,_t).
  All of our results will hold equally if the debiasing coefficients in
  (<ref>) are replaced by their limits b_ts^∞, or by any
  consistent estimates of these limits.
  
  We make two observations regarding this construction:
  
  
    * From the lower-triangular form of _t, one may check
  that the upper-left (t-1) × (t-1) submatrix of (_t^j)^⊤
  is (_t-1^j)^⊤, and similarly the upper-left (t-1) × (t-1)
  submatrix of _t^(j) is _t-1^(j). Thus, the upper-left
  submatrices of _t and _t coincide with
  _t-1 and _t-1.
  
    * For each iteration t ≥ 1, _t
  depends on  only via its first t free cumulants
  κ_1,…,κ_t, and _t depends on 
  only via its first 2t free cumulants κ_1,…,κ_2t.
  
  
  
  In the Gaussian setting of ∼GOE(n), where  has
  independent (0,1/n) entries above the diagonal and (0,2/n)
  entries on the diagonal, the limit spectral distribution of  is the Wigner
  semicircle law. The limits of the free cumulants κ_1,κ_2,… in
  this case are
  
      κ_1^∞=0,     κ_2^∞=1,     κ_j^∞=0      for all  j ≥ 2.
  
  This yields simply
  
      _t^∞=(_t^∞)^⊤,     _t^∞=_t^∞.
  
  If we further specialize to an algorithm where each _̆t depends only on
  the previous iterate _t-1, then ⟨∂_s _̆t ⟩=0 for
  s ≠ t-1, and this yields the Gaussian AMP algorithm
  
      _t   =_̆t-⟨∂_t-1_̆t ⟩_̆t-1
      _̆t+1   =u_t+1(_t,)
  
  as studied in <cit.> and
  <cit.>. Furthermore, the state
  evolution is such that each iterate _t has the empirical limit
  (0,σ_tt^∞), where
  σ_tt^∞=lim_n →∞⟨_̆t^2 ⟩.
  
  Note that outside of this Gaussian setting, we do not in general have
  the identity _t^∞=_t^∞, i.e. the empirical second
  moments of _1,…,_t do not coincide with those of _̆1,…,_̆t
  in the large-n limit, even if  is scaled so that κ_2=1.
  
  
  
  
   §.§ Main result
  
  
  We impose the following assumptions on the model (<ref>)
  and the AMP iterates (<ref>–<ref>). Note that here, we do not
  require the functions u_t+1(·) to be Lipschitz, but instead impose only
  the assumption (<ref>) of polynomial growth.
  
  
  
  
  
    * Ø∈^n × n is a random and Haar-uniform orthogonal matrix.
  
    * ∈^n is independent of Ø and satisfies
  Λ almost surely as n →∞, for a random variable
  Λ having finite moments of all orders.
  
    * _̆1 ∈^n and ∈^n × k are independent of Ø
  and satisfy (_̆1,)  (U_1,E) almost surely
  as n →∞, for a random vector
  (U_1,E) ≡ (U_1,E_1,…,E_k) having finite moments of all orders.
  
    * Each function u_t+1:^t+k→ satisfies (<ref>) for
  some C>0 and p ≥ 1.
  Writing its argument as (z,e) where z ∈^t and e ∈^k,
  u_t+1 is weakly differentiable in z and continuous in e.
  For each s=1,…,t, ∂_s u_t+1 also
  satisfies (<ref>) for some C>0 and p ≥ 1,
  and ∂_s u_t+1(z,e) is continuous
  at Lebesgue-a.e. z ∈^t for every e ∈^k.
  
    * [Λ]>0 and [U_1^2]>0. Letting (Z_1,…,Z_t) ∼(0,_t^∞) be independent of (U_1,E), each function
  u_t+1 is such that there do not exist constants
  α_1,…,α_t,β_1,…,β_t for which
  
      u_t+1(Z_1,…,Z_t,E)=∑_s=1^t α_s Z_s+β_1 U_1+∑_s=2^t
      β_s U_s(Z_1,…,Z_s-1,E)
  
  with probability 1 over (U_1,E,Z_1,…,Z_t).
  
  
  
  We clarify that Theorem <ref> below establishes the existence of the
  limit _t^∞ provided that condition (e) holds for the
  functions u_2,…,u_t, and this limit _t^∞ then defines
  condition (e) for the next function u_t+1. This condition (e) is a
  non-degeneracy assumption that holds if each function u_t+1 has a
  non-linear dependence on the preceding iterate z_t.
  
  
  Under Assumption <ref>, for each fixed t ≥ 1,
  almost surely as n →∞: _t →_t^∞ for 
  a deterministic non-singular matrix _t^∞, and
  
      (_̆1,…,_̆t+1,_1,…,_t,) 
      (U_1,…,U_t+1,Z_1,…,Z_t,E)
  
  where (Z_1,…,Z_t) ∼(0,_t^∞),
  this vector (Z_1,…,Z_t) is independent of (U_1,E), and
  U_s=u_s(Z_1,…,Z_s-1,E) for each s=2,…,t+1.
  
  
  The proof of this result is provided in Appendix
  <ref>. The limit _t^∞ is
  given by replacing ⟨_̆s_̆s'⟩,
  ⟨∂_s'_̆s ⟩, and κ_k in the
  definitions (<ref>) and (<ref>) with [U_sU_s'],
  [∂_s' u_s(Z_1,…,Z_s-1,E)], and the free cumulants
  κ_k^∞ of the limit spectral distribution Λ.
  
  
  
   §.§ Removing the non-degeneracy assumption
  
  
  The following corollary provides a version of Theorem <ref>
  without the non-degeneracy condition of Assumption <ref>(e),
  under the stronger condition that each
  function u_t+1 is continuously-differentiable and Lipschitz.
  Note that the convergence established is only in W_2, rather than
  in W_p for every order p ≥ 1 as in Theorem <ref>.
  
  The proof follows the idea of
  <cit.> by studying a perturbed AMP sequence and then taking
  the limit of this perturbation to 0. We provide this proof in
  Appendix <ref>.
  
  
  Suppose Assumption <ref>(a–c) holds, lim sup_n →∞_∞<∞, each function
  u_t+1:^t+k→ is continuously-differentiable, and
  
      |u_t+1(z,e)-u_t+1(z',e)| ≤ Cz-z'
  
  for a constant C>0 and all z,z' ∈^t and e ∈^k.
  Then for each fixed t ≥ 1, almost surely as n →∞:
  _t →_t^∞ for a deterministic (possibly singular)
  matrix _t^∞, and
  
      (_̆1,…,_̆t+1,_1,…,_t,) 
      (U_1,…,U_t+1,Z_1,…,Z_t,E)
  
  where (U_1,…,U_t+1,Z_1,…,Z_t,E) is as defined in
  Theorem <ref>.
  
  
  
  
  § AMP ALGORITHM FOR RECTANGULAR MATRICES
  
  
  In this section, we describe the form of the general AMP algorithm for
  a rectangular matrix
  
      =Ø^⊤∈^m × n,     =()
  
  and state a formal theorem for its state evolution. We denote
  
      =(λ_1,…,λ_min(m,n)) ∈^min(m,n)
  
  as the diagonal entries of , which are the singular values of .
  
  We consider an initialization _̆1 ∈^m, and two matrices of side
  information
  
      ∈^m × k     and     ∈^n ×ℓ
  
  for fixed dimensions k,ℓ≥ 0, all independent of .
  (We may take k,ℓ=0 if there is no
  such side information.) Starting from this initialization, the AMP algorithm
  takes the form
  
      _t   =^⊤_̆t-b_t1_̌1-b_t2_̌2-…-b_t,t-1_̌t-1
      _̌t   =v_t(_1,…,_t,)
      _t   =_̌t-a_t1_̆1-a_t2_̆2-…-a_tt_̆t
      _̆t+1   =u_t+1(_1,…,_t,)
  
  for functions v_t:^t+ℓ→ and u_t+1:^t+k→.
  In the first iteration t=1, (<ref>) is simply _1=^⊤_̆1.
  The debiasing coefficients a_t1,…,a_tt and
  b_t1,…,b_t,t-1 are defined to ensure that
  
      (_1,…,_t) (0,_t^∞)      and      (_1,…,_t) (0,_t^∞)
  
  as m,n →∞. We describe these debiasing coefficients and
  state evolution in the next section, in terms of the rectangular free cumulants
  of —these were also derived recently in <cit.>.
  
  
  
   §.§ Debiasing coefficients and limit covariance
  
  
  Define the t × t matrices
  
      _t=[  ⟨_̆1^2 ⟩ ⟨_̆1_̆2 ⟩         ⋯ ⟨_̆1_̆t ⟩; ⟨_̆2_̆1 ⟩  ⟨_̆2^2 ⟩         ⋯ ⟨_̆2_̆t
      ⟩;         ⋮         ⋮         ⋱         ⋮; ⟨_̆t_̆1 ⟩ ⟨_̆t_̆2 ⟩         ⋯  ⟨_̆t^2 ⟩ ],     _t=[           0           0           ⋯           0           0;  ⟨∂_1 _̆2 ⟩           0           ⋯           0           0;  ⟨∂_1 _̆3 ⟩  ⟨∂_2 _̆3 ⟩           ⋯           0           0;           ⋮           ⋮           ⋱           ⋮           ⋮;  ⟨∂_1 _̆t ⟩  ⟨∂_2 _̆t ⟩           ⋯ ⟨∂_t-1_̆t ⟩           0 ],
      
      _t=[  ⟨_̌1^2 ⟩ ⟨_̌1_̌2 ⟩         ⋯ ⟨_̌1_̌t ⟩; ⟨_̌2_̌1 ⟩  ⟨_̌2^2 ⟩         ⋯ ⟨_̌2_̌t
      ⟩;         ⋮         ⋮         ⋱         ⋮; ⟨_̌t_̌1 ⟩ ⟨_̌t_̌2
      ⟩         ⋯  ⟨_̌t^2 ⟩ ],     _t=
      [ ⟨∂_1 _̌1 ⟩          0          ⋯          0; ⟨∂_1 _̌2 ⟩ ⟨∂_2 _̌2 ⟩          ⋯          0;          ⋮          ⋮          ⋱          ⋮; ⟨∂_1 _̌t ⟩ ⟨∂_2 _̌t ⟩          ⋯ ⟨∂_t _̌t ⟩ ].
  
  For each j ≥ 0, define
  
      _t^(j)   =∑_i=0^j (_t_t)^i _t
      (_t^⊤_t^⊤)^j-i
      +∑_i=0^j-1 (_t_t)^i_t _t_t^⊤
      (_t^⊤_t^⊤)^j-1-i,
      _t^(j)   =∑_i=0^j (_t_t)^i _t
      (_t^⊤_t^⊤)^j-i
      +∑_i=0^j-1 (_t_t)^i_t _t_t^⊤
      (_t^⊤_t^⊤)^j-1-i.
  
  The second summations of
  (<ref>) and (<ref>) are not present
  for j=0. So for example,
  
      _t^(0)   =_t
      _t^(1)   =_t_t_t+_t _t _t^⊤+
      _t_t^⊤_t^⊤
      _t^(2)   =_t_t_t_t_t+_t_t_t
      _t _t^⊤+_t_t_t_t^⊤_t^⊤
         +_t_t _t^⊤_t^⊤_t^⊤
      +_t_t^⊤_t^⊤_t^⊤_t^⊤
      _t^(0)   =_t
      _t^(1)   =_t_t_t+_t_t_t^⊤
      +_t_t^⊤_t^⊤
      _t^(2)   =_t_t_t_t_t+_t_t_t_t_t^⊤
      +_t_t_t_t^⊤_t^⊤
         +_t_t_t^⊤_t^⊤_t^⊤
      +_t_t^⊤_t^⊤_t^⊤_t^⊤.
  
  
  Let {κ_2k}_k ≥ 1 be the rectangular free cumulants
  with aspect ratio γ=m/n corresponding to the sequence of even moments
  
      m_2k=1/m∑_i=1^min(m,n)λ_i^2k,
  
  as defined in Section <ref>. Note that we always use the
  normalization 1/m, so these are the moments of  padded by m-n
  additional 0's if m>n.
  
  Define the t × t matrices
  
      _t=(∑_j=0^∞κ_2(j+1)_t(_t_t)^j)^⊤,     _t=(γ∑_j=0^∞κ_2(j+1)_t(_t_t)^j)^⊤,
  
  
      _t=∑_j=0^∞κ_2(j+1)_t^(j),     _t=γ∑_j=0^∞κ_2(j+1)_t^(j).
  
  These are in fact finite series, as it may be verified that
  
      _t(_t_t)^j   =0  for  j ≥ t+1
      _t(_t_t)^j   =0  for  j ≥ t
      _t^(j)   =0  for  j ≥ 2t
      _t^(j)   =0  for  j ≥ 2t-1.
  
  So for example,
  
      _1=κ_2_1^⊤,     _2=κ_2_2^⊤+κ_4(_2_2_2)^⊤,   …
         _1=0,     _2=γκ_2 _2^⊤,
          _3=γκ_2_3^⊤+γκ_4(_3_3_3)^⊤,   …
         _1=κ_2 _1^(0)+κ_4_1^(1),
          _2=κ_2_2^(0)+κ_4_2^(1)+κ_6_2^(2)+κ_8_2^(3),   …
         _1=γκ_2 _1^(0),
          _2=γκ_2_2^(0)+γκ_4_2^(1)+γκ_6_2^(2),   …
  
  
  The matrices _t and _t are upper-triangular, with the forms
  
      _t=[ a_11 a_21    ⋯ a_t1;      a_22    ⋯ a_t2;              ⋱    ⋮;                a_tt ], 
          _t=[       0    b_21    b_31       ⋯    b_t1;               0    b_32       ⋯    b_t2;                       ⋱       ⋱       ⋮;                               0 b_t,t-1;                                       0 ].
  
  The debiasing coefficients a_t1,…,a_tt,b_t1,…,b_t,t-1 in
  (<ref>) and (<ref>) are defined as the last columns of
  _t and _t. Under the conditions to be imposed in Assumption
  <ref>, these matrices all have deterministic t × t limits
  
      (_t^∞,_t^∞,_t^∞,_t^∞,
      _t^∞,_t^∞,_t^∞,_t^∞)
      =lim_m,n →∞ (_t,_t,_t,_t,_t,_t,
      _t,_t).
  
  The matrices _t^∞ and _t^∞ are the covariances
  in the state evolutions for (_1,…,_t) and (_1,…,_t).
  As in the symmetric square setting, the debiasing coefficients in
  (<ref>) and (<ref>) may be replaced by their limits
  a_ts^∞ and b_ts^∞, or by any consistent estimates of these
  limits.
  
  We make the following observations about the above definitions:
  
  
    * The upper-left (t-1) × (t-1) submatrices of
  _t,_t,_t,_t coincide with the matrices
  _t-1,_t-1,_t-1,_t-1.
  
    * For each t ≥ 1, _t,_t,_t,_t depend respectively
  only on the rectangular free cumulants of  up to the orders
  κ_2t,κ_2t-2,κ_4t,κ_4t-2.
  
    * The matrices _t,_t depend on
  _̆1,…,_̆t,_̌1,…,_̌t and their derivatives.
  The matrices _t,_t depend on
  _̆1,…,_̆t,_̌1,…,_̌t-1 and their derivatives, but they
  do not depend on _̌t or its derivatives. (Thus the debiasing coefficients
  and state evolution for _t in (<ref>) are well-defined before
  defining _̌t in (<ref>).)
  
  The first two statements are analogous to our observations in the
  symmetric square setting. The third statement holds from the definitions of
  _t and _t in (<ref>–<ref>),
  because the last column of
  _t is 0, so _t_t does not depend on the last row of
  _t, and _t_t_t^⊤ does not depend on the last row or
  column of _t.
  
  
  In the Gaussian setting where  has i.i.d. (0,1/n) entries, the
  limit spectral distribution of ^⊤ is the Marcenko-Pastur law, with
  limiting rectangular free cumulants
  
      κ_2^∞=1,     κ_2j^∞=0    for all 
      j ≥ 2.
  
  This yields simply
  
      _t^∞=(_t^∞)^⊤,     _t^∞=γ
      (_t^∞)^⊤,     _t^∞=_t^∞,
          _t=γ_t^∞.
  
  If we further specialize to an algorithm where v_t depends only on z_t and
  u_t+1 depends only on y_t, then ⟨∂_s _̆t ⟩=0 for
  all s ≠ t-1 and ⟨∂_s _t ⟩=0 for all s ≠ t.
  This yields the Gaussian AMP algorithm
  
      _t   =^⊤_̆t-γ⟨∂_t-1_̆t ⟩_̌t-1
      _̌t   =v_t(_t,)
      _t   =_̌t-⟨∂_t _̌t ⟩_̆t
      _̆t+1   =u_t+1(_t,)
  
  as studied in <cit.>. Furthermore,
  the state evolution is such that _t has the
  empirical limit (0,ω_tt^∞) where
  ω_tt^∞=lim_m,n →∞γ·⟨_̆t^2 ⟩,
  and _t has the empirical limit (0,σ_tt^∞) where
  σ_tt^∞=lim_m,n →∞⟨_̌t^2
  ⟩.
  
  Note that outside of this Gaussian setting, in general we do not have
  the identities _t=_t and _t=γ_t even when
   is normalized such that κ_2=1.
  
  
  
  
   §.§ Main result
  
  
  We impose the following assumptions on the model
  (<ref>–<ref>) and the AMP
  iterates (<ref>–<ref>). Again, we do not
  require here v_t(·) and u_t+1(·) to be Lipschitz.
  
  
  
  
  
    * m,n →∞ such that m/n=γ∈ (0,∞) is a fixed
  constant.
  
    * Ø∈^m × m and ∈^n × n are independent
  random and Haar-uniform orthogonal matrices.
  
    * ∈^min(m,n) is independent of Ø, and satisfies
  Λ almost surely as m,n →∞, for a random
  variable Λ having finite moments of all orders.
  
    * _̆1 ∈^m, ∈^m × k, and ∈^n ×ℓ are independent of Ø,
  and satisfy (_̆1,)  (U_1,E) and F almost surely
  as m,n →∞, where
  (U_1,E) ≡ (U_1,E_1,…,E_k) and F ≡ (F_1,…,F_ℓ)
  are random vectors having finite moments of all orders.
  
    * Each function v_t:^t+ℓ→ and
  u_t+1:^t+k→ satisfies (<ref>) for some C>0 and p
  ≥ 1. Writing their arguments
  as (z,f) and (y,e) where z,y ∈^t, f ∈^ℓ, and e ∈^k, v_t is weakly differentiable in z and continuous in f, and
  u_t+1 is weakly differentiable in y and continuous in e. For each
  s=1,…,t, ∂_s v_t and ∂_s u_t+1 also
  satisfy (<ref>) for some C>0 and p ≥ 1, where
  ∂_s v_t(z,f) is continuous at Lebesgue-a.e. z
  ∈^t for every f ∈^ℓ, and
  ∂_s u_t+1(y,e) is continuous at Lebesgue-a.e. y ∈^t for
  every e ∈^k.
  
    * [Λ]>0 and [U_1^2]>0. Letting (Z_1,…,Z_t) ∼(0,_t^∞) be independent of F, there do not exist constants
  α_1,…,α_t,β_1,…,β_t-1 for which
  
      v_t(Z_1,…,Z_t,F)=∑_s=1^t α_s Z_s+∑_s=1^t-1β_s v_s(Z_1,…,Z_s,F)
  
  with probability 1 over (F,Z_1,…,Z_t). Letting
  (Y_1,…,Y_t) ∼(0,_t^∞) be independent of (U_1,E),
  there do not exist constants
  α_1,…,α_t,β_1,…,β_t for which
  
      u_t+1(Y_1,…,Y_t,E)=∑_s=1^t α_s Y_s+β_1 U_1
      +∑_s=2^t β_s u_s(Y_1,…,Y_s-1,E)
  
  with probability 1 over (U_1,E,Y_1,…,Y_t).
  
  
  
  As in the symmetric square setting, we clarify that Theorem <ref> below
  establishes the existence of _t^∞ when condition (f) holds for
  u_1,…,u_t and v_1,…,v_t-1, and this limit _t^∞
  then defines condition (f) for v_t. Similarly, the theorem establishes
  the existence of _t^∞ when condition (f) holds for
  u_1,…,u_t and v_1,…,v_t, and this limit _t^∞
  then defines the condition for u_t+1. This condition (f) is a non-degeneracy
  assumption that will hold as long as u_t+1(·) and v_t(·) depend
  non-linearly on y_t and z_t, respectively.
  
  
  Under Assumption <ref>, for each fixed t ≥ 1, almost surely as
  n →∞: _t →_t^∞ and
  _t →_t^∞ for some deterministic non-singular
  matrices _t^∞ and _t^∞. Also,
  
      (_̆1,…,_̆t+1,_1,…,_t,)    
      (U_1,…,U_t+1,Y_1,…,Y_t,E)
      
      (_̌1,…,_̌t,_1,…,_t,)    
      (V_1,…,V_t,Z_1,…,Z_t,F)
  
  where (Y_1,…,Y_t) ∼(0,_t^∞) is independent of
  (U_1,E); (Z_1,…,Z_t) ∼(0,_t^∞) is independent of F;
  U_s=u_s(Z_1,…,Z_s-1,E) for each s=2,…,t+1;
  and V_s=v_s(Z_1,…,Z_s,F) for each s=1,…,t.
  
  
  The limits _t^∞ and _t^∞ are
  given by replacing ⟨_̆s_̆s'⟩, ⟨_̌s_̌s'⟩,
  ⟨∂_s'_̆s ⟩, ⟨∂_s'_̌s ⟩,
  and κ_2k
  in the definitions (<ref>–<ref>) and
  (<ref>) with
  [U_sU_s'], [V_sV_s'],
  [∂_s' u_s(Y_1,…,Y_s-1,E)],
  [∂_s' v_s(Z_1,…,Z_s,F)], and κ_2k^∞.
  
  The proof of this result is provided in Appendix
  <ref>.
  As in Corollary <ref>, we may remove the non-degeneracy
  condition in Assumption <ref>(f) if v_t and u_t+1 are
  continuously-differentiable and Lipschitz.
  This is stated in the following corollary. The proof
  follows the same argument as that of Corollary <ref>, and we omit
  this for brevity.
  
  
  Suppose Assumption <ref>(a–d) holds, lim sup_n →∞_∞<∞, each function v_t:^t+ℓ→ and
  u_t+1:^t+k→ is continuously-differentiable, and
  
      |v_t(z,f)-v_t(z',f)| ≤ Cz-z',     
      |u_t+1(y,e)-u_t+1(y',e)| ≤ Cy-y'
  
  for a constant C>0 and all z,z',y,y' ∈^t, e ∈^k, and f ∈^ℓ. Then for each fixed t ≥ 1, almost surely as n →∞:
  _t →_t^∞ and _t →_t^∞ for some
  deterministic (possibly singular) matrices _t^∞ and
  _t^∞, and
  
      (_̆1,…,_̆t+1,_1,…,_t,)    
      (U_1,…,U_t+1,Y_1,…,Y_t,E)
      
      (_̌1,…,_̌t,_1,…,_t,)    
      (V_1,…,V_t,Z_1,…,Z_t,F)
  
  where these limits are as defined in Theorem <ref>.
  
  
  
  
  § PROOF IDEAS
  
  
  We describe here the main ideas of the proofs. In the
  setting of a symmetric square matrix ∈^n × n, the basic
  strategy is to write =Ø^⊤Ø, and to express the AMP iterations
  (<ref>–<ref>) in an expanded form as
  
      _̊t   =Ø_̆t
      _t   =Ø^⊤_̊t
      _t   =_t-b_t1_̆1-…-b_tt_̆t
      _̆t+1   =u_t+1(_1,…,_t).
  
  All analyses are performed conditional on _̆1 and , so that the
  only randomness is in the Haar-orthogonal matrix Ø. We apply Bolthausen's
  conditioning technique <cit.>, analyzing sequentially
  each iterate _̊1,_1,_1,_̆2,_̊2,… conditional on all preceding
  iterates. This requires understanding the law of Ø conditional on events of
  the form
  
      Ø=,
  
  which was shown in <cit.> to be 
  
      Ø|_Ø=L=(^⊤)^-1^⊤+_^⊥Ø̃_^⊥^⊤.
  
  Here, _^⊥ and _^⊥ are matrices with
  orthonormal columns spanning the orthogonal complements of
  the column spans of  and , and Ø̃ is an independent
  Haar-orthogonal matrix. Applying
  (<ref>) to the appearances of Ø in
  (<ref>–<ref>), we will exhibit decompositions
  
      _̊t=_̊∥+_̊⊥,     _t=_∥+_⊥.
  
  The vectors _̊⊥ and _⊥ arise from the second term of
  (<ref>) and have empirical distributions that are
  approximately Gaussian conditional on the preceding iterates. The vectors
  _̊∥ and _∥ arise from the first term of
  (<ref>), are deterministic conditional on the preceding
  iterates, and represent biases respectively in the directions of
  (_̊1,…,_̊t-1,_̊t-1,…,_̊t-1) and
  (_̆1,…,_̆t,_1,…,_t-1). The Onsager correction
  by b_t1_̆1+…+b_tt_̆t in (<ref>) is defined to
  exactly cancel the component of this bias _∥ in (_̆1,…,_̆t), so that (_1,…,_t) has
  an approximate joint Gaussian law. When the spectrum of  converges to
  Wigner's semicircle law,
  the forms of _̊∥ and _∥ and variances of
  _̊⊥ and _⊥ are more straightforward to track across iterations,
  and this produces a slightly different proof of the AMP analyses in
  <cit.>.
  
  When the spectrum of  does not converge to the
  semicircle law, two difficulties arise in carrying out this conditional
  analysis. First, the forms of _̊∥,_̊⊥,_∥,_⊥
  in iteration T will depend on
  
      n^-1_̆s^⊤^k_̆t ≡ n^-1_̊s^⊤^k_̊t
           for  k=1,2  and  s,t ≤ T.
  
  These values will in turn depend on
  
      n^-1_̆s^⊤^k _̆t ≡ n^-1_̊s^⊤^k_̊t
           for  k=1,…,4  and  s,t ≤ T-1,
  
  which will in turn depend on
  
      n^-1_̆s^⊤^k _̆t ≡ n^-1_̊s^⊤^k_̊t
           for  k=1,…,6  and  s,t ≤ T-2,
  
  and so forth. The final dependence is
  on n^-1_̆1^⊤^k _̆1 for k=1,…,2T, whose large-n
  limits are given by
  the first 2T moments of the limit spectral distribution of , because
  the initialization _̆1 is independent of  which is rotationally invariant
  in law. The free cumulants of  that appear in the final forms
  of the Onsager correction and state evolution emerge by
  tracking these dependences. To provide an inductive argument that can describe
  these dependences for arbitrary iterations, our proof establishes a
  precise form of
  
      lim_n →∞ n^-1_̆s^⊤^k _̆t
  
  for every fixed moment k ≥ 0 and all fixed iterates s,t ≥ 1.
  These forms depend on combinatorial coefficients that we call “partial
  moment coefficients”, defined by summing over certain subsets of the
  non-crossing partition lattice, and which interpolate between the moments and
  free cumulants of the spectral distribution of .
  We define these coefficients in Appendix <ref>.
  
  A second technical difficulty which arises is that for the resulting
  conditioning events Ø=, the form of the matrix ^⊤ in
  (<ref>) becomes complicated, depending on series of
  matrices with these partial moment coefficients, and (^⊤)^-1 does not admit a tractable description. Instead, we handle
  matrix-vector products (^⊤)^-1$̌ arising in the computation by
  “guessing” the formfor this product, and then verifying that(^⊤)=$̌. This type of verification is contained in Lemma <ref>,
  and relies on combinatorial identities for these partial moment coefficients.
  
  The proof ideas in the rectangular setting are similar: We write =Ø^⊤ and express (<ref>–<ref>) in an
  expanded form analogous to (<ref>–<ref>) above.
  A key component of the proof is then to identify the large-(m,n) limits of the
  four quantities
  
      m^-1_̆s^⊤ (^⊤)^k_̆t,
         m^-1_̌s^⊤^⊤(^⊤)^k_̆t,
  
  
      n^-1_̆s^⊤(^⊤)^k_̌t,
         n^-1_̌s^⊤ (^⊤)^k_̌t
  
  for all fixed moments k ≥ 0 and iterates s,t ≥ 1. These will depend
  on certain partial moment coefficients that interpolate between the moments and
  rectangular free cumulants of the limit singular value distribution
  of , and which are defined by summing over subsets of the lattice of
  non-crossing partitions of sets with even cardinality. These coefficients are
  defined in Appendix <ref>, and the corresponding
  identities involving (^⊤)^-1 are contained in Lemma
  <ref>.
  
  For the analyses of the Bayes-AMP algorithms for PCA in Section <ref>,
  part (a) of Theorems <ref> and <ref> are straightforward
  consequences of the results for the general AMP algorithms. Part (b) of these
  theorems require an analysis of the state evolutions for the single-iterate
  posterior mean denoisers, which we carry out in Appendix
  <ref>. This analysis applies a contractive
  mapping argument to show that for sufficiently large signal strengths,
  the matrices _t, _t, _t, and _t all converge
  as t →∞ in a space of “infinite matrices” equipped with a
  weighted ℓ_∞ metric.
  
  
  
  
  
  § PROOF FOR SYMMETRIC SQUARE MATRICES
  
  
  In this appendix, we prove Theorem <ref>.
  Recalling =Ø^⊤Ø where =(),
  we may write the iterations (<ref>–<ref>) equivalently as
  
      _̊t   =Ø_̆t
      _t   =Ø^⊤_̊t
      _t   =_t-b_t1_̆1-…-b_tt_̆t
      _̆t+1   =u_t+1(_1,…,_t,)
  
  
  As discussed in Section <ref>, we will wish to
  identify the almost-sure limits
  
      lim_n →∞ n^-1_̊s^⊤^k _̊s'≡lim_n →∞ n^-1_̆s^⊤^k_̆s'
  
  for each fixed pair s,s' ≥ 1 and fixed order k ≥ 0.
  In Section <ref> below, we first
  define certain “partial moment” coefficients c_k,j corresponding to the
  free cumulants {κ_k}_k ≥ 1 of . We then define,
  for each iteration t ≥ 1 and each order k ≥ 0, the t × t matrix
  
      Ł_t^(k)=∑_j=0^∞ c_k,j_t^(j)
  
  where _t^(j) is defined by (<ref>). As in the definitions
  (<ref>), this series is in fact finite because _t^(j)=0
  for j ≥ 2t-1. The limits (<ref>) will be identified as the
  entries of Ł_t^(k,∞)=lim_n →∞Ł_t^(k).
  
  
  
   §.§ Coefficients for “partial moments”
  
  
  Let {m_k}_k ≥ 1 and {κ_k}_k ≥ 1 be the moments and
  free cumulants of , as defined in
  Section <ref>. For notational convenience, we identify
  
      κ_0=1.
  
  We then define a doubly-indexed sequence of coefficients
  (c_k,j)_k,j ≥ 0 by
  
      c_0,0=1,    c_0,j=0  for  j ≥ 1,
         c_k,j=∑_m=0^j+1 c_k-1,m κ_j+1-m for 
      k ≥ 1.
  
  These coefficients admit the following combinatorial interpretation: Let
  
      (k,ℓ)={π∈(k):S ∩{1,…,ℓ}≠ S
       for all  S ∈π}.
  
  This is the subset of non-crossing partitions π∈(k)
  where no set S ∈π is contained in {1,…,ℓ}.
  For ℓ=0, (k,0)=(k) is the set of all non-crossing partitions.
  The following lemma shows that
  c_k,j corresponds to the part of the sum (<ref>)
  that enumerates only over the partitions
  belonging to the subset (k+j,j) of (k+j).
  
  
  For each k ≥ 1,
  
      c_k,j=∑_π∈(k+j,j)∏_S ∈πκ_|S|.
  
  In particular, c_1,j=κ_j+1 for each j ≥ 0, and c_k,0=m_k
  for each k ≥ 1.
  
  
  For k=1, the only non-zero term in the sum (<ref>)
  corresponds to m=0. This gives c_1,j=κ_j+1.
  The only partition of {1,…,j+1} where
  no set belongs to {1,…,j} is the partition consisting of a single
  set with all j+1 elements. Thus (j+1,j) consists of this single
  partition, so the right side of (<ref>) is simply κ_j+1.
  This verifies (<ref>) for k=1.
  
  Suppose inductively that (<ref>) holds for k-1 (and all j).
  Consider
  
      c_k,j=c_k-1,j+1+∑_m=0^j c_k-1,mκ_j+1-m.
  
  By this induction hypothesis, the first term is
  
      c_k-1,j+1=∑_π∈(k+j,j+1)∏_S ∈πκ_|S|.
  
  To analyze the second term of (<ref>), note that
  if π∈(k+j,j) but π∉(k+j,j+1), then there is some set
  S ∈π containing j+1 and also belonging to {1,…,j+1}.
  This set S ∈π must consist of consecutive elements of
  {1,…,j+1}, because if there is a gap in the elements of S, then the
  elements in this gap must form their own sets of π as
  π is non-crossing, and this contradicts π∈(k+j,j). Thus
  S={m+1,…,j+1} for some m ∈{0,…,j}.
  Removing S from π establishes a
  bijection between such partitions π and the
  non-crossing partitions π' ∈(k-1+m,m) of the k-1+m remaining
  elements, such that
  no set of π' is contained in {1,…,m}. Summing over such partitions
  π' and applying the induction hypothesis, we have
  
      c_k-1,m=∑_π' ∈(k-1+m,m)∏_S' ∈π'κ_|S'|.
  
  Then applying this bijection and including back {m+1,…,j+1}
  (of size j+1-m) into π,
  
      c_k-1,mκ_j+1-m=∑_π∈(k+j,j) ∖(k+j,j+1)_{m+1,…,j+1}∈π  ∏_S ∈πκ_|S|.
  
  Summing this over all possible values m ∈{0,…,j} gives
  
      ∑_m=0^j c_k-1,mκ_j+1-m=∑_π∈(k+j,j) ∖(k+j,j+1)  ∏_S ∈πκ_|S|,
  
  and combining with (<ref>) and
  (<ref>) yields (<ref>). This completes
  the induction, establishing (<ref>) for all k.
  
  Finally, the statement c_k,0=m_k follows from specializing
  (<ref>) to j=0, and applying
  (k,0)=(k) and the moment-cumulant
  relations (<ref>).
  
  
  
  
   §.§ Partial moment identities
  
  
  Recalling the definition of Ł_t^(k) in (<ref>), 
  we now establish several identities that are derived from the recursion for
  c_k,j in (<ref>).
  
  
  For every t ≥ 1,
  
      Ł_t^(0)   =_t
      Ł_t^(1)   =_t_t+_t_t
         =_t^⊤_t+_t_t^⊤
      Ł_t^(2)   =_t^⊤_t_t
      +_t^⊤_t_t+_t_t^⊤_t+_t
  
  
  
  For k=0, we have c_0,0=1 and c_0,j=0 for all j ≥ 1. We also have
  _t^(0)=_t. Hence (<ref>) follows from (<ref>).
  
  For k=1, we have c_1,j=κ_j+1 by Lemma <ref>. Then
  
      Ł_t^(1)=∑_j=0^∞κ_j+1_t^(j)
      =∑_j=0^∞κ_j+1∑_i=0^j _t^i _t
      (_t^j-i)^⊤.
  
  Note that all series throughout this proof are actually finite, so we may freely
  exchange orders of summation.
  Separating the terms that begin with _t from those that begin with
  _t,
  
      Ł_t^(1)   =∑_j=0^∞κ_j+1_t (_t^j)^⊤
      +∑_j=1^∞κ_j+1∑_i=1^j _t^i _t
      (_t^j-i)^⊤
         =_t_t+_t∑_j=0^∞κ_j+2∑_i=0^j _t^i
      _t(_t^j-i)^⊤=_t_t+_t_t.
  
  Since Ł_t^(1), _t, and _t are symmetric,
  we must also have Ł_t^(1)=_t^⊤_t+_t_t^⊤, and this yields both identities in
  (<ref>).
  
  For k=2, applying c_1,m=κ_m+1 and the recursion (<ref>),
  we have
  
      Ł_t^(2)=∑_j=0^∞ c_2,j_t^(j)
      =∑_j=0^∞(∑_m=0^j+1κ_m+1κ_j+1-m)
      ·(∑_i=0^j _t^i _t(_t^j-i)^⊤).
  
  Collecting terms by powers of _t and _t^⊤,
  
      Ł_t^(2)   =∑_i=0^∞∑_p=0^∞(∑_m=0^i+p+1κ_m+1κ_i+p+1-m)
      _t^i _t (_t^p)^⊤.
  
  Substituting q=i+p-m, we may write
  
      ∑_m=0^i+p+1κ_m+1κ_i+p+1-m
      =κ_i+1κ_p+1+κ_i+p+2κ_0
      +∑_m=0^i-1κ_m+1κ_i+p+1-m
      +∑_q=0^p-1κ_i+p+1-qκ_q+1
  
  where the last two sums may be empty if i=0 or p=0. Recalling the notation
  κ_0=1 from (<ref>), and identifying
  
      _t^⊤_t_t   =(∑_i=0^∞κ_i+1_t^i)
      _t(∑_p=0^∞κ_p+1 (_t^p)^⊤)
      =∑_i=0^∞∑_p=0^∞κ_i+1κ_p+1_t^i _t (_t^p)^⊤
      _t   =∑_j=0^∞κ_j+2∑_i=0^j _t^i _t
      (_t^j-i)^⊤=∑_i=0^∞∑_p=0^∞κ_i+p+2κ_0 _t^i _t (_t^p)^⊤
      _t^⊤_t_t   =
      (∑_m=0^∞κ_m+1_t^m)_t
      (∑_j=0^∞κ_j+2∑_p=0^j
      _t^j-p_t(_t^p)^⊤)
         =∑_i=1^∞∑_p=0^∞(∑_m=0^i-1κ_m+1κ_i+p+1-m)
      _t^i _t (_t^p)^⊤
      _t_t^⊤_t   =(∑_j=0^∞κ_j+2∑_i=0^j _t^i _t (_t^j-i)^⊤)
      _t^⊤(∑_q=0^∞κ_q+1 (_t^q)^⊤)
         =∑_i=0^∞∑_p=1^∞(∑_q=0^p-1κ_i+p+1-qκ_q+1)
      _t^i _t (_t^p)^⊤,
  
  this yields (<ref>).
  
  
  
  Define
  
      _t=[        _t _t_t+_t_t;      _t^⊤   _t^⊤_t+ ].
  
  For every t ≥ 1 and k ≥ 0,
  
      [   Ł_t^(k) Ł_t^(k+1) ]   =[     ∑_j=0^∞ c_k,j_t^j ∑_j=0^∞ c_k,j+1_t^(j) ]_t
      [   Ł_t^(k) Ł_t^(k+1); Ł_t^(k+1) Ł_t^(k+2) ]   =c_k,0[ Ł_t^(0) Ł_t^(1); Ł_t^(1) Ł_t^(2) ]
      +_t^⊤[                       0 ∑_j=0^∞
      c_k,j+1(_t^j)^⊤;     ∑_j=0^∞ c_k,j+1_t^j   ∑_j=0^∞
      c_k,j+2_t^(j) ]_t
  
  
  
  
  Applying (<ref>) and the definitions of Ł_t^(1) and _t,
  and recalling the notation κ_0=1
  from (<ref>) and c_1,j=κ_j+1 from Lemma
  <ref>,
  
      _t^⊤=[      _t      _t; Ł_t^(1) _t^⊤_t+ ]
      =[                 _t                 _t; ∑_j=0^∞κ_j+1_t^(j)    ∑_j=0^∞κ_j _t^j ].
  
  
  For (<ref>), applying the definition of _t^(j),
  we compute
  
      [ _t _t ][ ∑_j=0^∞ c_k,j(_t^j)^⊤; ∑_j=0^∞ c_k,j+1_t^(j) ]   =_t ·∑_j=0^∞ c_k,j (_t^j)^⊤
      +_t ·∑_j=0^∞ c_k,j+1∑_i=0^j _t^i _t
      (_t^j-i)^⊤
         =∑_j=0^∞ c_k,j∑_i=0^j _t^i
      _t(_t^j-i)^⊤
         =∑_j=0^∞ c_k,j_t^(j)=Ł_t^(k).
  
  We also compute
  
      [ ∑_j=0^∞κ_j+1_t^(j)    ∑_j=0^∞κ_j _t^j ][ ∑_j=0^∞ c_k,j(_t^j)^⊤; ∑_j=0^∞ c_k,j+1_t^(j) ]
         =(∑_j=0^∞κ_j+1∑_i=0^j _t^i _t
      (_t^j-i)^⊤)
      ·(∑_p=0^∞ c_k,p(_t^p)^⊤)
         +(∑_j=0^∞κ_j _t^j)
      ·(∑_p=0^∞ c_k,p+1∑_q=0^p _t^p-q_t
      (_t^q)^⊤)
         =∑_i=0^∞∑_j=i^∞∑_p=0^∞κ_j+1c_k,p_t^i _t(_t^j-i+p)^⊤
      +∑_q=0^∞∑_p=q^∞∑_j=0^∞κ_j c_k,p+1_t^j+p-q_t(_t^q)^⊤
         =∑_i=0^∞∑_r=0^∞(∑_j=i^r+iκ_j+1c_k,r-j+i)
      _t^i _t(_t^r)^⊤
      +∑_q=0^∞∑_ℓ=0^∞(∑_p=q^ℓ+qκ_ℓ-p+qc_k,p+1) _t^ℓ_t(_t^q)^⊤
         =∑_i=0^∞∑_r=0^∞((κ_i+1c_k,r+κ_i+2c_k,r-1+
      +…+κ_i+r+1c_k,0)
         +(κ_ic_k,r+1+κ_i-1c_k,r+2
      +…+κ_0c_k,i+r+1))_t^i_t(_t^r)^⊤
         =∑_i=0^∞∑_r=0^∞(∑_j=0^i+r+1κ_j c_k,i+r+1-j)
      _t^i_t(_t^r)^⊤.
  
  From the recursion for c_k,j in (<ref>), this is equal to
  
      ∑_i=0^∞∑_r=0^∞
      c_k+1,i+r_t^i_t(_t^r)^⊤=Ł_t^(k+1).
  
  Combining this with (<ref>) and (<ref>) and taking the
  transpose yields (<ref>).
  
  For (<ref>), applying (<ref>),
  first observe that
  
      _t^⊤[                   0; ∑_j=0^∞ c_k,j+1_t^j ]   =[                ∑_j=0^∞ c_k,j+1_t^j+1; ∑_j=0^∞κ_j _t^j ·∑_p=0^∞
      c_k,p+1_t^p ]
         =[                  ∑_ℓ=1^∞ c_k,ℓ_t^ℓ; ∑_ℓ=0^∞(∑_j=0^ℓκ_j c_k,ℓ-j+1)
      _t^ℓ ]
      =[   ∑_ℓ=0^∞ (c_k,ℓ-c_k,0c_0,ℓ)_t^ℓ; ∑_ℓ=0^∞ (c_k+1,ℓ-c_k,0c_1,ℓ)_t^ℓ ]
  
  where the last equality applies c_0,0=1, c_0,ℓ=0 for ℓ≥ 1,
  c_1,ℓ=κ_ℓ+1, and the recursion (<ref>).
  Next, applying the same computations as leading to
  (<ref>) and (<ref>), we obtain
  
      _t^⊤[ ∑_j=0^∞ c_k,j+1(_t^j)^⊤;   ∑_j=0^∞ c_k,j+2_t^(j) ]   =[                                  ∑_j=0^∞ c_k,j+1_t^(j); ∑_i=0^∞∑_r=0^∞∑_j=0^i+r+1κ_j c_k,i+r+2-j_t^i_t(_t^r)^⊤ ]
         =[   ∑_ℓ=0^∞ (c_k,ℓ+1-c_k,0c_0,ℓ+1)_t^(ℓ); ∑_ℓ=0^∞ (c_k+1,ℓ+1-c_k,0c_1,ℓ+1)_t^(ℓ) ]
  
  where the second equality again applies c_0,ℓ+1=0 for ℓ≥ 0,
  c_1,ℓ+1=κ_ℓ+2, and the recursion (<ref>).
  Combining these two identities, we get
  
      _t^⊤[                       0 ∑_j=0^∞ c_k,j+1(_t^j)^⊤;     ∑_j=0^∞ c_k,j+1_t^j   ∑_j=0^∞ c_k,j+2_t^(j) ]
         =[       ∑_ℓ=0^∞ c_k,ℓ_t^ℓ   ∑_ℓ=0^∞ c_k,ℓ+1_t^(ℓ);     ∑_ℓ=0^∞ c_k+1,ℓ_t^ℓ ∑_ℓ=0^∞ c_k+1,ℓ+1_t^(ℓ) ]
      -c_k,0[     ∑_ℓ=0^∞ c_0,ℓ_t^ℓ ∑_ℓ=0^∞ c_0,ℓ+1_t^(ℓ);     ∑_ℓ=0^∞ c_1,ℓ_t^ℓ ∑_ℓ=0^∞ c_1,ℓ+1_t^(ℓ) ]
  
  Then (<ref>) follows from multiplying on the right
  by _t, and applying (<ref>) to the right side
  with k and also with 0,1,k+1 in place of k.
  
  
  
  
   §.§ Conditioning argument
  
  
  We now prove Theorem <ref>, applying the conditioning argument
  described in Section <ref>. Theorem <ref> follows directly
  from the following extended lemma, where part (b) identifies the
  limits (<ref>) with the limit of Ł_t^(k).
  
  
  Suppose Assumption <ref> holds. Almost surely
  for each t=1,2,3,…:
  
  
    * There exist deterministic matrices
  (_t^∞,_t^∞,_t^(j,∞),_t^∞,_t^∞,Ł_t^(k,∞))
  for all fixed j,k ≥ 0 such that
  
      (_t^∞,_t^∞,_t^(j,∞),_t^∞,_t^∞,Ł_t^(k,∞))=lim_n
      →∞ (_t,_t,_t^(j),_t,_t,Ł_t^(k)).
  
  
    * For some random variables R_1,…,R_t having finite moments of all
  orders,
  
      (_̊1,…,_̊t,)  (R_1,…,R_t,Λ).
  
  Furthermore, for each k ≥ 0,
  
      [(R_1,…,R_t)^⊤Λ^k(R_1,…,R_t)]
      ≡lim_n →∞ n^-1
      (_̊1,…,_̊t)^⊤^k(_̊1,…,_̊t)=Ł_t^(k,∞).
  
  
    * We have
  
      (_̆1,…,_̆t+1,_1,…,_t,) 
      (U_1,…,U_t+1,Z_1,…,Z_t,E)
  
  as described in Theorem <ref>.
  
    * The matrix
  
      [         _t^∞     _t^∞_t^∞; _t^∞(_t^∞)^⊤         _t^∞ ]
  
  is non-singular.
  
  
  
  Denote by t^(a),t^(b),t^(c),t^(d) the claims of parts (a–d) up to and
  including iteration t. We induct on t.
  Note that since Λ by Assumption
  <ref>(b), the empirical
  moments m_k of  satisfy m_k → m_k^∞≡[Λ^k] for each k ≥ 0. Then also
  
      κ_k →κ_k^∞,      c_k,j→ c_k,j^∞
  
  for all j,k ≥ 0, where κ_k^∞ and c_k,j^∞ are the free
  cumulants and partial moment coefficients of Λ.
  
  
  Step 1: t=1. We have
  _1=⟨_̆1^2 ⟩→_1^∞≡[U_1^2] by Assumption <ref>(c),
  κ_k →κ_k^∞ and c_k,j→
  c_k,j^∞ by the above, and _1=0. Then 1^(a) follows from the
  definitions.
  
  Noting that _̊1=Ø_̆1 and
  applying Proposition <ref> with =, we have
  
      (,_̊1)  (Λ,R_1)
  
  where R_1 ∼(0,[U_1^2]) is independent of Λ. Then
  for any k ≥ 0,
  
      n^-1_̊1^⊤^k _̊1
      =n^-1∑_i=1^n λ_i^k r_i1^2
      →[Λ^kR_1^2]=m_k^∞[U_1^2].
  
  Note that m_k^∞=c_k,0^∞ by Lemma
  <ref>. Furthermore, _1=0 so that
  _1^(0)=_1=⟨_̆1^2 ⟩ and
  _1^(j)=0 for all j ≥ 1. Hence
  Ł_1^(k,∞)=m_k^∞[U_1^2] for each k ≥ 0.
  This shows 1^(b).
  
  For 1^(c), conditioning on _̆1,_̊1,,, 
  the conditional law of Ø is that of Ø conditioned on the event
  
      _̊1=Ø_̆1.
  
  Since n^-1_̊1^2 →[R_1^2]=[U_1^2], and this is non-zero by
  Assumption <ref>(e), we must have _̊1 ≠ 0
  for all large n.
  Then by Proposition <ref>, this conditional law of
  Ø is equal to
  
      _̊1(_̊1^⊤_̊1)^-1_̆1^⊤+__̊1^⊥Ø̃__̆1^⊥^⊤
  
  where Ø̃∈^(n-1) × (n-1) is Haar-uniform and independent of
  (_̆1,_̊1,,), and Π__̊1^⊥,Π__̆1^⊥∈^n × (n-1) have orthonormal columns spanning the orthogonal
  complements of _̊1,_̆1. Thus, to analyze the joint behavior of
  (_̆1,_̆2,_1,), we may replace the update _1=Ø^⊤_̊1
  in this first iteration t=1 by the update
  
      _1   =_∥+_⊥
      _∥   =_̆1(_̊1^⊤_̊1)^-1_̊1^⊤_̊1
      _⊥   =__̆1^⊥Ø̃^⊤__̊1^⊥^⊤_̊1
  
  as this will not change the joint law of (_̆1,_̆2,_1,).
  
  For _∥, applying 1^(b),
  we have n^-1_̊1^⊤_̊1/(n^-1_̊1^⊤_̊1)
  → m_1^∞[R_1^2]/[R_1^2]=κ_1^∞. Then applying
  (_̆1,)  (U_1,E) and Proposition <ref>,
  
      (_̆1,,_∥)  (U_1,E,S_∥),     
      S_∥=κ_1^∞ U_1.
  
  For _⊥, applying 1^(b) again and identifying
  κ_2^∞=m_2^∞-(m_1^∞)^2, observe that
  
      n^-1__̊1^⊥^⊤_̊1^2
      =n^-1_̊1^2-(n^-1_̊1^⊤_̊1)^2/n^-1_̊1^2→ m_2^∞[U_1^2]
      -(m_1^∞[U_1^2])^2/[U_1^2]
      =κ_2^∞[U_1^2].
  
  Then applying Proposition <ref>,
  
      _⊥ S_⊥∼(0,κ_2^∞[U_1^2]),
  
  where this limit S_⊥ is independent of
  (U_1,E). Observe that _1=κ_1, so
  
      _1=_1-κ_1 _̆1=(_∥-κ_1 _̆1)+_⊥.
  
  Applying κ_1 →κ_1^∞, S_∥=κ_1^∞ U_1,
  and Propositions <ref> and <ref>, we obtain
  
      (_1,_̆1,)  (Z_1,U_1,E),      Z_1=S_⊥.
  
  Then also (_1,_̆1,_̆2,)  (Z_1,U_1,U_2,E)
  where U_2=u_2(Z_1,E), by Proposition <ref> and the polynomial
  growth condition for u_2(·) in Assumption <ref>(d).
  Identifying _1^∞=κ_2^∞[U_1^2] as the variance of
  S_⊥, this shows 1^(c).
  
  Finally, we have _1^∞=0, and κ_2^∞=[Λ]>0
  and [U_1^2]>0 by Assumption <ref>(e). This implies 1^(d).
  
  
  Step 2: Analysis of _̊t+1. Suppose that
  t^(a),t^(b),t^(c),t^(d) all hold,
  and consider iteration t+1. Note that t^(c) implies ⟨_̆s_̆s'⟩→[U_sU_s'] for all s,s' ≤ t+1,
  so _t+1→_t+1^∞.
  By Assumption <ref>(d), for all s'<s ≤ t+1, each
  derivative ∂_s' u_s
  satisfies the growth condition (<ref>) and is also continuous on a set
  of probability 1 under (Z_1,…,Z_s-1,E), since _s-1^∞ is
  non-singular by t^(d). Then by t^(c) and Proposition
  <ref>, we also have ⟨∂_s'_̆s ⟩→[∂_s' u_s(Z_1,…,Z_s-1,E)], so _t+1→_t+1^∞. Combining with the
  convergence κ_k →κ_k^∞ and c_k,j→ c_k,j^∞ and
  the definitions, this yields t+1^(a).
  
  Let us now show t+1^(b) by analyzing the iterate _̊t+1.
  We define the n × t matrices
  
      _t=(_̆1,…,_̆t),     _t=(_̊1,…,_̊t),
          _t=(_1,…,_t).
  
  Then the updates (<ref>–<ref>) up to iteration t may be
  written as
  
      _t=Ø_t,     _t=Ø^⊤_t-_t_t,
  
  or equivalently,
  
      _t=Ø_t,     Ø_t=_t-_t_t.
  
  Thus, conditioning on _t,_t,_t,_̆t+1,,,
  the law of Ø is conditioned on the event
  
      [ _t _t ][     -_t;   0     ]
      =Ø[ _t _t ].
  
  
  Let us introduce
  
      _t=n^-1[ _t^⊤_t _t^⊤_t; _t^⊤_t _t^⊤_t ].
  
  By t^(c), we have n^-1_t^⊤_t →_t^∞
  and n^-1_t^⊤_t →_t^∞.
  Applying Proposition <ref> (derived from Stein's lemma) entrywise to
  n^-1_t^⊤_t, and recalling the definition of _t in
  (<ref>), we also have
  n^-1_t^⊤_t →_t^∞_t^∞. So
  
      _t →_t^∞
      =[         _t^∞     _t^∞_t^∞; _t^∞(_t^∞)^⊤         _t^∞ ].
  
  This limit _t^∞ is invertible by t^(d). Then _t
  must have full rank 2t for all large n, so (_t,_t) also has full
  column rank 2t for all large n. Then by Proposition
  <ref>, the above conditional law of Ø is given by
  
      [ _t _t ][     -_t;   0     ]_t^-1· n^-1[ _t^⊤; _t^⊤ ]
      +_(_t,_t)^⊥Ø̃_(_t,_t)^⊥^⊤
  
  where Ø̃∈^(n-2t) × (n-2t) is again an independent Haar-orthogonal matrix.
  To analyze (_̊1,…,_̊t+1,), we may then replace the update
  _̊t+1=Ø_̆t+1 by
  
      _̊t+1   =_̊∥+_̊⊥
      _̊∥   =[ _t _t ][     -_t;   0     ]_t^-1· n^-1[ _t^⊤; _t^⊤ ]_̆t+1
      _̊⊥   =_(_t,_t)^⊥Ø̃_(_t,_t)^⊥^⊤_̆t+1,
  
  as this does not change the joint law of (_̊1,…,_̊t+1,).
  
  To analyze _̊∥, let us define
  
      _t^∞=[ [U_1U_t+1];          ⋮; [U_tU_t+1] ],
          _t^∞=[ [∂_1 u_t+1(Z_1,…,Z_t,E)];                        ⋮; [∂_t u_t+1(Z_1,…,Z_t,E)] ].
  
  These are the last columns of _t+1^∞ and
  (_t+1^∞)^⊤ with their last entries removed.
  Then, applying again t^(c) and Proposition <ref>,
  
      n^-1_t^⊤_̆t+1→_t^∞,      n^-1_t^⊤_̆t+1→_t^∞_t^∞.
  
  Noting that _t^∞ is invertible by t^(d), this yields
  
      [     -_t;   0     ]_t^-1· n^-1[ _t^⊤; _t^⊤ ]_̆t+1
         →([                   0;         0 (_t^∞)^-1 ]_t^∞[      _t^∞;    0      ])^-1[ _t^∞; _t^∞ ]
      =(_t^∞)^-1[ _t^∞; _t^∞ ]
  
  where _t^∞ is the limit of _t
  defined in (<ref>). This shows also that
  _t^∞ is invertible.
  Then applying Proposition <ref> and
  (_̊1,…,_̊t,) → (R_1,…,R_t,Λ)
  by t^(b), we have
  
      _̊∥ R_∥
      =[   R_1     ⋯   R_t Λ R_1     ⋯ Λ R_t ] (_t^∞)^-1[ _t^∞; _t^∞ ].
  
  
  For _̊⊥, observe that
  
      n^-1_(_t,_t)^⊥^⊤_̆t+1^2
         =n^-1_̆t+1^2-n^-1_̆t+1^⊤[ _t _t ]·_t^-1· n^-1[ _t^⊤; _t^⊤ ]_̆t+1
         →[U_t+1^2]
      -([   _t; _t_t ]^⊤[     _t  _t _t; _t_t^⊤     _t ]^-1[   _t; _t_t ])^∞
  
  where we use the condensed notation (⋯)^∞ to indicate that all
  quantities in the parentheses are evaluated at their
  n →∞ limits. Then by Proposition <ref>,
  
      _̊⊥ R_⊥∼(0, [U_t+1^2]
      -([   _t; _t_t ]^⊤[     _t  _t _t; _t_t^⊤     _t ]^-1[   _t; _t_t ])^∞),
  
  where this limit R_⊥ is independent of (R_1,…,R_t,Λ).
  Combining these, we have
  (_̊1,…,_̊t+1,)  (R_1,…,R_t+1,Λ) where
  
      R_t+1=[   R_1     ⋯   R_t Λ R_1     ⋯ Λ R_t ] (_t^∞)^-1[ _t^∞; _t^∞ ]+R_⊥.
  
  
  We will require later in the argument that [R_⊥] given by
  (<ref>) is strictly positive. Let us verify this here:
  Identifying the entries of
  
      _t^∞,     _t^∞_t^∞,      and     _t^∞
  
  as the quantities [U_sU_s'], [Z_sU_s'], and [Z_sZ_s'] for
  indices 1 ≤ s,s' ≤ t+1, observe that this variance of R_⊥ given by
  (<ref>) is the variance of the residual of the
  projection of U_t+1 onto the
  linear span of the random variables (Z_1,…,Z_t,U_1,…,U_t) with
  respect to the L_2-inner-product (X,Y) ↦[XY]. Thus if
  [R_⊥]=0, then there would exist scalar constants
  α_1,…,α_t,β_1,…,β_t such that
  
      U_t+1=α_1Z_1+…+α_tZ_t+β_1U_1+…+β_tU_t
  
  almost surely, but this contradicts Assumption <ref>(e). So
  
      [R_⊥]>0.
  
  
  Let us now introduce a block notation for Ł_t+1^(k,∞) (with blocks
  of sizes t and 1) given by
  
      Ł_t+1^(k,∞)=[       Ł_t^(k,∞)       ł_t^(k,∞);   (ł_t^(k,∞))^⊤ l_t+1,t+1^(k,∞) ].
  
  To conclude the proof of t+1^(b),
  it remains to compute the two quantities
  
      [[ R_1   ⋯ R_t ]^⊤Λ^k
      R_t+1]      and     [Λ^kR_t+1^2]
  
  and show that they are given by ł_t^(k,∞) and
  l_t+1,t+1^(k,∞).
  
  For the first quantity, observe that [R_s Λ^k R_⊥]=0 for all s
  ≤ t, because R_⊥ has mean 0 and is independent of (R_s,Λ).
  Then applying (<ref>) and t^(b),
  
      [[ R_1   ⋯ R_t ]^⊤Λ^k
      R_t+1]
      =([   Ł_t^(k) Ł_t^(k+1) ]_t^-1[ _t; _t ])^∞.
  
  Applying the identity (<ref>), we get
  
      [[ R_1   ⋯ R_t ]^⊤Λ^k
      R_t+1]
         =(∑_j=0^∞ c_k,j_t^j _t
      +∑_j=0^∞ c_k,j+1_t^(j)_t)^∞
         =(∑_j=0^∞ c_k,j_t+1^j_t+1
      +∑_j=0^∞ c_k,j+1_t+1^(j)_t+1^⊤)_1:t, t+1^∞.
  
  Here, we use the notation (·)_1:t, t+1 to indicate the entries of rows
  1 to t of column t+1. This last equality holds by writing
  _t+1^j_t+1 and _t+1^(j)_t+1^⊤ in block
  form, and noting that we have the blocks
  
      _t+1^j=[ _t^j    0;         * ],
          _t+1=[ *  ;   * ],
          _t+1^(j)=[ _t^(j)      *;             * ],     _t+1^⊤
      =[  * _t;     0 ].
  
  Finally, from the definitions
  of _t+1^(j) and Ł_t+1^(k), the above is simply
  
      [[ R_1   ⋯ R_t ]^⊤Λ^k
      R_t+1]=(Ł_t+1^(k,∞))_1:t, t+1=ł_t^(k,∞).
  
  
  For [Λ^k R_t+1^2], we again apply (<ref>) and the
  independence of R_⊥ and (R_1,…,R_t,Λ) to obtain
  similarly
  
      [Λ^k R_t+1^2]
         =([ _t; _t ]^⊤
      (_t^-1)^⊤[   Ł_t^(k) Ł_t^(k+1); Ł_t^(k+1) Ł_t^(k+2) ]_t^-1[ _t; _t ])^∞+[Λ^kR_⊥^2].
  
  Applying independence of Λ and R_⊥ and
  taking the expected square on both sides of (<ref>), we also have
  
      [Λ^k R_⊥^2]
         =m_k^∞[R_⊥^2]
         =m_k^∞([R_t+1^2]-[(
      [   R_1     ⋯   R_t Λ R_1     ⋯ Λ R_t ] (_t^∞)^-1[ _t^∞; _t^∞ ])^2])
         =c_k,0^∞([U_t+1^2]
      -([ _t; _t ]^⊤
      (_t^-1)^⊤[ Ł_t^(0) Ł_t^(1); Ł_t^(1) Ł_t^(2) ]_t^-1[ _t; _t ])^∞),
  
  the last line identifying m_k^∞=c_k,0^∞ by Lemma <ref>
  and using
  
      [R_t+1^2]=lim_n →∞ n^-1_̊t+1^2=
      lim_n →∞ n^-1_̆t+1^2=[U_t+1^2].
  
  Applying this to (<ref>),
  and then applying the identity (<ref>), we get
  
      [Λ^k R_t+1^2]
         =c_k,0^∞[U_t+1^2]
      +([ _t; _t ]^⊤[                        0 ∑_j=0^∞ c_k,j+1 (_t^j)^⊤;      ∑_j=0^∞ c_k,j+1_t^j    ∑_j=0^∞
      c_k,j+2_t^(j) ][ _t; _t ])^∞
         =c_k,0^∞[U_t+1^2]
      +(∑_j=0^∞ c_k,j+1_t^⊤ (_t^j)^⊤_t
      +c_k,j+1_t^⊤_t^j _t+
      c_k,j+2_t^⊤_t^(j)_t)^∞
         =(c_k,0_t+1
      +∑_j=0^∞ c_k,j+1_t+1 (_t+1^j+1)^⊤
      +c_k,j+1_t+1^j+1_t+1+c_k,j+2_t+1_t+1^(j)_t+1^⊤)^∞_t+1,t+1.
  
  Here, we use (·)_t+1,t+1 to denote the lower-right entry, and this last
  equality follows again from writing the matrix products in block form and
  observing that
  
      _t+1=[         *        _t;      _t^⊤ [U_t+1^2] ],
         (_t+1^j+1)^⊤=[          * (_t^j)^⊤_t;                     0 ],
        _t+1^j+1=[        *        *; _t^⊤_t^j        0 ], 
        _t+1^(j)=[ _t^(j)      *;             * ].
  
  Applying the definitions of _t+1^(j) and Ł_t+1^(k),
  this is just
  
      [Λ^k R_t+1^2]=(∑_j=0^∞ c_k,j_t+1^(j))_t+1,t+1^∞=l_t+1,t+1^(k,∞).
  
  This concludes the proof of t+1^(b).
  
  
  Step 3: Analysis of _t+1.
  Assuming t+1^(a),t+1^(b),t^(c),t^(d),
  we now show t+1^(c) and t+1^(d). Define the (t+1) × t matrices
  
      _t=[   _t; _t^⊤ ],
          _t=[ _t;    ].
  
  These are the first t columns of _t+1 and
  _t+1, and we have _t_t=_t+1_t.
  Conditional on _t,_t,_t,_̆t+1,_̊t+1,,, the law of Ø
  is conditioned on the event
  
      [ _t+1   _t ][     -_t;   0     ]
      =Ø[ _t+1   _t ].
  
  Let _t^∞ and _t^∞ be the n →∞
  limits of _t and _t, and let us introduce
  
      _t=n^-1[ _t+1^⊤_t+1   _t+1^⊤_t;   _t^⊤_t+1     _t^⊤_t ].
  
  Then by t^(c) and Proposition <ref>,
  
      _t →_t^∞
      =[        _t+1^∞      _t^∞_t^∞; _t^∞ (_t^∞)^⊤          _t^∞ ].
  
  
  To check that this limit _t^∞
  is invertible, observe that its 2t × 2t
  submatrix removing row and column t+1 is just _t^∞ from
  (<ref>), which is invertible by t^(d). The
  Schur-complement of the (t+1,t+1) entry is exactly (<ref>), which we
  have shown is positive in (<ref>). Thus
  _t^∞ is invertible,
  so (_t+1,_t) has full column rank 2t+1
  for all large n. Then by Proposition
  <ref>, the conditional law of Ø is
  
      [ _t+1   _t ][     -_t;         ]_t^-1· n^-1[ _t+1   _t ]^⊤
      +_(_t+1,_t)^⊥Ø̃_(_t+1,_t)^⊥^⊤
  
  where Ø̃ is an independent Haar-orthogonal matrix. Thus, to analyze
  the joint behavior of (_̆1,…,_̆t+2,_1,…,_t+1,),
  we may replace the update _t+1=Ø^⊤_̊t+1 by
  
      _t+1   =_∥+_⊥
      _∥   =[ _t+1   _t ]_t^-1[           0; -_t^⊤       ]· n^-1[ _t+1^⊤;   _t^⊤ ]_̊t+1
      _⊥   =_(_t+1,_t)^⊥Ø̃^⊤_(_t+1,
      _t)^⊥^⊤_̊t+1.
  
  
  To analyze _∥, recall the notation ł_t^(k,∞)
  from (<ref>) and set
  
      Ł̃_t^(k,∞)=[     Ł_t^(k,∞); (ł_t^(k,∞))^⊤ ],     ł̃_t^(k,∞)=[       ł_t^(k,∞); l_t+1,t+1^(k,∞) ].
  
  Then applying t+1^(b),
  
      n^-1[  _t+1^⊤_̊t+1; _t^⊤^2 _̊t+1 ]→[ ł̃_t^(1,∞);  ł_t^(2,∞) ].
  
  Introducing the matrices
  
      _t=[          _t+1   _t+1_t+_t_t;          _t^⊤ _t^⊤_t+_t × t ],     _t^∞=lim_n →∞_t,
  
  we have
  
      _t^-1[           0; -_t^⊤       ]· n^-1[ _t+1^⊤;   _t^⊤ ]_̊t+1   →([                 0; (_t^∞)^⊤          ]_t)^-1[ ł̃_t^(1,∞);  ł_t^(2,∞) ]
         =[                   0;         0 (_t^∞)^-1 ]
      ((_t^∞)^-1)^⊤[ ł̃_t^(1,∞);  ł_t^(2,∞) ].
  
  This also shows that _t^∞ is invertible.
  
  Let us introduce the block notations
  
      _t+1=[        _t       _̱t;         0 b_t+1,t+1 ],
          _t=[       _̱t; b_t+1,t+1 ],
          _t+1=[        _t        _t;      _t^⊤ σ_t+1,t+1 ]
  
  and denote with ^∞ their n →∞ limits.
  Defining _t+1 by (<ref>) and writing this in block
  form, it may be checked that
  
      _t+1=[ _t  *;  0  * ]
  
  where _t constitutes the first 2t+1 rows and columns.
  Applying the identity (<ref>) with t+1 and k=1 yields
  
      [ Ł_t+1^(1); Ł_t+1^(2) ]
      =_t+1^⊤[ ∑_j=0^∞ c_1,j(_t+1^j)^⊤; ∑_j=0^∞ c_1,j+1_t+1^(j) ]
      =_t+1^⊤[ _t+1; _t+1 ],
  
  the second equality identifying c_1,j=κ_j+1 and applying
  the definitions of _t+1 and _t+1 in (<ref>).
  Then equating the first 2t+1 entries of the last column on both sides, and
  taking the limit n →∞, we get
  
      [ ł̃_t^(1,∞);  ł_t^(2,∞) ]
      =(_t^∞)^⊤[ _t^∞; _t^∞ ].
  
  Inverting (_t^∞)^⊤ and applying this to
  (<ref>),
  
      _∥[   U_1     ⋯ U_t+1 ]_t^∞+[ Z_1   ⋯ Z_t ]
      (_t^∞)^-1_t^∞.
  
  
  For _⊥, note that we have shown _t^∞ in
  (<ref>) is invertible.
  Applying the definition of _t and the
  identity (<ref>), we also have
  
      _t^∞   =lim_n →∞ n^-1[           0; -_t^⊤     0 ][ _t+1^⊤_t+1   _t+1^⊤_t;   _t^⊤_t+1  _t^⊤^2 _t ][     -_t;   0     ]
         =[                   0; -(_t^∞)^⊤         0 ][    Ł_t+1^(0,∞)     Ł̃_t^(1,∞); (Ł̃_t^(1,∞))^⊤      Ł_t^(2,∞) ][       -_t^∞;     0       ].
  
  Thus the matrices
  
      n^-1[ _t+1^⊤_t+1   _t+1^⊤_t;   _t^⊤_t+1  _t^⊤^2 _t ],     [    Ł_t+1^(0,∞)     Ł̃_t^(1,∞); (Ł̃_t^(1,∞))^⊤      Ł_t^(2,∞) ]
  
  are also invertible (the former almost surely
  for all large n). Observe then that
  
      n^-1_(_t+1,_t)^⊥^⊤_̊t+1^2
             =n^-1_̊t+1^2-n^-1[  _t+1^⊤_̊t+1; _t^⊤^2 _̊t+1 ]^⊤[ _t+1^⊤_t+1   _t+1^⊤_t;   _t^⊤_t+1  _t^⊤^2 _t ]^-1[  _t+1^⊤_̊t+1; _t^⊤^2 _̊t+1 ]
             →(l_t+1,t+1^(2)-
      [ ł̃_t^(1);  ł_t^(2) ]^⊤[    Ł_t+1^(0)     Ł̃_t^(1); (Ł̃_t^(1))^⊤      Ł_t^(2) ]^-1[ ł̃_t^(1);  ł_t^(2) ])^∞
  
  Then by Proposition <ref>,
  
      _⊥ S_⊥∼(0, (l_t+1,t+1^(2)-
      [ ł̃_t^(1);  ł_t^(2) ]^⊤[    Ł_t+1^(0)     Ł̃_t^(1); (Ł̃_t^(1))^⊤      Ł_t^(2) ]^-1[ ł̃_t^(1);  ł_t^(2) ])^∞)
  
  where this limit S_⊥ is independent of
  (U_1,…,U_t+1,Z_1,…,Z_t,E).
  Combining the above, we obtain
  
      _t+1 S_t+1=
      [   U_1     ⋯ U_t+1 ]_t^∞+[ Z_1   ⋯ Z_t ]
      (_t^∞)^-1_t^∞+S_⊥.
  
  Then, since
  
      _t+1=_t+1-(_̆1,…,_̆t+1)[ _t ],
  
  applying Propositions <ref> and <ref>,
  this shows
  
      (_̆1,…,_̆t+2,_1,…,_t+1,)
       (U_1,…,U_t+2,Z_1,…,Z_t+1,E)
  
  where U_t+2=u_t+2(Z_1,…,Z_t+1,E) and
  
      Z_t+1=[ Z_1   ⋯ Z_t ]
      (_t^∞)^-1_t^∞+S_⊥.
  
  In particular, (Z_1,…,Z_t+1) has a multivariate normal limit 
  independent of (U_1,E).
  
  To conclude the proof of t+1^(c), it remains to compute
  
      [[ Z_1   ⋯ Z_t ]^⊤ Z_t+1],     [Z_t+1^2]
  
  and show that these are given by _t^∞ and σ_t+1,t+1^∞. Observe
  that [Z_s S_⊥]=0 for all s ≤ t, since S_⊥ has mean 0 and is
  independent of Z_s. Then
  
      [[ Z_1   ⋯ Z_t ]^⊤ Z_t+1]
      =_t^∞ (_t^∞)^-1_t^∞=_t^∞.
  
  
  To compute [Z_t+1^2], note that (<ref>) may be written as
  
      S_t+1=[   U_1     ⋯ U_t+1 ]_t^∞+Z_t+1.
  
  Taking the expected square on both sides,
  
      [S_t+1^2]=[([   U_1     ⋯ U_t+1 ]_t^∞)^2]+2[([   U_1     ⋯ U_t+1 ]_t^∞)Z_t+1]+[Z_t+1^2].
  
  Since _t+1=Ø^⊤_̊t+1, we have
  
      [S_t+1^2]=lim_n →∞
      n^-1_t+1^2=lim_n →∞
      n^-1_̊t+1^⊤^2 _̊t+1=(Ł_t+1^(2,∞))_t+1,t+1.
  
  Identifying _t as the last column of _t+1, we have also
  
      [([   U_1     ⋯ U_t+1 ]_t^∞)^2]
      =(_t^∞)^⊤_t+1^∞_t^∞
      =((_t+1^∞)^⊤_t+1^∞_t+1^∞)_t+1,t+1.
  
  Applying
  
      [(U_1,…,U_t+1)^⊤ Z_t+1]
      =lim_n →∞ n^-1(_̆1,…,_̆t+1)^⊤_t+1
      =_t^∞_t^∞,
  
  we get
  
      [([   U_1     ⋯ U_t+1 ]_t^∞)Z_t+1]
         =(_t^∞)^⊤_t^∞_t^∞
      =((_t+1^∞)^⊤_t+1^∞_t+1^∞)_t+1,t+1
         =(_t^∞)^⊤(_t^∞)^⊤_t^∞
      =(_t+1^∞(_t+1^∞)^⊤_t+1^∞)_t+1,t+1
  
  Thus
  
      [Z_t+1^2]   =((Ł^(2)_t+1-_t+1^⊤_t+1_t+1
      -_t+1^⊤_t+1_t+1
      -_t+1_t+1^⊤_t+1)^∞)_t+1,t+1
         =(_t+1^∞)_t+1,t+1=σ_t+1,t+1^∞.
  
  where the second equality applies the identity (<ref>).
  This concludes the proof of t+1^(c).
  
  Finally, to show t+1^(d), observe that
  
      ([             0; _t+1^⊤        ][       _t+1   _t+1_t+1; _t+1_t+1^⊤       _t+1 ][      _t+1;    0      ])^∞
      =[ Ł_t+1^(0,∞) Ł_t+1^(1,∞); Ł_t+1^(1,∞) Ł_t+1^(2,∞) ]
  
  by Lemma <ref>. The upper-left (2t+1) × (2t+1)
  submatrix of (<ref>) is exactly the second matrix of
  (<ref>), which we have already shown is invertible. So to check
  invertibility of (<ref>), it suffices to show that the Schur
  complement of the lower-right entry is non-zero. By (<ref>), this Schur
  complement is equal to [S_⊥]. Thus, we must show that
  [S_⊥]>0.
  
  Interpreting the (s,s')
  entry of Ł_t^(k,∞) as [Λ^k R_s R_s'], note that
  [S_⊥] in (<ref>) is the variance of the residual of
  the projection of Λ R_t+1 onto the linear span of
  (R_1,…,R_t+1,Λ R_1,…,Λ R_t) with respect to the
  L_2-inner-product (X,Y) ↦[XY].  
  Thus, if [S_⊥]=0, then
  
      Λ R_t+1=α_1 R_1+…+α_t+1R_t+1
      +β_1 Λ R_1+…+β_t Λ R_t
  
  for some scalar constants α_1,…,α_t+1,β_1,…,β_t
  almost surely. Substituting (<ref>) and rearranging to isolate
  R_⊥, we get
  
      (Λ-α_t+1)R_⊥=f(R_1,…,R_t,Λ)
  
  for some quantity f(R_1,…,R_t,Λ) that does not depend on R_⊥.
  By Assumption <ref>(e), Λ is not a constant random
  variable, so on an event of positive probability, we have Λ≠α_t+1. Then conditioning on (R_1,…,R_t,Λ) and on this
  event, we have
  R_⊥=f(R_1,…,R_t,Λ)/(Λ-α_t+1), implying that
  the conditional law of R_⊥ is constant. Recall
  that R_⊥ is independent of (R_1,…,R_t,Λ)—thus R_⊥
  must be a constant random variable unconditionally. However, R_⊥ is a
  mean-zero normal variable with positive variance by
  (<ref>). This is a contradiction, so
  [S_⊥]>0. This shows t+1^(d), concluding the induction.
  
  
  
  
  § PROOF FOR RECTANGULAR MATRICES
  
  
  In this appendix, we prove Theorem <ref> using similar ideas.
  Let us write the iterations (<ref>–<ref>) as
  
      _̊t   =Ø_̆t
      _t   =^⊤^⊤_̊t
      _t   =_t-b_t1_̌1-…-b_t,t-1_̌t-1
      _̌t   =v_t(_1,…,_t,)
      _t   =_̌t
      _t   =Ø^⊤_t
      _t   =_t-a_t1_̆1-…-a_tt_̆t
      _̆t+1   =u_t+1(_1,…,_t,)
  
  Note that _̆t,_̊t,_t,_t ∈^m while _̌t,_t,_t,_t ∈^n.
  
  In the proof, we will identify the limits of the quantities
  
      m^-1_̊s^⊤ (^⊤)^k _̊s'   ≡ m^-1_̆s^⊤ (^⊤)^k _̆s'
      
      m^-1_s^⊤^⊤ (^⊤)^k _̊s'   ≡ m^-1_̌s^⊤^⊤ (^⊤)^k _̆s'
      
      n^-1_̊s^⊤ (^⊤)^k _s'   ≡ n^-1_̆s^⊤(^⊤)^k _̌s'
      
      n^-1_s^⊤ (^⊤)^k _s'   ≡ n^-1_̌s^⊤ (^⊤)^k _̌s'.
  
  In addition to the matrices _t^(j) and _t^(j) in
  (<ref>–<ref>), let us define
  
      _t^(j)=∑_i=0^j (_t_t)^i _t _t
      (_t^⊤_t^⊤)^j-i
      +∑_i=0^j (_t_t)^i_t _t^⊤
      (_t^⊤_t^⊤)^j-i.
  
  For example,
  
      _t^(0)   =_t_t+_t_t^⊤
      _t^(1)   =_t_t_t_t+_t_t_t_t^⊤
      +_t_t_t^⊤_t^⊤
      +_t_t^⊤_t^⊤_t^⊤.
  
  Corresponding to (<ref>–<ref>),
  we then define four families of matrices
  
      _̋t^(2k)=∑_j=0^∞ c_2k,j_t^(j),     _t^(2k+1)=∑_j=0^∞ c_2k+1,j_t^(j),
      _t^(2k+1)=∑_j=0^∞c̅_2k+1,j(_t^(j))^⊤,     Ł_t^(2k)=∑_j=0^∞c̅_2k,j_t^(j)
  
  where c_2k,j,c_2k+1,j,c̅_2k,j,c̅_2k+1,j are certain
  rectangular partial moment coefficients, defined in
  Section <ref> below. We show in (<ref>) below
  that c̅_2k+1,j=γ· c_2k+1,j, so that
  
      _t^(2k+1)=γ· (_t^(2k+1))^⊤.
  
  The limits of (<ref>–<ref>)
  will be identified as the entries of
  lim_m,n →∞_̋t^(2k),_t^(2k+1),_t^(2k+1),Ł_t^(2k).
  
  
  
   §.§ Coefficients for “partial moments”
  
  
  Let {κ_2k}_k ≥ 1 be the rectangular free cumulants for
  the moment sequence (<ref>) with aspect ratio γ=m/n.
  Recall from Section <ref>
  the second cumulant sequence κ̅_2k=γ·κ_2k for
  all k ≥ 1. For notational convenience, we set
  
      κ_0=1,     κ̅_0=1.
  
  We define four sequences of combinatorial coefficients, denoted by
  
      c_2k,j,    c̅_2k,j,
           c_2k+1,j,     c̅_2k+1,j
  
  for integers k,j ≥ 0. These sequences are defined by the initializations
  
      c_0,0=c̅_0,0=1,
           c_0,j=c̅_0,j=0    for  j ≥ 1
  
  and by the recursions, for all j,k ≥ 0,
  
      c_2k+1,j   =∑_m=0^j+1 c_2k,mκ_2(j+1-m)
      c̅_2k+1,j   =∑_m=0^j+1c̅_2k,mκ̅_2(j+1-m)
      
      c_2k+2,j   =∑_m=0^j c_2k+1,mκ̅_2(j-m)
      c̅_2k+2,j   =∑_m=0^j
      c̅_2k+1,mκ_2(j-m).
  
  
  Let
  
      '(2k,ℓ)={π∈'(2k):S ∩{1,…,ℓ}≠ S
       for all  S ∈π}
  
  be the subset of non-crossing partitions π∈'(2k) where no
  set S ∈π is contained in {1,…,ℓ}. We set
  '(2k,0)='(2k). Recall the moment-cumulant
  relations (<ref>),
  where e(π) and o(π) count the number of sets S ∈π whose smallest
  element is even and odd, and m̅_2k is defined from m_2k by
  (<ref>). Then these coefficients admit the following interpretations.
  
  
  For each k ≥ 0,
  
      c_2k+1,j=∑_π∈'(2k+2j+2,2j+1)γ^e(π)∏_S ∈πκ_|S|,
          c̅_2k+1,j=∑_π∈'(2k+2j+2,2j+1)γ^o(π)∏_S ∈πκ_|S|,
  
  and for each k ≥ 1,
  
      c_2k,j=∑_π∈'(2k+2j,2j)γ^e(π)∏_S ∈πκ_|S|,     c̅_2k,j=∑_π∈'(2k+2j,2j)γ^o(π)∏_S ∈πκ_|S|.
  
  In particular, for all j,k ≥ 0, we have
  
      c_1,j=κ_2(j+1),     c̅_1,j=κ̅_2(j+1),
           c_2k,0=m_2k,     c̅_2k,0=m̅_2k.
  
  Finally, for all j,k ≥ 0, we have
  
      c̅_2k+1,j=γ· c_2k+1,j.
  
  
  
  Let us show (<ref>–<ref>)
  by induction on k. By the initialization (<ref>)
  and the recursions (<ref>) and (<ref>), we have
  c_1,j=κ_2(j+1) and c̅_1,j=κ̅_2(j+1)
  for all j ≥ 0. Since the sets of each partition in '(2j+2) must have
  even cardinality, '(2j+2,2j+1) consists of only the
  partition π with the single set {1,…,2j+2} and this partition has
  e(π)=0 and o(π)=1. Applying κ̅_2j+2=γ·κ_2j+2, this shows both identities of
  (<ref>) for k=0.
  
  Assuming that (<ref>) holds for some k ≥ 0, we now
  check (<ref>) for k+1.
  If π∈'(2k+2j+2,2j) ∖'(2k+2j+2,2j+1), then there is a set
  S ∈π containing 2j+1 that is a subset of {1,…,2j+1}. Since
  π is non-crossing and S has even cardinality, this set must be of the form
  S={2m+2,…,2j+1} for some m ∈{0,…,j-1}. 
  This set S has cardinality 2(j-m), and its smallest element is even.
  Removing S from π yields a bijection between all such partitions π
  and the partitions π' ∈(2k+2m+2,2m+1). Thus,
  applying the induction hypothesis (<ref>) with m in
  place of j,
  
      c_2k+1,m·γκ_2(j-m)   =
      ∑_π∈'(2k+2j+2,2j) ∖'(2k+2j+2,2j+1)
      _{2m+2,…,2j+1}∈πγ^e(π)∏_S ∈πκ_|S|,
      c̅_2k+1,m·κ_2(j-m)   =
      ∑_π∈'(2k+2j+2,2j) ∖'(2k+2j+2,2j+1)
      _{2m+2,…,2j+1}∈πγ^o(π)∏_S ∈πκ_|S|.
  
  Summing over m=0,…,j-1, combining with the induction hypothesis
  (<ref>) applied for j,
  and recalling that κ_0=κ̅_0=1
  while κ̅_2j=γ·κ_2j for j ≥ 1, we obtain
  
      ∑_m=0^j c_2k+1,mκ̅_2(j-m)   =∑_π∈'(2k+2j+2,2j)γ^e(π)∏_S ∈πκ_|S'|,
      ∑_m=0^j c̅_2k+1,mκ_2(j-m)   =∑_π∈'(2k+2j+2,2j)γ^o(π)∏_S ∈πκ_|S'|.
  
  Recognizing the left sides as c_2(k+1),j and c̅_2(k+1),j
  by (<ref>–<ref>),
  this shows (<ref>) for k+1.
  
  Now assuming that (<ref>) holds for some k ≥ 1, we
  check (<ref>) for k.
  If π∈'(2k+2j+2,2j+1) ∖'(2k+2j+2,2j+2), then similar to
  the above, there is some
  set S={2m+1,…,2j+2}∈π for some m ∈{0,…,j},
  with cardinality 2(j-m)+2 and whose smallest element is odd. Removing S
  from π yields a bijection between such partitions π
  and the partitions π' ∈(2k+2m,2m). Then applying the induction
  hypothesis (<ref>) with m in place of k,
  
      c_2k,m·κ_2(j-m)+2   =∑_π∈'(2k+2j+2,2j+2) ∖'(2k+2j+2,2j+1)
      _{2m+1,…,2j+2}∈πγ^e(π)∏_S ∈πκ_|S|,
      c̅_2k,m·γκ_2(j-m)+2   =∑_π∈'(2k+2j+2,2j+2) ∖'(2k+2j+2,2j+1)
      _{2m+1,…,2j+2}∈πγ^o(π)∏_S ∈πκ_|S|.
  
  Summing over m=0,…,j, combining with the induction hypothesis
  (<ref>) applied for j+1, and
  applying again κ_0=κ̅_0=1 and κ̅_2j=γ·κ_2j for j ≥ 1, we obtain
  (<ref>) for k. This concludes the induction,
  showing (<ref>) for all k ≥ 0
  and (<ref>) for all k ≥ 1.
  
  The statements c_1,j=κ_2(j+1) and
  c̅_1,j=κ̅_2(j+1) are already shown.
  The statements c_2k,0=m_2k and c̅_2k,0=m̅_2k
  follow from '(2k,0)='(2k), together with the
  moment-cumulant relations (<ref>). Finally, for the
  identity (<ref>), note that this holds for
  k=0 because κ̅_2(j+1)=γ·κ_2(j+1). Supposing
  that it holds for k-1, we may compose (<ref>) and
  (<ref>) to get
  
      c_2k+1,j   =∑_m=0^j+1∑_p=0^m c_2k-1,pκ̅_2(m-p)κ_2(j+1-m)
         =∑_p=0^j+1 c_2k-1,p∑_m=p^j+1κ̅_2(m-p)κ_2(j+1-m)
      =∑_p=0^j+1 c_2k-1,p∑_m=0^j-p+1κ̅_2mκ_2(j-p+1-m).
  
  Similarly
  
      c̅_2k+1,j=∑_p=0^j+1c̅_2k-1,p∑_m=0^j-p+1κ_2mκ̅_2(j-p+1-m).
  
  Comparing these two expressions and applying c̅_2k-1,p=γ·
  c_2k-1,p for each p=0,…,j+1, we get c̅_2k+1,j=γ·
  c_2k+1,j. This shows (<ref>) for all k.
  
  
  
  
   §.§ Partial moment identities
  
  
  Recalling the definitions of
  _̋t^(2k),_t^(2k+1),_t^(2k+1),Ł_t^(2k) from (<ref>),
  we now collect several identities derived from the recursions for
  c_2k,j,c_2k+1,j,c̅_2k,j,c̅_2k+1,j.
  
  
  For every t ≥ 1,
  
      _̋t^(0)   =_t
      Ł_t^(0)   =_t
      _t^(1)   =_t^⊤_t+_t_t^⊤
         =γ^-1· (_t_t+_t_t)
      _t^(1)   =_t^⊤_t+_t_t^⊤
         =γ· (_t_t+_t_t)
      Ł_t^(2)   =γ· (_t^⊤_t_t+_t^⊤_t_t
      +_t_t^⊤_t+_t)
      _̋t^(2)   =γ^-1·
      (_t^⊤_t_t+_t^⊤_t_t
      +_t_t^⊤_t+_t)
  
  
  
  The identities (<ref>) and (<ref>) follow immediately from
  the initializations c_0,0=c̅_0,0=1 and
  c_0,j=c̅_0,j=0 for all j ≥ 1, and the
  observations _t^(0)=_t and _t^(0)=_t.
  
  For (<ref>), let us separate the terms of _t^(1) ending with
  _t from those ending with _t^⊤. Applying
  c_1,j=κ_2(j+1), this yields
  
      _t^(1)=∑_j=0^∞κ_2(j+1)((_t_t)^j _t_t
      +∑_i=0^j-1 (_t_t)^i _t_t
      (_t^⊤_t^⊤)^j-i
      +∑_i=0^j (_t_t)^i_t_t^⊤
      (_t^⊤_t^⊤)^j-i).
  
  Observe that
  
      ∑_j=0^∞κ_2(j+1) (_t_t)^j _t_t
      =_t^⊤_t,
  
  while
  
      ∑_j=0^∞κ_2(j+1)(∑_i=0^j-1 (_t_t)^i _t_t
      (_t^⊤_t^⊤)^j-i
      +∑_i=0^j (_t_t)^i_t_t^⊤
      (_t^⊤_t^⊤)^j-i)
         =∑_j=0^∞κ_2(j+1)(∑_i=0^j-1
      (_t_t)^i _t_t_t^⊤(_t^⊤_t^⊤)^j-1-i+∑_i=0^j (_t_t)^i
      _t(_t^⊤_t^⊤)^j-i)_t^⊤
      =_t_t^⊤.
  
  Thus we obtain (<ref>). The identity (<ref>) follows
  analogously by separating the terms of _t^(1) starting with _t
  from those starting with _t. (The factor γ^-1 cancels the factor
  of γ in the definitions of _t and _t.) The identities
  (<ref>–<ref>) follow from
  (<ref>–<ref>) and the relation
  _t^(1)=γ· (_t^(1))^⊤ from (<ref>).
  
  For (<ref>), applying (<ref>) and the identity
  c̅_1,m=κ̅_2(m+1)=γ·κ_2(m+1), we have
  
      Ł_t^(2)=∑_j=0^∞c̅_2,j_t^(j)   =γ·∑_j=0^∞(∑_m=0^j
      κ_2(m+1)κ_2(j-m))
             ·(∑_i=0^j (_t_t)^i
      _t(_t^⊤_t^⊤)^j-i
      +∑_i=0^j-1 (_t_t)^i
      _t_t_t^⊤(_t^⊤_t^⊤)^j-1-i).
  
  Collecting terms by powers of _t_t and its transpose, this is
  
      Ł_t^(2)   =γ·∑_i=0^∞∑_p=0^∞[
      (∑_m=0^i+pκ_2(m+1)κ_2(i+p-m))
      (_t_t)^i_t(_t^⊤_t^⊤)^p
         
      +(∑_m=0^i+p+1κ_2(m+1)κ_2(i+p+1-m))
      (_t_t)^i_t_t_t^⊤(_t^⊤_t^⊤)^p
      ].
  
  From the definitions and the notation κ_0=1, we now identify
  
      _t^⊤_t_t   =∑_i=0^∞∑_p=0^∞κ_2(i+1)κ_2(p+1)
      (_t_t)^i_t_t_t^⊤(_t^⊤_t^⊤)^p
      _t   =∑_i=0^∞∑_p=0^∞κ_2(i+p+1)κ_0
      (_t_t)^i_t(_t^⊤_t^⊤)^p
      +∑_i=0^∞∑_p=0^∞κ_2(i+p+2)κ_0
      (_t_t)^i_t_t_t^⊤(_t^⊤_t^⊤)^p
      _t^⊤_t_t   =∑_m=0^∞κ_2(m+1)
      (_t_t)^m+1
         ·∑_q=0^∞κ_2(q+1)(∑_p=0^q
      (_t_t)^q-p_t(_t^⊤_t^⊤)^p
      +∑_p=0^q-1
      (_t_t)^q-1-p_t_t_t^⊤
      (_t^⊤_t^⊤)^p)
         =∑_i=0^∞∑_p=0^∞(∑_m=0^i-1κ_2(m+1)κ_2(i+p-m))
      (_t_t)^i_t(_t^⊤_t^⊤)^p
         +∑_i=0^∞∑_p=0^∞(∑_m=0^i-1κ_2(m+1)κ_2(i+p+1-m))
      (_t_t)^i_t_t_t^⊤(_t^⊤_t^⊤)^p
      _t _t^⊤_t   =(_t^⊤_t_t)^⊤
         =∑_i=0^∞∑_p=0^∞(∑_m=i^i+p-1κ_2(m+1)κ_2(i+p-m))
      (_t_t)^i_t(_t^⊤_t^⊤)^p
         +∑_i=0^∞∑_p=0^∞(∑_m=i+1^i+pκ_2(m+1)κ_2(i+p+1-m))
      (_t_t)^i_t_t_t^⊤(_t^⊤_t^⊤)^p
  
  Summing these four expressions and comparing with (<ref>) 
  yields the identity (<ref>).
  
  For (<ref>), we may write similarly
  
      _̋t^(2)   =γ·∑_j=0^∞(∑_m=0^j
      κ_2(m+1)κ_2(j-m)γ^-{m=j})
             ·(∑_i=0^j (_t_t)^i
      _t(_t^⊤_t^⊤)^j-i
      +∑_i=0^j-1 (_t_t)^i
      _t_t_t^⊤(_t^⊤_t^⊤)^j-1-i),
  
  where the factor γ^-{m=j} comes from the fact that
  κ̅_2(j-m)=γκ_2(j-m) if m<j, but
  κ̅_2(j-m)=κ_2(j-m)=1 for m=j.
  This is matched by the observation that _t^⊤_t_t,
  _t^⊤_t_t, and _t_t^⊤_t on the right of
  (<ref>) have a factor of γ^2, whereas the last term _t
  has a factor of only γ. Then (<ref>) follows from an argument
  analogous to the above, and we omit this for brevity.
  
  
  
  Define
  
      _t=[         _t _t _t+_t_t;       _t^⊤    _t^⊤_t+ ],
          _t=[         _t _t _t+_t_t;       _t^⊤    _t^⊤_t+ ].
  
  Then for every t ≥ 1 and k ≥ 0,
  
      [        _̋t^(2k)   (_t^(2k+1))^⊤;       _t^(2k+1) γ^-1·Ł_t^(2k+2) ]
         =[     ∑_j=0^∞ c_2k,j (_t_t)^j ∑_j=0^∞ c_2k,j+1 (_t^(j))^⊤; ∑_j=0^∞ c_2k+1,j (_t_t)^j_t      ∑_j=0^∞ c_2k+1,j_t^(j) ]_t
      
         =c_2k,0[      _̋t^(0)   (_t^(1))^⊤;       _t^(1) γ^-1·Ł_t^(2) ]
      +_t^⊤[                                 0 ∑_j=0^∞ c_2k,j+1_t^⊤ (_t^⊤_t^⊤)^j;       ∑_j=0^∞ c_2k,j+1 (_t_t)^j_t            ∑_j=0^∞ c_2k,j+1_t^(j) ]_t,
  
  and
  
      [      Ł_t^(2k) (_t^(2k+1))^⊤;     _t^(2k+1)  γ·_̋t^(2k+2) ]
         =[     ∑_j=0^∞c̅_2k,j (_t_t)^j      ∑_j=0^∞c̅_2k,j+1_t^(j); ∑_j=0^∞c̅_2k+1,j (_t_t)^j_t      ∑_j=0^∞c̅_2k+1,j_t^(j) ]_t
      
         =c̅_2k,0[    Ł_t^(0) (_t^(1))^⊤;     _t^(1)  γ·_̋t^(2) ]
      +_t^⊤[                                 0 ∑_j=0^∞c̅_2k,j+1_t^⊤ (_t^⊤_t^⊤)^j;       ∑_j=0^∞c̅_2k,j+1 (_t_t)^j_t            ∑_j=0^∞c̅_2k,j+1_t^(j) ]_t.
  
  
  
  The arguments are similar to those of Lemma <ref>.
  Applying (<ref>), the definitions of _t^(1) and _t from
  (<ref>) and (<ref>), and the
  notation κ_0=1, we have
  
      _t^⊤=[      _t      _t;  _t^(1) _t^⊤_t+ ]
      =[                    _t                    _t; ∑_j=0^∞κ_2(j+1)_t^(j)  ∑_j=0^∞κ_2j (_t_t)^j ].
  
  Then applying the definitions of _t^(j), _t^(j), and
  _t^(j) from (<ref>–<ref>) and (<ref>),
  we may compute
  
      [ _t _t ][ ∑_j=0^∞ c_2k,j(_t^⊤_t^⊤)^j;     ∑_j=0^∞ c_2k,j+1_t^(j) ]
         =∑_j=0^∞ c_2k,j_t (_t^⊤_t^⊤)^j
      +∑_j=0^∞ c_2k,j+1(∑_i=0^j
      (_t_t)^i+1_t(_t^⊤_t^⊤)^j-i
      +∑_i=0^j (_t_t)^i_t_t_t^⊤
      (_t^⊤_t^⊤)^j-i)
         =∑_j=0^∞ c_2k,j_t^(j)=_̋t^(2k),
         [ _t _t ][ ∑_j=0^∞ c_2k+1,j_t^⊤(_t^⊤_t^⊤)^j;           ∑_j=0^∞ c_2k+1,j_t^(j) ]
         =∑_j=0^∞ c_2k+1,j(
      _t_t^⊤(_t^⊤_t^⊤)^j
      +∑_i=0^j (_t_t)^i
      _t_t(_t^⊤_t^⊤)^j-i
      +∑_i=0^j-1 (_t_i)^i+1_t_i^⊤
      (_t^⊤_t^⊤)^j-1-i)
         =∑_j=0^∞
      c_2k+1,j(_t^(j))^⊤=(_t^(2k+1))^⊤.
  
  We may also compute, analogously to (<ref>),
  
      [ ∑_j=0^∞κ_2(j+1)_t^(j)  ∑_j=0^∞κ_2j (_t_t)^j ][ ∑_j=0^∞ c_2k,j(_t^⊤_t^⊤)^j;     ∑_j=0^∞ c_2k,j+1_t^(j) ]
           =(∑_j=0^∞κ_2(j+1)∑_i=0^j (_t_t)^i
      (_t_t+_t_t^⊤)(_t^⊤_t^⊤)^j-i)
      ·(∑_p=0^∞ c_2k,p(_t^⊤_t^⊤)^p)
                 +(∑_j=0^∞κ_2j (_t_t)^j)
      (∑_p=0^∞ c_2k,p+1∑_q=0^p (_t_t)^q
      (_t_t+_t_t^⊤)(_t^⊤_t^⊤)^p-q)
           =∑_i=0^∞∑_r=0^∞(∑_m=0^i+r+1κ_2mc_2k,i+r+1-m)
      (_t_t)^i (_t_t+_t_t^⊤)
      (_t^⊤_t^⊤)^r
           =∑_i=0^∞∑_r=0^∞ c_2k+1,i+r
      (_t_t)^i (_t_t+_t_t^⊤)
      (_t^⊤_t^⊤)^r=_t^(2k+1).
  
  In the last two equalities above, we used the recursion (<ref>)
  and the definitions of _t^(j) and _t^(2k+1). Similarly,
  
      [ ∑_j=0^∞κ_2(j+1)_t^(j)  ∑_j=0^∞κ_2j (_t_t)^j ][ ∑_j=0^∞ c_2k+1,j_t^⊤(_t^⊤_t^⊤)^j;           ∑_j=0^∞ c_2k+1,j_t^(j) ]
           =(∑_j=0^∞κ_2(j+1)∑_i=0^j (_t_t)^i
      (_t_t+_t_t^⊤)(_t^⊤_t^⊤)^j-i)
      ·(∑_p=0^∞ c_2k+1,p_t^⊤(_t^⊤_t^⊤)^p)
             +(∑_j=0^∞κ_2j (_t_t)^j)
      ·∑_p=0^∞ c_2k+1,p(∑_q=0^p
      (_t_t)^q_t(_t^⊤_t^⊤)^p-q
         +∑_q=0^p-1 (_t_t)^q_t_t_t^⊤
      (_t^⊤_t^⊤)^p-1-q)
           =∑_i=0^∞∑_r=0^∞(
      (∑_m=0^i+r+1κ_2m
      c_2k+1,i+r+1-m)(_t_t)^i_t_t_t^⊤
      (_t^⊤_t^⊤)^r
         +(∑_m=0^i+rκ_2m
      c_2k+1,i+r-m)(_t_t)^i_t(_t^⊤_t^⊤)^r
      )
           =γ^-1∑_i=0^∞∑_r=0^∞(c̅_2k+2,i+r+1(_t_t)^i_t_t_t^⊤
      (_t^⊤_t^⊤)^r
      +c̅_2k+2,i+r(_t_t)^i_t(_t^⊤_t^⊤)^r)
           =γ^-1Ł_t^(2k+2).
  
  In the last two equalities above, we used the identity
  c_2k+1,j=γ^-1c̅_2k+1,j, the recursion
  (<ref>), and the definitions of _t^(j) and
  Ł_t^(2k+2). Recalling (<ref>),
  stacking (<ref>), (<ref>),
  (<ref>), and (<ref>),
  and taking the transpose yields (<ref>).
  
  For (<ref>), applying again
  the form (<ref>) for _t^⊤, we may compute
  
      _t^⊤[                          0; ∑_j=0^∞ c_2k,j+1(_t_t)^j_t ]
      =[                ∑_j=0^∞ c_2k,j+1(_t_t)^j+1; ∑_ℓ=0^∞(∑_j=0^ℓκ_2jc_2k,ℓ+1-j)
      (_t_t)^ℓ_t ].
  
  Let us apply c_0,0=1, c_0,ℓ=0 for ℓ≥ 1,
  c_1,ℓ=κ_2(ℓ+1), and the recursion (<ref>) in the
  form
  
      c_2k+1,ℓ-c_2k,0κ_2(ℓ+1)=∑_j=0^ℓ
      c_2k,ℓ+1-jκ_2j.
  
  This gives
  
      _t^⊤[                          0; ∑_j=0^∞ c_2k,j+1(_t_t)^j_t ]   =[     ∑_ℓ=0^∞ (c_2k,ℓ-c_2k,0c_0,ℓ)
      (_t_t)^ℓ; ∑_ℓ=0^∞ (c_2k+1,ℓ-c_2k,0c_1,ℓ)
      (_t_t)^ℓ_t ]
  
  Applying the same computations as leading to (<ref>) and
  (<ref>), with the recursion (<ref>) in the forms
  
      ∑_m=0^i+r+1κ_2mc_2k,i+r+2-m
      =c_2k+1,i+r+1-κ_2(i+r+2)c_2k,0,
        ∑_m=0^i+rκ_2mc_2k,i+r+1-m
      =c_2k+1,i+r-κ_2(i+r+1)c_2k,0
  
  replacing the final two steps of (<ref>),
  we have also
  
      _t^⊤[ ∑_j=0^∞ c_2k,j+1_t^⊤
      (_t^⊤_t^⊤)^j;            ∑_j=0^∞ c_2k,j+1_t^(j) ]   =[              ∑_j=0^∞ c_2k,j+1(_t^(j))^⊤; ∑_j=0^∞
      (c_2k+1,j-κ_2(j+1)c_2k,0)_t^(j) ]
         =[ ∑_ℓ=0^∞ (c_2k,ℓ+1-c_2k,0c_0,ℓ+1)
      (_t^(ℓ))^⊤;        ∑_ℓ=0^∞ (c_2k+1,ℓ-c_2k,0c_1,ℓ)_t^(ℓ) ].
  
  Stacking (<ref>) and (<ref>), multiplying on the
  right by _t, and then applying
  (<ref>) for k and also for k=0, yields
  
      _t^⊤[                                 0 ∑_j=0^∞ c_2k,j+1_t^⊤
      (_t^⊤_t^⊤)^j;        ∑_j=0^∞ c_2k,j+1(_t_t)^j_t            ∑_j=0^∞ c_2k,j+1_t^(j) ]_t
                 =[        _̋t^(2k)   (_t^(2k+1))^⊤;       _t^(2k+1) γ^-1·Ł_t^(2k+2) ]
      -c_2k,0[      _̋t^(0)   (_t^(1))^⊤;       _t^(1) γ^-1·Ł_t^(2) ].
  
  Rearranging this yields (<ref>).
  
  The identities (<ref>) and (<ref>) follow from writing
  
      _t^⊤=[      _t      _t;  _t^(1) _t^⊤_t+ ]
      =[                         _t                         _t; ∑_j=0^∞κ̅_2(j+1)(_t^(j))^⊤       ∑_j=0^∞κ̅_2j(_t_t)^j ]
  
  and applying the same arguments, which we omit for brevity.
  
  
  
  
   §.§ Conditioning argument
  
  
  We now prove Theorem <ref> using a conditioning argument similar to the
  symmetric square setting.
  Recall the definition of ∈^min(m,n) from
  (<ref>). Let us define
  
      _m ∈^m,     _n ∈^n
  
  to be this vector extended by m-n and n-m additional 0's,
  respectively. Thus =_m if m ≤ n, and
  =_n if n ≤ m. By Assumption <ref>(c), we then
  have
  
      _m Λ_m,     _n Λ_n
  
  where Λ_m denotes a mixture of Λ and the point mass at 0
  when γ>1, and Λ_n denotes such a mixture when γ<1.
  
  Let _t-1^(2k+1)∈^(t-1) × t denote the first t-1
  rows of _t^(2k+1)∈^t × t. The following extended
  lemma implies Theorem <ref>, where parts (b) and (e) identify the
  almost sure limits of (<ref>–<ref>).
  
  
  Suppose Assumption <ref> holds. Almost surely for each
  t=1,2,3,…:
  
  
    * For all fixed j,k ≥ 0, there exist deterministic limit matrices
  
      (_t^∞,_t^∞,_t^(j,∞),_t^∞,_t^∞,_̋t^(2k,∞),_t-1^(2k+1,∞))
         =lim_m,n →∞
      (_t,_t,_t^(j),_t,_t,_̋t^(2k),_t-1^(2k+1))
  
  
    * For some random variables R_1,…,R_t,P̅_1,…,P̅_t-1
  with finite moments of all orders,
  
      (_̊1,…,_̊t,_1,…,_t-1,_m) 
      (R_1,…,R_t,P̅_1,…,P̅_t-1,Λ_m).
  
  For each k ≥ 0,
  
      [(R_1,…,R_t)^⊤Λ_m^2k(R_1,…,R_t)]   =_̋t^(2k,∞)
      [(P̅_1,…,P̅_t-1)^⊤Λ_m^2k
      (R_1,…,R_t)]   =_t-1^(2k+1,∞).
  
  
    * (_̌1,…,_̌t,_1,…,_t,)  (V_1,…,V_t,Z_1,…,Z_t,F)
  as described in Theorem <ref>.
  
    * For all fixed j,k ≥ 0, there exist deterministic limit matrices
  
      (_t^∞,_t^∞,_t^(j,∞),_t^∞,_t^∞,Ł_t^(2k,∞),_t^(2k+1,∞))
         =lim_m,n →∞
      (_t,_t,_t^(j),_t,_t,Ł_t^(2k),_t^(2k+1))
  
  
    * For some random variables P_1,…,P_t,R̅_1,…,R̅_t with
  finite moments of all orders,
  
      (_1,…,_t,^⊤_̊1,…,^⊤_̊t,_n)
       (P_1,…,P_t,R̅_1,…,R̅_t,Λ_n).
  
  For each k ≥ 0,
  
      [(R̅_1,…,R̅_t)^⊤Λ_n^2k(P_1,…,P_t)]
         =_t^(2k+1,∞),
      [(P_1,…,P_t)^⊤Λ_n^2k(P_1,…,P_t)]
         =Ł_t^(2k,∞).
  
  
    * (_̆1,…,_̆t+1,_1,…,_t,)
   (U_1,…,U_t+1,Y_1,…,Y_t,E)
  as described in Theorem <ref>.
  
    * The matrices
  
      [          _t^∞      _t^∞_t^∞; _t^∞ (_t^∞)^⊤          _t^∞ ],
          [          _t^∞      _t^∞_t^∞; _t^∞ (_t^∞)^⊤          _t^∞ ]
  
  are both non-singular.
  
  
  
  
  Denote by t^(a),t^(b),… the claims of part (a), part (b), etc. up
  to and including iteration t. We induct on t.
  We will omit details of the argument that are similar to the proof
  of Theorem <ref>.
  
  
  Step 1: t=1. We have
  _1 →_1^∞=[U_1^2], _1=0, _1=0,
  and κ_2k→κ_2k^∞ where
  κ_2k^∞ are the rectangular free cumulants defined by the limiting
  moments
  
      m_2k^∞=lim_m,n →∞1/m∑_i=1^min(m,n)λ_i^2k=[Λ_m^2k].
  
  Then also (c_2k,j,c_2k+1,j,c̅_2k,j,c̅_2k+1,j) →
  (c_2k,j^∞,c_2k+1,j^∞,c̅_2k,j^∞,
  c̅_2k+1,j^∞),
  and claim 1^(a) follows from the definitions.
  
  Since _̊1=Ø_̆1, by Proposition <ref>,
  
      (_̊1,_m)  (R_1,Λ_m)
  
  where R_1 ∼(0,[U_1^2]) is independent of Λ_m. Then
  [Λ_m^2kR_1^2]=[Λ_m^2k][R_1^2]
  =m_2k^∞[U_1^2]. Note that _1^(j)=0 for j ≥ 1, so
  _̋1^(2k)=c_2k,0_1^(0)=m_2k⟨_̆1^2 ⟩,
  the last equality using m_2k=c_2k,0 by Lemma <ref>.
  Thus _̋1^(2k,∞)=m_2k^∞[U_1^2], and 
  this shows 1^(b).
  
  For 1^(c), note that c_2,0=m_2=κ_2. Then 1^(b) implies
  
      1/n^⊤_̊1^2
      =m/n·1/m∑_i=1^min(m,n)λ_i^2 r_i1^2 →γ·[Λ_m^2R_1^2]
      =γκ_2^∞_1^∞.
  
  Since _1=^⊤^⊤_̊1,
  Proposition <ref> shows
  
      (_1,)  (Z_1,F)
  
  where Z_1 ∼(0,γκ_2^∞_1^∞) is
  independent of
  F. Identifying _1^∞=γκ_2^∞_1^∞ and applying Proposition <ref> for the joint 
  convergence with _̌1=v_1(_1,), this shows 1^(c).
  
  Observe that 1^(c) implies
  _1 →_1^∞=[V_1^2]. As ∂_1 v_1
  satisfies (<ref>) and is continuous on a set of probability 1 under
  the limit law (Z_1,F), we have also
  _1 →_1^∞=[∂_1 v_1(Z_1,F)]
  by Proposition <ref>. Then 1^(d) follows from the
  definitions.
  
  For 1^(e), define _1 ∈^n as the first n entries of
  _̊1 if n ≤ m, or _̊1 extended by n-m additional i.i.d.(0,[U_1^2]) random variables if n>m. By Proposition
  <ref>(b),
  
      (_1,_n)  (Ř_1,Λ_n)
  
  where Ř_1 ∼(0,[U_1^2]) is independent of Λ_n. Note
  that ^⊤_̊1 ∈^n may be written as the entrywise product of
  _n with _1, in both cases n ≤ m and n>m. Thus
  
      (^⊤_̊1,_n)  (R̅_1,Λ_n),
          R̅_1=Λ_n Ř_1.
  
  To analyze the joint convergence with _1, we now condition on
  _̆1,_̊1,_1,_̌1,,,.
  The law of  is then conditioned on the event
  ^⊤_̊1=_1. As [Z_1]=γκ_2^∞[U_1^2]>0 by the given assumptions, we have
  n^-1_1^⊤_1 ≠ 0 for all large n.
  Then by Proposition <ref>,
  the conditional law of  is
  
      ^⊤_̊1(_1^⊤_1)^-1_1^⊤+_(^⊤_̊1)^⊥__1^⊥^⊤
  
  where ∈^(n-1) × (n-1) is an independent
  Haar-orthogonal matrix. So we may replace the update
  _1=_̌1 by
  
      _1=_∥+_⊥,     _∥=^⊤_̊1(_1^⊤_1)^-1_1^⊤_̌1,     _⊥=_(^⊤_̊1)^⊥__1^⊥^⊤_̌1.
  
  By 1^(c) and Proposition <ref>,
  n^-1_1^⊤_1 →_1^∞ and n^-1_1^⊤_̌1 →_1^∞_1^∞. Then by Proposition <ref>,
  
      (_∥,^⊤_̊1,_n) 
      (P_∥,R̅_1,Λ_n),      P_∥=_1^∞·R̅_1.
  
  For _⊥, observe that
  
      n^-1__1^⊥^⊤_̌1^2
      =n^-1(_̌1^2-(_̌1^⊤_1)^2/_1^2)
      →_1^∞-(_1^∞)^2_1^∞,
  
  so Proposition <ref> shows
  
      _⊥ P_⊥∼(0, _1^∞-(_1^∞)^2_1^∞)
  
  where P_⊥ is independent of (R̅_1,Λ_n). Then
  
      (_1,^⊤_̊1,_n)  (P_1,R̅_1,Λ_n),     
      P_1=_1^∞·R̅_1+P_⊥.
  
  Since P_⊥ has mean 0 and is independent of (R̅_1,Λ_n), we
  have
  
      [Λ_n^2kP_1R̅_1]
         =_1^∞·[Λ_n^2kR̅_1^2]
         =_1^∞·lim_n →∞ n^-1_̊1^⊤(^⊤)^k^⊤_̊1
         =γ_1^∞·lim_n →∞ m^-1_̊1^⊤ (^⊤)^k+1_̊1
         =γ_1^∞·[Λ_m^2k+2R_1^2]
      =γ c_2k+2,0^∞_1^∞_1^∞
  
  where the last equality applies 1^(b).
  By the recursion (<ref>) and identity (<ref>), we have
  c_2k+2,0=c_2k+1,0=γ^-1c̅_2k+1,0. We have also
  _1^(0)=_1_1 and _1^(j)=0 for j ≥ 1, because
  _1=0. Then _1^(2k+1)=c̅_2k+1,0_1_1,
  so the above is simply
  
      [Λ_n^2kP_1R̅_1]=_1^(2k+1,∞).
  
  Similarly, applying the above identity and also
  [Λ_n^2k]=m̅_2k^∞=c̅_2k,0^∞ by Lemma
  <ref>, we have
  
      [Λ_n^2kP_1^2]   =(_1^∞)^2 ·[Λ_n^2kR̅_1^2]+[Λ_n^2kP_⊥^2]
         =c̅_2k+1,0^∞ (_1^∞)^2_1^∞
      +c̅_2k,0^∞·
      (_1^∞-(_1^∞)^2_1^∞).
  
  Identifying _1=γκ_2 _1=κ̅_2_1,
  applying c̅_2k+1,0=c̅_2k,0κ̅_2
  +c̅_2k,1 by (<ref>), and then identifying
  _1^(0)=_1, _1^(1)=_1^2_1,
  and _1^(j)=0 for j ≥ 2, this is
  
      [Λ_n^2kP_1^2]=
      (c̅_2k+1,0^∞-c̅_2k,0^∞κ̅_2^∞)(_1^∞)^2_1^∞
      +c̅_2k,0^∞_1^∞
      =c̅_2k,1^∞_1^(1,∞)+c̅_2k,0^∞_1^(0,∞)=Ł_1^(2k,∞).
  
  This shows 1^(e).
  
  For 1^(f), we now condition on
  _̆1,_̊1,_1,_̌1,_1,,,. Then the law of Ø is
  conditioned on the event _̊1=Ø_̆1. By assumption,
  [U_1^2]>0, so m^-1_̆1^⊤_̆1 ≠ 0 for all large m.
  Then the conditional law of Ø is
  
      _̊1(_̆1^⊤_̆1)^-1_̆1^⊤+__̊1^⊥Ø̃__̆1^⊥^⊤.
  
  So the update for _1=Ø^⊤_1 may be replaced by
  
      _1=_∥+_⊥,     _∥=_̆1(_̆1^⊤_̆1)^-1_̊1^⊤_1,     _⊥=__̆1^⊥Ø̃^⊤__̊1^⊥^⊤_1.
  
  Applying 1^(e) and the above observation
  _1^(1)=c̅_1,0_1_1=γκ_2 _1_1,
  we have
  
      1/m_̊1^⊤_1
         =n/m·1/n_̊1^⊤_1
      →γ^-1_1^(1,∞)=
      κ_2^∞_1^∞_1^∞
      =a_11^∞_1^∞,
      1/m__̊1^⊥^⊤_1^2
         =n/m·1/n(_1^2
      -(_̊1^⊤_1)^2/_̊1^2)
         →γ^-1·(Ł_1^(2,∞)-(_1^(1,∞))^2/γ·_1^∞)
      =(γ^-1·(c̅_2,1_1^(1)+c̅_2,0_1^(0)
      -γκ_2^2 _1^2_1))^∞.
  
  Recalling from the above that _1^(1)=_1^2_1, and applying
  γ^-1c̅_2,1=γ^-1(c̅_1,0κ_2+c̅_1,1)
  =κ_2^2+κ_4 and
  γ^-1c̅_2,0=γ^-1c̅_1,0=κ_2, this yields
  
      1/m__̊1^⊥^⊤_1^2
      →κ_4^∞_1^(1,∞)+κ_2^∞_1^(0,∞)=_1^∞.
  
  So _⊥→ Q_⊥∼(0,_1^∞) where this is independent
  of (U_1,E), and
  
      (_1,_̆1,)  (a_11^∞ U_1+Q_⊥,U_1,E).
  
  Then applying _1=_1-a_11_̆1, a_11→ a_11^∞ by 1^(d),
  and Propositions <ref> and <ref>,
  
      (_̆1,_̆2,_1,)  (U_1,U_2,Y_1,E)
  
  where Y_1=Q_⊥∼(0,_1^∞) and U_2=u_2(Y_1,E). This
  yields 1^(f).
  
  For 1^(g), observe first that _1^∞=[U_1^2]>0, _1=0,
  and _1^∞=γκ_2^∞[U_1^2]>0 by the given
  assumptions. The Schur-complement
  _1^∞-(_1^∞)^2_1^∞ in the second matrix of
  1^(g)
  is the residual variance of projecting of V_1 onto the span of Z_1, which is
  positive by Assumption <ref>(f), so the second matrix of 1^(g)
  is invertible. By (<ref>), this shows also that
  
      [P_⊥]>0.
  
  For the first matrix of 1^(g), it remains to show that _1^∞>0.
  Note that by (<ref>), _1^∞ is the residual variance
  of projecting P̅_1 onto the span of R_1. If this were 0, then
  P̅_1=α R_1 for some constant α∈ with probability 1.
  Applying 1^(b) and 1^(e), we then have
  
      0=[Λ_m^2(P̅_1-α R_1)^2]
      =lim_m,n →∞ m^-1^⊤_1-α^⊤_̊1^2
      =γ^-1·[(Λ_n^2 P_1-αR̅_1)^2],
  
  so also Λ_n^2P_1=αR̅_1 with probability 1.
  Recalling P_1=_1^∞·R̅_1+P_⊥, this shows
  Λ_n^2 P_⊥=(α-_1^∞·Λ_n^2)R̅_1.
  Since Λ_n^2 is not identically 0, and P_⊥ is
  independent of (Λ_n,R̅_1), we must then have that P_⊥ is
  constant with probability 1, but this contradicts that P_⊥∼(0,[P_⊥]) where [P_⊥]>0 as argued above. So
  _1^∞>0, and the first matrix of 1^(g) is also invertible.
  
  
  Step 2: Analysis of _̊t+1. Suppose that t^(a-g) hold. To show
  t+1^(a), observe that the limits _t+1^∞ and
  _t+1^∞ exist by t^(f),
  Proposition <ref>, and the given
  conditions for the functions ∂_s' u_s. Furthermore,
  by t^(d) and t^(f), the limits
  _t+1^(j,∞), _t+1^∞, _t+1^∞, and
  _̋t+1^(2k,∞) also exist because these matrices do not depend on
  _̌t+1 or its derivatives. Each term constituting _t+1^(j)
  in (<ref>)
  may be written as either _t+1 or _t+1_t+1^⊤ times
  a matrix that depends only on _t+1, _t+1_t+1,
  and _t+1_t+1_t+1^⊤.
  Then the first t rows of _t+1^(j) also do not depend on _̌t+1 or
  its derivatives, so _t^(2k+1,∞) exists. This establishes
  t+1^(a).
  
  Let us now show t+1^(b). Define the matrices
  
      _t=[ _̆1   ⋯ _̆t ],
        _t=[ _̊1   ⋯ _̊t ], 
        _t=[ _1  ⋯ _t ],
  
  
      _t=[ _̌1   ⋯ _̌t ],
        _t=[ _1  ⋯ _t ], 
        _t=[ _1  ⋯ _t ].
  
  Conditional on the AMP iterates up to _̆t+1,
  the law of Ø is conditioned on
  
      [ _t _t ][     -_t;         ]
      =Ø[ _t _t ].
  
  Let us introduce
  
      _t=m^-1[ _t^⊤_t _t^⊤_t; _t^⊤_t _t^⊤_t ],
  
  noting that by t^(f), t^(g), and Proposition <ref>,
  
      _t →_t^∞
      =[          _t^∞      _t^∞_t^∞; _t^∞ (_t^∞)^⊤          _t^∞ ]
  
  where _t^∞ is invertible. Then by
  Proposition <ref>, for all large n,
  the conditional law of Ø is
  
      [ _t _t ][     -_t;         ]_t^-1· m^-1[ _t^⊤; _t^⊤ ]
      +_(_t,_t)^⊥Ø̃_(_t,_t)^⊥^⊤
  
  where Ø̃∈^(m-2t) × (m-2t) is an independent
  Haar-orthogonal matrix. We may thus replace the
  update for _̊t+1 by
  
      _̊t+1   =_̊∥+_̊⊥
      _̊∥   =[ _t _t ][     -_t;         ]_t^-1· m^-1[ _t^⊤; _t^⊤ ]_̆t+1
      _̊⊥   =_(_t,_t)^⊥Ø̃_(_t,_t)^⊥^⊤_̆t+1.
  
  
  For _̊∥, define
  
      _t^∞=[ [U_1U_t+1];          ⋮; [U_tU_t+1] ]∈^t,     _t^∞=[ [∂_1 u_t+1(Y_1,…,Y_t,E)];                        ⋮; [∂_t u_t+1(Y_1,…,Y_t,E)] ]∈^t,
  
  which are the last columns of _t+1^∞ and
  (_t+1^∞)^⊤ with their last entries removed. Observe that
  
      m^-1_t^⊤_̆t+1→_t^∞,
           m^-1_t^⊤_̆t+1→_t^∞_t^∞.
  
  Then by arguments similar to the proof of Theorem <ref>,
  
      _̊∥ R_∥=[  R_1    ⋯  R_t P̅_1    ⋯ P̅_t ]
      (_t^∞)^-1[ _t^∞; _t^∞ ]
  
  and _t^∞ is the limit of _t defined in
  (<ref>). Also,
  
      _̊⊥ R_⊥∼(0, [U_t+1^2]
      -([   _t; _t_t ]^⊤[     _t  _t _t; _t_t^⊤     _t ]^-1[   _t; _t_t ])^∞)
  
  where this limit R_⊥ is independent of
  (R_1,…,R_t,P̅_1,…,P̅_t,Λ_m). So
  
      (_̊1,…,_̊t+1,_1,…,_t,_m)
       (R_1,…,R_t+1,P̅_1,…,P̅_t,Λ_m), 
           R_t+1=R_∥+R_⊥.
  
  We have
  
      [R_⊥]>0
  
  because this is the residual variance of projecting
  U_t+1 onto the span of (U_1,…,U_t,Y_1,…,Y_t), which is positive
  by Assumption <ref>(f).
  
  Let us now introduce the block notation
  
      _̋t+1^(2k,∞)=[       _̋t^(2k,∞)        _t^(2k,∞);    (_t^(2k,∞))^⊤ h_t+1,t+1^(2k,∞) ],     _t^(2k+1,∞)=[  _t^(2k+1,∞) ı_t^(2k+1,∞) ].
  
  To conclude the proof of t+1^(b), it remains to show that
  
      [(R_1,…,R_t)^⊤Λ_m^2kR_t+1]
         =_t^(2k,∞)
      [(P̅_1,…,P̅_t)^⊤Λ_m^2kR_t+1]
         =ı_t^(2k+1,∞)
      [Λ_m^2kR_t+1^2]   =h_t+1,t+1^(2k,∞).
  
  
  For (<ref>), observe that by t^(b) and t^(e), we have
  
      [(R_1,…,R_t)^⊤Λ_m^2k (R_1,…,R_t)]
      =_̋t^(2k,∞)
  
  and
  
      [(R_1,…,R_t)^⊤Λ_m^2k (P̅_1,…,P̅_t)]
         =lim_m,n →∞1/m_t^⊤ (^⊤)^2k_t
         =γ^-1lim_m,n →∞1/n_t^⊤ (^⊤)^2k_t
         =γ^-1[(R̅_1,…,R̅_t)^⊤Λ_n^2k (P_1,…,P_t)]
         =γ^-1_t^(2k+1,∞)=(_t^(2k+1,∞))^⊤,
  
  the last equality applying (<ref>).
  Then applying (<ref>),
  (<ref>), and the independence of R_⊥ from
  (R_1,…,R_t,P̅_1,…,P̅_t,Λ_m), we have
  
      [(R_1,…,R_t)^⊤Λ_m^2kR_t+1]
      =([      _̋t^2k (_t^2k+1)^⊤ ]_t^-1[ _t; _t ])^∞.
  
  Applying the first row of the identity (<ref>),
  and the definitions of _t+1^(j), _t+1^(j),
  and _̋t+1^(2k) from (<ref>), (<ref>),
  and (<ref>),
  
      [(R_1,…,R_t)^⊤Λ_m^2kR_t+1]
         =(∑_j=0^∞ c_2k,j (_t_t)^j _t
      +∑_j=0^∞ c_2k,j+1 (_t^(j))^⊤_t)^∞
         =(∑_j=0^∞ c_2k,j (_t+1_t+1)^j _t+1
      +∑_j=0^∞ c_2k,j+1 (_t+1^(j))^⊤_t+1^⊤)_1:t, t+1^∞
         =(∑_j=0^∞ c_2k,j_t+1^(j))_1:t, t+1^∞
      =_t^(2k,∞).
  
  
  For (<ref>), observe that also by t^(e),
  
      [(P̅_1,…,P̅_t)^⊤Λ_m^2k (P̅_1,…,P̅_t)]
         =lim_m,n →∞1/m_t^⊤^⊤
      (^⊤)^2k_t
         =γ^-1lim_m,n →∞1/n_t^⊤(^⊤)^2k+2_t
         =γ^-1·[(P_1,…,P_t)^⊤Λ_n^2k+2(P_1,…,P_t)]
      =γ^-1·Ł_t^(2k+2,∞).
  
  So
  
      [(P̅_1,…,P̅_t)^⊤Λ_m^2kR_t+1]
      =([       _t^(2k+1) γ^-1·Ł_t^(2k+2) ]_t^-1[ _t; _t ])^∞.
  
  Applying the second row of the identity (<ref>),
  
      [(P̅_1,…,P̅_t)^⊤Λ_m^2kR_t+1]
         =(∑_j=0^∞ c_2k+1,j (_t_t)^j _t _t
      +∑_j=0^∞ c_2k+1,j_t^(j)_t)^∞
         =(∑_j=0^∞ c_2k+1,j (_t+1_t+1)^j
      _t+1_t+1+∑_j=0^∞ c_2k+1,j_t+1^(j)_t+1^⊤)_1:t, t+1^∞
         =(∑_j=0^∞ c_2k+1,j_t+1^(j))_1:t, t+1^∞
      =ı_t^(2k+1,∞).
  
  
  For (<ref>), applying again (<ref>) and a 
  computation similar to the proof of Theorem <ref>, we have
  
      [Λ_m^2kR_t+1^2]
      =([ _t; _t ]^⊤
      (_t^-1)^⊤[        _̋t^(2k)   (_t^(2k+1))^⊤;       _t^(2k+1) γ^-1·Ł_t^(2k+2) ]_t^-1[ _t; _t ])^∞
      +[Λ_m^2kR_⊥^2]
  
  where, by independence of R_⊥ and Λ_m,
  
      [Λ_m^2kR_⊥^2]
      =c_2k,0^∞([U_t+1^2]-
      ([ _t; _t ]^⊤
      (_t^-1)^⊤[      _̋t^(0)   (_t^(1))^⊤;       _t^(1) γ^-1·Ł_t^(2) ]_t^-1[ _t; _t ])^∞).
  
  Combining these and applying the identity (<ref>),
  
      [Λ_m^2kR_t+1^2]
         =c_2k,0^∞[U_t+1^2]+(∑_j=0^∞ c_2k,j+1(_t^⊤_t^⊤ (_t^⊤_t^⊤)^j _t
      +_t^⊤ (_t_t)^j_t_t
      +_t^⊤_t^(j)_t))^∞
         =(c_2k,0_t+1
      +∑_j=1^∞ c_2k,j(_t+1(_t+1^⊤_t+1^⊤)^j
      +(_t+1_t+1)^j _t+1
      +_t+1_t+1^(j-1)_t+1^⊤))_t+1,t+1^∞
         =(∑_j=0^∞ c_2k,j_t+1^(j))_t+1,t+1^∞
      =h_t+1,t+1^(2k,∞).
  
  This completes the proof of t+1^(b).
  
  Let us make here the following additional observation: This also shows
  
      (_1,…,_t,^⊤_̊1,…,^⊤_̊t+1,
      _n)  (P_1,…,P_t,R̅_1,…,R̅_t+1,Λ_n)
  
  for a certain limit R̅_t+1, which is part of the claim in t+1^(e).
  This is because from the decomposition
  _̊t+1=_̊∥+_̊⊥, we have
  ^⊤_̊t+1=^⊤_̊∥+^⊤_̊⊥.
  From the form of _̊∥ and claim t^(e), we have
  
      ^⊤_̊∥R̅_∥=[      R̅_1         ⋯      R̅_t Λ_n^2 P_1         ⋯ Λ_n^2 P_t ]
      (_t^∞)^-1[ _t^∞; _t^∞ ].
  
  For ^⊤_̊⊥, let us define _⊥∈^n
  to be the first n entries of _̊⊥ if n ≤ m, or
  _̊⊥ extended by an additional n-m
  i.i.d. (0,[R_⊥]) variables if n>m.
  By Proposition <ref>(b), _⊥Ř_⊥
  in both cases, where this limit Ř_⊥ has the same normal law as
  R_⊥ above,
  and is independent of (P_1,…,P_t,R̅_1,…,R̅_t,Λ_n). 
  Since ^⊤_̊⊥ is the entrywise product of _n
  with _⊥, this shows that (<ref>) holds where
  
      R̅_t+1=R̅_∥+Λ_n Ř_⊥.
  
  Furthermore,
  
      [(P_1,…,P_t)^⊤Λ_n^2k(R̅_1,…,R̅_t+1)]
         =lim_m,n →∞ n^-1_t^⊤ (^⊤)^2k^⊤_t+1
         =γ·[(P̅_1,…,P̅_t)^⊤Λ_m^2k(R_1,…,R_t+1)]
         =γ·_t^(2k+1,∞),
      [(R̅_1,…,R̅_t+1)^⊤Λ_n^2k
      (R̅_1,…,R̅_t+1)]
         =lim_m,n →∞ n^-1_t+1^⊤(^⊤)^k ^⊤_t+1
         =γ·[(R_1,…,R_t+1)^⊤Λ_m^2k+2(R_1,…,R_t+1)]
         =γ·_̋t+1^(2k+2,∞).
  
  
  Step 3: Analysis of _t+1. We now show t+1^(c).
  Conditioning on the iterates up to _̊t+1, the law of  is conditioned on
  
      [   _t ^⊤_t ][     -_t;         ]
      =[ _t _t ]
  
  Set
  
      _t=n^-1[ _t^⊤_t _t^⊤_t; _t^⊤_t _t^⊤_t ],
  
  and note that by t^(c), t^(g), and Proposition <ref>,
  
      _t →_t^∞
      =[          _t^∞      _t^∞_t^∞; _t^∞ (_t^∞)^⊤          _t^∞ ]
  
  where _t^∞ is invertible. Thus, for all large n,
  the conditional law of  is
  
      [   _t ^⊤_t ][     -_t;         ]_t^-1· n^-1[ _t^⊤; _t^⊤ ]
      +_(_t,^⊤_t)^⊥_(_t,_t)^⊥^⊤
  
  where ∈^(n-2t) × (n-2t) is an independent
  Haar-orthogonal matrix. So we may replace the update
  for _t+1=^⊤^⊤_̊t+1 by
  
      _t+1   =_∥+_⊥
      _∥   =[ _t _t ]_t^-1[            ; -_t^⊤       ]· n^-1[ _t^⊤; _t^⊤ ]^⊤_̊t+1,
      _⊥   =_(_t,_t)^⊥^⊤_(_t,^⊤_t)^⊥^⊤^⊤_̊t+1
  
  Applying t+1^(b) shown above, and recalling the block notation
  (<ref>), observe that
  
      n^-1_t^⊤^⊤_̊t+1   →γ·[(P̅_1,…,P̅_t)^⊤ R_t+1]
      =γ·ı_t^(1,∞)
      
      n^-1_t^⊤^⊤_̊t+1   →γ·[(R_1,…,R_t)^⊤Λ_m^2 R_t+1]
      =γ·_t^(2,∞).
  
  Then by a computation analogous to the proof of Theorem <ref>,
  
      _t^-1[            ; -_t^⊤       ]· n^-1[ _t^⊤; _t^⊤ ]^⊤_̊t+1→([           0;     0 _t^-1 ] (_t^-1)^⊤[ γ·ı_t^(1);  γ·_t^(2) ])^∞
  
  where _t is as defined in (<ref>). Applying the second row of
  the identity (<ref>) with t+1 and with k=0,
  and recalling c̅_1,j=κ̅_2(j+1)=γ·κ_2(j+1), we have
  
      [ (_t+1^(1))^⊤;  γ·_̋t+1^(2) ]
      =_t+1^⊤[ ∑_j=0^∞γ·κ_2(j+1)_t+1^⊤ (_t+1^⊤_t+1^⊤)^j;                ∑_j=0^∞γ·κ_2(j+1)_t+1^(j) ]
      =_t+1^⊤[ _t+1; _t+1 ].
  
  Writing the block forms
  
      _t+1=[  _t _̱t;   0   0 ],
          _t+1=[        _t        _t;      _t^⊤ ω_t+1,t+1 ],
  
  and applying (_t+1^(1))^⊤=γ·_t+1^(1),
  this yields
  
      γ·[ ı_t^(1);  _t^(2) ]
      =(_t+1^⊤[ _t+1; _t+1 ])_(1:t)∪ (t+2:2t+1), t+1
      =_t^⊤[ _̱t;  _t ],
  
  where the second equality follows because _t+1 is 0 in its lower-right
  entry while _t+1^⊤ is 0 in rows 1:t and t+2:2t+1 of its last
  column. Inverting _t^⊤ and applying this above,
  
      _∥ S_∥=[ V_1   ⋯ V_t ]_̱t^∞+[ Z_1   ⋯ Z_t ]
      (_t^∞)^-1_t^∞.
  
  Similar to the proof of Theorem <ref>, we have also
  
      _⊥ S_⊥∼(0, (γ· h_t+1,t+1^(2)-
      [ γ·ı_t^(1);  γ·_t^(2) ]^⊤[    Ł_t^(0) (_t^(1))^⊤;     _t^(1)  γ·_̋t^(2) ]^-1[ γ·ı_t^(1);  γ·_t^(2) ])^∞)
  
  where S_⊥ is independent of (V_1,…,V_t,Z_1,…,Z_t). So
  
      _t+1 S_t+1=[ V_1   ⋯ V_t ]_̱t^∞+[ Z_1   ⋯ Z_t ]
      (_t^∞)^-1_t^∞+S_⊥.
  
  Since _t+1=_t+1-_t_̱t, this yields
  
      (_̌1,…,_̌t+1,_1,…,_t+1,)
       (V_1,…,V_t+1,Z_1,…,Z_t+1,F)
  
  where V_t+1=v_t+1(Z_1,…,Z_t+1,F) and
  
      Z_t+1=[ Z_1   ⋯ Z_t ]
      (_t^∞)^-1_t^∞+S_⊥.
  
  
  Thus (Z_1,…,Z_t,Z_t+1) has a multivariate normal distribution.
  To compute the covariance, observe that
  since S_⊥ is independent of (Z_1,…,Z_t), we have
  
      [(Z_1,…,Z_t)^⊤
      Z_t+1]=_t^∞(_t^∞)^-1_t^∞=_t^∞.
  
  For [Z_t+1^2], squaring both sides of (<ref>),
  applying
  
      [S_t+1^2]=lim_m,n →∞ n^-1_t+1^2
      =lim_m,n →∞ n^-1^⊤_̊t+1^2
      =γ· h_t+1,t+1^(2,∞),
  
  and rearranging yields
  
      [Z_t+1^2]=((γ·_̋t+1^(2)
      -_t+1^⊤_t+1_t+1
      -_t+1^⊤_t+1_t+1
      -_t+1_t+1^⊤_t+1)^∞)_t+1,t+1.
  
  Applying the identity (<ref>), this gives
  [Z_t+1^2]=ω_t+1,t+1^∞, and this concludes the proof of
  t+1^(c).
  
  Let us make here the following additional observation: We have
  
      [S_⊥]>0
  
  above. This is because by (<ref>) and
  (<ref>) for k=0,
  the quantity [S_⊥] above may be seen to be the residual variance of
  projecting R̅_t+1 onto the linear span of
  (P_1,…,P_t,R̅_1,…,R̅_t). If this residual variance were 0,
  then for some constants α_1,…,α_t,β_1,…,β_t
  we would have R̅_t+1=α_1 P_1+…+α_t P_t
  +β_1 R̅_1+…+β_t R̅_t with probability 1, so that
  
      0   =[Λ_n^2 · (R̅_t+1-α_1 P_1-…-α_t P_t
      -β_1 R̅_1-…-β_t R̅_t)^2]
         =lim_m,n →∞
      n^-1^⊤_̊t+1-α_1 _1-…-α_t
      _t-β_1 ^⊤_̊1-…-β_t^⊤_̊t^2
         =γ·[(Λ_m^2 R_t+1-α_1P̅_1-…
      -α_tP̅_t-β_1Λ_m^2 R_1-…-β_t Λ_m^2 R_t)^2].
  
  Thus also
  
      Λ_m^2 R_t+1=α_1P̅_1+…
      +α_tP̅_t+β_1Λ_m^2 R_1+…+β_t Λ_m^2 R_t
  
  with probability 1. However, recall the decomposition
  R_t+1=R_∥+R_⊥ where
  R_⊥ is independent of
  (R_1,…,R_t,P̅_1,…,P̅_t,Λ_m). Thus we have
  
      Λ_m^2 R_⊥=f(R_1,…,R_t,P̅_1,…,P̅_t,Λ_m)
  
  where the quantity on the right does not depend on R_⊥. Since Λ_m
  is not identically 0 by the condition [Λ]>0 in
  Assumption <ref>(f), this implies that R_⊥ must be constant
  almost surely, contradicting (<ref>) already shown. Thus,
  (<ref>) holds.
  
  
  Step 4: Analysis of _t+1. Note that t^(f), t+1^(c), and
  the given conditions for the functions ∂_s' v_s
  imply the existence of all limits in t+1^(d). Let us now show
  t+1^(e). The joint convergence with ^⊤_̊t+1 has been
  established already in (<ref>), so we proceed to analyze
  _t+1.
  
  For this, let
  
      _t=[  _t _̱t ]∈^t ×
      (t+1),
          _t=[ _t    ]∈^t
      × (t+1)
  
  be the first t rows of _t+1 and _t+1, and let
  
      _t=[ _t;  0 ]∈^(t+1) × t,
          _t=[     _t; (_t)^⊤ ]∈^(t+1) × t
  
  be the first t columns of _t+1 and _t+1. Conditional on
  the iterates up to _̌t+1, the law of  is now conditioned on
  
      [     _t ^⊤_t+1 ][     -_t;         ]
      =[   _t _t+1 ].
  
  Let us denote
  
      _t=n^-1[     _t^⊤_t   _t^⊤_t+1;   _t+1^⊤_t _t+1^⊤_t+1 ],
  
  noting that by t+1^(c) and Proposition <ref>,
  
      _t →_t^∞=[            _t^∞      _t^∞_t+1^∞; _t+1^∞
      (_t^∞)^⊤          _t+1^∞ ].
  
  The upper-left 2t × 2t submatrix of _t^∞ is
  _t^∞, which is
  invertible by t^(g). The Schur-complement of its lower-right entry is
  the residual variance of projecting Z_t+1 onto the span of
  (V_1,…,V_t,Z_1,…,Z_t). Since _t+1=_t+1-_t_̱t, this is
  the same as the residual variance of projecting S_t+1 onto the 
  span of (V_1,…,V_t,Z_1,…,Z_t), which is exactly [S_⊥] by
  the convergence (<ref>) and the fact that S_⊥ is a
  mean-zero variable independent of (V_1,…,V_t,Z_1,…,Z_t). This was
  shown to be non-zero in (<ref>), so _t^∞
  is invertible. Thus for all large n, the conditional law of  is
  
      [     _t ^⊤_t+1 ][     -_t;         ]_t^-1· n^-1[   _t^⊤; _t+1^⊤ ]
      +_(_t,^⊤_t+1)^⊥_(_t,_t+1)^⊥^⊤
  
  So we may replace the update for _t+1=_̌t+1 by
  
      _t+1   =_∥+_⊥
      _∥   =[     _t ^⊤_t+1 ][     -_t;         ]_t^-1· n^-1[   _t^⊤; _t+1^⊤ ]_̌t+1
      _⊥   =_(_t,^⊤_t+1)^⊥_(_t,_t+1)^⊥^⊤_̌t+1
  
  
  For _∥, define
  
      _t^∞=[ [V_1V_t+1];          ⋮; [V_tV_t+1] ]∈^t,     _t^∞=[   [∂_1 v_t+1(Z_1,…,Z_t+1,F)];                            ⋮; [∂_t+1 v_t+1(Z_1,…,Z_t+1,F)] ]∈^t+1
  
  and observe that
  
      n^-1_t^⊤_̌t+1→_t^∞,     
      n^-1_t+1^⊤_̌t+1→_t+1^∞_t^∞.
  
  Then by a computation similar to the proof of t+1^(b) above,
  
      _∥[    P_1      …    P_t   R̅_1      … R̅_t+1 ]
      (_t^∞)^-1[ _t^∞; _t^∞ ]
  
  where
  
      _t=[           _t _t
      _t+_t_t+1;         _t^⊤       _t^⊤_t ].
  
  Also,
  
      _⊥ P_⊥∼(0, [V_t+1^2]
      -([     _t; _t+1_t ]^⊤[       _t   _t_t+1; _t+1_t^⊤     _t+1 ]^-1[     _t; _t+1_t ])^∞)
  
  where P_⊥ is independent of
  (P_1,…,P_t,R̅_1,…,R̅_t+1,Λ_n). So
  
      (_1,…,_t+1,^⊤_̊1,…,^⊤_̊t+1,_n)  (P_1,…,P_t+1,R̅_1,…,R̅_t+1,
      Λ_n),      P_t+1=P_∥+P_⊥.
  
  We have
  
      [P_⊥]>0
  
  because this is the residual variance of projecting V_t+1 onto the span of
  (V_1,…,V_t,Z_1,…,Z_t+1), which is positive by Assumption
  <ref>(f).
  
  Let us introduce the block notation
  
      Ł_t+1^(2k)=[       Ł_t^(2k)       ł_t^(2k);   (ł_t^(2k))^⊤ l_t+1,t+1^(2k) ],     _t+1^(2k+1)=[ γ· (_t^(2k+1))^⊤      j̃_t^(2k+1) ]
  
  where _t^(2k+1)∈^t × (t+1) forms the first t
  rows of _t+1^(2k+1) as previously defined, and thus
  γ· (_t^(2k+1))^⊤ forms the first t columns
  of _t+1^(2k+1) by the identity (<ref>). To conclude the proof of
  t+1^(e), it remains to show that
  
      [(R̅_1,…,R̅_t+1)^⊤Λ_n^2kP_t+1]
         =j̃_t^(2k+1,∞)
      [(P_1,…,P_t)^⊤Λ_n^2k P_t+1]   =ł_t^(2k,∞)
      [Λ_n^2kP_t+1^2]   =l_t+1,t+1^(2k,∞).
  
  
  The arguments are similar to those for t+1^(b): For
  (<ref>), by the convergence (<ref>), the form
  of P_∥, and the
  identities (<ref>–<ref>), we have
  
      [(R̅_1,…,R̅_t+1)^⊤Λ_n^2kP_t+1]
      =([ γ· 
      (_t^(2k+1))^⊤    γ·_̋t+1^(2k+2) ]_t^-1[ _t; _t ])^∞.
  
  Applying the second row of the identity (<ref>),
  
      [    _t+1^(2k+1) γ·_̋t+1^(2k+2) ]
      =[ ∑_j=0^∞c̅_2k+1,j(_t+1_t+1)^j_t+1         ∑_j=0^∞c̅_2k+1,j_t+1^(j) ]_t+1.
  
  Note that _t defined in (<ref>) is the submatrix of
  _t+1 with row and column t+1 removed. 
  Furthermore, column t+1 of _t+1 is 0. Thus, removing column t+1
  from both sides of this identity yields
  
      [ γ· (_t^(2k+1))^⊤   γ·_̋t+1^(2k+2) ]
      =[ ∑_j=0^∞c̅_2k+1,j (_t+1_t+1)^j
      _t         ∑_j=0^∞c̅_2k+1,j_t+1^(j) ]_t.
  
  Taking the limit m,n →∞, 
  inverting _t^∞, and applying this above,
  
      [(R̅_1,…,R̅_t+1)^⊤Λ_n^2kP_t+1]
         =(∑_j=0^∞c̅_2k+1,j(
      (_t+1_t+1)^j_t_t
      +_t+1^(j)_t))^∞
         =(∑_j=0^∞c̅_2k+1,j (_t+1^(j))^⊤)_1:t+1, t+1^∞=j̃_t^(2k+1,∞).
  
  Here, the last two equalities apply again the fact that the last column of
  _t+1 is 0, and the definitions of _t+1^(j) and
  _t+1^(2k+1) in (<ref>) and (<ref>).
  
  For (<ref>), we have
  
      [(P_1,…,P_t)^⊤Λ_n^2k P_t+1]
      =([    Ł_t^(2k) γ·_t^(2k+1) ]_t^-1[ _t; _t ])^∞.
  
  Applying the first row of the identity (<ref>) with t+1,
  and a similar argument of removing the t+1^th rows and columns from
  both sides, we obtain
  
      [    Ł_t^(2k) γ·_t^(2k+1) ]=[ ∑_j=0^∞c̅_2k,j
      (_t_t)^j  ∑_j=0^∞c̅_2k,j+1_t^(j) ]_t
  
  where _t^(j) are the first t rows of _t+1^(j). Then
  
      [(P_1,…,P_t)^⊤Λ_n^2k P_t+1]
      =(∑_j=0^∞ (c̅_2k,j
      (_t_t)^j_t+∑_j=0^∞c̅_2k,j+1_t^(j)_t))^∞
      =ł_t+1^(2k,∞).
  
  
  For (<ref>), squaring both sides of (<ref>)
  and recalling [Λ_n^2k]=m̅_2k^∞=c̅_2k,0^∞
  from Lemma <ref>, we have
  
      [Λ_n^2kP_t+1^2]
         =([ _t; _t ]^⊤
      (_t^-1)^⊤[         Ł_t^(2k)      γ·_t^(2k+1); γ· (_t^(2k+1))^⊤     γ·_̋t^(2k+2) ]_t^-1[ _t; _t ])^∞
      +[Λ_n^2kP_⊥^2],
      [Λ_n^2kP_⊥^2]
         =c̅_2k,0^∞([V_t+1^2]
      -[ _t; _t ]^⊤
      (_t^-1)^⊤[       Ł_t^(0)      γ·_t^(1); γ· (_t^(1))^⊤     γ·_̋t^(2) ]_t^-1[ _t; _t ])^∞.
  
  Applying (<ref>) with t+1, and removing the t+1^th rows
  and columns from both sides, we have
  
      [         Ł_t^(2k)      γ·_t^(2k+1); γ· (_t^(2k+1))^⊤   γ·_̋t+1^(2k+2) ]-c̅_2k,0[       Ł_t^(0)      γ·_t^(1); γ· (_t^(1))^⊤   γ·_̋t+1^(2) ]
         =_t^⊤[                                     0 ∑_j=0^∞c̅_2k,j+1_t^⊤
      (_t+1^⊤_t+1^⊤)^j;        ∑_j=0^∞c̅_2k,j+1(_t+1_t+1)^j_t              ∑_j=0^∞c̅_2k,j+1_t+1^(j) ]_t.
  
  Then combining the above,
  
      [Λ_n^2kP_t+1^2]
         =(c̅_2k,0[V_t+1^2]
      +∑_j=0^∞c̅_2k,j+1(_t^⊤_t^⊤
      (_t+1^⊤_t+1^⊤)^j _t
      +_t^⊤ (_t+1_t+1)^j_t_t
      +_t^⊤_t+1^(j)_t))^∞
         =(∑_j=0^∞c̅_2k,j_t+1^(j))_t+1,t+1^∞
      =l_t+1,t+1^(2k,∞).
  
  This concludes the proof of t+1^(e).
  
  
  Step 5: Analysis of _t+1.
  Finally, let us show t+1^(f) and t+1^(g). Conditional on
  the iterates up to _t+1, the law of Ø is now conditioned on
  
      [ _t+1   _t ][     -_t;         ]
      =Ø[ _t+1   _t ].
  
  Let us set
  
      _t=m^-1[ _t+1^⊤_t+1   _t+1^⊤_t;   _t^⊤_t+1     _t^⊤_t ],
  
  noting that by t+1^(c),
  
      _t →_t^∞
      =[    _t+1   _t _t; _t _t^⊤      _t ].
  
  This limit is invertible because its submatrix removing row and column
  t+1 is _t^∞ which is
  invertible by t^(g), and the Schur complement of the (t+1,t+1)
  entry is exactly [R_⊥] from (<ref>), which we have shown
  is non-zero in (<ref>). Then for all large n,
  the conditional law of Ø is
  
      [ _t+1   _t ][     -_t;         ]_t^-1· m^-1[ _t+1^⊤;   _t^⊤ ]
      +_(_t+1,_t)^⊥Ø̃_(_t+1,_t)^⊥^⊤
  
  So we may replace the update for _t+1=Ø^⊤_t+1 by
  
      _t+1   =_∥+_⊥
      _∥   =[ _t+1   _t ]_t^-1[            ; -_t^⊤       ]· m^-1[ _t+1^⊤; _t^⊤^⊤ ]_t+1
      _⊥   =_(_t+1,_t)^⊥Ø̃^⊤_(_t+1,_t)^⊥^⊤_t+1
  
  Setting
  
      _t=[        _t+1 _t+1_t+_t_t;        _t^⊤      _t^⊤_t ]
  
  and recalling j̃_1^(1) from (<ref>),
  by a computation similar to the proof of t+1^(c) above, we have
  
      _t^-1[            ; -_t^⊤       ]· m^-1[ _t+1^⊤; _t^⊤^⊤ ]_t+1→(γ^-1·[           0;     0 _t^-1 ]
      (_t^-1)^⊤[ j̃_t^(1);  ł_t^(2) ])^∞.
  
  Applying the second row of (<ref>) with t+1 and with k=0,
  and recalling
  _t+1^(2k+1)=γ^-1· (_t+1^(2k+1))^⊤ and
  c_1,j=κ_2(j+1), we get
  
      γ^-1·[  _t+1^(1); Ł_t+1^(2) ]
      =_t+1^⊤[ ∑_j=0^∞κ_2(j+1)_t+1^⊤ (_t+1^⊤_t+1^⊤)^j;                ∑_j=0^∞κ_2(j+1)_t+1^(j) ]
      =_t+1^⊤[ _t+1; _t+1 ].
  
  Hence, writing
  
      _t+1=[ _t _t ],
          _t+1=[        _t        _t;      _t^⊤ σ_t+1,t+1 ],
  
  noting that _t is the matrix _t+1 with the
  last row and column removed, and that _t+1^⊤ is
  0 in entries 1:2t+1 of its last column, this yields
  
      γ^-1·[ j̃_t^(1);  ł_t^(2) ]=_t^⊤[ _t; _t ].
  
  Taking the limit m,n →∞, inverting
  (_t^∞)^⊤, and applying this above,
  
      _∥[   U_1     ⋯ U_t+1 ]_t^∞+[ Y_1   ⋯ Y_t ]
      (_t^∞)^∞_t^∞.
  
  We have also
  
      _⊥ Q_⊥∼(0, (γ^-1· l_t+1,t+1^(2)-[ γ^-1·j̃_t^(1);  γ^-1·ł_t^(2) ]^⊤[   _̋t+1^(0)  (_t^(1))^⊤;      _t^(1) γ^-1Ł_t^(2) ]^-1[ γ^-1·j̃_t^(1);  γ^-1·ł_t^(2) ])^∞)
  
  where _⊥ is independent of
  (U_1,…,U_t+1,Y_1,…,Y_t). Then
  
      _t+1 Q_t+1=[   U_1     ⋯ U_t+1 ]_t^∞+[ Y_1   ⋯ Y_t ]
      (_t^∞)^-1_t^∞+Q_⊥.
  
  Recalling _t+1=_t+1-_t+1_t, this yields
  
      (_̆1,…,_̆t+2,_1,…,_t+1,) 
      (U_1,…,U_t+2,Y_1,…,Y_t+1,E)
  
  where U_t+2=u_t+2(Y_1,…,Y_t+1,E) and
  
      Y_t+1=[ Y_1   ⋯ Y_t ]
      (_t^∞)^-1_t^∞+Q_⊥.
  
  So (Y_1,…,Y_t,Y_t+1) has a multivariate normal limit.
  To compute the covariance, note that
  
      [(Y_1,…,Y_t)^⊤ Y_t+1]=_t^∞(_t^∞)^-1_t^∞=_t^∞.
  
  Squaring both sides of (<ref>), applying [Q_t+1^2]=lim_m,n
  →∞ m^-1_t+1^2=γ^-1
  l_t+1,t+1^(2,∞),
  and rearranging,
  
      [Y_t+1^2]=(γ^-1·Ł_t+1^(2)-_t+1^⊤_t+1_t+1-_t+1^⊤_t+1_t+1
      -_t+1_t+1^⊤_t+1)^∞_t+1,t+1.
  
  Applying (<ref>), this is σ_t+1,t+1^∞.
  This concludes the proof of t+1^(f).
  
  Finally, for the invertibility claim of t+1^(g), let us first observe that
  
      [Q_⊥]>0
  
  above. This is because [Q_⊥] is the residual
  variance of projecting P̅_t+1 onto the span of
  (R_1,…,R_t+1,P̅_1,…,P̅_t). If this were 0, then for some
  constants α_1,…,α_t+1,β_1,…,β_t, we would have
  
      0   =[Λ_m^2 · (P̅_t+1-α_1 R_1-…-α_t+1
      R_t+1-β_1P̅_1-…-β_tP̅_t)^2]
         =lim_m,n →∞ m^-1^⊤_t+1
      -α_1^⊤_̊1-…-α_t+1^⊤_̊t+1
      -β_1^⊤_1-…-β_t^⊤_t^2
         =γ^-1·[(Λ_n^2 P_t+1-α_1
      R̅_1-…-α_t+1R̅_t+1-β_1Λ_n^2 P_1-…
      -β_nΛ_n^2P_t)^2].
  
  So
  
      Λ_n^2P_⊥=f(P_1,…,P_t,R̅_1,…,R̅_t+1,Λ_n)
  
  for some quantity on the right not depending on P_⊥. This contradicts the
  independence of P_⊥ from
  P_1,…,P_t,R̅_1,…,R̅_t+1,Λ_n, the
  assumption [Λ_n]>0, and the condition [P_⊥]>0 already
  shown in (<ref>). So (<ref>) holds.
  
  To show the invertibility of
  
      [            _t+1^∞      _t+1^∞_t+1^∞; _t+1^∞ (_t+1^∞)^⊤            _t+1^∞ ],
  
  note that its upper-left (2t+1) × (2t+1) submatrix
  is _t^∞, which we have shown is invertible.
  The Schur-complement of its lower-right entry is the residual variance of
  projecting Y_t+1 onto (U_1,…,U_t,Y_1,…,Y_t+1). As
  _t+1=_t+1-_t+1_t, this is equivalently the residual
  variance of projecting Q_t+1 onto (U_1,…,U_t,Y_1,…,Y_t+1),
  which is exactly [Q_⊥] by (<ref>) and the fact that
  Q_⊥ is a mean-zero variable independent of
  (U_1,…,U_t,Y_1,…,Y_t+1). Since [Q_⊥]>0, this shows that
  (<ref>) is invertible.
  
  To show the invertibility of
  
      [            _t+1^∞      _t+1^∞_t+1^∞; _t+1^∞ (_t+1^∞)^⊤            _t+1^∞ ],
  
  note that its submatrix removing row and column t+1 is 
  _t^∞, which we have also shown is invertible. The
  Schur-complement of the (t+1,t+1) entry is the residual variance of projecting
  V_t+1 onto the span of (V_1,…,V_t,Z_1,…,Z_t+1), which is
  non-zero by Assumption <ref>(f). Thus (<ref>) is
  invertible. This shows t+1^(g), and concludes the induction and the proof.
  
  
  
  
  § ANALYSIS OF AMP FOR PCA
  
  
  In this appendix, we prove Theorems <ref> and <ref>.
  We also complete the verification of Eq. (<ref>), showing
  that the rectangular AMP algorithm achieves lower matrix 
  mean-squared error than the sample PCs.
  
  
  
   §.§ State evolution for PCA
  
  
  We prove Theorems <ref>(a) and <ref>(a), using the general
  results of Corollaries <ref> and <ref>.
  
  
  We may write the AMP iterations (<ref>–<ref>) as
  
      _t=_̆* · (α/n)_̆*^⊤_̆t+_̆t
      -b_t1_̆1-…-b_tt_̆t,     _̆t+1=u_t+1(_t).
  
  Approximating (α/n)_̆*^⊤_̆t ≈α·[U_*U_t]=μ_t^∞, we consider the auxiliary AMP sequence
  initialized at _1=_̆1 and defined by
  
      _t=_t-b̃_t1_1-…
      -b̃_tt_t,     _t+1=ũ_t+1(_t,_̆*)
      ≡ u_t+1(_t+μ_t^∞_̆*).
  
  Here, the debiasing coefficients b̃_t1,…,b̃_tt are
  the values of the last column of _t, defined by (<ref>)
  and (<ref>) with the iterates _t and the
  free cumulants κ̃_k of . The partial derivatives that define
  (<ref>) are given by ∂_s
  ũ_t+1(·)=ũ_t+1'(·) if s=t and
  ∂_s ũ_t+1(·)=0 otherwise, where ũ_t+1'(·)
  denotes the derivative in its first argument z̃_t.
  
  This auxiliary AMP sequence is of the general form
  (<ref>–<ref>) with side information =_̆*.
  By the given differentiability and Lipschitz assumption for u_t+1(·),
  the conditions of Corollary <ref> are satisfied for
  ũ_t+1(·), so we have for each fixed T ≥ 1 that
  
      (_1,…,_T+1,_1,…,_T,_̆*)
      
      (Ũ_1,…,Ũ_T+1,Z̃_1,…,Z̃_T,U_*).
  
  Here, (Z̃_1,…,Z̃_T) ∼(0,_T^∞)
  where _T^∞ is defined by (<ref>) and
  (<ref>) for this auxiliary AMP sequence, Ũ_1=U_1,
  Ũ_s+1=ũ_s+1(Z̃_s,U_*) for s ≥ 1,
  and (Z̃_1,…,Z̃_T) is independent of (Ũ_1,U_*). 
  Defining
  
      _t=_t+μ_t^∞_̆*,
          F̃_t=Z̃_t+μ_t^∞ U_*,
  
  this implies
  
      (_1,…,_T+1,_1,…,_T,_̆*)
      
      (Ũ_1,…,Ũ_T+1,F̃_1,…,F̃_T,U_*).
  
  Since each derivative ∂_s ũ_t+1 is non-zero only for s=t,
  the covariance matrix _T^∞ has the entries
  
      σ̃_st^∞=∑_j=0^s-1∑_k=0^t-1κ̃_j+k+2^∞(∏_i=s-j+1^s
      [ũ_i'(Z̃_i-1,U_*)])
      (∏_i=t-k+1^t [ũ_i'(Z̃_i-1,U_*)])
      [Ũ_s-jŨ_t-k]
  
  where the summand for (j,k) corresponds to
  _T^j_T(_T^k)^⊤ in the definitions (<ref>) and
  (<ref>).
  
  We conclude the proof by showing that the joint law of this limit in
  (<ref>) coincides with the limit described in Theorem
  <ref>(a), and that (_̆1,…,_̆T+1,_1,…,_T,_̆*) for
  the original AMP algorithm converges to the same limit.
  Observe first that the n-1 smallest eigenvalues of  are interlaced with
  the n eigenvalues of . Letting m_k be as defined in
  (<ref>), and denoting the moments of the empirical spectral
  distribution of  by
  
      m̃_k=1/n∑_i=1^n λ_i^k,
  
  this interlacing and the
  condition ≤ C_0 imply |m_k-m̃_k| → 0 and m_k,m̃_k
  → m_k^∞=[Λ^k] for each fixed k ≥ 1 as n →∞.
  Hence also for each fixed k ≥ 1,
  
      |κ_k-κ̃_k| → 0    and   κ_k,κ̃_k →κ_k^∞
  
  where {κ_k^∞}_k ≥ 1 are the free cumulants of Λ.
  
  We now check inductively that, almost surely for each fixed
  T=0,1,2,… as n →∞,
  
      n^-1_̆s-_s^2 → 0  for all  s ≤ T+1,     
      n^-1_s-_s^2 → 0  for all  s ≤ T,
  
  and
  
      (_̆1,…,_̆T+1,_1,…,_T,_̆*)
       (U_1,…,U_T+1,F_1,…,F_T,U_*)
  
  where the joint law of this limit is as in Theorem <ref> and
  coincides with the limit in (<ref>)
  
  For the base case T=0, we have _̆1-_1=0,
  (_̆1,_̆*)  (U_1,U_*)=(Ũ_1,U_*)
  by assumption, and the remaining claims are
  vacuous. Assume inductively that these claims hold for T-1. Then for all s,s'
  ≤ T,
  
      lim_n →∞⟨_̆s _̆s'⟩=[U_sU_s']
      =[Ũ_sŨ_s']=lim_n →∞⟨_s
      _s'⟩,
  
  and similarly for all s ≤ T,
  
      lim_n →∞⟨ u_s'(_s-1) ⟩=[u_s'(F_s-1)]
      =[ũ_s'(Z̃_s-1,U_*)]
      =lim_n →∞⟨∂_s-1_s ⟩.
  
  Combining with (<ref>) and comparing (<ref>)
  with (<ref>),
  this shows that _T^∞ coincides with _T^∞,
  and hence that the limit laws in (<ref>)
  and (<ref>) coincide for T.
  
  Comparing (<ref>) with the general definition of _T from
  (<ref>), this also shows that |b_st-b̃_st| → 0
  as n →∞, for all s,t ≤ T. Denoting
  
      _t=_̆t-b_t1_̆1-…-b_tt_̆t,
  
  and applying also ≤ C_0 and
  n^-1_̆s-_s^2 → 0 for all s ≤ T by the
  induction hypothesis, we obtain n^-1_T-_T^2 → 0.
  Since _T=_̆* · (α/n)_̆*^⊤_̆T+_T,
  _T=_̆* ·μ_T^∞+_T, and
  (α/n)_̆*^⊤_̆T →α·[U_*U_T]=μ_T^∞
  by the induction hypothesis (<ref>), this shows
  
      n^-1_T-_T^2 → 0.
  
  Then, as the function u_T+1 is Lipschitz,
  
      n^-1_̆T+1-_T+1^2
      =n^-1u_T+1(_T)-u_T+1(_T)^2 → 0.
  
  This shows (<ref>) for T.
  Applying Proposition <ref>, this implies that
  (_̆1,…,_̆T+1,_1,…,_T,_̆*) must have the same empirical
  limit in W_2 as
  (_1,…,_T+1,_1,…,_T,_̆*).
  Together with (<ref>) and the coincidence of the two
  joint limit laws in (<ref>) and (<ref>)
  that was already established, this shows (<ref>)
  for T, concluding the induction and the proof.
  
  
  
  We may write the AMP iterations (<ref>–<ref>) as
  
      _t=_̌* · (α/m)_̆*^⊤_̆t+^⊤_̆t-b_t1_̌1-…-b_t,t-1_̌t-1,         _̌t=v_t(_t)
  
  
      _t=_̆* · (α/m)_̌*^⊤_̌t+_̌t-a_t1_̆1-…-a_tt_̆t,
              _̆t+1=u_t+1(_t).
  
  Approximating (α/m)_̆*^⊤_̆t ≈α·[U_*U_t]=ν_t^∞ and (α/m)_̌*^⊤_̌t ≈α/γ·[V_*V_t]=μ_t^∞, we consider the
  auxiliary AMP sequence initialized at _1=_̆1 and defined by
  
      _t=^⊤_t-b̃_t1_1-…
      -b̃_t,t-1_t-1,         _t=ṽ_t(_t,_̌*)
      ≡ v_t(_t+ν_t^∞_̌*),
  
  
      _t=_t-ã_t1_t-…-ã_tt_t,
              _t+1=ũ_t+1(_t,_̆*)
      ≡ u_t+1(_t+μ_t^∞_̆*).
  
  Here, the debiasing coefficients are the last columns of _t and
  _t defined by (<ref>)
  with the iterates _t,_t and with
  rectangular free cumulants κ̃_2k of . The partial
  derivatives in (<ref>) and (<ref>) are
  given by ∂_s ũ_t+1(·)=ũ_t+1'(·)
  if s=t and 0 otherwise, and ∂_s
  ṽ_t(·)=ṽ_t'(·) if s=t and 0 otherwise,
  where ũ_t+1'(·) and ṽ_t'(·) denote their
  derivatives with respect to the first arguments ỹ_t and z̃_t.
  
  This auxiliary AMP sequence
  is of the form (<ref>–<ref>) with side information
  =_̆* and =_̌*. Setting
  
      _t=_t+μ_t^∞_̆*,     _t=_t+ν_t^∞_̌*,
  
  Corollary <ref> then implies for each fixed T ≥ 1 that
  
      (_1,…,_T,_1,…,_T,_̌*)
          (Ṽ_1,…,Ṽ_T,G̃_1,…,G̃_T,V_*)
      
      (_1,…,_T+1,_1,…,_T,_̆*)
          (Ũ_1,…,Ũ_T+1,F̃_1,…,F̃_T,U_*)
  
  where these limits are described by
  F̃_t=Ỹ_t+μ_t^∞ U_*,
  G̃_t=Z̃_t+ν_t^∞ V_*,
  (Ỹ_1,…,Ỹ_T) ∼(0,_T^∞)
  and (Z̃_1,…,Z̃_T) ∼(0,_T^∞).
  Here, the forms for _T^∞ and _T^∞
  are given by (<ref>) and (<ref>) defined for
  this auxiliary sequence: In the definitions (<ref>),
  summing the first terms of _T^(j) and
  _T^(j) in (<ref>–<ref>) yields
  the terms with coefficient κ_2(j+k+1) in
  (<ref>) and (<ref>), while
  summing the second terms of (<ref>–<ref>)
  yields the terms with coefficient κ_2(j+k+2).
  
  The proof is concluded by a similar comparison argument as in the preceding
  proof of Theorem <ref>(a), showing that these joint limit laws
  coincide with those of Theorem <ref> and that the original AMP
  sequence converges also to these joint laws. We omit the details for brevity.
  
  
  
  
   §.§ Analysis of state evolutions
  
  
  We now prove Theorems <ref>(b) and <ref>(b).
  For notational simplicity, we will drop all
  superscripts ^∞ in this section, so that
  κ_k,_T,_T etc. are all understood as their
  deterministic n →∞ limits. We use the entrywise notation
  
      _T=(δ_st)_s,t=1^T,
       _T=(σ_st)_s,t=1^T,  _T=(μ_t)_t=1^T,
       _T=(γ_st)_s,t=1^T,
       _T=(ω_st)_s,t=1^T,  _T=(ν_t)_t=1^T.
  
  
  The proofs will apply a contractive mapping argument to show that
  the matrices _T, _T, _T, and _T all
  converge in a certain normed space. Fix an arbitrary constant ζ∈ (0,1),
  say
  
      ζ=1/2.
  
  We consider the space of “infinite matrices”
  =(x_st:s,t ≤ 0), indexed by the non-positive integers. The index
  (0,0) should be interpreted as the lower-right corner of . We equip this
  space with a weighted ℓ_∞-norm
  
      _ζ=sup_s,t ≤ 0ζ^|s| ∨ |t| |x_st|,
           |s| ∨ |t|=max(|s|,|t|).
  
  Thus the weight is ζ^k for the 2k+1 coordinate pairs
  
      (s,t)=(-k,0),(-k,-1),…,(-k,-k+1),(-k,-k),(-k+1,-k),…,
      (-1,-k),(0,-k).
  
  Define ={:_ζ<∞},
  and observe that  is complete under ·_ζ.
  For any compact interval I ⊂, denote
  
      _I={:_st∈ I  for all  s,t ≤ 0}⊂.
  
  Then _I is closed in , and hence _I is also
  complete under the norm ·_ζ.
  
  We will embed the matrices _T,_T,_T,_T as
  elements ,,,∈, with the coordinate identifications
  
      δ_st=x_s-T,t-T,     σ_st=y_s-T,t-T,
          γ_st=z_s-T,t-T,     ω_st=w_s-T,t-T
  
  
      x_st=y_st=z_st=w_st=0    if   
      s ≤ -T  or  t ≤ -T.
  
  Thus _T, _T, _T, and _T fill out the
  lower-right T × T corners of the corresponding sequences in ,
  with their lower-right (T,T) entries identified with the coordinate (0,0)
  of . Zero-padding is applied for the remaining entries of (,,,)
  not belonging to this corner. The proofs will then have two main steps:
  
  
    * For large T, the state evolution that maps these matrices from
  iterate T to iterate T+1 will be approximated by a fixed map that is
  independent of T, where the approximation is in the norm ·_ζ.
  
    * This map will be shown to be contractive over certain sub-domains of
   with respect to ·_ζ, and hence these matrices will
  converge to a fixed point of this map.
  
  
  
  
    §.§.§ Symmetric square matrices
  
  
  We first show Theorem <ref>(b). Recall that the AMP algorithm is given
  by (<ref>–<ref>), where we take u_t+1(·)
  to be the single-iterate posterior mean denoiser in (<ref>).
  Differentiating (<ref>) in f, we obtain
  
      ∂/∂ fη(f |μ,σ^2)=[U_*, ∂/∂ f(-(f-μ U_*)^2/2σ^2) |F=f]
      =μ/σ^2[U_* | F=f].
  
  Then
  
      u_t+1'(f_t)=μ_t/σ_tt[U_* | F_t=f_t].
  
  Observe that for all t ≥ 1,
  
      [[U_* | F_t]]=[(U_*-U_t+1)^2]
      =[U_*^2]-[U_t+1^2]=1-δ_t+1,t+1.
  
  So (<ref>) may be written more explicitly as
  
      σ_st=∑_j=0^s-1∑_k=0^t-1κ_j+k+2(∏_i=s-j+1^s μ_i-1/σ_i-1,i-1(1-δ_ii))
      (∏_i=t-k+1^t μ_i-1/σ_i-1,i-1(1-δ_ii))δ_s-j,t-k.
  
  In this expression, we have
  
      μ_1=α·[U_1U_*]=α,     μ_i=α·[U_iU_*]=α·[U_i^2]=αδ_ii for  i ≥ 2.
  
  
  For a sufficiently large constant C>0 depending on C_0 and ,
  we define the intervals
  
      I_Δ=[1-C/α^2, 1],
           I_Σ=[1/2κ_2,3/2κ_2],
  
  and the corresponding domains
  _I_Δ,_I_Σ⊂ by (<ref>).
  Motivated by the forms (<ref>) and (<ref>),
  we will approximate the map (_T,_T-1)
  ↦_T by a fixed map h^Σ:_I_Δ×_I_Σ→, defined entrywise by
  
      h^Σ_st(,)=∑_j=0^∞∑_k=0^∞κ_j+k+2(∏_i=s-j+1^s α x_i-1,i-1/y_ii(1-x_ii))
      (∏_i=t-k+1^t α x_i-1,i-1/y_ii(1-x_ii))x_s-j,t-k.
  
  (Note that the embedding of _T-1 in  has indices that are
  offset from those of _T and _T by 1, so y_ii appears
  instead of y_i-1,i-1.)
  We will approximate the map (_T,_T) ↦_T+1 by a
  fixed map
  h^Δ:_I_Δ×_I_Σ→, defined as
  
      h^Δ_st(,)=_,[_,[U_* | F_s]
      _,[U_* | F_t]]
  
  where the expectations are with respect to the (,)-dependent joint law
  
      (F_s,F_t)=(α x_ss,α x_tt)U_*
      +(Z_s,Z_t),      (Z_s,Z_t) ∼(0, Π[ y_ss y_st; y_ts y_tt ])  independent of  U_*.
  
  Here, we denote by Π:I_Σ^2 × 2→ I_Σ^2 × 2 the
  map
  
      Π[ a b; c d ]
      =[                a min(√(bc),√(ad)); min(√(bc),√(ad))                d ]
  
  whose image is always symmetric positive-semidefinite, so that the above
  bivariate normal law is always well-defined. (If M is already
  symmetric positive-semidefinite, then Π(M)=M.)
  
  The following lemma establishes the Lipschitz bounds for
  h^Σ and h^Δ.
  
  
  In the setting of Theorem <ref>(b), there exist constants C,α_0>0
  such that for all α>α_0 and
  (,),(',') ∈_I_Δ×_I_Σ:
  
  
    * h^Σ(,) ∈_I_Σ and
  h^Σ(,)-h^Σ(',')_ζ≤ Cα-'_ζ
  +(C/α)-'_ζ.
  
    * h^Δ(,) ∈_I_Δ and
  h^Δ(,)-h^Δ(',')_ζ≤ (C/α)-'_ζ
  +(C/α^2)-'_ζ.
  
  
  
  Let C,C',c,… denote constants depending only on C_0 and  and
  changing from instance to instance.
  For part (a), let us write (<ref>) as
  
      h^Σ_st(,)=∑_j,k=0^∞κ_j+k+2
      h_st^(j,k)(,).
  
  Observe that for (j,k)=(0,0), we have simply
  h_st^(0,0)(,)=x_st.
  By the given domains of  and , for all other (j,k), we have the bounds
  x_i-1,i-1≤ 1, y_ii≥κ_2/2, and 1-x_ii≤ C/α^2
  in the products defining (<ref>). There are j+k factors of
  the form (α x_i-1,i-1/y_ii)(1-x_ii), yielding
  
      |h_st^(j,k)(,)| ≤ (C/α)^j+k.
  
  Applying |κ_j+k+2| ≤ (16C_0)^j+k+2 by
  Proposition <ref>, for α>α_0 sufficiently large,
  this implies
  
      |h^Σ_st(,)-κ_2 x_st|
      =|∑_(j,k) ≠ (0,0)κ_j+k+2
      h_st^(j,k)(,)| ≤κ_2/3.
  
  Applying x_st∈ [1-C/α^2,1], this yields
  h^Σ_st(,) ∈ I_Σ=[κ_2/2,3κ_2/2] for
  α>α_0 sufficiently large. Then h^Σ(,) ∈_I_Σ.
  
  To show the Lipschitz bound for h^Σ,
  for λ∈ [0,1] we set
  
      ^λ=λ+(1-λ)',     ^λ
      =λ+(1-λ)'.
  
  Then
  
      |h^Σ_st(,)-h^Σ_st(',')|
      =|∫_0^1 d/dλ
      h^Σ_st(^λ,^λ)dλ|
      ≤sup_λ∈ [0,1]|d/dλ
      h^Σ_st(^λ,^λ)|.
  
  By the chain rule,
  
      d/dλ h^Σ_st(^λ,^λ)
      =∑_p,q ≤ 0 (x_pq-x_pq')∂ h^Σ_st/∂ x_pq (^λ,^λ)
      +∑_p ≤ 0 (y_pp-y_pp')∂ h^Σ_st/∂ y_pp
      (^λ,^λ).
  
  
  We establish a uniform bound for these partial derivatives.
  For any ,∈_I_Δ×_I_Σ,
  applying the above bound for (α x_i-1,i-1/y_ii)(1-x_ii),
  we have
  
      |∂ h_st^(j,k)/∂ x_pp|
         ≤
      Cα^2 (C/α)^j+k    if  p ∈{s-j+1,…,s}∪{t-k+1,…,t}
      
      (C/α)^j+k    if  p=s-j  or  p=t-k
      
      |∂ h_st^(j,k)/∂ y_pp|
         ≤ (C/α)^j+k   if  p ∈{s-j+1,…,s}∪{t-k+1,…,t}
      |∂ h_st^(j,k)/∂ x_s-j,t-k|
         ≤ (C/α)^j+k,
  
  and all other partial derivatives of h_st^(j,k) are 0.
  Multiplying by κ_j+k+2, applying |κ_j+k+2| ≤
  (16C_0)^j+k+2, and summing over j,k ≥ 0, this implies
  
      |∂ h^Σ_st/∂ x_pp|
         ≤∑_j,k ≥ 0 |κ_j+k+2| ·|∂ h_st^(j,k)/∂ x_pp|
         ≤{p ≤ s}∑_k ≥
      0C(C'/α)^(s-p)+k+
      {p ≤ t}∑_j ≥ 0 C(C'/α)^j+(t-p)
         +{p ≤ s}∑_j ≥ s+1-p∑_k ≥ 0
      Cα^2(C'/α)^j+k
      +{p ≤ t}∑_k ≥ t+1-p∑_j ≥ 0
      Cα^2(C'/α)^j+k
  
  where the first two terms are the contributions from
  p=s-j and p=t-k, and the latter two terms are the contributions
  from p ∈{s-j+1,…,s} and p ∈{t-k+1,…,t}.
  For α>α_0 sufficiently large, this simplifies to the bound
  
      |∂ h^Σ_st/∂ x_pp|
      ≤ Cα({p ≤ s}·(C'/α)^s-p
      +{p ≤ t}·(C'/α)^t-p).
  
  We have similarly
  
      |∂ h^Σ_st/∂ y_pp|
         ≤∑_j,k ≥ 0 |κ_j+k+2| ·|∂ h_st^(j,k)/∂ y_pp|
         ≤{p ≤ s}∑_j ≥ s+1-p∑_k ≥ 0
      C(C'/α)^j+k
      +{p ≤ t}∑_k ≥ t+1-p∑_j ≥ 0
      C(C'/α)^j+k
         ≤C/α({p ≤ s}·(C'/α)^s-p
      +{p ≤ t}·(C'/α)^t-p)
  
  and, for p ≠ q,
  
      |∂ h^Σ_st/∂ x_pq|
      ≤∑_j,k ≥ 0 |κ_j+k+2| ·|∂ h_st^(j,k)/∂ x_pq|
      ≤ C ·{p ≤ s  and  q ≤ t}·(C'/α)^s-p+t-q.
  
  
  Applying these bounds to (<ref>) and (<ref>),
  
      |h^Σ_st(,)-h^Σ_st(',')|
         ≤ Cα∑_p ≤ s|x_pp-x_pp'|
      (C'/α)^s-p
      +Cα∑_p ≤ t|x_pp-x_pp'|(C'/α)^t-p
         
      +C/α∑_p ≤ s|y_pp-y_pp'|(C'/α)^s-p
      +C/α∑_p ≤ t|y_pp-y_pp'|(C'/α)^t-p
         +C∑_p ≤ s∑_q ≤ t |x_pq-x_pq'|
      (C'/α)^s-p+t-q.
  
  For α>α_0 large enough, we may bound the terms above using
  
      ∑_p ≤ s |x_pp-x_pp'| (C'/α)^s-p   ≤sup_p ≤ s |x_pp-x_pp'|ζ^|p|·∑_p ≤ sζ^-|p| (C'/α)^s-p≤ C-'_ζ·ζ^-|s|,
      ∑_p ≤ s |y_pp-y_pp'| (C'/α)^s-p   ≤sup_p ≤ s |y_pp-y_pp'|ζ^|p|·∑_p ≤ sζ^-|p| (C'/α)^s-p≤ C-'_ζ·ζ^-|s|,
      ∑_p ≤ s∑_q ≤ t
      |x_pq-x_pq'|(C'/α)^s-p+t-q   ≤sup_p ≤ s  and  q ≤ t
      |x_pq-x_pq'|ζ^|p| ∨ |q|·∑_p ≤ s∑_q ≤ tζ^-(|p| ∨ |q|)(C'/α)^s-p+t-q
         ≤ C-'_ζ·ζ^-(|s| ∨ |t|).
  
  Then
  
      h^Σ(,)-h^Σ(',')_ζ
      =sup_s,t ≤ 0 |h^Σ_st(,)-h^Σ_st(',')|ζ^|s| ∨ |t|≤ Cα-'_ζ+C/α-'_ζ,
  
  yielding the Lipschitz bound in part (a).
  
  For part (b), let us denote
  
      η(f |μ,σ^2)=[U_* | F=f],     
      v(f |μ,σ^2)=[U_* | F=f]
  
  in the scalar model F=μ· U_*+Z, where Z ∼(0,σ^2) is
  independent of U_*. Observe that
  
      [U_* | F]=[μ^-1(F-Z) | F]=(1/μ)^2 [Z | F],
  
  so that
  
      [v(F |μ,σ^2)]
      =[[U_* | F]]=(1/μ)^2 [[Z | F]]
      ≤ (1/μ)^2 [Z]=(σ/μ)^2.
  
  
  For ease of notation, let us write  for _, in the
  definition of the function h^Δ. We denote
  
      U_t=[U_* | F_t]=η(F_t |α x_tt,y_tt).
  
  Observe that the above then implies
  
      [(U_*-U_t)^2]=[v(F_t |α x_tt,y_tt)]
      ≤ y_tt/(α x_tt)^2.
  
  For ∈_I_Δ and ∈_I_Σ,
  applying y_tt≤ (3/2)κ_2 and x_tt^2 ≥ 3/4 for
  all α>α_0 sufficiently large, this shows
  
      [(U_*-U_t)^2] ≤ 2κ_2/α^2.
  
  Now applying [U_*U_t]=[U_t^2] and
  [U_*^2]=1, we may write
  
      [U_sU_t]-1   =[(U_s-U_*)(U_t-U_*)]+([U_s^2]-1)+
      ([U_t^2]-1)
         =[(U_s-U_*)(U_t-U_*)]-[(U_s-U_*)^2]
      -[(U_t-U_*)^2]
         ≥ -3/2([(U_s-U_*)^2]+[(U_t-U_*)^2]).
  
  So [U_sU_t] ≥ 1-6κ_2/α^2. By Cauchy-Schwarz, also
  [U_sU_t] ≤[U_s^2]^1/2[U_t^2]^1/2≤[U_*^2]=1,
  so h^Δ(,) ∈_I_Δ.
  
  To show the Lipschitz bound for h^Δ, from (<ref>) we have
  
      |h_st^Δ(,)-h_st^Δ(',')|
         =|[U_sU_t]-[U_s'U_t']|
         ≤|[(U_s-U_*)(U_t-U_*)]-[(U_s'-U_*)(U_t'-U_*)]|
         +|[(U_s-U_*)^2]-[(U_s'-U_*)^2]|
      +|[(U_t-U_*)^2]-[(U_t'-U_*)^2]|
  
  so it suffices to bound these three terms individually. We demonstrate the
  bound for the first term: Let us write
  
      [ F_s; F_t ]=[ μ_s; μ_t ]U_*+[ β_ss β_st; β_st β_tt ][ W_s; W_t ]
  
  where
  
      (μ_s,μ_t)=(α x_ss,α x_tt),     [ β_ss β_st; β_st β_tt ]=(Π[ y_ss y_st; y_ts y_tt ])^1/2,      (W_s,W_t) ∼(0,).
  
  Here (·)^1/2 denotes the positive-semidefinite matrix square-root,
  given explicitly for 2 × 2 matrices by
  
      M^1/2=1/√( M+2√( M))(M+√( M)·_2 × 2).
  
  Then
  
      U_s=η(F_s |α x_ss,y_ss)
      =η(F_s |μ_s,β_ss^2+β_st^2),
           U_t=η(F_t |α x_tt,y_tt)
      =η(F_t |μ_t,β_st^2+β_tt^2).
  
  For λ∈ [0,1], writing the same forms for F_s',F_t',U_s',U_t',
  we define the linear interpolations
  
      μ_s^λ=λμ_s+(1-λ)μ_s',
          β_ss^λ=λβ_ss+(1-λ) β_ss',
           F_s^λ=λ F_s+(1-λ) F_s',
  
  and similarly for μ_t,β_st,β_tt,F_t. Finally, we define
  
      σ_ss^λ=(β_ss^λ)^2+(β_st^λ)^2,
          σ_tt^λ=(β_st^λ)^2+(β_tt^λ)^2,
  
  
      U_s^λ=η(F_s^λ|μ_s^λ,σ_ss^λ),
          
      U_t^λ=η(F_t^λ|μ_t^λ,σ_tt^λ).
  
  Denoting ∂_λ as the derivative in λ, we then have
  
      [(U_s-U_*)(U_t-U_*)]-[(U_s'-U_*)(U_t'-U_*)]
      =∫_0^1 ∂_λ[(U_s^λ-U_*)(U_t^λ-U_*)]dλ
  
  where this latter expectation is over the underlying random variables
  (U_*,W_s,W_t). The law of
  (U_*,W_s,W_t) does not depend on λ, so we may take the derivative
  inside this expectation, yielding
  
      |[(U_s-U_*)(U_t-U_*)]-[(U_s'-U_*)(U_t'-U_*)]|
         ≤∫_0^1
      [|∂_λ U_s^λ·
      (U_t^λ-U_*)|+|(U_s^λ-U_*) ·∂_λ U_t^λ|] dλ
  
  Observe from (<ref>) and the condition
  y_ss,y_st,y_ts,y_tt>0 that β_ss,β_st,β_tt>0. Then
  
      σ_ss^λ   ≤ 2λ^2(β_ss^2+β_st^2)
      +2(1-λ)^2(β_ss'^2+β_st'^2)
      =2λ^2 y_ss+2(1-λ)^2y_ss'
      σ_ss^λ   ≥λ^2(β_ss^2+β_st^2)
      +(1-λ)^2(β_ss'^2+β_st'^2)
      =λ^2 y_ss+(1-λ)^2y_ss'.
  
  Recalling that ∈_I_Δ and ∈_I_Σ,
  we get the bounds
  
      μ_s^λ≤α,     μ_s^λ≥α-C/α,
          σ_ss^λ≤ C,     σ_ss^λ≥ c.
  
  for some constants C,c>0.
  Applying these bounds together with (<ref>) yields
  [(U_s^λ-U_*)^2] ≤ C/α^2. The same argument holds for
  [(U_t^λ-U_*)^2]. Then applying Cauchy-Schwarz to
  (<ref>),
  
      |[(U_s-U_*)(U_t-U_*)]-[(U_s'-U_*)(U_t'-U_*)]|
      ≤C/α∫_0^1
      ([(∂_λ U_s^λ)^2
      ]^1/2+[(∂_λ
      U_t^λ)^2]^1/2)dλ.
  
  
  We proceed to bound [(∂_λ U_s^λ)^2]:
  Recall η(f |μ,σ^2) and v(f |μ,σ^2)
  from (<ref>), and define in addition
  
      k(f |μ,σ^2)=[U_*,U_*^2 | F=f].
  
  Differentiating the explicit form for η(f |μ,σ^2)
  in (<ref>) yields
  
      ∂/∂ fη(f |μ,σ^2)
         =[U_*, ∂/∂ f(-(f-μ
      U_*)^2/2σ^2) | F=f]
      =μ/σ^2· v(f |μ,σ^2)
      ∂/∂μη(f |μ,σ^2)
         =[U_*, ∂/∂μ(
      -(f-μ U_*)^2/2σ^2)|F=f]
      =f/σ^2v(f |μ,σ^2)-μ/σ^2k(f |μ,σ^2)
      ∂/∂σ^2η(f |μ,σ^2)
         =[U_*, ∂/∂σ^2(
      -(f-μ U_*)^2/2σ^2)|F=f]
      =-fμ/σ^4v(f |μ,σ^2)
      +μ^2/2σ^4k(f |μ,σ^2).
  
  Let us write as shorthand
  
      V_s^λ=v(F_s^λ|μ_s^λ,σ_ss^λ),
           K_s^λ=k(F_s^λ|μ_s^λ,σ_ss^λ).
  
  Then applying the chain rule to differentiate (<ref>),
  
      ∂_λ U_s^λ   =∂η/∂ f·(∂_λμ_s^λ· U_*
      +∂_λβ_ss^λ· W_s
      +∂_λβ_st^λ· W_t)
      +∂η/∂μ·∂_λμ_s^λ
      +∂η/∂σ^2·(2β_ss^λ·∂_λβ_ss^λ
      +2β_st^λ·∂_λβ_st^λ)
         =(μ_s^λ/σ_ss^λ
      U_*V_s^λ+1/σ_ss^λF_s^λ V_s^λ
      -μ_s^λ/σ_ss^λK_s^λ)
      ∂_λμ_s^λ
      +(μ_s^λ/σ_ss^λW_sV_s^λ
      -2β_ss^λμ_s^λ/(σ_ss^λ)^2
      F_s^λ V_s^λ+(μ_s^λ)^2
      β_ss^λ/(σ_ss^λ)^2K_s^λ)
      ∂_λβ_ss^λ
         +(μ_s^λ/σ_ss^λW_tV_s^λ
      -2β_st^λμ_s^λ/(σ_ss^λ)^2
      F_s^λ V_s^λ+(μ_s^λ)^2
      β_st^λ/(σ_ss^λ)^2K_s^λ)
      ∂_λβ_st^λ
         ≡I·∂_λμ_s^λ
      +II·∂_λβ_ss^λ
      +III·∂_λβ_st^λ.
  
  
  We bound the expected squares of these coefficients
  I,II,III:
  Applying (<ref>) and Cauchy-Schwarz,
  
      [(U_*V_s^λ)^2]    ≤[U_*^4]^1/2[(V_s^λ)^4]^1/2≤ C ·[(V_s^λ)^4]^1/2
      [(F_s^λ V_s^λ)^2]
         ≤[(F_s^λ)^4]^1/2[(V_s^λ)^4]^1/2≤ Cα^2 [(V_t^λ)^4]^1/2.
  
  Then the coefficient for ∂_λμ_s^λ in
  (<ref>) has expected square bounded as
  
      [I^2]=[(μ_s^λ/σ_ss^λ
      U_*V_s^λ+1/σ_ss^λF_s^λ V_s^λ
      -μ_s^λ/σ_ss^λK_s^λ)^2]
      ≤ Cα^2([(V_s^λ)^4]^1/2+[(K_s^λ)^2]).
  
  Recalling the identity
  
      v(F |μ,σ^2)
      =[U_* | F]=[μ^-1(F-σ W) | F]=(σ/μ)^2
      [W | F],
  
  we obtain
  
      [v(F |μ,σ^2)^4]
         =(σ/μ)^8 [[W | F]^4]
         =(σ/μ)^8 [[(W-[W | F])^2 | F]^4]
         ≤ (σ/μ)^8 [(W-[W | F])^8] ≤
      C(σ/μ)^8,
  
  the last inequality applying W ∼(0,1) and
  Proposition <ref>. So [(V_s^λ)^4]
  ≤ Cα^-8. We also have the identity
  
      k(F |μ,σ^2)
      =[U_*,U_*^2 | F]   =μ^-3[F-σ W,(F-σ W)^2 | F]
         =(2σ^2F/μ^3)[W | F]-(σ^3/μ^3)[W,W^2 |
      F],
  
  so
  
      [k(F |μ,σ^2)^2]
         ≤ (8σ^4/μ^6)[F^2[W | F]^2]
      +(2σ^6/μ^6)[[W,W^2 | F]^2]
         ≤ (8σ^4/μ^6)[F^4]^1/2[[W | F]^4]^1/2
      +(2σ^6/μ^6)[[W | F]^2]^1/2[[W^2 | F]^2]^1/2
         ≤ C(σ^4/μ^6)[F^4]^1/2+C(σ/μ)^6
  
  by similar arguments. Then [(K_s^λ)^2] ≤ Cα^-4, and
  we obtain
  
      [I^2] ≤ C/α^2.
  
  
  For the coefficients of ∂_λβ_ss^λ 
  and ∂_λβ_st^λ in (<ref>), we first
  apply a cancellation of the leading-order term: Comparing the identities
  (<ref>) and (<ref>), we have
  
      K_s^λ=2/μ_s^λF_s^λ V_s^λ
      -(σ_ss^λ)^3/2/(μ_s^λ)^3[W,W^2 | F_s^λ]
  
  where W ∼(0,1) is the Gaussian variable such that
  F_s^λ=μ_s^λ· U_*+√(σ_ss^λ)· W.
  Then the coefficient of ∂_λβ_ss^λ in
  (<ref>) is
  
      II=μ_s^λ/σ_ss^λW_sV_s^λ
      -2β_ss^λμ_s^λ/(σ_ss^λ)^2
      F_s^λ V_s^λ+(μ_s^λ)^2
      β_ss^λ/(σ_ss^λ)^2K_s^λ 
      =μ_s^λ/σ_ss^λW_sV_s^λ
      -β_ss^λ/μ_s^λ(σ_ss^λ)^1/2[W,W^2 | F_s^λ].
  
  Similar arguments as above yield [(W_s^2V_s^λ)^2]
  ≤ Cα^-4 and [[W,W^2 | F_s^λ]^2] ≤ C.
  Thus we obtain the bound
  
      [II^2] ≤ C/α^2.
  
  For the coefficient of ∂_λβ_st^λ, the same argument
  shows
  
      [III^2] ≤ C/α^2.
  
  Applying (<ref>), (<ref>), and (<ref>)
  to (<ref>) yields
  
      [(∂_λ U_s^λ)^2]
      ≤C/α^2((∂_λμ_s^λ)^2
      +(∂_λβ_ss^λ)^2+(∂_λβ_st^λ)^2).
  
  The same argument applies for U_t^λ to show
  
      [(∂_λ U_t^λ)^2]
      ≤C/α^2((∂_λμ_t^λ)^2
      +(∂_λβ_st^λ)^2+(∂_λβ_tt^λ)^2).
  
  By the definition of our linear interpolation,
  ∂_λμ_s^λ=μ_s-μ_s' which does not depend on
  λ, and similarly for the other derivatives above. Then applying this to
  (<ref>) yields a bound of
  
      C/α^2(|μ_s-μ_s'|+|μ_t-μ_t'|
      +|β_ss-β_ss'|+|β_st-β_st'|
      +|β_tt-β_tt'|)
  
  for the first term in (<ref>).
  
  The same argument applied with s=t bounds the other two terms
  of (<ref>), and we obtain
  
      |h^Δ_st(,)-h^Δ_st(',')|    ≤C/α^2(|μ_s-μ_s'|+|μ_t-μ_t'|
      +|β_ss-β_ss'|+|β_st-β_st'|
      +|β_tt-β_tt'|)
  
  Observe that |μ_s-μ_s'|=α|x_ss-x_ss'|. Furthermore, it may be
  verified from (<ref>) and (<ref>) that the map
  (y_ss,y_st,y_ts,y_tt) ↦ (β_ss,β_st,β_tt)
  is Lipschitz over y_ss,y_st,y_ts,y_tt∈ I_Σ,
  since I_Σ is bounded away from 0. Then
  
      |h^Δ_st(,)-h^Δ_st(',')|    ≤C/α(|x_ss-x_ss'|+|x_tt-x_tt'|)
         
      +C/α^2(|y_ss-y_ss'|+|y_st-y_st'|+|y_ts-y_ts'|
      +|y_tt-y_tt'|).
  
  This implies
  
      h^Δ(,)-h^Δ(',')_ζ
      =sup_s,t ≤ 0 |h^Δ(,)_s,t-h^Δ(',')_s,t|ζ^|s| ∨ |t|≤C/α-'_ζ+C/α^2-'_ζ,
  
  which shows part (b).
  
  
  The next lemma establishes the approximation of the state evolution for
  _T by the fixed map h^Σ, and the state evolution for _T
  by the fixed map h^Δ.
  
  
  In the setting of Theorem <ref>(b), there exist constants C,α_0>0
  such that for all α>α_0 and T ≥ 1:
  
  
    * δ_st∈ I_Δ and σ_st∈ I_Σ for all s,t
  ∈{2,…,T} whereas δ_st∈ [-1,1] and σ_st∈
  [-3κ_2/2, 3κ_2/2] if s=1 and t ∈{1,…,T} or t=1 and
  s ∈{1,…,T}.
  
    * Consider _T,_T as elements of
  , with the coordinate identifications and zero-padding
  of (<ref>). Then for any
  (,) ∈_I_Δ×_I_Σ,
  
      _T-h^Σ(,)_ζ≤ Cα-_T_ζ
      +(C/α)-_T-1_ζ+Cζ^T/2.
  
  
    * Similarly, for any
  (,) ∈_I_Δ×_I_Σ,
  
      _T+1-h^Δ(,)_ζ≤ (C/α)-_T_ζ
      +(C/α^2)-_T_ζ+Cζ^T.
  
  
  
  
  For part (a), the arguments are similar to those in the proof of
  Lemma <ref>: We induct on T.
  Note that δ_11=[U_1^2] ≤ 1 so the claim holds for _1.
  Suppose that the claims hold for _T and _T-1.
  To establish the claim for _T,
  for any s,t ≥ 1 we may write (<ref>) as
  
      σ_st=∑_j=0^s-1∑_k=0^t-1κ_j+k+2σ_st^(j,k).
  
  Observe that σ_st^(0,0)=δ_st, which belongs to I_Δ if
  s,t ≥ 2 and to [-1,1] otherwise. Observe also that
  |σ_st^(j,k)| ≤ (C/α)^j+k for all (j,k) ≠ (0,0)
  by the same argument as in Lemma <ref>(a).
  (Here, each factor 1-δ_ii has an index i ≥ 2, so this is at most
  C/α^2.)
  For sufficiently large α, applying |κ_j+k+2| ≤
  (16C_0)^j+k+2 and summing over (j,k) ≠ (0,0) shows the claim
  for _T.
  Now suppose that the claims of part (a) hold for _T
  and _T, and consider _T+1. For s=1 or t=1, we apply
  |δ_s,T+1|=|[U_1U_T+1]| ≤[U_1^2]^1/2[U_T+1^2]^1/2≤ 1.
  For s,t ≥ 2, the argument of (<ref>) in
  Lemma <ref>(b) shows
  
      δ_st-1 ≥ -3/2(σ_s-1,s-1/μ_s-1^2+σ_t-1,t-1/μ_t-1^2).
  
  We recall that μ_j=αδ_jj if j ≥ 2 and
  μ_1=α. For sufficiently large α, this implies
  δ_st∈ I_Δ in both cases (where the constant C defining
  I_Δ depends on ), so the claim holds for
  _T+1. This concludes the induction and establishes part (a).
  
  For part (b), let us now index the entries of
  _T,_T-1,_T by
  s,t ≤ 0, to coincide with the indices of . Let
  ' ∈_I_Δ be _T with each coordinate projected onto
  the interval I_Δ, and let ' ∈_I_Σ
  be the analogous projection of _T-1 onto
  I_Σ. We first bound _T-h^Σ(',')_ζ.
  Observe that by part (a) already shown, ' must coincide
  with _T in the lower-right (T-1) × (T-1) corner,
  and ' must coincide with _T-1 in the lower-right
  (T-2) × (T-2) corner.
  Applying again |κ_j+k+2| ≤ (16C_0)^j+k+2, we may write
  
      (_T)_st=∑_0 ≤ j,k<T/4κ_j+k+2(_T)_st^(j,k)+R_st
  
  where this remainder satisfies |R_st| ≤ C(C'/α)^T/4 for
  α>α_0 sufficiently large. Similarly, we may write
  
      h^Σ_st(',')=∑_0 ≤ j,k<T/4κ_j+k+2h_st^(j,k)(',')+R_st(',')
  
  where |R_st(',')| ≤ C(C'/α)^T/4. Comparing the forms of
  (<ref>) and (<ref>), observe that for s,t>-T/2
  and j,k<T/4, we have (_T)_st^(j,k)=h_st^(j,k)(',')
  because these are identical functions of the entries of the lower-right
  (3T/4) × (3T/4) sub-matrices of _T and _T-1. So
  
      |(_T)_st-h_st^Σ(',')| ≤
      C(C'/α)^T/4 for all  s,t>-T/2.
   
  Applying the trivial bound
  
      |(_T)_st-h^Σ_st(',')|
      ≤ |(_T)_st|+|h^Σ_st(',')| ≤ 3κ_2
       for  s ≤ -T/2  or  t ≤ -T/2,
  
  we obtain
  
      _T-h^Σ(',')_ζ≤ C(C'/α)^T/4+Cζ^T/2≤ Cζ^T/2
  
  for α>α_0 large enough. By Lemma <ref>(a) and
  the definitions of ',', we have also
  
      h^Σ(',')-h^Σ(,)_ζ   ≤
      Cα-'_ζ+(C/α)-'_ζ
         ≤ Cα-_T_ζ+(C/α)-_T-1_ζ,
  
  and combining these shows part (b).
  
  For part (c), now let ' and ' be the coordinate-wise projections of
  _T and _T onto I_Δ and I_Σ. We bound
  _T+1-h^Δ(',')_ζ. Observe that ' and '
  coincide with _T and _T in their lower-right
  (T-1) × (T-1) corners, and each 2 × 2 principal minor of
  _T must be positive-semidefinite because _T is a covariance
  matrix. Thus
  
      (_T+1)_st=h^Δ_st(',')  for all  s,t ∈{-T+2,…,0}.
  
  Applying the trivial bound |(_T+1)_st-h^Δ(',')_st| ≤ 2 for the remaining s,t, and
  
      h^Δ(,)-h^Δ(',')_ζ   ≤ (C/α)-'_ζ
      +(C/α^2)-'_ζ
         ≤ (C/α)-_T_ζ+(C/α^2)-_T_ζ
  
  similar to the above, we obtain part (c).
  
  
  
  Theorem <ref>(a) shows
  
      lim_n →∞ n^-1_̆T^2=[U_T^2]=δ_TT,    lim_n →∞ n^-1_̆T^⊤_̆*=[U_TU_*]=[U_T^2]=δ_TT.
  
  Thus, it suffices to show that δ_TT→Δ_* as T →∞,
  where (Δ_*,Σ_*) ∈ I_Δ× I_Σ is the unique fixed
  point of (<ref>).
  
  Consider the map G:_I_Δ×_I_Σ→_I_Δ×_I_Σ that is the successive composition of
  
      (,) ↦ (,h^Σ(,)),     
      (,) ↦ (h^Δ(,),)
  
  which approximates (_T,_T-1) ↦
  (_T+1,_T). Writing its components as G=(G_x,G_y),
  Lemma <ref> implies
  
      G_y(,)-G_y(',')_ζ   ≤
      Cα-'_ζ+(C/α)-'_ζ
      G_x(,)-G_x(',')_ζ   ≤
      (C/α)-'_ζ+(C/α^2)G_y(,)-G_y(',')_ζ
         ≤ (C'/α)-'_ζ+(C'/α^3)-'_ζ.
  
  Then defining the norm ·_ζ,α on the product space
  _I_Δ×_I_Σ by
  
      (,)_ζ,α=_ζ+(1/α^2)_ζ,
  
  this shows
  
      G(,)-G(',')_ζ,α≤ (C/α)(,)-(',')_ζ,α≤τ(,)-(',')_ζ,α
  
  for some constant τ∈ (0,1) and all α>α_0 sufficiently large.
  Thus G is a contraction on _I_Δ×_I_Σ in this
  norm, and admits a unique fixed point (_*,_*)
  ∈_I_Δ×_I_Σ by the Banach fixed point theorem.
  
  We claim that this fixed point is such that _* equals a constant Δ_*
  ∈ I_Δ and _* equals a constant Σ_* ∈ I_Σ
  in every coordinate. By the definitions of the functions h^Δ and
  h^Σ, such a pair is a fixed point if and only if
  
      Σ_*=∑_j,k=0^∞κ_j+k+2(αΔ_*/Σ_*(1-Δ_*))^j+kΔ_*,     Δ_*=[[U_* | F]^2]
  
  in the model F=αΔ_* U_*+Z where Z ∼(0,Σ_*).
  These equations may be rewritten as
  
      Σ_*   =∑_k=0^∞ (k+1)κ_k+2(αΔ_*/Σ_*(1-Δ_*))^k Δ_*
      =Δ_* R'(αΔ_*(1-Δ_*)/Σ_*)
      
      1-Δ_*   =[(U_*-[U_* | F])^2]
      =(α^2Δ_*^2/Σ_*)
  
  which is exactly the pair of fixed point equations (<ref>).
  
  To argue that such a fixed point exists and is unique in I_Δ×
  I_Σ, consider the pair of scalar maps
  
      h^Σ_sc(Δ_*,Σ_*)=Δ_* R'(αΔ_*(1-Δ_*)/Σ_*),
           h^Δ_sc(Δ_*,Σ_*)=1-(α^2Δ_*^2/Σ_*)=[[U_* | F]^2].
  
  Denote their composition as G_sc(Δ_*,Σ_*).
  Specializing Lemma <ref> to pairs (,) and
  (',') where ,',,' are each equal to a constant in every
  coordinate, our preceding arguments
  imply that G_sc:I_Δ× I_Σ→ I_Δ× I_Σ is a contraction with respect to the norm
  (Δ,Σ)=|Δ|+(1/α^2)|Σ|. Then there exists a unique
  fixed point (Δ_*,Σ_*) ∈ I_Δ× I_Σ for
  G_sc, by the Banach fixed point theorem applied to this scalar
  setting. So the fixed point (_*,_*) for G must be such that
  _* is constant and equal to Δ_*, and _* is constant and equal to
  Σ_*, by uniqueness of (_*,_*).
  
  Finally, to conclude the proof, fix any >0. Let G^T_0=G ∘…∘ G denote the T_0-fold composition of G. For the above contraction
  rate τ of this function G, the Banach fixed point theorem implies
  quantitatively, for any ,∈_I_Δ×_I_Σ,
  
      G^T_0(,)-(_*,_*)_ζ,α≤τ^T_0(,)-(_*,_*)_ζ,α≤ Cτ^T_0
  
  where the second inequality holds because
  _I_Δ×_I_Σ is bounded under
  ·_ζ,α. Then for all large enough T_0, we have
  
      G^T_0(,)-(_*,_*)_ζ,α</2.
  
  By Lemma <ref>(b) and (c), for any (,) ∈_I_Δ×_I_Σ and any T ≥ 1, also
  
      (_T+1,_T)-G(,)_ζ,α
         =_T+1-h^Δ(,h^Σ(,))_ζ+(1/α^2)
      _T-h^Σ(,)_ζ
         ≤ (C/α)-_T_ζ+Cζ^T
      +[(C/α^2)+(1/α^2)]
      (Cα-_T_ζ+(C/α)-_T-1_ζ
      +Cζ^T/2)
         ≤τ(_T,_T-1)-(,)_ζ,α
      +C'ζ^T/2.
  
  Iterating this bound,
  
      (_T+T_0,_T+T_0-1)-G^T_0(,)_ζ,α≤τ^T_0(_T,_T-1)-(,)_ζ,α
      +C'ζ^T/2(1-τ)^-1.
  
  For all large enough T_0 and T, this is also at most /2. Thus,
  combining with the above,
  
      lim sup_T →∞ |δ_TT-Δ_*| ≤lim sup_T →∞(_T+1,_T)-(_*,_*)_ζ,α≤.
  
  Here >0 is arbitrary, so |δ_TT-Δ_*| → 0 as desired.
  
  
  
  
    §.§.§ Rectangular matrices
  
  
  We now prove Theorem <ref>(b) using a similar argument.
  
  Recall that the AMP algorithm is
  given by (<ref>–<ref>), where v_t(·) and
  u_t+1(·) are the single-iterate posterior-mean denoisers
  in (<ref>). As in the symmetric square setting, we have
  
      u_t+1'(f_t)   =μ_t/σ_tt[U_* | F_t=f_t]
      =μ_t/σ_tt(1-δ_t+1,t+1)
      
      v_t'(g_t)   =ν_t/ω_tt[V_* | G_t=g_t]
      =ν_t/ω_tt(1-γ_tt).
  
  Here,
  
      μ_t=(α/γ) γ_tt,     ν_t=α    if  t=1 
      αδ_tt    if  t ≥ 2.
  
  
  For a sufficiently large constant C>0, we define the intervals
  
      I_Δ=I_Γ=[1-C/α^2, 1],   
      I_Σ=[1/2κ_2,3/2κ_2],   
      I_Ω=[1/2γκ_2,3/2γκ_2 ]
  
  and the corresponding domains
  _I_Δ,_I_Σ,_I_Γ,_I_Ω.
  We then define four maps h^Ω,h^Γ,h^Σ,h^Δ:
  _I_Δ×_I_Σ×_I_Γ×_I_Ω→ that respectively
  approximate the state evolution functions
  
      (_T,_T-1,_T-1,_T-1)    ↦_T
      
      (_T,_T-1,_T-1,_T)    ↦_T
      
      (_T,_T-1,_T,_T)    ↦_T
      
      (_T,_T,_T,_T)    ↦_T+1.
  
  Substituting the above forms of the derivatives into
  (<ref>) and (<ref>), and
  identifying (,,,) ↔
  (,,,) with the appropriate offsets of indices,
  we may define these maps to have the entries
  
      h^Ω_st(,,,)
         =γ∑_j=0^∞∑_k=0^∞((∏_i=s-j+1^s ×∏_i=t-k+1^t)
      α z_ii/γ y_ii
      (1-x_ii)α x_i-1,i-1/w_ii(1-z_ii))
      (κ_2(j+k+1)x_s-j,t-k
         +κ_2(j+k+2)α z_s-j,s-j/γ y_s-j,s-j (1-x_s-j,s-j)
      α z_t-k,t-k/γ y_t-k,t-k (1-x_t-k,t-k)z_s-j,t-k),
      
      h^Γ_st(,,,)   =_,[_,[V_* | G_s]
      _,[V_* | G_t]]
      
      h^Σ_st(,,,)
         =∑_j=0^∞∑_k=0^∞((∏_i=s-j+1^s ×∏_i=t-k+1^t)
       α x_ii/w_ii(1-z_ii)α z_i-1,i-1/γ y_ii(1-x_ii) )
      (κ_2(j+k+1) z_s-j,t-k
         +κ_2(j+k+2)α x_s-j,s-j/w_s-j,s-j(1-z_s-j,s-j)
      α x_t-k,t-k/w_t-k,t-k(1-z_t-k,t-k)x_s-j,t-k),
      
      h^Δ_st(,,,)   =_,[_,[U_* | F_s]
      _,[U_* | F_t]],
  
  where these expectations are taken with respect to the (,,,)-dependent
  joint laws
  
      (G_s,G_t)   =(α x_ss,α x_tt)V_*+
      (0, Π[ w_ss w_st; w_ts w_tt ])
      
      (F_s,F_t)   =((α/γ) z_ss,(α/γ)
      z_tt)U_*+(0, Π[ y_ss y_st; y_ts y_tt ])
  
  and Π(·) is as defined in (<ref>).
  Note that h^Γ depends only on (,), while h^Δ depends only on
  (,).
  
  The following establishes Lipschitz bounds for these functions,
  and is analogous to Lemma <ref>.
  
  
  In the setting of Theorem <ref>(b),
  there exist constants C,α_0>0
  such that for all α>α_0 and (,,,),(',',',') ∈_I_Δ×_I_Σ×_I_Γ×_I_Ω:
  
  
    * h^Σ(,,,) ∈_I_Σ, h^Ω(,,,) ∈_I_Ω,
  and
  
      h^Σ(,,,)-h^Σ(',',',')_ζ   ≤ C(-'_ζ+-')_ζ+(C/α^2)(-'_ζ
      +-'_ζ),
      h^Ω(,,,)-h^Ω(',',',')_ζ   ≤ C(-'_ζ+-')_ζ+(C/α^2)(-'_ζ
      +-'_ζ).
  
  
    * h^Δ(,,,) ∈_I_Δ, h^Γ(,,,) ∈_I_Γ,
  and
  
      h^Δ(,,,)-h^Δ(',',',')_ζ   ≤
      (C/α)-'_ζ+(C/α^2)-'_ζ,
      h^Γ(,,,)-h^Γ(',',',')_ζ   ≤
      (C/α)-'_ζ+(C/α^2)-'_ζ.
  
  
  
  
  For part (a), the argument is similar to Lemma <ref>(a).
  We denote by C,C',c,… constants that depend only on C_0,,γ.
  Let us write
  
      h_st^Σ(,,,)=∑_j,k=0^∞κ_2(j+k+1)h_st,0^(j,k)(,,,)
      +κ_2(j+k+2)h_st,1^(j,k)(,,,).
  
  For both a=0 and a=1, we have
  
      |h_st,a^(j,k)| ≤ C(C'/α)^2(j+k+a).
  
  Applying |κ_2j| ≤ C^2j from
  Proposition <ref>(b), for α>α_0 large enough,
  we obtain h_st^Σ(,,,) ∈ I_Σ. We may also verify the 
  bounds, for both a=0 and a=1,
  
      |∂ h_st,a^(j,k)/∂ y_pp|
         ≤ C(C'/α)^2(j+k+a)   if  p ∈{s-j+1,…,s} or  p ∈{t-k+1,…,t}
      |∂ h_st,a^(j,k)/∂ w_pp|
         ≤ C(C'/α)^2(j+k+a)   if  p ∈{s-j+1-a,…,s} or  p ∈{t-k+1-a,…,t}
      |∂ h_st,0^(j,k)/∂ x_pp|
         ≤ Cα^2(C'/α)^2(j+k)   if  p ∈{s-j+1,…,s} or  p ∈{t-k+1,…,t}
      |∂ h_st,1^(j,k)/∂ x_pp|
         ≤ Cα^2(C'/α)^2(j+k+1)    if  p ∈{s-j+1,…,s} or  p ∈{t-k+1,…,t}
      
      C(C'/α)^2(j+k+1)    if  p=s-j  or  p=t-k
      
      |∂ h_st,0^(j,k)/∂ z_pp|
         ≤ Cα^2(C'/α)^2(j+k)    if  p ∈{s-j+1,…,s} or  p ∈{t-k+1,…,t}
      
      C(C'/α)^2(j+k)    if  p=s-j  or  t-k
      
      |∂ h_st,1^(j,k)/∂ z_pp|
         ≤ Cα^2(C'/α)^2(j+k+1)   if  p ∈{s-j,…,s} or  p ∈{t-k,…,t}
      |∂ h_st,0^(j,k)/∂ z_s-j,t-k|
         ≤ C(C'/α)^2(j+k)
      |∂ h_st,1^(j,k)/∂ x_s-j,t-k|
         ≤ C(C'/α)^2(j+k+1)
  
  and all other partial derivatives are 0. Multiplying by κ_2(j+k+1) and
  κ_2(j+k+2), applying the bound |κ_2j| ≤ C^2j from
  Proposition <ref>(b), and summing over j,k ≥ 0, we
  obtain
  
      |∂ h_st^Σ/∂ x_pp|,
      |∂ h_st^Σ/∂ z_pp|
         ≤ C({p ≤ s}(C'/α)^2(s-p)
      +{p ≤ t}(C'/α)^2(t-p))
      |∂ h_st^Σ/∂ y_pp|,
      |∂ h_st^Σ/∂ w_pp|
         ≤ C({p ≤ s}(C'/α)^2(s-p+1)
      +{p ≤ t}(C'/α)^2(s-p+1))
      |∂ h_st^Σ/∂ z_pq|
         ≤ C ·{p ≤ s  and  q ≤
      t}(C'/α)^2(s-p+t-q)
      |∂ h_st^Σ/∂ x_pq|
         ≤ C ·{p ≤ s  and  q ≤
      t}(C'/α)^2(s-p+t-q+1).
  
  Then applying the same argument as in Lemma <ref>(a),
  we obtain
  
      h^Σ(,,,)-h^Σ(',',',')_ζ≤ C(-'_ζ+-')_ζ+(C/α^2)(-'_ζ
      +-'_ζ).
  
  The proof for h^Ω is analogous, and part (a) follows.
  
  Part (b) is a direct consequence of Lemma <ref>(b), since
  h^Δ(,,,) is a function only of (,) that has the same form
  as h^Δ(,) in Lemma <ref>(b), and similarly for
  h^Γ.
  
  
  The next lemma now follows from Lemma <ref> via
  the same argument as Lemma <ref>, and we omit the proof for
  brevity.
  
  
  In the setting of Theorem <ref>(b), there exist constants 
  C,α_0>0 such that for all α>α_0 and T ≥ 1:
  
  
    * Each entry of _T, _T, _T, and _T
  belongs respectively to I_Δ, I_Σ, I_Γ, and I_Ω,
  except for entries in the first row or column of _T and _T
  which belong to [-1,1] and [-3γκ_2/2,3γκ_2/2].
  
    * For any (,,,) ∈_I_Δ×_I_Σ×_I_Γ×_I_Ω, we have
  
      _T-h^Ω(,,,)_ζ   ≤ C(-_T_ζ+-_T-1_ζ)
         +(C/α^2)(-_T-1_ζ
      +-_T-1_ζ)+Cζ^T/2
      _T-h^Γ(,,,)_ζ   ≤ (C/α)-_T_ζ
      +(C/α^2)-_T_ζ+Cζ^T
      _T-h^Σ(,,,)_ζ   ≤ C(-_T_ζ+-_T_ζ)
         +(C/α^2)(-_T-1_ζ
      +-_T_ζ)+Cζ^T/2
      _T+1-h^Δ(,,,)_ζ   ≤ (C/α)-_T_ζ
      +(C/α^2)-_T_ζ+Cζ^T
  
  
  
  
  
  Given Theorem <ref>(a), it suffices to show that
  δ_TT→Δ_* and γ_TT→Γ_* as T →∞, where (Δ_*,Σ_*,Γ_*,Ω_*,X_*)
  ∈ I_Δ× I_Σ× I_Γ× I_Ω× is
  the unique fixed point of (<ref>).
  
  We define the map G:_I_Δ×_I_Σ×_I_Γ×_I_Ω→_I_Δ×_I_Σ×_I_Γ×_I_Ω as the successive
  composition of the four maps
  
      (,,,)    ↦ (,,,h^Ω(,,,))
      
      (,,,)    ↦ (,,h^Γ(,,,),)
      
      (,,,)    ↦ (,h^Σ(,,,),,)
      
      (,,,)    ↦ (h^Δ(,,,),,,)
  
  Writing its components as G=(G_x,G_y,G_z,G_w),
  Lemma <ref> may be applied to show that
  
      G_w(,,,)-G_w(',',',')_ζ   ≤ C-'_ζ+C-'_ζ
      +C/α^2-'_ζ+C/α^2-'_ζ
      G_z(,,,)-G_z(',',',')_ζ   ≤C/α-'_ζ+C/α^2-'_ζ
      +C/α^4-'_ζ+C/α^4-'_ζ
      G_y(,,,)-G_y(',',',')_ζ   ≤ C-'_ζ+C/α^2-'_ζ
      +C/α^2-'_ζ+C/α^4-'_ζ
      G_x(,,,)-G_x(',',',')_ζ   ≤C/α^2-'_ζ+C/α^3-'_ζ
      +C/α^4-'_ζ+C/α^5-'_ζ.
  
  Then defining the norm ·_ζ,α on _I_Δ×_I_Σ×_I_Γ×_I_Ω by
  
      (,,,)_ζ,α=_ζ+(1/α)_ζ
      +(1/α^2)_ζ+(1/α^3)_ζ,
  
  we obtain
  
      G(,,,)-G(',',',')_ζ,α   ≤ (C/α^2)(,,,)-(',',',')_ζ,α
         ≤τ(,,,)-(',',',')_ζ,α
  
  for some τ∈ (0,1) and α>α_0 sufficiently large. So G
  admits a unique fixed point (_*,_*,_*,_*) ∈_I_Δ×_I_Σ×_I_Γ×_I_Ω.
  
  By the same argument as in the proof of Theorem <ref>(b) for
  the symmetric square setting, this fixed point
  (_*,_*,_*,_*) must be equal to scalar constants
  (Δ_*,Σ_*,Γ_*,Ω_*) in every coordinate,
  where these constants satisfy
  
      Σ_*   =∑_j,k=0^∞(α^2 Δ_*Γ_*(1-Δ_*)(1-Γ_*)/γΣ_*Ω_*)^j+k(κ_2(j+k+1)Γ_*+κ_2(j+k+2)α^2 Δ_*^3(1-Γ_*)^2/Ω_*^2)
      Ω_*   =∑_j,k=0^∞(α^2 Δ_*Γ_*(1-Δ_*)(1-Γ_*)/γΣ_*Ω_*)^j+k(γκ_2(j+k+1)Δ_*+γκ_2(j+k+2)α^2 Γ_*^3(1-Δ_*)^2/γΣ_*^2)
      Δ_*   =1-(α^2Γ_*^2/γ^2Σ_*),
          Γ_*=1-(α^2Δ_*^2/Ω_*).
  
  (The fixed point (Δ_*,Σ_*,Γ_*,Ω_*) to these equations
  exists by the Banach fixed point theorem specialized to the scalar setting.)
  Writing
  
      R'(x)   =∑_k=1^∞κ_2k· kx^k-1
      =∑_k=0^∞κ_2(k+1)· (k+1)x^k
      
      S(x)   =(R(x)/x)'
      =∑_k=2^∞κ_2k· (k-1)x^k-2
      =∑_k=0^∞κ_2(k+2)· (k+1)x^k,
  
  we see that the above equations are equivalent to the fixed point equations
  (<ref>). The proof is concluded using the same
  contractive mapping argument as in Theorem <ref>(b).
  
  
  
  
   §.§ Verification of
  Eq. (<ref>)
  
  
  Denote T(z)=(1+z)(1+γ z).
  Let us first show that the values Δ_ and Γ_ in
  (<ref>–<ref>) may be written equivalently as
  
      Δ_=T(R(x))-xT'(R(x))R'(x)/1+γ R(x),
          Γ_=T(R(x))-xT'(R(x))R'(x)/1+R(x).
  
  To see this, let us define φ(z), φ̅(z), and D(z) as in
  (<ref>), and define also
  
      M(z)=∑_k=1^∞ m_2k^∞ z^k.
  
  From <cit.>,
  the rectangular R-transform is given by R(z)=U(z(D^-1(z))^2-1),
  where U(z) is a function defined such that T(U(z-1))=z. Thus
  T(R(z))=z(D^-1(z))^2, and differentiating on both sides yields
  
      T(R(z))-zT'(R(z))R'(z)=-2z^2D^-1(z)/D'(D^-1(z)).
  
  Next, applying series expansions for φ(z) and φ̅(z) 
  and substituting into D(z), we obtain
  
      φ(z)=z^-1(1+M(z^-2)),
          φ̅(z)=z^-1(1+γ M(z^-2)),
           D(z)=z^-2T(M(z^-2)).
  
  By (<ref>), the rectangular
  R-transform satisfies the identity M(z)=R(zT(M(z))).
  Then M(z^-2)=R(D(z)), so
  
      D(z)=z^-1(1+R(D(z))) ·φ̅(z)
      =φ(z) · z^-1(1+γ R(D(z))).
  
  Hence, applying this with D^-1(z) in place of z and rearranging,
  
      z/1+R(z)=φ̅(D^-1(z))/D^-1(z)),
          z/1+γ R(z)=φ(D^-1(z))/D^-1(z).
  
  Applying these identities (<ref>) and (<ref>) for
  z=x, we see that (<ref>) coincides with the definitions
  (<ref>–<ref>), as desired. 
  
  Now we proceed to verify (<ref>).
  As in Remark <ref>, applying the mmse
  inequality (<ref>) to the second and third
  fixed point equations of (<ref>) and rearranging, we obtain
  
      Σ_* ≥Σ_≡α^2Γ_*^2(1-Δ_*)/γ^2Δ_*,     Ω_* ≥Ω_≡α^2Δ_*^2(1-Γ_*)/Γ_*.
  
  We apply the following argument to “substitute” these inequalities into the
  remaining fixed-point equations: Fixing Δ_* ∈ I_Δ and Γ_*
  ∈ I_Γ, denote
  
      X(Σ,Ω)   =α^2 Δ_*Γ_*(1-Δ_*)(1-Γ_*)/γΣΩ
      
      f(Σ,Ω)   =Δ_* R'(X(Σ,Ω))
      +α^2Δ_*^4(1-Γ_*)^2/Γ_*Ω^2S(X(Σ,Ω)),
      
      g(Σ,Ω)   =γΓ_*R'(X(Σ,Ω))
      +α^2Γ_*^4(1-Δ_*)^2/γΔ_*Σ^2
      S(X(Σ,Ω)).
  
  The fourth and fifth fixed point equations of (<ref>) may
  be written as (Δ_*/Γ_*)Σ_*=f(Σ_*,Ω_*) and
  (Γ_*/Δ_*)Ω_*=g(Σ_*,Ω_*). So for any constant
  η∈, (Σ_*,Ω_*) solves the equation
  
      0=f(Σ,Ω)+η· g(Σ,Ω)
      -Δ_*/Γ_*Σ-η·Γ_*/Δ_*Ω.
  
  
  Let us denote
  
      x=X(Σ_,Ω_)=γ/α^2
  
  and pick this constant η to solve the linear equation
  
      (ηα^2-R'(x)-ηγ^3/α^2S(x))
      (1+R(x))=(α^2/γ^2-ηγ R'(x)
      -1/α^2S(x))(1+γ R(x)).
  
  Note that for all α>α_0 sufficiently large, we have η≈
  [(1+γ R(x))/γ^2]/(1+R(x)) ≈ 1/γ^2, which is of constant
  order. We claim that for any Δ_* ∈ I_Δ and
  Γ_* ∈ I_Γ, the right side of (<ref>) is decreasing
  as a function of Σ∈ [Σ_,∞) and Ω∈
  [Ω_,∞).
  To see this, observe first that since 1-Δ_* ≤ C/α^2 and
  1-Γ_* ≤ C/α^2, we have |X(Σ,Ω)| ≤ C/α^2 for
  parameters in these domains. Then to compute the derivatives of
  f(Σ,Ω) and g(Σ,Ω), we may apply the series expansions
  
      R'(X(Σ,Ω))   =κ_2^∞+2κ_4^∞·α^2 Δ_*Γ_*(1-Δ_*)(1-Γ_*)/γΣΩ+…
      
      S(X(Σ,Ω))   =κ_4^∞+2κ_6^∞·α^2 Δ_*Γ_*(1-Δ_*)(1-Γ_*)/γΣΩ+…
  
  which are convergent for α>α_0 sufficiently large, and
  differentiate these term-by-term. We may thus verify the bounds
  
      |∂_Σ R'(X(Σ,Ω))|,
      |∂_Ω R'(X(Σ,Ω))|,
      |∂_Σ S(X(Σ,Ω))|,
      |∂_Ω S(X(Σ,Ω))| ≤C/α^2,     
      |S(X(Σ,Ω))| ≤ C,
  
  which imply
  
      |∂_Σ f(Σ,Ω)|,|∂_Ω f(Σ,Ω)|,
      |∂_Σ g(Σ,Ω)|,|∂_Ω g(Σ,Ω)|
      ≤C/α^2.
  
  Then the derivatives in (Σ,Ω) of the right
  side of (<ref>) are negative for all α>α_0
  sufficiently large, yielding the desired monotonicity.
  
  Since (Σ_*,Ω_*) satisfies (<ref>) with equality, we
  may then substitute (<ref>) to obtain
  
      0 ≤ f(Σ_,Ω_)+η· g(Σ_,Ω_)
      -Δ_*/Γ_*Σ_-η·Γ_*/Δ_*Ω_.
  
  Applying the forms of Σ_, Ω_, f, and g and
  rearranging, we arrive at
  
      (ηα^2-R'(x)-ηγ^3/α^2S(x))Δ_*
      +(α^2/γ^2-ηγ R'(x)-1/α^2S(x)
      )Γ_*
      ≤(α^2/γ^2+ηα^2)Δ_*Γ_*.
  
  Now applying the identity (<ref>), we may write this as
  
      A(x)((1+γ R(x))Δ_*+(1+R(x))Γ_*) ≤ 
      B(x)Δ_*Γ_*.
  
  Here, solving explicitly the equation (<ref>) for η
  and applying also S(x)=R'(x)/x-R(x)/x^2, these values A(x) and B(x)
  may be computed after some algebraic simplification to be
  
      A(x)   =(1+R(x))(1+γ R(x))-x(1+γ+2γ R(x))R'(x)/γ x[(1+R(x))(1+γ R(x))+xR'(x)(1-γ)]
      
      B(x)   =2(1+R(x))(1+γ R(x))/γ x[(1+R(x))(1+γ
      R(x))+xR'(x)(1-γ)].
  
  Note that for α>α_0 sufficiently large (and hence small
  x=γ/α^2), the numerators and denominators of A(x) and B(x) are
  all positive. Then clearing the denominators of A(x) and B(x) in
  (<ref>) and applying to the left side
  
      (1+γ R(x))Δ_*+(1+R(x))Γ_*
      ≥ 2√((1+γ R(x))(1+R(x))Δ_*Γ_*),
  
  we obtain
  
      √((1+R(x))(1+γ R(x))Δ_*Γ_*)≥
      (1+R(x))(1+γ R(x))-x(1+γ+2γ R(x))R'(x).
  
  Recalling the notation T(z)=(1+z)(1+γ z), this may be rewritten as
  
      Δ_*Γ_* ≥[T(R(x))-xT'(R(x))R'(x)]^2/T(R(x)),
  
  where the right side coincides with Δ_Γ_ by
  (<ref>). This establishes (<ref>).
  
  The inequalities in the preceding argument stem from (<ref>)
  and (<ref>). If both U_* ∼(0,1) and V_* ∼(0,1),
  then (<ref>) holds with equality. In this case, we have
  (Δ_*/Γ_*)Σ_=f(Σ_,Ω_) and
  (Γ_*/Δ_*)Ω_=g(Σ_,Ω_) in the preceding
  argument, and these two equations may be solved to yield Δ_*=Δ_
  and Γ_*=Γ_. Then equality holds in (<ref>).
  (Note that equality also holds in (<ref>) because
  (1+γ R(x))Δ_=(1+R(x))Γ_ by
  (<ref>).) Conversely, if either U_* or V_* is not
  distributed as (0,1), then at least one of the inequalities in
  (<ref>) is strict. Then the inequality
  (<ref>) is also strict, implying that
  (<ref>) holds with strict inequality as well.
  
  
  
  § REMOVING THE NON-DEGENERACY ASSUMPTION
  
  
  In this appendix, we prove Corollary <ref>.
  We follow a similar approach to <cit.> and
  construct a perturbed version of the AMP sequence:
  Let ,_1,_2,…∈^n be random vectors independent of each
  other and of all other quantities, where  has i.i.d.Uniform(-1,1) entries and each _t has
  i.i.d. (0,1) entries. For a fixed small parameter >0, consider the
  perturbed noise matrix
  
      ^=Ø^⊤(+)Ø,
  
  the perturbed initialization
  
      _̆1^=_̆1+_1,
  
  and the perturbed AMP iterations
  
      _t^   =^_̆t^-b_t1^_̆1^-…
      -b_tt^_̆t^
      _̆t+1^   =u_t+1(_1^,…,_t^,)+_t+1.
  
  We define
  _t^,_t^,_t^,_t^ by (<ref>)
  and (<ref>) using this perturbed sequence,
  and the above coefficients
  (b_t1^,…,b_tt^) are the last column of _t^.
  
  Note that for any fixed >0 and up to any fixed iteration T, these
  perturbed iterations are an example of the general iterations
  (<ref>–<ref>) applied with noise matrix ^, by defining
  the augmented side-information matrix ^=(,_2,…,_T+1) and
  considering the functions
  
      u_t+1^(_1,…,_t,^)
      =u_t+1(_1,…,_t,)+_t+1.
  
  By Propositions <ref> and <ref>, we have
  
      +Λ^≡Λ+Γ,   
      (_̆1^,,_2,…,_T+1)  (U_1+ W_1,E,W_2,…,W_T+1),
  
  where Γ∼Uniform(-1,1) is independent of Λ and
  (W_1,…,W_T+1) ∼(0,) is independent of (U_1,E).
  It is then clear that Assumption <ref> including part (e) holds
  for this perturbed sequence, so Lemma <ref> applies.
  
  Define the almost-sure limits
  
      (_t^,∞,_t^,∞,_t^,∞,_t^,∞)=lim_n
      →∞ (_t^,_t^,_t^,_t^),
  
  as guaranteed by Lemma <ref>.
  We let _̆1,_1,_̆2,_2,… continue to denote the original
  AMP sequence. We now establish inductively the following
  two claims, almost surely for each t=1,2,3,…, where the second claim
  implies the corollary:
  
  
    * 
      lim sup_n →∞ n^-1_t-1^2<∞,
          lim_→ 0 lim sup_n →∞ n^-1_t-1^-_t-1^2=0,
  
  
      lim sup_n →∞ n^-1_̆t^2<∞,
          lim_→ 0 lim sup_n →∞ n^-1_̆t^-_̆t^2=0.
  
  
    * (_̆1,…,_̆t,_1,…,_t-1,) 
  (U_1,…,U_t,Z_1,…,Z_t-1,E). The deterministic limits
  
      (_t^∞,_t^∞,_t^∞,_t^∞)=lim_n
      →∞ (_t,_t,_t,_t)
  
  all exist, where
  (_t^∞,_t^∞,_t^∞,_t^∞)
  =lim_→ 0
  (_t^,∞,_t^,∞,_t^,∞,_t^,∞).
  
  
  Let t^(a),t^(b) denote these claims up to and including iteration t.
  We induct on t.
  
  For 1^(a), we have n^-1_̆1^2 →[U_1^2]<∞ by Assumption
  <ref>(c). We also have _̆1^-_̆1=_1
  and n^-1_1^2 → 1. Thus
  lim_→ 0lim sup_n →∞ n^-1_̆1^-_̆1^2=0.
  
  For 1^(b), we have (_̆1,)  (U_1,E)
  and _1 →[U_1^2] also by Assumption <ref>(c).
  Since _1=0 and κ_k →κ_k^∞ (the k^th free
  cumulant of Λ) for each k ≥ 1, this shows the existence of all four
  limits _1^∞,_1^∞,_1^∞,_1^∞.
  Note that _1^,∞=[U_1^2]+^2, so that
  _1^,∞→_1=[U_1^2] as → 0.
  Letting κ_k^,∞ be the
  free cumulants of Λ^, note that the moments of Λ^
  converge to those of Λ as → 0, so also
  κ_k^,∞→κ_k^∞. Since
  _1^,∞=0=_1, this shows the last statement of 1^(b).
  
  Suppose now that t^(a) and t^(b) hold. To show t+1^(a), observe that
  
      _t≤_̆t+∑_s=1^t |b_ts| _̆s.
  
  Applying =_∞,
  lim sup_n →∞_∞<∞,
  and lim sup_n →∞ n^-1_̆s^2<∞
  and lim_n →∞ |b_ts|=|b_ts^∞| by t^(a) and t^(b),
  this shows
  
      lim sup_n →∞ n^-1_t^2<∞.
  
  Now comparing (<ref>) with (<ref>),
  
      _t^-_t≤^-_̆t^+_̆t^-_̆t
      +∑_s=1^t |b_ts^-b_ts|_̆s^+|b_ts|
      _̆s^-_̆s.
  
  Applying also ^-≤,
  lim_→ 0 |b_ts^|=|b_ts|, and
  lim_→ 0lim sup_n →∞ n^-1_̆s^-_̆s^2=0
  by t^(a) and t^(b), this shows
  
      lim_→ 0 lim sup_n →∞
      n^-1_t^-_t^2=0.
  
  For _̆t+1, we have
  
      n^-1∑_i=1^n (u_t+1(z_i1,…,z_it,E)
      -u_t+1(0,…,0,E))^2 ≤ Cn^-1(_1^2+…+_t^2)
  
  by the Lipschitz assumption for u_t+1. Then applying t^(a) to bound the
  right side,
  
      lim sup_n →∞ n^-1_̆t+1^2<∞.
  
  Now comparing (<ref>) and (<ref>),
  
      n^-1_̆t+1^-_̆t+1^2
         ≤ 2^2 · n^-1_t+1^2+2n^-1u_t+1(_1,…,_t,)-u_t+1(_1^,…,_t^,)^2
         ≤ 2^2 · n^-1_t+1^2
      +2C∑_s=1^t n^-1_s^-_s^2.
  
  Then applying t^(a) and (<ref>) to bound the right side, we obtain
  
      lim_→ 0 lim sup_n →∞
      n^-1_̆t+1^-_̆t+1^2=0.
  
  This shows t+1^(a).
  
  For t+1^(b), let
  
      x_i   =(_̆1,…,_̆t+1,_1,…,_t,)_i
      
      x_i^   =(_̆1^,…,_̆t+1^,_1^,…,_t^,)_i
  
  be the i^th rows of these matrices.
  Let X=(U_1,…,U_t+1,Z_1,…,Z_t,E) be the limit to be shown, and
  let X^ be the limit of the perturbed sequence.
  To show the desired W_2 convergence, it suffices to check the convergence
  
      1/n∑_i=1^n f(x_i) →[f(X)]
  
  for all Lipschitz functions f(x) and for f(x)=x^2. Let us write
  
      lim sup_n →∞|1/n∑_i=1^n f(x_i)-[f(X)]|
         ≤lim_→ 0lim sup_n →∞(|1/n∑_i=1^n f(x_i)-f(x_i^)|
      +|1/n∑_i=1^n f(x_i^)-[f(X^)]|
      +|[f(X^)]-[f(X)]|).
  
  For the first term of (<ref>), note that any such 
  function f satisfies the pseudo-Lipschitz condition
  
      |f(x)-f(x')| ≤ C(1+x+x')x-x'
  
  for some constant C>0. Then by this and Cauchy-Schwarz,
  
      |1/n∑_i=1^n f(x_i)-f(x_i^)|
         ≤C/n∑_i=1^n (1+x_i+x_i^)x_i-x_i^
         ≤ C'(1/n∑_i=1^n (1+x_i^2+x_i^^2))^1/2(1/n∑_i=1^n x_i-x_i^^2)^1/2.
  
  Recalling the definitions of x_i and
  x_i^ in (<ref>) and (<ref>) and
  applying t+1^(a), this term converges to 0
  in the limits n →∞ followed by → 0.
  The second term of (<ref>) converges to 0 as n →∞ for any
  fixed >0, by Lemma <ref>. For the third term of
  (<ref>), note that as → 0, we have 
  
      U_1^→ U_1,      (Z_1^,…,Z_t^) → (Z_1,…,Z_t)
  
  in the Wasserstein space W_2, where the second convergence follows from
  _t^-_t→ 0 in
  t^(b). Since the functions u_2,…,u_t+1 are Lipschitz, this implies
  X^→ X in W_2, so [f(X^)] →[f(X)].
  Combining these establishes (<ref>), and hence
  
      (_̆1,…,_̆t+1,_1,…,_t,) 
      (U_1,…,U_t+1,Z_1,…,Z_t,E).
  
  This implies the existence of the limits _t+1^∞.
  Each function u_s is Lipschitz and continuously-differentiable by assumption,
  so each partial derivative ∂_s' u_s is bounded and continuous.
  Then this also implies the existence of _t+1^∞, and hence of
  _t+1^∞ and _t+1^∞. As → 0,
  since X^→ X in W_2 as shown above, we also have
  _t+1^,∞→_t+1^∞ and
  _t+1^,∞→_t+1^∞, and hence
  _t+1^,∞→_t+1^∞ and
  _t+1^,∞→_t+1^∞. This concludes the
  proof of t+1^(b).
  
  
  
  § PROPERTIES OF EMPIRICAL WASSERSTEIN
  CONVERGENCE
  
  
  In this appendix, we collect several properties of the notation  and
   for empirical Wasserstein convergence from Section
  <ref>.
  
  We will use below the following fact: To verify ℒ where ∈^n × k, it suffices to check that (<ref>) holds for every
  function f:^k → satisfying, for some constant C>0,
  the pseudo-Lipschitz condition
  
      |f(v)-f(v')| ≤ C(1+v^p-1+v'^p-1)v-v'.
  
  This is because by <cit.>, it suffices to
  check (<ref>) for f(v)=v^p, together with the usual
  weak convergence which is equivalent to (<ref>) holding for bounded
  Lipschitz functions. Note that this condition (<ref>)
  implies the polynomial growth condition (<ref>).
  
  
  Fix any p ≥ 1, t ≥ 1, and k ≥ 0. Let ∈^n × k
  be a deterministic matrix satisfying E, and let
  ∈^n × t be random with i.i.d. rows equal in law to
  V ∈^t, where [V^p]<∞. Then the joint convergence
  
      (,)  (V,E)
  
  holds almost surely, where V is independent of E in the limit (V,E).
  
  
  For k=0, the result V follows from the strong law of large
  numbers applied to any function f satisfying
  (<ref>), as [|f(V)|] ≤ C(1+[V^p])<∞.
  
  For k>0, we proceed by approximating E with a discrete random variable, and
  then applying the law of large numbers for each discrete value of E. In
  detail: Fix any function f satisfying (<ref>), and fix
  any ∈ (0,1). Let {(v_i,e_i)}_i=1^n be the rows of (,).
  Since (<ref>) implies (<ref>),
  for any R>0 we have
  
      1/n∑_i:e_i>R |f(v_i,e_i)|
      ≤C/n∑_i:e_i>R (1+v_i^p+e_i^p).
  
  Note that as E, the uniform integrability condition of
  <cit.> shows
  
      lim_R →∞lim sup_n →∞1/n∑_i:e_i>R 1 ≤lim_R →∞lim sup_n →∞1/n∑_i:e_i>Re_i^p=0.
  
  This bounds the first and third terms on the right side of
  (<ref>). For the middle term, we consider two cases.
  If V≤ K almost surely for some K>0, then by this and the convergence
  V,
  
      lim_n →∞1/n∑_i=1^n min(v_i,2K)^p
      =[min(V,2K)^p]=[V^p]=
      lim_n →∞1/n∑_i=1^n v_i^p.
  
  Applying also |{i:v_i>2K}|/n → 0, this implies
  
      lim_n →∞1/n∑_i:v_i>2Kv_i^p
      =lim_n →∞(1/n∑_i=1^n (v_i^p
      -min(v_i,2K)^p)+(2K)^p ·|{i:v_i>2K}|/n)=0.
  
  In this case, we may bound
  
      lim_R →∞lim sup_n →∞1/n∑_i:e_i>Rv_i^p ≤lim_R →∞lim sup_n →∞1/n(∑_i:e_i>R (2K)^p+∑_i:v_i>2Kv_i^p)=0.
  
  Conversely, if the support of V is unbounded, then let
  v_(1)≥…≥v_(n) be the ordered values of
  {v_i}_i=1^n. Note that for each R>0, we have
  |{i:e_i>R}|/n →δ(R) for some δ(R) ≥ 0, where
  δ(R) → 0 as R →∞. Then
  
      lim_R →∞lim sup_n →∞1/n∑_i:e_i>Rv_i^p ≤lim_δ→ 0lim sup_n →∞1/n∑_i=1^δ nv_(i)^p.
  
  Now applying V and the corresponding uniform
  integrability condition for ,
  
      lim_δ→ 0lim sup_n →∞1/n∑_i=1^δ nv_(i)^p
      =lim_R →∞lim sup_n →∞1/n∑_i:v_i>Rv_i^p=0.
  
  Combining the above and applying this to (<ref>),
  
      lim_R →∞lim sup_n →∞1/n∑_i:e_i>R |f(v_i,e_i)|=0.
  
  So we may pick a bounded set ℬ⊂^k large enough
  such that
  
      lim sup_n →∞1/n∑_i:e_i ∉ℬ |f(v_i,e_i)|<.
  
  Applying also
  
      [|f(V,E)| ·{E ∉ℬ}]
      ≤[C(1+V^p+E^p){E ∉ℬ}]
  
  and the integrability of V^p and E^p,
  we may pick ℬ large enough such that
  
      [|f(V,E)| ·{E ∉ℬ}]<.
  
  
  Now let {U_α}_α=1^M be any finite partition of ℬ
  such that each set U_α has diameter at most , and the boundary of
  U_α has probability 0 under the law of E.
  (For example, take ℬ=[-K,K]^k to be a hyperrectangle in
  ^k, and construct this partition by dividing [-K,K] along each axis
  into small enough intervals. Take -K, K, and these
  interval boundaries to have probability 0
  under the univariate marginal distribution of each coordinate of E.) Pick a
  point u_α∈ U_α for each α=1,…,M. For each e ∈ℬ, define u(e)=u_α where α is the index such that
  e ∈ U_α. Then applying (<ref>) and u(e)^p-1≤ C(e^p-1+1),
  
      1/n∑_i:e_i ∈ℬ |f(v_i,e_i)-f(v_i,u(e_i))|
      ≤C/n∑_i=1^n
      (1+v_i^p-1+e_i^p-1) ·
  
  for a constant C>0 independent of . Since V
  and E, this yields
  
      lim sup_n →∞1/n∑_i:e_i ∈ℬ
      |f(v_i,e_i)-f(v_i,u(e_i))| ≤ C'.
  
  Similarly,
  
      [|f(V,E)-f(V,u(E))|
      ·{E ∈ℬ}] ≤ C'.
  
  
  Finally, let us write
  
      1/n∑_i:e_i ∈ℬ f(v_i,u(e_i))
      =∑_α=1^M 1/n∑_i:e_i ∈ U_α f(v_i,u_α).
  
  Observe that for each fixed α=1,…,M,
  since the boundary of U_α has probability 0 under E,
  by weak convergence
  we have |{i:e_i ∈ U_α}|/n →[E ∈ U_α].
  Then by the law of large numbers applied to the function f(·,u_α),
  almost surely
  
      1/n∑_i:e_i ∈ U_α
      f(v_i,u_α) →[E ∈ U_α] ·[f(V,u_α)].
  
  Summing over α=1,…,M and applying the independence of V and E,
  
      1/n∑_i:e_i ∈ℬ f(v_i,u(e_i))
         →∑_α=1^M [E ∈ U_α] ·[f(V,u_α)]
         =∑_α=1^M [f(V,u_α) ·{E ∈ U_α}]
      =[f(V,u(E)) ·{E ∈ℬ}].
  
  Combining (<ref>), (<ref>), (<ref>), (<ref>),
  and (<ref>), we obtain
  
      lim sup_n →∞|1/n∑_i=1^n f(v_i,e_i)
      -[f(V,E)]| ≤ C
  
  for a constant C>0 independent of . As this holds for all >0, this
  shows n^-1∑_i=1^n f(v_i,e_i) →[f(V,E)], which concludes the
  proof.
  
  
  
  Fix p,p' ≥ 1 and k,ℓ≥ 1.
  If ∈^n × k satisfies W_p+p'→ V, and
  g:^k →^ℓ is any continuous function satisfying
  g(v)≤ C(1+v^p') for some C>0 and all v ∈^k,
  then g()  g(V).
  
  
  This follows from Definition <ref>, since for any continuous function
  f:^ℓ→ satisfying (<ref>) for the order
  p, the composition f ∘ g:^k → is continuous and
  satisfies (<ref>) for the order p+p'.
  
  
  
  Fix p ≥ 1 and k ≥ 0. Suppose ∈^n × k satisfies
  V, and f:^k → is a function satisfying
  (<ref>) that is continuous everywhere except on a set having
  probability 0 under the law of V. Then
  
      1/n∑_i=1^n f()_i →[f(V)].
  
  
  
  Let f be such a function. For any M>0, consider the bounded function
  f^M(v)=max(-M,min(f(v),M)). Let v_i be the i^th row of .
  The condition V implies the usual weak
  convergence of the empirical distribution of {v_i}_i=1^n to V,
  so n^-1∑_i=1^n f^M(v_i) →[f^M(V)] even when f^M is
  discontinuous on a set of probability 0 under V. Now taking M →∞,
  we have [f^M(V)] →[f(V)] by the bound
  |f^M(v)| ≤ C(1+v^p) and the dominated convergence theorem.
  By this bound, we also have
  
      lim_M →∞lim sup_n →∞1/n∑_i=1^n
      |f^M(v_i)-f(v_i)|
         ≤lim_M →∞lim sup_n →∞1/n∑_i:|f(v_i)|>M
      |f(v_i)|
         ≤lim_R →∞lim sup_n →∞1/n∑_i:v_i>R C(1+v_i^p)=0,
  
  where the last limit is 0 by <cit.>.
  Then n^-1∑_i=1^n f(v_i) →[f(V)] as desired.
  
  
  
  Fix p ≥ 1 and k,ℓ≥ 1.
  If ∈^n × k, ∈^n ×ℓ, and
  _n,∈^k ×ℓ satisfy
  (V_1,…,V_k), 0, and _n →
  as n →∞, then
  
      _n+[ V_1   … V_k ].
  
  
  
  Let f:^ℓ→ satisfy (<ref>). Then
  (v_1 ⋯ v_k) ↦
  f((v_1 ⋯ v_k)) is continuous and satisfies
  (<ref>) with the order p, so by the convergence
  (V_1 ⋯ V_k), we have
  
      1/n∑_i=1^n f()_i →[f((V_1 ⋯ V_k))].
  
  
  Let v_i,w_i be the i^th rows of  and .
  Note that (V_1 ⋯ V_k) implies
  n^-1∑_i=1^n v_i^p →[(V_1 ⋯ V_k)^p]<∞.
  Similarly n^-1∑_i=1^n w_i^p → 0.
  Then applying Jensen's inequality, Holder's inequality,
  and the bound v_i+w_i^p-1≤ C(v_i^p-1+w_i^p-1),
  we have for some constants C,C'>0 depending on  that
  
      1/n∑_i=1^n |f()_i-f(+)_i|
         ≤C/n∑_i=1^n (1+v_i^p-1+v_i+w_i^p-1)w_i
         ≤ C' ·1/n(w_i+v_i^p-1w_i
      +w_i^p)
         ≤ C'[(1/n∑_i=1^n w_i^p)^1/p
      +(1/n∑_i=1^n v_i^p)^(p-1)/p(1/n∑_i=1^n w_i^p)^1/p
      +(1/n∑_i=1^n w_i^p)] → 0.
  
  Similarly,
  
      1/n∑_i=1^n |f(_n+)_i-f(+)_i|
         ≤C/n∑_i=1^n
      (1+v_i_n+w_i^p-1+v_i+w_i^p-1) ·v_i(_n-)
         ≤ C'_n-·1/n∑_i=1^n
      (v_i+v_i^p+w_i^p-1v_i) → 0.
  
  Combining the above yields the proposition.
  
  
  The following is an empirical form of Stein's lemma.
  
  
  Fix p ≥ 2.
  Suppose (_1,…,_t,) ∈^n × (t+k) are such that
  
      (_1,…,_t,)  (Z_1,…,Z_t,E)
  
  where, for some non-singular covariance matrix Σ∈^t × t,
  (Z_1,…,Z_t) ∼(0,Σ) and this
  is independent of E. Suppose u:^t+k→ is weakly
  differentiable in its first t arguments and satisfies (<ref>)
  for the order p-1.
  Then, almost surely as n →∞,
  
      1/n[ _1^⊤;    ⋮; _t^⊤ ]
      u(_1,…,_t,)
      →Σ·[ [∂_1 u(Z_1,…,Z_t,E)];                    ⋮; [∂_t u(Z_1,…,Z_t,E)] ].
  
  
  
  Note that for each s=1,…,t, the function (z_1,…,z_t,e)
  ↦ z_su(z_1,…,z_t,e) is continuous and
  satisfies (<ref>) with order p, so
  
      1/n(_1,…,_t)^⊤
      u(_1,…,_t,) →[(Z_1,…,Z_t) ·
      u(Z_1,…,Z_t,E)].
  
  To show that the right side of (<ref>) is equivalent to the
  given expression, we apply Stein's lemma:
  Let us condition on a realization E=e for any fixed e ∈^k,
  and denote Z=(Z_1,…,Z_t). We may write
  Z=Σ^1/2X where X ∼(0,), and define
  
      v_e(x)=u(Σ^1/2x,e).
  
  Since Σ is non-singular, the maps
  X ↦Σ^1/2X and Z ↦Σ^-1/2Z are both Lipschitz.
  Then by the chain rule for weak differentiability under bi-Lipschitzian
  changes of coordinates, see <cit.>,
  v_e(x) is weakly differentiable with
  
      ∇ v_e(x)=Σ^1/2·∇ u(Σ^1/2x,e)
  
  a.e. over x ∈^t.
  (We denote by ∇ (·) the vector of partial derivatives.)
  Applying Stein's lemma for weakly differentiable functions, see <cit.>, we have for each s=1,…,t that
  [X_s v_e(X)]=[∂_s v_e(X)]. Then
  
      [Z · u(Z,e)]=[Σ^1/2 X · v_e(X)]
      =Σ^1/2·[∇ v_e(X)]=Σ·[∇ u(Z,e)].
  
  Taking the expectation over E and applying this to (<ref>)
  concludes the proof.
  
  
  
  
  § AUXILIARY LEMMAS
  
  
  This appendix collects several auxiliary results that were used in the preceding
  arguments.
  
  
  
   §.§ Properties of Haar-orthogonal matrices
  
  
  The following result was established as <cit.>.
  
  
  Fix k ≥ 1, and let ,∈^n × k be deterministic matrices
  with rank k,
  such that = for some orthogonal matrix ∈^n × n. If
  Ø∈^n × n is a random Haar-uniform orthogonal matrix,
  then the law of Ø conditioned on =Ø is equal to the law of
  
      (^⊤)^-1^⊤+_^⊥Ø̃_^⊥^⊤
      =(^⊤)^-1^⊤+_^⊥Ø̃_^⊥^⊤.
  
  Here, Ø̃∈^(n-k) × (n-k) is a Haar-uniform
  orthogonal matrix, and
  _^⊥,_^⊥∈^n × (n-k)
  are matrices with orthonormal columns spanning the orthogonal complements of
  the column spans of  and .
  
  
  
  Fix any p ≥ 1 and k,ℓ≥ 0. Let Ø∈^(n-ℓ) ×
  (n-ℓ) be a
  random Haar-uniform orthogonal matrix. Let
  ∈^n × k and ∈̌^n-ℓ be deterministic
  and satisfy E and n^-1^2 →σ^2,
  and let ∈^n × (n-ℓ) be any deterministic
  matrix with orthonormal columns.
  
  
    * Almost surely as n →∞,
  
      (Ø,̌)  (Z,E)
  
  where Z ∼(0,σ^2) is independent of E.
  
    * Consider a second dimension m such that m,n →∞ simultaneously.
  Fix j ≥ 0, and let ∈^m × j be deterministic and satisfy
  F. Let ∈^m be the first m entries of ΠØ$̌
  ifm ≤n, orΠØ$̌ extended by m-n i.i.d. entries with
  distribution (0,σ^2) if m>n. Then almost
  surely as m,n →∞,
  
      (,)  (Ž,F)
  
  where Ž∼(0,σ^2) is independent of F.
  
  
  
  For part (a), observe that Ø$̌ is a
  random vector uniformly distributed on the sphere of radius. Thus, we
  may introduce a Gaussian vector∼(0,_n ×n)so thatΠ^⊤∼(0,_(n-ℓ) ×(n-ℓ)), 
  and writeØ=̌Π^⊤·/Π^⊤. Then
  
      ΠØ=̌ΠΠ^⊤·/Π^⊤
      =·/Π^⊤-Π^⊥·/Π^⊤
  
  whereΠ^⊥=-ΠΠ^⊤∈^n ×nis a projection onto a subspace of fixed
  dimensionℓ. By Proposition <ref>,
  
      (,)  (Z̃,E)
  
  whereZ̃ ∼(0,1).
  We haven^-1^2 →σ^2by assumption andn^-1Π^⊤^2 →1almost surely, so
  
      /Π^⊤→σ.
  
  We also have the equality in lawΠ^⊥=_̆1w_1+…+_̆ℓw_ℓfor some
  orthonormal unit vectors_̆1,…,_̆ℓ∈^nspanning the range ofΠ^⊥, and forw_1,…,w_ℓiid∼ (0,1).
  Letting{u_ij}_i=1^nbe the entries of_̆j,
  for eachj=1,…,ℓand any fixedp ≥1, we have
  
      1/n∑_i=1^n |u_ij w_j|^p
      ≤ |w_j|^p ·1/√(n)→ 0
  
  almost surely asn →∞. Thus also
  
      1/n∑_i=1^n |(Π^⊥)_i|^p → 0,
  
  soΠ^⊥·/Π^⊤ 0.
  Combining this with (<ref>), (<ref>), and
  (<ref>) and applying
  Proposition <ref>, we obtain part (a).
  
  For part (b), let ∈^mbe the firstmentries ofifm
  ≤n, orextended bym-nadditional(0,1)entries ifm>n.
  Let_̊1 ∈^mbe the firstmentries of^⊥·/Π^⊤ifm ≤n, or this vector extended bym-nadditional 0's ifm>n. Let_̊2 ∈^mbe 0 ifm ≤n, or equal
  to 0 in the firstnentries and equal to ·(/Π^⊤-σ)in the lastm-nentries
  ifm>n. Then we may write
  
      =·/Π^⊤-_̊1-_̊2.
  
  The same argument as in part (a) shows
  
      (,)  (Z̃,F)
  
  whereZ̃ ∼(0,1)is independent ofF, and_̊1 0.
  Whenm>n, we also have
  
      1/m∑_i=n+1^m
      |((/Π^⊤-σ))_i|^p
      ≤|/Π^⊤-σ|^p ·1/m∑_i=1^m |()_i|^p → 0
  
  almost surely. So_̊2 0. Then applying
  Proposition <ref> shows part (b).
  
  
   §.§ Properties of moments and free cumulants
  
  Let Λ be a random variable with finite moments of all orders, such that
  [|Λ|^k] ≤ M^k for some M>0 and all integers k ≥ 1.
  
    *  Let {κ_k}_k ≥ 1 be the free cumulants of Λ. Then
  for all k ≥ 1,
  
      |κ_k| ≤ (16M)^k.
  
  Thus the R-transform of Λ is analytic on the domain |x|<1/(16M),
  where it may be defined by the convergent series
  
      R(x)=∑_k=1^∞κ_k x^k-1.
  
    *  Let {κ_2k}_k ≥ 1 be the rectangular free cumulants of
  Λ with aspect ratio γ. Then for all k ≥ 1,
  
      |κ_2k| ≤max(γ^k,1) · (16M)^2k.
  
  Thus the rectangular R-transform of Λ is analytic on the domain
  |x|<min(γ^-1,1)/(16M)^2, where it may be defined by the convergent
  series
  
      R(x)=∑_k=1^∞κ_2k x^k.
  
  For part (a), the free cumulants may be expressed explicitly
  by Möbius inversion of the
  moment-cumulant relations (<ref>), yielding
  
      κ_k=∑_π∈(k) m_π·μ(π,1_k),
           m_π=∏_S ∈π m_|S|,
  
  where μ(·,·) are the Möbius functions on the non-crossing
  partition lattice and 1_k is the trivial partition consisting of the single
  set {1,…,k}. We have |μ(π,1_k)|
  ≤ 4^k and |(k)| ≤ 4^k—see the proof of <cit.>. Combining with |m_π| ≤ M^k for all π∈(k), part (a) follows.
  
  For part (b), we apply a similar argument in the rectangular probability
  space (𝒜,p_m,p_n,φ_m,φ_n) from which the rectangular
  free cumulants are defined—see <cit.> for
  definitions. Here, p_m,p_n ∈𝒜 are orthogonal projections
  satisfying p_m+p_n=1, and φ_m and φ_n are traces on
  p_m𝒜p_m and p_n𝒜p_n that satisfy φ_m(p_m)=1,
  φ_n(p_n)=1, and γ/(1+γ) ·φ_m(xy)=1/(1+γ)
  ·φ_n(yx) for x ∈ p_m𝒜p_n and y ∈
  p_n𝒜p_m. Let E:𝒜→𝒟 be the conditional
  expectation onto the sub-algebra 𝒟 generated by (p_m,p_n), given
  by E(x)=φ_m(p_mxp_m)p_m+φ_n(p_nxp_n)p_n.
  For k ≥ 1 and partitions π∈(k), let
  κ_π^𝒟 be the 𝒟-valued
  free cumulants defined by the moment-cumulant relations
  
      E(a_1… a_k)=∑_π∈(k)κ_π^𝒟(a_1,…,a_k).
  
  
  If a ∈ p_m𝒜p_n is an element such that
  φ_m((aa^*)^k)=[Λ^2k],
  then the rectangular free cumulant κ_2k of Λ is given by
  
      κ_2k· p_m=κ_1_2k^𝒟(a,a^*,…,a,a^*).
  
  (Compare <cit.> with
  <cit.>, the latter being the definition of
  rectangular free cumulants that we have reviewed in
  Section <ref> and used throughout this work.)
  From the Möbius inversion
  
      κ_1_2k^𝒟(a,a^*,…,a,a^*)
      =∑_π∈(2k) m_π^𝒟(a,a^*,…,a,a^*)
      ·μ(π,1_2k)
  
  where m_π^𝒟 is the 𝒟-valued joint moment function
  associated to π, we obtain
  
      |κ_2k|=
      |φ_m(κ_1_2k^𝒟(a,a^*,…,a,a^*))|
      ≤ 16^2kmax_π∈(2k)|φ_m(m_π^𝒟(a,a^*,…,a,a^*))|.
  
  Here, it may be checked that when
  φ_m(m_π^𝒟(a,a^*,…,a,a^*)) is non-zero, it must be a
  product of φ_m((aa^*)^i_1),…,φ_m((aa^*)^i_a) and
  φ_n((a^*a)^j_1),…,φ_n((a^*a)^j_b)
  where the elements of π have cardinalities
  2i_1,…,2i_a,2j_1,…,2j_b. Then applying
  φ_n((a^*a)^j)=γφ_m((aa^*)^j) and
  φ_m((aa^*)^j)=[Λ^2j] ≤ M^2j, we obtain
  |φ_m(m_π^𝒟(a,a^*,…,a,a^*))| ≤max(γ^k,1)M^2k, which yields part (b).
  
  Let k be a positive integer. Then for any random variable X and any
  sigma-algebra ,
  
      [(X-[X |])^k] ≤ 2^k[|X|^k].
  
  Write as shorthand Y=[X |]. We expand the left side and apply
  Hölder's inequality to obtain
  
      [(X-Y)^k]=∑_j=0^k kj[X^jY^k-j]
      ≤∑_j=0^k kj[|X|^k]^j/k[|Y|^k]^(k-j)/k.
  
  By Jensen's inequality,
  
      [|Y|^k]=[|[X |]|^k] ≤[[|X|^k |]]=[|X|^k],
  
  and the result follows from ∑_j=0^k kj=2^k.
  
  
  § ACKNOWLEDGEMENTS
  
  I am grateful to my advisor Andrea Montanari, who first introduced me to the
  beautiful worlds of both free probability and AMP. I would like to thank Keigo
  Takeuchi and Galen Reeves for helpful discussions and pointers to related
  literature, and Yufan Li for pointing out an error in a previous version
  of the manuscript. This research is supported in part by NSF Grant DMS-1916198.
  
  
  alpha