1

Performance Analysis of LMS Filters for
Estimation of Cyclostationary Signals

arXiv:1708.00635v1 [] 2 Aug 2017

Nir Shlezinger and Koby Todros

Abstract
The least mean-square (LMS) filter is one of the most common adaptive linear estimation algorithms. In many practical scenarios, and particularly in digital communications systems, the signal of
interest (SOI) and the input signal are jointly wide-sense cyclostationary. Previous works analyzing
the performance of LMS filters for this important case assume specific probability distributions of
the considered signals or specific models that relate the input signal and the SOI. In this work, we
provide a general transient and steady-state performance analysis that is free of specific distributional
or model assumptions. We obtain conditions for convergence and derive analytical expressions for the
non-asymptotic and steady-state mean-squared error. The accuracy of our analysis is demonstrated in
simulation studies that correspond to practical communications scenarios.

I. I NTRODUCTION
The least mean-squares (LMS) is a widely used algorithm for adaptive linear estimation
of a signal of interest (SOI) based on an input signal. The LMS algorithm is a stochastic
approximation of the iterative steepest descent based implementation of the Wiener filter, when
the SOI and the input signal are jointly wide-sense stationary (JWSS) [1]–[3]. This stochastic
approximation involves a simple update equation which can be implemented in practical systems
with low computational complexity [1, Ch. 9], [2, Ch. 10]. As the LMS is designed for JWSS
signals, many works have been devoted to analyze its performance in this setup, see, e.g., [3]–
[7], and also [1, Ch. 9], [2, Ch. 24]. Nonetheless, man-made signals, and specifically digitally
modulated communications signals, are typically wide-sense cyclostationary (WSCS) [8, Sec. 57], [9, Ch. 1], [10]. Thus, in many practical communications systems, the considered signals are
The authors are with the department of ECE, Ben-Gurion University, Be’er-Sheva, Israel (e-mail: nirshl@post.bgu.ac.il;
todros@ee.bgu.ac.il).

August 3, 2017

DRAFT

2

jointly WSCS (JWSCS) [8, Sec. 3.6.2] rather than JWSS. Examples include interference-limited
communications [11], [12] and cognitive radio [13]. Another important example is narrowband
(NB) power line communications (PLC) systems, where the channel input is a digitally modulated
WSCS signal [14, Ch. 5], the channel transfer function is periodically time-varying [15], and the
additive channel noise is commonly modeled as a WSCS process that is mutually independent of
the channel input [14, Ch. 2], [16]. Hence, in this case, the channel input and the noisy channel
output are JWSCS.
Despite the importance of the WSCS scenario, only a few works have studied the performance
of adaptive algorithms in the presence of JWSCS input signal and SOI. In [17] it was shown
that the LMS filter coefficients are mean convergent only when the step-size approaches zero.
In this case, the filter coefficients converge to the minimal time-averaged (TA) mean-squared
error (MSE) filter. The performance of the LMS when applied to the adaptation of frequency
shift (FRESH) filters with WSCS inputs was studied in [18]–[20]. Specifically, [18] focused on
interference rejection in the presence of cyclostationary digitally modulated signals; The work
[19] proposed a scheme for blind adaptation of FRESH filters using the LMS and the recursive
least-squares algorithms; The work [20] studied the effect of errors in the frequency shifts on
the performance of LMS-based adaptive FRESH filters with a temporally independent input
signal. In [21], the LMS performance was studied for the identification of a linear system whose
coefficients obey a random walk model with a WSCS Gaussian input and an additive wide-sense
stationary (WSS) Gaussian noise. We note that when the random walk effect is negligible, the
linear system considered in [21] becomes linear time-invariant (LTI). The work [22] analyzed
the LMS performance when applied to adaptive line enhancement/cancellation for a WSCS input
consisting of a Gaussian process with periodic variance plus a sine wave with random phase.
Excluding [17], which studied mean convergence only, all the works mentioned above assume
specific signal distributions or specific models that relate the input and SOI.
In the works [23], [24] we developed a different adaptive filter for JWSCS signals based on the
TA-MSE criterion and analyzed its performance. As the adaptive filter in [23], [24] specializes
the LMS only when the signals are JWSS, the performance study conducted in [23], [24] cannot
be used to characterize the LMS behavior when the signals are JWSCS. In this context, it is

August 3, 2017

DRAFT

3

important to note that the empirical performance of the LMS presented in [24, Sec. V], in which
some of the scenarios considered correspond to practical communications scenarios, cannot be
predicted by any existing analytical LMS performance study. This is mainly due to the fact that
in [24, Sec. V], non-Gaussian signals and periodically time-varying channels (as encountered,
e.g., in practical NB-PLC systems) were considered, that do not satisfy the specific distributional
and model assumptions made in existing LMS performance analysis tools. The lack of a reliable
characterization of the behavior of LMS filters in such practical communications setups further
motivates the analysis of the LMS performance for general JWSCS signals.

Main Contributions: In this work, we provide a new general performance analysis of the
LMS algorithm in the presence of JWSCS input and SOI. We obtain a closed-form expression for
the transient MSE, derive conditions for convergence, including sufficient conditions on the LMS
step-size, and characterize the steady-state MSE. Unlike [18]–[22], we do not impose a specific
distribution or model on the considered signals. We only make the basic assumptions that are
commonly used in the analysis of adaptive algorithms. Specifically, the SOI is related to the input
signal only through a general stochastic representation which arises from the linear minimum
MSE (LMMSE) filter, that represents a linear periodically time-varying (LPTV) system when the
SOI and the input signal are JWSCS [25, Ch. 17.5.1]. Unlike [17], we also analyze mean-square
convergence and provide analytic expressions for the non-asymptotic and steady-state MSE.
Furthermore, unlike [17], in the paper, convergence of the LMS filter is characterized accurately,
without small step-size approximations. This is carried out by introducing a generalized definition
for convergence using the theory of asymptotically periodic sequences [26].
We verify our analysis in simulation examples. In particular, we demonstrate the accuracy of
our analysis in scenarios for which the LMS performance was not yet characterized, including
LPTV system identification and practical NB-PLC signal recovery scenarios. Our simulation
results show an excellent agreement between the theoretical and empirical performance measures.
Furthermore, we demonstrate that our analysis generalizes the results of [21] when applied to
LTI system identification in the presence of non-Gaussian signals.
The paper is organized as follows: Section II states the considered problem and presents the
general signal model assumed, Section III details the transient and steady state performance

August 3, 2017

DRAFT

4

analysis, and Section IV presents simulation examples. Lastly, Section V provides concluding
remarks. Complete proofs for the lemmas and theorems stated throughout the paper are provided
in the Appendix.
II. P RELIMINARIES

AND

P ROBLEM F ORMULATION

A. Notations
We denote column vectors with lower-case boldface letters, e.g., x. Matrices are denoted with
upper-case boldface letter, e.g., X. In denotes the n×n identity matrix and 0n×m denotes the allzero n×m matrix. Hermitian transpose, transpose, complex conjugate, and stochastic expectation
are denoted by (·)H , (·)T , (·)∗ , and E{·}, respectively. The real part of x is denoted by Re {x},
((n))m denotes the remainder of n when divided by m, and ⊗ denotes the Kronecker product.
The set of non-negative integers is denoted by N. For an n×n matrix X, λmax (X) denotes the
largest real eigenvalue of X, given that such exists, ρ(X) denotes the spectral radius of X, and
x = vec (X) denotes the n2 ×1 column vector obtained by stacking the columns of X, which is

recovered from x via X = vec−1 (x). For an n×1 vector y and an n2 ×1 vector x, kyk2 , yH y

denotes the squared Euclidean norm and kyk2x , yH vec−1 (x) y denotes its weighted version,

when vec−1 (x) is positive-definite. Lastly, for a set of n×n matrices {Xk } and integers l, m,
m
Q
Xk is the product Xm Xm−1 · · · Xl when m ≥ l and In when m < l.
k=l

B. Wide-Sense Cyclostationary Stochastic Processes
A discrete-time proper-complex (PC) multivariate process x[n] is said to be WSCS if both its


mean E x[n]} and autocorrelation function E x[n+l]xH[n]} are periodic with some period, N0 ,
w.r.t. n [8, Sec. 3.5]. A pair of jointly proper-complex (JPC) processes x1 [n], x2 [n] are said to

be JWSCS with period N0 if each process is WSCS with period N0 and their cross-correlation

function E x1 [n+l]xH
2 [n]} is periodic with period N0 w.r.t. n [8, Sec. 3.6.2]. Note that when

x1 [n], x2 [n] are WSCS with different periods, say N1 and N2 , and their cross-correlation function
is periodic with period N1,2 , then they are also JWSCS with a period which equals the least
common multiple of N1 , N2 and N1,2 .

August 3, 2017

DRAFT

5

C. Problem Formulation
We wish to characterize the performance of the LMS filter for linear estimation of a scalar
SOI d[n] based on an M × 1 multivariate input signal x[n], where x[n] and d[n] are zero-mean,
JPC, and JWSCS with period N0 . Let h[n] denote the M × 1 random coefficients vector of the
LMS filter at time instance n. For a step-size µ and initial guess h[0], the LMS update equation
is given by [1, Eq. 9.5]

∗
ˆ
h[n + 1] = h[n] + µ · x[n] d[n] − d[n] ,

n ≥ 0,

(1)

ˆ , hH [n]x[n] is the linear estimate of the SOI at time instance n.
where d[n]
In order to characterize the performance of the LMS algorithm, we formulate the relationship between d[n] and x[n] using the LMMSE estimator. To that aim, let hM [n] denote the
deterministic coefficients vector of the LMMSE estimator of d[n] based on x[n]. Clearly, hM [n]
satisfies the Wiener-Hopf equations [2, Ch. 3.5] at each time instance n. Thus, letting v[n] be
the estimation error of the LMMSE filter, the SOI d[n] can be written as
d[n] = hH
M [n]x[n] + v[n].

(2)

By the orthogonality principle [2, Ch. 4.2], the estimation error of the LMMSE filter is orthogonal
to the input signal, i.e., E{x[n]v ∗ [n]} = 0M ×1 . Note that the relationship between d[n] and x[n]
in (2) is a general stochastic representation of the SOI that does not arise from a specific model.
The fact that the SOI d[n] and the input signal x[n] are zero-mean, JPC, and JWSCS, results
in the following properties of the LMMSE filter coefficients vector hM [n] and the corresponding
estimation error v[n]:
1) Since d[n] and x[n] are JWSCS with period N0 , it follows from [25, Ch. 17.5.1] that hM[n]
defines an M ×1 periodic sequence, i.e., the LMMSE filter represents an LPTV system.
2) As d[n] and x[n] are also JPC and zero-mean, it follows from (2) that v[n] is a PC, zeromean, WSCS process, whose variance σv2 [n] , E{|v[n]|2 } is periodic with period N0 .
Similarly to the standard approach used for analyzing the LMS algorithm for JWSS signals,
e.g., [1]–[5], we make the following assumptions on the signals in (2):

August 3, 2017

DRAFT

6

AS1 The estimation error of the LMMSE estimator, v [n1 ], and the input signal, x [n2 ], are
mutually independent ∀n1 , n2 , see also [2, Ch. 15.2], [4, Sec. B.2]. This assumption is
H
satisfied, e.g., when d[n] = gN
[n]x[n] + z[n] where gN0 [n] is a deterministic LPTV filter
0

and z[n] is a PC WSCS process independent of x[n]. In this case the LMMSE filter is
hM[n] = gN0 [n] and its estimation error is v[n] = z[n]. This assumption also holds when
d[n] and x[n] are jointly Gaussian and temporally uncorrelated.
AS2 The random coefficients vector h[n] in (1) is independent of the instantaneous input x[n],
see [1, Pg. 392], [2, Ch. 16.4]. This is satisfied when, e.g., x[n] is a temporally independent
process.
AS3 The input x[n] is fourth-order cyclostationary1, i.e., its fourth-order moments are bounded
and periodic with period N0 . This is satisfied when, e.g., x[n] is Gaussian and WSCS.
Assumptions AS1-AS3 are utilized in the following section to obtain explicit convergence
conditions and to derive closed-form expressions for the non-asymptotic and steady-state MSE.
In Section IV, we show that the analysis carried out under these assumptions provides a reliable
characterization of the LMS performance in practical communications scenarios, where AS1-AS3
do not necessarily hold.
III. LMS P ERFORMANCE A NALYSIS
To facilitate the analysis, we first define several quantities that are summarized in Table I. We
then derive an expression for the non-asymptotic time-evolution of the MSE, which is utilized
to derive conditions for convergence. Under these conditions, we obtain an analytic expression
for the steady-state MSE.
A. Time-Evolution of the MSE
In order to analyze the MSE performance of the LMS filter, we first define its instantaneous
estimation error:
e[n] , d[n] − hH [n]x[n].

(4)

1
A similar assumption was made in the analysis of LMS with non-Gaussian WSS inputs [2, Ch. 24], where it was assumed
that the fourth-order moments are time-invariant [2, Eq. (24.9)] and bounded [2, Pg. 361].

August 3, 2017

DRAFT

7

TABLE I
Q UANTITIES USED IN

THE PERFORMANCE ANALYSIS OF THE

LMS.

Rx [n] , IM −µx [n] xH [n]
g[n] , hM [n] − hTA

Cx [n] , E x [n] xH [n]

(3a)
(3b)

cx [n] , vec (Cx [n])

(3d)

R̃x [n] , E {Rx [n]} = IM − µCx [n]

F[n] , E RTx [n] ⊗ Rx [n]

A[n] , CTx [n] ⊗ IM + (IM ⊗ Cx [n])
n
o
T
B[n] , E x [n] xH [n] ⊗ x [n] xH [n]


1 A[n] −B[n]
H[n] , 2
2IM 2 0M 2 ×M 2
n
o
T
P[n] , E x [n] xH [n] ⊗ Rx [n]

(3e)

LR̃
k1 ,k2

,

N0 −1+k
Q 2

R̃x [((l))N0 ]
l=k1
N0P
−1+k
LR̃
fk ,
l+1,k Cx [((l))N0 ] g [((l))N0 ]
l=k
−1

R̃
sk , IM −Lk,k
fk
LF
k1 ,k2 ,

N0 −1+k
Q 2

F [((l))N0 ]

(3c)

(3f )
(3g)
(3h)
(3i)
(3j)
(3k)
(3l)
(3m)
(3n)

l=k1

zk , 2PT [k](g[k]⊗s∗k )+BT [k](g[k]⊗g∗[k]) + σv2 [k]cx [k]

(3o)

The instantaneous MSE at time index n is given by E{|e[n]|2 }. To characterize the MSE timeevolution, we first obtain recursive relations for the first and (weighted) second-order statistical
moments of the coefficients error vector, which, similarly to [20, Eq. (24)], is defined as:
h̄ [n] , hTA − h [n] ,

(5)

where hTA is the M × 1 coefficients vector of the linear minimal TA-MSE (LMTA-MSE)
estimator obtained from the time-averaged Wiener-Hopf equations [20, Eq. (5)]. A recursive
relation for the expected coefficients error vector is given in the following lemma:
Lemma 1 (Mean relation). The expected coefficients error vector (5) satisfies the following
recursive relation for n ≥ 0

August 3, 2017

DRAFT

8



E h̄ [n + 1] = R̃x [n]E h̄ [n] − µ·Cx[n]g[n],

(6)

where g[n], Cx [n], and R̃x [n] are defined in (3b), (3c), and (3e), respectively.
[A proof is given in Appendix A]
The mean-square deviation (MSD) in filter coefficients is defined as the stochastic expectation
o
n
2
, where h̄[n] is the coefficients error vector defined in (5). Furthermore, we define
E h̄ [n]
o
n
2
the weighted MSD as E h̄ [n] q , where q is some M × 1 vector, such that Q , vec−1 {q}

is a Hermitian positive semi-definite matrix. A recursive relation for the weighted MSD is stated
in the following lemma:
Lemma 2 (Variance relation). The weighted MSD satisfies the following recursion for n ≥ 0:
E

n

h̄ [n+1]

2
q

o

=E

n

h̄ [n]

2
F[n]q

o

+µ2 kg [n]k2B[n]q



−2µ · gT [n]⊗E h̄ [n] P[n]q+µ2σv2 [n]cTx [n]q,

(7)

where g[n], cx [n], F [n], B [n], and P [n] are defined in (3b), (3d), (3f), (3h), and (3j), respectively.
[A proof is given in Appendix B]
Notice that for a period N0 = 1, the input signal and the SOI are JWSS. In this case, the
LMMSE filter coincides with the LMTA-MSE filter, introduced below (5), and therefore, by Eq.
(3b), the deviation vector g[n] satisfies g[n] ≡ 0M×1 . In this case, as expected, Lemma 1 coincides
with the LMS mean relation for JWSS signals [2, Eq. (24.2)], and Lemma 2 coincides with the
LMS variance relation for JWSS signals [2, Eq. (24.11)]. Another special case of the variance
relation (7) is obtained under the following scenario: Consider the problem of identifying an
LTI system with WSCS input x[n] whose output is corrupted by additive WSCS noise v[n],
uncorrelated with the input. Under this scenario, one can verify that the LMMSE filter is time
invariant, and therefore, the deviation vector g[n] satisfies g[n] ≡ 0M×1 . In this case, if both x[n]
and v[n] are Gaussian, and v[n] is WSS (recall that WSCS processes specialize WSS processes),
then, by setting q = vec−1 (IM ), one can verify that (7) specializes the recursive characterization

August 3, 2017

DRAFT

9

of the MSD in [21, Eq. (15)]. We note that this specialization holds only when the random walk
effect in [21] is neglected, i.e., the linear system considered in [21] becomes LTI2 .
In the following theorem, the recursive relations in Lemmas 1–2 are used to obtain an explicit
characterization of the non-asymptotic MSE of LMS filters with JWSCS input and SOI:
Theorem 1 (MSE time-evolution). The instantaneous MSE of the LMS algorithm (1) satisfies
n
o

2
2
E{|e[n]| } = E h̄ [n] cx [n] + 2Re gH [n]Cx [n]E{h̄[n]}
+ kg [n]k2cx [n] + σv2 [n],

∀n ≥ 0,

(8)

where the quantities g[n], Cx [n], and cx [n] are defined in (3b)–(3d), respectively.
[A proof is given in Appendix C]
n

Note that the expectations E h̄[n] and E h̄ [n]

2
cx [n]

o

in (8) can be recursively computed

using (6) and (7), respectively. Also notice that when g[n] ≡ 0M×1 , i.e., the LMMSE filter is
LTI, the MSE time-evolution (8) specializes the corresponding result for the LMS with JWSS
signals in [2, Pg. 363].
B. Convergence Analysis and Steady-State MSE
Here, we derive the conditions for convergence of the LMS filter with JWSCS signals, and
characterize its steady-state performance. We begin by stating the definition for asymptotically
periodic sequence, which is equivalent to the one in [26, Def. 3.1].
Definition 1 (Asymptotically periodic sequence). A sequence p[n] is said to asymptotically
periodic with period N0 if for every k ∈ {0, 1, . . . , N0 − 1} , N0 , the subsequence pk [n] ,
p[n · N0 + k] converges as n → ∞.
Comment 1. Note that when the N0 subsequences all converge to the same limit, the definition
of asymptotically periodic sequences specializes the definition of convergent sequences.
2
While we consider the general setup of adaptive estimation of JWSCS signals, for which the LMMSE filter coefficients
vary periodically in time, the work [21] studied the identification of a linear system and modeled the temporal variations in
the system coefficients via a non-stationary random walk process. As a result, the variance relation in [21, Eq. (15)] includes
an additive variance component due to these random variations. Consequently, for LTI system identification, Eq. (7) specializes
[21, Eq. (15)] when this variance term is neglected, as carried out in part of the analysis conducted in [21].

August 3, 2017

DRAFT

10

Comment 2. It follows from Def. 1 that if p[n] is asymptotically periodic with period N0 , then
pk [n] = p[n · N0 + k] converges for every finite k ∈ N.
Based on Def. 1, we consider the following definitions for convergence:
Definition 2 (Convergence in the mean). An adaptive filter with coefficients error vector h̄ [n]

is said to be mean convergent if E h̄ [n] is asymptotically periodic.

Definition 3 (Mean-square stability). An adaptive filter with coefficients error vector h̄ [n] is
n
o
2
said to be mean-square stable if E h̄ [n]
is asymptotically periodic.
Defs. 2–3 generalize the traditional definitions for mean convergence and mean-square stability

for JWSS SOI and input signal [2, Ch. 23.2, 23.4]. These traditional definitions require the mean
n
o

2
coefficients error E h̄ [n] to converge to 0M ×1 and the MSD E h̄ [n]
to be convergent.

However, as was shown in [17, Cmt. 2], when no specific model relating the JWSCS SOI d[n]

and input signal x[n] is assumed, then lim E h̄ [n] = 0M ×1 only when the step-size µ → 0.
n→∞

Consequently, to be able to specify a non-infinitesimal step-size region which guarantees that
the algorithm does not diverge, we use Defs. 2–3.
Using Lemma 1, we obtain the following necessary and sufficient condition for the LMS to

be mean convergent:
Proposition 1 (Necessary and sufficient condition for mean convergence). The LMS algorithm
is mean convergent if and only if LR̃
k,k , defined in (3k), satisfies


R̃
ρ Lk,k < 1, ∀k ∈ N0 .

(9)

[A proof is given in Appendix D]

Comment 3. For a given µ > 0, it is shown in Appendix D that lim E h̄ [n] = 0M ×1 , namely,
n→∞

the LMS satisfies the traditional mean-convergence definition used in [17], if and only if, in

addition to (9), the vector sk defined in (3m) satisfies sk = 0M ×1 , ∀k ∈ N0 . According to (3l)
and (3m), this occurs, e.g., when the deviation vector g[n] defined in (3b) satisfies g[n] ≡ 0M ×1 ,
i.e., the LMMSE filter is LTI, as in the case of JWSS signals. However, since the LMMSE filter for

August 3, 2017

DRAFT

11

JWSCS signals is LPTV [25, Ch. 17.5.1], g[n] is generally non-zero. Consequently, as noted in
[17, Cmt. 2], the LMS filter with JWSCS signals generally does not satisfy traditional definition
for mean convergence for any fixed step-size µ > 0.
By (3k), one can verify that (9) is satisfied if (but not only if) the spectral norm of R̃x [l]

satisfies ρ R̃x [l] < 1, ∀l ∈ N0 . Hence, we obtain the following sufficient condition on the
step-size µ which guarantees mean convergence:

Corollary 1 (Sufficient condition for mean convergence). The LMS algorithm is mean convergent
if the step-size satisfies
0<µ<

2
,
λmax (Cx [k])

∀k ∈ N0 .

(10)

[A proof is given in Appendix E]


Note that in the WSS case, N0 = 1, Eq. (9) can be written as ρ R̃x [0] < 1, thus (10)

becomes also a necessary condition for mean convergence, and coincides with the standard
mean convergence condition for WSS signals in [2, Ch. 24.2].

Next, based on the recursive relations in Lemmas 1-2, we obtain the following sufficient
condition for mean-square stability:
Theorem 2 (Mean-square stability). The LMS algorithm is both mean convergent and meansquare stable if the following conditions are satisfied:
1) Condition (10) is satisfied.
2) The covariance matrix Cx [k] is non-singular and its entries are bounded ∀k ∈ N0 .
3) The step-size µ satisfies3 for all k ∈ N0 :

µ < min

1
,
−1
λmax (A [k]B[k]) λmax (H[k])
1



,

(11)

where A[k], B[k], and H[k] are defined in (3g)–(3i), respectively.
[A proof is given in Appendix F]
3

If H[k] does not have any real positive eigenvalues, then condition (11) is replaced with µ <

August 3, 2017

1
.
λmax(A−1 [k]B[k])

DRAFT

12

Note that since the covariance matrix Cx [k] is required to be non-singular, then A[k] defined
(3g) is non-singular, thus (11) is well-defined. We also note that when the signals are JWSS,
i.e., N0 = 1, then Thm. 2 specializes the sufficient condition for stability of LMS with JWSS
non-Gaussian signals in [2, Eq. (24.24)].
Lastly, as stated in the following theorem, when the conditions in Thm. 2 are satisfied, and
the number of iterations n approaches infinity, the transient MSE (8) converges to a periodic
sequence that characterizes the steady-state MSE and takes the following form:
Theorem 3 (Steady-state MSE). If the conditions stated in Thm. 2 are satisfied, then the transient
MSE (8) is an asymptotically periodic sequence such that ξ[k] , lim E {|e [n · N0 + k] |2 } takes
n→∞

the form:
ξ[k] = µ

2

N0X
+k−1
l=k

F
zT((l))N LF
l+1,k IM 2 − Lk,k
0

−1

cx [k]


−2µRe gH [k]Cx [k]sk + kg [k]k2cx [k] + σv2 [k],

(12)

where g[n], Cx [n], cx [n], sk , LF
k,l , and zk are defined in (3b)–(3d) and (3m)–(3o), respectively.
[A proof is given in Appendix G]
Notice that for N0 = 1, i.e., x[n] and d[n] are JWSS, then, as noted in the discussion following
Lemma 2, g[n] ≡ 0M ×1 . In this case, the second and third summands of (12) vanish, and the
first summand reduces to the excess steady-state MSE of the LMS with JWSS signals in [2,
Thm. 24.1]. Thus, Thm. 3 specializes the steady-state MSE characterization in [2, Thm. 24.1]
when the SOI and input are JWSS.
C. Discussion
First, we note that, as discussed in the previous subsection, when the signals are JWSS, the
generalized performance analysis presented in this paper coincides with the standard performance
analysis of the LMS with JWSS signals in [2, Ch. 24]. We further note that when the signals
are not JWSS, then the vector g[n] (3b), which represents the deviation of the LMMSE filter
from an LTI system, and essentially, the periodic dynamic of the signals (and is thus zero for the
JWSS setup), has a dominant effect on the LMS behavior. This is observed, e.g., in the temporal

August 3, 2017

DRAFT

13

statistical moments of the coefficients error characterized by the recursive relations in (6) and
(7), and also in the instantaneous MSE in (8). Consequently, the presented performance analysis
quantifies the effect of these periodic dynamic on the performance of the LMS filter, compared
to the JWSS setup. Finally, note that the deviation vector g[n] does not appear in [21]. This is
due to the fact that for the specific system model and the additive noise considered in [21], it
can be shown that the LMMSE estimator of the SOI is LTI, and thus g[n] is the zero vector.
IV. N UMERICAL E XAMPLES
Here, we demonstrate the theoretical analysis in a simulation study. We focus on two scenarios:
1) A scenario which satisfies AS1-AS3, whose purpose is to verify the theoretical analysis and
to numerically compare our analysis with the state-of-the-art.
2) An NB-PLC signal recovery scenario, which demonstrates the accuracy of the analysis in
a practical scenario when AS1-AS3 are not necessarily satisfied.
All empirical performance measures were obtained via 10000 Monte Carlo simulations.
A. Scenario 1: Theoretical Analysis Verification
We begin with an example in which AS1-AS3 are satisfied. To that aim, we construct JWSCS
input signal x[n] and SOI d[n], under the following settings: Let Nu , Nx , Nh , and Nv be positive
integers, and let u[n] be an M ×1 i.i.d. random process with M = 8 i.i.d. entries that are uniformly
distributed over the unit circle. The input signal x[n] is a zero-mean WSCS given by
1
x[n] , √
Nu





2πn
1+0.5 cos
Nx

 NX
u −1
k=0

u[Nu · n+k].

We set the M × 1 LMMSE coefficients vector hM [n] in (2) such that


1
· ((n))Nh e−0.5|k| .
(hM [n])k , 1+
5 · Nh
Furthermore, we set the estimation error process v[n] in (2) to be a zero-mean temporally
uncorrelated Gaussian process independent of x[n], with variance
2

0.1
2
−6
.
σv [n] , 10 · 1+
((n))Nv +1

August 3, 2017

DRAFT

14

8

Empirical, µ = 0.01
Theoretical, µ = 0.01
Empirical, µ = 0.04
Theoretical, µ = 0.04

6
4

MSE [dB]

2
0
−2
−4
−6
−8
−10

0

50

100

150

200

250

300

350

Iteration

Fig. 1. The theoretical and the empirical instantaneous MSEs when AS1-AS3 are satisfied.

The SOI d[n] is related to x[n] via (2). Note that d[n] and x[n] are JWSCS with period N0 ,
which is the least common multiple of Nx , Nh , and Nv , and that this setup satisfies AS1-AS3,
for any selection of Nu , Nx , Nh , and Nv .
In order to verify the theoretical analysis in Thms. 1-3, we first fix Nu = 8, Nx = 40, Nh = 10,
and Nv = 5. Note that under this setting, the input signal x[n] is non-Gaussian, the LMMSE
filter hM [n] is an LPTV system, and the estimation error v[n] is a WSCS process. Fig. 1 depicts
the theoretical MSE for this setting, computed via (8), as compared to the empirical MSE, for
step sizes µ = {0.01, 0.04}. Fig. 2 depicts the theoretical steady-state TA-MSE for this setting
(recall that the MSE converges to a periodic sequence) computed by time-averaging (12) over the
period N0 , the stability threshold, computed via (11), and the empirical steady-state TA-MSE.
It is illustrated in Fig. 1 that, when AS1-AS3 are satisfied, the time-evolution of the MSE is
accurately characterized by the theoretical analysis in Thm. 1. An excellent agreement between
the empirical and theoretical performance measures is also observed in Fig. 2. One sees that,
indeed, the conditions in Thm. 2 guarantee mean-square stability when the step-size is not larger
than the threshold. Furthermore, one can notice that (12) accurately characterizes the steady-state
performance of the LMS filter after convergence is obtained.
Next, we compare our analysis to the one provided in [21], for the problem of LTI system

August 3, 2017

DRAFT

15

10

Empirical
Theoretical

8
6
Steady−state TA−MSE [dB]

−5
4
−6
2

Stability
threshold
(Thm. 2)

−7

0

−8

−2

−9
−10
0.02

−4

0.04

0.06

0.08

−6
−8
−10

0

0.02

0.04

0.06
0.08
Step−size

0.1

0.12

0.14

Fig. 2. The theoretical and the empirical steady-state TA-MSEs when AS1-AS3 are satisfied.

identification. We note that in [21], the system input is assumed to be a Gaussian WSCS process
and the additive noise in the system output is assumed to be a WSS Gaussian process. Note that
for Nh = Nv = 1, the SOI d[n] and the input signal x[n] defined above are related via an LTI filter
whose output is corrupted by additive WSS Gaussian noise. Also note that by the multivariate
central limit theorem [27, Pg. 912], as Nu increases, the input vector x[n] converges in distribution
to a multivariate Gaussian random vector. Thus, for sufficiently large Nu , the LMS performance
for this setting can be characterized by [21, Eq. (15)]4 . As [21] focused on characterizing the
o
n
2
, Figs. 3–4 depict the empirical filter coefficients
MSD in the LMS coefficients, E h̄ [n]
MSD, compared to the theoretical MSD, evaluated using Lemma 2 with q = vec−1 (IM ) and

using [21, Eq. (15)], for step-size µ = 0.1, Nx = 40, and Nu = {64, 1}. In Fig. 3 we observe
that for Nu = 64, i.e., when the distribution of the input signal is approximately Gaussian, then,
as expected, both our analysis and [21, Eq. (15)] accurately characterize the filter coefficients
MSD. However, in Fig. 4 we observe that for Nu = 1, [21, Eq. (15)] does not provide an
accurate characterization of the empirical performance, as the distribution of the input signal
is not Gaussian. On the other hand, an excellent agreement between the empirical MSD and
4

As noted in footnote 2, [21] modeled temporal variations in the coefficients of a linear system using as a random walk
process. Thus, [21, Eq. (15)] includes an additive variance component due to these variations. Consequently, [21, Eq. (15)]
characterizes the LMS performance for LTI system identification when this variance term is neglected, as performed in part of
the analysis in [21].

August 3, 2017

DRAFT

16

10

10
Empirical
Theoretical
[20, Eq. (15)]

0

0
−10
MSD [dB]

MSD [dB]

−10
−20
−30

−20
−30

−40

−40

−50

−50

−60
0

Empirical
Theoretical
[20, Eq. (15)]

50

100

150
200
Iteration

250

300

−60
0

350

Fig. 3. The theoretical and the empirical filter coefficients
MSDs, LTI system identification scenario, Nu = 64.

50

100

150
200
Iteration

250

300

350

Fig. 4. The theoretical and the empirical filter coefficients
MSDs, LTI system identification scenario, Nu = 1.

the theoretical MSD computed using Lemma 2 is observed, as unlike [21], our analysis is not
limited to a specific distribution of the input signal.
In summary, the agreement between the empirical and theoretical performance measures
observed in this numerical study illustrate the validity of the theoretical study when AS1-AS3
are satisfied. Furthermore, the comparison with [21] for LTI system identification demonstrates
the benefit of our general analysis compared to previous works.
B. Scenario 2: NB-PLC Signal Recovery
Next, we evaluate the theoretical analysis in Thms. 1-3 in a practical scenario which corresponds to signal recovery in NB-PLC channels. The SOI d[n] is the input signal to an
NB-PLC channel. Here, d[n] is an orthogonal frequency division multiplexing signal with 36
subcarries, each modulated via a QPSK constellation, with 12 cyclic prefix samples. It follows
from [10] that d[n] is WSCS with period N0 = 48. The channel output, r[n], is given by
∞
P
r[n] =
g[n, l]d[n − l] + w[n] [28, Sec. III], where g[n, l] is an LPTV filter with period N0 ,
l=0

generated as in [28] following the IEEE P1901.2 standard [29]. The additive channel noise
w[n] (note that this is not the estimation error v[n] in (2)) is a WSCS Gaussian process with
period5 N0 , generated using the model [30], with a set of parameters taken from [30, Tbl.
2]. This scenario is illustrated in Fig 5. The input to the LMS filter, x[n], is obtained via
5

In NB-PLC channels, the periods of the WSCS information signal, LPTV channel transfer function, and WSCS noise, are not
necessarily the same [15]. However, as these periods are typically commensurate [29], the statistical moments of the considered
signals and the LPTV channel can be treated as if they have the same period, which equals the least common multiple of the
different periods.

August 3, 2017

DRAFT

17

Fig. 5. A schematic description of the NB-PLC signal recovery scenario (Scenario 2).

multivariate preprocessing of r[n] that produces (x[n])k = r[n − k], k ∈ {0, 1, . . . , M − 1},
with M = 8. The LMMSE filter hM [n] in (2) is obtained from the orthogonality principle,

i.e., E {x [n] d∗ [n]} = E x [n] xH [n] hM [n], and v[n] is obtained as the estimation error for

hM [n]. We note that the entire set of assumptions AS1-AS3 is not satisfied in this scenario. The
NP
0−1
E{|r[n−k]−w[n−k]|2 }
k=0
, is set to 12 dB.
signal-to-noise ratio, defined as
NP
0−1
E{|w[n−k]|2 }
k=0
We note that the analysis in [21] is not applicable under this scenario due to the following
reasons: 1) The work [21] is designated for system identification, while the considered NBPLC scenario corresponds to signal recovery; 2) NB-PLC channels are modeled as LPTV
systems with additive WSCS noise, while in [21] a linear non-periodically time-varying system
is assumed, where the temporal variations in the system coefficients obey a random walk process.
Furthermore, in [21] the additive noise is a WSS process.
Fig. 6 depicts the empirical transient MSE and its theoretical values, computed via (8), for step
sizes µ = {0.01, 0.04}. The empirical steady-state TA-MSE is depicted in Fig. 7, compared to
the stability threshold, computed via (11), and to the theoretical steady-state TA-MSE, computed
by time-averaging (12). The results in Fig. 6 demonstrate that even in practical scenarios where
AS1-AS3 are not necessarily satisfied, there is a very good agreement between the theoretical
and the empirical performance. Additionally, it is observed in Fig. 6 that due to the dominant
periodic dynamics in the NB-PLC scenario, the LMS exhibits significant periodic variations
in the MSE. In Fig. 7 we observe that Thm. 2 provides a reliable prediction of the stability
threshold of the LMS filter. Furthermore, one can notice that Thm. 3 accurately characterizes
the empirical steady-state performance, and that there is only a small gap between the theoretical
and empirical measures, which arises from the fact AS1-AS3 are not satisfied here. To conclude,

August 3, 2017

DRAFT

18

Empirical, µ = 0.01
Theoretical, µ = 0.01
Empirical, µ = 0.04
Theoretical, µ = 0.04

0

MSE [dB]

−2

−4

−6

−8

−10
0

50

100

150

200

250

300

350

Iteration

Fig. 6. The theoretical and the empirical instantaneous MSEs, NB-PLC signal recovery scenario.

300

Empirical
Theoretical

Steady−state TA−MSE [dB]

250

200

−3
−4
−5

150

−6

Stability
threshold
(Thm. 2)

−7
100
−8
−9
0.02

50

0.04

0.06

0.08

0.1

0
0

0.02

0.04

0.06

0.08

0.1
0.12
Step−size

0.14

0.16

0.18

0.2

Fig. 7. The theoretical and the empirical steady-state TA-MSEs, NB-PLC signal recovery scenario.

these results illustrate the accuracy of the proposed analysis in practical scenarios in which
assumptions AS1-AS3 are not satisfied.
V. C ONCLUSIONS
This paper provides a general performance analysis of the LMS algorithm for estimation
of WSCS signals. A complete characterization of the time-evolution of the MSE and the first
and second-order statistical moments of the coefficients error vector was provided. Sufficient

August 3, 2017

DRAFT

19

conditions for convergence and stability were obtained, and the steady-state MSE was derived.
The simulation results demonstrate the accuracy of the theoretical performance measures and
the stability thresholds.
A PPENDIX
The following properties are repeatedly used in the sequel:
1) For any matrix triplet A1 , A2 , A3 of compatible dimensions, it holds that (see [32, Ch. 9.2]),

vec (A1 A2 A3 ) = AT3 ⊗ A1 vec (A2 ) .

(13)


Tr AT1 A2 = vec (A1 )T vec (A2 ) .

(14)

2) For any pair of square matrices A1 , A2 of identical dimensions, it holds that (see [32, Ch.
9.2]),

3) Since x[n] is WSCS, if follows from AS3 that Cx [n], cx [n], R̃x [n], F[n], A[n], B[n], H[n],
and P[n], defined in (3c)-(3j), are all periodic with period N0 .
A. Proof of Lemma 1
From (2) it follows that the instantaneous estimation error e[n] (4) can be written as
H

e[n] = h̄ [n] x [n] + gH [n] x [n] + v [n] .

(A.1)

From the definition of the coefficients error vector (5) it follows that ∀n ∈ N
h̄[n+1] = hTA −h[n+1]
(a)

= hTA −h[n]−µ · x[n]e∗ [n]

∗
(b)
H
= h̄[n]−µ · x[n] h̄ [n] x [n]+gH [n] x [n]+v [n]
(c)

= Rx [n]h̄[n]−µ · x[n]xH [n]g[n]−µ · x[n]v ∗ [n] ,

(A.2)

where (a) follows from (1); (b) follows from (A.1); and (c) follows from the definition of Rx [n]
in (3a). Applying the stochastic expectation to (A.2) yields

August 3, 2017

DRAFT

20



E{h̄[n+1]} = E Rx [n]h̄[n] −µ · E x [n] xH g [n]−µ · E {x [n] v ∗ [n]}
(a)

= R̃x [n]E{h̄[n]}−µ · Cx [n]g[n],

(A.3)

where (a) follows since h̄[n] and Rx [n] are mutually independent by AS2, and since x[n] and
v[n] are zero-mean and mutually independent by AS1.
B. Proof of Lemma 2
Set Q = vec−1 {q}. The weighted squared Euclidean norm can be formulated as
h̄ [n+1]

2
q

H

= h̄ [n+1] Qh̄ [n+1]
(a)

H
= Rx [n] h̄ [n]−µ · x [n] xH [n] g [n]+v ∗ [n] Q
× Rx [n] h̄ [n]−µ · x [n] xH [n] g [n]+v ∗ [n]
H

= h̄ [n] Rx [n] QRx [n] h̄ [n]





+µ2 · gH [n] x [n]+v [n] xH [n] Qx [n] xH [n] g [n]+v ∗ [n]
n
o
H
−2µ · Re h̄ [n]Rx [n] Qx [n] xH [n]g[n]+v ∗ [n] ,

(B.1)

where (a) follows from (A.2). Applying the stochastic expectation to (B.1), combined with AS2
yields
E

n

h̄ [n+1]

2
q

o

n
o
H
= E h̄ [n] E{Rx [n] QRx [n]} h̄ [n]
n n
o 
o
H
−2µ · Re E h̄ [n] E Rx [n] Qx [n] xH [n] g [n]

+µ2 · gH [n] E x [n] xH [n] Qx [n] xH [n] g [n]

+µ2 · E |v [n]|2 xH [n] Qx [n] .

(B.2)

In the following we simplify the expressions for each of the summands in (B.2). First we note
that combining (13) with definitions (3f), (3h), and (3j) yields

August 3, 2017

E{Rx [n] QRx [n]} = vec−1 {F [n] q} ,

(B.3a)


E Rx [n] Qx [n] xH [n] = vec−1 {B [n] q} ,

(B.3b)

DRAFT

21

and

E x [n] xH [n] Qx [n] xH [n] = vec−1 {P [n] q} ,

(B.3c)

respectively. Thus, combining (13) with (B.3a) yields
n
o
n
H
E h̄ [n] E{Rx [n]QRx [n]}h̄[n] = E h̄[n]

2
F[n]q

o

.

(B.4)

Furthermore, combining (13) with (B.3c) yields
n


n
o

H
H
T
E h̄ [n] E Rx [n] Qx [n] x [n] g [n] = g [n] ⊗ E h̄ [n] P [n] q,
H

o

(B.5)

and (13) combined with (B.3b) yields

gH [n] E x [n] xH [n] Qx [n] xH [n] g [n] = kg [n]k2B[n]q .

(B.6)

Lastly, from AS1 it follows that


E |v [n]|2 xH [n] Qx [n] = σv2 [n] E xH [n] Qx [n]
(a)

= σv2 [n] Tr {Cx [n] Q}

(b)

= σv2 [n] cTx [n] q,

(B.7)

where (a) follows from the definition of Cx [n] (3c), and (b) follows from (14). Plugging (B.4),
(B.5), (B.6), and (B.7) into (B.2) yields (7).

C. Proof of Theorem 1
From (A.1) it follows that

E |e [n]|2 = E



H

H

2



h̄ [n] x [n]+g [n] x [n]+v [n]


2

H
(a)
+E |v [n]|2
=E
h̄ [n]+g [n] x [n]


2



(b)
H
+2Re gH [n] E x [n] xH [n] E h̄ [n]
= E h̄ [n] x [n]

+gH [n] E x [n] xH [n] g [n]+σv2 [n] ,

August 3, 2017

(C.1)

DRAFT

22

where (a) follows from AS1, and (b) follows from AS2. From AS2 it also follows that
E



H

h̄ [n] x [n]

2



n
o

H
H
= E h̄ [n] E x [n] x [n] h̄ [n]

(a)

=E

n

h̄ [n]

2
cx [n]

o

,

(C.2)

where (a) follows from the definition of cx [n] (3d). Hence, plugging (C.2) into (C.1) yields
(8).

D. Proof of Proposition 1
First, as in [24, Appendix F], we define the decimated components decomposition [25, Sec.
17.2] of the coefficients error vector as follows:
h̄k [n] , h̄[n · N0 +k],

k ∈ N0 , n ∈ N.

(D.1)

It thus follows that ∀n > N0
E{h̄k [n+1]} = E{h̄[n · N0 +N0 +k]}
(a)

= R̃x [((k−1))N0 ] E{h̄[n · N0 +N0 +k−1]}−µ·Cx [((k−1))N0 ] g [((k−1))N0 ]

(b)

= LR̃
k,k E{h̄k [n]}−µ·fk ,

(D.2)

where (a) follows from (6) and since R̃x [n], Cx [n], and g[n], defined in (3e), (3c), and (3b),
are all periodic with period N0 ; (b) follows from repeating the recursion N0 times and plugging
the expressions for LR̃
k,k and fk defined in (3k) and (3l). Repeating (D.2) n times yields
n 
n+1

m
X
R̃
R̃
E{h̄k [n+1]} = Lk,k
Lk,k fk .
E{h̄k [0]}−µ

(D.3)

m=0



Therefore, E h̄k [n] converges regardless of the initial value E h̄k [0] if and only if
lim

n→∞



LR̃
k,k

n

= 0M ×M ,



R̃
which is satisfied if and only if ρ Lk,k < 1 [31, Ch. 7.10]. This proves condition (9).
m 
−1
n 
P
R̃
LR̃
Additionally, we note that when (9) is satisfied, then lim
=
I
−L
[31,
M
k,k
k,k
n→∞ m=0

August 3, 2017

DRAFT

23

Ch. 7.10], thus
−1


R̃
lim E h̄k [n] = −µ · IM −Lk,k
fk

(D.4)

n→∞

=−µ · sk ,

(D.5)


where sk is defined in (3m). It follows from (D.4) that lim E h̄k [n] = 0M ×1 if and only if
n→∞

fk = 0M ×1 , which proves the statement in Comment 3.

E. Proof of Corollary 1
It follows from (3k) that ∀k ∈ N0 ,




ρ LR̃
k,k = ρ

NY
0−1+k
l=k

0−1+k
(a) NY

≤

l=k

!

R̃x [((l))N0 ]

0−1

 NY


ρ R̃x [((l))N0 ]
ρ R̃x [l] ,

(E.1)

l=0

where (a) follows from [31, Ch. 5.2]. From (E.1) it follows that condition (9) is satisfied if (but

not only if) ρ R̃x [l] < 1, ∀l ∈ N0 . Since R̃x [l] = IM − µ · Cx [l], where Cx [l] is Hermitian

positive semi-definite, it follows that ρ R̃x [l] < 1 if and only if (10) is satisfied.
F. Proof of Theorem 2

From the proof of Prop. 1 it follows that if (9) is satisfied, then E h̄k [n] converges to −µ·sk ,

where h̄k [n] is defined in (D.1). For k ∈ N0 , n ∈ N, define


b̃[nN0 + k] , E h̄k [n] +µ · sk ,

(F.1)



p̃[n] , PT [n] g[n] ⊗ b̃∗ [n] .

(F.2)

and

As the entries of Cx [k] are bounded ∀k ∈ N0 , it follows from (D.3) combined with AS3 that,
when (10) is satisfied, then, the entries of p̃[n] are bounded ∀n ∈ N and lim p̃[n] = 0M 2 ×1 .
n→∞

August 3, 2017

DRAFT

24

From Lemma 2 it follows that for any M 2 × 1 vector q such that Q = vec−1 {q} is Hermitian
positive semi-definite, and ∀n > N0 , we have that
E

n

h̄k [n+1]

2
q

o

=E
=E

n

h̄ [nN0 +N0 +k]

2
q

n

o

2
F[((k−1))N0 ]q

o

h̄ [nN0 +N0 +k−1]


P [((k−1))N0 ] q
+2µ2 · gT [((k−1))N0 ] ⊗ sH
((k−1))N
0

+µ2 · σv2 [((k−1))N0 ] cTx [((k−1))N0 ] q +µ2kg [((k−1))N0 ] k2B[((k−1)) ]q
N0


−2µ · gT [((k−1))N0 ]⊗ b̃H [nN0 +N0 +k−1] P [((k−1))N0 ] q
o
n
(a)
2
= E h̄ [nN0 +N0 +k−1] F[((k−1)) ]q
N0

+µ2 · zT((k−1))N q − 2µ · p̃T [nN0 +N0 +k−1] q,

(F.3)

0

where (a) follows by plugging (3o) and (F.2), recalling that

kg [n]k2B[n]q = gT [n] ⊗ gH [n] B [n] q.
Using (F.3), we next show that if (11) is satisfied, then E
finite value for n → ∞, ∀k ∈ N0 . To that aim, define:
ak ,

N0X
−1+k

n

zT((l))N LF
l+1,k
0

l=k

h̄k [n]

bk [n] , 2

T

p̃ [nN0 +

l=k

o

converges to a fixed and

!T

,

and
N0X
−1+k

2
q

l]LF
l+1,k

(F.4a)

!T

.

(F.4b)

Again, since the entries of Cx [k] are bounded and from AS3 it follows that ak and bk [n] are
bounded ∀k ∈ N0 . Using these definitions, repeating (F.3) N0 times results in
E

n

h̄k [n+1]

2
q

o

=E

n

h̄k [n]

2
LF
k,k q

o

+µ2 · aTk q−2µ · bTk [n] q.

(F.5)

Following [2, Ch. 24.2] and [24, Appendix F], we use (F.5) to formulate M 2 state-space

August 3, 2017

DRAFT

25

recursions for each k ∈ N0 as follows:
E

n

2

o

h̄k [n+1] (LF ) q = E
k,k
l

n

2

h̄k [n] (LF )
k,k

l+1

q

o

2

l

T
F l
+µ2aTk LF
k,k q−2µbk [n] Lk,k q,

(F.6)

M −1
l ∈ {0, 1, . . . , M 2 − 1}. Let {αl }l=0
be the coefficients of the characteristic polynomial of LF
k,k

[31, Pg. 492]. It follows from the Cayley-Hamilton theorem [31, Pg. 532] and from the linearity
of the weighted Euclidean norm [2, Eq. (23.31)] that
E



h̄k [n]

2

(LFk,k )

M2

q



2−1
M
X

=−

αl E

l=0

n

o
2
h̄k [n] (LF )l q .
k,k

Therefore, by defining the M 2 × 1 vectors h̄k [n], ak , and bk [n], such that
˜
˜
˜
o
n

2
l
h̄k [n] l , E h̄k [n] (LF ) q ,
k,k
˜
l
(ak )l , aTk LF
k,k q,
˜
and
l
(bk [n])l , bTk [n] LF
k,k q,
˜
l ∈ {0, 1, . . . , M 2−1}, and the M 2 × M 2 matrix F̄k such that F̄Tk is the companion matrix of the
˜
˜
characteristic polynomial of LF
[31,
Pg.
648],
the
state-space
recursions (F.6) can be written
k,k
as a set of N0 multivariate difference equations
h̄k [n + 1] = F̄k h̄k [n] + µ2 ak − 2µbk [n],
˜
˜ ˜
˜
˜

(F.7)

k ∈ N0 , n ≥ 0. Note that ∀k ∈ N0 , Eq. (F.7) represents an M 2 × M 2 multivariate LTI system
with input signal µ2 ak −2µbk [n] and output signal h̄k [n]. Since the entries of p̃[n] are bounded
˜
˜
˜
∀n ∈ N and lim p̃[n] = 0M 2 ×1 , it follows that the entries of bk [n] are also bounded ∀n ∈ N and
n→∞
˜

that lim bk [n] = 0M 2 ×1 . It therefore follows from [2, Ch. 23.4] that if ρ F̄k < 1 then h̄k [n] is
n→∞ ˜
˜
o˜
n
2
bounded and tends to a steady-state value for n → ∞, thus E h̄k [n] q is convergent.
So far we have shown that if condition (9) is satisfied, the entries of Cx [k] are bounded

∀k ∈ N0 , and ρ F̄k < 1 for all k ∈ N0 , then h̄[n] is mean-square stable. We now show that
˜
the latter condition is satisfied when the constraints on A[k]−1 B[k] and H[k] in (11). Note that

August 3, 2017

DRAFT

26

it follows from [2, Pg. 346] that the eigenvalues of F̄k are the eigenvalues of LF
k,k . Similarly to
˜
NQ
0 −1

(E.1), it can be shown that ∀k ∈ N0 , ρ LF
ρ (F[l]). Therefore, if ρ (F[k]) < 1 for all
k,k ≤
l=0

k ∈ N0 , then mean-square stability is obtained. Note that F[k] is Hermitian, thus its eigenvalues
are real, and ρ (F[k]) < 1 if and only if all the eigenvalues of F[k] are in the (−1, 1) interval.
Also note that F[k] = IM 2 − µA[k] + µ2 B[k], where A and B are positive semi-definite, and
µ > 0. Note that all the eigenvalues of F[k] are smaller than 1 if and only if IM 2 − F[k] is
positive definite, or equivalently, A[k] − µB[k] is positive definite. This is obtained when A[k]
is positive definite, which is obtained when Cx [k] is positive definite, and the step-size satisfies
µ<

1
λmax(A−1 [k]B[k])

[7, Appendix A]. Next, note that all the eigenvalues of F[k] are larger than

−1 if and only if 2IM 2 −µA[k]+µ2 B[k] is positive definite. It follows from [7, Appendix A] that
1
this is satisfied for all µ > 0 when H[k] has no real positive eigenvalues, and for µ < λmax(H[k])

when H[k] has at least one real positive eigenvalue. Combining the conditions used in the proof
yields the theorem.
G. Proof of Theorem 3
For k ∈ N0 , n ∈ N, using definition (D.1) and the periodicity of g[n], Cx [n], and σv2 [n], the
instantaneous MSE (8) can be written as
n

2
E |e [n · N0 + k] | = E h̄k [n]

2
cx [k]

o


+ 2Re gH [k]Cx [k]E{h̄k [n]}

+ kg [k]k2cx [k] + σv2 [k].

(G.1)

When Thm. 2 is satisfied, then the adaptive filter is mean convergent and mean-square stable as
in Defs. 2–3, respectively. Thus, letting n → ∞ in (G.1) yields
n

lim E |e [n · N0 +k] |2 = lim E h̄k [n]

n→∞

n→∞

2
cx [k]

+kg [k]k2cx [k] +σv2 [k].

o

n
o
+2Re gH[k]Cx [k] lim E{h̄k [n]}
n→∞

(G.2)

Next, recalling the definitions of ak and bk [n] stated in (F.4), it follows from (F.5) that

August 3, 2017

DRAFT

27

∀k ∈ N0 , for n → ∞ it holds that
lim E

n→∞

n

h̄k [n]

2
q

o

= lim E
n→∞

(a)

n

= lim E
n→∞

h̄k [n]

2
LF
k,k q

h̄k [n]

2
LF
k,k

n

o

+µ2 · aTk q − 2µ· lim bTk [n] q
n→∞
o
+µ2 · aTk q,
q

o
n
2
where (a) follows since lim bk [n] = 0M ×1 . Thm. 2 guarantees that lim E h̄k [n] q and
n→∞
n
o n→∞
2
lim E h̄k [n] LF q both exist and are finite. Thus, from the linearity of the weighted Eu-

n→∞

k,k

clidean norm [2, Eq. (23.31)] we have
lim E

n→∞

∀k ∈ N0 . Setting q = IM 2 − LF
k,k
lim E

n→∞

n

h̄k [n]

o

n

h̄k [n] (I 2 −LF )q = µ2 aTk q,
M
k,k

−1

cx [k] in (G.3) yields

2
cx [k]

o

2

= µ2 aTk IM 2 −LF
k,k

(a)

=

−1

(G.3)

cx [k]

NX
0+k−1
2
µ
zT((l))N LF
l+1,k
0
l=k

IM 2 −LF
k,k

−1

cx [k],

(G.4)

where (a) follows from (F.4a). Plugging (G.4) and (D.5) into (G.2) yields (12).
R EFERENCES
[1] S. Haykin. Adaptive Filter Theory. Prentice Hall, 2003.
[2] A. H. Sayed. Adaptive Filters. IEEE Press, 2008.
[3] B. Widrow, J. M. McCool, M. G. Larimore, and C. R. Johnson, Jr. “Stationary and nonstationary learning characteristics
of the LMS adaptive filter”. Proc. IEEE, vol. 64, no. 8, Aug. 1976, pp. 1151-1162.
[4] W. A. Gardner. “Learning characteristics of stochastic-gradient-descent algorithms: A general study, analysis, and critique”.
Signal Processing, vol. 6, no. 2, Apr. 1984, pp. 113–133.
[5] A. Feuer and E. Weinstein. “Convergence analysis of LMS filters with uncorrelated Gaussian data”. IEEE Trans. Acoustics,
Speech, and Signal Processing, vol. 33, no. 1, Jan. 1985, pp. 1373–1385.
[6] M. Rupp. “The behavior of LMS and NLMS algorithms in the presence of spherically invariant processes”. IEEE Trans.
Signal Processing, vol. 41, no. 3, Mar. 1993, pp. 1149–1160.
[7] T. Y. Al-Naffouri and A. H. Sayed. “Transient analysis of data-normalized adaptive filters”. IEEE Trans. Signal Processing,
vol. 51, no. 3, Mar. 2003, pp. 639–652.
[8] W. A. Gardner, A. Napolitano, and L. Paura. “Cyclostationarity: Half a century of research”. Signal Processing, vol. 86,
Apr. 2006, pp. 639–697.
[9] W. A. Gardner (Editor). Cyclostationarity in Communications and Signal Processing. IEEE Press, 1994.

August 3, 2017

DRAFT

28

[10] R. W. Heath Jr. and G. B. Giannakis. “Exploiting input cyclostationarity for blind channel identification in OFDM systems”.
IEEE Trans. Signal Processing, vol. 47, no. 3, Mar. 1999, pp. 848–856.
[11] P. Yang, Y. Lan, X. Li, Y. Zhang, Y. Tao, and L. You. “Taming cross-technology interference for WiFi and ZigBee
coexistence networks”. IEEE Trans. Mobile Computing, vol. 15, no. 4, Apr. 2016, pp. 1009–1021.
[12] J. Campbell, A. Gibbs, and B. Smith. “The cyclostationary nature of crosstalk interference from digital signals in multipair
cable - Part I: Fundamentals”. IEEE Trans. Communications, vol. 31, no. 5, May 1983, pp. 629–637.
[13] X. Hong, Z. Chen, C. X. Wang, S. A. Vorobyov, and J. S. Thompson. “Cognitive radio networks”. Vehicular Technology
Magazine, vol. 4, no. 4, Dec. 2009, pp. 76–84.
[14] H. C. Ferreira, L. Lampe. J. Newbury, and T. G. Swart. Power Line Communications - Theory and Applications for
Narrowband and Broadband Communications over Power Lines. Wiley and Sons, Ltd., 2010.
[15] F. J. Cañete, J. A. Cortés, L. Dı́ez, and J. T. Entrambasaguas. “Analysis of the cyclic short-term variation of indoor power
line channels”. IEEE Journal on Selected Areas in Communications, vol. 24, no. 7, Jul. 2006, pp. 1327–1338.
[16] M. Nassar, J. Lin, Y. Mortazavi, A. Dabak, I. H. Kim, and B. L. Evans. “Local utility power line communications in the
3–500 kHz band: channel impairments, noise, and standards”. IEEE Signal Processing Magazine, vol. 29, no. 5, Aug.
2012, pp. 116–127.
[17] D. C. McLernon. “Analysis of LMS algorithm with inputs from cyclostationary random processes”. Electronics Letters,
vol. 27, no. 2, Jan. 1991, pp. 136–138.
[18] J. H. Reed and T. C. Hsia. “The performance of time-dependent adaptive filters for interference rejection”. IEEE Trans.
Acoustics, Speech, and Signal Processing, vol. 38, no. 8, Aug. 1990, pp. 1373–1385.
[19] J. Zhang, K. M. Wong, Z. Q. Luo, and P. C. Ching. “Blind adaptive FRESH filtering for signal extraction”. IEEE Trans.
Signal Processing, vol. 47, no. 5, May 1999, pp. 1397–1402.
[20] O. A. Yeste-Ojeda and J. Grajal. “Adaptive-FRESH filters for compensation of cycle-frequency errors”. IEEE Trans.
Signal Processing, vol. 58, no. 1, Jan. 2010, pp. 1–10.
[21] N. J. Bershad, E. Eweda, and J. C. M. Bermudez. “Stochastic analysis of the LMS and NLMS algorithms for cyclostationary
white Gaussian inputs”. IEEE Trans. Signal Processing, vol. 62, no. 9, May 2014, pp. 2238–2249.
[22] N. J. Bershad, E. Eweda, and J. C. M. Bermudez. “Stochastic analysis of an adaptive line enhancer/canceler with a
cyclostationary input”. IEEE Trans. Signal Processing, vol. 64, no. 1, Jan. 2016, pp. 104–109.
[23] N. Shlezinger, K. Todros, and R. Dabora. “Adaptive LMS-type filter for cyclostationary signals”. Proceedings of the
International Symposium on Wireless Communications Systems (ISWCS), Poznan, Poland, Sep. 2016, pp. 37–41.
[24] N. Shlezinger, K. Todros, and R. Dabora. “Adaptive filtering based on time-averaged MSE for cyclostationary signals”.
IEEE Trans. Communications, vol. 65, no. 4, Apr. 2017, pp. 1746–1761.
[25] G. B. Giannakis. “Cyclostationary signal analysis”. Digital Signal Processing Handbook, CRC Press, 1998, pp. 17.1–17.31.
[26] R. P. Agarwal and J. Popenda. “Periodic solutions of first order linear difference equations”. J. Math. Comput. Modelling,
vol. 22, no. 1, Jul. 1995, pp. 11–19.
[27] W. Greene. Econometric Analysis. Prentice Hall, 2003.
[28] N. Shlezinger and R. Dabora. “On the capacity of narrowband PLC channels”. IEEE Trans. Communications, vol. 63, no.
4, Apr. 2015, pp. 1191–1201.

August 3, 2017

DRAFT

29

[29] IEEE Standards Association. “P1901.2/D0.09.00 draft standard for low frequency (less than 500 kHz) narrow band power
line communications for smart grid applications”. Jun. 2013.
[30] M. Katayama, T. Yamazato and H. Okada. “A mathematical model of noise in narrowband power line communications
systems”. IEEE Journal on Selected Areas in Communications, vol. 24, no. 7, Jul. 2006, pp. 1267–1276.
[31] C. D. Meyer. Matrix Analysis and Applied Linear Algebra. Society for Industrial and Applied Mathematics, 2000.
[32] K. B. Petersen and M. S. Pedersen. The Matrix Cookbook. Technical University of Denmark, 2008.

August 3, 2017

DRAFT

