1

Diffusion leaky LMS algorithm: analysis and implementation
Lu Lu, Haiquan Zhao*
School of Electrical Engineering, Southwest Jiaotong University, Chengdu, China.
https://doi.org/10.1016/j.sigpro.2017.05.015
COMMENT: This paper has been published in Signal Processing

ABSTRACT—The diffusion least-mean square (dLMS) algorithms have attracted much attention owing to its robustness for
distributed estimation problems. However, the performance of such filters may change when they are implemented for suppressing
noises from speech signals. To overcome this problem, a diffusion leaky dLMS algorithm is proposed in this work, which is
characterized by its numerical stability and small misadjustment for noisy speech signals when the unknown system is a lowpass
filter. Finally, two implementations of the leaky dLMS are introduced. It is demonstrated that the leaky dLMS can be effectively
introduced into a noise reduction network for speech signals.
Keywords: Distributed estimation; Diffusion LMS; Leaky LMS; Noisy speech signal.

1. Introduction

Since the simple structure and low computational burden, the least-mean square (LMS) algorithm has become a widely used
adaptive filter. However, in practice, it is well known that direct implementation of the conventional LMS algorithm can be
problematic. Such problems can be outlined as follows. (i) Numerical problem, caused by inadequacy of excitation in the input
signal; (ii) stagnation behaviour, often occur in low input signal scenarios, making the performance degradation of the algorithm. To
solve these problems, the leaky LMS algorithm [1] as its variants were proposed in diverse fields, such as active noise control (ANC)
[2], channel estimation [3], and nonlinear acoustic echo cancellation (NLAEC) [4].
Recently, to estimate some parameters of interest from the data collected at nodes distributed over a geographic region, several
distributed estimation algorithms were developed, including incremental [5-6], and diffusion algorithms [7-10], etc. In the
incremental strategy, a cyclic path is required the definition over the nodes, and this technique is also sensitive to link failures [11].
On the other hand, the diffusion method, is widely used because of its ease of implementation. In this strategy, each node

E-mail addresses: lulu@my.swjtu.edu.cn (L. Lu), hqzhao@home.swjtu.edu.cn (H. Zhao)

2
communicates with a subset of its neighbours, which achieves more data from their neighbours with moderate amount of
communications. Particularly, in [12], two versions of the diffusion LMS (dLMS) algorithms, adapt-then-combine (ATC) and
combine-then-adapt (CTA) were proposed, based on the different orders of adaptation and combination steps. Note that the ATC
method outperforms the CTA method in all cases, and better performance can be achieved if the measurement are shared. Due to its
merit, the ATC version of the dLMS algorithm (ATC dLMS) has been introduced to subband adaptive filter [13], information
theoretic learning (ITL) [14] to obtain improved performance.
Motivated by these considerations, in this paper, we proposed a leaky dLMS algorithm with two combination strategies-ATC and
CTA, resulting in ATC leaky dLMS and CTA leaky dLMS algorithms. Very recently, a distributed incremental leaky LMS was
proposed to surmount the drift problem of dLMS algorithm [15]. Unfortunately, it is derived from the incremental approach, which
prohibits its practical applications. Compared with the existing algorithms, the leaky dLMS algorithm is derived by minimizing the
instantaneous leaky objective function rather than the mean square error cost function. Moreover, it has a superiority performance
via ATC strategy.
Notation: Throughout this paper, we use {}
⋅ to represent a set, | ⋅ | denotes absolute value of a scalar, (⋅)T denotes transposition,

E{}
⋅ denotes expectation, tr{}
⋅ denotes trace operator, || ⋅ ||22 denotes l2-norm, and {}
⋅ o is the real value of parameter. Besides, we
use boldface and normal letters to denote the random quantities and deterministic quantities, respectively.

2. Diffusion LMS strategies

Consider a network of N sensor nodes distributed over a geographic area. At each time instant i, each sensor node k ∈ {1, 2,..., N }
has access to the realization {d k (i ), uk .i } of some zero-mean random process {d k (i ), u k .i } , and d k (i ) is a scalar and u k .i is an
regression vector with length M. Suppose these measurements follow a standard model given by:

d k (i) = u k ,i wo + ν k (i)

(1)

where wo ∈ ℂ M ×1 is the unknown parameter vector, and ν k (i ) is the measurement noise with variance σ v2, k . Here, we assume that
u k ,i and ν k (i ) are spatially independent and independent identically distributed (i.i.d.), and ν k (i ) is independent of u k ,i .

The dLMS algorithm is obtain by minimizing a linear combination of the local mean square error (MSE):

J kloc ( w) =

∑a

l ,k

l∈ N k

E | el ,i |2 =

∑a

l ,k

E | d l (i) − ul ,i w |2

(2)

l ∈N k

where N k is the set of nodes with which node k shares information (including k itself). The weighting coefficient {al , k } are real,

3
N

non-negative, and satisfy: ∑ al , k = 1 .
l =1

The dLMS algorithm obtains the estimation via two steps, adaptation and combination. According to the order of these two steps,
the diffusion LMS algorithm is classified into the ATC dLMS and CTA dLMS algorithms. The updation equation of ATC dLMS
can be expressed as

ϕk ,i = wk ,i −1 + µ ∑ cl , k ulT,i (d l (i ) − ul ,i wk ,i −1 ) (adaptation)

l ∈N k

(combination)
 wk ,i = ∑ al , k ϕl ,i
l∈ N k


(3)

where µ is the step size (learning rate), and ϕk is the local estimates at node k. The weighting coefficients {cl ,k } is the real,
non-negative, satisfying the condition cl , k = al , k = 0 if l ∉ N k . Similarly, the CTA dLMS algorithm can be given as

ϕk ,i −1 = ∑ al , k wl ,i −1
(combination)

l ∈N k

T
 wk ,i = ϕk ,i −1 + µ ∑ cl , k ul ,i (d l (i) − ul ,iϕ k ,i −1 ) (adaptation).
l ∈N k


(4)

3. Proposed diffusion leaky LMS algorithm

In this section, the leaky dLMS is proposed to address the two problems mentioned above. For each node k, we seek an estimate
of wo by minimizing the following cost function:

∑a

J kloc ( w) =

l ,k

E | el ,i |2 + γ wT w

l∈ N k

=

(5)

∑a

2

l ,k

T

E | dl (i ) − u l ,i w | + γ w w

l∈ N k

where γ>0 is the leakage coefficient, and w is the estimate of wo . Using the steepest descent algorithm, yields

∇ w J kloc ( w) =

∂{E | el ,i |2 +γ wT w}

∑a

l ,k

∂w

l ∈N k

.

(6)

Therefore, the updating of proposed algorithm for estimating wo at node k can be derived from the steepest descent recursion

wk ,i = wk ,i −1 − µ ∑ al , k

∂{E | el ,i |2 +γ wT w}
∂w

l ∈N k

|wk .i−1 .

(7)

Under the linear combination assumption [12,14], let us define the linear combination wk ,i at node k as
wk ,i −1 =

∑a

l ,k

l ∈N k

ϕl ,i −1 .

(8)

4
Introducing (8) to (7), we can obtain an iterative formula for the intermediate estimate

ϕl ,i = ϕl ,i −1 − µ
≈ ϕl ,i −1 − µ

∂E | el ,i |2 +γ wT w
∂w
∂E | el ,i |2 +γ wT w

∂w
= (1 − γµ )ϕl ,i −1 + µ ulT.i el ,i .

|wk .i−1
|wl .i−1

(9)

Hence, we obtain the leaky dLMS algorithm by transforming the steepest-descent type iteration of (7) into a two-step iteration


∂E | ek ,i |2 +γ wT w
=
w
−
|wk .i −1
ϕ
µ
 k ,i
k , i −1
∂w

 wk ,i = ∑ al , k ϕl ,i .
l∈ N k


(10)

In adaptation step of (10), the local estimate ϕk ,i −1 is replaced by linear combination wk ,i −1 . Such substitution is reasonable,
because the linear combination contains more data information from neighbor nodes than ϕk ,i −1 [12]. Then, we extend the leaky
dLMS algorithm to its ATC and CTA forms.
ATC leaky dLMS algorithm:

●

ϕk ,i = (1 − µγ ) wk ,i −1 + µ ∑ cl , k ulT,i ( dl (i) − ul ,i wk ,i −1 ) (adaptation)

l ∈N k

(combination).
 wk ,i = ∑ al , k ϕl ,i
l∈ N k


(11)

ϕk ,i −1 = ∑ al , k wl ,i −1
(combination)

l ∈N k

T
 wk ,i = (1 − µγ )ϕ k ,i −1 + µ ∑ cl , k ul ,i (d l (i ) − ul ,iϕk ,i −1 ) (adaptation).
l ∈N k


(12)

CTA leaky dLMS algorithm:

●

4. Simulation Results

In the following, we evaluate the performance of the proposed algorithm for noise reduction network with 20 nodes. The topology
of the network is shown in Fig. 1. The unknown vector of interest is a lowpass filter of order M=5, whose coefficients and
normalized frequency response are shown in Fig. 2(a-b). The regressors are zero-mean white Gaussian distributed with covariance
matrices Ru , k = σ uk2 I , with σ uk2 shown in Fig. 2(c). The performance of the different algorithms is measured in terms of the network
N

mean square deviation (MSD) MSD = 10 log10 (∑ || wl − wo || / N ) . All network MSD curves are obtained by ensemble averaging
l =1

over 50 independent trials.

5

Fig. 1. Network topology.

5.1 Example 1: Gaussian input
In this example, the Gaussian signal with zero mean and unit variance is employed as the input signal, al , k = 1/ nk is used for

{al , k } , and cl , k = 1 / nk (uniform, [16]), where nk is the degree of node k. The signal-to-noise ratio (SNR) of each node is SNR=0dB,

Amplitude
(dB)

Amplitude

corresponding to the high background noise power.
0.5

-0.5

0.064

-0.008

0

1

1.5

0.064

0.443

2

2.5

3

3.5

(a)

-0.008

4

4.5

5

Tap

-5

(b)
-10
0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Regressor
variances

Normalized Frequency (×π rad/sample)
1
0.8
0.6
0.4
0.2
0

(c)

2

4

6

8

10

12

14

16

18

20

Node number
Fig. 2. (a) Coefficients of the filter. (b) Frequency responses of the unknown system. (c) Regressor variances.

0
baseline

MSD(dB)

-5

-10

-15
node 7

node 20

node 5

CTA leaky dLMS
ATC leaky dLMS

-20
0.1

0.2

0.3

0.4

0.5

0.6

0.7

Step size

Fig. 3. Network MSD of the proposed algorithms versus different step sizes

6

-6
CTA leaky
CTA leaky
CTA leaky
ATC leaky
ATC leaky
ATC leaky

-8

MSD(dB)

-10

dLMS(γ=0.001)
dLMS(γ=0.002)
dLMS(γ=0.1)
dLMS(γ=0.001)
dLMS(γ=0.002)
dLMS(γ=0.1)

-12

-14

-16

-18

-20
0

200

400

600

800

1000

Number of iterations

Fig. 4. Network MSD of the proposed algorithms versus different γ (µ=0.08).

Fig. 3 shows the steady-state network MSD of the leaky dLMS with different step-size. As one can see, the steady-state MSDs of
the proposed algorithms increase as the step sizes increase. For µ>0.668 (node 5), the CTA leaky dLMS fails to work. Besides, by
using (31), we can obtain the theory upper bounds of step sizes from different nodes. The calculating data indicate that, the upper
bound of step size from node 1 is 1.1 (maximum upper bound among 20 nodes), and upper bound of step size from node 7 is about
0.4 (minimum upper bound among 20 nodes). To guarantee all the nodes work well, µ must be µ<0.4 to ensure that the stability. As
a result, we select µ=0.08 in the following simulations. Then, we investigate the performance of the algorithm in different γ, as
shown in Fig. 4. It can be easily observed that the CTA/ATC leaky dLMS algorithm is not sensitive to this choice, but it turns out that
the best option is γ=0.002. Fig. 5 displays a comparison of MSE from the proposed algorithm and existing algorithms. One can see
that the proposed algorithm outperforms the conventional dLMS algorithms in terms of steady-state error. After the about iteration
50, all the algorithm reach steady state. In addition, it is obviously shown from these curves that the proposed algorithms reach low
misadjustment (−14dB and −18dB) in about iteration 60, whereas the CAT dLMS and ATC dLMS algorithms finally settle at a
network MSD value of −12dB and −16dB. All the ATC-based algorithms achieve improved performance as compared with
CTA-based algorithms, with the similar initial convergence rate.

7

-6
CTA dLMS(µ=0.1)
CTA leaky dLMS(µ=0.08,γ=0.002)

-8

ATC dLMS(µ=0.1)
ATC leaky dLMS(µ=0.08,γ=0.002)

MSD(dB)

-10
-12

-14
-16
-18

-20
0

100

200

300

400

500

600

700

800

900

1000

Number of iterations

Fig. 5. Network MSD learning curves for Gaussian input.

5.2 Example 2: Speech input signal
In the second example, the input signal is a speech input (See Fig. 6). In piratical application, different nodes have different
speech signals. Therefore, we consider a input signal generated as u u , k = σ uk2 i speech input . The environment and parameters of all
the algorithms are same as those of example 1.
In Fig. 7, it demonstrates a comparison of network MSD performance of different algorithms. In this cases, superior performance
of the proposed algorithms are obtained because that the leaky method can stabilize the system [1]. The proposed algorithm, which
adopts the ATC and CTA approaches, reduces the misadjustment when the speech is selected as the input signal. Meanwhile, the fast
convergence rate is obtained. Furthermore, it can be obviously observed from this figure that the misadjustment for the ATC version
is less than that of CTA version. To further demonstrate the performance of proposed algorithm, Fig. 8 depicts the waveforms of the
simulation result at node 14 ( σ uk2 = 0.35 ). The ATC leaky dLMS algorithm, inherits the advantages of the leaky algorithm, and
achieves a stability performance in the presence of strong disturbances.
5

Amplitude

3
1
-1
-3
-5

0

100

200

300

400

500

600

700

Number of iterations

Fig. 6. Input speech signals

800

900

1000

8

20
CTA dLMS(µ=0.1)
CTA leaky dLMS(µ=0.08,γ=0.002)
ATC dLMS(µ=0.1)
ATC leaky dLMS(µ=0.08,γ=0.002)

15
10

MSD(dB)

5
0
-5
-10
-15
-20

0

100

200

300

400

500

600

700

800

900

1000

Number of iterations

Fig. 7. Network MSD learning curves for speech input.
3

(a)
1

Amplitude

-1
-3

0

100

200

300

400

500

600

700

800

900

1000

3

(b)
1
-1
-3

0

100

200

300

400

500

600

700

800

900

1000

Number of iterations
Fig. 8. Waveforms of the simulation result at node 14. (a) Noisy speech (green line: noise signal with SNR=0dB; blue line: clean speech). (b) Filtered speech with
the ATC leaky dLMS algorithm.
Table 1 Computation time
Algorithms

Computation
Time(s)

ATC dLMS

0.6962

ATC leaky dLMS
CTA dLMS

1.0026
1.1945

CTA leaky dLMS

1.5701

To evaluate the computational burden, we have measured the average run execution time of the algorithm on a 2.1-GHz AMD
processor with 8GB of RAM, running Matlab R2013a on Windows 7. The computation time of the algorithms is outlined in Table.
1. One can see that the ATC-dLMS algorithm is the fastest one among these distributed algorithms. The proposed algorithm slightly
increases the execution time, but the final performance is lower than other algorithms. The CTA algorithms achieve the slower than
the ATC algorithms, still with an affordable computation time.

9

5. Conclusion

In this paper, a leaky dLMS algorithm with ATC and CTA strategies has been proposed for distributed estimation. The proposed
algorithm updates the local estimation based on leaky method, which can not only maintain the stability for distributed system, but
also enhance the performance for noise suppression. Simulation results in the context of noise-reduction system show that the
proposed algorithm achieves an improved performance as compared with existing algorithms.

Acknowledgement
This work was partially supported by National Science Foundation of P.R. China (Grant nos: 61271340, 61571374, 61433011).
The first author would also like to acknowledge the China Scholarship Council (CSC) for providing him with financial support to
study abroad (No. 201607000050).

References
[1] K. Mayyas, T. Aboulnasr, Leaky LMS algorithm: MSE analysis for Gaussian data, IEEE Trans. Signal Process. 45 (4) (1997) 927–934.
[2] J. Cheer, S.J. Elliott, Active noise control of a diesel generator in a luxury yacht, Appl. Acoust. 105 (2016) 209–214.
[3] D.B. Bhoyar, C.G. Dethe, M.M. Mushrif, A.P. Narkhede, Leaky least mean square (LLMS) algorithm for channel estimation in
BPSK-QPSK-PSK MIMO-OFDM system, International Multi-Conference on Automation, Computing, Communication, Control and
Compressed Sensing., Kottayam, 2013, pp. 623–629.
[4] J.M. Gil-Cacho, M. Signoretto, T.V. Waterschoot, M. Moonen, S.H. Jensen, Nonlinear acoustic echo cancellation based on a
sliding-window leaky kernel affine projection algorithm, IEEE Trans. Audio Speech Lang. Process. 21 (9) (2013) 1867–1878.
[5] L. Li, J.A. Chambers, C.G. Lopes, A.H. Sayed, Distributed estimation over an adaptive incremental network based on the affine projection
algorithm, IEEE Trans. Signal Process. 58 (1) (2010) 151–164.
[6] Y. Liu, W.K.S. Tang, Enhanced incremental LMS with norm constraints for distributed in-network estimation, Signal Process. 94 (2014)
373–385.
[7] N. Takahashi, I. Yamada, A.H. Sayed, Diffusion least-mean squares with adaptive combiners: Formulation and performance analysis, IEEE
Trans. Signal Process. 58 (9) (2010) 4795–4810.
[8] J. Chen, A.H. Sayed, Diffusion adaptation strategies for distributed optimization and learning over networks, IEEE Trans. Signal Process. 60
(8) (2012) 4289–4305.
[9] R. Abdolee, B. Champagne, A.H. Sayed, Estimation of space-time varying parameters using a diffusion LMS algorithm, IEEE Trans. Signal
Process. 62 (2) (2014) 403–418.
[10] R. Abdolee, B. Champagne, A.H. Sayed, Diffusion adaptation over multi-agent networks with wireless link impairments, IEEE Trans.
Mobile Computing 2015 DOI: 10.1109/TMC.2015.2460251.

10
[11] R. Abdolee, B. Champagne, Diffusion LMS strategies in sensor networks with noisy input data, IEEE/ACM Trans. Networking 24 (1) (2016)
3–14.
[12] F.S. Cattivelli, A.H. Sayed, Diffusion LMS strategies for distributed estimation, IEEE Trans. Signal Process. 58 (3) (2010) 1035–1048.
[13] J. Ni, Diffusion sign subband adaptive filtering algorithm for distributed estimation, IEEE Signal Process. Lett. 22 (11) (2015) 2029–2033.
[14] C. Li, P. Shen, Y. Liu, Z. Zhang, Diffusion information theoretic learning for distributed estimation over network, IEEE Trans. Signal
Process. 61 (16) (2013) 4011–4024.
[15] M. Sowjanya, A.K. Sahoo, S. Kumar, Distributed incremental Leaky LMS, International Conference on Communications and Signal
Processing (ICCSP), Melmaruvathur, 2015, pp. 1753–1757.
[16] V.D. Blondel, J.M. Hendrickx, A. Olshevsky, J.N. Tsitsiklis, Convergence in multiagent coordination, consensus, and flocking, in Proc.
Joint 44th IEEE Conf. Decision Control Eur. Control Conf.(CDC-ECC), Seville, Spain, 2005, pp. 2996–3000

