Statistical Information Fusion for Multiple-View Sensor
Data in Multi-Object Tracking

arXiv:1702.08641v1 [cs.SY] 28 Feb 2017

Xiaoying Wanga , Reza Hoseinnezhadb,∗, Amirali K. Gostarb ,
Tharindu Rathnayakeb , Benlian Xua , Alireza Bab-Hadiasharb
a Changshu

Institute of Technology, Changshu, Soochow, P. R. of China.
of Engineering, RMIT University, Victoria, Australia.

b School

Abstract
This paper presents a novel statistical information fusion method to integrate
multiple-view sensor data in multi-object tracking applications. The proposed
method overcomes the drawbacks of the commonly used Generalized Covariance Intersection method, which considers constant weights allocated for sensors. Our method is based on enhancing the Generalized Covariance Intersection with adaptive weights that are automatically tuned based on the amount
of information carried by the measurements from each sensor. To quantify information content, Cauchy-Schwarz divergence is used. Another distinguished
characteristic of our method lies in usage of the Labeled Multi-Bernoulli filter
for multi-object tracking, in which the weight of each sensor can be separately
adapted for each Bernoulli component of the filter. The results of numerical
experiments show that our proposed method can successfully integrate information provided by multiple sensors with different fields of view. In such scenarios,
our method significantly outperforms the state of art in terms of inclusion of all
existing objects and tracking accuracy.
Keywords: Multi-sensor fusion, random finite sets, labeled multi-Bernoulli
filter, Cauchy-Schwarz divergence.

∗ Corresponding

author
Email address: rezah@rmit.edu.au (Reza Hoseinnezhad)

Preprint submitted to Elsevier

March 1, 2017

1. Introduction
Statistical sensor fusion methods combine the information received from
multiple sensors to propagate statistical density and estimate the state(s) of
object(s) with enhanced accuracy compared to using a single sensor [1]. The
emergence of new sensors, advanced processing techniques, and improved processing hardware has made real-time multi-sensor fusion increasingly viable and
rapidly evolving. By means of multi-sensor information fusion, multi-object
tracking systems can not only enlarge spatial/temporal surveillance coverage,
but also improve system reliability and robustness [2, 3].
Popular statistical multi-object tracking techniques include Multiple-Hypotheses
Tracking (MHT) [4], Joint Probabilistic Data Association (JPDA) [5], and Random Finite Set (RFS) filters [6]. Recent studies on RFS theory have led to
various filters with different implementations (based on simplifying but reasonable assumptions on the multi-object distribution) such as probability hypothesis density (PHD) filter and its cardinalized version (CPHD) [6], the multiBernoulli filter (MB) and its cardinality-balanced version (CB-MeMBer) [7], and
the newly derived δ-GLMB(Generalized Labeled Multi-Bernoulli) [8, 9] and its
special case, the Labeled Multi-Bernoulli (LMB) filters [10, 11]. A robust version
of multi-Bernoulli filter that can handle unknown clutter intensity and detection
profile has also been introduced in [12]. The most recent family of RFS filters
are based on using labeled random finite set densities which have been shown
to admit conjugacy with the general multiple point measurement set likelihood
and to be closed under the Chapman-Kolmogorov equation [10, 11].
Various implementations of RFS filters based on Sequential Monte Carlo
(SMC) and Gaussian-mixture approximations have been presented. These methods have been utilized to solve different tracking problems in various applications such as track-before-detect visual tracking applications [13–15], and sensor
management in target tracking applications [16–19].
In multi-sensor applications, a common approach is to use the above filters in
each sensor node and fuse the posterior densities that computed locally in every

2

node with each other. The fusion operation can be executed in a distributed or
central manner. In this context, multi-sensor fusion refers to combining several
multi-object posteriors that depending on the type of the filter used in each
sensor node, can have a different mathematical form. To combine the local
posteriors, the most common method proposed in the RFS filtering literature is
using the Generalized Covariance Intersection (GCI) rule. Examples include the
fusion of Poisson multi-object posteriors of multiple local PHD filters [20], i.d.d.
clusters densities of several local CPHD filters [21], multi-Bernoulli densities of
local multi-Bernoulli filters [22], and LMB or δ-GLMB filters [23].
The multi-sensor fusion rules that have been formulated based on the GCIrule in [20–23] all lead to consensus fusion (also called competitive fusion) of
multiple densities. The fused density is formed with more emphasis on objects
that are included in all local posteriors. In many applications, where different sensors have different fields of view, some objects may not be detected by
all sensors. In a consensus/competitive sensor fusion scheme, e.g. using the
GCI-rule, such objects may not be represented well in the fused multi-object
density, and may be lost from the fused multi-object state estimate (i.e. from
the tracking results).
Figure 1 shows an example where multiple targets are detected by a multitransceiver suite. Due to the distance between, and limited field of view of
the sensors, each target is detected only by some sensors. In this scenario,
the multi-sensor fusion solution should be capable of combining complementary
information provided by various sensors. Examples of complementary fusion
solutions include the usage of linear-like complementary filters for attitude estimation [24], using an extended Kalman filter for complementary fusion of
multi-sensor data in mobile robotics [25], and using an unscented Kalman filter
for complementary fusion of multiple Poisson densities from local PHD filters
in robotic applications [26].
This paper presents a novel strategy for combining random set posteriors
from sensor nodes with different and limited fields of view. The proposed sensor
fusion method is fundamentally different from the GCI-rule as used in [20–23], in
3

,_.,

·:}�,

.l..'?

��'

.2-v

.��'

.
.. . .
.· ·. ·.. .:

Figure 1: Due to the far distance between the sensors (transceivers) and their limited field of
view, no target is consistently detected by all sensors. There is no unanimous agreement on
any target between all sensors, and a consensus-based fusion algorithm may fail to include all
target state estimates.

the sense that it performs not only consensus fusion (thus, enhanced accuracy
when sensors are in agreement on the same object), but also complementary
fusion (thus, extended coverage when different sensors cover different regions).
Our method is particularly useful for fusion of LMB posteriors, with which
the GCI-rule leads to separately combining the densities of Bernoulli components (possible objects) with the same label. We give adaptive weights to each
Bernoulli component label, so that the more informative local posteriors for
that label influence the fused density more. The information content of each
local posterior can be quantified using information theoretic divergence.
In a Bayes filtering context, divergence functionals are commonly used to
quantify the expected information gain from prior to posterior densities. The
commonly used divergence functions in the stochastic filtering literature include
the Shannon entropy [27], the Kullback-Leibler (KL) divergence [20, 28, 29], and
Rényi divergence [30, 31, 18] which has been the dominant choice of divergence
in the random set filtering literature. These divergences have significant computational cost and can only be computed analytically for simple cases. Recently,
Cauchy-Schwarz divergence was shown to admit a simple closed-form expression for Poisson multi-object densities [32]. This was followed by an increasing

4

uptake of Cauchy-Schwarz divergence to solve multi-object tracking problems
using labeled random set filters [33–36]. In this work, we use Cauchy-Schwarz
divergence to quantify the information content of each local posterior in relation to each possibly existing object (Bernoulli component of the fused LMB
posterior).
The major contributions of this paper include:
1. A complete framework is proposed for statistical multi-sensor fusion in
multi-object systems, formulated for LMB filters running locally in each
sensor node of the system. The solution can handle both consensus (competitive) fusion of sensors that detect the same object, and complementary
fusion of sensors with limited field of view that cover different parts of the
object state space, or a mix of competitive and complementary fusion in
cases where the sensors’ fields of view overlap and objects can be detected
by more than one sensor.
2. Formulation of Cauchy Schwarz divergence from prior to posterior is presented for each component of an LMB density, in addition to its incorporation into the fusion formula.
3. Detailed Sequential Monte Carlo (SMC) implementation of the entire
framework using LMB filters is presented via a step-by-step algorithm.
The rest of this paper is organized as follows. Section 2 briefly reviews the
necessary background material on labeled multi-Bernoulli filter and CauchySchwarz divergence. The proposed multi-sensor fusion method and its full implementation are then presented in section 3. Numerical experiments are given
in section 4. They demonstrate that in scenarios involving sensors with unlimited fields of view (hence all objects being detectable, and consensus fusion
being suitable), our method performs very similar to the GCI-rule. However,
with sensors having limited (and overlapping) fields of view, our method is capable of tracking all the objects while the state of art method (GCI-rule method)
leads to targets being lost. Section 5 concludes the paper.

5

2. Background
This section briefly presents background material on Bayesian multi-object
tracking, labeled RFS and Cauchy-Schwarz divergence which are necessary for
developing our proposed method.
2.1. Bayesian Multi-Object Tracking
For notational purposes, we use lower-case letters (e.g. x, x) to denote singleobject states and upper-case letters (e.g. X, X) to denote multi-object states.
In order to distinguish labeled states and distributions from unlabeled ones, we
adopt bold letters to represent labeled entities and variables (e.g. X, x, π). Furthermore, blackboard bold letters represent spaces for variables (e.g. X, Z, L).
Assume that at time k, the labeled multi-object state is denoted by Xk ⊂ X
and the multi-object observation is denoted by Zk ⊂ Z where X and Z denote the
space of single-object states and single-object measurements, respectively. Both
Xk and Zk are modeled as random finite sets. In a Bayes multi-object filtering
scheme, the multi-object random set distribution is recursively predicted and
updated.
Let us denote the labeled multi-object prior density (at time k − 1) by
πk−1 (·|Z1:k−1 ), where Z1:k−1 is the collection of finite measurements up to time
k − 1. The multi-object prediction at time k is given by Chapman-Kolmogorov
equation,
Z
πk|k−1 (Xk |Z1:k−1 ) = fk|k−1 (Xk |X)πk−1 (X|Z1:k−1 )δX,

(1)

where fk|k−1 (Xk |X) is the multi-object transition kernel from time k − 1 to
time k and the integral is the labeled set integral defined in [11, 33].
The update step of the Bayes multi-object filter at time k is based on Bayes’
rule and returns the following multi-object posterior:
πk (Xk |Z1:k ) = R

gk (Zk |Xk )πk|k−1 (Xk |Z1:k−1)
,
gk (Zk |X)πk|k−1 (X|Z1:k−1) δX

(2)

where Zk is the set of observations received at time k, and it is comprised of
some measurements each associated with an object (with some objects possibly
6

missed), and some false alarms or clutter. Both the number of object-related
measurements and the number of false alarms randomly vary with time. Hence,
Zk is a random set with its stochastic variations characterized by a multi-object
likelihood function gk (Zk |Xk ).
2.2. Labeled Multi-Bernoulli Filter
To implement the general Bayes’ filtering steps presented in previous section,
for the sake of computational tractability, a specific family of distribution is
assumed for the multi-object state, and the filter is implemented to predict and
update the parameters of that specific multi-object distribution. Labeled multiBernoulli (LMB) is an example of the recently introduced labeled random set
distributions. The Bayes’ filter that operates based on assuming LMB prior and
posterior is called an LMB filter.
The LMB distribution is completely described by its components π = {(r(`) , p(`) (·))}`∈L
where r(`) is the probability of existence of a track with label ` ∈ L, and p(`) (x)
is the probability density of the object’s state x ∈ X conditional on its existence.
The LMB RFS density is given by
X

π(X) = ∆(X)w(L(X)) [p] ,

(3)

where L(X) is the set of all labels extracted from labeled states in X, and


1 if |X| = |L(X)|
(4)
∆(X) ,

0 otherwise,
in which | · | means “the cardinality of”, and
Y

[p]X ,

p(`) (x),

(5)

 Y 1 (`)r(`)
L
(1 − r(`) )
`∈L

(6)

(x,`)∈X

and
w(L)

=

Y

1 − r(i)

i∈L

is the probability of joint existence of all objects with labels ` ∈ L and nonexistence of all other labels [11].
7

In the LMB filter, suppose that the prior is an LMB with parameters
{(r(`) , p(`) (·))}`∈L . In the prediction step, the LMB prior is turned into the
following new LMB density with evolved particles and probabilities of existence
including the LMB birth components:
π+ =

n

(`)

(`)

r+,S , p+,S

o

∪
`∈L

n

(`)

(`)

rB , pB

o

(7)
`∈B

where
(`)

r+,S

= ηS (`) r(`)

(8)

(`)
p+,S

= hpS (·, `)f (x|·, `), p(`) (·)i/ηS (`)

(9)

and
ηS (`) = hpS (·, `), p(`) (·)i.
(`)

(10)
(`)

We denote the predicted LMB parameters by {(r+ , p+ (·))}`∈L+ where L+ =
L ∪ B.
Assume that at the sensor node i, a measurement set denoted by Zi is
acquired, and denote the LMB posterior (locally updated at sensor node i) as
πi (·|Zi ) =

n

o
(`) (`)
ri , pi (·)

.
`∈L+

The parameters of each local LMB posterior are given by [11]:
(`)

ri

w(I+ ,θ) (Z) 1I+ (`)

P

=

(I+ ,θ)∈F (L+ )×ΘI+
(`)

pi (x)

=

1
r (`)

w(I+ ,θ) (Z) 1I+ (`)p(θ) (x, `)

P

(11)

(I+ ,θ)∈F (L+ )×ΘI+

where
w(I+ ,θ) (Z) ∝
p(θ) (x, `)

=

ηZ (`)

(θ)

=

ψZ (x, `; θ)

=

(θ)

w+ (I+ )[ηZ ]I+
(`)
p+ (x)ψZ (x,`;θ)
(θ)
ηZ (`)

(`)

hp+ (x), ψZ (x, `; θ)i

 pD (x,`)g(zθ(`) |x,`) , if
κ(zθ(`) )
 1 − p (x, `),
if
D

8

(12)
θ(`) > 0
θ(`) = 0

and ΘI+ is the space of mappings θ : I+ → {0, 1, . . . , |Z|} such that θ(i) =
θ(i0 ) > 0 implies i = i0 , and the weight term, w+ (I+ ), is given by:
w+ (I+ ) =

Y 

(i)

1 − r+

i∈L+

 Y 1 (`)r(`)
L+
+
(`)

`∈I+

1 − r+

.

(13)

When the filter is implemented using SMC method, the density function of
each component with label (`) is approximated by J (`) number of particles and
weights
(`)

(`)

p

(x) ≈

J
X

(`)

(`)

wj δ(x − xj )

(14)

j=1

where δ(·) is the Dirac delta function. This makes it easy to compute the inner
products and integrals involved in prediction and update equations (integrals
are replaced with sums over particles).
2.3. Cauchy-Schwarz divergence
The Cauchy-Schwarz divergence is based on the Cauchy-Schwarz inequality
for inner products. The Cauchy-Schwarz divergence between the probability
densities f and g of two random vectors with respect to the reference measure
µ is given by:
DCS (f, g) = − ln

hf, giµ
kf kµ kgkµ

(15)

R
where hf, giµ , X f (x)g(x)µdx denotes standard inner product of two densities,
p
and kf kµ , hf, f iµ . Note that DCS (f, g) is symmetric and positive unless
f = g, in which case DCS (f, g) = 0.
Similar to random vector densities, the Cauchy-Schwarz divergence between
two densities π1 and π2 is given by:
DCS (π1 , π2 ) = − ln

hπ1 , π2 iµ
kπ1 kµ kπ2 kµ

(16)

where the norms are computed via set integrals [32]. The computational cost
of the above divergence term is usually too expensive for real-time applications.
However, as Hoang et al. [32] showed, in the special case of two Poisson RFSs,
the Cauchy-Schwarz divergence equals half the squared distance between their
9

intensity functions. Indeed, denoting the two Poisson RFS densities by π1 and π2
and their intensity functions by v1 (·) and v2 (·), the Cauchy-Schwarz divergence
between them is given by:
DCS (π1 , π2 ) =
where kv1 − v2 k2 =

R

K
kv1 − v2 k2
2

(17)

|v1 (x) − v2 (x)|2 dx, and K is the unit of hyper-volume.

3. Multiple-View Sensor Fusion with LMB Filter
Consider an LMB filter that operates on the measurements received from
ns sensors for tracking an unknown number of objects. The predicted LMB is
independent from sensor measurements and can be directly computed by (7)–
(10). In each sensor node i, a local update returns a posterior from the prior
(`)

(`)

(predicted) LMB, denoted by {(ri , pi (·))}`∈L+ . The sensor fusion problem is
focused on how to combine all these local posteriors into a fused LMB posterior
that conveys all the information in a concise yet comprehensive manner.
The most common method for sensor fusion with random set filters is the
Generalized Covariance Intersection (GCI) method. Let us assume that local
posteriors π1 , . . . , πns are fused into the multi-object density π. In GCI method,
the following weighted average of Kullback-Leibler distances from local posteriors to the fused posterior should be minimized [23]:
πfused = arg min
π

where

Pns

i=1

ns
X

ωi DKL (πi , π).

(18)

i=1

s
ωi = 1. The weights {ωi }ni=1
are constant and quantify our confi-

dence on validity of the measurements acquired from each sensor.
When all the local and fused posteriors are LMB, with the above mentioned
parametrization, the following GCI-rule is derived [23, 36]:
R Qns  (`) (`) ωi
dx
i=1 ri pi (x)

ωi R Q

ωi
r̃(`) = Q
(`)
(`) (`)
ns
ns
+
r
p
(x)
dx
1
−
r
i
i
i
i=1
i=1


ωi
Qns
(`)
i=1 pi (x)
(`)

ωi .
p̃ (x) = R Q
(`)
ns
p
(x)
dx
i
i=1

10

(19)

(20)

where ωi is a weight indicating the strength of the emphasis on sensor si in the
Pns
ωi = 1.
fusion process. These weights should be normalized, i.e. i=1
In an SMC implementation, let us denote the predicted LMB parameters by
(`)

(`)
(`)
(`) J+
{(r+ , {w+j , x+j }j=1
)}`∈L+

(`)

(`)

During the update step of each local LMB filter, the particles do not change,
and only their weights evolve. Thus, for each label ` we have,
(`)

Ji

(`)

(`)

xi,j = x`+j .

= J+ ;

In other words, the predicted LMB prior and all the updated LMB posteriors will
have the same particles but with different weights and existence probabilities.
This makes the fusion of the posteriors generated at each sensor straightforward.
Substituting each density with its particle approximation turns the integrals
into weighted sums over the particles, and the fused existence probability is
given by:
PJ+(`) Qns 
r̃(`) =

j=1

Qns 
i=1

(`) J

(`)

i
)}`∈L+ .
and the local LMB posterior at node i by {(ri , {wij , xij }j=1

(`)

1 − ri

i=1

ωi

+

(`)

(`)

ri wi

ωi

PJ+(`) Qns 
j=1

i=1

(`)

(`)

ri wi

ωi .

(21)

The fused densities also take the form of weighted sums of Dirac deltas:
J

p̃(`) (x) =

(`)

+
X

(`)

w̃j

(`)

δ(x − x+j )

(22)

j=1

where

ωi
(`)
wi,j
=
PJ+(`) Qns  (`) ωi
j=1
i=1 wi,j
Q ns 

(`)

w̃j

i=1

(23)

is the fused weight of each fused posterior particle.
Remark 1. By minimizing the average distance from all local posteriors, the
GCI fusion method is considered a consensus-based fusion method. The fused
posterior should be as similar as possible to all local posteriors, hence, would
convey the information that are overlapped (regions of consensus) between all
s
sensors. This is correct regardless of what the constant weights {ωi }ni=1
are as

long as they are non-zero.
11

Remark 2. From equations (21) and (23), it is evident that during the GCI
fusion, parameters of each Bernoulli component with label ` are only fused with
those with the same label. Indeed, the fusion could be implemented separately
for each label, and there is no cross-label computing.
Based on the observation made in Remark 1, GCI fusion in its current form
is consensus fusion. As it was explained earlier (see Figure 1 and the associated
explanation in section 1), consensus fusion cannot be successfully applied to
combine the measurements acquired from sensors with different fields of view.
In the following, we present a novel solution that resolves this issue.
Inspired by the observation that in GCI fusion, for each label, the multisensor data are fused separately (as stated in Remark 2), we suggest that each
weight ωi associated with each sensor si is adaptively tuned separately for each
label `. For this purpose, we propose that for each label `, the weight associated
(`)

with sensor si , denoted by ωi , should be continuously tuned to exponentially
increase with the information content of that sensor relevant to the possible object with label `. To quantify the information content of a sensor measurement,
we look at the Cauchy Schwarz divergence between its local posterior and the
prior, with only label ` considered.
(`)

(`)

Suppose the predicted LMB distribution is π+ = {(r+ , p+ (·))}`∈L , and
at Sensor Si the measurement set Zi is acquired and used in a local LMB
(`)

(`)

update leading to the local posterior πi = {(ri , pi (·))}`∈L . Associate with a
particular label `, there is a prior Bernoulli RFS with its density parametrized
(`)

(`)

(`)

as π+ = {(r+ , p+ (·))} and a local posterior Bernoulli RFS with its density
(`)

parametrized as πi

(`)

(`)

(`)

= {(ri , pi (·))}. The fusion weight ωi

is then given by:

(`)

(`)

ωi
(`)

where Di

Di

∝ exp

(24)

(`)

mini0 ∈[1,ns ] Di0

(`)

(`)

is the Cauchy-Schwarz divergence between π+ and πi . The follow-

ing proposition presents how the divergence between two Bernoulli RFSs can
be computed.
Proposition 1. A first moment approximation of the Cauchy-Schwarz diver12

gence between two Bernoulli random set densities π1 and π2 , parametrized by
{(r1 , p1 (·))} and {(r2 , p2 (·))} respectively, is given by:
1
2
K kr1 p1 (·) − r2 p2 (·)k
2

DCS (π1 , π2 ) u
where kv1 (·) − v2 (·)k2 =

R

(25)

|v1 (x) − v2 (x)|2 dx, and K is the unit of hyper-volume.

Proof. Mahler [6] has shown that in terms of Kullback-Leibler distance, the
closest Poisson density πPoiss. to any RFS density π is the one that matches
the first moment (intensity function) of π. On the other hand, the intensity
function of a Bernoulli RFS with parameters {(r, p(·))} is given by: [6]
v(x) = r p(x).

(26)

Hence, the best first moment approximation of the Bernoulli densities π1 and
π2 are Poisson densities with intensity functions
v1 (x) = r1 p1 (x);

v2 (x) = r2 p2 (x).

(27)

Approximating the two Bernoulli RFSs with their first moment approximations
(Poisson RFSs), the Cauchy-Schwarz divergence between them is then given
by (17). Substituting the above intensity functions in (17) leads to equation (25).

When the LMB filter is implemented using particle approximations for Bernoulli
densities, the inner product integration turns into a weighted sum over particles. Thus, in the context of weight tuning for multi-sensor fusion, the Cauchy(`)

(`)

(`)

(`)

J

(`)

+
Schwarz divergence between the Bernoulli prior π+ = {(r+ , {w+,j , x+,j }j=1
)}

(`)

and a local Bernoulli posterior πi

J

(`)
Di

1 Note

(`)

(`)

(`)

J

(`)

+
= {(ri , {wi,j , x+,j }j=1
)} is given by:1

(`)

+
X
1
(`) (`)
(`) (`)
ri wi,j − r+ w+,j
= K
2
j=1

(`)

2

.

(28)

(`)

that the total number of particles J+ and their locations x+,j are the same in the

prior and the local posterior, only particle weights and existence probabilities are updated.

13

In a step-by-step algorithm, for each possibly existing object with label `,
(`)

we first compute the Cauchy-Schwarz divergences Di
[1, ns ]. We then find the minimum divergence

(`)
Dmin

for all sensor nodes i ∈
(`)

, mini∈[1,ns ] Di

and

compute the non-normalized weights as



(`)
(`)
(`)
Ωi = exp Di
Dmin
and finally normalize the weights as
(`)

(`)

ωi

Ω
= Pn i
s

(`)

i0 =1

.

Ωi

With these label-dependent weights that continuously evolve with time (depending on the worth of sensor data relevant to each possible object label), the local
posteriors are then fused to form a new LMB with the following probabilities of
existence:
PJ+(`) Qns 
j=1

r̃(`) =
Qns

i=1



(`)

1 − ri

ωi(`)

i=1

+

(`)

(`)

ri wi

ωi(`)

PJ+(`) Qns 
j=1

i=1

(`)

(`)

ri wi

ωi(`)

(29)

and the following weights for the particles in the fused Bernoulli component
densities:

ωi(`)
(`)
w
i,j
i=1
=
.
(`)
(`)
PJ+ Qns  (`) ωi
j=1
i=1 wi,j
Qns 

(`)

w̃j

(30)

Remark 3. To maintain tractability, LMB components with extremely small
existence probabilities should be pruned. This is performed only after GCI-fusion
of the local posteriors.
3.1. Step-by-step Algorithm
Algorithm 1 shows the complete pseudo code for our proposed approach
to multiple-view multi-sensor fusion with the LMB filter. Starting with an
LMB prior (which is the fused LMB posterior from previous time), the function
Predict(·) implements the LMB prediction step and the function Update(·)

14

implements each sensor’s local LMB update step. Fusion(·) denotes the multisensor fusion process which is separately detailed in algorithm 2. There, the
parameters of locally updated posteriors for each label are separately fused using GCI-fusion formulae (29)-(30), in which the weights of sensors have been
adaptively tuned according to the information contents of each sensor measurement relevant to the object label.
In Algorithm 1, after the fused LMB posterior π̃ is acquired, Bernoulli components with small existence probabilities are pruned and finally the number of
objects and their states are estimated.
Algorithm 1 LMB filtering with multiple-view multi-sensor fusion
1: procedure LMB Filter(LMB prior π)
2:

π+ ← Predict(π)

3:

for i = 1 : ns do

4:

. Using Eq. (7)-(10).

πi ← Update(π+ , Zi )

. Using Eq. (11)-(13).

5:

end for

6:

s
π̃ ← Fusion({πi }ni=1
, π+ )

7:

Pruning

. Ref. [8, 11]

8:

Estimatation

. Ref. [8, 11]

9:

end procedure

4. Numerical Experiments
We conducted extensive experiments involving various scenarios with different numbers of targets, sensors, target motion models, sensor detection profile
models and sensors field of view. In each experiment, we compared the performance of the proposed sensor fusion method with the state of art method,
in terms of Optimal Sub-Pattern Assignment (OSPA) errors [37]. This section
includes the representative set of our simulation results which show how the
proposed method works in different clutter rates and sensors’ fields of view. All
scenarios share the following parameters.

15

Algorithm 2 Multi-sensor fusion over separate object labels.
ns
, π+ )
1: procedure Fusion({πi }i=1
2:
3:
4:

for ` ∈ L+ do
for i = 1 : ns do
(`)

Di

←CSD(πi , π+ )

. Using Eq. (28)

5:

end for

6:

Dmin ← mini∈[1,ns ] Di

7:
8:

for i = 1 : ns do


(`)
(`)  (`)
Ωi ← exp Di Dmin

9:

end for

(`)

(`)

11:

for i = 1 : ns do
(`)
(`) Pns
(`)
ωi ← Ωi
i0 =1 Ωi0

12:

end for

13:

s
π̃ (`) ←GCI-Fusion({πi , ωi }ni=1
)

10:

(`)

. Normalizing
(`)

. Using Eq. (29)-(30)
14:
15:

end for
end procedure

The targets maneuver in an area of 1600 m × 1200 m. The single target
state x is comprised of its label and unlabeled state. The label is formed as ` =
(kB , iB ) where kB is the birth time of the target and iB is an index to distinguish
targets born at the same time. The unlabeled state is four-dimensional and
includes the Cartesian coordinates of the target and its speed in those directions,
denoted by x = [px ṗx py ṗy ]> . Each target moves according to the NearlyConstant Velocity (NCV) model. With this model, the transition density is
fk|k−1 (xk |xk−1 , `) = N (xk ; Fk xk−1 , Qk ), where the transition matrix is


1 T 0 0




0 1 0 0 
,
Fk = 


0 0 1 T 


0 0 0 1

16

and the covariance matrix is


B

2 
Qk = σw
02


0.1
 ; B=
B
0.1

02



0.001
0.001


.

The variance parameter σw = 5 m/s2 and T is the sampling interval (in our
experiments T = 1 s). The probability of survival is fixed at pS (x, `) = 0.99 and
the LMB component’s pruning threshold is γp = 10−4 . For each sensor si , with
its location denoted by [xsi ysi ]> , each target (if detected) leads to a bearing
and range measurement vector with the measurement noise density given by
N (·; [0 0]> , R) in which R = diag(σθ2 , σr2 ) with σθ2 = 2π/180 rad and σr2 = 10 m2
being the scales of range and bearing noise. Thus, the single target likelihood
function is g(zi |x, `) = N (z; µi (x), R), where


 q
>
px −xsi
µi (x) = arctan
(px −xsi )2 + (py −ysi )2 .
py −ysi

(31)

In all scenarios, the density p(`) (·) of each labeled Bernoulli component in
the filter is approximated by J (`) = 1000 particles. All simulation experiments
were implemented in MATLAB R2015b and ran on an Intel Core i7-4770 CPU
@3.40GHz, and 8 GB memory.
Figure 2 shows a challenging scenario involving 10 targets maneuvering while
being detected by five sensors that limited fields of view (with overlaps). The
targets enter and exit the scene mainly from the sides of the scene, with different
birth/death times and locations. The birth process is simply chosen to have a
single LMB component, hypothesizing that with a constant existence probability
of 0.05, one object may appear at each time step with its location uniformly
distributed within the surveillance area. The five sensors fields of view and a
sample of 10 target trajectories are also shown in Fig. 2. Each sensor has a
60◦ wide field of view, except in studies investigating the effect of the width of
sensor’s field of view (see section 4.3).
4.1. Probability of detection profile
To model the detection profile of each sensor in our numerical studies, we
adopt a model similar to the one presented in [38] for sonar detectors used
17

5 sensor

1200

y coordinate (m)

1000

800

600

400

200
Sensor 1

Sensor 2

Sensor 3

Sensor 4

Sensor 5

0
0

200

400

600

800

1000

1200

1400

1600

x coordinate (m)

Figure 2: A challenging simulated scenario involving 10 targets appearing and disappearing in
different locations and times. Multi-target tracking needs fusion of the multiple sensors that
have different fields of view. Black dots are ground truths targets trajectories with starting
points denoted by

•

and end points by . The red circles are the location estimates at each

iteration. The figure is best viewed in color.

for submarine location estimation. The detection profile of each sensor is not
only range-dependent (decreases with increasing sensor-target distance), but
also angle-dependent and has no detection ability when the targets are outside
the sensor’s field of view.
Assuming a Rayleigh fading signal model, with a threshold Γth , the probability of detection is

pD,i = exp −


Γth
,
1 + c0 f1 (ri )f2 (ψi )

(32)

where c0 is a constant, f1 (ri ) describes the dependence on the propagation distance between sensor and object, which has inversely proportional relationship
p
f1 (ri ) = ri−1 in which ri (x) = (px − xsi )2 + (py − ysi )2 , and f2 (ψi ) specifies the dependence on the propagation angle. Bearing dependence function is
modeled similar to a Butterworth filter,
f2 (ψi ) =

1
ψi 2H
1 + (W
)

,

(33)

where 2W is the 3dB bandwidth corresponding to the sensor’s field of view, and
18

Sensor Location
(1000,0)

P.D. has decreased from ~1 to 0.8

Prob. of Detection

1

Field of View

0.8
0.6
0.4

Outside FoV
P.D. ~ 0.0

0.2
0
0

2000
1000

500

1000

1500

2000

y (meters)

0

x (meters)

Figure 3: Sensor’s detection probability distribution for W = 30◦ .

H is the filter order, and ψi is the i-th sensor-target vertical bearing given by
ψi (x) = arctan

 p −x 
x
si
.
py − ysi

(34)

When a target is outside the FoV of the i-th sensor, the false alarm rate is
pf a,i = exp(−Γth ),

(35)

which can be controlled by adjusting the threshold Γth .
Choices of the above parameters depend on the specific application environment. In our case, we select a higher order value H = 40 to ensure that sensors
have a good detection ability especially for the boundary of each sensor’s field
of view. We also set Γth = 4.6 and c0 = 4.6 × 104 to ensure each sensor’s PD,i
is between 0.99 and 0.01. Figure 3 shows one sensor’s detection probability distribution for W = 30. Applying this detection profile to sensors in the scenario
shown in Fig. 2 leads to different probabilities of detection by different sensors
for the same target. For instance, for a target located at (400,1000), we have
pD,s1 = 0.9221, pD,s2 = 0.9238, pD,s3 = 0.9156, pD,s4 = 0.01 and pD,s5 = 0.01.
4.2. Different clutter analysis
Each measurement set acquired from each sensor also includes Poisson distributed clutter with varying clutter rates. In our first experiment presented
here, we examined the tracking performance of our multi-sensor fusion method
with different clutter rates including λc = 5, λc = 15 and λc = 30. We ran 200
19

70
6 c=5

OSPA (m)(c=100, p=2)

60

6 c=15
6 c=30

50
40
30
20
10
0
5

10

15

20

25

30

35

40

45

50

Time (s)

Figure 4: OSPA distances under low clutter rate λc = 5, middle clutter rate λc = 15 and high
clutter rate λc = 30 (averaged over 200 MC runs.) The figure is best viewed in color.

Monte Carlo repetitions for the same object trajectories but with independently
generated different clutter and measurement noise realizations. We then computed the average OSPA errors with order and cut-off parameters p = 2 and
c = 100 (see [37, Eqs. (3)-(4)] for definition of OSPA and its parameters).
Figures 4 and 5 show the OSPA errors and cardinality statistics for each
case. The OSPA peaks at k = 10, k = 15, k = 30 and k = 35 are due to
the objects’ appearance and disappearance at those times. The results show
that our proposed method exhibit desirable tracking performance that does not
substantially degrade in severely high clutter rate environments. They also
demonstrate that variances of cardinality statistics are small which is evidence
for desirable reliability of detection capability of the proposed tracking solution.
Note that the state of art (Consensus fusion for labeled random set distribution using GCI [23]) failed to track all the existing targets, because at any time
there were targets that were not detected by one or more sensors, hence there
was no consensus between all sensors about them and they were omitted from
the fused results.

20

11
10

Cardinality Statistics

9
8
7

Ground Truth
Mean 6 c =5

6

StDev 6 c =5
Mean 6 c =15

5

StDev 6 c =15

4

Mean 6 c =30
StDev 6 c =30

3
2
5

10

15

20

25

30

35

40

45

50

Time (s)

Figure 5: Cardinality statistics under low clutter rate λc = 5, middle clutter rate λc = 15 and
high clutter rate λc = 30 (averaged over 200 MC runs.) The figure is best viewed in color.

4.3. Comparative Analysis
In order to compare the performance of our method with the state of art GCI
fusion method as introduced in [23] and used in [36, 22], we examined the OSPA
errors and cardinality statistics returned by our proposed sensor fusion method
with GCI fusion. To implement GCI fusion (with fixed and equal weights, as
proposed in [36, 22, 23]), we simply used the bandwidth W = 90◦ and disabled
the weight adaptation part of our code (replacing it with all weights equal to
1/ns ). A bandwidth of W = 90◦ means unlimited sensors’ field of view. Hence,
all sensors should have consensus on detecting an existing target, and the GCI
fusion method (which is consensus-based) should work. We also recorded the
results returned by GCI fusion method in cases where the sensors’ field of view
is limited (to demonstrate that the GCI fusion method fails while ours performs
satisfactorily).
Figures 6 and 7 present the OSPA errors and cardinality statistics of each
method in each scenario, averaged over 200 Monte Carlo experiments in each
case. The following observations can be made from the results presented in
Figures 6 and 7:

21

70
W=30
W=50

60

OSPA (m)(c=100, p=2)

W=70
W=90

50

W=30 consensus fusion [21]
W=90 consensus fusion [21]

40
30
20
10
0
5

10

15

20

25

30

35

40

45

50

Time (s)

Figure 6: OSPA distances with different FoV of sensors while λc = 5 (averaged over 200 MC
runs.). The figure is best viewed in color.

• When the sensors have unlimited field of view (W = 90◦ ), our method and
the state of art GCI fusion method [23] perform similarly. But our method
returns slightly lower estimation errors, because within its sensor fusion
scheme, it prioritizes sensor measurements that convey more information.
• When the field of view is limited (e.g. W = 30◦ in our experiments),
the GCI fusion method [23] fails to detect some targets (it returns cardinality estimates that are far below the ground truth in Figure 7) and
consequently returns large OSPA errors–see Figure 6. This is while with
the same limited field of view, our method returns acceptable cardinality
estimates (detecting all targets in most of the 200 Monte Carlo runs) and
small OSPA errors.
• The performance of our proposed method (in terms of detection - cardinality - and overall OSPA errors) is robust to variations in the field of view.
There is not much difference in data plotted for our method in figures 6
and 7 for different W values.

22

Cardinality Statistics

10

8
Ground Truth
Mean W=30
StDev W=30
Mean W=50
StDev W=50
Mean W=70
StDev W=70
Mean W=90
StDev W=90
Mean W=30 consensus fusion [21]
StDev W=30 consensus fusion [21]
Mean W=90 consensus fusion [21]
StDev W=90 consensus fusion [21]

6

10.5

4

10

2
9.5
20

0
5

10

25

30

15

20

25

30

35

40

45

50

Time (s)

Figure 7: Cardinality statistics with different FoV while λc = 5 (averaged over 200 MC runs.).
The figure is best viewed in color.

5. Conclusions
A novel multiple-view multi-sensor fusion method was introduced for LMB
filters. Through the Cauchy-Schwarz divergence evaluation for each LMB component on all sensors, the proposed method overcomes the drawbacks of commonly used Generalized Covariance Intersection (GCI) method, which only considers a constant weight to each sensor during the whole tracking and fusion
process. With a proper sensor importance weights selection scheme, a stepby-step SMC fusion implementation algorithm was detailed with LMB filters
running on each sensor node. Numerical experiments involving a challenging
multi-target tracking scenario, showed that our method can properly fuse information received from sensors with different fields of view, while the state of the
art sensor fusion method fails.

Acknowledgment
This project was supported by the Australian Research Council through
ARC Discovery grant DP160104662, as well as National Nature Science Foundation of China grant 61673075.
23

References
References
[1] M. E. Liggins, D. L. Hall, J. Llinas., Handbook of multisensor data fusion:
theory and practice, second edition, CRC Press, 2008.
[2] D. L. Hall, S. A. H. McMullen, Mathematical Techniques in Multisensor
Data Fusion (Artech House Information Warfare Library), Artech House,
Inc., Norwood, MA, USA, 2004.
[3] C. Fantacci, F. Papi, Scalable multisensor multitarget tracking using the
marginalized δ–GLMB density, IEEE Signal Processing Letters 23 (6)
(2016) 863–867. doi:10.1109/LSP.2016.2557078.
[4] D. Reid, An algorithm for tracking multiple targets, IEEE Transactions on
Automatic Control 24 (6) (1979) 843–854. doi:10.1109/TAC.1979.1102177.
[5] S. Blackman, R. Popoli, Design and Analysis of Modern Tracking Systems,
Artech House, 1999.
[6] R. P. S. Mahler, Advances in Statistical Multisource-Multitarget Information Fusion, Artech House, 2014.
[7] B. T. Vo, B. N. Vo, A. Cantoni, The cardinality balanced multi-target
multi-Bernoulli filter and its implementations, IEEE Transactions on Signal
Processing 57 (2) (2009) 409–423. doi:10.1109/TSP.2008.2007924.
[8] B. N. Vo, B. T. Vo, D. Phung, Labeled random finite sets and the Bayes
multi-target tracking filter, IEEE Transactions on Signal Processing 62 (24)
(2014) 6554–6567. doi:10.1109/TSP.2014.2364014.
[9] F. Papi, B. N. Vo, B. T. Vo, C. Fantacci, M. Beard, Generalized labeled multi-Bernoulli approximation of multi-object densities,
IEEE Transactions on Signal Processing 63 (20) (2015) 5487–5497.
doi:10.1109/TSP.2015.2454478.

24

[10] B. T. Vo, B. N. Vo, Labeled random finite sets and multi-object conjugate
priors, IEEE Transactions on Signal Processing 61 (13) (2013) 3460–3475.
doi:10.1109/TSP.2013.2259822.
[11] S. Reuter, B. T. Vo, B. N. Vo, K. Dietmayer, The labeled multi-Bernoulli
filter, IEEE Transactions on Signal Processing 62 (12) (2014) 3246–3260.
doi:10.1109/TSP.2014.2323064.
[12] B. T. Vo, B. N. Vo, R. Hoseinnezhad, R. P. S. Mahler, Robust multiBernoulli filtering, IEEE Journal of Selected Topics in Signal Processing
7 (3) (2013) 399–409.
[13] B. N. Vo, B. T. Vo, N. T. Pham, D. Suter, Joint detection and estimation
of multiple objects from image observations, IEEE Transactions on Signal
Processing 58 (10) (2010) 5129–5141. doi:10.1109/TSP.2010.2050482.
[14] R. Hoseinnezhad, B.-N. Vo, B.-T. Vo, D. Suter, Visual tracking of numerous
targets via multi-Bernoulli filtering of image data, Pattern Recognition
45 (10) (2012) 3625–3635.
[15] R. Hoseinnezhad, B. N. Vo, B. T. Vo, Visual tracking in background subtracted image sequences via multi-Bernoulli filtering, IEEE Transactions
on Signal Processing 61 (2) (2013) 392–397.
[16] A. K. Gostar, R. Hoseinnezhad, A. Bab-Hadiashar, Multi-Bernoulli sensor
control via minimization of expected estimation errors, IEEE Transactions
on Aerospace and Electronic Systems 51 (3) (2015) 1762–1773.
[17] A. K. Gostar, R. Hoseinnezhad, A. Bab-Hadiashar, Robust multibernoulli sensor selection for multi-target tracking in sensor networks, IEEE Signal Processing Letters 20 (12) (2013) 1167–1170.
doi:10.1109/LSP.2013.2283735.
[18] A. K. Gostar, R. Hoseinnezhad, A. Bab-Hadiashar, Multi-Bernoulli sensorselection for multi-target tracking with unknown clutter and detection profiles, Signal Processing 119 (2016) 28–42. doi:10.1016/j.sigpro.2015.07.007.
25

[19] H. G. Hoang, B. T. Vo, Sensor management for multi-target tracking via multi-Bernoulli filtering, Automatica 50 (4) (2014) 1135–1142.
doi:10.1016/j.automatica.2014.02.007.
[20] G. Battistelli, L. Chisci, C. Fantacci, A. Farina, R. P. S. Mahler, Distributed
fusion of multitarget densities and consensus PHD/CPHD filters, in: Proceedings of SPIE, Signal Processing, Sensor/Information Fusion, and Target Recognition, Vol. 9474, Baltimore, 2015. doi:10.1117/12.2176948.
[21] G. Battistelli, L. Chisci, C. Fantacci, A. Farina, A. Graziano, Consensus CPHD filter for distributed multitarget tracking, IEEE Journal of Selected Topics in Signal Processing 7 (3) (2013) 508–520.
doi:10.1109/JSTSP.2013.2250911.
[22] W. Bailu, Y. Wei, R. Hoseinnezhad, L. Suqi, K. Lingjiang, Y. Xiaobo, Distributed fusion with multi-Bernoulli filter based on generalized covariance
intersection, IEEE Transactions on Signal Processing 65 (2017) 242–255.
[23] C. Fantacci, B. N. Vo, B. T. Vo, G. Battistelli, L. Chisci, Consensus labeled
random finite set filtering for distributed multi-object tracking, ArXiv eprintshttp://arxiv.org/abs/1501.01579.
[24] L. Benziane, A. E. Hadri, A. Seba, A. Benallegue, Y. Chitour, Attitude estimation and control using linearlike complementary filters: Theory and experiment, IEEE Transactions on Control Systems Technology 24 (6) (2016)
2133–2140. doi:10.1109/TCST.2016.2535382.
[25] J. Simanek, M. Reinstein, V. Kubelka, Evaluation of the EKFbased estimation architectures for data fusion in mobile robots,
IEEE/ASME Transactions on Mechatronics 20 (2) (2015) 985–990.
doi:10.1109/TMECH.2014.2311416.
[26] J. Gan, M. Vasic, A. Martinoli, Cooperative multiple dynamic object tracking on moving vehicles based on sequential monte carlo probability hypothesis density filter, in: 2016 IEEE 19th International Conference on Intel26

ligent Transportation Systems (ITSC), IEEE, Rio de Janeiro, 2016, pp.
2163–2170. doi:10.1109/ITSC.2016.7795906.
[27] K. J. Hintz, E. S. McVey, Multi-process constrained estimation, IEEE
Transactions on Systems, Man, and Cybernetics 21 (1) (1991) 237–244.
doi:10.1109/21.101154.
[28] G. Battistelli, L. Chisci, Kullback-Leibler average, consensus on probability densities, and distributed state estimation with guaranteed stability,
Automatica 50 (3) (2014) 707–718. doi:10.1016/j.automatica.2013.11.042.
[29] G. Battistelli, L. Chisci, C. Fantacci, A. Farina, B. N. Vo, Average
Kullback-Leibler divergence for random finite sets, in: 2015 18th International Conference on Information Fusion, Fusion 2015, 2015, pp. 1359–1366.
[30] B. Ristic, B. N. Vo, Sensor control for multi-object state-space estimation using random finite sets, Automatica 46 (11) (2010) 1812–1818.
doi:10.1016/j.automatica.2010.06.045.
[31] B. Ristic, B. N. Vo, D. Clark, A note on the reward function for PHD
filters with sensor control, IEEE Transactions on Aerospace and Electronic
Systems 47 (2) (2011) 1521–1529. doi:10.1109/TAES.2011.5751278.
[32] H. G. Hoang, B. N. Vo, B. T. Vo, R. P. S. Mahler, The Cauchy-Schwarz
divergence for poisson point processes, IEEE Transactions on Information
Theory 61 (8) (2015) 4475–4485.
[33] M. Beard, B. T. Vo, B. N. Vo, S. Arulampalam, Sensor control for multitarget tracking using Cauchy-Schwarz divergence, in: Fusion 2015, Washton, D.C, 2015, pp. 937–944.
[34] M. Beard, B.-T. Vo, B.-N. Vo, S. Arulampalam, Void probabilities and
Cauchy-Schwarz divergence for generalized labeled multi-bernoulli models,
ArXiv e-printshttps://arxiv.org/abs/1510.05532.

27

[35] A. K. Gostar, R. Hoseinnezhad, A. Bab-Hadiashar, Multi-bernoulli sensor
control using Cauchy-Schwarz divergence, in: FUSION 2016, Heidelberg,
2016, pp. 651–657.
[36] M. Jiang, W. Yi, L. Kong, Multi-sensor control for multi-target tracking
using Cauchy-Schwarz divergence, ArXiv e-printshttp://arxiv.org/abs/
1603.08355.
[37] D. Schuhmacher, B. T. Vo, B. N. Vo, A consistent metric for performance
evaluation of multi-object filters, IEEE Transactions on Signal Processing
56 (8 I) (2008) 3447–3457. doi:10.1109/TSP.2008.920469.
[38] S. Zhou, P. Willett, Submarine location estimation via a network of
detection-only sensors, IEEE Transactions on Signal Processing 55 (6)
(2007) 3104–3115. doi:10.1109/TSP.2007.893970.

28

