Abstract
This paper proposes a novel audio-visual tracking approach that exploits constructively audio and visual modalities in order to estimate trajectories of multiple people in a joint state space. The tracking problem is modeled using a sequential Bayesian filtering framework. Within this framework, we propose to represent the posterior density with a Gaussian Mixture Model (GMM). To ensure that a GMM representation can be retained sequentially over time, the predictive density is approximated by a GMM using the Unscented Transform. While a density interpolation technique is introduced to obtain a continuous representation of the observation likelihood, which is also a GMM. Furthermore, to prevent the number of mixtures from growing exponentially over time, a density approximation based on the Expectation Maximization (EM) algorithm is applied, resulting in a compact GMM representation of the posterior density. Recordings using a camcorder and microphone array are used to evaluate the proposed approach, demonstrating significant improvements in tracking performance of the proposed audio-visual approach compared to two benchmark visual trackers.
INTRODUCTION
Awareness of the surrounding environment is a prerequisite for interaction between humans and autonomous systems. In particular for Human-Robot Interaction (HRI), knowledge of the directions of sound sources in the surrounding acoustic environment is crucial in order to approach, look at, focus on, and engage with users.
However, in realistic conditions, speech radiated in enclosed environments is subject to reverberation due to reflections off surrounding walls and objects. Dominant early reflections therefore often lead to spurious false detections, whilst late reverberation causes localization errors. Furthermore, human talkers are highly dynamic sources, such that Directions-of-Arrival (DOAs) are both spatiallyand time-varying. Therefore, source tracking approaches constructively exploit temporal models of the source dynamics to estimate and smooth the trajectory of corresponding DOAs. Whilst acoustic source tracking approaches, such as [1] , propagate source trajectories through natural periods of speech inactivity within sentences, prolonged inactivity during dialogues often leads to track deletions.
Nevertheless, especially in the case of robotics, acoustic sensors are often coupled with synchronized camera systems. Therefore, visual information can be exploited constructively to disambiguate
The research leading to these results has received funding from the European Unions Seventh Framework Programme (FP7/2007-2013) under grant agreement no. 609465. the practical challenges of acoustic signal processing. Recent contributions in the audio-visual community therefore utilize features extracted from images to complement audio processing tasks and vice versa. The attention control system for mobile robots in [2, 3] uses separate but parallel audio and visual processing subsystems to identify salient events. In [4] , a visual tracker is used to estimate the positions and velocities of people for blind source separation. Furthermore, a multi-person visual tracker is used for speaker diarization in multi-party dialogues in [5] .
Nevertheless, to fully exploit the information of both modalities, audio-visual fusion -rather than disambiguation -is necessary. An audio-visual tracking system is proposed in [6, 7] that estimates the trajectories of talkers from the joint signals of several distributed cameras and a large microphone array. Nevertheless, joint audiovisual tracking approaches for compact configurations, such as for robot audition, are a novel contribution to the literature. The primary challenge for joint estimation from both modalities is that audiovisual observations are typically highly non-linear, non-Gaussian and multi-modal. The posterior Probability Density Function (pdf) is therefore analytically intractable. Nevertheless, classical approximations for non-linear systems, such as the Extended Kalman Filter (EKF) [8] and Unscented Kalman Filter (UKF) [9] are only valid for the estimation and propagation of unimodal Gaussian densities.
In this paper, we therefore propose to approximate the posterior density by a Gaussian Mixture Model (GMM). To ensure that a GMM representation can be retained over time, the predictive density is approximated by a GMM using the Unscented Transform (UT) [10] . The UT calculates the statistics of a random variable which undergoes a non-linear transformation and builds on the principle that it is easier to approximate a probability distribution than an arbitrary nonlinear function [10, 11] . Furthermore, we propose to use a density interpolation technique to approximate the observation likelihood function by a GMM. To prevent the number of mixtures from growing exponentially over time, a density approximation based on the Expectation Maximization (EM) algorithm is applied, resulting in a compact GMM representation of the posterior pdf.
This paper is structured as follows: Section 2 introduces the audio-visual state and measurement models. The proposed tracking framework is detailed in Section 3. Experimental results are evaluated in Section 4, and conclusions drawn in Section 6.
SYSTEM MODEL
Consider a stream of synchronized sensory inputs, i.e., an image sequence and multi-channel microphone signals. Let t denote the time-step index of the audio-visual stream of data. The audio-visual tracking problem is modeled as a discrete-time Dynamic State Space Model (DSSM). The hidden system state, Xt ∈ R D , with initial distribution, P(X0), represents the two-dimensional (2D) location of N human talkers on the image plane at time step t. The system state evolves over time as an unobserved first-order Markov process according to the state transition model given by the conditional pdf, P(Xt|Xt−1). The audio-visual observations at t, denoted by Zt = {Z a t , Z v t }, are conditionally independent from all other variables given the state Xt and are generated according to the pdf:
where Z a t denotes the auditory observations extracted from the audio frame and Z v t denotes the visual observations extracted from the image at time-step t. The DSSM can also be written as a set of system equations representing the process and observation models as:
( visual observation model) (4) where Vt denotes the process noise that drives the dynamic system through a nonlinear state transition function, f, and where
denote the auditory and visual observation noise corrupting the observation of the system state through the nonlinear observation functions ha and hv, respectively.
The face detector proposed in [12] , and implemented in OpenCV [13] , is used to obtain the visual observations Z v t . However, the face detector is only reliable when a frontal face is presented and uninformative if a person turned his/her face away from the camera. The multi-source localization approach in [14] is used to obtain auditory observations, Z a t , in the form of DOA estimates in azimuth and inclination.
A major challenge in audio-visual processing is the representation of auditory and visual observations in a common space. In this paper, a geometric transformation [3] is therefore applied to the DOAs in order to map the source directions from spherical space to a pixel position on the image plane. Therefore, auditory DOAs, visual facial detections, and the desired system states can be treated in the same mathematical space.
PROPOSED METHOD
Given all available observations Z1:t−1 = {Z1, . . . , Zt−1} up to time-step t − 1, the current posterior at t is predicted using the state transition model and prior, P(Xt−1|Z1:t−1), as:
As the audio-visual observations, Zt, become available at t, the state can be updated using Bayes's theorem, i.e.,
where C = P(Zt|Z1:t−1) = P(Zt|Xt)P(Xt|Z1:t−1) dXt
is a normalization constant. The posterior P(Xt|Z1:t), also referred to as the filtering distribution, is used as the prior distribution at the next time-step t + 1.
Predicted pdf
Assume that the prior pdf at t is given by a GMM with nt−1 components, i.e.,
where
are the mixing weights with
are the mean vectors with
are the covariance matrices with Σ i t−1 ∈ R D×D . The UT enables the propagation of the means and covariance matrices through a nonlinear function. To calculate the statistics of Xt that undergoes a nonlinear transformation, a set of 2D + 1 weighted samples or sigma points, {S (i,j) } 2D,n t−1 j=0,i=1 , are carefully chosen so that they capture the mean and covariance of the system state. A selection scheme that satisfies this requirement is [15] :
, j= 0 (9)
where w (i,j) is the weight associated with the jth sigma point such that Gaussian component in the predicted distribution are approximated using a weighted sample mean and covariance of the sigma points. Finally, the GMM approximating the predicted pdf is given by:
where
w (i,j) S (i,j) , and
Here, Q t is the process noise covariance matrix.
Likelihood
The observation likelihood is a measure for evaluating which configuration of Xt best matches the observations Zt at time-step t. In order to efficiently explore the possible configuration space of Xt and obtain a good approximation for the posterior distribution, m samples {Xi}
, are generated from the Gaussian Mixture (GM) distribution given in (10) . For each sample Xi, the audio-visual likelihood li is computed as:
where lapp is function that evaluate appearance similarity between tracked targets and image regions indicated by the hypothesis state configuration Xi. la is a function that measure the distance between Xi and auditory observations. lv measures the distance between Xi and visual observations. {βi} 3 i=1 are parameters that control the influence of the likelihood functions. We set a large value for β1 since the appearance similarity is more reliable compared to other two distance measures. We note that the likelihood in (11) increases as audio-visual observations become available. However, the likelihood does not increase for missing detections. This property makes the model robust to absence of audio-visual observations, e.g., in time of missed face detections and speech inactivity.
To obtain a continuous approximation of the likelihood given the discrete samples {Xi} m i=1 and associated likelihood {li} m i=1 , the Radial Basis Function (RBF) [16] is used for interpolation. Therefore, a Gaussian kernel is assigned to each sample i = 1, . . . , m, such that the likelihood of Xj induced by the i th kernel is given by
where the sample location, Xi, is used as the mean and the covariance, or the kernel bandwidth, Pi, is set to k-nearest neighbors (KNN) distance, i.e.,
where c is a constant that depends on the number of samples and the dimensionality, I is the D-dimensional identity matrix, and KNN i j (k) is the KNN distance of sample i in the j th dimension. Therefore, the observation likelihood P(Zt|Xt) is approximated by nt m Gaussians, such that
The weight, wi, of kernel i = 1, . . . , nt, is computed by solving the constrained non-negative least square problem [16] :
where A ∈ R m×m is a design matrix with each element (i, j) given by Pi(Xj), the matrix b ∈ R m×1 contains li for each row i = 1, . . . , m, and w = [w1, w2, . . . , wm] is the kernel weight column-vector.
Posterior pdf
Both the predicted density in (10) and the observation likelihood in (14) are represented by GMs. Therefore, the posterior pdf in (6) is equivalent to a product of two GMs, and is therefore equivalent to a GM with an exponentially increasing number of components, i.e.,
where
The resulting density function in (16) is a weighted mixture of Gaussians with nt−1 × nt components.
To mitigate an exponential explosion in the number of components, we first note that typically many of the kernels correspond to weights close to zero and are therefore negligible. To remove stochastically irrelevant kernels, the weighted data EM in [18] is used to fit a GMM to (16) . The optimal number of GM components, mt, for fitting is selected in a principled manner based on the 1 The approach can be naturally extended to more complex approaches to kernel bandwidth selection, discussed in, e.g., [17] . information-theoretic Minimum Message Length (MML) principle. We note that although the weighted data EM is not a general density approximation method, it does preserve significant mode locations and approximates well-separated GMs accurately. Thus, the final posterior distribution at time-step t is given by:
where mt is the number of components [18] .
EXPERIMENTAL SETUP
The camera-microphone setup shown in Fig. 1 is used to gather audio-visual recordings in order to evaluate the performance of the proposed model. The setup consists of a HD video camera and a 32 channel spherical microphone array (Eigenmike). The camera provides video frames with a resolution of 1920 × 1200 pixels at a rate of 25 Frames-Per-Second (FPS). The original audio signals are captured at 48 kHz, and are downsampled to 16 kHz. The acoustic impulse generated by a "clapperboard" is used for audio-visual synchronization. Two scenarios are recorded in order to test the robustness of the proposed tracking model to dynamic scenes. The scenarios are referred to in the following as S1 and S2. Both recordings are taken in the meeting room illustrated in Fig. 1b . In both scenarios, two talkers are speaking whilst moving within the room. The talkers occasionally occlude each other and move in and out of the camera field of view. In S1, the participants take speech turns, whilst the talkers speak simultaneously most of the time in S2. Scenario S1 has 3000 video frames (2 minutes); S2 has 2000 video frames (1 minute and 20 seconds). For both scenarios, the ground truth is identified by hand-annotating in each video frame the talker positions with bounding boxes around faces, 2D locations inferred from the bounding boxes, and target ID.
Face detection yields fully automatic initialization. The algorithm initializes a new tracker for a person that has subsequent detections with overlapping bounding boxes, which are neither occluded nor associated to an already existing tracker. The initial mean positions are drawn from a Gaussian distribution centered at the detection center. The initial size corresponds to the detection size. An appearance model based on RGB color histogram is initialized from the image region around the initial bouding box. Conversely, if not enough detections are found for the same target within T consecutive frames, the track is automatically terminated. Since addition and termination of targets, respectively, increases and decreases the system state dimension, this procedure is performed at the start of each-time step, prior to density approximation. The Hungarian algorithm [19] is used to obtain the optimal face detection to existing target association. The association cost matrix is based on the pairwise similarity of the RGB histogram between existing targets and detections.
The common CLEAR MOT metrics [20] consisting of multiple metrics are used for evaluation. The Multiple Object Tracking Precision (MOTP) evaluates the intersection area over the union area of bounding boxes. The Multiple Object Tracking Accuracy (MOTA) calculates the accuracy composed of false negatives, false positives and identity switching. Furthermore, the Optimal MAss Transfer (OMAT) metric [21] and the Optimal Sub-Pattern Assignment (OSPA) metric [22] are used in order to evaluate tracking performance independent of track labelling. These metrics compute set distances between the ground truth set of objects present in the scene and the set of objects estimated by the tracker.
EXPERIMENTAL VALIDATION
The performance of the proposed tracking model using auditory and visual observations (P-AV) is compared against the proposed tracking model without the auditory data (P-V) and the multi-person visual tracker (OV) in [23] .
The results are summarized in Table 1 , whilst the audio-visual tracking results for the proposed P-AV are illustrated for video frames 351, 352, 451, and 452 of S1 in Fig. 2 . The full videos, Matlab code and additional examples are available online 2 . The comparison between P-AV and P-V highlights that constructive exploitation of the auditory observations leads to improved disambiguation of the states of the moving talkers. An improvement of 11.3 percentage points in MOTA is achieved using P-AV compared to P-V for S1. For S2, the improvement for the audiovisual tracker corresponds to 3.5 percentage points. The difference in results between S1 and S2 can be explained by the difficulty of the scenarios. In S2, the position of the talkers is mostly static, mainly affected by body and head rotations. Therefore, as facial occlusions are rare, face detection is highly reliable. In S1, frequent visual occlusions and obscurations as well as crossing speaker paths lead to less reliable visual detections. Furthermore, the talkers frequently speak simultaneously. Hence, audio DOAs for both talkers are exploited constructively for audio-visual disambiguation in frames where faces are visually occluded.
The results also highlight that the proposed approach outperforms the benchmark OV in both scenarios. This is due to propagation of multiple Gaussians for each target at each time-step by the proposed model, leading to increased robustness to measurement ambiguity. For S1, P-AV therefore results in an improvement of 37.9 and 32.7 percentage points in MOTA and MOTP respectively compared to OV. For S2, an improvement of 45.5 and 32.7 percentage points is achieved.
DISCUSSION AND CONCLUSION
This paper presented a novel approach to audio-visual tracking that jointly utilizes multiple DOAs obtained by sound source localization and facial detections to estimate the trajectories of multiple talkers in pixel space. Within a Bayesian framework, the posterior multisource pdf is propagated using a GMM. To retain the Gaussianity over time, the non-linear observation likelihood function is ap- proximated by a density interpolation technique. Furthermore, to avoid the exponential explosion of Gaussians, an EM algorithm is used to cluster Gaussians, thereby retaining only statistically relevant components. Results based on measurements with a camcorder and eigenmike demonstrate signficant improvements in performance for audio-visual tracking compared to using only visual data.

{'paper_id': '6712155', 'title': 'Audio-visual tracking by density approximation in a sequential Bayesian filtering framework', 's2_url': 'https://api.semanticscholar.org/CorpusID:6712155', 'source': 'S2ORC'}