title: FCN Approach for Dynamically Locating Multiple Speakers
authors: Hodaya Hammer Shlomo E. Chazan Jacob Goldberger Sharon Gannot
text: 
  
  DeepPrognosis: Preoperative Prediction of Pancreatic Cancer Survival and Surgical Margin via Contrast-Enhanced CT Imaging
      Jiawen Yao1 Yu Shi2 Le Lu1 Jing Xiao3 Ling Zhang1
      December 30, 2023
  =========================================================================================================================
  
  
  
  
  In this paper, we present a deep neural network-based online multi-speaker localisation algorithm. Following the W-disjoint orthogonality principle in the spectral domain, each TF bin is dominated by a single speaker, and hence by a single DOA. A fully convolutional network is trained with instantaneous spatial features to estimate the DOA for each TF bin. The high resolution classification enables the network to accurately and simultaneously localize and track multiple speakers, both static and dynamic. Elaborated experimental study using both simulated and real-life recordings in static and dynamic scenarios, confirms that the proposed algorithm outperforms both classic and recent deep-learning-based algorithms.
  
  
  
  
  
  § INTRODUCTION
  
  
      
  Locating multiple sound sources recorded with a microphone array in an acoustic environment is an essential component in various cases such as source separation and  scene analysis.  The relative location of a sound source with respect to a microphone array is generally given in the term of the DOA of the sound wave originating from that location.  DOA estimation and tracking are generating interest lately, due to the need for far-field enhancement and recognition in smart home devices.
  
  In real-life environments, sound sources are captured by the microphones together with acoustic reverberation. While propagating in an acoustic enclosure, the sound wave undergoes reflections from the room facets and from various objects. These reflections deteriorate speech quality and, in extreme cases, its intelligibility. Furthermore, reverberation increases the time dependency between speech frames, making source DOA estimation a very challenging task.
  
  
  
  A plethora of classic signal processing-based approaches have been proposed throughout the years for the task of broadband DOA estimation.  
  
  The MUSIC algorithm <cit.> applies a subspace method that was later adapted to the challenges of speech processing in <cit.>.
  
  
  
  The SRP-PHAT algorithm <cit.> uses generalizations of  cross-correlation methods for DOA estimation. 
  
  
  These methods are still widely in use. However, in high reverberation enclosures, their performance is not satisfactory. 
  
  
  
  Supervised learning methods encompass an advantage for this task since they are data-driven. Deep-learning methods can be trained to find the DOA in different acoustic conditions. Moreover, if a network is trained using rooms with different acoustic conditions and multiple noise types, it can be made robust against noise and reverberation even for rooms which were not in the training set.
  
  Deep learning methods have recently been proposed for sound source localization.
  In <cit.> simple feed-forward DNN were trained using GCC-based audio features, demonstrating improved performance as compared with classic approaches. Yet, this method is mainly designed to deal with  a single sound source at a time. In 
  <cit.> the authors trained a DNN for multi-speaker DOA estimation. In high reverberation conditions, however, their performance is not satisfactory. In <cit.>  time domain features were  used and they have shown performance improvement in highly-reverberant enclosures. 
  
  In <cit.>, a CNN based classification method was applied in the STFT domain for broadband DOA estimation, assuming that only a single speaker is active per time frame. The phase component of the STFT coefficients of the input signal were directly provided as input to the CNN. This work was extended in <cit.> to estimate multiple speakers' DOA, and has shown high DOA classification performances. In this approach, the DOA is estimated for each frame independently.
  The main drawback of most DNN-based approaches, however, is that they only use low-resolution supervision, namely only time frame or even utterance-based labels. In speech signals, however, each time-frequency bin is dominated by a single speaker, a property referred to as WDO <cit.>.  Adopting this model results in higher resolution, which might be beneficial for the task at hand.
  This model was also utilized in <cit.> for speech separation  where the authors recast the separation problem as a DOA classification at the TF domain. A FCN was trained using spatial features to infer the DOA at every TF bin. Although the DOA resolution was relatively low, it was sufficient for the separation task at low reverberation conditions. 
  When applying this method in high-reverberation enclosures or to separate adjacent speakers, a performance degradation was observed.
  
  
  
      
  In this work, we present a multi-speaker DOA estimation algorithm. According to the WDO property of speech signals <cit.>, each TF bin is dominated by (at most) a single speaker. This TF bin can therefore be associated with a single DOA. 
  We use instantaneous spatial cues from the microphone signals. These features are used to train a FCN to infer the DOA of each TF bin. The FCN is trained to address  various reverberation conditions. The TF-based classification facilitates the tracking ability for multiple moving speakers. In addition, unlike many other supervised domains, the DOA domain lacks a standard benchmark. The LOCATA dataset <cit.> was recorded in one room with relatively low reverberation (RT_60=0.55). Furthermore, a training dataset with high TF labels is not publicly available.  Therefore,  we generated training and test datasets simulating various real-life scenarios. 
  We tested the proposed method on simulated data, using publicly available RIR recorded in a real room <cit.>, as well as real-life experiments. 
  We show that the proposed algorithm significantly outperforms state-of-the-art competing methods.
  
  
  
  
  
  
  
  
  The main contribution of this paper is the A high resolution TF-based approach that improves  DOA estimation performances w.r.t. the SOTA approaches, which are frame-based, and enables simultaneously tracking multiple moving speakers. 
           
           
  
  
  
  
   
  
  
  § MULTIPLE SPEAKER' LOCATION ALGORITHM
  
  
  
  
  
   §.§ Time-frequency features
  
  
  Consider an array with M microphones acquiring a mixture of N speech sources in a reverberant environment. The i-th speech signal  s^i(t) propagates through the acoustic channel before being acquired by the m-th microphone:
  
      z_m(t)=∑_i=1^N s^i(t)*h_m^i(t),  m =1,…,M
      ,
  
  where h_m^i is the RIR relating the i-th speaker and the m-th microphone. In the STFT domain (<ref>) can be written as (provided that the frame-length is sufficiently large w.r.t. the filter length):
  
      z_m(l,k)=∑_i=1^N s^i(l,k)h_m^i(l,k),
  
  where l and k, are the time frame and the frequency indices,  respectively.
  
  
  
  
  The STFT (<ref>) is complex-valued and hence comprises both spectral and phase information. It is clear that the spectral information alone is insufficient for DOA estimation. It is therefore a common practice to use the phase of the TF representation of the received microphone signals, or their respective phase-difference, as they are directly related to the DOA in non-reveberant environments.
  
  
  We decided to use an alternative feature, which is generally independent of the speech signal and is mainly determined by the spatial information. For that, we have selected the RTF <cit.> as our feature, since it is known to encapsulate the spatial fingerprint for each sound source. Specifically, we use the iRTF, which is the bin-wise ratio between the m-th microphone signal and the reference microphone signal z_ref(l,k):
  
      iRTF(m,l,k) = z_m(l,k)/z_ref(l,k).
  
  Note, that the reference microphone is arbitrarily chosen. Reference microphone selection is beyond the scope of this paper (see <cit.> for a reference microphone selection method). The input feature set extracted from the recorded signal is thus a 3D tensor ℛ: 
  
      ℛ(l,k,m) = [ℜ𝔢(iRTF(m,l,k)), 
                  ℑ𝔪(iRTF(m,l,k))
                  ] .
  
  The matrix ℛ  is  constructed from L × K bins, where L is the number of time frames and K is the number of frequencies. Since the iRTF are normalized by the reference microphone, it is excluded from the features. Then for each TF bin (l,k), there are  P=2(M-1) channels, where the multiplication by 2 is due to the real and imaginary parts of the complex-valued feature.
   For each TF bin the spatial features were normalized to have a zero mean and a unit variance.
  
  
  
  Recall that the WDO assumption <cit.> implies that each TF bin (l,k) is dominated by a single speaker. 
  Consequently, as the speakers are spatially separated, i.e. located at different DOA, each TF bin is dominated by a single DOA. 
  
   Our goal in this work is to accurately estimate the speaker direction at every TF bin from the given mixed recorded signal. 
  
  
  
   §.§ FCN for DOA estimation
   
  We formulated the DOA estimation  as a classification task by discretizing the DOA range. The resolution was set to 5^∘, such that the DOA candidates are in the set Θ={0^∘,5^∘,10^∘,…,180^∘}.
  
  
  Let D_l,k be a r.v. representing the active dominant direction, recorded at bin (l,k). Our task boils down to deducing the conditional distribution of the discrete set of DOA in Θ for each TF bin, given the recorded mixed signal:
  
      p_l,k(θ) = p( D_l,k=θ|ℛ),   θ∈Θ.
  
  For this task, we use a DNN. 
  The network output is an L× K× |Θ| tensor, where |Θ| is the cardinality of the set Θ.
  Under this construction of the feature tensor and output probability tensor, a pixel-to-pixel approach for mapping a 3D  input `image', ℛ and a 3D output `image', p_l,k(θ), can be utilized.  
   An FCN is used to compute (<ref>) for each TF bin. The pixel-to-pixel method is beneficial in two ways. First, for each TF bin in our input image the network estimates the DOA distribution separately. Second, the TF supervision is carried out with the spectrum of the different speakers. The FCN hence takes advantage of the spectral structure and the continuity of the sound sources in both the time and frequency axes. These structures contribute to the pixel-wise classification task, and prevent discontinuity in the DOA decisions over time.  
  In our implementation, we used a U-net architecture, similar to the one described in <cit.>. We dub our algorithm TF-DOAnet.
  
  The input to the network is the feature matrix ℛ (<ref>). 
  In our U-net architecture, the input  shape is (L,K,P) where K=256 is the number of frequency bins, L=256 is the number of frames, and P=2M-2 where M is the number of microphones. The overlap between successive STFT frames is set to 75%. This allows to improve the estimation accuracy of the RTF, by averaging  three consecutive frames both in the numerator and denominator of (<ref>), without  sacrificing the instantaneous nature of the RTF.
  
  
  
  TF bins in which there is no active speech are non-informative. Therefore, the estimation is carried out only on speech-active  TF bins. As we assume that the acquired signals are noiseless, we define a TF-based VAD as follows:
  
      VAD(l,k) = {[              1 |z_ref(l,k)|≥ϵ;              0           o.w.;                ].  ,
  
  where ϵ is a threshold value. In noisy scenarios, we can use a robust SPP estimator instead of the VAD <cit.>.
  
  The task of DOA estimation only requires time frame estimates. Hence, we aggregate over all active frequencies at a given time frame to obtain a frame-wise probability:
  
      p_l(θ)=1/K'∑_k=1^K p_l,k(θ)VAD(l,k).
  
  where K' is the number of active frequency bands at the l-th time frame.
  We thus obtain for each time frame a posterior distribution over all possible DOA. 
  If the number of speakers is known in advance, we can choose the directions corresponding to the highest posterior probabilities. If an estimate of the number of speakers is also required, it can be determined by applying a suitable threshold. Figure <ref> summarizes the TF-DOAnet in a block diagram. 
  
  
  
  
  
  
  
  
  
  
   §.§ Training phase
  
  
  The supervision in the training phase is based on the WDO assumption in which  each TF bin is dominated by (at most) a single speaker.
  The training is based on simulated data generated by a publicly availble RIR generator software[Available online at ], efficiently implementing the image method <cit.>. A four microphone linear array was simulated with (8,8,8) cm inter-microphones distances. Similar microphone inter-distances were used in the test phase. For each training sample, the acoustic conditions were randomly drawn from one of the simulated rooms of different sizes and different reverberation levels RT_60 as described in Table <ref>. The microphone array was randomly placed in the room in one out of six arbitrary positions. 
  
  
   
  
  
   
  
  
  
  
  For each scenario, two clean signals were randomly drawn from the WSJ1 database <cit.> and then convolved with RIR corresponding to two different DOA in the range Θ={0,5,…,180}. 
  The sampling rate of all signals and RIR was set to 16KHz. The speakers were positioned in a radius of r=1.5m from the center of the microphone array. To enrich the training diversity, the radius of the speakers was perturbed by a Gaussian noise with a variance of  0.1 m. The DOA of each speaker was calculated w.r.t. the center of the microphone array.
  
   The contributions of the two sources were then summed with a random SIR selected in the range of SIR∈[-2,2] to obtain the received microphone signals. Next, we calculated the STFT of both the mixture and the STFT of the separate signals with a frame-length  K=512 and an overlap of 75% between two successive frames. 
  
  We then constructed the audio feature matrix R as described in Sec. <ref>. In the training phase, both the location and a clean recording of each speaker were known, hence they could be used to generate the labels. For each TF bin (l,k), the dominating speaker was determined by:
  
      ←_i | s^i(l,k)h_ref^i(l,k) |.
  
  The ground-truth label D_l,k is the DOA of the dominant speaker. The training set comprised four hours of recordings with 30000 different scenarios of mixtures of two speakers. It is worth noting that as the length of each speaker recording was different, the utterances could  also include non-speech or single-speaker frames. 
  The network was trained to minimize the cross-entropy between the correct and the estimated DOA. The cross-entropy cost function was summed over all the images in the training set.
  The network was implemented in Tensorflow with the ADAM optimizer <cit.>.
  The number of epochs was set to be 100, and the training stopped after the validation loss increased for 3 successive epochs. The mini-batch size was set to be 64 images. 
  
  
  
  
  
  
  § EXPERIMENTAL STUDY
  
  
  
  
   §.§ Experimental setup
  
  
  In this section we evaluate the TF-DOAnet and compare its performance to classic and DNN-based algorithms.
  To objectively evaluate the performance of the TF-DOAnet, we first simulated 2 unfamiliar test rooms. Then, we tested our TF-DOAnet with real RIR recordings in different rooms. Finally, a real-life scenario with fast moving speakers was recorded and tested.    
  
  
  
  For each test scenario, we selected two speakers from the test set of the WSJ1 database <cit.>, placed them at two different angles between 0^∘ and 180^∘ relative to the microphone array, at a distance of either 1m or 2m. The signals were generated by convolving the signals with RIR corresponding to the source positions and with either simulated or recorded acoustic scenarios.
  
  
  Performance measures Two different measures to objectively evaluate the results were used: the MAE and the localization accuracy (Acc.). The MAE, computed between the true and estimated DOA for each evaluated acoustic condition, is given by 
  
      MAE(^∘)=1/N· C∑_c=1^Cmin_π∈ S_N∑_n=1^N|θ_n^c-θ̂_π(n)^c|  ,
  
  where N is the number of simultaneously active speakers and C is the total number of speech mixture segments considered for evaluation for a specific acoustic condition. In our experiments N=2. The true and estimated DOA for the
  n-th speaker in the c-th mixture are denoted by θ_n^c and θ̂_n^c, respectively.
  
  The localization accuracy is given by
  
      Acc.(%)=Ĉ_acc./C×100
  
  where Ĉ_acc. denotes the number of speech mixtures for which the localization of the speakers is accurate. We considered the
  localization of speakers for a speech frame to be accurate if the distance between the true and the estimated DOA for all the speakers was less than or equal to 5^∘.
  
   Compared algorithms
  We compared the performance of the TF-DOAnet with two frequently used baseline methods, namely the MUSIC and SRP-PHAT algorithms. In addition, we  compared its performance  with the  CMS-DOA estimator <cit.>.[the trained model is available here <https://github.com/Soumitro-Chakrabarty/Single-speaker-localization>]
  To facilitate the comparison, the MUSIC pseudo-spectrum was computed for each frequency sub-band and for each STFT time frame, with an angular resolution of 5^∘ over the entire DOA domain. Then, it was averaged over all  frequency subbands to obtain a broadband pseudo-spectrum followed by averaging over all the time frames L. Next, the two DOA with the highest values were selected as the final DOA estimates. Similar post-processing was applied to the computed SRP-PHAT pseudo-likelihood for each time frame.
  
  
  
  
   §.§ Speaker localization results
  
  
  
    
  Static simulated scenario
  We first generated a test dataset with simulated RIR. 
  
  Two different rooms were used, as described in Table <ref>. For each scenario, two speakers (male or female) were randomly drawn from the WSJ1 test database, and placed at two different DOA within the range {0,5,…,180} relative to the microphone array. The microphone array was similar to the one used in the training phase. Using the RIR generator, we generated the RIR for the given scenario and convolved it with the speakers' signals. 
  
  The results for the TF-DOAnet compared with the competing methods  are depicted in Table <ref>. The tables shows that the deep-learning approaches outperformed the classic approaches. The TF-DOAnet achieved very high scores and outperformed the DNN-based CMS-DOA algorithm in terms of both MAE and accuracy.
  
  
  
  
  
  
  
  
    
  Static real recordings scenario
  The best way to evaluate the capabilities of the TF-DOAnet is testing it with real-life scenarios. For this purpose, we first carried out experiments with real measured RIR from a multi-channel impulse response database <cit.>. The database comprises  RIR measured in an acoustics lab for three different reverberation times of RT_60 = 0.160, 0.360, and 0.610 s. The lab dimensions are 6× 6× 2.4 m. 
  
  The recordings were carried out with different DOA positions in the range of    [0^∘, 180^∘], in steps of 15^∘. The sources were positioned at distances of 1 m and 2 m from the center of the microphone array.
  The recordings were carried out with a linear microphone array consisting of 8 microphones with three different microphone spacings. For our experiment, we chose
  the [8, 8, 8, 8, 8, 8, 8] cm setup. 
  In order to construct an array setup identical to the one in the training phase, we selected a sub-array of the four center microphones out of the total 8 microphones in the original setup. Consequently, we used a ULA with M = 4 elements with an inter-microphone distance of 8 cm.
  
  The results for the TF-DOAnet compared with the competing methods are depicted in Table <ref>. Again, the TF-DOAnet outperforms all competing methods, including the CMS-DOA algorithm. Interestingly, for the 1 m case, the best results for the TF-DOAnet were obtained for the highest reverberation level, namely RT_60=610 ms, and for the 2 m case, for  RT_60=360 ms. While surprising at first glance, this can be explained using the following arguments. There is an accumulated evidence that reverberation, if properly addressed, can be beneficial in speech processing, specifically for multi-microphone speech enhancement and source extraction <cit.> and  for speaker localization <cit.>. In reverberant environments, the intricate acoustic propagation pattern constitutes a specific “fingerprint” characterizing the location of the speaker(s). When reverberation level increases, this fingerprint becomes more pronounced and is actually more informative than its an-echoic counterpart. An inference methodology that is capable of extracting the essential driving parameters of the RIR will therefore improve when the reverberation is higher. If the acoustic propagation becomes even more complex, as is the case of high reverberation and a remote speaker, a slight performance degradation may occur, but as evident from the localization results, for sources located 2 m from the array, the performance for RT_60=610 ms was still better than the performance for RT_60=160 ms.
  
  
  
  
  
  
    
  Real-life dynamic scenario
  To further evaluate the capabilities of the TF-DOAnet, we also carried out real dynamic scenarios experiments.
  
  The room dimensions are 6 × 6 × 2.4 m. The room reverberation level can be adjusted and we set the RT_60 at two levels,  390 ms and 720 ms, respectively. The microphone array consisted of 4 microphones with an inter-microphone spacing of 8 cm.  The speakers walked naturally on an arc at a distance of about 2.2 m from the center of the microphone array. For each RT_60 two experiments were recorded. The two speakers started at the angles 20^∘ and 160^∘ and walked until they reached 70^∘ and 100^∘, respectively, turned around and walked back to their starting point. This was done several times throughout the recording. Figure <ref> depicts the real-life experiment setup and Fig. <ref> depicts a schematic diagram of the setup of this experiment. The ground truth labels of this experiment were measured with the Marvelmind indoor 3D tracking set.[<https://marvelmind.com/product/starter-set-ia-02-3d/>]
  
  Figures <ref> and <ref> depict the results of the two experiments. It is clear that the TF-DOAnet outperformed the CMS-DOA algorithm, especially for the high RT_60 conditions. Whereas the CMS-DOA fluctuated rapidly, the TF-DOAnet output trajectory was smooth and noiseless.
  
  	
  
  
  
  
  
  
  
  
  
  
  
   
  
  
   
  
  
  
   §.§ Ablation study
  
  In our implementation, we used the real and imaginary part of the RTF (<ref>). Other approaches might be beneficial. For example, in <cit.>, the cos and the sin of the phase of the RTF were used. In other approaches, the spectrum was added to the spatial features <cit.>.  
  
  In this section, the different features were tested with the same model. We compared the proposed features with two other features. First, we used the proposed features as described in (<ref>). The second approach was a variant of our approach with the  spectrum added (`TF-DOAnet with Spec.'). The third, used the cos and the sin features as presented in <cit.> (`Cos-Sin').
  All features were crafted from the same training data described in Sec. <ref>. We tested the different approaches in the test conditions described in <ref>. 
  
  First, it is clear that all the features with our high resolution TF model outperformed the frame-based CMS-DOA algorithm, as reported in Table <ref>. This confirms that the TF supervision is beneficial for the task at hand. Second, the proposed features were shown to be better than the Cos-Sin features. Finally, it is very interesting to note that the addition of the spectrum features slightly deteriorated the results for this task.  
  
  
  
  
  
  § CONCLUSIONS
  
  A FCN approach was presented in this paper for the DOA estimation task. Instantaneous RTF features were used to train the model. The high TF resolution facilitated the tracking of multiple moving speakers simultaneously. A comprehensive experimental study was carried out with simulated and real-life recordings. The proposed approach outperformed both the classic and CNN-based SOTA algorithms in all experiments.
  Training and test datasets which represent different real-life scenarios were constructed as a DOA benchmark and will become available after publication.
  
  
  
  
  § BROADER IMPACT
  
  Several modern technologies can benefit from the proposed localization algorithm. We already mentioned the emerging technology of smart speakers in the Introduction. These devices are equipped with multiple microphones and are implementing location-specific tasks, e.g. the extraction of the speaker of interest.
  Of particular interest are socially assistive robots (SARs), as they are likely to play an important role in healthcare and psychological well-being, in particular during non-medical phases inherent to any hospital process. 
  
  The algorithm neither uses the content nor the identity of the speakers and hence does not to violate the privacy of the users. Moreover, since normally speech signal cannot propagate over long distances, the algorithm application is limited to small enclosures. 
  plain
  
  
  
  