Separating Varying Numbers of Sources with Auxiliary Autoencoding Loss
Yi Luo Nima Mesgarani
eess.AS cs.LG cs.SD


RTLola Cleared for Take-Off: 
Monitoring Autonomous AircraftThis work was partially supported by the German Research Foundation (DFG) as part of the Collaborative Research Center Foundations of Perspicuous Software Systems (TRR 248, 389792660), and by the European Research Council (ERC) Grant OSARES (No. 683300).
    Jan Baumeister10000-0002-8891-7483Bernd Finkbeiner10000-0002-4280-8441 Sebastian Schirmer2 

Maximilian Schwenger10000-0002-2091-7575 Christoph Torens2

    December 30, 2023
==========================================================================================================================================================================================================================================================================================================================




Many recent source separation systems are designed to separate a fixed number of sources out of a mixture. In the cases where the source activation patterns are unknown, such systems have to either adjust the number of outputs or to identify invalid outputs from the valid ones. Iterative separation methods have gain much attention in the community as they can flexibly decide the number of outputs, however (1) they typically rely on long-term information to determine the stopping time for the iterations, which makes them hard to operate in a causal setting; (2) they lack a “fault tolerance” mechanism when the estimated number of sources is different from the actual number. In this paper, we propose a simple training method, the auxiliary autoencoding permutation invariant training (A2PIT), to alleviate the two issues. A2PIT assumes a fixed number of outputs and uses auxiliary autoencoding loss to force the invalid outputs to be the copies of the input mixture, and detects invalid outputs in a fully unsupervised way during inference phase. Experiment results show that A2PIT is able to improve the separation performance across various numbers of speakers and effectively detect the number of speakers in a mixture.

Index Terms: speech separation, permutation invariant training, auxiliary autoencoding loss



§ INTRODUCTION

Many recent source separation systems assume that the number of active sources in a mixture is known in advance during both training and inference phases <cit.>. Such assumption can be valid when there is additional information, such as visual cue or source locations <cit.>, however for a general blind source separation system it is typically not straightforward to obtain such information, especially in inference phase. In problems such as the separation of shorter streams or chunks in a long mixture, e.g. real-world conversations or music recordings, the number of active sources can vary from chunk to chunk. The estimation of the number of valid sources in a mixture is thus an important problem towards the successful deployment of separation systems into such applications.

Various methods have been proposed to tackle the problem of separating varying numbers of sources. A most simple way is to assume a maximum number of sources in a mixture, which is denoted by N, and let the model to always generate N outputs <cit.>. For mixtures having M sources where M<N, N-M outputs are invalid and need to be properly designed and effectively detected. The invalid outputs are typically forced to have a significantly smaller energy than the valid outputs, and a energy threshold can then be applied to filter out those outputs. Another approach first estimates the speaker embedding for each active source with an output-length-free model, e.g. a sequence-to-sequence generative model, and then performs speaker extraction based on the embeddings <cit.>. A third category of methods perform separation in an iterative way, where in each iteration only one target source is separated from the residual mixture <cit.>. The iteration stops when there is no source left, and the stop time can be determined by either an energy threshold or another trained discriminator. It has been shown that under various circumstances, the number of sources in the mixture can be effectively estimated and the separation performance can be guaranteed.
 
On the other hand, there are various drawbacks in each category of the existing methods. For the fixed-output-number method, the training targets for the invalid outputs are typically low- or zero-energy signals. However, such targets cannot be jointly used with energy-invariant training objectives, such as scale-invariant signal-to-distortion ratio (SI-SDR) <cit.>, which has proven to be a better training objective in many scenarios <cit.>. Moreover, the detection of invalid outputs typically relies on a pre-defined energy threshold, which may cause trouble when the mixture also has a very low energy. For the speaker extraction method, the speaker embeddings are typically estimated at utterance-level and require a long enough context, which makes the method hard to apply in online or causal systems. For methods that utilize additional target speaker enrollments for speaker embedding extraction, the generalization ability on unseen speakers is also limited. For the iterative method, the run-time complexity linearly increases as the number of sources increases, and stop time detection is typically performed at utterance-level as well. When there is noise in the mixture, it is also unclear in which iteration should the noise be cancelled. Moreover, none of the methods have a “fault tolerance” mechanism when the estimated number of sources is different than the actual number. What should the model append to the output if it estimates fewer sources than the actual case? How should the model delete invalid outputs if it generates more? How can such decision process or control flow be effectively incorporated into the training of the model? These questions are important for a practical and robust system.
 
In this paper, we propose a simple training method based on the fixed-output assumption by designing proper training targets for the invalid outputs. We adopt the fixed-output-number assumption as in real-world conversations such as meeting scenarios, the maximum number of simultaneously active speakers is almost always fewer than three <cit.>, thus a maximum number of speakers can typically be pre-assumed. Instead of using low-energy auxiliary targets for invalid outputs, we use the mixture itself as auxiliary targets to force the invalid outputs to perform autoencoding. With the permutation invariant training (PIT) framework <cit.> for speech separation, we refer to it as the auxiliary autoencoding permutation invariant training (A2PIT). A2PIT not only allows the model to perform valid output detection in a self-supervised way without additional modules, but also achieves “fault tolerance” by the “do nothing is better than do wrong things” principle. As the mixture itself can be treated as the output of a null separation model, i.e. perform no separation at all, the auxiliary targets force the model to generate outputs not worse than doing nothing. Moreover, the detection of invalid outputs in A2PIT can be done at frame-level based on the similarity between the outputs and the mixture, which makes it possible to perform single-pass separation and valid source detection in real-time.

The rest of the paper is organized as follows. Section <ref> first makes a quick overview on the PIT framework and then introduces the proposed A2PIT method. Section <ref> provides the experiment configurations and discusses the results. Section <ref> concludes the paper.




§ AUXILIARY AUTOENCODING PERMUTATION INVARIANT TRAINING



 §.§ Permutation Invariant Training


Permutation Invariant Training (PIT) is currently the most widely used training method for speech separation systems. PIT aims at solving the output permutation problem in supervised learning setting, where the correct label permutation of the training targets is unknown with respect to the model outputs. Unlike methods that explicitly use the label permutation information inside the model <cit.>, PIT calculates the loss between the outputs and all possible permutations of the targets, and select the one that corresponds to the minimum loss for back-propagation.

Models using PIT for training often have a fixed number of outputs, which we denote the number as N. For the problem of separating varying numbers of sources where the actual number of sources are M≤ N, N-M auxiliary targets need to be properly designed. A typical way is to use low-energy random Gaussian noise as targets and detect invalid outputs by using a simple energy threshold <cit.>, and it has shown that in certain datasets this energy-based method can achieve reasonable performance.



 §.§ Auxiliary Autoencoding for Invalid Outputs


There are two main issues in the energy-based method for invalid output detection. First, it cannot be jointly used with energy-invariant objective functions like SI-SDR. Second, once the detection of invalid speakers fails and the noise signals are selected as the targets, the outputs can be completely uncorrelated with any of the targets, which is unpreferred for applications that require high perceptual quality or low distortion. We define this as the problem of lacking “fault tolerance” mechanism for unsuccessful separation.

To allow the models to use any objective functions and to have such “fault tolerance” ability, we select the mixture signal itself as the auxiliary targets instead of random noise signals. For mixtures with N outputs and M<N targets, N-M mixture signals are appended to the targets and PIT is applied to find the best output permutation with respect to the targets. The A2PIT loss with the best permutation then becomes:

    ℒ_obj = ℒ_sep + ℒ_AE

where ℒ_sep∈ℝ is the loss for the valid outputs and ℒ_AE∈ℝ is the auxiliary autoencoding loss for the invalid outputs with the input mixture as targets. As autoencoding is in general a much simpler task than separation, proper gradient balancing method should be applied on the two loss terms for successful training. Recall that SI-SDR is defined as:

    SI-SDR(x⃗, x̂⃗̂) = 10 log_10||αx⃗||_2^2/||x̂⃗̂ - αx⃗||_2^2

where α = x̂⃗̂x⃗^⊤ / x⃗x⃗^⊤ corresponds to the optimal rescaling factor towards the estimated signal. Let a ≜x⃗x⃗^⊤, b ≜x̂⃗̂x⃗^⊤ and c ≜x̂⃗̂x̂⃗̂^⊤, we can rewrite the definition as:

    SI-SDR(x⃗, x̂⃗̂)    = 10 log_10(b^2 / a/c-2b^2 / a + b^2 / a) 
       = 10 log_10(1/ac / b^2-1) 
       ≜ 10 log_10(c(x⃗, x̂⃗̂)^2/1-c(x⃗, x̂⃗̂)^2)

where c(x⃗, x̂⃗̂) ≜ b/√(ac) = x̂⃗̂x⃗^⊤/√((x⃗x⃗^⊤)(x̂⃗̂x̂⃗̂^⊤)) is the cosine similarity between x⃗ and x̂⃗̂. The scale-invariance behavior of SI-SDR can be easily observed by the nature of cosine similarity, and SI-SDR(x⃗, x̂⃗̂) → +∞ as |c(x⃗, x̂⃗̂)| → 1. It's easy to see that the second term in |∂ SI-SDR(x⃗, x̂⃗̂) / ∂  c(x⃗, x̂⃗̂)| approaches infinity as |c(x⃗, x̂⃗̂)| approaches 1. Using it for ℒ_AE may let the system to easily collapse to a local minimum which have very high performance on the auxiliary autoencoding term while fail to separate the sources. Based on this concern, we propose an α-skewed SI-SDR (α-SI-SDR):

    α-SI-SDR(x⃗, x̂⃗̂) ≜ 10 log_10(c(x⃗, x̂⃗̂)^2/1+α-c(x⃗, x̂⃗̂)^2)

where the scale of the gradient with respect to the cosine similarity term is controlled by α≥ 0, and α=0 corresponds to the standard SI-SDR. For multiple-speaker utterances, we empirically set α=0.3 for ℒ_AE and α=0 for ℒ_sep. For single-speaker utterances, the training target for separation is equivalent (when there is no noise) or very close (when there is noise) to the input mixture. In this case, we also set α=0.3 for ℒ_sep.



 §.§ Detection of invalid outputs



During inference phase, the detection of invalid outputs can be performed by calculating the similarity, e.g. SI-SDR score, between all outputs and the input mixture, and a threshold calculated from the training set can be used for the decision. For the “fault tolerance” mechanism, the following method is applied for selecting the valid outputs:

    
  * If the estimated number of outputs K is smaller than the actual number M, M-K additional outputs are randomly selected from the N-K remaining outputs.
    
  * If the estimated number of outputs K is larger than the actual number M, M outputs are randomly selected from the K outputs.


Another benefit for A2PIT is that it also allows frame-level detection of the invalid outputs for causal applications. Frame-level detection calculates accumulated similarity starting from the first frame of the outputs, and is able to dynamically change the selected valid outputs as the similarity scores become more reliable. For streaming-based applications that require a real-time playback of the separation outputs, e.g. hearable devices, the change of the output tracks can also be easily done by switching the outputs at frame-level. We leave it as a future task and focus on utterance-level detection in Section <ref>.




§ EXPERIMENTS





 §.§ Dataset


We simulate a single-channel noisy speech separation dataset with the Librispeech dataset <cit.>. 40 hours of training data (), 20 hours of validation data, and 12 hours of test data are generated from the 100-hour training set, development set, and test set, respectively. The number of speakers are evenly sampled between 1 and 4 to make sure the dataset is balanced to the varying numbers of speakers. All utterances are 6-second long with a sample rate of 16k Hz. For utterances with more than one speaker, an overlap ratio between all the speakers is uniformly sampled between 0% and 100% and the speech signals are shifted accordingly. The speech signals are then rescaled to a random absolute energy between -2.5 and 2.5 dB. A noise signal is randomly selected from the 100 Nonspeech Corpus <cit.>, and is repeated if its length is less than 6 seconds. The noise signal is then rescaled to a random absolute energy between -20 and -10 dB. We use both the clean and noisy mixtures to report the performance of A2PIT in the two scenarios.



 §.§ Model configurations


We adopt the time-domain audio separation network (TasNet) with dual-path RNN (DPRNN) <cit.> for all experiments. We use the same hyperparameter settings as in <cit.> for the 2 ms window configuration, with the only difference that we use 3 instead of 6 DPRNN blocks. The total number of parameters is thus 1.3M. The baseline model uses the standard SI-SDR as the training objective, and all other models use the proposed A2PIT together with α-SI-SDR proposed in Section <ref>. All models are trained for a maximum of 100 epochs with the Adam optimizer <cit.>. The initial learning rate is 1e-3 and is decayed by a factor of 0.98 for every two epochs. No other regularizers or training tricks are applied.

We train the DPRNN-TasNet for each of the speaker count configurations as the baseline models. These results represents how well the models can achieve when the number of speakers is known and a specific model is trained on such mixtures. For separating varying numbers of sources, we train DPRNN-TasNet models on three configurations:

    
  * 2+3 speakers: the 2 and 3 speaker mixtures are used for both training and evaluation, and the number of outputs N is set to 3. This is to mimic the behavior under certain cases when the maximum number of active sources is bounded by 3 (e.g. meeting scenarios). We denote it as the 2+3 model.
    
  * 2+3+4 speakers: the 2, 3 and 4 speaker mixtures are used for both training and evaluation, and the number of outputs N is set to 4. This is to increase the difficulty of both the separation and speaker count. We denote it as the 2+3+4 model.
    
  * 1+2+3+4 speakers: all training and evaluation datasets are used. We denote it as the 1+2+3+4 model.


Each configuration contains both the clean and noisy scenarios, which results in a total of 6 different configurations.



 §.§ Evaluation metrics


We evaluate the separation performance by the SI-SDR improvement (SI-SDRi) with respect to the unprocessed mixture. To evaluate the accuracy for speaker number detection, we report the confusion matrix of the predicted and oracle numbers of speakers in the test set.



 §.§ Results and discussions




  §.§.§ Determine the similarity threshold through the training set


The similarity threshold described in Section <ref> needs to be determined through the training set. Figure <ref> shows the autoencoding SI-SDR, i.e. the SI-SDR between the outputs and the input mixture, for different configurations. We can see that in the clean separation task, most of the auxiliary outputs have a significantly higher autoencoding SI-SDR than the valid outputs. The high SI-SDR utterances in the 1+2+3+4 model are mainly due to the single-speaker samples. This allows us to draw a clear boundary to distinguish them. We empirically set the threshold for all models for clean separation tasks to be 20, i.e. outputs with autoencoding SI-SDR higher than 20 dB will be treated as invalid outputs, and set the thresholds for the 2+3 model, 2+3+4 model, 1+2+3+4 model for the noisy separation tasks to be 12, 12, and 8, respectively.



  §.§.§ Accuracy of speaker counting






Table <ref> and <ref> show the confusion matrices for all 6 configurations. Note that each of the speaker number has a test set of 1800 utterances. We first notice that for the 2+3 model, the prediction of speaker count can be done with a very high accuracy in both clean and noisy separation tasks. For the 2+3+4 model, the detection of 3 speaker mixtures is worse than that of both 2 and 4 speaker mixtures, and the error mostly comes from the misclassification into 4 speaker mixtures. For the 1+2+3+4 model, we find that the detection of the 1 speaker mixtures almost always fail (detects no speakers in the mixture). With the autoencoding threshold, the model predicts no valid outputs for most of the times. This is somehow expected as in the clean separation task, the mixture itself is equivalent to the separated output, and in the noisy separation task, the separated output may still have very high similarity score with respect to the mixture because of our high SNR configuration. For tasks such as automatic speech recognition, this will not be an issue as the acoustic models are typically noise robust, while for tasks that require perceptual quality, the outputs need to be further evaluated. Beyond the 1 speaker mixtures, the accuracy for speaker counting for other cases remains high.

Another interesting observation is that the models occasionally predict zero speakers (e.g. 2-speaker utterances in all models for noisy separation). This can only happen when the autoencoding SI-SDR of all outputs are larger than the pre-defined threshold. It indicates that in certain utterances the separation may completely fail and the model converges to always perform autoencoding. A better solution to this issue is left for future works.



  §.§.§ Performance of speech separation






Table <ref> and <ref> provide the separation performance on the clean and noisy separation tasks, respectively. For the one speaker utterances in the clean separation task, SI-SDR instead of SI-SDRi is reported as the input is already the clean target itself. We observe that A2PIT can almost always improve the separation performance on all configurations with both clean and noisy data, and the gains for 3 and 4 speaker cases are significant. We can conclude from the results that A2PIT is able to achieve on par or better overall separation performance on both clean and noisy separation tasks. Even with predicted output selection, the fault tolerance ability introduced by A2PIT allows the model to control the performance degradation. These results confirms the effectiveness of A2PIT.




§ CONCLUSION

In this paper, we proposed a simple method for separating varying numbers of speakers in a mixture with “fault tolerance” ability, which we referred to as the auxiliary autoencoding permutation invariant training (A2PIT). A2PIT assumed a fixed number of outputs N and appended mixture signals to the training targets of the utterances whose number of valid outputs M was smaller than N. Fault tolerance was achieved by treating the auxiliary outputs as the outputs of a “null” separation which directly passed the input to the output. We call this the “do nothing is better than do wrong things” principle. During inference time, a similarity threshold between the mixture and the outputs was used to determine valid outputs in a fully unsupervised way. Experiment results showed that A2PIT was able to effectively perform speaker count in various scenarios, and maintained on par or better separation performance than baseline systems trained for specific datasets with both oracle and predicted speaker count.




§ ACKNOWLEDGMENTS

This work was funded by a grant from the National Institute of Health, NIDCD, DC014279; a National Science Foundation CAREER Award; and the Pew Charitable Trusts.


IEEEtran


