Student-Teacher Learning for BLSTM Mask-based Speech Enhancement
Aswin Shanmugam Subramanian Szu-Jui Chen Shinji Watanabe


Cross-validation in high-dimensional spaces: a lifeline for least-squares models and multi-class LDA
    Matthias S. Treder
    December 30, 2023
====================================================================================================


Spectral mask estimation using bidirectional long short-term memory (BLSTM) neural networks has been widely used in various speech enhancement applications, and it has achieved great success when it is applied to multichannel enhancement techniques with a mask-based beamformer.
However, when these masks are used for single channel speech enhancement they severely distort the speech signal and make them unsuitable for speech recognition. 
This paper proposes a student-teacher learning paradigm for single channel speech enhancement. 
The beamformed signal from multichannel enhancement is given as input to the teacher network to obtain soft masks. 
An additional cross-entropy loss term with the soft mask target is combined with the original loss, so that the student network with single-channel input is trained to mimic the soft mask obtained with multichannel input through beamforming. 
Experiments with the CHiME-4 challenge single channel track data shows improvement in ASR performance.


Index Terms: Speech enhancement, speech recognition, mask estimation, BLSTM, student-teacher learning



§ INTRODUCTION


The presence of background noise and reverberation degrades the performance of an automatic speech recognition (ASR) system. 
The performance of noise robust ASR was greatly improved by using multiple microphones instead of just using a single microphone <cit.>.
Especially, mask-based beamforming techniques have shown outstanding results in this scenario <cit.>, and many of the top systems in the CHiME-4 challenge <cit.> use these techniques for speech enhancement. 
For example, <cit.> use a bidirectional long short-term memory (BLSTM) neural network to accurately predict speech (and noise) masks, given the noisy spectrogram.
In <cit.>, both speech and noise masks are in turn used to calculate cross power spectral density (PSD) matrices of the noise and the target speech, which are used by the generalized eigenvalue (GEV) beamformer to get the beamforming filter <cit.>.

However, compared with the success of multichannel speech enhancement, single-channel speech enhancement is not well established, especially when the method is combined with other speech processing applications including ASR and speaker recognition <cit.>.



Single-channel speech enhancement, especially based on deep neural networks (DNNs), has been studied by many research groups <cit.>
with promising performance improvement in terms of the speech enhancement metrics or hearing purposes.
However, we often observe a degradation in performance when we use single-channel enhancement as a preprocessing step for ASR due to the spectral distortions induced by the enhancement.
This paper focuses on the single-channel speech enhancement to overcome this issue.

Our idea is to fill out the gap between single-channel and multichannel speech enhancement by using a well known DNN technique called student-teacher learning.
We propose to train a teacher network, which takes the beamformed signal obtained from multichannel speech enhancement as input and predicts high-quality speech masks. 
Then, a student network, which takes the noisy signal as input, is trained to mimic the mask predicted by the teacher network. 
Conventionally student-teacher learning <cit.> is used to reduce the complexity of the network. 
For example, a less complex student model with fewer parameters tries to mimic the soft targets of a more complex teacher model. 
In <cit.>, student-teacher training is used for self-supervised learning of an ASR acoustic model, where the student model tries to mimic the effects of multichannel speech enhancement by using only single channel inputs.

This was achieved by training the student network with noisy training data as input to mimic the soft posteriors of the teacher network trained on the enhanced speech. 
In addition, the benefits of using soft targets compared to hard targets for DNN acoustic models are also shown in <cit.>, which is advocated by <cit.> as knowledge distillation.

This paper follows BLSTM-mask based beamforming proposed in <cit.>, which has two cross-entropy loss terms - one for speech mask target and the other for noise mask target.
The speech mask predicted by the model can also be used for single-channel enhancement, which this paper uses as the baseline.
We have an additional loss term based on the cross entropy with the soft mask from the teacher network as a target.
The effectiveness of the proposed method is investigated by using the single channel track of the CHiME-4 dataset <cit.>. 
The CHiME-4 dataset consists of both real and simulated data and it is common to use only the simulation data to prepare the clean and noise targets.
As the proposed method uses the soft mask from the teacher network as the target, which can be obtained even for the real data, it can use both real and simulation data in the training stage. This is also an unique aspect of the proposed method.
Additionally, four different speech enhancement metrics - perceptual evaluation of speech quality (PESQ) <cit.>,  short-time objective intelligibility measure (STOI) <cit.>, extended STOI (eSTOI) <cit.> and speech distortion ratio (SDR) <cit.> are used as part of the experiments to discuss the performance of speech enhancement in addition to the ASR performance.




§ BLSTM MASK-BASED SPEECH ENHANCEMENT

This section first describes a mask prediction method by using a binary cross entropy loss, which is a basic component of this paper.
It also describes a mask-based beamformer to estimate beamforming filters based on the estimated masks.



 §.§ Binary cross entropy loss


The noise-aware training of BLSTM mask proposed in <cit.> is explained in this section. 
The network estimates two masks: the first is the speech mask which denotes the degree of how dominant the speech component is in each time-frequency bin, while the second is the noise mask which denotes that of the noise component <cit.>. 
The ideal binary speech mask target IBM_X(t,b) ∈{0, 1} at frame t in frequency bin b is defined based on the signal-to-noise (SNR) ratio with thresholding as:

    IBM_X(t,b) =
    
    1,    if ‖ x (t, b) ‖/‖ n (t, b) ‖ > threshold 
    
    0,    otherwise 
    ,

where, ‖ x (t, b) ‖∈ℝ _≥ 0 is the power spectrum of the clean speech signal and ‖ n (t, b) ‖∈ℝ _≥ 0 is the power spectrum of the noise signal at each time-frequency bin (t, b).
The ideal binary noise mask IBM_N (t,b) ∈{0, 1} is also calculated in a similar way.

Given a sequence of T-length noisy speech magnitude spectra Y = ({‖ y (t, b) ‖} _b=1 ^B | t =1, ⋯, T), the BLSTM network predicts the speech mask w _X (t, b) ∈ [0, 1] and noise mask w _N (t, b) ∈ [0, 1] at each time-frequency bin (t, b), as follows:

    w _v (t, b) = σ(Lin _v (BLSTM (Y))),  where  v ∈{X, N},

where σ(·), Lin _v(·), and BLSTM(·) are sigmoid activation, linear, and output of the masking network given in Table <ref>, respectively.


The binary cross entropy is used as the loss function to train the network. 
The overall loss function is defined by combining the speech and noise binary loss functions (loss_X and loss_N) as follows:

    loss    =  loss_X + loss_N,
       ≜1/T*B∑ _t, b∑ _v ∈{X, N}CE (IBM_v (t, b), w _v (t, b)),

where CE(a, a') is the binary cross entropy defined as follows:

    CE(a, a')    ≜ a log a' + (1 - a) log (1 - a') 
        = 
    
     a log a'       a = 0 
    
     (1 - a) log (1 - a')        a = 1 
    .

Note that since a ∈{0, 1}, either first or second term in the right hand side of Eq. (<ref>) becomes zero.
This type of target is particular called as a hard label in the student-teacher learning <cit.> or knowledge distillation <cit.> context.

When we apply the estimated speech mask w _X (t, b) to the original signal y(t, b), we can perform single-channel speech enhancement as follows:

    x ^(se) (t, b) = w _X (t, b) y(t, b),

where x ^(se) (t, b) is an enhanced signal (the superscript (se) denotes single-channel enhancement).
Although the single channel enhancement only requires the prediction of the speech mask, the noise mask is used to obtain the noise PSD matrix in a multichannel scenario, which will be discussed in the next section.







 §.§ Mask-based beamformer

This section extends to deal with a multichannel signal with M as a number of channels. 
When the single channel mask estimation in the previous section is applied to the multichannel signal, the corresponding speech mask w _m, X (t, b) and noise mask w _m, N (t, b) for each channel m can be obtained.
With these multichannel masks, the following time-frequency mask is obtained from a median operation: 

    w̅ _v (t, b) = Median ({w _m, v (t, b)} _m=1 ^M)  where  v ∈{X, N}.

With the speech and noise masks w̅ _X (t, b) and w̅ _N (t, b), the PSD matrices, which are 
Φ_X(b) ∈ℂ ^M× M of speech at frequency bin b, and Φ_N (b) ∈ℂ ^M × M of noise at frequency bin b, can be estimated as follows:

    Φ_v (b) = ∑_t=1^T w̅_v(t,b)𝐲(t, b)𝐲(t, b)^H  where   v ∈{X, N}


where 𝐲(t, b) ∈ℂ ^M is the M-dimensional complex spectral value at time (frame) t and frequency bin b.

From these PSD matrices, the M-dimensional complex beamforming filter 𝐟 (b) ∈ℂ ^M can be estimated.
This paper adopts the GEV beamformer, which is obtained by maximizing the expected SNR with respect to 𝐟 (b) for each frequency bin as given by the equation below:

    𝐟_GEV (b) = _𝐟  (b)𝐟^H (b)Φ_X (b) 𝐟 (b)/𝐟^H  (b) Φ_N (b) 𝐟 (b).

This optimization is equivalent to solving the following eigenvalue problem:

    (Φ_N (b) ) ^-1Φ_X (b) 𝐟  (b) = λ𝐟  (b),

where 𝐟  (b) is the eigenvector and λ is the corresponding eigenvalue. 

Once we obtain the beamforming filter 𝐟_GEV (b), we can perform multichannel speech enhancement as follows:

    x ^(me) (t, b) = 𝐟_GEV^H (b)  𝐲(t, b),

where x ^(me) (t, b) is an enhanced signal  (the superscript (me) denotes multichannel enhancement).

In general, x ^(me) (t, b) can be well-denoised by making use of spacial information and beamforming operation, compared to the single-channel enhanced signal x ^(se) (t, b) introduced in Section <ref>.
This paper proposes to use these single- and multi-channel speech enhancement properties, and designs a new objective function for single-channel enhancement, by using the mask obtained by multichannel enhancement as a better soft label.








§ STUDENT-TEACHER MODEL

This section explains in detail the proposed single-channel enhancement technique by using a student-teacher model.



 §.§ Teacher model

Firstly, the beamformed signal x ^(me) (t, b) is given as input to the teacher model. 
The architecture of the network is the same as the one explained in Section <ref> except that the output is only the speech mask w _X ^(me) (t, b) and there is no noise mask. 
Hence the binary loss function for a teacher network is 

    loss = 1/T*B∑ _t, bCE (IBM_X (t, b), w _X ^(me) (t, b))).


Since the beamformed signal is already well-denoised, the above teacher network provides a high-quality speech mask.
Alternatively, the original clean signal x (t, b) can be used as the input instead of the beamformed signal to train the teacher model.
However, the problem of estimating a speech mask given a clean signal is too trivial, and the network is not well-trained in such a trivial condition.









 §.§ Student model


With the speech mask w _X ^(me) (t, b) obtained by the above teacher network, we can additionally consider the following student-teacher loss function:

    loss _st =  1/T*B∑ _t, bCE (w _X ^(me) (t, b)), w _X ^(se) (t, b))).

Compared with the binary cross entropy case in Eq. (<ref>), the target w _X ^(me) (t, b)) ∈ [0, 1] is a bounded continuous value, and provides richer supervisions to the student network.

The final loss function is expressed with the student-teacher loss and the original loss (Eq. (<ref>)) as:

    loss = λ_1 loss _st + 
    λ_2 loss _X + λ_3 loss _N

where λ_1, λ_2, and λ_3 are linear interpolation weights. 
The student model is trained with this loss function, which expects to learn the multichannel enhancement ability within a single-channel enhancement framework. Figure <ref> illustrates the proposed student-teacher training paradigm. 




 §.§ Student model with real data


All the above formulations are discussed given that we have parallel clean speech (x) and noise (n), which are not usually obtained in the real recording.
However, the proposed network can obtain the soft target of the real data by simply applying multichannel speech enhancement for the real data.
In this scenario, it has only one student-teacher loss term loss _st (Eq. (<ref>)), which exactly mimics the teacher network.
With this setup, the loss function can be obtained by switching the student-teacher loss (Eq. (<ref>)) and the combined loss (Eq. (<ref>)) for real and simulation data as follows:

    loss = 
    loss _st    for real
    λ_1 loss _st + 
    λ_2 loss _X + λ_3 loss _N    for simulation

This is our final loss function, which fully makes use of the benefit of the student-teacher learning paradigm in speech enhancement.



§ EXPERIMENTS

To verify the effectiveness of the proposed single-channel experiments, the 1 channel track in the CHiME-4 challenge <cit.> was used.











 §.§ Speech recognition

The Kaldi speech recognition toolkit <cit.> was used for ASR experiments. The GMM baseline provided by the CHiME-4 challenge <cit.> was used for comparing different mask models.
The mask network was implemented using the Chainer neural network toolkit <cit.>. The cross-validation loss was calculated using the development data after every epoch during the training of the teacher network. 
The training was stopped when the loss did not decrease anymore after 5 epochs of patience and the model with the least cross-validation loss was chosen. 
The 5th channel data was used to estimate the target masks for the beamformed data. 
The student network was trained in a batch mode where a minibatch comprised the frames of all 6 channels of one utterance.

To compare the performance of different masking models, we first prepared an ASR baseline that would be neutral to the enhancement method by using only the noisy data of all 6 channels as the training data. 
We performed enhancement on the development and evaluation sets of the official 1-channel track CHiME-4 dataset, with different models and decoded their enhanced speech with the above ASR systems.
The results of these ASR experiments are shown in Table <ref> (rows 1–3).
The performance of the teacher model was better than that of the original BLSTM mask as expected, although both degraded the performance from the non-enhanced noisy speech.

The second experiments focused on our proposed student models (rows 4–11 in Table <ref>), and we found that most of the different configurations of student models performed better than that of the teacher model.
Note that in this experiment, we fixed the number of epochs for several models as 3.
This is because we additionally investigated the dependency between the validation loss and WER, as shown in Figure <ref>, and we found that it did not necessarily give us the best results for the best validation score with different hyper-parameter λ, as introduced in Eq. (<ref>).
We empirically found that 3 epochs in most of the cases seemed to give good performance. 
This result indicates that choosing the epoch based on the WER of the development data seems to be a better criterion than the enhancement-driven cross-validation loss (related discussions will be shown in the next section). 
With this setup (3 epochs), we found that the parameter choice of λ_1 = 0.35, λ_2 = 0.15 and λ_3 = 0.5 gave the best performance amongst the different values we tried. 
This is an interesting finding, since this result indicates that soft targets obtained by the teacher model would effect the performance more than that of supervised targets.
We also trained the student model with additional real data, as discussed in Section <ref> (row 11 in Table <ref>), and this shows the improvement for the real test set owing to the inclusion of the real training data, which is a desired property in real environments.

When the 5th channel of the training data was also enhanced in the same way as the development and evaluation data using the baseline BLSTM mask and included as part of the ASR training (row 12), the performance was slightly better than using the original noisy data for the evaluation data but it was slightly worse for the development data. When the development and evaluation data was enhanced using our best student models (rows 13 and 14), the performance improved significantly compared to using the original noisy data in the all conditions. Also, we observed the similar improvement for the real test set again when we trained the student model with real data.



 §.§ Speech enhancement scores

The four different scores described in Section <ref> - PESQ, STOI, eSTOI and SDR were computed for several enhancement methods, as shown in Table <ref>. 
The 5th channel clean signal from the 6ch track convolved with room impulse response was used as the reference signal. 
All the different  masking models gave significantly better scores in all four metrics compared to using the noisy data without any enhancement although we did not observe any considerable difference in the scores  amongst the models.
In addition, by comparing Tables <ref> and <ref>, we did not observe clear correlations between the speech enhancement scores and WERs, which also suggests that we need some careful investigation on the objective function of BLSTM-based enhancement for the ASR purpose.



§ CONCLUSION

We proposed a new training paradigm for mask estimation using BLSTMs based on student-teacher training for single channel speech enhancement. 
We showed that the proposed student-teacher technique improved the ASR performance from the original noisy speech and the enhanced speech obtained by conventional BLSTM masking. 
Our future work is to evaluate our speech enhancement techniques with a strong ASR backend including a time delay neural network (TDNN) <cit.> with the lattice-free version of the maximum mutual information (LF-MMI) <cit.>.



IEEEtran





















