title: Length- and Noise-aware Training Techniques for Short-utterance Speaker Recognition
authors: Wenda Chen Jonathan Huang Tobias Bocklet
text: 
  
  The Complexity of L(p,q)-Edge-Labelling
      [
      December 30, 2023
  =======================================
  
  
  
  
  Speaker recognition performance has been greatly improved with the emergence of deep learning. Deep neural networks show the capacity to effectively deal with impacts of noise and reverberation, making them attractive to far-field speaker recognition systems. The x-vector framework is a popular choice for generating speaker embeddings in recent literature due to its robust training mechanism and excellent performance in various test sets. In this paper, we start with early work on including invariant representation learning (IRL) to the loss function and modify the approach with centroid alignment (CA) and length variability cost (LVC) techniques to further improve robustness in noisy, far-field applications. This work mainly focuses on improvements for short-duration test utterances (1-8s). We also present improved results on long-duration tasks. In addition, this work discusses a novel self-attention mechanism. On the VOiCES far-field corpus, the combination of the proposed techniques achieves relative improvements of 7.0% for extremely short and 8.2% for full-duration test utterances on equal error rate (EER) over our baseline system.
  
  
  Index Terms: speaker recognition, invariant representation learning, centroid alignment, x-vector, far-field
  
  
  
  § INTRODUCTION
  
  
  Speaker recognition system have been popularized in consumer devices such as smart speakers and smartphones.  In these use cases, the speech utterance of the user during an interaction with the device can be used to perform voice matching.  These use cases are particularly challenging with respect to channel degradation. Furthermore, the interactions tend to be short, making recognition even harder. In this paper, we propose techniques to address these issues. 
  Recently, a successful application of deep neural networks to the domain of speaker recognition helped to improve the accuracy significantly <cit.>. Here, the network is trained end-to-end via triplet loss.
  Another state of the art system is x-vector extraction <cit.>. A frame-wise neural network is used as feature generator followed by a segment-based network using pooled frame-wise data.
  Training is performed via cross-entropy loss on multiple thousand speakers. For recognition, speaker embeddings are extracted and compared with a similarity measure, e.g., cosine distance. 
  Noise robustness is achieved by PLDA <cit.> and simulation of noisy training data <cit.>. 
  
  Recent research evaluated various extensions of the x-vector approach focusing also on higher noise robustness. The statistical pooling was substituted by attention mechanism <cit.> in order to learn the weighted pooling in a data-driven manner. 
  Angular softmax <cit.> and large margin losses <cit.> showed in combination with noise-augmented training data very good results. 
  <cit.> presented a scoring approach mimicing the PLDA-scoring by discriminative training. 
  <cit.> added residual blocks in order to allow deeper networks. 
  <cit.> motivated loss functions in order to teach the network to learn an implicit cleaning of encodings.
  
  Besides noise, the duration of utterances is a factor of high importance for the accuracy. 
  <cit.> described the use of gammatone features in combination with i-vector on short utterances, <cit.> trained deep convolutional networks specifically on short utterances.
  <cit.> described the use of inception networks by transforming utterances to fixed length spectrograms and training via triplet loss.
  <cit.> used stacked gated recurrent units (GRU) to extract utterance-level features followed by residual convolution neural networks (ResCNNs) trained with speaker identity subspace loss that focuses on transforming utterances from the same speaker to the same subspace.
  <cit.> used adversarial learning techniques to learn enhanced embeddings from short and long embedding pairs from the same speaker.
  <cit.> argued that pooling and phoneme-aware training is harmful, especially for short-utterance SID and used adversarial training to remove phoneme-variability from the speaker embeddings.
  
  This work focuses on enhanced training techniques for x-vector embeddings with a strong focus on short-duration speech segments in heavy far-field and noisy conditions. 
  We start with our previous work <cit.> on adding additive margin softmax (AM-softmax) and Invariant Representation Learning (IRL) to an x-vector-based SID system. Due to recent improvements, we substitute the statistical pooling by an attention mechanism. We then modify the idea of IRL and introduce improved training techniques: Centroid Alignment (CA) and Length Variability Cost (LVC). These techniques address the variability in utterances due to channel noises and utterance duration, by making the embedding space less sensitive to such perturbations. CA modifies IRL by enforcing small distances of the training utterance to the average embedding (centroid). LVC tries to keep the distance between a speaker's two randomly-selected utterances with different lengths small. The proposed techniques show improvements on both short and full-length VOiCES test utterances which were collected in noisy and reverberant environments <cit.>. 
  The rest of the paper is organized as follows:  Section <ref> provides the intuition behind our training techniques.  Section <ref> details the modifications we made to the x-vector architecture as well as our attention mechanism.  Section <ref> describes the model training techniques.  Section <ref> shows the results and discussions of our experiments. We finish the paper with a short conclusion in Section <ref>.
  
  
  
  
  § MOTIVATION
      
  
  The intuition behind our model training techniques can be best illustrated by visualizing the 2-D t-SNE plots of a well-trained speaker embedding (Fig. <ref>). The embedding was been generated from the baseline system described in Section <ref>. The plots show embeddings extracted from random utterances of 10 speakers in LibriSpeech <cit.>. The full-length clean speech utterances shows distinct, well-separated clusters; no confusion with other speakers is visible. When noise is added to the full-length utterances [b], the speaker clusters are more blurred. Adding clean speech segments truncated to 2s [c] shows higher confusion than [b]. Combining noise and short duration amplifies the effect [d].  To also quantify the growing deviation, we calculated the average standard deviation of the x-vector embeddings (see line baseline in  <ref>): noisy embeddings show a higher deviation compared to clean, embeddings generated on short utterances have a higher deviation than noisy ones and noisy short further increases the standard deviation. Based on the simple observation that clean speech is `better' than noisy, and full-length is `better' than truncated, we hypothesize that we can use the `better' samples as an additional training target. This idea is the basis for our auxiliary training objectives:  for IRL, we align the noisy speech with its corresponding clean version; for LVC, we align a short duration speech with its full-length version; and for CA, we align the noisy or short-duration speech with the cluster centroid of clean full-length speech for each speaker.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  § SYSTEM ARCHITECTURES
  
  
  
  
   §.§ Baseline X-vector Architecture
  
  
  Although our contributions in attention and model training work for other embedding architectures, we show our findings using the popular x-vector architecture <cit.>. 
  The topology we used is a slight modification of the original x-vector description.  We chose 40-dimension log mel-filterbank as features. The features are mean-normalized on a 3-second sliding window. In  <ref>, layers 1-6 are unmodified from the original paper.  We removed one of the dense layers before the output, and extract the embedding at layer 7 with 256 dimensions. Our experiments have shown consistently better results with these modifications.
  
  
  
  
  
  
   §.§ Multi-headed Self-attention for X-vector
  
  
  Attention mechanisms have been proposed in several previous studies <cit.> to give different weightings to the frames within an speech utterance.  In the context of the x-vector embedding, the stats pooling layer is replaced with an attentive pooling layer.  The input to the attentive pooling is the hidden representation of layer 5, which we define here as 𝐡_t at time step t.  In a departure from previous literature, we compute different attention heads for parts of 𝐡_t.  Our hypothesis is that the dimensions in this hidden representation correspond to different aspects of the signal which should not be treated with the same weighting.  Concretely, we break the vector 𝐡_t into K contiguous, equal-length, smaller sub-vectors 𝐡_t,k, where k=1,...,K is the sub-vector index.  The attention weight corresponding to these sub-vectors can be computed by
  
      e_t,k = f(𝐰^T_k𝐡_t+b_k),
  
  where 𝐖_k ∈ℝ^1500 × 1500/K and  b_k ∈ℝ^1500/K × 1 are the trainable weight matrix and bias, respectively, and f is the non-linear activation.  We found that the Sigmoid activation here gives the best performance.  The frames of the attention weights are normalized by a Softmax 
  
      α_t,k = exp(e_t,k)/∑_t=1^Texp(e_t,k)
  
  The attentive mean and standard deviation pooling for each sub-vector across the entire utterances for t=1,...,T time steps is
  
      μ_k = ∑_t=1^Kα_t,k𝐡_t,k,
  
  and 
  
      σ_k = √(∑_t=1^Kα_t,k𝐡_t,k⊗𝐡_t,k
      	- μ_k ⊗μ_k
      	),
  
  respectively. ⊗ denotes the element-wise multiply operation.  Finally, the output of the attentive pooling is formed by a concatenation of the results from <ref> and <ref>, for all k,
  
      𝐳 = [μ_1^T,...,μ_K^T,
      	σ_1^T,...,σ_K^T]^T
  
  as a 3000-dimension vector for the utterance.
  
  
  
  
  § TRAINING TECHNIQUES
  
  
  
  
  
   §.§ Baseline Training
  
  
  In the original x-vector paper, the output of the last layer with N speaker labels is fed into a Softmax loss.  Because Softmax is specifically designed for classification, others focused on improving the speaker recognition objective with other loss functions <cit.> or even completely different training infrastructure <cit.>. In our experiments, the use of Additive Margin Softmax (AM-softmax) loss <cit.> was consistently superior to basic Softmax training in the far-field test set. The systems trained with AM-softmax did not benefit from having a PLDA backend <cit.>. We use cosine similarity as the scoring mechanism. This then reflects our baseline system.
  
  For the proposed learning techniques which are describing next, the weights are initialized with the weights of a baseline systems. We used two different methodologies to train our baseline systems: 1. with fixed length 8s long utterances, 2. with utterances of variable length between 0.5 and 8.5s. In the result section, these variants are differentiated by the suffix -long for the former and -varied for the later variant.
  
  
  
   §.§ Invariant Representation Learning
  
  
  Fig. <ref> shows the IRL training procedure.  At each training iteration, clean (x) and noisy (x') features of the same utterance are fed into the network one after another, resulting the computation of two AM-softmax loss values.  At the layer where the speaker embedding is extracted, we align the two sides by imposing cosine similarity and MSE loss.  Mathematically, the combined loss is sumarized as 
  
      L_IRL(x,x')=L_AM(x)+α L_AM(x')-γ L_cos(x,x')+λ L_2(x,x')
  
  where L_AM is the categorical loss, L_cos is the cosine similarity (<ref>), and L_2 is the means square loss (<ref>)
  
      L_cos(x, x') =   
      	ϕ(x) ·ϕ(x')/ϕ(x)ϕ(x')
  
  
      L_2(x, x') =  (ϕ(x)-ϕ(x'))^2
  
  
  
  
  
  The embedding layer representation is denoted by ϕ_l.  The parameters α, γ, and λ control the contributions from the losses.  This training tries to learn a representation invariant to noise. For IRL we achieved best results when initializing with a baseline trained on fixed 8s utterances.
  
  
  
   §.§ Length Variability Cost
  
  LVC is in fact a special case of IRL.  Here, x is a long-duration utterance (i.e. 8s), and x' is a truncated version of the same utterance (i.e. randomly chosen length between 0.5-8.5s). This training tries to learn a representation invariant to utterance duration. α =1 for LVC and IRL in Eq. <ref>. For LVC we achieved best results when initializing with a baseline trained by the fixed length methodology.
  
  
  
   §.§ Centroid Alignment
  
  The intuition behind CA is that the clean full-length utterances, and especially their centroid of each speaker, is a desirable training target in the embedding space. After each epoch of training, we compute the embeddings for all clean full-length segments, and average the length-normalized embeddings for each speaker label to form the centroids.  For the subsequent epoch, the speaker centroid is the target to do alignment against each training sample.  Concretely, for Eq. <ref>, we compute the AM-softmax for x only, and set α to zero because we do not use training pairs.  Instead of doing alignment with another training sample, we use the centroid of speaker for x as the alignment target x'. 
  
  
  
  § EXPERIMENTAL RESULTS
  
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
  
  
   
  
  
  
      
  
  
  
  
  
  
  
  
   §.§ Experimental Settings
  
  Our systems are trained on the Voxceleb 1 and 2 corpora <cit.>. The training material is prepared by applying 10x data augmentation. For each augmented speech file we convolve a randomly chosen room impulse response (RIR) from 100 artificially generated RIRs by Pyroomacoustics <cit.> and 100 randomly selected real RIRs from the Aachen Impulse Response Database <cit.>. Afterwards, the data is mixed in with randomly chosen clips from Google's Audioset under Creative Commons <cit.>. The SNR for mixing was uniformly distributed between 0 and 18 dB.  
  We evaluate the proposed loss functions on the VOiCES Dev and Eval data <cit.> and compare against a baseline system trained with AM-softmax loss. 
  We found optimal weighting factors γ and λ for CA  with γ=0.5 and λ=0.01, meaning the cosine loss is weighted significantly higher. For both LVC and IRL ideal values have been found with γ=λ=0.5. We see very consistent results between Dev and Eval. For the sake of readability we present numbers on VOiCES dev EER and  minDCF (P_tar=0.01) metrics.
  
  
  
   
  
       
  
  
  
  
  
   §.§ Self-attention Model
  
  Fig. <ref> visualizes the behaviour of the attention mechanism on an audio segment containing speech, music, and silence.  If we look at 5 randomly chosen attention heads, their weights are different in the speech segments, indicating they are learning different aspects of the signal.  The average of the attention weights across all heads shows that the silence and noisy segments of the audio are suppressed compared to the speech.  Empirically we saw improvements in increasing the number of attention heads to 100, but not beyond.  We use 100 heads for the results here. In  <ref> the positive influence of attention is visible in the recognition rates. x-vector denotes our baseline system with the topology shown in  <ref>. For the sake of completeness, we also show results of ext-x-vector with and without attention. ext-x-vector is an extended x-vector system as defined in <cit.>. The temporal context is slightly
  wider for the frame-level layers and dense layers are added in between the T-DNN layers. The x-vector variants with attention are consistently better than the non-attention variants. The extended variant  significantly outperforms the baseline. For the sake of less training time and limited model size and power, we decided to use the smaller topology x-vector-att for the evaluation of the proposed loss functions.
  
  
  
   §.§ LVC and CA loss
  
  The models are evaluated on various speech durations: 1s, 2s, 4s, 8s, and full (as in VOiCES). Results are summarized in  <ref>. For AM-softmax we report results on the training methodologies -long (8s training samples) and -varied (0.5-8.5s training samples). For AM-softmax, the results on short utterances (1-4s) are superior when trained with variable length methodology. On 8s and full duration, training via long methodology achieved the better results. This confirms the intuition: training a system on long utterances gives better results on longer test utterances. 
  CA shows an advantage in extremely short test utterances (1s and 2s). CA tries to bring short utterances closer to the speaker centroid which is estimated on 8s long utterances and thus helps to stabilize results on short utterances. On 1s test utterances the EER is reduced by 7.0 % compared to AMSM-varied.  
  LVC seems especially helpful for 4s and 8s durations.  LVC tries to diminish the distance between long, i.e., 8s and randomly a shorter version (randomly chosen 0.5-8.5s) of the same utterances. 
  Thus, LVC helps to stabilize the training process by removing general speaker variability, which is also visible in  <ref>. On 8s segments, LVC improves the EER result by 12.2 % compared to AMSM-long. IRL works best in the full length condition (8s pairs of utterances from the same speaker are aligned) and is close to LVC in the 8s test condition. Trying to tie a clean and noisy utterances of the same size close together, is especially helpful for longer duration utterances: IRL achieves an EER reduction of 8.2 % on the full duration utterances compared to AMSM-long.
  
  
  
   <ref> shows the average standard deviations of embedding for the 10 speakers  plotted in  <ref> in clean, noisy and short scenarios. Compared to the AM-softmax baseline, CA and LVC help to reduce the variability especially for short and combined (noisy and short) scenarios. For the baseline, the average standard deviations increases for noisy, short and the combination. Applying CA and LVC, brings noisy and short utterances closer to the centroids and removes variation within utterances of one and the same speaker. Hence, making the speaker separation more accurate. While the clean cases also get closer to their cluster centroids, the average relative improvement rate is especially high for the short and noisy cases at 4.6%. 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  § CONCLUSION
   
  In this paper, we proposed to extend the x-vector based SID with self-attention pooling and apply new loss functions based on IRL, CA and LVC. These techniques are targeted for improving speaker recognition on short utterances in reverberant and noisy conditions. Both the proposed attention mechanism and the improved loss functions show a reduction in error rate. Our experiments and analysis proved that the centroids and embedding length variations are good regularization references for the AM-softmax losses. Combining the approaches leads to an EER reduction of 7.0 % on 1s and 8.2 % on full duration utterances.
  IEEEtran
  
  
  
  
  
  
  