Empirical Evaluation of Speaker Adaptation on DNN based Acoustic Model
Ke Wang Junbo Zhang Yujun Wang Lei Xie

0.90

Learning to Branch[Authors' addresses: ]
    Maria-Florina Balcan Travis Dick Tuomas Sandholm Ellen Vitercik
    December 24, 2016
===================================================================



Speaker adaptation aims to estimate a speaker specific acoustic model from a speaker independent one to minimize the mismatch between the training and testing conditions arisen from speaker variabilities.  A variety of neural network adaptation methods have been proposed since deep learning models have become the main stream.
But there still lacks an experimental comparison between different methods, especially when DNN-based acoustic models have been advanced greatly. In this paper, we aim to close this gap by providing an empirical evaluation of three typical speaker adaptation methods: LIN, LHUC and KLD. Adaptation experiments, with different size of adaptation data, are conducted on a strong TDNN-LSTM acoustic model. More challengingly, here, the source and target we are concerned with are standard Mandarin speaker model and accented Mandarin speaker model. We compare the performances of different methods and their combinations. Speaker adaptation performance is also examined by speaker's accent degree. 



Index Terms: Speaker adaptation, deep neural networks, LIN, KLD, LHUC



§ INTRODUCTION


Speech recognition accuracy has been significantly improved since the use of deep learning models (DLMs), or more specifically, deep neural networks (DNNs) <cit.>. Various models, such as convolutional neural networks (CNNs) <cit.>, time-delay neural networks (TDNNs) <cit.>, long short-term memory (LSTM) recurrent neural networks (RNNs) <cit.> and their variants <cit.> and combinations <cit.>, have been developed to further improve the performance. However, the accuracy of an automatic speech recognition (ASR) system in real applications still lags behind that in controlled testing conditions. This raises the old and unsolved problem called training-testing mismatch, i.e., the training set cannot match the new acoustic conditions or fails to generalize to new speakers. Thus a variety of acoustic model compensation and adaptation methods have been proposed, to better deal with unseen speakers and mismatched acoustic conditions.

This study specifically focuses on speaker adaptation, i.e., modifying a general model, commonly a speaker-independent acoustic model (SI AM), to work better for a specific new speaker, though the same adaptation technique can be applied to other mismatched conditions. The history of acoustic model speaker adaptation can be traced back to the GMM-HMM era <cit.>, while the focus has been shifted to neural networks since the rise of DLMs. Various approaches have been developed for neural network acoustic model adaptation <cit.> and they can be roughly categorized into three classes: speaker-adapted layer insertion, subspace method and direct model adapting.





In the category of speaker-adapted layer insertion, linear transformation, which augments the original network with certain speaker-specific linear layer(s), is a simple-but-effective approach. Common methods include linear input network (LIN) <cit.>, linear hidden network (LHN) <cit.>, and linear output network (LOH) <cit.>, just to name a few. Among them, LIN is the most popular one. Learning hidden unit contribution (LHUC) <cit.> is another type of speaker-adapted layer insertion method that makes the SI network parameters to be speaker-specific by inserting special layers to control the amplitude of the hidden layers. 

Another category, subspace method, aims to find a low dimensional speaker subspace that is used for adaptation. The most straightforward application is to use subspace-based features, e.g., i-vectors <cit.>, as a supplement of acoustic features in the neural network for acoustic model training, or speaker adaptive training (SAT). Another approach, serving the same purpose with auxiliary features, is called speaker codes <cit.>. A specific set of network units for each speaker is connected and optimized with the original SI network. Note that i-vector based SAT has become a standard in the training of deep neural network acoustic models <cit.> as this simple trick can bring small-but-consistent improvement.

A straightforward idea is to use new speaker's data to adapt the DNN parameters directly. Retraining/fine-tuning the SI model using the new data is the simplest way, which is also called retrained speaker independent (RSI) adaptation <cit.>. To avoid over-fitting, conservative training, such as Kullback-Leibler divergence (KLD) regularization <cit.> is further introduced. This approach tries to force the posterior distribution of the adapted model to be closer to that estimated from the SI model, by adding a KLD regularization term to the original cross entropy cost function to update the network parameters. Although quite effective, this approach results in an individual neural network for each speaker.

To the best of our knowledge, there still lacks a thorough experimental comparison between different speaker adaptation methods in the literature, especially when the DNN-based acoustic models (AMs) have been advanced greatly since the introduction of these adaptation techniques. In this paper, we aim to close this gap by providing an empirical evaluation of three typical speaker adaptation methods: LIN, LHUC and KLD. Adaptation experiments are conducted on a strong TDNN-LSTM acoustic model (well trained i-vector based SAT-DNN acoustic model with cMLLR <cit.>) tested with different size of adaptation data. More challengingly, here, the source and target we are concerned with are standard Mandarin speaker model and accented Mandarin speaker model. We compare the performance of different methods and their combinations. The speaker adaptation performance is also examined by speaker's accent degree. In a word, we would like to provide readers a big picture on the selection of speaker adaptation techniques.





The rest of this paper is organized as follows. In Section <ref>, we briefly introduce LIN, KLD, LHUC and give a discussion on their abilities. Next, we describe a series of experiments and report the results in Section <ref>. Finally, some conclusions are drawn in Section <ref>.



§ SPEAKER ADAPTATION ALGORITHMS




 §.§ LIN




Linear input network (LIN) <cit.> is a classical input transformation approach for neural network adaptation. As shown in Figure <ref>, LIN assumes that the mismatch between training and testing can be captured in the feature space by employing a trainable linear input layer which maps speaker dependent speech to speaker independent network (i.e., acoustic model). The inserted layer usually has the same dimension as the original input layer and is initialized to an identity weight matrix and 0 bias. Unlike other layers of the neural network, linear activation function f(x)=x is used for this additional layer.

During adaptation, standard error back-propagation (BP) is used to update the LIN's parameters while keeping all other network parameters fixed, by minimizing the loss function (e.g., cross entropy, mean square error) of the original AM. After adaptation, each speaker-specific LIN captures the relations between the speaker and the training space. Finally, for each testing speaker, the corresponding LIN is selected to do feature transformation and the transformed vector is directly fed to the original unadapted AM for speech recognition.



 §.§ KLD Regularization




As a popular conservative training adaptation technique, Kullback-Leibler divergence (KLD) <cit.> regularization tries to force the posterior distribution of the adapted model to be closer to that estimated from the SI model. By contrast, the L_2 regularization aims to keep the parameters of adapted model to be closer to those of the SI model.

For acoustic model training, it is typical to minimize the cross entropy (CE)


    ℱ_CE = - 1/N∑_t=1^N∑_y=1^Sp̃(y|x_t) log p(y|x_t) ,

where N is the number of training samples, S is the total number of states, p̃(y|x_t) is the target probability and p(y|x_t) is neural network's output posteriors. We usually use a hard alignment from an existing ASR system as the training labels and set p̃(y|x_t)=δ(y=s_t), where δ is the Kronecker delta function and s_t is the label of t-th sample. By adding the KLD term to Eq. (<ref>) we get the following optimization criterion:


    ℱ_CE   = (1-ρ) ℱ_CE - ρ1/N∑_t=1^N∑_y=1^Sp^SI(y|x_t) log p(y|x_t) 
       = - 1/N∑_t=1^N∑_y=1^S[ (1-ρ) p̃(y|x_t) + ρ p^SI(y|x_t) ] log p(y|x_t) 
       = - 1/N∑_t=1^N∑_y=1^Sp̂(y|x_t) log p(y|x_t) ,

where ρ is regularization weight and we have defined


    p̂(y|x_t) ≜ (1-ρ) p̃(y|x_t) + ρ p^SI(y|x_t) .



By comparing Eq. (<ref>) and Eq. (<ref>), we can find that applying KLD is equivalent to changing the target distribution in the conventional BP algorithm. When ρ=0, we can regard this configuration as RSI, i.e., retraining the SI model directly using the traditional CE loss.



 §.§ LHUC




As shown in Figure <ref>, learning hidden unit contribution (LHUC) <cit.> modifies the SI model by defining a set of speaker dependent parameters θ for a specific speaker, where θ={r^1,⋯,r^L} and r^l is the vector of speaker dependent parameters for l-th hidden layer. Then the element-wise function a(·) is adopted to constrain the range of r^l and the speaker dependent hidden layer output can be defined as the following function:


    h^l = a(r^l) ∘ϕ^l(W^l⊤h^l-1) ,

where ∘ is an element-wise multiplication and a(·) is typically defined as a sigmoid with amplitude 2, i.e.,



    a(r^l) ≜2/1+exp(-r^l) ,

to constrain the range of r's elements to [0,2 ].

LHUC, given adaptation data, actually rescales the contributions (amplitudes) of the hidden units in the model without actually modifying their feature receptors.  At the training stage, θ is optimized with the standard BP algorithm while keeping all the other parameters fixed for a specific speaker. During the testing stage, the corresponding θ is chosen to constrain the amplitudes of hidden units in order to get more accurate posterior probability for the speaker.



 §.§ Discussion and Combination








We compare the three speaker adaptation approaches in terms of adapted parameter size and modification on the AM.


  * Size of Adapted Parameters: LHUC has minimal adapted parameters, followed by LIN. For KLD regularization, since each speaker has a fully adapted neural network AM, it results in the largest size of adapted parameters.

  * Modification on AM: In the KLD regularization based adaptation, we do not need to change the original AM network structure, while only changing the loss function. By contrast, we need to adjust the network structure, e.g., inserting layers in the use of LIN and LHUC. However, we need to take extra burden to find an appropriate regularization weight ρ in the KLD regularization based adaptation, which is searched through the validation set.


The three approaches perform network adaptation from different aspects and thus can be integrated to expect some extra benefits. LIN and LHUC, both without changing the parameters of the original SI network, can be directly integrated. In other words, LIN's parameters and the speaker dependent parameters θ are updated using the target speaker's data while keeping the parameters of the original network intact. As the KLD adaptation itself needs to update the parameters of the original network, in the integration of LIN/LHUC with KLD, we only use Eq. (<ref>) as the loss function to update LIN's parameters or/and the speaker dependent parameters θ while still keeping the original SI network parameters unchanged.





§ EXPERIMENTS








 §.§ Experimental setup


In the experiments, we used a Mandarin corpus that consists of 3,000 speaker (about 1000hrs) with standard accent to build a baseline TDNN-LSTM AM. Before NN model training, the alignments were achieved from a GMM-HMM AM, combined with fMLLR,  trained using the same dataset. Our speaker adaptation dataset consists of 10 Mandarin speakers from Hubei Province of China and each speaker contributes 450 utterances (about 0.5hr/speaker). Note that the 10 speakers have different levels of accents and we expect that a good speaker adaptation technique should handle different levels of accents. We randomly selected 50 utterances as the cross validation set, 100 utterances as the test set and the others as the training set. In the adaptation experiments, we varied the number of training utterances from 5 to 300 to observe the performances of different data size.

For the baseline SI acoustic model, 40-dimensional Mel filter-bank cepstral coefficients (MFCCs) spliced with 2 left, 2 right frames and 100-dimensional i-vector, further transformed to 300-dimension with linear discriminate analysis (LDA), were used as the network input. The output softmax layer has 5,795 units representing senones. Moreover, the TDNN-LSTM model has 6 TDNN layers (520 neurons) and 3 LSTMP layers <cit.> (520 cells with 130 recurrent nodes and 130 non-recurrent nodes). Network training started from an initial learning rate of 0.0003 [More details about this architecture can be found in Kaldi: egs/wsj/s5/local/nnet3/run_tdnn_lstm.sh]. A trigram language model (LM) was used in evaluating both the baseline and the adapted models. Moreover, all of our experiments were based on Kaldi <cit.>.



 §.§ Results of Baseline Model


Table <ref> shows the character error rate (CER) for each speakers, tested with the baseline AM. We can see that the baseline model performs differently for each speaker and the average CER is 24.86%. The CER has a wide range from 3% to 56.62%. We manually checked the recordings from different speakers and found that speaker S05 had heavy accent and speaker S01 had slight Mandarin accent. This huge difference gives the speaker adaptation methods a big challenge. We will report the adaptation results in terms of accent levels later in Section <ref>. 



 §.§ Comparison of LIN, RSI, KLD and LHUC


We investigated the adaptation ability of LIN, RSI, KLD and LHUC using different amount of adaptation data. Previous studies on LHUC <cit.> have demonstrated that adapting more layers in the network can get continuously better accuracy. Hence we inserted LHUC parameters after each hidden layers. For LIN, models were adapted with a small learning rate of 0.00001, while 0.001 and 0.01 were used as an initial learning rate for KLD and LHUC, respectively. From the results shown in Figure <ref>, we can see that KLD achieves the best performance and is more stable than RSI on different amount of adaptation data for all speakers. LIN, as simple layer-insertion method, is also helpful, but its performance is not as good as the other two. For RSI and LHUC, their performances are comparable in most cases, but over-fitting is occurred for RSI when the adaptation data size exceeds 200.

Furthermore, similar with <cit.>, we gave a deep investigation on KLD-based adaptation and results are shown in Figure <ref>. First, unlike the results in <cit.>, where using small amount of data (5 or 10 utterances) for KLD adaptation is unfortunately harmful, we still can obtain apparent CER reduction when the same size of data are used for adaptation. We believe that this is because our testing speakers have noticeable accents, i.e., the difference between the SI data and the target speaker data is significant. The comparison of different ρ in the range of [0.0625, 0.5] also indicates that reasonable CER reduction can be obtained even with a small ρ for different size of adaptation data.  The figure also clearly shows that a medium regularization weight (e.g., 0.25) is preferred for larger and smaller adaptation sets and a smaller regularization weight (e.g., 0.0625) is better used for medium size of adaptation set. We also compared the performances between different speakers. Results from Figure <ref> shows that KLD works for every testing speaker and the speaker with highest CER on the SI model ( i.e., S5, with the heaviest accent) achieves the largest CER reduction. But with the increase of adaptation data, the gain on each speaker becomes smaller and smaller.



 §.§ Combinations


We further experimented on method combinations and results are summarized in Figure <ref>. We can see the combinations of different methods cannot bring salient improvements and the best performance is achieved by KLD only. Even badly, any combination with LIN will drag the performance to LIN.  Combining LHUC with KLD can obtain slightly better result than the vanilla LHUC for very small (less than 10) and large (more than 200) adaptation dataset. But for small adaptation data size (20∼80), LHUC itself performs better.









 §.§ Different degrees of accent


As shown in Table <ref> earlier, the baseline AM's performance varies on different speakers. It's necessary to compare different adaptation methods in terms of accent level. We manually categorized the 10 speakers into 3 accented groups: slight, medium and heavy according to their performances on the baseline AM in Table <ref>. According to the accent level, results are summarized in Figure <ref> (slight), Figure <ref> (medium) and Figure <ref> (heavy). From Figure <ref>, we can see that LHUC performs consistently the best for the adaptation on slight-accent speakers, while KLD and RSI are not stable. We believe that this is because the baseline model is trained using data mostly from Mandarin speakers with standard accent and the baseline model itself is robust enough; in this case, direct update on the network parameters may be harmful. Observing Figure <ref>, for medium-accent speakers, we can see that KLD and LHUC can get comparable performances with much lower CER than LIN. RSI is still not stable and over-fitting happens when a large adaptation data set is used. If memory footprint is a major consideration, we suggest to use LHUC as its has a small set of adapted parameters for each speaker; otherwise LHUC and KLD can be both  considered for medium-accent speakers. As shown in Figure <ref>, for heavy-accent speakers, KLD can get absolutely the best performance among the three methods, followed by LHUC, while LIN still performs the worst. We believe that KLD's superior performance is because the posterior distribution of the heavy-accent speech is far away from that of the unaccented speech; in this case, directly updating the network parameters or dragging the two distributions closer, is the most effective means. This also clarifies why RSI is better than LHUC and why we cannot observe over-fitting in this condition.



§ CONCLUSIONS


In this work, we have systematically compared the performance of three widely-used speaker adaptation methods on a challenging dataset with accented speakers. We show that i-vector based SAT-DNN AM is already strong enough to slight-accent speakers but performs badly to medium- and heavy-accent speakers. By using LIN,  KLD, LHUC, we can further improve the speech recognition performance not only for medium- and heavy-accent speakers, but also for slight-accent speakers. Moreover, the experimental results show that, in general, KLD and LHUC consistently outperform LIN and KLD demonstrates the best performance. The combination of different methods cannot bring salient improvements. For the adaptation on slight-accent speakers, LHUC is preferred with consistent improvement, while KLD and RSI are not stable. For medium-accent speakers, KLD and LHUC can get comparable performances with much lower CER than LIN. For heavy-accent speakers, KLD can get absolutely the best performance, followed by LHUC, while LIN still performs the worst. 




§ ACKNOWLEDGEMENTS


The authors would like to thank Jian Li, Mengfei Wu and Yongqing Wang for their supports on this work. The research work is supported by the National Key Research and Development Program of China (Grant No.2017YFB1002102) and the National Natural Science Foundation of China (Grant No.61571363).

IEEEtran




