Pretrained Semantic Speech Embeddings for End-to-End Spoken Language Understanding via Cross-Modal Teacher-Student Learning

Spoken language understanding is typically based on pipeline architectures including speech recognition and natural language understanding steps. Therefore, these components are optimized independently from each other and the overall system suffers from error propagation. In this paper, we propose a novel training method that enables pretrained contextual embeddings such as BERT to process acoustic features. In particular, we extend it with an encoder of pretrained speech recognition systems in order to construct end-to-end spoken language understanding systems. Our proposed method is based on the teacher-student framework across speech and text modalities that aligns the acoustic and the semantic latent spaces. Experimental results in three benchmark datasets show that our system reaches the pipeline architecture performance without using any training data and outperforms it after fine-tuning with only a few examples.


Introduction
Recent developments in the fields of electronics, computations and data processing have led to an increased interest in smart assistants with speech interfaces. It is likely driven by the fact that usually people can learn to use speech for interaction intuitively without any special training [1] and make it a primary medium of information exchange. However, speech poses a major challenge to a machine when it comes to the task of extraction of information intended to be transmitted by a human speaker, also known as Spoken Language Understanding (SLU) [2]. The key difficulty here is that speech is highly variable, e.g. depending on room acoustics, recording equipments, and contains rich information about speakers, e.g. their cultural background and physiology [3]. Some of them are not useful for SLU. For this reason, the information extraction task is often performed on the text representation using Natural Language Understanding (NLU) methods [4], while Automatic Speech Recognition (ASR) systems [5,6] conduct the conversion from speech to text. ASR step removes redundant information from the input and provides some kind of normalized form on the output. However, at the same time it causes loss of potentially useful information that can not be encoded in the text representation, such as prosody, loudness and speech rate. Besides, the operation of finding the most probable sequence of words for speech input is computationally expensive. This is partly solved by various heuristics avoiding exploration of less probable hypothesis [7,8,9], what in turn introduces additional errors propagated to NLU component. Finally, the sequential design of pipeline approach leads to unavoidable source of latency, because NLU component can not start its work before ASR is finished, and it is not desirable in the interactive context of smart assistant. The problems of pipeline approach described above can be solved by end-to-end SLU methods.
Existing works on end-to-end SLU modeling either focus on supervised downstream task, for example dialog act classification [10], intent detection [11,12,13] and slot filling [14], or target a generic semantic embedding [15,16,17] usually inspired by such successful models as word embeddings Word2Vec [18] and contextual text embeddings BERT [19]. Highly variable and complex nature of speech leads to large amounts of both data and computational resources required for SLU training compared to NLU training, especially for recently popular approach based on contextual embeddings. While data requirements could be satisfied for unsupervised approaches, computational resources are still a problem. Fortunately, most of the modern language processing methods, including ASR and NLU, are based on neural networks and deep learning. Deep learning offers an easy way to transfer knowledge between learned tasks. This technique is referred as transfer learning and it is successfully applied in both ASR [20,21] and NLU [22,19]. Therefore, transfer learning should be a promising direction to explore for SLU as well. Several reports [23,12,13,24,14,17] indicate that transfer learning from audio modality through pretraining on ASR task or, alternatively, speech autoencoding, is helpful for downstream SLU tasks. Transfer learning from text modality, however, has been applied only for Speech2Vec [16] and SpeechBERT [17] so far.
We propose a novel method that combines parameters transfer from well trained end-to-end ASR systems [25] such as pretrained ESPNet [26] and end-to-end NLU models such as pretrained BERT [19] with Teacher-Student learning [27,28] for final alignment of SLU output space to NLU output space in order to construct end-to-end SLU model allowing few-shot transfer of downstream tasks from text to speech. By doing so, we enable pretrained end-to-end contextual embeddings such as BERT to process acoustic features. In particular, we aim to generate fixed length vectors with semantic representation from speech segments of variable length. Transfer learning from both text and audio modalities makes our approach mostly similar to [17] and [16]. Unlike these approaches, we are not concerned with the difficult question of language units in the input and always produce single utterance level representation. We evaluate our model on several downstream tasks of utterance classification, what relates this work to previous research [29,24]. It is one of the first attempts of cross-modal transfer for language understanding and pretrained NLU model adaptation in SLU. It also appears to be the first report on purely Transformer based SLU model, although it is not the main interest of our study.   Figure 1: End-to-end SLU using cross-modal T-S learning.

Method
we use Encoder block of pretrained end-to-end ASR model [25] in order to covert acoustic features of speech signal to hidden representation. Second, we feed the hidden representation through a learnable linear mapping to pretrained masked language model [19], fine-tuned to produce semantic sentence embedding, which serves as NLU model. Finally, we utilize teacher-student learning method in order to align output of our SLU model to output of pretrained NLU model. Both ASR and NLU models are based on Transformer architecture [30] widely used for sequence processing.

End-to-end ASR
ASR model implements sequence-to-sequence approach and contains two major blocks called encoder and decoder. Encoder takes sequence X with acoustic features from the input utterance on input and outputs encoded sequence Xǫ with hidden representation. Decoder takes the encoded sequence Xǫ on input and outputs target sequence Y with text tokens representing transcription of the input utterance. ASR model is trained to minimize weighted sum of cross-entropy objective function calculated from decoder outputŶ and ground truth transcription Y with CTC objective function calculated from learnable linear mapping of encoder output Xǫ and ground truth transcription Y .

NLU
NLU model is a neural network that takes sequence X with text tokens on input and produces encoded sequence Xǫ. Semantic sentence embedding vector y is obtained by applying pooling operation to the encoded sequence Xǫ. The model's parameters are initially pretrained with the tasks of masked token and next sentence prediction from the encoded sequence Xǫ representing contextual text token embeddings, as it is done with BERT model [19]. After that, the model is extended with pooling operation over the encoded sequence Xǫ, producing pooled output y, and is fine-tuned on specialized datasets to encode more semantic information to the pooled output y.

Teacher-Student learning
Teacher-Student learning minimizes distance-based objective function between outputs of two models on same or equivalent inputs with the aim to update Student model's parameters so that its output becomes more similar to the output of Teacher model.

NLU model
We use pretrained bert-base-nli-stsb-mean-tokens Sentence-BERT model [34]. The model itself is fine-tuned from the well-known pretrained bert-base-uncased model [19]. Transformer network has attention dimension 768, feedforward inner dimension 3072, 12 heads and 12 blocks. Input text is tokenized to 30,000 subword units. The model is pretrained with masked LM and next sentence prediction tasks on BooksCorpus and English Wikipedia datasets. Pooling operation MEAN is added to obtain the sentence embedding y from the encoded sequence Xǫ. The sentence embedding is first fine-tuned on SNLI and MultiNLI datasets for 3-way classification between contradiction, entailment and neutral classes for a given pair of sentences using cross-entropy objective function. After that, the sentence embedding is fine-tuned on STSb dataset for prediction of cosine similarity for a given pair of sentences using mean-squared-error objective function.

SLU model
SLU model is constructed by combining ASR model's encoder with self-attention blocks of NLU model, so that NLU model recieves the hidden representation from ASR encoder instead of the output of input embedding layer of NLU model. Linear layer is added between ASR encoder and NLU blocks to map the dimension of hidden representation from 512 to 768. Finetuning is performed using Teacher-Student approach by minimizing the distance between output of SLU model for speech recordings and output of NLU model for corresponding transcriptions. We conduct fine-tuning experiments with cosine, L2 and L1 distance based objective functions. SLU model acts as a Student, and we select empirically, which parameters to update during the fine-tuning. NLU model acts as a Teacher, and we freeze its parameters. We employ smaller acoustic dataset consisting of LibriTTS, Common Voice 3, and M-AILABS corpora with a total amount of 1453 hours for the fine-tuning. Our motivation here is to utilize richer transcriptions with punctuation available in these datasets and to supply NLU model with extra information for potentially semantically finer sentence embeddings. We use the transcriptions as is and do not apply any text preprocessing that is usually done in ASR training, including our end-to-end ASR model. Validation data is the validation subset of LibriTTS corpus with a total duration of 15 hours. We do not apply SpecAugment during the fine-tuning, because it yielded worse results in our early experiments.

Evaluation
SLU model is evaluated on two downstream tasks, dialog act (DA) classification and intent classification, both of which are utterance classification tasks. DA classification is evaluated on two corpora: ICSI Meeting Recorder Dialog Act Corpus (MRDA) and NXT-format Switchboard Corpus (SwDA). Intent classification is evaluated on Fluent Speech Commands (FSC) corpus. Table 1 summarizes the datasets. In order to perform utterance classification, we first train a one layer feed-forward classifier on sentence embeddings, produced by the NLU model from the ground truth transcriptions of training subset, using cross-entropy objective function. After that, we test the classifier on semantic utterance embeddings, extracted from the recordings of testing subset using the SLU model. We report accuracy values as a percentage of correctly classified utterances from the total number of utterances.

Baseline
Traditional approach to SLU tasks is a pipeline of ASR followed by NLU, and we adopt it as a baseline while employing the same ASR and NLU models as in the rest of the experiments. Table 2 reports the results of NLU on ASR output as well as on the ground truth transcriptions. This is done to separate the degradation introduced by ASR errors from imperfectness of NLU component and potential labeling mistakes.

. Layers for fine-tuning
Our first set of experiments is designed to determine which layers of SLU model should be fine-tuned after the combination of parameters transferred from ASR encoder and NLU. As mentioned before, we insert a linear mapping layer between former ASR and NLU layers because of the difference in dimensionality. It is initialized randomly and its parameters are always updated during the fine-tuning step. In addition to that, we try to fine-tune various amount of layers closest to the mapping layer, meaning top layers of former ASR encoder and bottom layers of former NLU. We do so, because for these layers the output (for ASR encoder) or the input (for NLU) is expected to change after the parameters transfer in contrast to bottom layers of former ASR encoder and top layers of former NLU, where input and output should not change.
We run fine-tuning for 10 epochs using square root learning rate scheduling [30] with 300,000 warmup steps and learning rate coefficient 50, and use cosine distance based objective function. The results are given in Table 3. While it is not completely clear how many layers should be fine-tuned, we can conclusively tell that fine-tuning of former ASR encoder layers is more beneficial than former NLU layers. We decide to fine-tune the two top former ASR encoder layers. The results also illustrate that the optimization of SLU model for smaller distance of its output from the output of NLU model is general enough and translates to accuracy improvements in the downstream tasks, although not in all cases.

Learning rate schedule
After deciding which layers to fine-tune, we run a series of experiments to determine the best learning rate schedule. Table 4 presents the combinations of learning rate constant and number of warmup steps explored by us. When we increase number of warmup steps, we notice positive effect from slower learning rate ramp up. However, as number number of warmup steps becomes close to the total number of fine-tuning steps, we have to increase number of epochs from 10 to 20 in order to see the whole fine-tuning process.   Table  5. Overall, these results indicate that the evaluated objective functions behave similarly in this task, however L1 distance based objective function yields slightly better results.

Further supervised fine-tuning on downstream tasks
The resulting end-to-end utterance classification model is fully differentiable and can be further optimized for a downstream task by applying standard supervised neural network training methods with few labeled speech samples. This feature should be helpful for the full exploitation of information that is relevant for the task and is encoded in speech, what is less trivial to implement in the traditional ASR and NLU pipeline setup. We examine whether it is useful in practice by running standard supervised classifier training on few samples from training subsets. In each downstream task, the starting model is a combination of pretrained SLU and pretrained linear classifier. SLU model is a longer fine-tuned (61 epochs instead of 20) L1 configuration from the previous experiment. Linear classifier is pretrained on training transcriptions of the corresponding corpus, as in the previous experiments. Table 6 compares the results of fine-tuning of the output layer alone and together with two former ASR encoder layers. We conclude that end-to-end approach indeed can overcome the error propagation problem of pipeline SLU approach by the automatic propagation of the error signal back to relevant parts of SLU system.

Qualitative Analysis
We attempt to assess the differences between the pipeline and end-to-end SLU approaches in greater detail by looking at the accuracy values on groups of utterances split by the WER lev- els of the baseline ASR system. Our hypothesis is that the pipeline system would make more mistakes on the more challenging recordings characterized by higher WER values, because ASR systems are optimized for phonetic or graphemic similarity to the ground truth and are more likely to lose semantic information in case of errors, compared to a model explicitly optimized for the semantic information extraction. Figure 2 shows the accuracy values of the baseline pipeline system and the best end-to-end SLU model without fine-tuning for downtream tasks. The utterances are grouped by WER of the baseline ASR in 10% ranges. The ranges with WER > 100% are not included for brevity, they account for 487, 3847 and 44 utterances in SwBD, MRDA and FSC testing subsets respectively. The results confirm fully our hypothesis on FSC dataset and confirm it to some extent on SwBD and MRDA datasets.

Conclusions
We proposed to combine parameters transfer from well trained ASR and NLU models with Teacher-Student learning for final alignment of SLU output space to NLU output space in order to construct end-to-end SLU model allowing few-shot transfer of downstream tasks from text to speech. We outlined necessary steps and settings for the practical pretrained NLU model adaptation in SLU via cross-modal transfer. Experimental results in three benchmark datasets showed that our system reaches to the pipeline architecture performance without using any training data and outperforms it when fine-tuning with a few examples. The results of this research support the idea that text pretrained contextual embeddings can be useful for tasks outside of text modality. The present study also adds new tasks to the growing body of research of language processing methods using Transformer neural networks.