{'fraction_non_alphanumeric': 0.029621125143513203, 'fraction_numerical': 0.011251435132032146, 'mean_word_length': 5.057371349095966, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Knowledge Distillation and Data Selection for Semi-Supervised Learning in CTC Acoustic Models

Semi-supervised learning (SSL) is an active area of research which aims to utilize unlabelled data in order to improve the accuracy of speech recognition systems. The current study proposes a methodology for integration of two key ideas: 1) SSL using connectionist temporal classification (CTC) objective and teacher-student based learning 2) Designing effective data-selection mechanisms for leveraging unlabelled data to boost performance of student models. Our aim is to establish the importance of good criteria in selecting samples from a large pool of unlabelled data based on attributes like confidence measure, speaker and content variability. The question we try to answer is: Is it possible to design a data selection mechanism which reduces dependence on a large set of randomly selected unlabelled samples without compromising on Word Error Rate (WER)? We perform empirical investigations of different data selection methods to answer this question and quantify the effect of different sampling strategies. On a semi-supervised ASR setting with 40000 hours of carefully selected unlabelled data, our CTC-SSL approach gives 17% relative WER improvement over a baseline CTC system trained with labelled data. It also achieves on-par performance with CTC-SSL system trained on order of magnitude larger unlabeled data based on random sampling.


Introduction
In recent years, introduction of Recurrent Neural Networks (RNNs), in particular, Long Short Term Memory (LSTM) RNNs [1] have been shown to outperform feed-forward neural networks [2] and significantly improve the performance of automatic speech recognition (ASR) systems. More recently, it has been shown that LSTMs when trained with CTC loss followed by sequence discriminative training can achieve state-ofthe-art performance on various large-scale acoustic modelling tasks [3,4]. However, a primary bottleneck of these frameworks is their dependence on availability of large amounts of labeled data. Since collection and transcription of large amounts of speech data is costly and time-consuming, techniques to leverage large amounts of unlabelled data for acoustic modeling are explored under the framework of semi-supervised learning (SSL) [5,6].
SSL for ASR involves automatic generation of labels for unlabelled data and leveraging them along with labelled data for building ASR systems. Self-training has been shown to be an effective SSL framework where an initial system is trained on labelled data to generate machine transcriptions for unlabelled data [7,8,9,10,11]. However, the effectiveness of such systems gets constrained by the "goodness" of the initial system and "quality" of machine transcripts. Teacher-Student based learning [12] originally proposed in the context of knowledge distillation (KD) for model compression is another approach used for SSL. KD involves using a strong teacher model trained on labelled data to generate labels which are used to train a student model to match teacher output distribution.
The effectiveness of KD has been well established in speech recognition tasks [13,14,15,16]. However, conventional frame-level based KD approaches may not be directly feasible for CTC trained models due to 'alignment-free' nature of CTC loss function. Modifications have been proposed in recent studies which either attempt to align teacher and student models' spike timings [17,18] or propose KD at the level of sequences instead of frames [19,20]. The cost associated with aligning output spikes from teacher and student model make the framelevel based approaches computationally expensive when using large amounts of unlabelled data. Existing studies on sequencelevel KD for CTC entail a certain degree of complexity in sampling n-best hypothesis [20] or computing forward-backward posteriors from teacher model [19]. A simplified version of sequence-level KD has been studied in the context of neural machine translation (NMT) [21], which essentially collapses the entire label sequence space to a single 1-best sequence. In this work, we simplify this assumption further by approximating the entire label sequence space by a single 1-best sequence obtained by concatenating frame-level argmax output from teacher model. Our results demonstrate that such a framework provides better WER as compared to a standard self-training based approach.
In a recent study in KD [22], it was shown that augmenting labelled data with 1 million hours of randomly selected unlabelled data improves relative WER by 10-20%. However, processing such large quantities of unlabelled data and subsequent training requires access to powerful compute and storage resources and leads to significant increase in training time. The prime motivation for this work is to seek answer to the following question: Instead of large amounts of randomly selected unlabelled data, is it possible to design an intelligent data selection scheme which can achieve similar WER gains but with lesser data?
Majority of data selection approaches studied in the past have relied on confidence scores, entropy based confidence measure and local-global entropy minimization among others for bootstrapping initial seed model with additional unlabelled data [23,24] in the self-learning framework. To the best of our knowledge, data selection in the context of teacher-student learning has not been explored extensively. We present an empirical study of the role of different data selection mecha-nisms based on confidence and Natural Language Understanding (NLU) based domain distribution as well as speaker and content diversity on WER of CTC-SSL based student model. Overall, our CTC-SSL framework achieves 17% relative WER improvement over a baseline CTC system with labelled data only and on-par performance with a CTC-SSL model trained on an order of magnitude higher randomly sampled unlabelled data.

SSL via Knowledge Distillation for CTC
One of the tasks of speech recognition involves mapping a sequence of frame-level labels (called 'path' and denoted as π) into a label sequence (denoted as h) of length equal to or less than the number of frames. In the CTC framework [3], a path is converted into a label sequence by introducing deletion of repeated as well as blank labels. This conversion is referred to as 'CTC mapping' with function B, where h = B(π). Since multiple possible paths can be mapped into an identical label sequence, conditional probability of label sequence h given input sequence x is defined by Equation (1).
Posterior probability P (πt|xt) of label πt at frame t is typically modeled with a RNN. NX denotes the number of frames in utterance X. In this case, training objective is to minimize negative log-likelihood of label sequence given input frames as given by LCT C = −ln(P (h|X)).
For unlabelled dataset, our aim is to generate pseudo-labels from a strong teacher model. Teacher-Student learning based knowledge distillation (KD) has been studied for CTC acoustic models primarily from a model compression standpoint [20]. The objective of KD as shown in Equation (2) is to train a smaller student model using the output of a stronger teacher model as training labels.
Where PT (πt|xt), PS(πt|xt) denote the teacher and student model output respectively and Z denotes the set of all labels. This approach is referred to as frame-level KD. In sequence-level based KD a student model is trained by minimizing cross entropy loss between probability distributions of label sequences h generated by teacher and student model as shown in Equation (3).
Where H denotes the set of all possible label sequences. Substituting Equation (1) into Equation (3) for PT (h|X), we see that computation of LCT C−KDseq involves a summation over all paths π such that B(π) ∈ H which is a computationally expensive operation. Hence, similar to [21], we approximate the summation by its maximum value i.e. a single 1-best path to get Equation (4).
For further simplification, we apply the greedy decoding approximation suggested by Equation (4) in [3] which approximates the best path by a path formed by concatenating the most probable label at each time step. Due to the spiky nature of CTC posteriors, PT (πt|xt) can be approximated by a Kronecker delta function at arg max PT (πt|xt). Hence, substituting π * as path formed by concatenating the most probable labels π * t at each time frame t from teacher model's output, Equation (4) can be transformed into Equation (5).
Overall, this approximation to sequence-level knowledge distillation for unlabelled data can be described in following steps: 1. Sample frame level labels π * t for unlabelled data by taking an argmax over teacher posterior vector at each time step t.

2.
Convert frame-level label sequence π * to CTC labels by applying CTC mapping operator B(π * ) = h * 3. Use this label sequence h * for conventional CTC model training.
Combining loss functions for labelled data DL and unlabelled data DU , we get the overall training loss function as shown in Equation (6).

Data Selection
We discuss data selection strategies used in our SSL experiments below:

Utterance-level Confidence Scores
Utterance-level confidence scores, as described in our earlier work [25], are used to select data for SSL experiments (details will be discussed in the next section). Utterance-level confidence scores are produced by a logistic regression classifier, which takes hand-crafted decoder features of an utterance as input and produces a posterior probability estimate that indicates whether ASR hypothesis corresponding to utterance is correct (WER=0) or not (WER =0). Input features for training the confidence model contain ASR decoder information such as utterance posterior, acoustic cost, language model cost, number of words, maximum depth of n-best output, frame length and statistics on the number of arcs and nodes explored in decoding trellis. The confidence model is trained using the cross-entropy criterion.

Utterance-level Domain Information
Utterance-level domain information, estimated by our internal Natural Language Understanding (NLU) system, is also used to select data for SSL experiments in this study (details will be discussed in the next section). This is because ASR accuracies vary across domains (such as Music, Weather and Information etc.) due to domains having varying complexity of utterances. We demonstrate the effect of selecting domain-specific SSL data on per-domain ASR accuracies.

Common filters
Apart from utterance-level confidence and domain information, additional filters are also employed to optimize the amount of unlabelled data: •

Experiments and Results
Experimental setup used in this study is described in Table 1. AM training consists of cross-entropy training, which produces a seed AM for the final stage CTC training. The training data for building baseline AMs consist of anonymized labelled dialectspecific English data from Alexas production traffic. In case of SSL based AMs, additional unlabelled data is interleaved with the labelled data used for building baseline AM. In addition, the teacher AM training leverages multi-dialect CE and CTC training phases, where 45000 hours of labelled data is pooled across multiple English dialects, prior to dialect-specific sMBR training to boost its accuracy. For decoding, we use dialect-specific Language Models (LMs) that are trained on both production and external data sources.

CTC-SSL with random sampling
The results of British (en-GB) and Indian (en-IN) English dialect-specific baseline and teacher models are shown in Table  2. It confirms that the teacher is indeed better than the baseline, and is a good candidate for CTC-SSL experiments. As a next step, we randomly select 250000 hours of data (similar to [22]), produce pseudo-labels using appropriate teacher model, and build corresponding student model using CTC-SSL technique discussed in Section 2. The results indicate that leveraging 250K hours of randomly selected unlabelled data and 8K-16K hours of labelled data using CTC-SSL framework yields 17% WERR compared to a baseline that is trained on labelled data alone. This result also serves as an additional data point to evaluate the efficiency of data selection methods described in Section 3.  We also compared our proposed Teacher-Student based approach with Self-training framework for en-IN. For Selftraining, we built a multi-dialect unidirectional LSTM AM (similar to students architecture) using 45000 hours of labelled data which is used to machine-label 250000 hours of SSL data. As shown in Table. 3, Teacher-Student based CTC-SSL outperforms Self-training by yielding 6.3% additional WERR. Therefore, we use BiLSTM based teacher AM for rest of SSL experiments in this paper.  Table 3: WERR comparison between Teacher-Student and Selftraining based SSL models for en-IN

Utterance confidence based sampling
In this section, we provide an empirical analysis of selecting unlabelled data based on confidence distribution for CTC-SSL training. The entire unlabelled data is divided into 10 uniformly spaced bins within [0-1000] ([0-1] scaled by 1000) based on confidence values extracted from anonymized production traffic.We sample 5000 hours of unlabelled data within each bin for en-GB. However, due to anonymized data sparsity constraints, 8000 hours of unlabelled data are sampled within [500-1000] for en-IN such that the total SSL data distribution remains similar across both the locales (∼80% less than randomly sampled 250k). Apart from utterance confidence, common filters described in Section 3.3 are also applied for data selection within each bin. We build CTC-SSL models for each of confidence bins following procedure discussed earlier. Figure 1 quantifies the effect of confidence based unlabelled data selection from each bin on the performance of CTC-SSL system (in terms of WERR) compared to baseline trained on labelled data only. Figure 1: WERR comparison across student models after adding unlabelled data from utterance confidence bins Figure 1 clearly suggests that adding unlabelled data from low to mid range confidence distribution to CTC training positively impacts the performance of ASR system compared to baseline. As the sampling of unlabelled data moves from low to high confidence, it is observed that the contribution of SSL data to student performance drops from 11.7% to 6% for en-GB and 9.7% to 5.2% for en-IN. With respect to the student model, samples drawn from high confidence bins can be considered as 'easy' and hence adding them to existing labelled data is not beneficial. On the other hand, mid to low confidence bins contain 'difficult' samples which when added to student model training will enable better generalization and improved KD from the teacher.
Next, we attempt to come up with a strategy to effectively combine data across multiple confidence bins. The approaches explored are: • Natural distribution (ND): Sample according to production pipeline distribution • Uniform distribution (UD): Sample equally from all bins. • Weighted sampling (WS): Sample based on WERRs in Table 1 The results, including large scale random selection (discussed earlier), are listed in Table 4. 40000 hours of unlabelled data combined using WS and UD methods yield better results than that of ND. Interestingly, these smart unlabelled data selection methods with order of magnitude less data yield results comparable to 250K hours of randomly sampled experiment which reinforces the importance of data selection for semi-supervised learning in ASR.

Utterance domain based sampling
We present empirical analysis of the effect of domain based sampling of unlabelled data in boosting the performance of ASR. Based on NLU domain extracted from anonymized production traffic, we sample 10000 hours and 8000 hours from 5 individual domains [D1· · · D5] in en-GB and en-IN respectively and build CTC-SSL model by augmenting labelled data with individual domain unlabelled data. Apart from using domain  Table 4: WERR comparison across CTC-SSL models with different confidence bin based combination methods information, other filters as described in Section 3.3 are also applied while selecting data for these experiments.

Conclusions
This paper presents an empirical study on large-scale semisupervised learning for CTC acoustic models where a strong offline teacher model is used to generate pseudo-labels for unlabelled data. The unlabelled data is selected based on confidence and domain distribution as well as speaker and content variability. Experimental results on two different dialects reinforce the efficacy of teacher generated pseudo labels and the importance of intelligent data selection methods. It is observed that domain-specific unlabelled data has a strong impact on corresponding WER with little cross-domain impact signifying the importance of such a sampling strategy in boosting the performance of low resource domains. Future work in this direction would be to devise a strategy to leverage both confidence as well as domain diversity in a combined data sampling strategy for SSL. Another important future direction will be to study the impact of word level decoding which incorporates both lexicon and strong language model in improving the quality of teacher generated pseudo-labels.