Automatic Speech Recognition System-Independent Word Error Rate Estimatio
Chanho Park Mingjie Chen Thomas Hain






§ INTRODUCTION

Automatic quality estimation (QE) of machine learning models has gained increasing popularity. Many types of quality estimation have been studied <cit.>, where the aim is to estimate the quality of outputs of models. It was found to be useful for measuring model performance for practical tasks, such as image classification <cit.>, machine translation <cit.>, automatic speech recognition (ASR) <cit.>, speech synthesis <cit.> and speech enhancement <cit.>, especially when there is no oracle references or ground-truth labels.

This paper mainly focuses on quality estimation of ASR system output. ASR research <cit.> has achieved impressive successes and made huge progress on transcription accuracy and computational efficiency. Even so, errors still occur and it remains essential to evaluate the quality of ASR transcripts independently at low cost, as oracle references are not always available, especially in real-world applications or production scenarios.

Several methods to estimate the quality of ASR transcripts have been proposed. As summarised in <cit.>, these methods can be roughly categorised into two classes: method using glass-box and black-box features. For the glass-box methods, to estimate the quality of ASR outputs, intermediate features of ASR systems, such as confidence scores <cit.> are typically used. Another type is a black-box method that directly estimates the word error rate (WER) of transcripts and audio <cit.>. The former methods using a confidence score have been used widely for different applications such as semi-supervised learning <cit.> and speaker-adaptation <cit.>. However, a common issue of ASR systems derived from confidence scores is that they can be overconfident  <cit.>, which leads to low performance of quality estimation. Compared to confidence score-based methods, WER estimators using black-box features are not dependent on the intermediate features from specific ASR systems. Hence, the WER estimators can have computational efficiency advantages at inference because they do not require running expensive ASR decoding. However, a common issue of most previous WER estimators <cit.> is that their training is dependent on factors such as the nature of hypotheses or training datasets. These dependencies may be causes of performance degradation or result in a narrow range of use of WER estimators.

Typically, the training datasets of WER estimators are generated based on speech datasets with ground-truth references. These datasets are composed of pairs of a speech utterance and a hypothesis, as well as the target WER. The latter is derived by comparing a hypothesis with the reference transcript and computing the so-called edit distance. The hypotheses are generated using a specific ASR system. Hence, the WER estimators <cit.> are dependent on the ASR systems and the speech datasets used to generate the training datasets. In this paper, for simplicity, these WER estimators are named system-dependent WER estimators. When estimating the quality of output from another ASR system, system-dependent WER estimators do not perform well and need to be re-trained, which reduces the usefulness of ASR system-dependent WER estimators. In addition, they are also likely to suffer performance degradation on out-of-domain test data, where errors occur that previously have not been observed.

The work in this paper tries to address the issues above by proposing a System-Independent WER Estimation (SIWE) method. Instead of generating training datasets using ASR systems, a range of data augmentation methods are proposed that allow to generate plausible hypotheses. The data augmentation methods deduce hypotheses from ground-truth references by inserting errors. Three types of strategies are used for error insertion: producing insertion, deletion and substitution errors. The training datasets generated from the proposed methods help SIWE to achieve state-of-the-art WER estimation performance. On in-domain test data, SIWE reaches the same level as performance of the system-dependent WER estimators. Furthermore, SIWE outperformed the system-dependent WER estimators on out-of-domain test data.

The main contributions of this paper can be summarised as follows:

	
  * This paper proposes a System-Independent WER estimator, which enjoys a broader range of use than system-dependent WER estimators.
	
  * This paper proposes a new data augmentation method for generating training datasets for WER estimators.
	
  * The proposed SIWE model reaches the same level of performance as system-dependent estimators on in-domain test data and it further outperformed the system-dependent WER estimators on out-of-domain test data.






§ RELATED WORKS



 §.§ Word Error Rate Estimation

<cit.> proposed an estimation method for ASR QE without manual reference transcripts. They trained a WER estimation model using both glass-box and black-box features. <cit.> also proposed a feed-forward neural network called e-WER for predicting the number of errors and word count per spoken utterance. In <cit.>, the features extracted from a phone recogniser were used to improve the performance. Moreover, e-WER3 in <cit.> extended the previous model to predict WER on multiple language data including English. In the study, only black-box features were employed. In addition to these methods, the Fast WER estimator (Fe-WER), has been proposed by <cit.>. This model reduced the computational cost without performance degradation by adopting self-supervised learning representations for speech and text as black-box features. For evaluation, mean absolute error (MAE), root mean square error (RMSE) and Pearson correlation coefficient (PCC) were measured.



 §.§ Automatic Speech Recognition Systems


ASR has experienced impressive success and made huge progress in recent years <cit.>. They have significant differences in many ways, including model architectures, input features and training objectives. One type of ASR systems is based on so-called hybrid modelling <cit.>. The system consists of multiple independent modules and is trained through a training objective, such as lattice-free maximum mutual information (LF-MMI) <cit.>. End-to-end (E2E) ASR systems have gained much attention. For E2E to be robust to the long context, recurrent neural network (RNN) transducer <cit.> has been adopted by <cit.>. Its encoder is jointly trained with prediction networks that depend on previous labels. Moreover, Transformer has been integrated with E2E models: wav2vec 2.0 <cit.>; Conformer <cit.>; Whisper <cit.>. First, wav2vec 2.0 is pre-trained to learn contextualised representation. Then, representation is linearly projected into output tokens. The model is optimised by minimising the connectionist temporal classiﬁcation loss. Second, Conformer is a model based on a transformer and convolutional neural networks for feature extraction. Last, Whisper is an ASR system trained on large amounts of transcripts collected from the internet. It is trained on multiple tasks, such as speech recognition and language identification.



§ ASR SYSTEM-INDEPENDENT WER ESTIMATOR



 §.§ WER Estimation

Figure <ref> depicts ASR system-independent WER estimation. The first step is generating training data for WER estimation. Spoken utterances can be transcribed by an ASR system for baseline models or a simulator for proposed models. A training instance for WER estimation consists of a speech utterance, a hypothesis and a WER between the hypothesis and the reference transcript. In previous studies <cit.>, the hypotheses are produced from an ASR system, which, by implication, makes the WER estimation models dependent on the ASR system. This work proposes to derive hypotheses from reference transcripts via data generation. At training, the WER estimator is usually optimised as a regression model through the MSE loss function. 



 §.§ Fe-WER



  §.§.§ Model Architecture

The WER estimator is built upon the method suggested by <cit.>. The model is based on a two-tower architecture <cit.>, where one tower is a speech representation model and the other is a text representation model as described in Figure <ref>. 


The tower models extract frame-level features from a spoken utterance and token-level features from a transcript. The features are averaged over frames or tokens. The aggregated features for speech and text are concatenated as an input to the multi-layer perceptrons (MLPs). Then, the MLPs output a WER estimate. 



  §.§.§ Training Objective

The model is trained to minimise mean squared error (MSE). 

    MSE = 1/N∑_i=1^N (WER - WER)^2

where N is the number of instances, WER is an actual WER and WER is an estimate. 



  §.§.§ Evaluation Metrics

RMSE is used as an evaluation metric for WER estimation as well as PCC. These metrics have been adopted for WER estimation in the recent studies <cit.>. RMSE measures the average difference between estimates and actual WERs, while PCC measures the relationship between them. PCC is from -1 to 1 and if it is 1, then the estimate tends to increase when the target increases, while it is 0 if there is no relationship between them. 




 §.§ Hypothesis Generation Strategy

There are three main strategies for hypothesis generation: random selection, phonetic similarity, linguistic probability. These approaches generate the errors of an ASR, an acoustic and a language model, respectively. 

Random Selection Strategy Positions for insertions, deletions and substitutions are selected randomly, aiming for a specific target WER. 

Phonetic Similarity Strategy An ASR model produces ASR errors between phonetically similar words, such as grief and brief. Thus, the phonetic similarity between the reference word and the other words can be considered when the substitution is generated. First of all, a word in a reference transcript is converted into a phoneme sequence, e.g. speech to S P IY CH. Then, the edit distance between two words is calculated. After calculating all the distances to the other words in a vocabulary list, the top n words similar to the reference word are listed. When a word is substituted with another word, the word for substitution is selected from the phonetic similar word list. 

Linguistic Probability Strategy In the case of insertion, it can be caused by grammatical corrections. Here is an example of the insertion related to the context rather than the acoustic similarity: [I do not] know if he is alive. The words in [] are inserted by an ASR model with a language model, which is used to get the probability of the word in context.

The relation between Hypothesis generation strategies and edit types is described in Table <ref>.






 §.§ Hypothesis Generation Method




  §.§.§ Random Sampling of Transcripts

The first approach to hypothesis generation is to draw samples randomly from the transcripts of the ASR corpus. In other words, the spoken utterances and transcripts are paired randomly. This method guarantees that the vocabulary list, the number of words and the distribution of words in the training, validation and evaluation dataset do not change during hypothesis generation. 



  §.§.§ Random Sampling of Words

Another random sampling can be at a word level. The words in a reference are replaced with the words randomly chosen according to the vocabulary distribution of the datasets. With this method, the total number of words and the distribution of words in the dataset can be maintained in addition to the length of a transcript. However, with the random sampling methods, the WER of the hypotheses can not be targeted. For example, hypotheses of 10% of WER. To address these issues, a method for controlling WER is introduced in the following section. 



  §.§.§ Edit Generation

In contrast to the random sampling methods, individual edits on transcripts can be generated to achieve a target WER for a dataset. WER is the ratio of the number of deletion, substitution and insertion errors in hypotheses to the number of words in references. As the reference does not change, the WER of a dataset can be controlled by the total number of errors.

To keep the number of edits as close to the target as possible, the tokens of deletion and substitution, [del] and [sub], respectively, replace the words to be deleted or substituted. Then, the [del] tokens are deleted from the reference and the [sub] tokens are replaced by other words. For substitution, a phonetic similarity matrix is built using edit distance between phoneme sequences of two words. Then, [sub] tokens could be substituted by phonetically similar ones, e.g., born replaced by borne. Among the words in the phonetic similarity matrix, a word for the substitution is randomly selected according to the similarity. 

The total number of each edit type could change whenever shorter paths are found as a result of the insertion. For example, a sequence of a deletion and an insertion converts into a substitution. To minimise the unexpected change in the number of individual edits, a token for insertion, [ins], is inserted between correct words after generating deletions and insertions. For the [ins] token, it is replaced by linguistically probable ones, e.g., it can be inserted into the end of because of. In a similar way to the substitution, the word is drawn from the most probable word list. The probability is obtained by the language model trained on the reference transcripts of a training dataset. 

The example of the hypothesis generation is described in Table <ref>.



  §.§.§ Data augmentation

Training data are augmented by merging hypothesis sets of different WERs. Each set is generated individually. When they are combined, some instances can be duplicated. To maintain the same amount of data generated, they are not removed. Therefore, if the same number of datasets are merged, the amount of training data will be the same.



§ EXPERIMENTAL SETUP



 §.§ ASR Corpora

Ted-Lium 3 (TL3) <cit.> was used as a corpus for training WER estimators. This corpus has been used for the WER estimation task in previous studies <cit.>. While the TL3 test dataset was used for in-domain evaluation, three evaluation datasets from different domains were used for the evaluation of the models on out-of-domain data. First, FCASC is an evaluation dataset for AMI <cit.>. AMI is a multiparty meeting corpus recorded in business meetings of three or four participants. These participants played the roles of employees in a business situation with or without scenarios. Second, 2000 HUB5 English Evaluation Speech [https://catalog.ldc.upenn.edu/LDC2002S09] is an English conversational telephone speech dataset. It consists of 20 unreleased conversations from the Switchboard study <cit.> as well as 20 conversations from CALLHOME American English Speech [https://catalog.ldc.upenn.edu/LDC97S42]. The Switchboard conversations are between two people on daily topics. The CALLHOME conversations are between family members or close friends. Finally, Wall Street Journal (WSJ) <cit.> is a read speech corpus based on mainly WSJ materials. One of the features of WSJ is the transcripts with verbalised punctuation, e.g., COMMA. For WSJ, there are multiple evaluation datasets with different vocabulary sizes: eval92 5k, eval92 20k, eval93 5k, eval93 20k. For the following experiment, the four evaluation datasets for WSJ were merged into one dataset. The evaluation datasets from different domains were named as AMI eval, SWB/CH eval and WSJ eval. 



 §.§ ASR Systems


Training data for WER estimation were generated by transcribing the TL3 train set through different ASR systems in Section <ref>: Whisper <cit.>, wav2vec 2.0 <cit.>, Chain <cit.>, Conformer <cit.>, Transducer <cit.>. These models and their weights were downloaded from online resources[https://github.com/openai/whisper][https://github.com/facebookresearch/fairseq][https://github.com/kaldi-asr/kaldi][https://huggingface.co/speechbrain][https://github.com/speechbrain/speechbrain] except Chain. The Chain model was trained on 100 hours of LibriSpeech <cit.> using LF-MMI <cit.> and its augmented versions by changing speed and volume. The details of each model size and their training data are described in Table <ref>.







 §.§ Training Datasets
 

The training datasets for WER estimation were generated in two ways as described in Figure <ref>: transcribing the TL3 train dataset with an ASR model and simulating the ASR output. First, the training dataset was transcribed using the ASR models in Section <ref>. To prevent imbalanced data from the low WER, the number of instances whose WER was 0 was limited to the sum of the second and the third common WER, e.g. the total number of instances in the range of WER of 3% and 4%. 

Second, the hypotheses were generated by hypothesis generation methods described in Section <ref> and Table <ref>. For GEN1, a spoken utterance and a transcript were paired randomly and uniformly. For GEN2, each word was replaced with another word randomly sampled from the word distribution of the TL3 train dataset. GEN3 was a method to generate hypotheses aiming for target WERs of 2%, 4%...100%. From GEN4 to GEN7, phonetic similarity was adopted, while the linguistic probability was employed by GEN8. When the target WER is given, the target number for each edit is distributed almost equally. For example, the WER of insertions, deletions and substitutions generated using GEN7 were 3.06%, 3.33% and 3.29% when the target WER was 10%. For substitution, the length of the phonetically similar word list was different from 10, 30, 50 and 100: GEN4 to GEN7, respectively. Moreover, for linguistic probability, a 3-gram language model was trained on the TL3 train with the SRI Language Modeling toolkit[http://www.speech.sri.com/projects/srilm/]. Additionally, several datasets were merged for data augmentation. For example, the generated hypotheses of 10%, 20%...100% WER were merged into the dataset called GEN7W10-100 when the hypothesis generation method is GEN7. The datasets generated by ASR systems and hypothesis generations are organised in Table <ref>. 




 §.§ Evaluation Datasets

The evaluation datasets were generated by the ASR systems in Table <ref>. The instances of 0 WER were filtered out as described in Section <ref>. As a result, the total duration of audio could be reduced differently according to an ASR system. The WER estimation models were also evaluated on the out-of-domain test sets, AMI eval, SWB eval and WSJ eval. Each test set was transcribed by ASR1–5. WER of the evaluation datasets are summarised in Table <ref>. 









 §.§ WER Estimator

MLPs are employed to predict WER. The layer sizes of the model were [2048, 600, 32, 1]. The outputs of hidden layers are dropped out at a rate of 0.1. The Adam optimiser is used with a learning rate of 0.007. The cosine scheduler with 15 iterations is used as an annealing factor of the learning rate. For feature extraction, HuBERT large <cit.> and XLM-R large <cit.> are adopted. The averaged representations are 1024-dimensional features. 



§ RESULTS



 §.§ Evaluation on In-Domain Datasets


The performance of the WER estimators trained on the ASR hypotheses is summarised in Table <ref>. The performance of the estimators was best when the evaluation dataset was generated with the same ASR used to transcribe the training dataset. For example, when the estimator was trained on HYP1 train (TL3), then the performance of the estimator was the best on HYP1 eval (TL3).

For ASR system-independent WER estimation, WER estimators were trained on the data augmented with the datasets in Table <ref>. SIWE1 and SIWE2 are the estimation models trained on the GEN1 and GEN2 train, respectively. These random sampling methods for data generation showed relatively poor results. However, the RMSE and PCC of SIWE3–8 are comparable to those of ASR system-dependent WER estimators, Fe-WER1–5. SIWE8 outperformed the other SIWE models on most evaluation datasets.

For comparison between Fe-WERs and SIWEs, the RMSE and PCC values are averaged over HYP1–5 eval without a duration weight, the mean RMSE and PCC of the SIWE8 model were better than Fe-WER1 by 0.0056 and 0.0289, respectively.
The results are shown in Table <ref>. 



 §.§ Evaluation on Out-of-Domain Datasets

The estimators were evaluated on out-of-domain datasets in Table <ref>. For simplicity, the RMSE and PCC values are averaged over HYP1–5 eval without a duration weight. The mean RMSE and PCC are shown in Table <ref>. In terms of the means of RMSE and PCC, the performance of SIWE7 and SIWE8 on AMI and SWB/CH eval are better than those of all Fe-WERs, while the results on WSJ eval are mixed. On WSJ eval, the RMSE and the PCC mean values of all the SIWE models are better than those of Fe-WER1, while they are worse than those of Fe-WER4. 







§ DISCUSSION



 §.§ Data Generation Method

In this section, we will discuss the effect of each generation method for WER estimation on in-domain and out-of-domain data in detail. First, the edit generation method, GEN3, improved the performance of the estimators significantly compared to GEN1 and GEN2. In Table <ref>, the mean RMSE of SIWE3 was 0.1542, while those of SIWE1 and SIWE2 were 0.8730 and 0.8797, respectively. The amount of the training data would be helpful to improve the performance of the estimator because the training dataset for SIWE3 was the merged dataset of GEN3W10–100. Moreover, all the PCC values of the estimators trained on GEN3–7 were over 0.5, while those of SIWE1 and SIWE2 were below 0.3. Second, using phonetic similarity for substitution also brought an additional gain in the performance of the estimators. Additionally, as the size of the phonetically similar word list increased from 10 to 100, RSME and PCC of SIWE improved. Furthermore, when the linguistic probability was added to the generation method, GEN8, the performance was notably improved on all evaluation datasets. For example, the mean RMSE and PCC of the SIWE8 estimator on HYP1–5 eval were 0.1298 and 0.6860, while those of the SIWE7 estimator were 0.1400 and 0.6434, respectively. 

However, the performance improvement by GEN8 could not be achieved on out-of-domain data. The results in Table <ref> show that all the results of SIWE8 are behind the SIWE3–7 on AMI and SWB/CH evals. These results mean that the linguistic information obtained from the TL3 corpus might not be helpful for WER estimation on different corpora. 



 §.§ Data Augmentation

The performance of SIWEs was behind that of Fe-WERs on out-of-domain data except for WSJ eval. For further analysis, SIWEs were trained on the datasets consisting of different ranges of target WER. For example, training data of WER of 2%, 4%...20%. The amount of data does not change from the datasets used in Table <ref> as the same number of datasets were merged. The performance improvement was observed when the WER range of the training dataset was close to the WER of evaluation datasets. For example, SIWE7W2–20 performed better than the others in terms of both mean RMSE and PCC when the mean WER of HYP1–5 eval (WSJ) was 0.1110. Similarly, SIWE7W42–60 outperformed the others on SWB/CH eval whose mean of WER on HYP1–5 eval (SWB/CH) was 0.4367. Moreover, those results in Table <ref> are better than those of SIWE7 in Table <ref>. 



§ CONCLUSION

Hypothesis generation and data augmentation for ASR system-independent WER estimation are proposed in this paper. The estimator trained on the hypotheses generated by this proposed method outperforms the baselines on out-of-domain datasets. As a hypothesis generation strategy, linguistic information helps improve the performance of WER estimators when they are evaluated on in-domain datasets. The phonetic similarity for substitution improved the performance of the estimators on both in-domain and out-of-domain datasets. On out-of-domain datasets, the performance of estimators has further improved marginally when the hypotheses are generated with a target WER close to the evaluation dataset. 



§ ACKNOWLEDGEMENTS



This work was conducted at the Voicebase/Liveperson Centre of Speech and Language Technology at the University of Sheffield which is supported by Liveperson, Inc..




§ BIBLIOGRAPHICAL REFERENCES



lrec-coling2024-natbib



