Review on automatic speech recognition quality estimation (ASR QE).The application of ASR QE in ASR system combination for both single-microphone multiple-ASR system task and on multiple-microphone multiple-ASR system task.Ranking the system combination inputs based on predicted quality.Management of tied ranks.Automatically finding the optimum level of combination for each segment. Recognizer Output Voting Error Reduction (ROVER) has been widely used for system combination in automatic speech recognition (ASR). In order to select the most appropriate words to insert at each position in the output transcriptions, some ROVER extensions rely on critical information such as confidence scores and other ASR decoder features. This information, which is not always available, highly depends on the decoding process and sometimes tends to overestimate the real quality of the recognized words. In this paper we propose a novel variant of ROVER that takes advantage of ASR quality estimation (QE) for ranking the transcriptions at segment level instead of: i) relying on confidence scores, or ii) feeding ROVER with randomly ordered hypotheses. We first introduce an effective set of features to compensate for the absence of ASR decoder information. Then, we apply QE techniques to perform accurate hypothesis ranking at segment-level before starting the fusion process. The evaluation is carried out on two different tasks, in which we respectively combine hypotheses coming from independent ASR systems and multi-microphone recordings. In both tasks, it is assumed that the ASR decoder information is not available. The proposed approach significantly outperforms standard ROVER and it is competitive with two strong oracles that exploit prior knowledge about the real quality of the hypotheses to be combined. Compared to standard ROVER, the absolute WER improvements in the two evaluation scenarios range from 0.5 to 7.3 .
@cite_42: We address the problem of estimating the quality of Automatic Speech Recognition (ASR) output at utterance level, without recourse to manual reference transcriptions and when information about system’s confidence is not accessible. Given a source signal and its automatic transcription, we approach this problem as a regression task where the word error rate of the transcribed utterance has to be predicted. To this aim, we explore the contribution of different feature sets and the potential of different algorithms in testing conditions of increasing complexity. Results show that our automatic quality estimates closely approximate the word error rate scores calculated over reference transcripts, outperforming a strong baseline in all the testing conditions.
In the ASR field, confidence estimation strongly relies on knowledge about the inner workings of the decoder that produces the transcriptions. Such knowledge includes acoustic and language model scores and word timing information associated to arcs of word lattices. QE, instead, approaches the problem as a system-independent task, in which this information is not necessarily accessible. In the first work along this direction @cite_42 , we explored ASR QE as a supervised regression problem in which the WER of an utterance transcription has to be automatically predicted. Given a set of (, , ) triplets as training instances, different algorithms were evaluated on test data consisting of unseen (, ) pairs. All models used an effective set of signal textual hybrid features, named black-box features in contrast with the glass-box ones that model the internal behavior of the decoder. Results showed that ASR QE predictions can closely approximate the true'' WER scores calculated over reference transcripts, outperforming a strong baseline in a variety of test conditions of increasing complexity. Interestingly, when combined with the glass-box features ( confidence scores), the predictions are better than those obtained by using confidence scores alone.