How to evaluate the quality of speech-to-text data without access to the true labels? I am dealing with a data set of transcribed call center data, where customers are being recorded when interacting with the agent. This is then automatically transcribed by an external transcription system. I want to automatically assess the quality of these transcriptions.
Sadly, the quality seems to be disastrous. In some cases it's little more than gibberish, often due to different dialects the machine is not able to handle. We have no access to the original recordings (data privacy), so there is no way whatsoever to get or create the true labels. The system cannot be replaced as we are committed to it.
Again to the question: is there any way to automatically assess the quality of transcriptions with NLP methods? We want to quantify and compare transcription quality to filter out the best samples for semantic inference of our customers' input in a downstream task. I am thinking about something like a coherence measure in order to find the sentences which make the most sense, grammatically or semantically. Sadly, things as BLEU, WER or Rouge do not work in this case.
I'd be grateful for anything pointing in the right direction. Most importantly again, we have no labels and it needs to be scalable.
Thanks a lot!
There is at least one way:

Create/Acquire a grammar model for the language spoken (there are several such models for various languages used in NLP)
Test the transcripts for beign grammaticaly/syntacticaly correct.
This assesment will at least rule out gibberish and most of transcripts that do not correspond to valid sentences of the language spoken