How is an ASR's output compared to ground truth for validation?
The answer I was looking for is Word Error Rate. It is the most standard way of comparing ASR transcriptions wrt ground truth. It is less granular than what I had in mind, it is basically Levenshtein Distance on a word level rather than a character level.
jiwer in python has a few other metrics and is easy to use.