Spaces:
Runtime error
Runtime error
update custom message
Browse files- constants.py +4 -4
constants.py
CHANGED
@@ -97,20 +97,20 @@ are ranked based on their average WER scores, from lowest to highest.
|
|
97 |
For more details on the individual datasets and how models are evaluated to give the ESB score, refer to the [ESB paper](https://arxiv.org/abs/2210.13352).
|
98 |
"""
|
99 |
|
100 |
-
CUSTOM_MESSAGE = """##
|
101 |
This space is a fork of the original [hf-audio/open_asr_leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard). It aims to demonstrate how the CommonVoice Test Set provides a relatively accurate approximation of the average WER/CER (Word Error Rate/Character Error Rate) at a significantly lower computational cost.
|
102 |
|
103 |
#### Why is this useful?
|
104 |
This opens way the to achieve standardized test set for most languages, enabling us to programmatically select a reasonably effective model for any language supported by CommonVoice.
|
105 |
|
|
|
|
|
106 |
Columns `Model`, `RTF`, and `Average WER` were sourced from [hf-audio/open_asr_leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard) using the version from September 7, 2023.
|
107 |
|
108 |
-
Models are sorted by consistancy in
|
109 |
|
110 |
### Results
|
111 |
The CommonVoice Test provides a Word Error Rate (WER) within a 20-point margin of the average WER.
|
112 |
|
113 |
While not perfect, this indicates that CommonVoice can be a useful tool for quickly identifying a suitable ASR model for a wide range of languages in a programmatic manner. However, it's important to note that it is not sufficient as the sole criterion for choosing the most appropriate architecture. Further considerations may be needed depending on the specific requirements of your ASR application.
|
114 |
-
|
115 |
-
For more context, [here](https://gist.github.com/wasertech/400ca3dd61f2d6f7f4f5495afbb32ef3) is the output of my ASR server when running without any specified model to load for various languages. It tries to score the most suitable model for any given language. Since metrics are mostly self-reported, sometimes in different format, it consistently picks an unadequate model.
|
116 |
"""
|
|
|
97 |
For more details on the individual datasets and how models are evaluated to give the ESB score, refer to the [ESB paper](https://arxiv.org/abs/2210.13352).
|
98 |
"""
|
99 |
|
100 |
+
CUSTOM_MESSAGE = """## Using CommonVoice to approximate average WER for open domain transcriptions
|
101 |
This space is a fork of the original [hf-audio/open_asr_leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard). It aims to demonstrate how the CommonVoice Test Set provides a relatively accurate approximation of the average WER/CER (Word Error Rate/Character Error Rate) at a significantly lower computational cost.
|
102 |
|
103 |
#### Why is this useful?
|
104 |
This opens way the to achieve standardized test set for most languages, enabling us to programmatically select a reasonably effective model for any language supported by CommonVoice.
|
105 |
|
106 |
+
For more context, [here](https://gist.github.com/wasertech/400ca3dd61f2d6f7f4f5495afbb32ef3) is the output of my ASR server when running without any specified model to load for various languages. It tries to score the most suitable model for any given language. Since metrics are mostly self-reported, sometimes in different format, it consistently picks an unadequate model.
|
107 |
+
|
108 |
Columns `Model`, `RTF`, and `Average WER` were sourced from [hf-audio/open_asr_leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard) using the version from September 7, 2023.
|
109 |
|
110 |
+
Models are sorted by consistancy in their results across testsets. (by increasing order of absolute delta between average WER and CommonVoice WER)
|
111 |
|
112 |
### Results
|
113 |
The CommonVoice Test provides a Word Error Rate (WER) within a 20-point margin of the average WER.
|
114 |
|
115 |
While not perfect, this indicates that CommonVoice can be a useful tool for quickly identifying a suitable ASR model for a wide range of languages in a programmatic manner. However, it's important to note that it is not sufficient as the sole criterion for choosing the most appropriate architecture. Further considerations may be needed depending on the specific requirements of your ASR application.
|
|
|
|
|
116 |
"""
|