Merge branch 'main' of https://huggingface.co/Nhut/wav2vec2-large-xlsr-vietnamese

README.md

@@ -2,7 +2,8 @@
 language: vi
 datasets:
 - common_voice
-- TODO: https://data.mendeley.com/datasets/k9sxg2twv4/4
+- FOSD: https://data.mendeley.com/datasets/k9sxg2twv4/4
+- VIVOS: https://ailab.hcmus.edu.vn/vivos
 metrics:
 - wer
 tags:
@@ -24,24 +25,18 @@ model-index:
     metrics:
        - name: Test WER
          type: wer
-         value: 54.55
+         value: 52.48
 ---
-
 # Wav2Vec2-Large-XLSR-53-Vietnamese
-
-Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on Vietnamese using the [Common Voice](https://huggingface.co/datasets/common_voice), and [FOSD](https://data.mendeley.com/datasets/k9sxg2twv4/4).
+Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) on Vietnamese using the [Common Voice](https://huggingface.co/datasets/common_voice), [FOSD](https://data.mendeley.com/datasets/k9sxg2twv4/4) and [VIVOS](https://ailab.hcmus.edu.vn/vivos).
 When using this model, make sure that your speech input is sampled at 16kHz.
-
 ## Usage
-
 The model can be used directly (without a language model) as follows:
-
 ```python
 import torch
 import torchaudio
 from datasets import load_dataset
 from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
-
 ENCODER = {
     "ia ": "iê ",
     "ìa ": "iề ",
@@ -149,39 +144,26 @@ def decode_string(x):
   for k, v in list(reversed(list(ENCODER.items()))):
     x = x.replace(v, k)
   return x
-
-
 test_dataset = load_dataset("common_voice", "vi", split="test[:2%]") 
-
 processor = Wav2Vec2Processor.from_pretrained("Nhut/wav2vec2-large-xlsr-vietnamese")
 model = Wav2Vec2ForCTC.from_pretrained("Nhut/wav2vec2-large-xlsr-vietnamese")
 resampler = torchaudio.transforms.Resample(48_000, 16_000)
-
 # Preprocessing the datasets.
 # We need to read the aduio files as arrays
 def speech_file_to_array_fn(batch):
   speech_array, sampling_rate = torchaudio.load(batch["path"])
   batch["speech"] = resampler(speech_array).squeeze().numpy()
   return batch
-
 test_dataset = test_dataset.map(speech_file_to_array_fn)
 inputs = processor(test_dataset["speech"][:2], sampling_rate=16_000, return_tensors="pt", padding=True)
-
 with torch.no_grad():
   logits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits
-
 predicted_ids = torch.argmax(logits, dim=-1)
-
-print("Prediction:", decode_string(processor.batch_decode(predicted_ids)))
+print("Prediction:", [decode_string(x) for x in processor.batch_decode(predicted_ids)])
 print("Reference:", test_dataset["sentence"][:2])
 ```
-
-
 ## Evaluation
-
 The model can be evaluated as follows on the Vietnamese test data of Common Voice.
-
-
 ```python
 import torch
 import torchaudio
@@ -291,31 +273,36 @@ ENCODER = {
     "ưỡ": "\xa6",
     "ượ": "\xa7",
 }
-  
+
 def decode_string(x):
   for k, v in list(reversed(list(ENCODER.items()))):
     x = x.replace(v, k)
   return x
 
-
 test_dataset = load_dataset("common_voice", "vi", split="test")
 wer = load_metric("wer")
-
-processor = Wav2Vec2Processor.from_pretrained(MODEL)
-model = Wav2Vec2ForCTC.from_pretrained(MODEL)
+processor = Wav2Vec2Processor.from_pretrained("Nhut/wav2vec2-large-xlsr-vietnamese")
+model = Wav2Vec2ForCTC.from_pretrained("Nhut/wav2vec2-large-xlsr-vietnamese")
 model.to("cuda")
 
 chars_to_ignore_regex = '[\\\+\@\ǀ\,\?\.\!\-\;\:\"\“\%\‘\”\�]'
 resampler = torchaudio.transforms.Resample(48_000, 16_000)
 
+# Preprocessing the datasets.
+# We need to read the aduio files as arrays
+def speech_file_to_array_fn(batch):
+  batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
+  speech_array, sampling_rate = torchaudio.load(batch["path"])
+  batch["speech"] = resampler(speech_array).squeeze().numpy()
+  return batch
+
+test_dataset = test_dataset.map(speech_file_to_array_fn)
 # Preprocessing the datasets.
 # We need to read the aduio files as arrays
 def evaluate(batch):
   inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)
-
   with torch.no_grad():
     logits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits
-
   pred_ids = torch.argmax(logits, dim=-1)
   batch["pred_strings"] = processor.batch_decode(pred_ids)
   # decode_string: We replace the encoded letter with the initial letters
@@ -325,12 +312,7 @@ def evaluate(batch):
 result = test_dataset.map(evaluate, batched=True, batch_size=8)
 print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))
 ```
-
-**Test Result**: 54.55 % 
-
-
+**Test Result**: 52.48 % 
 ## Training
-
-The Common Voice `train`, `validation`, and FOSD datasets were used for training as well.
-
+The Common Voice `train`, `validation` and FOSD datasets and VIVOS datasets were used for training as well.
 The script used for training can be found [here](https://colab.research.google.com/drive/11pP4uVJj4SYZTzGjlCUtOHywlhYqs0cPx)