pytorch tf wav2vec2 automatic-speech-recognition transformers generated_from_keras_callback model-index
---
tags:
- generated_from_keras_callback
model-index:
- name: wav2vec2-xls-r-300m-mixed
  results: []
---

<!-- This model card has been generated automatically according to the information Keras had access to. You should
probably proofread and complete it, then remove this comment. -->

# wav2vec2-xls-r-300m-mixed

Finetuned https://huggingface.co/facebook/wav2vec2-xls-r-300m on https://github.com/huseinzol05/malaya-speech/tree/master/data/mixed-stt

**Update 2022-07-02, https://huggingface.co/mesolitica/wav2vec2-xls-r-300m-mixed slightly better accuracy**.

This model was finetuned on 3 languages,

1. Malay
2. Singlish
3. Mandarin

**This model trained on a single Tesla V100 32GB VRAM, provided by https://keyreply.com/**. 

## Evaluation set

Evaluation set from https://github.com/huseinzol05/malaya-speech/tree/master/pretrained-model/prepare-stt with sizes,

```
len(malay), len(singlish), len(mandarin)
-> (765, 3579, 614)
```

It achieves the following results on the evaluation set based on [evaluate-wav2vec2-xls-r-300m-mixed.ipynb](evaluate-wav2vec2-xls-r-300m-mixed.ipynb):

Mixed evaluation,

```
CER: 0.048555454439612775
WER: 0.14151468058308714
CER with LM: 0.03977501945111893
WER with LM: 0.09809135311921899
```

Malay evaluation,

```
CER: 0.05372605571018908
WER: 0.23714922876687583
CER with LM: 0.03508559320616622
WER with LM: 0.1294898148329521
```

Singlish evaluation,

```
CER: 0.0488366183589853
WER: 0.1294114484378467
CER with LM: 0.04119293317615
WER with LM: 0.09411106530063
```

Mandarin evaluation,

```
CER: 0.04047435404966954
WER: 0.09291050873816364
CER with LM: 0.037352703254831865
WER with LM: 0.08217217867571727
```

Language model from https://huggingface.co/huseinzol05/language-model-bahasa-manglish-combined