|
|
--- |
|
|
language: |
|
|
- ms |
|
|
- en |
|
|
- zh |
|
|
- ta |
|
|
datasets: |
|
|
- mesolitica/Malaysian-STT-Whisper |
|
|
- malaysia-ai/STT-Whisper |
|
|
base_model: |
|
|
- openai/whisper-large-v3-turbo |
|
|
--- |
|
|
|
|
|
# Malaysian Finetune Whisper Large V3 Turbo |
|
|
|
|
|
Finetune Whisper Large V3 Turbo on Malaysian context. |
|
|
|
|
|
## Improvement |
|
|
|
|
|
1. Distilled from Whisper Large V3 on Malaysian and Science context. |
|
|
2. Better translation for Malay, Manglish, Mandarin, Tamil and Science context. |
|
|
3. Word level timestamp, introduced `<|transcribeprecise|>` token, **a new task!** |
|
|
|
|
|
## how we finetuned it? |
|
|
|
|
|
We done 2 phases, |
|
|
|
|
|
1. Finetune on [mesolitica/Malaysian-STT-Whisper](https://huggingface.co/datasets/mesolitica/Malaysian-STT-Whisper) |
|
|
- Revision [267552e0f093068519a816112c2741939d057f48](https://huggingface.co/mesolitica/malaysian-whisper-large-v3-turbo-v3/commit/267552e0f093068519a816112c2741939d057f48) |
|
|
- WanDB at https://wandb.ai/huseinzol05/malaysian-whisper-large-v3-turbo-v3 |
|
|
2. Stage 2, Annealing on 5% from [mesolitica/Malaysian-STT-Whisper](https://huggingface.co/datasets/mesolitica/Malaysian-STT-Whisper) and 100% from [mesolitica/Malaysian-STT-Whisper-Stage2](https://huggingface.co/datasets/mesolitica/Malaysian-STT-Whisper-Stage2) |
|
|
- Revision [5f6ca0596f01527ec2d013662cfc168d5f754461](https://huggingface.co/mesolitica/malaysian-whisper-large-v3-turbo-v3/commit/5f6ca0596f01527ec2d013662cfc168d5f754461) |
|
|
- WanDB at https://wandb.ai/huseinzol05/malaysian-whisper-large-v3-turbo-v3-stage2 |
|
|
|
|
|
## Benchmark |
|
|
|
|
|
 |
|
|
|
|
|
## Example |
|
|
|
|
|
We use for examples, |
|
|
|
|
|
1. [assembly.mp3](https://github.com/mesolitica/malaya-speech/blob/master/speech/assembly.mp3) |
|
|
|
|
|
2. [toodia.mp3](https://github.com/mesolitica/malaya-speech/blob/master/speech/podcast/toodia.mp3) |
|
|
|
|
|
### Whisper Large v3 |
|
|
|
|
|
```python |
|
|
from transformers import ( |
|
|
WhisperFeatureExtractor, |
|
|
WhisperForConditionalGeneration, |
|
|
WhisperProcessor, |
|
|
WhisperTokenizerFast |
|
|
) |
|
|
from datasets import Audio |
|
|
import torch |
|
|
|
|
|
def get_audio(f): |
|
|
return audio.decode_example(audio.encode_example(f))['array'] |
|
|
|
|
|
sr = 16000 |
|
|
audio = Audio(sampling_rate=sr) |
|
|
feature_extractor = WhisperFeatureExtractor.from_pretrained( |
|
|
'openai/whisper-large-v3' |
|
|
) |
|
|
processor = WhisperProcessor.from_pretrained( |
|
|
'openai/whisper-large-v3' |
|
|
) |
|
|
tokenizer = WhisperTokenizerFast.from_pretrained( |
|
|
'openai/whisper-large-v3' |
|
|
) |
|
|
model = WhisperForConditionalGeneration.from_pretrained( |
|
|
'openai/whisper-large-v3', |
|
|
torch_dtype = torch.bfloat16, |
|
|
).cuda() |
|
|
|
|
|
assembly = get_audio('assembly.mp3') |
|
|
assembly = assembly[: 16000 * 30] |
|
|
toodia = get_audio('toodia.mp3') |
|
|
toodia = toodia[: 16000 * 30] |
|
|
|
|
|
with torch.no_grad(): |
|
|
p = processor([assembly], return_tensors='pt') |
|
|
p['input_features'] = p['input_features'].to(torch.bfloat16) |
|
|
r = model.generate( |
|
|
p['input_features'].cuda(), |
|
|
output_scores=True, |
|
|
return_dict_in_generate=True, |
|
|
return_timestamps=True, |
|
|
task = 'transcribe', |
|
|
) |
|
|
|
|
|
tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(r['sequences'][0])) |
|
|
``` |
|
|
|
|
|
Output, |
|
|
|
|
|
``` |
|
|
<|startoftranscript|><|ms|><|transcribe|> |
|
|
<|0.00|> Assembly on Aging di Vienna, Australia yang telah diadakan pada tahun 1982<|6.42|> |
|
|
<|6.42|> dan berasaskan unjuran tersebut maka Jabatan Perangkaan Malaysia menganggarkan<|11.58|> |
|
|
<|11.58|> menjelang tahun 2035 sejumlah 15% penduduk kita adalah daripada kalangan warga emas.<|18.70|> |
|
|
<|18.70|> Untuk makluman Tuan Yang Putua dan juga Aliam Bohmat, pembangunan sistem pendaftaran warga emas<|24.02|> |
|
|
<|24.02|> ataupun kita sebutkan IWEN adalah usaha kerajaan ke arah merealisasikan objektif<|28.70|> |
|
|
<|29.40|><|endoftext|> |
|
|
``` |
|
|
|
|
|
```python |
|
|
with torch.no_grad(): |
|
|
p = processor([toodia], return_tensors='pt') |
|
|
p['input_features'] = p['input_features'].to(torch.bfloat16) |
|
|
r = model.generate( |
|
|
p['input_features'].cuda(), |
|
|
output_scores=True, |
|
|
return_dict_in_generate=True, |
|
|
return_timestamps=True, |
|
|
task = 'transcribe', |
|
|
) |
|
|
|
|
|
tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(r['sequences'][0])) |
|
|
``` |
|
|
|
|
|
Output, |
|
|
|
|
|
``` |
|
|
<|startoftranscript|><|ms|><|transcribe|> |
|
|
<|0.00|> Kerana dia tak ada keseimbangan dalam hidup.<|4.00|> |
|
|
<|4.00|> Jika anda mencari keseimbangan dalam hidup,<|6.00|> |
|
|
<|6.00|> anda akan menemukan dupe.<|7.00|> |
|
|
<|7.00|> Tak ada?<|8.00|> |
|
|
<|8.00|> Tak ada.<|9.00|> |
|
|
<|9.00|> Tapi jika anda menikmati, anda akan baik-baik saja.<|13.00|> |
|
|
<|15.00|> Seperti semua rahsia yang saya buat,<|17.00|> |
|
|
<|17.00|> pada masa yang sama, saya menambahkan.<|19.00|> |
|
|
<|19.00|> Pada masa yang sama, saya menambahkan.<|20.00|> |
|
|
<|20.00|> Oh, jadi seperti kelabu-kelabu.<|22.00|> |
|
|
<|22.00|> Lepas itu, saya menambahkan rahsia di Langkawi.<|25.00|> |
|
|
<|25.00|> Selepas itu, kita mengambil pesawat.<|26.00|> |
|
|
<|26.00|> Selepas itu, kita pergi dan mengambil pesawat ke KL.<|28.00|> |
|
|
<|28.00|><|endoftext|> |
|
|
``` |
|
|
|
|
|
### Whisper Large v3 Turbo |
|
|
|
|
|
```python |
|
|
from transformers import ( |
|
|
WhisperFeatureExtractor, |
|
|
WhisperForConditionalGeneration, |
|
|
WhisperProcessor, |
|
|
WhisperTokenizerFast |
|
|
) |
|
|
from datasets import Audio |
|
|
import torch |
|
|
|
|
|
def get_audio(f): |
|
|
return audio.decode_example(audio.encode_example(f))['array'] |
|
|
|
|
|
sr = 16000 |
|
|
audio = Audio(sampling_rate=sr) |
|
|
feature_extractor = WhisperFeatureExtractor.from_pretrained( |
|
|
'openai/whisper-large-v3-turbo' |
|
|
) |
|
|
processor = WhisperProcessor.from_pretrained( |
|
|
'openai/whisper-large-v3-turbo' |
|
|
) |
|
|
tokenizer = WhisperTokenizerFast.from_pretrained( |
|
|
'openai/whisper-large-v3-turbo' |
|
|
) |
|
|
model = WhisperForConditionalGeneration.from_pretrained( |
|
|
'openai/whisper-large-v3-turbo', |
|
|
torch_dtype = torch.bfloat16, |
|
|
).cuda() |
|
|
|
|
|
assembly = get_audio('assembly.mp3') |
|
|
assembly = assembly[: 16000 * 30] |
|
|
toodia = get_audio('toodia.mp3') |
|
|
toodia = toodia[: 16000 * 30] |
|
|
|
|
|
with torch.no_grad(): |
|
|
p = processor([assembly], return_tensors='pt') |
|
|
p['input_features'] = p['input_features'].to(torch.bfloat16) |
|
|
r = model.generate( |
|
|
p['input_features'].cuda(), |
|
|
output_scores=True, |
|
|
return_dict_in_generate=True, |
|
|
return_timestamps=True, |
|
|
task = 'transcribe', |
|
|
) |
|
|
|
|
|
tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(r['sequences'][0])) |
|
|
``` |
|
|
|
|
|
Output, |
|
|
|
|
|
``` |
|
|
<|startoftranscript|><|ms|><|transcribe|> |
|
|
<|0.00|> Assembly on Aging di Vienna, Australia<|3.72|> |
|
|
<|3.72|> yang telah diadakan pada tahun 1982<|6.36|> |
|
|
<|6.36|> dan berasaskan unjuran tersebut<|8.80|> |
|
|
<|8.80|> maka Jabatan Perangkaan Malaysia<|10.38|> |
|
|
<|10.38|> menganggarkan menjelang tahun 2035<|13.68|> |
|
|
<|13.68|> sejumlah 15% penduduk kita adalah<|17.30|> |
|
|
<|17.30|> daripada kalangan warga emas.<|18.70|> |
|
|
<|19.26|> Untuk makluman Tuan Niri Putua dan juga Aliam Buhumat<|22.08|> |
|
|
<|22.08|> pembangunan sistem pendaftaran warga emas<|24.02|> |
|
|
<|24.02|> ataupun kita sebutkan EWEN<|25.36|> |
|
|
<|25.36|> adalah usaha kerajaan kearah merealisasikan<|28.38|> |
|
|
<|28.38|><|endoftext|> |
|
|
``` |
|
|
|
|
|
```python |
|
|
with torch.no_grad(): |
|
|
p = processor([toodia], return_tensors='pt') |
|
|
p['input_features'] = p['input_features'].to(torch.bfloat16) |
|
|
r = model.generate( |
|
|
p['input_features'].cuda(), |
|
|
output_scores=True, |
|
|
return_dict_in_generate=True, |
|
|
return_timestamps=True, |
|
|
task = 'transcribe', |
|
|
) |
|
|
|
|
|
tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(r['sequences'][0])) |
|
|
``` |
|
|
|
|
|
Output, |
|
|
|
|
|
``` |
|
|
<|startoftranscript|><|ms|><|transcribe|> |
|
|
<|0.00|> My line of work, dia takde work life balance.<|3.74|> |
|
|
<|4.36|> If you cari work life balance, dupe camp.<|7.16|> |
|
|
<|7.36|> Takde.<|7.70|> |
|
|
<|8.26|> Memang takde.<|8.94|> |
|
|
<|9.86|> But, you know, if you enjoy it, then you're okay with it.<|12.70|> |
|
|
<|15.40|> Macam lahsia semua aku buat pun, at the same time aku shoot.<|18.90|> |
|
|
<|19.16|> At the same time aku ada pun motion.<|20.16|> |
|
|
<|20.88|> Jadi macam kelang kabut.<|22.12|> |
|
|
<|22.24|> Lepas tu lah lahsia pula aku shoot dekat Langkawi.<|24.50|> |
|
|
<|25.16|> So, ni kita take flight.<|25.92|> |
|
|
<|26.12|> Lepas tu pergi, take flight, balik ke L, lepas tu balik ke L.<|28.00|> |
|
|
<|28.00|> Jadi macam tu pun nak.<|29.12|> |
|
|
<|29.28|> Kelang kabut.<|29.76|> |
|
|
<|endoftext|> |
|
|
``` |
|
|
|
|
|
### mesolitica/Malaysian-whisper-large-v3-turbo-v3 |
|
|
|
|
|
```python |
|
|
import torch |
|
|
from transformers.models.whisper import tokenization_whisper |
|
|
|
|
|
tokenization_whisper.TASK_IDS = ["translate", "transcribe", 'transcribeprecise'] |
|
|
|
|
|
from transformers import ( |
|
|
WhisperFeatureExtractor, |
|
|
WhisperForConditionalGeneration, |
|
|
WhisperProcessor, |
|
|
WhisperTokenizerFast |
|
|
) |
|
|
from datasets import Audio |
|
|
|
|
|
def get_audio(f): |
|
|
return audio.decode_example(audio.encode_example(f))['array'] |
|
|
|
|
|
sr = 16000 |
|
|
audio = Audio(sampling_rate=sr) |
|
|
feature_extractor = WhisperFeatureExtractor.from_pretrained( |
|
|
'mesolitica/Malaysian-whisper-large-v3-turbo-v3' |
|
|
) |
|
|
processor = WhisperProcessor.from_pretrained( |
|
|
'mesolitica/Malaysian-whisper-large-v3-turbo-v3' |
|
|
) |
|
|
tokenizer = WhisperTokenizerFast.from_pretrained( |
|
|
'mesolitica/Malaysian-whisper-large-v3-turbo-v3' |
|
|
) |
|
|
model = WhisperForConditionalGeneration.from_pretrained( |
|
|
'mesolitica/Malaysian-whisper-large-v3-turbo-v3', |
|
|
torch_dtype = torch.bfloat16, |
|
|
).cuda() |
|
|
assembly = get_audio('assembly.mp3') |
|
|
assembly = assembly[: 16000 * 30] |
|
|
|
|
|
toodia = get_audio('toodia.mp3') |
|
|
toodia = toodia[: 16000 * 30] |
|
|
|
|
|
with torch.no_grad(): |
|
|
p = processor([assembly], return_tensors='pt') |
|
|
p['input_features'] = p['input_features'].to(torch.bfloat16) |
|
|
r = model.generate( |
|
|
p['input_features'].cuda(), |
|
|
output_scores=True, |
|
|
return_dict_in_generate=True, |
|
|
return_timestamps=True, |
|
|
task = 'transcribeprecise', |
|
|
) |
|
|
|
|
|
tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(r['sequences'][0])) |
|
|
``` |
|
|
|
|
|
Output, |
|
|
|
|
|
``` |
|
|
<|startoftranscript|><|ms|><|transcribeprecise|> |
|
|
<|0.00|> Example<|1.20|> |
|
|
<|1.56|> on<|1.62|> |
|
|
<|1.74|> Aging<|2.04|> |
|
|
<|2.14|> di<|2.20|> |
|
|
<|2.26|> Vienna,<|2.50|> |
|
|
<|2.70|> Austria<|3.78|> |
|
|
<|4.28|> yang<|4.38|> |
|
|
<|4.42|> telah<|4.58|> |
|
|
<|4.62|> diadakan<|5.08|> |
|
|
<|5.16|> pada<|5.30|> |
|
|
<|5.36|> tahun<|5.60|> |
|
|
<|5.62|> 1982<|7.02|> |
|
|
<|7.12|> dan<|7.24|> |
|
|
<|7.32|> berasaskan<|7.86|> |
|
|
<|7.96|> unjuran<|8.36|> |
|
|
<|8.42|> tersebut,<|8.80|> |
|
|
<|8.88|> maka<|9.04|> |
|
|
<|9.12|> Jabatan<|9.48|> |
|
|
<|9.54|> Perangkaan<|9.98|> |
|
|
<|10.04|> Malaysia<|10.36|> |
|
|
<|10.84|> menganggarkan<|11.56|> |
|
|
<|11.96|> menjelang<|12.34|> |
|
|
<|12.40|> tahun<|12.64|> |
|
|
<|12.66|> 2035,<|14.02|> |
|
|
<|14.50|> sejumlah<|14.96|> |
|
|
<|14.98|> 15%<|15.92|> |
|
|
<|16.26|> penduduk<|16.60|> |
|
|
<|16.68|> kita<|16.88|> |
|
|
<|17.00|> adalah<|17.30|> |
|
|
<|17.40|> daripada<|17.80|> |
|
|
<|17.86|> kalangan<|18.16|> |
|
|
<|18.20|> warga<|18.38|> |
|
|
<|18.44|> emas.<|18.66|> |
|
|
<|19.24|> Untuk<|19.40|> |
|
|
<|19.46|> makluman<|19.86|> |
|
|
<|20.64|> Tuan<|20.76|> |
|
|
<|20.80|> dan<|20.88|> |
|
|
<|20.94|> Pertua<|21.14|> |
|
|
<|21.20|> dan<|21.28|> |
|
|
<|21.34|> juga<|21.50|> |
|
|
<|21.58|> Aliam<|21.74|> |
|
|
<|21.80|> Bohmat,<|22.08|> |
|
|
<|22.22|> pembangunan<|22.66|> |
|
|
<|22.72|> sistem<|23.00|> |
|
|
<|23.06|> pendaftaran<|23.48|> |
|
|
<|23.54|> warga<|23.72|> |
|
|
<|23.78|> emas<|23.98|> |
|
|
<|24.06|> ataupun<|24.36|> |
|
|
<|24.42|> kita<|24.56|> |
|
|
<|24.64|> sebutkan<|24.96|> |
|
|
<|25.08|> EWEN<|25.34|> |
|
|
<|25.86|> adalah<|26.08|> |
|
|
<|26.16|> usaha<|26.46|> |
|
|
<|26.60|> kerajaan<|27.04|> |
|
|
<|27.16|> ke<|27.20|> |
|
|
<|27.28|> arah<|27.44|> |
|
|
<|27.50|> merealisasikan<|28.36|> |
|
|
<|28.88|> objektif<|29.36|> |
|
|
<|29.42|> yang<|29.50|> |
|
|
<|29.56|> telah<|29.70|> |
|
|
<|29.76|> digariskan<|29.98|> |
|
|
<|endoftext|> |
|
|
``` |
|
|
|
|
|
```python |
|
|
with torch.no_grad(): |
|
|
p = processor([toodia], return_tensors='pt') |
|
|
p['input_features'] = p['input_features'].to(torch.bfloat16) |
|
|
r = model.generate( |
|
|
p['input_features'].cuda(), |
|
|
output_scores=True, |
|
|
return_dict_in_generate=True, |
|
|
return_timestamps=True, |
|
|
task = 'transcribeprecise', |
|
|
) |
|
|
|
|
|
tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(r['sequences'][0])) |
|
|
``` |
|
|
|
|
|
Output, |
|
|
|
|
|
``` |
|
|
<|startoftranscript|><|ms|><|transcribeprecise|> |
|
|
<|0.00|> Pada<|0.70|> |
|
|
<|0.70|> barisan<|1.00|> |
|
|
<|1.04|> kerja,<|1.24|> |
|
|
<|1.96|> dia<|2.08|> |
|
|
<|2.20|> takde<|2.48|> |
|
|
<|2.76|> keseimbangan<|2.90|> |
|
|
<|2.94|> kerja-hidup<|3.20|> |
|
|
<|3.28|> keseimbangan.<|3.72|> |
|
|
<|4.68|> Jika<|4.78|> |
|
|
<|4.84|> anda<|4.92|> |
|
|
<|4.96|> cari<|5.20|> |
|
|
<|5.30|> keseimbangan<|5.40|> |
|
|
<|5.44|> kerja-hidup<|5.66|> |
|
|
<|5.72|> keseimbangan,<|6.20|> |
|
|
<|6.52|> dupe<|6.72|> |
|
|
<|6.80|> kem.<|7.08|> |
|
|
<|7.36|> Takde.<|7.64|> |
|
|
<|7.92|> Takde.<|8.10|> |
|
|
<|8.52|> Memang<|8.74|> |
|
|
<|8.78|> takde.<|8.88|> |
|
|
<|9.80|> Takde.<|10.02|> |
|
|
<|10.16|> Tetapi,<|10.34|> |
|
|
<|10.68|> anda<|10.74|> |
|
|
<|10.78|> tahu,<|10.88|> |
|
|
<|10.98|> jika<|11.04|> |
|
|
<|11.08|> anda<|11.12|> |
|
|
<|11.18|> menikmatinya,<|11.52|> |
|
|
<|11.64|> ya,<|11.76|> |
|
|
<|11.94|> maka<|12.06|> |
|
|
<|12.24|> anda<|12.30|> |
|
|
<|12.36|> okay<|12.48|> |
|
|
<|12.52|> dengan<|12.60|> |
|
|
<|12.64|> itu.<|12.68|> |
|
|
<|15.64|> Macam<|15.84|> |
|
|
<|15.88|> lahsia<|16.16|> |
|
|
<|16.24|> semua<|16.36|> |
|
|
<|16.40|> aku<|16.54|> |
|
|
<|16.80|> buat<|16.96|> |
|
|
<|17.00|> pun,<|17.12|> |
|
|
<|18.00|> pada<|18.12|> |
|
|
<|18.16|> masa<|18.28|> |
|
|
<|18.32|> sama<|18.44|> |
|
|
<|18.48|> aku<|18.56|> |
|
|
<|18.64|> shoot.<|18.88|> |
|
|
<|19.16|> Pada<|19.24|> |
|
|
<|19.28|> sama<|19.42|> |
|
|
<|19.46|> masa<|19.56|> |
|
|
<|19.60|> aku<|19.68|> |
|
|
<|19.72|> ada<|19.76|> |
|
|
<|19.80|> pusing<|19.92|> |
|
|
<|19.96|> motion.<|20.24|> |
|
|
<|20.36|> Oh.<|20.48|> |
|
|
<|20.56|> Oh.<|20.62|> |
|
|
<|20.68|> Jadi<|21.40|> |
|
|
<|21.50|> macam<|21.68|> |
|
|
<|21.72|> kelangkabut.<|22.12|> |
|
|
<|22.18|> Lepas<|22.38|> |
|
|
<|22.42|> lahsia<|22.90|> |
|
|
<|22.98|> pula<|23.08|> |
|
|
<|23.16|> aku<|23.22|> |
|
|
<|23.30|> shoot<|23.44|> |
|
|
<|23.48|> dekat<|23.72|> |
|
|
<|24.10|> Langkawi.<|24.44|> |
|
|
<|25.14|> Jadi<|25.24|> |
|
|
<|25.28|> kita<|25.44|> |
|
|
<|25.48|> ambil<|25.60|> |
|
|
<|25.64|> penerbangan.<|25.92|> |
|
|
<|26.08|> Lepas<|26.20|> |
|
|
<|26.24|> tu<|26.30|> |
|
|
<|26.34|> pergi<|26.52|> |
|
|
<|26.64|> ambil<|26.72|> |
|
|
<|26.80|> penerbangan,<|26.96|> |
|
|
<|27.00|> balik<|27.32|> |
|
|
<|27.36|> KL,<|27.58|> |
|
|
<|27.62|> lepas<|27.72|> |
|
|
<|27.76|> tu<|27.78|> |
|
|
<|27.78|> pulak<|27.88|> |
|
|
<|27.92|> KL.<|27.98|> |
|
|
<|28.04|> Kita<|28.12|> |
|
|
<|28.16|> jadi<|28.56|> |
|
|
<|28.60|> macam<|28.72|> |
|
|
<|28.76|> lah.<|28.80|> |
|
|
<|28.84|> Itu<|28.92|> |
|
|
<|28.96|> penat.<|29.32|> |
|
|
<|29.36|> Kelangkabut<|29.72|> |
|
|
<|endoftext|> |
|
|
``` |
