
# byt5-dv

Pretrained from scratch on Dhivei (language of the Maldives)
with ByT5, Google's new byte-level tokenizer strategy.

Corpus: dv.wikipedia.org as of March 2020 (TFDS)

Notebook - Pretraining on Wikipedia: https://colab.research.google.com/drive/19Afq7CI6cOi1DaTpnQhBbEbnBzLSFHbH

## Demo

Notebook - Finetuning on Maldivian news classification task: https://colab.research.google.com/drive/11u5SafR4bKICmArgDl6KQ9vqfYtDpyWp

Current performance:

- mBERT: 52%
- **byt5-dv**: 81%
- dv-wave (ELECTRA): 89%
- dv-muril: 90.7%
- dv-labse: 91.3-91.5%

Source of dataset: https://github.com/Sofwath/DhivehiDatasets

## Work in progress - todos

The Wikipedia corpus is too small for this language. In the future I would add
OSCAR and Sofwath's Maldivian corpus, if I can rewrite the script to accept those
as one TFDS dataset.

This is based on ByT5-small ... we should try a larger model

This needs more time for pretraining
{"language": "dv"}
transformers pytorch tf t5 text2text-generation dv autotrain_compatible endpoints_compatible text-generation-inference region:us
TAGS
#transformers #pytorch #tf #t5 #text2text-generation #dv #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 


# byt5-dv

Pretrained from scratch on Dhivei (language of the Maldives)
with ByT5, Google's new byte-level tokenizer strategy.

Corpus: URL as of March 2020 (TFDS)

Notebook - Pretraining on Wikipedia: URL

## Demo

Notebook - Finetuning on Maldivian news classification task: URL

Current performance:

- mBERT: 52%
- byt5-dv: 81%
- dv-wave (ELECTRA): 89%
- dv-muril: 90.7%
- dv-labse: 91.3-91.5%

Source of dataset: URL

## Work in progress - todos

The Wikipedia corpus is too small for this language. In the future I would add
OSCAR and Sofwath's Maldivian corpus, if I can rewrite the script to accept those
as one TFDS dataset.

This is based on ByT5-small ... we should try a larger model

This needs more time for pretraining
# byt5-dv

Pretrained from scratch on Dhivei (language of the Maldives)
with ByT5, Google's new byte-level tokenizer strategy.

Corpus: URL as of March 2020 (TFDS)

Notebook - Pretraining on Wikipedia: URL ## Demo

Notebook - Finetuning on Maldivian news classification task: URL

Current performance:

- mBERT: 52%
- byt5-dv: 81%
- dv-wave (ELECTRA): 89%
- dv-muril: 90.7%
- dv-labse: 91.3-91.5%

Source of dataset: URL ## Work in progress - todos

The Wikipedia corpus is too small for this language. In the future I would add
OSCAR and Sofwath's Maldivian corpus, if I can rewrite the script to accept those
as one TFDS dataset.

This is based on ByT5-small ... we should try a larger model

This needs more time for pretraining
TAGS
#transformers #pytorch #tf #t5 #text2text-generation #dv #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 
 # byt5-dv

Pretrained from scratch on Dhivei (language of the Maldives)
with ByT5, Google's new byte-level tokenizer strategy.

Corpus: URL as of March 2020 (TFDS)

Notebook - Pretraining on Wikipedia: URL ## Demo

Notebook - Finetuning on Maldivian news classification task: URL

Current performance:

- mBERT: 52%
- byt5-dv: 81%
- dv-wave (ELECTRA): 89%
- dv-muril: 90.7%
- dv-labse: 91.3-91.5%

Source of dataset: URL ## Work in progress - todos

The Wikipedia corpus is too small for this language. In the future I would add
OSCAR and Sofwath's Maldivian corpus, if I can rewrite the script to accept those
as one TFDS dataset.

This is based on ByT5-small ... we should try a larger model

This needs more time for pretraining
passage: TAGS
#transformers #pytorch #tf #t5 #text2text-generation #dv #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 
# byt5-dv

Pretrained from scratch on Dhivei (language of the Maldives)
with ByT5, Google's new byte-level tokenizer strategy.

Corpus: URL as of March 2020 (TFDS)

Notebook - Pretraining on Wikipedia: URL## Demo

Notebook - Finetuning on Maldivian news classification task: URL

Current performance:

- mBERT: 52%
- byt5-dv: 81%
- dv-wave (ELECTRA): 89%
- dv-muril: 90.7%
- dv-labse: 91.3-91.5%

Source of dataset: URL## Work in progress - todos

The Wikipedia corpus is too small for this language. In the future I would add
OSCAR and Sofwath's Maldivian corpus, if I can rewrite the script to accept those
as one TFDS dataset.

This is based on ByT5-small ... we should try a larger model

This needs more time for pretraining