
# dv-labse

This is an experiment in cross-lingual transfer learning, to insert Dhivehi word and
word-piece tokens into Google's LaBSE model.

- Original model weights: https://huggingface.co/setu4993/LaBSE
- Original model announcement: https://ai.googleblog.com/2020/08/language-agnostic-bert-sentence.html

This currently outperforms dv-wave and dv-MuRIL (a similar transfer learning model) on 
the Maldivian News Classification task https://github.com/Sofwath/DhivehiDatasets

- mBERT: 52%
- dv-wave (ELECTRA): 89%
- dv-muril: 90.7%
- dv-labse: 91.3-91.5% (may continue training)

## Training

- Start with LaBSE (similar to mBERT) with no Thaana vocabulary
- Based on PanLex dictionaries, attach 1,100 Dhivehi words to Sinhalese or English embeddings
- Add remaining words and word-pieces from dv-wave's vocabulary to vocab.txt
- Continue BERT pretraining on Dhivehi text

CoLab notebook: 
https://colab.research.google.com/drive/1CUn44M2fb4Qbat2pAvjYqsPvWLt1Novi

{"language": "dv"}
transformers pytorch jax bert fill-mask dv autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #jax #bert #fill-mask #dv #autotrain_compatible #endpoints_compatible #region-us 


# dv-labse

This is an experiment in cross-lingual transfer learning, to insert Dhivehi word and
word-piece tokens into Google's LaBSE model.

- Original model weights: URL
- Original model announcement: URL

This currently outperforms dv-wave and dv-MuRIL (a similar transfer learning model) on 
the Maldivian News Classification task URL

- mBERT: 52%
- dv-wave (ELECTRA): 89%
- dv-muril: 90.7%
- dv-labse: 91.3-91.5% (may continue training)

## Training

- Start with LaBSE (similar to mBERT) with no Thaana vocabulary
- Based on PanLex dictionaries, attach 1,100 Dhivehi words to Sinhalese or English embeddings
- Add remaining words and word-pieces from dv-wave's vocabulary to URL
- Continue BERT pretraining on Dhivehi text

CoLab notebook: 
URL

# dv-labse

This is an experiment in cross-lingual transfer learning, to insert Dhivehi word and
word-piece tokens into Google's LaBSE model.

- Original model weights: URL
- Original model announcement: URL

This currently outperforms dv-wave and dv-MuRIL (a similar transfer learning model) on 
the Maldivian News Classification task URL

- mBERT: 52%
- dv-wave (ELECTRA): 89%
- dv-muril: 90.7%
- dv-labse: 91.3-91.5% (may continue training) ## Training

- Start with LaBSE (similar to mBERT) with no Thaana vocabulary
- Based on PanLex dictionaries, attach 1,100 Dhivehi words to Sinhalese or English embeddings
- Add remaining words and word-pieces from dv-wave's vocabulary to URL
- Continue BERT pretraining on Dhivehi text

CoLab notebook: 
URL
TAGS
#transformers #pytorch #jax #bert #fill-mask #dv #autotrain_compatible #endpoints_compatible #region-us 
 # dv-labse

This is an experiment in cross-lingual transfer learning, to insert Dhivehi word and
word-piece tokens into Google's LaBSE model.

- Original model weights: URL
- Original model announcement: URL

This currently outperforms dv-wave and dv-MuRIL (a similar transfer learning model) on 
the Maldivian News Classification task URL

- mBERT: 52%
- dv-wave (ELECTRA): 89%
- dv-muril: 90.7%
- dv-labse: 91.3-91.5% (may continue training) ## Training

- Start with LaBSE (similar to mBERT) with no Thaana vocabulary
- Based on PanLex dictionaries, attach 1,100 Dhivehi words to Sinhalese or English embeddings
- Add remaining words and word-pieces from dv-wave's vocabulary to URL
- Continue BERT pretraining on Dhivehi text

CoLab notebook: 
URL
passage: TAGS
#transformers #pytorch #jax #bert #fill-mask #dv #autotrain_compatible #endpoints_compatible #region-us 
# dv-labse

This is an experiment in cross-lingual transfer learning, to insert Dhivehi word and
word-piece tokens into Google's LaBSE model.

- Original model weights: URL
- Original model announcement: URL

This currently outperforms dv-wave and dv-MuRIL (a similar transfer learning model) on 
the Maldivian News Classification task URL

- mBERT: 52%
- dv-wave (ELECTRA): 89%
- dv-muril: 90.7%
- dv-labse: 91.3-91.5% (may continue training)## Training

- Start with LaBSE (similar to mBERT) with no Thaana vocabulary
- Based on PanLex dictionaries, attach 1,100 Dhivehi words to Sinhalese or English embeddings
- Add remaining words and word-pieces from dv-wave's vocabulary to URL
- Continue BERT pretraining on Dhivehi text

CoLab notebook: 
URL