
# roberta-base-bahasa-cased

Pretrained RoBERTa base language model for Malay.

## Pretraining Corpus

`roberta-base-bahasa-cased` model was pretrained on ~400 miliion words. Below is list of data we trained on,

1. IIUM confession, https://github.com/huseinzol05/malay-dataset/tree/master/dumping/clean
2. local Instagram, https://github.com/huseinzol05/malay-dataset/tree/master/dumping/clean
3. local news, https://github.com/huseinzol05/malay-dataset/tree/master/dumping/clean
4. local parliament hansards, https://github.com/huseinzol05/malay-dataset/tree/master/dumping/clean
5. local research papers related to `kebudayaan`, `keagaaman` and `etnik`, https://github.com/huseinzol05/malay-dataset/tree/master/dumping/clean
6. local twitter, https://github.com/huseinzol05/malay-dataset/tree/master/dumping/clean
7. Malay Wattpad, https://github.com/huseinzol05/malay-dataset/tree/master/dumping/clean
8. Malay Wikipedia, https://github.com/huseinzol05/malay-dataset/tree/master/dumping/clean

## Pretraining details

- All steps can reproduce from https://github.com/huseinzol05/Malaya/tree/master/pretrained-model/roberta.

## Example using AutoModelWithLMHead

```python
from transformers import AutoTokenizer, AutoModelForMaskedLM, pipeline

model = AutoModelForMaskedLM.from_pretrained('mesolitica/roberta-base-bahasa-cased')
tokenizer = AutoTokenizer.from_pretrained(
    'mesolitica/roberta-base-bahasa-cased',
    do_lower_case = False,
)
fill_mask = pipeline('fill-mask', model=model, tokenizer=tokenizer)
fill_mask('Permohonan Najib, anak untuk dengar isu perlembagaan <mask> .')
```

Output is,

```json
[{'score': 0.3368818759918213,
  'token': 746,
  'token_str': ' negara',
  'sequence': 'Permohonan Najib, anak untuk dengar isu perlembagaan negara.'},
 {'score': 0.09646568447351456,
  'token': 598,
  'token_str': ' Malaysia',
  'sequence': 'Permohonan Najib, anak untuk dengar isu perlembagaan Malaysia.'},
 {'score': 0.029483484104275703,
  'token': 3265,
  'token_str': ' UMNO',
  'sequence': 'Permohonan Najib, anak untuk dengar isu perlembagaan UMNO.'},
 {'score': 0.026470622047781944,
  'token': 2562,
  'token_str': ' parti',
  'sequence': 'Permohonan Najib, anak untuk dengar isu perlembagaan parti.'},
 {'score': 0.023237623274326324,
  'token': 391,
  'token_str': ' ini',
  'sequence': 'Permohonan Najib, anak untuk dengar isu perlembagaan ini.'}]
```



{"language": "ms"}
transformers pytorch tensorboard roberta fill-mask ms autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #tensorboard #roberta #fill-mask #ms #autotrain_compatible #endpoints_compatible #region-us 


# roberta-base-bahasa-cased

Pretrained RoBERTa base language model for Malay.

## Pretraining Corpus

'roberta-base-bahasa-cased' model was pretrained on ~400 miliion words. Below is list of data we trained on,

1. IIUM confession, URL
2. local Instagram, URL
3. local news, URL
4. local parliament hansards, URL
5. local research papers related to 'kebudayaan', 'keagaaman' and 'etnik', URL
6. local twitter, URL
7. Malay Wattpad, URL
8. Malay Wikipedia, URL

## Pretraining details

- All steps can reproduce from URL

## Example using AutoModelWithLMHead



Output is,





# roberta-base-bahasa-cased

Pretrained RoBERTa base language model for Malay. ## Pretraining Corpus

'roberta-base-bahasa-cased' model was pretrained on ~400 miliion words. Below is list of data we trained on,

1. IIUM confession, URL
2. local Instagram, URL
3. local news, URL
4. local parliament hansards, URL
5. local research papers related to 'kebudayaan', 'keagaaman' and 'etnik', URL
6. local twitter, URL
7. Malay Wattpad, URL
8. Malay Wikipedia, URL ## Pretraining details

- All steps can reproduce from URL ## Example using AutoModelWithLMHead



Output is,
TAGS
#transformers #pytorch #tensorboard #roberta #fill-mask #ms #autotrain_compatible #endpoints_compatible #region-us 
 # roberta-base-bahasa-cased

Pretrained RoBERTa base language model for Malay. ## Pretraining Corpus

'roberta-base-bahasa-cased' model was pretrained on ~400 miliion words. Below is list of data we trained on,

1. IIUM confession, URL
2. local Instagram, URL
3. local news, URL
4. local parliament hansards, URL
5. local research papers related to 'kebudayaan', 'keagaaman' and 'etnik', URL
6. local twitter, URL
7. Malay Wattpad, URL
8. Malay Wikipedia, URL ## Pretraining details

- All steps can reproduce from URL ## Example using AutoModelWithLMHead



Output is,
passage: TAGS
#transformers #pytorch #tensorboard #roberta #fill-mask #ms #autotrain_compatible #endpoints_compatible #region-us 
# roberta-base-bahasa-cased

Pretrained RoBERTa base language model for Malay.## Pretraining Corpus

'roberta-base-bahasa-cased' model was pretrained on ~400 miliion words. Below is list of data we trained on,

1. IIUM confession, URL
2. local Instagram, URL
3. local news, URL
4. local parliament hansards, URL
5. local research papers related to 'kebudayaan', 'keagaaman' and 'etnik', URL
6. local twitter, URL
7. Malay Wattpad, URL
8. Malay Wikipedia, URL## Pretraining details

- All steps can reproduce from URL## Example using AutoModelWithLMHead



Output is,