
## Indonesian RoBERTa Large

Indonesian RoBERTa Large is a masked language model based on the [RoBERTa](https://arxiv.org/abs/1907.11692) model. It was trained on the [OSCAR](https://huggingface.co/datasets/oscar) dataset, specifically the `unshuffled_deduplicated_id` subset. The model was trained from scratch and achieved an evaluation loss of 4.801 and an evaluation accuracy of 29.8%.

This model was trained using HuggingFace's Flax framework and is part of the [JAX/Flax Community Week](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104) organized by HuggingFace. All training was done on a TPUv3-8 VM, sponsored by the Google Cloud team.

All necessary scripts used for training could be found in the [Files and versions](https://huggingface.co/flax-community/indonesian-roberta-large/tree/main) tab, as well as the [Training metrics](https://huggingface.co/flax-community/indonesian-roberta-large/tensorboard) logged via Tensorboard.

## Model

| Model                      | #params | Arch.   | Training/Validation data (text)            |
| -------------------------- | ------- | ------- | ------------------------------------------ |
| `indonesian-roberta-large` | 355M    | RoBERTa | OSCAR `unshuffled_deduplicated_id` Dataset |

## Evaluation Results

The model was trained for 10 epochs and the following is the final result once the training ended.

| train loss | valid loss | valid accuracy | total time |
| ---------- | ---------- | -------------- | ---------- |
| 5.19       | 4.801      | 0.298          | 2:8:32:28  |

## How to Use

### As Masked Language Model

```python
from transformers import pipeline

pretrained_name = "flax-community/indonesian-roberta-large"

fill_mask = pipeline(
    "fill-mask",
    model=pretrained_name,
    tokenizer=pretrained_name
)

fill_mask("Budi sedang <mask> di sekolah.")
```

### Feature Extraction in PyTorch

```python
from transformers import RobertaModel, RobertaTokenizerFast

pretrained_name = "flax-community/indonesian-roberta-large"
model = RobertaModel.from_pretrained(pretrained_name)
tokenizer = RobertaTokenizerFast.from_pretrained(pretrained_name)

prompt = "Budi sedang berada di sekolah."
encoded_input = tokenizer(prompt, return_tensors='pt')
output = model(**encoded_input)
```

## Team Members

- Wilson Wongso ([@w11wo](https://hf.co/w11wo))
- Steven Limcorn ([@stevenlimcorn](https://hf.co/stevenlimcorn))
- Samsul Rahmadani ([@munggok](https://hf.co/munggok))
- Chew Kok Wah ([@chewkokwah](https://hf.co/chewkokwah))

{"language": "id", "license": "mit", "tags": ["indonesian-roberta-large"], "datasets": ["oscar"], "widget": [{"text": "Budi telat ke sekolah karena ia <mask>."}]}
transformers pytorch jax tensorboard safetensors roberta fill-mask indonesian-roberta-large id dataset:oscar arxiv:1907.11692 license:mit autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #jax #tensorboard #safetensors #roberta #fill-mask #indonesian-roberta-large #id #dataset-oscar #arxiv-1907.11692 #license-mit #autotrain_compatible #endpoints_compatible #region-us 

Indonesian RoBERTa Large
------------------------


Indonesian RoBERTa Large is a masked language model based on the RoBERTa model. It was trained on the OSCAR dataset, specifically the 'unshuffled\_deduplicated\_id' subset. The model was trained from scratch and achieved an evaluation loss of 4.801 and an evaluation accuracy of 29.8%.


This model was trained using HuggingFace's Flax framework and is part of the JAX/Flax Community Week organized by HuggingFace. All training was done on a TPUv3-8 VM, sponsored by the Google Cloud team.


All necessary scripts used for training could be found in the Files and versions tab, as well as the Training metrics logged via Tensorboard.


Model
-----



Evaluation Results
------------------


The model was trained for 10 epochs and the following is the final result once the training ended.



How to Use
----------


### As Masked Language Model


### Feature Extraction in PyTorch


Team Members
------------


* Wilson Wongso (@w11wo)
* Steven Limcorn (@stevenlimcorn)
* Samsul Rahmadani (@munggok)
* Chew Kok Wah (@chewkokwah)



### As Masked Language Model ### Feature Extraction in PyTorch


Team Members
------------


* Wilson Wongso (@w11wo)
* Steven Limcorn (@stevenlimcorn)
* Samsul Rahmadani (@munggok)
* Chew Kok Wah (@chewkokwah)
TAGS
#transformers #pytorch #jax #tensorboard #safetensors #roberta #fill-mask #indonesian-roberta-large #id #dataset-oscar #arxiv-1907.11692 #license-mit #autotrain_compatible #endpoints_compatible #region-us 
 ### As Masked Language Model ### Feature Extraction in PyTorch


Team Members
------------


* Wilson Wongso (@w11wo)
* Steven Limcorn (@stevenlimcorn)
* Samsul Rahmadani (@munggok)
* Chew Kok Wah (@chewkokwah)
passage: TAGS
#transformers #pytorch #jax #tensorboard #safetensors #roberta #fill-mask #indonesian-roberta-large #id #dataset-oscar #arxiv-1907.11692 #license-mit #autotrain_compatible #endpoints_compatible #region-us 
### As Masked Language Model### Feature Extraction in PyTorch


Team Members
------------


* Wilson Wongso (@w11wo)
* Steven Limcorn (@stevenlimcorn)
* Samsul Rahmadani (@munggok)
* Chew Kok Wah (@chewkokwah)