## Roberta-Large

This repo trains [roberta-large](https://huggingface.co/roberta-large) from scratch on the [Norwegian training subset of Oscar](https://oscar-corpus.com/) containing roughly 4.7 GB of data.

A ByteLevelBPETokenizer as shown in [this]( ) blog post was trained on the whole [Norwegian training subset of Oscar](https://oscar-corpus.com/).

Training is done on a TPUv3-8 in Flax. The training script as well as the script to create a tokenizer are attached below.

### Run 1

```
--weight_decay="0.01"
--max_seq_length="128"
--train_batch_size="1048"
--eval_batch_size="1048"
--learning_rate="1e-3"
--warmup_steps="2000"
--pad_to_max_length
--num_train_epochs="12"
--adam_beta1="0.9"
--adam_beta2="0.98"
```

Trained for 12 epochs with each epoch including 8005 steps => Total of 96K steps. 1 epoch + eval takes roughly 2 hours 40 minutes => trained in total for 1 day and 8 hours. Final loss was 3.695.

**Acc**:

![Acc](https://raw.githubusercontent.com/patrickvonplaten/scientific_images/master/flax_experiments/norwegian_large_acc_1.svg)

**Loss**:

![Loss](https://raw.githubusercontent.com/patrickvonplaten/scientific_images/master/flax_experiments/norwegian_large_loss_1.svg)

### Run 2

```
--weight_decay="0.01"
--max_seq_length="128"
--train_batch_size="1048"
--eval_batch_size="1048"
--learning_rate="5e-3"
--warmup_steps="2000"
--pad_to_max_length
--num_train_epochs="7"
--adam_beta1="0.9"
--adam_beta2="0.98"
```

Trained for 7 epochs with each epoch including 8005 steps => Total of 96K steps. 1 epoch + eval takes roughly 2 hours 40 minutes => trained in total for 18 hours. Final loss was 2.216 and accuracy 0.58.


**Acc**:

![Acc](https://raw.githubusercontent.com/patrickvonplaten/scientific_images/master/flax_experiments/norwegian_large_acc_2.svg)

**Loss**:

![Loss](https://raw.githubusercontent.com/patrickvonplaten/scientific_images/master/flax_experiments/norwegian_large_loss_2.svg)

{}
transformers tensorboard roberta fill-mask autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #tensorboard #roberta #fill-mask #autotrain_compatible #endpoints_compatible #region-us 

## Roberta-Large

This repo trains roberta-large from scratch on the Norwegian training subset of Oscar containing roughly 4.7 GB of data.

A ByteLevelBPETokenizer as shown in this blog post was trained on the whole Norwegian training subset of Oscar.

Training is done on a TPUv3-8 in Flax. The training script as well as the script to create a tokenizer are attached below.

### Run 1



Trained for 12 epochs with each epoch including 8005 steps => Total of 96K steps. 1 epoch + eval takes roughly 2 hours 40 minutes => trained in total for 1 day and 8 hours. Final loss was 3.695.

Acc:

!Acc

Loss:

!Loss

### Run 2



Trained for 7 epochs with each epoch including 8005 steps => Total of 96K steps. 1 epoch + eval takes roughly 2 hours 40 minutes => trained in total for 18 hours. Final loss was 2.216 and accuracy 0.58.


Acc:

!Acc

Loss:

!Loss

## Roberta-Large

This repo trains roberta-large from scratch on the Norwegian training subset of Oscar containing roughly 4.7 GB of data.

A ByteLevelBPETokenizer as shown in this blog post was trained on the whole Norwegian training subset of Oscar.

Training is done on a TPUv3-8 in Flax. The training script as well as the script to create a tokenizer are attached below. ### Run 1



Trained for 12 epochs with each epoch including 8005 steps => Total of 96K steps. 1 epoch + eval takes roughly 2 hours 40 minutes => trained in total for 1 day and 8 hours. Final loss was 3.695.

Acc:

!Acc

Loss:

!Loss ### Run 2



Trained for 7 epochs with each epoch including 8005 steps => Total of 96K steps. 1 epoch + eval takes roughly 2 hours 40 minutes => trained in total for 18 hours. Final loss was 2.216 and accuracy 0.58.


Acc:

!Acc

Loss:

!Loss
TAGS
#transformers #tensorboard #roberta #fill-mask #autotrain_compatible #endpoints_compatible #region-us 
 ## Roberta-Large

This repo trains roberta-large from scratch on the Norwegian training subset of Oscar containing roughly 4.7 GB of data.

A ByteLevelBPETokenizer as shown in this blog post was trained on the whole Norwegian training subset of Oscar.

Training is done on a TPUv3-8 in Flax. The training script as well as the script to create a tokenizer are attached below. ### Run 1



Trained for 12 epochs with each epoch including 8005 steps => Total of 96K steps. 1 epoch + eval takes roughly 2 hours 40 minutes => trained in total for 1 day and 8 hours. Final loss was 3.695.

Acc:

!Acc

Loss:

!Loss ### Run 2



Trained for 7 epochs with each epoch including 8005 steps => Total of 96K steps. 1 epoch + eval takes roughly 2 hours 40 minutes => trained in total for 18 hours. Final loss was 2.216 and accuracy 0.58.


Acc:

!Acc

Loss:

!Loss
passage: TAGS
#transformers #tensorboard #roberta #fill-mask #autotrain_compatible #endpoints_compatible #region-us 
## Roberta-Large

This repo trains roberta-large from scratch on the Norwegian training subset of Oscar containing roughly 4.7 GB of data.

A ByteLevelBPETokenizer as shown in this blog post was trained on the whole Norwegian training subset of Oscar.

Training is done on a TPUv3-8 in Flax. The training script as well as the script to create a tokenizer are attached below.### Run 1



Trained for 12 epochs with each epoch including 8005 steps => Total of 96K steps. 1 epoch + eval takes roughly 2 hours 40 minutes => trained in total for 1 day and 8 hours. Final loss was 3.695.

Acc:

!Acc

Loss:

!Loss### Run 2



Trained for 7 epochs with each epoch including 8005 steps => Total of 96K steps. 1 epoch + eval takes roughly 2 hours 40 minutes => trained in total for 18 hours. Final loss was 2.216 and accuracy 0.58.


Acc:

!Acc

Loss:

!Loss