pytorch bert fill-mask transformers generated_from_trainer model-index autotrain_compatible
---
tags:
- generated_from_trainer
model-index:
- name: bertlawbr
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# bertlawbr

This model is a fine-tuned version of [](https://huggingface.co/) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 1.0495

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- gradient_accumulation_steps: 8
- total_train_batch_size: 128
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-06
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 10000
- num_epochs: 20.0
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step   | Validation Loss |
|:-------------:|:-----:|:------:|:---------------:|
| 6.1291        | 0.22  | 2500   | 5.9888          |
| 4.8604        | 0.44  | 5000   | 4.4841          |
| 3.3321        | 0.66  | 7500   | 3.1190          |
| 2.7579        | 0.87  | 10000  | 2.6089          |
| 2.4135        | 1.09  | 12500  | 2.3029          |
| 2.2136        | 1.31  | 15000  | 2.1244          |
| 2.0735        | 1.53  | 17500  | 1.9931          |
| 1.9684        | 1.75  | 20000  | 1.8878          |
| 1.891         | 1.97  | 22500  | 1.8077          |
| 1.8215        | 2.18  | 25000  | 1.7487          |
| 1.7577        | 2.4   | 27500  | 1.6875          |
| 1.7113        | 2.62  | 30000  | 1.6444          |
| 1.6776        | 2.84  | 32500  | 1.6036          |
| 1.6203        | 3.06  | 35000  | 1.5608          |
| 1.6018        | 3.28  | 37500  | 1.5293          |
| 1.5602        | 3.5   | 40000  | 1.5044          |
| 1.5429        | 3.71  | 42500  | 1.4753          |
| 1.5148        | 3.93  | 45000  | 1.4472          |
| 1.4786        | 4.15  | 47500  | 1.4302          |
| 1.4653        | 4.37  | 50000  | 1.4128          |
| 1.4496        | 4.59  | 52500  | 1.3991          |
| 1.4445        | 4.81  | 55000  | 1.3943          |
| 1.5114        | 5.02  | 57500  | 1.4551          |
| 1.5054        | 5.24  | 60000  | 1.4525          |
| 1.4817        | 5.46  | 62500  | 1.4259          |
| 1.48          | 5.68  | 65000  | 1.4077          |
| 1.4526        | 5.9   | 67500  | 1.3912          |
| 1.4272        | 6.12  | 70000  | 1.3726          |
| 1.4078        | 6.34  | 72500  | 1.3596          |
| 1.399         | 6.55  | 75000  | 1.3450          |
| 1.386         | 6.77  | 77500  | 1.3328          |
| 1.3704        | 6.99  | 80000  | 1.3192          |
| 1.3538        | 7.21  | 82500  | 1.3131          |
| 1.3468        | 7.43  | 85000  | 1.2916          |
| 1.323         | 7.65  | 87500  | 1.2871          |
| 1.322         | 7.86  | 90000  | 1.2622          |
| 1.2956        | 8.08  | 92500  | 1.2624          |
| 1.2869        | 8.3   | 95000  | 1.2547          |
| 1.2763        | 8.52  | 97500  | 1.2404          |
| 1.275         | 8.74  | 100000 | 1.2305          |
| 1.2709        | 8.96  | 102500 | 1.2301          |
| 1.2514        | 9.18  | 105000 | 1.2179          |
| 1.2563        | 9.39  | 107500 | 1.2134          |
| 1.2487        | 9.61  | 110000 | 1.2111          |
| 1.2337        | 9.83  | 112500 | 1.2041          |
| 1.3215        | 10.05 | 115000 | 1.2879          |
| 1.3364        | 10.27 | 117500 | 1.2850          |
| 1.3286        | 10.49 | 120000 | 1.2779          |
| 1.3202        | 10.7  | 122500 | 1.2730          |
| 1.3181        | 10.92 | 125000 | 1.2651          |
| 1.2952        | 11.14 | 127500 | 1.2544          |
| 1.2889        | 11.36 | 130000 | 1.2506          |
| 1.2747        | 11.58 | 132500 | 1.2339          |
| 1.2729        | 11.8  | 135000 | 1.2277          |
| 1.2699        | 12.02 | 137500 | 1.2201          |
| 1.2508        | 12.23 | 140000 | 1.2163          |
| 1.2438        | 12.45 | 142500 | 1.2091          |
| 1.2445        | 12.67 | 145000 | 1.2003          |
| 1.2314        | 12.89 | 147500 | 1.1957          |
| 1.2188        | 13.11 | 150000 | 1.1843          |
| 1.2071        | 13.33 | 152500 | 1.1805          |
| 1.2123        | 13.54 | 155000 | 1.1766          |
| 1.2016        | 13.76 | 157500 | 1.1661          |
| 1.2079        | 13.98 | 160000 | 1.1625          |
| 1.1884        | 14.2  | 162500 | 1.1525          |
| 1.177         | 14.42 | 165000 | 1.1419          |
| 1.1793        | 14.64 | 167500 | 1.1454          |
| 1.173         | 14.85 | 170000 | 1.1379          |
| 1.1502        | 15.07 | 172500 | 1.1371          |
| 1.1504        | 15.29 | 175000 | 1.1295          |
| 1.146         | 15.51 | 177500 | 1.1203          |
| 1.1487        | 15.73 | 180000 | 1.1137          |
| 1.1329        | 15.95 | 182500 | 1.1196          |
| 1.1259        | 16.17 | 185000 | 1.1075          |
| 1.1287        | 16.38 | 187500 | 1.1037          |
| 1.126         | 16.6  | 190000 | 1.1042          |
| 1.1199        | 16.82 | 192500 | 1.0953          |
| 1.1072        | 17.04 | 195000 | 1.0885          |
| 1.1043        | 17.26 | 197500 | 1.0877          |
| 1.1007        | 17.48 | 200000 | 1.0835          |
| 1.0879        | 17.69 | 202500 | 1.0819          |
| 1.1           | 17.91 | 205000 | 1.0744          |
| 1.0863        | 18.13 | 207500 | 1.0774          |
| 1.087         | 18.35 | 210000 | 1.0759          |
| 1.0755        | 18.57 | 212500 | 1.0618          |
| 1.0832        | 18.79 | 215000 | 1.0628          |
| 1.0771        | 19.01 | 217500 | 1.0611          |
| 1.0703        | 19.22 | 220000 | 1.0555          |
| 1.069         | 19.44 | 222500 | 1.0552          |
| 1.0706        | 19.66 | 225000 | 1.0509          |
| 1.0633        | 19.88 | 227500 | 1.0465          |


### Framework versions

- Transformers 4.12.5
- Pytorch 1.10.1+cu113
- Datasets 1.17.0
- Tokenizers 0.10.3
