Hugging Face's logo

postbot
/
bert_uncased_L-12_H-128_A-2-mlm-multi-emails-hq

Fill-Mask
Transformers
PyTorch
bert
Generated from Trainer
Inference Endpoints
Model card Files Community
1
bert_uncased_L-12_H-128_A-2-mlm-multi-emails-hq / README.md
pszemraj's picture
pszemraj
Update README.md
5a8cc2a
preview code
|
raw
history blame contribute delete
2.66 kB
metadata
license: apache-2.0
tags:
  - generated_from_trainer
metrics:
  - accuracy
model-index:
  - name: bert_uncased_L-12_H-128_A-2-mlm-multi-emails-hq-r2
    results: []
widget:
  - text: Can you please send me the [MASK] by the end of the day?
    example_title: end of day
  - text: >-
      I hope this email finds you well. I wanted to follow up on our [MASK]
      yesterday.
    example_title: follow-up
  - text: The meeting has been rescheduled to [MASK].
    example_title: reschedule
  - text: Please let me know if you need any further [MASK] regarding the project.
    example_title: further help
  - text: >-
      I appreciate your prompt response to my previous email. Can you provide an
      update on the [MASK] by tomorrow?
    example_title: provide update
  - text: Paris is the [MASK] of France.
    example_title: paris (default)
  - text: The goal of life is [MASK].
    example_title: goal of life (default)

bert_uncased_L-12_H-128_A-2-mlm-multi-emails-hq

This model is a fine-tuned version of google/bert_uncased_L-12_H-128_A-2 on the None dataset. It achieves the following results on the evaluation set:

  • Loss: 2.4235
  • Accuracy: 0.5780

Model description

This is a 40 MB version of BERT that does surprisingly well!

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0003
  • train_batch_size: 8
  • eval_batch_size: 2
  • seed: 42
  • distributed_type: multi-GPU
  • gradient_accumulation_steps: 16
  • total_train_batch_size: 128
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.05
  • num_epochs: 8.0
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss Accuracy
2.9421 0.99 141 2.7769 0.5330
2.772 1.99 282 2.6669 0.5487
2.6997 2.99 423 2.5486 0.5621
2.6281 3.99 564 2.4865 0.5704
2.5626 4.99 705 2.4385 0.5766
2.5504 5.99 846 2.4421 0.5772
2.5434 6.99 987 2.4094 0.5818
2.5174 7.99 1128 2.4235 0.5780

Framework versions

  • Transformers 4.27.0.dev0
  • Pytorch 2.0.0.dev20230129+cu118
  • Datasets 2.8.0
  • Tokenizers 0.13.1