|
--- |
|
datasets: |
|
- HuggingFaceFW/fineweb |
|
language: |
|
- en |
|
--- |
|
# Encoder-Decoder model with DeBERTa encoder |
|
|
|
## pre-trained models |
|
|
|
- Encoder: `microsoft/deberta-v3-small` |
|
|
|
- Decoder: `deliciouscat/deberta-v3-base-decoder-v0.1` (6 transformer layers, 8 attention heads) |
|
|
|
-> 297511524(298M) params |
|
|
|
## Data used |
|
|
|
`HuggingFaceFW/fineweb` -> sampled 124800 |
|
|
|
## Training hparams |
|
|
|
- optimizer: AdamW, lr=2.3e-5, betas=(0.875, 0.997) |
|
|
|
- batch size: 12 (maximal on Colab pro A100 env) |
|
|
|
-> training on denoising objective (BART) |
|
|
|
## How to use |
|
|
|
``` |
|
from transformers import AutoTokenizer, EncoderDecoderModel |
|
|
|
model = EncoderDecoderModel.from_pretrained("deliciouscat/deberta-v3-base-encoder-decoder-v0.2") |
|
tokenizer = AutoTokenizer.from_pretrained("deliciouscat/deberta-v3-base-encoder-decoder-v0.2") |
|
``` |
|
|
|
## Future work! |
|
|
|
- train more scientific data |
|
|
|
- fine-tune on keyword extraction task |
