---
datasets:
- HuggingFaceFW/fineweb
language:
- en
---
# Encoder-Decoder model with DeBERTa encoder

## pre-trained models

- `deliciouscat/deberta-v3-base-encoder-decoder-v0.2`

-> 297511524(298M) params

## Data used

- `HuggingFaceFW/fineweb`

- AiHub ko-en translation corpus (English part)

- Some papers that I kept

## Training hparams

- optimizer: AdamW, lr=3e-5, betas=(0.875, 0.997)

- batch size: 12

-> training on denoising objective (BART), 29523 step

## How to use

```
from transformers import AutoTokenizer, EncoderDecoderModel

model = EncoderDecoderModel.from_pretrained("deliciouscat/deberta-v3-base-encoder-decoder-v0.3")
tokenizer = AutoTokenizer.from_pretrained("deliciouscat/deberta-v3-base-encoder-decoder-v0.3")
```

## Future work!

- train more scientific data

- fine-tune on keyword extraction task