
## DeBERTa: Decoding-enhanced BERT with Disentangled Attention

[DeBERTa](https://arxiv.org/abs/2006.03654) improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. It outperforms BERT and RoBERTa on  majority of NLU tasks with 80GB training data. 

Please check the [official repository](https://github.com/microsoft/DeBERTa) for more details and updates.

This model is the base DeBERTa model fine-tuned with MNLI task

#### Fine-tuning on NLU tasks

We present the dev results on SQuAD 1.1/2.0 and MNLI tasks.

| Model             | SQuAD 1.1 | SQuAD 2.0 | MNLI-m |
|-------------------|-----------|-----------|--------|
| RoBERTa-base      | 91.5/84.6 | 83.7/80.5 | 87.6   |
| XLNet-Large       | -/-       | -/80.2    | 86.8   |
| **DeBERTa-base**  | 93.1/87.2 | 86.2/83.1 | 88.8   |

### Citation

If you find DeBERTa useful for your work, please cite the following paper:

``` latex
@inproceedings{
he2021deberta,
title={DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION},
author={Pengcheng He and Xiaodong Liu and Jianfeng Gao and Weizhu Chen},
booktitle={International Conference on Learning Representations},
year={2021},
url={https://openreview.net/forum?id=XPZIaotutsD}
}
```

{"language": "en", "license": "mit", "tags": ["deberta-v1", "deberta-mnli"], "tasks": "mnli", "thumbnail": "https://huggingface.co/front/thumbnails/microsoft.png", "widget": [{"text": "[CLS] I love you. [SEP] I like you. [SEP]"}]}
transformers pytorch rust deberta text-classification deberta-v1 deberta-mnli en arxiv:2006.03654 license:mit autotrain_compatible endpoints_compatible has_space region:us
TAGS
#transformers #pytorch #rust #deberta #text-classification #deberta-v1 #deberta-mnli #en #arxiv-2006.03654 #license-mit #autotrain_compatible #endpoints_compatible #has_space #region-us 

DeBERTa: Decoding-enhanced BERT with Disentangled Attention
-----------------------------------------------------------


DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. It outperforms BERT and RoBERTa on majority of NLU tasks with 80GB training data.


Please check the official repository for more details and updates.


This model is the base DeBERTa model fine-tuned with MNLI task


#### Fine-tuning on NLU tasks


We present the dev results on SQuAD 1.1/2.0 and MNLI tasks.



If you find DeBERTa useful for your work, please cite the following paper:



#### Fine-tuning on NLU tasks


We present the dev results on SQuAD 1.1/2.0 and MNLI tasks.



If you find DeBERTa useful for your work, please cite the following paper:
TAGS
#transformers #pytorch #rust #deberta #text-classification #deberta-v1 #deberta-mnli #en #arxiv-2006.03654 #license-mit #autotrain_compatible #endpoints_compatible #has_space #region-us 
 #### Fine-tuning on NLU tasks


We present the dev results on SQuAD 1.1/2.0 and MNLI tasks.



If you find DeBERTa useful for your work, please cite the following paper:
passage: TAGS
#transformers #pytorch #rust #deberta #text-classification #deberta-v1 #deberta-mnli #en #arxiv-2006.03654 #license-mit #autotrain_compatible #endpoints_compatible #has_space #region-us 
#### Fine-tuning on NLU tasks


We present the dev results on SQuAD 1.1/2.0 and MNLI tasks.



If you find DeBERTa useful for your work, please cite the following paper: