
# longformer-base-4096-spanish

## [Longformer](https://arxiv.org/abs/2004.05150) is a Transformer model for long documents. 

`longformer-base-4096` is a BERT-like model started from the RoBERTa checkpoint (**BERTIN** in this case) and pre-trained for *MLM* on long documents (from BETO's `all_wikis`). It supports sequences of length up to 4,096! 

 

**Longformer** uses a combination of a sliding window (*local*) attention and *global* attention. Global attention is user-configured based on the task to allow the model to learn task-specific representations.


This model was made following the research done by [Iz Beltagy and Matthew E. Peters and Arman Cohan](https://arxiv.org/abs/2004.05150).

## Citation
If you want to cite this model you can use this:

```bibtex
@misc{mromero2022longformer-base-4096-spanish,
  title={Spanish LongFormer by Manuel Romero},
  author={Romero, Manuel},
  publisher={Hugging Face},
  journal={Hugging Face Hub},
  howpublished={\url{https://huggingface.co/mrm8488/longformer-base-4096-spanish}},
  year={2022}
}
```
{"language": ["es"], "license": "mit", "tags": ["Long documents", "longformer", "bertin", "spanish"], "datasets": ["spanish_large_corpus"], "widget": [{"text": "Manuel Romero ha creado con el equipo de BERTIN un modelo que procesa documentos <mask> largos."}]}
transformers pytorch tensorboard roberta fill-mask Long documents longformer bertin spanish es dataset:spanish_large_corpus arxiv:2004.05150 license:mit autotrain_compatible endpoints_compatible has_space region:us
TAGS
#transformers #pytorch #tensorboard #roberta #fill-mask #Long documents #longformer #bertin #spanish #es #dataset-spanish_large_corpus #arxiv-2004.05150 #license-mit #autotrain_compatible #endpoints_compatible #has_space #region-us 


# longformer-base-4096-spanish

## Longformer is a Transformer model for long documents. 

'longformer-base-4096' is a BERT-like model started from the RoBERTa checkpoint (BERTIN in this case) and pre-trained for *MLM* on long documents (from BETO's 'all_wikis'). It supports sequences of length up to 4,096! 

 

Longformer uses a combination of a sliding window (*local*) attention and *global* attention. Global attention is user-configured based on the task to allow the model to learn task-specific representations.


This model was made following the research done by Iz Beltagy and Matthew E. Peters and Arman Cohan.

If you want to cite this model you can use this:


# longformer-base-4096-spanish ## Longformer is a Transformer model for long documents. 

'longformer-base-4096' is a BERT-like model started from the RoBERTa checkpoint (BERTIN in this case) and pre-trained for *MLM* on long documents (from BETO's 'all_wikis'). It supports sequences of length up to 4,096! 

 

Longformer uses a combination of a sliding window (*local*) attention and *global* attention. Global attention is user-configured based on the task to allow the model to learn task-specific representations.


This model was made following the research done by Iz Beltagy and Matthew E. Peters and Arman Cohan.

If you want to cite this model you can use this:
TAGS
#transformers #pytorch #tensorboard #roberta #fill-mask #Long documents #longformer #bertin #spanish #es #dataset-spanish_large_corpus #arxiv-2004.05150 #license-mit #autotrain_compatible #endpoints_compatible #has_space #region-us 
 # longformer-base-4096-spanish ## Longformer is a Transformer model for long documents. 

'longformer-base-4096' is a BERT-like model started from the RoBERTa checkpoint (BERTIN in this case) and pre-trained for *MLM* on long documents (from BETO's 'all_wikis'). It supports sequences of length up to 4,096! 

 

Longformer uses a combination of a sliding window (*local*) attention and *global* attention. Global attention is user-configured based on the task to allow the model to learn task-specific representations.


This model was made following the research done by Iz Beltagy and Matthew E. Peters and Arman Cohan.

If you want to cite this model you can use this:
passage: TAGS
#transformers #pytorch #tensorboard #roberta #fill-mask #Long documents #longformer #bertin #spanish #es #dataset-spanish_large_corpus #arxiv-2004.05150 #license-mit #autotrain_compatible #endpoints_compatible #has_space #region-us 
# longformer-base-4096-spanish## Longformer is a Transformer model for long documents. 

'longformer-base-4096' is a BERT-like model started from the RoBERTa checkpoint (BERTIN in this case) and pre-trained for *MLM* on long documents (from BETO's 'all_wikis'). It supports sequences of length up to 4,096! 

 

Longformer uses a combination of a sliding window (*local*) attention and *global* attention. Global attention is user-configured based on the task to allow the model to learn task-specific representations.


This model was made following the research done by Iz Beltagy and Matthew E. Peters and Arman Cohan.

If you want to cite this model you can use this: