
# longformer-base-4096
[Longformer](https://arxiv.org/abs/2004.05150) is a transformer model for long documents. 

`longformer-base-4096` is a BERT-like model started from the RoBERTa checkpoint and pretrained for MLM on long documents. It supports sequences of length up to 4,096. 
 
Longformer uses a combination of a sliding window (local) attention and global attention. Global attention is user-configured based on the task to allow the model to learn task-specific representations.
Please refer to the examples in `modeling_longformer.py` and the paper for more details on how to set global attention.


### Citing

If you use `Longformer` in your research, please cite [Longformer: The Long-Document Transformer](https://arxiv.org/abs/2004.05150).
```
@article{Beltagy2020Longformer,
  title={Longformer: The Long-Document Transformer},
  author={Iz Beltagy and Matthew E. Peters and Arman Cohan},
  journal={arXiv:2004.05150},
  year={2020},
}
```

`Longformer` is an open-source project developed by [the Allen Institute for Artificial Intelligence (AI2)](http://www.allenai.org).
AI2 is a non-profit institute with the mission to contribute to humanity through high-impact AI research and engineering.
{"language": "en", "license": "apache-2.0"}
transformers pytorch tf rust longformer en arxiv:2004.05150 license:apache-2.0 endpoints_compatible has_space region:us
TAGS
#transformers #pytorch #tf #rust #longformer #en #arxiv-2004.05150 #license-apache-2.0 #endpoints_compatible #has_space #region-us 


# longformer-base-4096
Longformer is a transformer model for long documents. 

'longformer-base-4096' is a BERT-like model started from the RoBERTa checkpoint and pretrained for MLM on long documents. It supports sequences of length up to 4,096. 
 
Longformer uses a combination of a sliding window (local) attention and global attention. Global attention is user-configured based on the task to allow the model to learn task-specific representations.
Please refer to the examples in 'modeling_longformer.py' and the paper for more details on how to set global attention.


### Citing

If you use 'Longformer' in your research, please cite Longformer: The Long-Document Transformer.


'Longformer' is an open-source project developed by the Allen Institute for Artificial Intelligence (AI2).
AI2 is a non-profit institute with the mission to contribute to humanity through high-impact AI research and engineering.
# longformer-base-4096
Longformer is a transformer model for long documents. 

'longformer-base-4096' is a BERT-like model started from the RoBERTa checkpoint and pretrained for MLM on long documents. It supports sequences of length up to 4,096. 
 
Longformer uses a combination of a sliding window (local) attention and global attention. Global attention is user-configured based on the task to allow the model to learn task-specific representations.
Please refer to the examples in 'modeling_longformer.py' and the paper for more details on how to set global attention. ### Citing

If you use 'Longformer' in your research, please cite Longformer: The Long-Document Transformer.


'Longformer' is an open-source project developed by the Allen Institute for Artificial Intelligence (AI2).
AI2 is a non-profit institute with the mission to contribute to humanity through high-impact AI research and engineering.
TAGS
#transformers #pytorch #tf #rust #longformer #en #arxiv-2004.05150 #license-apache-2.0 #endpoints_compatible #has_space #region-us 
 # longformer-base-4096
Longformer is a transformer model for long documents. 

'longformer-base-4096' is a BERT-like model started from the RoBERTa checkpoint and pretrained for MLM on long documents. It supports sequences of length up to 4,096. 
 
Longformer uses a combination of a sliding window (local) attention and global attention. Global attention is user-configured based on the task to allow the model to learn task-specific representations.
Please refer to the examples in 'modeling_longformer.py' and the paper for more details on how to set global attention. ### Citing

If you use 'Longformer' in your research, please cite Longformer: The Long-Document Transformer.


'Longformer' is an open-source project developed by the Allen Institute for Artificial Intelligence (AI2).
AI2 is a non-profit institute with the mission to contribute to humanity through high-impact AI research and engineering.
passage: TAGS
#transformers #pytorch #tf #rust #longformer #en #arxiv-2004.05150 #license-apache-2.0 #endpoints_compatible #has_space #region-us 
# longformer-base-4096
Longformer is a transformer model for long documents. 

'longformer-base-4096' is a BERT-like model started from the RoBERTa checkpoint and pretrained for MLM on long documents. It supports sequences of length up to 4,096. 
 
Longformer uses a combination of a sliding window (local) attention and global attention. Global attention is user-configured based on the task to allow the model to learn task-specific representations.
Please refer to the examples in 'modeling_longformer.py' and the paper for more details on how to set global attention.### Citing

If you use 'Longformer' in your research, please cite Longformer: The Long-Document Transformer.


'Longformer' is an open-source project developed by the Allen Institute for Artificial Intelligence (AI2).
AI2 is a non-profit institute with the mission to contribute to humanity through high-impact AI research and engineering.