

# Cross-Document Language Modeling

CDLM: Cross-Document Language Modeling. 
Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew E Peters, Arie Cattan and Ido Dagan. In EMNLP Findings, 2021. [PDF](https://arxiv.org/pdf/2101.00406.pdf)


Please note that during our pretraining we used the document and sentence separators, which you might want to add to your data. The document and sentence separators are `<doc-s>`, `</doc-s>` (the last two tokens in the vocabulary), and `<s>`, `</s>`, respectively.


```python
from transformers import AutoTokenizer, AutoModel
# load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained('biu-nlp/cdlm')
model = AutoModel.from_pretrained('biu-nlp/cdlm')
```

The original repo is [here](https://github.com/aviclu/CDLM).

If you find our work useful, please cite the paper as:

```python
@article{caciularu2021cross,
  title={Cross-Document Language Modeling},
  author={Caciularu, Avi and Cohan, Arman and Beltagy, Iz and Peters, Matthew E and Cattan, Arie and Dagan, Ido},
  journal={Findings of the Association for Computational Linguistics: EMNLP 2021},
  year={2021}
}
```
{"language": "en", "license": "apache-2.0", "tags": ["longformer", "cdlm"], "inference": false}
transformers pytorch longformer fill-mask cdlm en arxiv:2101.00406 license:apache-2.0 autotrain_compatible region:us
TAGS
#transformers #pytorch #longformer #fill-mask #cdlm #en #arxiv-2101.00406 #license-apache-2.0 #autotrain_compatible #region-us 



# Cross-Document Language Modeling

CDLM: Cross-Document Language Modeling. 
Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew E Peters, Arie Cattan and Ido Dagan. In EMNLP Findings, 2021. PDF


Please note that during our pretraining we used the document and sentence separators, which you might want to add to your data. The document and sentence separators are '<doc-s>', '</doc-s>' (the last two tokens in the vocabulary), and '<s>', '</s>', respectively.




The original repo is here.

If you find our work useful, please cite the paper as:


# Cross-Document Language Modeling

CDLM: Cross-Document Language Modeling. 
Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew E Peters, Arie Cattan and Ido Dagan. In EMNLP Findings, 2021. PDF


Please note that during our pretraining we used the document and sentence separators, which you might want to add to your data. The document and sentence separators are '<doc-s>', '</doc-s>' (the last two tokens in the vocabulary), and '<s>', '</s>', respectively.




The original repo is here.

If you find our work useful, please cite the paper as:
TAGS
#transformers #pytorch #longformer #fill-mask #cdlm #en #arxiv-2101.00406 #license-apache-2.0 #autotrain_compatible #region-us 
 # Cross-Document Language Modeling

CDLM: Cross-Document Language Modeling. 
Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew E Peters, Arie Cattan and Ido Dagan. In EMNLP Findings, 2021. PDF


Please note that during our pretraining we used the document and sentence separators, which you might want to add to your data. The document and sentence separators are '<doc-s>', '</doc-s>' (the last two tokens in the vocabulary), and '<s>', '</s>', respectively.




The original repo is here.

If you find our work useful, please cite the paper as:
passage: TAGS
#transformers #pytorch #longformer #fill-mask #cdlm #en #arxiv-2101.00406 #license-apache-2.0 #autotrain_compatible #region-us 
# Cross-Document Language Modeling

CDLM: Cross-Document Language Modeling. 
Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew E Peters, Arie Cattan and Ido Dagan. In EMNLP Findings, 2021. PDF


Please note that during our pretraining we used the document and sentence separators, which you might want to add to your data. The document and sentence separators are '<doc-s>', '</doc-s>' (the last two tokens in the vocabulary), and '<s>', '</s>', respectively.




The original repo is here.

If you find our work useful, please cite the paper as: