# SciBERT Longformer

This is a Lonformer version of the [SciBERT uncased](https://huggingface.co/allenai/scibert_scivocab_uncased) model by Allen AI. The model is slower than SciBERT (~2.5x in my benchmarks) but can allow for 8x wider `max_seq_length` (4096 vs. 512) which is handy in the case of working with long texts, e.g. scientific full texts. 

The conversion to Longformer was performed with a [tutorial](https://github.com/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb) by Allen AI: see a [Google Colab Notebook](https://colab.research.google.com/drive/1NPTnMkeAYOF2MWH3_uJYesuxxdOzxrFn?usp=sharing) by [Yury](https://yorko.github.io/) which closely follows the tutorial. 

Note: 

- no additional MLM pretraining of the Longformer was performed, the [collab notebook](https://colab.research.google.com/drive/1NPTnMkeAYOF2MWH3_uJYesuxxdOzxrFn?usp=sharing) stops at step 3, and step 4 is not done. The model can be improved with this additional MLM pretraining, better to do so with scientific texts, e.g. [S2ORC](https://github.com/allenai/s2orc), again by Allen AI. 
- no extensive benchmarks of SciBERT Longformer vs. SciBERT were performed in terms of downstream task performance

Links:
 - the original [SciBERT repo](https://github.com/allenai/scibert)
 - the original [Longformer repo](https://github.com/allenai/longformer)


If using these models, please consider citing the following papers:
```
@inproceedings{beltagy-etal-2019-scibert,
    title = "SciBERT: A Pretrained Language Model for Scientific Text",
    author = "Beltagy, Iz  and Lo, Kyle  and Cohan, Arman",
    booktitle = "EMNLP",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/D19-1371"
}

@article{Beltagy2020Longformer,
  title={Longformer: The Long-Document Transformer},
  author={Iz Beltagy and Matthew E. Peters and Arman Cohan},
  journal={arXiv:2004.05150},
  year={2020},
}
```
{"language": ["en"], "license": "mit"}
transformers pytorch safetensors bert fill-mask en license:mit autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #safetensors #bert #fill-mask #en #license-mit #autotrain_compatible #endpoints_compatible #region-us 

# SciBERT Longformer

This is a Lonformer version of the SciBERT uncased model by Allen AI. The model is slower than SciBERT (~2.5x in my benchmarks) but can allow for 8x wider 'max_seq_length' (4096 vs. 512) which is handy in the case of working with long texts, e.g. scientific full texts. 

The conversion to Longformer was performed with a tutorial by Allen AI: see a Google Colab Notebook by Yury which closely follows the tutorial. 

Note: 

- no additional MLM pretraining of the Longformer was performed, the collab notebook stops at step 3, and step 4 is not done. The model can be improved with this additional MLM pretraining, better to do so with scientific texts, e.g. S2ORC, again by Allen AI. 
- no extensive benchmarks of SciBERT Longformer vs. SciBERT were performed in terms of downstream task performance

Links:
 - the original SciBERT repo
 - the original Longformer repo


If using these models, please consider citing the following papers:

# SciBERT Longformer

This is a Lonformer version of the SciBERT uncased model by Allen AI. The model is slower than SciBERT (~2.5x in my benchmarks) but can allow for 8x wider 'max_seq_length' (4096 vs. 512) which is handy in the case of working with long texts, e.g. scientific full texts. 

The conversion to Longformer was performed with a tutorial by Allen AI: see a Google Colab Notebook by Yury which closely follows the tutorial. 

Note: 

- no additional MLM pretraining of the Longformer was performed, the collab notebook stops at step 3, and step 4 is not done. The model can be improved with this additional MLM pretraining, better to do so with scientific texts, e.g. S2ORC, again by Allen AI. 
- no extensive benchmarks of SciBERT Longformer vs. SciBERT were performed in terms of downstream task performance

Links:
 - the original SciBERT repo
 - the original Longformer repo


If using these models, please consider citing the following papers:
TAGS
#transformers #pytorch #safetensors #bert #fill-mask #en #license-mit #autotrain_compatible #endpoints_compatible #region-us 
 # SciBERT Longformer

This is a Lonformer version of the SciBERT uncased model by Allen AI. The model is slower than SciBERT (~2.5x in my benchmarks) but can allow for 8x wider 'max_seq_length' (4096 vs. 512) which is handy in the case of working with long texts, e.g. scientific full texts. 

The conversion to Longformer was performed with a tutorial by Allen AI: see a Google Colab Notebook by Yury which closely follows the tutorial. 

Note: 

- no additional MLM pretraining of the Longformer was performed, the collab notebook stops at step 3, and step 4 is not done. The model can be improved with this additional MLM pretraining, better to do so with scientific texts, e.g. S2ORC, again by Allen AI. 
- no extensive benchmarks of SciBERT Longformer vs. SciBERT were performed in terms of downstream task performance

Links:
 - the original SciBERT repo
 - the original Longformer repo


If using these models, please consider citing the following papers:
passage: TAGS
#transformers #pytorch #safetensors #bert #fill-mask #en #license-mit #autotrain_compatible #endpoints_compatible #region-us 
# SciBERT Longformer

This is a Lonformer version of the SciBERT uncased model by Allen AI. The model is slower than SciBERT (~2.5x in my benchmarks) but can allow for 8x wider 'max_seq_length' (4096 vs. 512) which is handy in the case of working with long texts, e.g. scientific full texts. 

The conversion to Longformer was performed with a tutorial by Allen AI: see a Google Colab Notebook by Yury which closely follows the tutorial. 

Note: 

- no additional MLM pretraining of the Longformer was performed, the collab notebook stops at step 3, and step 4 is not done. The model can be improved with this additional MLM pretraining, better to do so with scientific texts, e.g. S2ORC, again by Allen AI. 
- no extensive benchmarks of SciBERT Longformer vs. SciBERT were performed in terms of downstream task performance

Links:
 - the original SciBERT repo
 - the original Longformer repo


If using these models, please consider citing the following papers: