# Skimformer

A collaboration between [reciTAL](https://recital.ai/en/) & [MLIA](https://mlia.lip6.fr/) (ISIR, Sorbonne Université)

## Model description
Skimformer is a two-stage Transformer that replaces self-attention with Skim-Attention, a self-attention module that computes attention solely based on the 2D positions of tokens in the page. The model adopts a two-step approach: first, the skim-attention scores are computed once and only once using layout information alone; then, these attentions are used in every layer of a text-based Transformer encoder. For more details, please refer to our paper: 

[Skim-Attention: Learning to Focus via Document Layout](https://arxiv.org/abs/2109.01078)
Laura Nguyen, Thomas Scialom, Jacopo Staiano, Benjamin Piwowarski, [EMNLP 2021](https://2021.emnlp.org/papers)

## Citation

``` latex
@article{nguyen2021skimattention,
    title={Skim-Attention: Learning to Focus via Document Layout}, 
    author={Laura Nguyen and Thomas Scialom and Jacopo Staiano and Benjamin Piwowarski},
    journal={arXiv preprint arXiv:2109.01078}
    year={2021},
}
```
{"license": "apache-2.0"}
transformers pytorch skimformer fill-mask arxiv:2109.01078 license:apache-2.0 autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #skimformer #fill-mask #arxiv-2109.01078 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #region-us 

# Skimformer

A collaboration between reciTAL & MLIA (ISIR, Sorbonne Université)

## Model description
Skimformer is a two-stage Transformer that replaces self-attention with Skim-Attention, a self-attention module that computes attention solely based on the 2D positions of tokens in the page. The model adopts a two-step approach: first, the skim-attention scores are computed once and only once using layout information alone; then, these attentions are used in every layer of a text-based Transformer encoder. For more details, please refer to our paper: 

Skim-Attention: Learning to Focus via Document Layout
Laura Nguyen, Thomas Scialom, Jacopo Staiano, Benjamin Piwowarski, EMNLP 2021


# Skimformer

A collaboration between reciTAL & MLIA (ISIR, Sorbonne Université)
TAGS
#transformers #pytorch #skimformer #fill-mask #arxiv-2109.01078 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #region-us 
 # Skimformer

A collaboration between reciTAL & MLIA (ISIR, Sorbonne Université)
passage: TAGS
#transformers #pytorch #skimformer #fill-mask #arxiv-2109.01078 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #region-us 
# Skimformer

A collaboration between reciTAL & MLIA (ISIR, Sorbonne Université)