
## Introduction

[Allenai's Longformer Encoder-Decoder (LED)](https://github.com/allenai/longformer#longformer).

As described in [Longformer: The Long-Document Transformer](https://arxiv.org/pdf/2004.05150.pdf) by Iz Beltagy, Matthew E. Peters, Arman Cohan, *led-base-16384* was initialized from [*bart-base*](https://huggingface.co/facebook/bart-base) since both models share the exact same architecture. To be able to process 16K tokens, *bart-base*'s position embedding matrix was simply copied 16 times.

This model is especially interesting for long-range summarization and question answering.

## Fine-tuning for down-stream task

[This notebook](https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v?usp=sharing) shows how *led-base-16384* can effectively be fine-tuned on a downstream task.

{"language": "en", "license": "apache-2.0"}
transformers pytorch tf led text2text-generation en arxiv:2004.05150 license:apache-2.0 autotrain_compatible endpoints_compatible has_space region:us
TAGS
#transformers #pytorch #tf #led #text2text-generation #en #arxiv-2004.05150 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #has_space #region-us 


## Introduction

Allenai's Longformer Encoder-Decoder (LED).

As described in Longformer: The Long-Document Transformer by Iz Beltagy, Matthew E. Peters, Arman Cohan, *led-base-16384* was initialized from *bart-base* since both models share the exact same architecture. To be able to process 16K tokens, *bart-base*'s position embedding matrix was simply copied 16 times.

This model is especially interesting for long-range summarization and question answering.

## Fine-tuning for down-stream task

This notebook shows how *led-base-16384* can effectively be fine-tuned on a downstream task.

## Introduction

Allenai's Longformer Encoder-Decoder (LED).

As described in Longformer: The Long-Document Transformer by Iz Beltagy, Matthew E. Peters, Arman Cohan, *led-base-16384* was initialized from *bart-base* since both models share the exact same architecture. To be able to process 16K tokens, *bart-base*'s position embedding matrix was simply copied 16 times.

This model is especially interesting for long-range summarization and question answering. ## Fine-tuning for down-stream task

This notebook shows how *led-base-16384* can effectively be fine-tuned on a downstream task.
TAGS
#transformers #pytorch #tf #led #text2text-generation #en #arxiv-2004.05150 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #has_space #region-us 
 ## Introduction

Allenai's Longformer Encoder-Decoder (LED).

As described in Longformer: The Long-Document Transformer by Iz Beltagy, Matthew E. Peters, Arman Cohan, *led-base-16384* was initialized from *bart-base* since both models share the exact same architecture. To be able to process 16K tokens, *bart-base*'s position embedding matrix was simply copied 16 times.

This model is especially interesting for long-range summarization and question answering. ## Fine-tuning for down-stream task

This notebook shows how *led-base-16384* can effectively be fine-tuned on a downstream task.
passage: TAGS
#transformers #pytorch #tf #led #text2text-generation #en #arxiv-2004.05150 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #has_space #region-us 
## Introduction

Allenai's Longformer Encoder-Decoder (LED).

As described in Longformer: The Long-Document Transformer by Iz Beltagy, Matthew E. Peters, Arman Cohan, *led-base-16384* was initialized from *bart-base* since both models share the exact same architecture. To be able to process 16K tokens, *bart-base*'s position embedding matrix was simply copied 16 times.

This model is especially interesting for long-range summarization and question answering.## Fine-tuning for down-stream task

This notebook shows how *led-base-16384* can effectively be fine-tuned on a downstream task.