title: Huggingface document summarization for long documents
body: <p>I would expect summarization tasks to generally assume long documents.  However, following documentation <a href="https://huggingface.co/course/chapter1/3?fw=pt" rel="nofollow noreferrer">here</a>, any of the simple summarization invocations I make say my documents are too long:</p>
  <pre><code>&gt;&gt;&gt; summarizer = pipeline(&quot;summarization&quot;)
  &gt;&gt;&gt; summarizer(fulltext)
  Token indices sequence length is longer than the specified maximum sequence length for this model (5620 &gt; 1024). Running this sequence through the model will result in indexing errors
  
  &gt;&gt;&gt; summarizer = pipeline(&quot;summarization&quot;, model=&quot;facebook/bart-large-cnn&quot;)
  &gt;&gt;&gt; summary = summarizer(fulltext)
  Token indices sequence length is longer than the specified maximum sequence length for this model (8084 &gt; 1024). Running this sequence through the model will result in indexing errors
  
  &gt;&gt;&gt; summarizer = pipeline(&quot;summarization&quot;, model=&quot;t5-base&quot;, tokenizer=&quot;t5-base&quot;)
  &gt;&gt;&gt; summary = summarizer(fulltext)
  Token indices sequence length is longer than the specified maximum sequence length for this model (5971 &gt; 512). Running this sequence through the model will result in indexing errors
  
  </code></pre>
  <p>What model or configuration choice makes this most automatic?  I've read other questions <a href="https://github.com/huggingface/transformers/issues/1791" rel="nofollow noreferrer">suggesting manually chunking the data</a> or <a href="https://stackoverflow.com/questions/66954682/token-indices-sequence-length-is-longer-than-the-specified-maximum-sequence-leng">truncation</a>, but the choice of boundaries and chunk length seem like they would make a difference in summaries.  What's the best practice for an arbitrary long document?  (Unbounded would be great, but let's say 50,000 tokens at a minimum.)</p>