title: Chunked tokenization in huggingface has an arrow error
body: <p>I'm following the code from <a href="https://youtu.be/8PmhEIXhBvI?t=86" rel="nofollow noreferrer">this video</a> at 1m25s, which shows:</p>
  <pre><code>def tokenize_and_chunk(texts):
    return tokenizer(
      texts[&quot;text&quot;], truncation=True, max_length=context_length,
      return overflowing_tokens=True
    )
  
  tokenized_datasets = raw_datasets.map(
    tokenize_and_chunk, batched=True, remove_columns=[&quot;text&quot;]
  )
  </code></pre>
  <p>Here's the error I get when I try to run this code:</p>
  <pre><code>model_name = &quot;t5-base&quot;
  tokenizer = AutoTokenizer.from_pretrained(model_name)
  model = AutoModel.from_pretrained(model_name)
  
  context_length = 1000
  
  def tokenize_and_chunk(texts):
      return tokenizer(
        texts[&quot;text&quot;], truncation=True, max_length=context_length,
        return_overflowing_tokens=True,
    )
  
  dataset = Dataset.from_pandas(pd.DataFrame([{&quot;id&quot;: &quot;123&quot;, &quot;text&quot;: &quot;Here are many words! &quot;*5000}]))  
  </code></pre>
  <p>Shows a fine data set:</p>
  <pre><code>Dataset({
      features: ['id', 'text'],
      num_rows: 1
  })
  </code></pre>
  <p>Ok,let's run the tokenizer:</p>
  <pre><code>toknized_datasets = dataset.map(tokenize_and_chunk, batched=True, remove_columns=[&quot;text&quot;])
  
   0%
  0/1 [00:00&lt;?, ?ba/s]
  
  ---------------------------------------------------------------------------
  ArrowInvalid                              Traceback (most recent call last)
  &lt;ipython-input-69-d1216744e2ab&gt; in &lt;module&gt;
  ----&gt; 1 toknized_datasets = dataset.map(tokenize_and_chunk, batched=True, remove_columns=[&quot;text&quot;])
  
  ArrowInvalid: Column 1 named id expected length 5 but got length 1
  </code></pre>