title: How to extract document embeddings from HuggingFace Longformer
body: <p>Looking to do something similar to</p>
  <pre class="lang-py prettyprint-override"><code>tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
  model = BertModel.from_pretrained('bert-base-uncased')
  input_ids = torch.tensor(tokenizer.encode(&quot;Hello, my dog is cute&quot;)).unsqueeze(0)  # Batch size 1
  outputs = model(input_ids)
  last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple
  </code></pre>
  <p>(from <a href="https://github.com/huggingface/transformers/issues/1950" rel="noreferrer">this thread</a>)
  using the longformer</p>
  <p>the documentation example seems to do something similar, but is confusing (esp. wrt. how to set the attention mask, I assume I'd want to set it to the <code>[CLS]</code> token, the example sets global attention to random values I think)</p>
  <pre class="lang-py prettyprint-override"><code>&gt;&gt;&gt; import torch
  &gt;&gt;&gt; from transformers import LongformerModel, LongformerTokenizer
  
  &gt;&gt;&gt; model = LongformerModel.from_pretrained('allenai/longformer-base-4096', return_dict=True)
  &gt;&gt;&gt; tokenizer = LongformerTokenizer.from_pretrained('allenai/longformer-base-4096')
  
  &gt;&gt;&gt; SAMPLE_TEXT = ' '.join(['Hello world! '] * 1000)  # long input document
  &gt;&gt;&gt; input_ids = torch.tensor(tokenizer.encode(SAMPLE_TEXT)).unsqueeze(0)  # batch of size 1
  
  &gt;&gt;&gt; # Attention mask values -- 0: no attention, 1: local attention, 2: global attention
  &gt;&gt;&gt; attention_mask = torch.ones(input_ids.shape, dtype=torch.long, device=input_ids.device) # initialize to local attention
  &gt;&gt;&gt; attention_mask[:, [1, 4, 21,]] = 2  # Set global attention based on the task. For example,
  ...                                     # classification: the &lt;s&gt; token
  ...                                     # QA: question tokens
  ...                                     # LM: potentially on the beginning of sentences and paragraphs
  &gt;&gt;&gt; outputs = model(input_ids, attention_mask=attention_mask)
  &gt;&gt;&gt; sequence_output = outputs.last_hidden_state
  &gt;&gt;&gt; pooled_output = outputs.pooler_output
  </code></pre>
  <p>(from <a href="https://huggingface.co/transformers/model_doc/longformer.html" rel="noreferrer">here</a>)</p>