title: padding and attention mask does not work as intended in batch input in GPT language model
body: <p>The following code is without batch:</p>
  
  <pre><code>from transformers import GPT2LMHeadModel, GPT2Tokenizer
  import torch
  
  tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
  model = GPT2LMHeadModel.from_pretrained('gpt2')
  model.eval()
  context=torch.tensor([tokenizer.encode("This is")])
  output, past = model(context)
  token = torch.argmax(output[..., -1, :])
  print(tokenizer.decode(token.item()))
  
  output: ' a'
  </code></pre>
  
  <p>This is working fine. Now, I extended this to batch setting:</p>
  
  <pre><code>from transformers import GPT2LMHeadModel, GPT2Tokenizer
  import torch
  
  tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
  model = GPT2LMHeadModel.from_pretrained('gpt2')
  model.eval()
  
  context=[torch.tensor(tokenizer.encode("This is ")),torch.tensor(tokenizer.encode("Hello How are "))]
  context=pad_sequence(context,batch_first=True)
  
  mask=torch.tensor([[1,1,0],[1,1,1]])
  output, past = model(context,attention_mask=mask)
  token = torch.argmax(output[..., -1, :],dim=1)
  tokenizer.decode(token)
  
  output: '\n you'
  </code></pre>
  
  <p>Here <code>\n</code> is next token for the first context and <code>you</code> is next token for second context of the batch.
  But The expected next token for the first context is <code>a</code>, since all the settings are same. Furthermore, if you reduce the second context to 2 token you will get <code>a</code> in this batch setting. So clearly, model can not understand the padding.
  Also, the attention mask does not work. Because,
  after padding the next token of sequence <code>this is</code> is 0 (zero). And according to the attention mask (<code>[1,1,0]</code>), this zero should be avoided and only the tokens <code>this</code> and <code>is</code> should be attended. The proofs that this attention masking is not working are:</p>
  
  <ul>
  <li><p>Use attention mask [1,1,1], that means attend even on the padding zero, you get the same output
  which is <code>\n</code>.</p></li>
  <li><p>Use the the string <code>this is!</code>. Here <code>!</code> has the zero index in the vocabulary matrix. Again you get the same output which is <code>\n</code>.</p></li>
  </ul>
  
  <p>Only time, it is possible to get desirable output is without the batch settings and attention mask ( now it seems, it does not matter because it has no effect anyway)</p>
  
  <p>Then I found <a href="https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.pad_token" rel="nofollow noreferrer">this</a>, which suggested to use pad_token. So I used like following:</p>
  
  <pre><code>from transformers import GPT2LMHeadModel, GPT2Tokenizer
  import torch
  from torch.nn.utils.rnn import pad_sequence  
  
  tokenizer = GPT2Tokenizer.from_pretrained("gpt2",pad_token="&lt;PAD&gt;")
  model = GPT2LMHeadModel.from_pretrained('gpt2')
  model.eval()
  
  context=[torch.tensor(tokenizer.encode("This is &lt;PAD&gt; ")),torch.tensor(tokenizer.encode("Hello How are"))]
  context=torch.stack(context)
  print(context)
  mask=torch.tensor([[1,1,0],[1,1,1]])
  
  output, past = model(context,attention_mask=mask)
  token = torch.argmax(output[..., -1, :],dim=1)
  tokenizer.decode(token)
  
  output: 'The you'
  </code></pre>
  
  <p>Here <code>The</code> is next token for the first context and <code>you</code> is next token for second context of the batch. This is also not working. Because <code>The</code> is not expected for the first context.</p>
  
  <p>How do I use variable length sequence in batch setting in gpt/gpt2 model?</p>