What is the meaning of the second output of Huggingface's Bert?
Using the vanilla configuration of base BERT model in the huggingface implementation, I get a tuple of length 2.

import torch

import transformers
from transformers import AutoModel,AutoTokenizer

bert_name="bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(bert_name)
BERT = AutoModel.from_pretrained(bert_name)

e=tokenizer.encode('I am hoping for the best', add_special_tokens=True)

q=BERT(torch.tensor([e]))

print (len(q)) #Output: 2


The first element is what I expect to receive - the 768 dimension embedding of each input token.

print (e) #Output : [101, 1045, 2572, 5327, 2005, 1996, 2190, 102] 
print (q[0].shape) #Output : torch.Size([1, 8, 768])


But what is the second element in the tuple?

print (q[1].shape) # torch.Size([1, 768])


It has the same size as the encoding of each token.
But what is it?

Maybe a copy of the [CLS] token, a representation for the classification of the entire encoded text?

Let's check.

a= q[0][:,0,:]
b=q[1]

print (torch.eq(a,b)) #Output : Tensor([[False, False, False, .... False]])


Nope!

What about a copy the embedding of the last token (for whatever reason)?

c= q[0][:,-1,:]
b=q[1]

print (torch.eq(a,c)) #Output : Tensor([[False, False, False, .... False]])


So, also not that.

The documentation talks about how changing the config can result in more tuple elements (like hidden states), but I did not find any description of this "mysterious" tuple element outputted by the default configuration.

Any ideas as to what is it and what is its usage?    

The output in this case is a tuple of (last_hidden_state, pooler_output). You can find documentation about what the returns could be here.
