How are contents of hidden_states tuple in BertModel in the transformers library arranged
model = BertModel.from_pretrained('bert-base-uncased', config=BertConfig.from_pretrained('bert-base-uncased',output_hidden_states=True))
outputs = model(input_ids) 
hidden_states = outputs[2]


hidden_states is a tuple of 13 torch.FloatTensors. Each tensor is of size: (batch_size, sequence_length, hidden_size).
According to the documentation, the 13 tensors are the hidden states of the embedding and the 12 encoder layers. 

My question: 

Is hidden_states[0] the embedding layer while hidden_states[12] is the 12th encoder layer or

Is hidden_states[0] the embedding layer while hidden_states[12] is the 1st encoder layer or

Is hidden_states[0] the 12th encoder layer while hidden_states[12] is the embedding layer or

Is hidden_states[0] the 1st encoder layer while hidden_states[12] is the embedding layer 

I havent found this found clearly stated anywhere else.

Looking at the source-code for BertModel, it can be concluded that hidden_states[0] contains the outputs of the initial embedding layer, and the rest of the elements in tuples contain the hidden states in the increasing order of each layer. Simply put, hidden_states[1] contains the outputs of the first layer of BERT and hidden_states[12] contains the last i.e. 12th layer.
