BERT embeddings in batches
I am following this post to extract embeddings for sentences and for a single sentence the steps are described as follows:
    text = &quot;After stealing money from the bank vault, the bank robber was seen &quot; \
           &quot;fishing on the Mississippi river bank.&quot;
    
    # Add the special tokens. 
    marked_text = &quot;[CLS] &quot; + text + &quot; [SEP]&quot;
    
    # Split the sentence into tokens. 
    tokenized_text = tokenizer.tokenize(marked_text)
    
    # Mark each of the 22 tokens as belonging to sentence &quot;1&quot;. 
    segments_ids = [1] * len(tokenized_text)
    
    # Convert inputs to PyTorch tensors 
    tokens_tensor = torch.tensor([indexed_tokens]) 
    segments_tensors = torch.tensor([segments_ids])
    
    # Load pre-trained model (weights) 
    model = BertModel.from_pretrained('bert-base-uncased',
                                 output_hidden_states = True,
                                      )
    
    # Put the model in &quot;evaluation&quot; mode, meaning feed-forward operation. 
    model.eval()
    
    with torch.no_grad():
        outputs = model(tokens_tensor, segments_tensors)
        hidden_states = outputs[2]

And I want to do this for a batch of sequences. Here is my example code:
seql = ['this is an example', 'today was sunny and', 'today was']
encoded = [tokenizer.encode(seq, max_length=5, pad_to_max_length=True) for seq in seql]

encoded
[[2, 2511, 1840, 3251, 3], 
 [2, 1663, 2541, 1957, 3], 
 [2, 1663, 2541, 3, 0]]

But since I'm working with batches, sequences need to have same length. So I introduce a padding token (3rd sentence) which confuses me about several points:

What should the segment id for pad_token (0) will be?

Should I use attention masking when feeding the tensors to the model so that padding is ignored? In the example only token and segment tensors are used.
outputs = model(tokens_tensor, segments_tensors)


If I don't work with batches but with individual sentences, then I might not need a padding token. Would it be better to do that compared to batches?



You could do all the work you need using one function ( padding,truncation)

encode_plus

check the parameters: the docs
The same you could do with a list of sequences

batch_encode_plus

docs
