TRANSFORMERS: Asking to pad but the tokenizer does not have a padding token
In trying to evaluate a bunch of transformers models sequentially with the same dataset to check which one performs better.
The list of models is this one:
MODELS = [
      ('xlm-mlm-enfr-1024'   ,&quot;XLMModel&quot;),
      ('distilbert-base-cased', &quot;DistilBertModel&quot;),
      ('bert-base-uncased'     ,&quot;BertModel&quot;),
      ('roberta-base'        ,&quot;RobertaModel&quot;),
      (&quot;cardiffnlp/twitter-roberta-base-sentiment&quot;,&quot;RobertaSentTW&quot;),
      ('xlnet-base-cased'     ,&quot;XLNetModel&quot;),
      #('ctrl'                ,&quot;CTRLModel&quot;),
      ('transfo-xl-wt103'    ,&quot;TransfoXLModel&quot;),
      ('bert-base-cased'       ,&quot;BertModelUncased&quot;),
      ('xlm-roberta-base'     ,&quot;XLMRobertaModel&quot;),
      ('openai-gpt'           ,&quot;OpenAIGPTModel&quot;),
      ('gpt2'                 ,&quot;GPT2Model&quot;)

All of them work fine until 'ctrl' model, which returns this error:
Asking to pad but the tokenizer does not have a padding token. Please select a token to use as 'pad_token' '(tokenizer.pad_token = tokenizer.eos_token e.g.)' or add a new pad token via 'tokenizer.add_special_tokens({'pad_token': '[PAD]'})'.
When tokenizing the sentences of my dataset.
The tokenizing code is
SEQ_LEN = MAX_LEN #(50)

for pretrained_weights, model_name in MODELS:

print(&quot;***************** INICIANDO &quot; ,model_name,&quot;, weights &quot;,pretrained_weights, &quot;********* &quot;)
print(&quot;carganzo el tokenizador ()&quot;)
tokenizer = AutoTokenizer.from_pretrained(pretrained_weights)
print(&quot;creando el modelo preentrenado&quot;)
transformer_model = TFAutoModel.from_pretrained(pretrained_weights)
print(&quot;aplicando el tokenizador al dataset&quot;)

##APLICAMOS EL TOKENIZADOR##

def tokenize(sentence):
  
  tokens = tokenizer.encode_plus(sentence, max_length=MAX_LEN,
                               truncation=True, padding='max_length',
                               add_special_tokens=True, return_attention_mask=True,
                               return_token_type_ids=False, return_tensors='tf')
  return tokens['input_ids'], tokens['attention_mask']

# initialize two arrays for input tensors
Xids = np.zeros((len(df), SEQ_LEN))
Xmask = np.zeros((len(df), SEQ_LEN))

for i, sentence in enumerate(df['tweet']):
    Xids[i, :], Xmask[i, :] = tokenize(sentence)
    if i % 10000 == 0:
        print(i)  # do this so we can see some progress


arr = df['label'].values  # take label column in df as array

labels = np.zeros((arr.size, arr.max()+1))  # initialize empty (all zero) label array
labels[np.arange(arr.size), arr] = 1  # add ones in indices where we have a value`

I have tried to define the padding tokens as the solution tells me, but then this error appears
could not broadcast input array from shape (3,) into shape (50,)

in line
Xids[i, :], Xmask[i, :] = tokenize(sentence)

I have also tried this solution and doesn't work neither.
If you have managed to read until here, thank you.
Any help is needed.

You can add the [PAD] token using add_special_tokens API.
tokenizer = AutoTokenizer.from_pretrained(pretrained_weights)
if tokenizer.pad_token is None:
    tokenizer.add_special_tokens({'pad_token': '[PAD]'})

