How to Suppress "Using bos_token, but it is not set yet..." in HuggingFace T5 Tokenizer
I'd like to turn off the warning that huggingface is generating when I use unique_no_split_tokens
In[2]   tokenizer = T5Tokenizer.from_pretrained(&quot;t5-base&quot;)
In[3]   tokenizer(&quot; &quot;.join([f&quot;&lt;extra_id_{n}&gt;&quot; for n in range(1,101)]), return_tensors=&quot;pt&quot;).input_ids.size()
Out[3]: torch.Size([1, 100])
    Using bos_token, but it is not set yet.
    Using cls_token, but it is not set yet.
    Using mask_token, but it is not set yet.
    Using sep_token, but it is not set yet.

Anyone know how to do this?

This solution worked for me:
tokenizer.add_tokens([f&quot;_{n}&quot; for n in range(1,100)], special_tokens=True)
model.resize_token_embeddings(len(tokenizer))
tokenizer.save_pretrained('pathToExtendedTokenizer/')
tokenizer = T5Tokenizer.from_pretrained(&quot;pathToExtendedTokenizer/&quot;)

