Huggingface error: AttributeError: 'ByteLevelBPETokenizer' object has no attribute 'pad_token_id'
I am trying to tokenize some numerical strings using a WordLevel/BPE tokenizer, create a data collator and eventually use it in a PyTorch DataLoader to train a new model from scratch.
However, I am getting an error

AttributeError: 'ByteLevelBPETokenizer' object has no attribute 'pad_token_id'

when running the following code
from transformers import DataCollatorForLanguageModeling
from tokenizers import ByteLevelBPETokenizer
from tokenizers.pre_tokenizers import Whitespace
from torch.utils.data import DataLoader, TensorDataset

data = ['4814 4832 4761 4523 4999 4860 4699 5024 4788 &lt;unk&gt;']

# Tokenizer
tokenizer = ByteLevelBPETokenizer()
tokenizer.pre_tokenizer = Whitespace()
tokenizer.train_from_iterator(data, vocab_size=1000, min_frequency=1, 
    special_tokens=[
        &quot;&lt;s&gt;&quot;,
        &quot;&lt;/s&gt;&quot;,
        &quot;&lt;unk&gt;&quot;,
        &quot;&lt;mask&gt;&quot;,
    ])

# Data Collator
data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=False
)

train_dataset = TensorDataset(torch.tensor(tokenizer(data, ......)))

# DataLoader
train_dataloader = DataLoader(
    train_dataset, 
    collate_fn=data_collator
)

Is this error due to not having configured the pad_token_id for the tokenizer? If so, how can we do this?
Thanks!
Error trace:
AttributeError: Caught AttributeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File &quot;/opt/anaconda3/envs/x/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py&quot;, line 198, in _worker_loop
    data = fetcher.fetch(index)
  File &quot;/opt/anaconda3/envs/x/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py&quot;, line 47, in fetch
    return self.collate_fn(data)
  File &quot;/opt/anaconda3/envs/x/lib/python3.8/site-packages/transformers/data/data_collator.py&quot;, line 351, in __call__
    if self.tokenizer.pad_token_id is not None:
AttributeError: 'ByteLevelBPETokenizer' object has no attribute 'pad_token_id'

Conda packages
pytorch                   1.7.0           py3.8_cuda10.2.89_cudnn7.6.5_0    pytorch
pytorch-lightning         1.2.5              pyhd8ed1ab_0    conda-forge
tokenizers                0.10.1                   pypi_0    pypi
transformers              4.4.2                    pypi_0    pypi


The error tells you that the tokenizer needs an attribute called pad_token_id. You can either wrap the ByteLevelBPETokenizer into a class with such an attribute (... and met other missing attributes down the road) or use the wrapper class from the transformers library:
from transformers import PreTrainedTokenizerFast

#your code
tokenizer.save(SOMEWHERE)
tokenizer = PreTrainedTokenizerFast(tokenizer_file=tokenizer_path)

