Huggingface Transformers ByteLevelBPETokenizer tokenizer not found
I'm trying to run through the (new) tutorial here: https://huggingface.co/blog/how-to-train, but hit an error trying to load the ByteLevelBPETokenizer. I started from an existing conda env and also tried with a totally fresh env, but both give the same error:

Exception has occurred: ImportError
cannot import name 'ByteLevelBPETokenizer' from 'tokenizers' (/home/james/anaconda3/envs/torch/lib/python3.7/site-packages/tokenizers/__init__.py)

Any thoughts as to what might be wrong?

I'm on Ubuntu 18.04, Python 3.7

Okay, turns out the transformers installer pulls an older version (0.0.11). So...

pip uninstall tokenizers
pip install tokenizers==0.4.2


...fixes it.
It does issues a warning: ERROR: transformers 2.4.1 has requirement tokenizers==0.0.11, but you'll have tokenizers 0.4.2 which is incompatible., but this can safely be ignored (this answer came from @julien-c at huggingface/tokenizers).
