Xenova

/

dbrx-instruct-tokenizer

like 5

It should be! We've tested it on the entire xnli dataset (all languages), and it produces the same result.

@Xenova Thank you. We made a base-tokenizer based on your code with slight modifications to included pad token and to tokens that the original tokenizer added dynamically but never added to encoder. https://huggingface.co/LnL-AI/dbrx-base-tokenizer