File size: 519 Bytes
5fa1a76 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
Loading from a JSON file
In order to load a tokenizer from a JSON file, let's first start by saving our tokenizer:
thon
tokenizer.save("tokenizer.json")
The path to which we saved this file can be passed to the [PreTrainedTokenizerFast] initialization
method using the tokenizer_file parameter:
thon
from transformers import PreTrainedTokenizerFast
fast_tokenizer = PreTrainedTokenizerFast(tokenizer_file="tokenizer.json")
This object can now be used with all the methods shared by the 🤗 Transformers tokenizers! |