Ahmadzei's picture
added 3 more tables for large emb model
5fa1a76
Use tokenizers from 🤗 Tokenizers
The [PreTrainedTokenizerFast] depends on the 🤗 Tokenizers library.