Ahmadzei's picture
added 3 more tables for large emb model
5fa1a76
Importantly, this means that the tokenization procedure has a direct impact on a model's perplexity which should always be taken into consideration when comparing different models.