Rename original/tokenizer.model to tokenizer.model
#6
by
winglian
- opened
tokenizer.model should be in the root to work with Huggingface Trainer
Also needs a similar PR for 70B
No because it's not supported by the standard slow tokenizers (it's tiktoken based not sentencepiece based)
winglian
changed pull request status to
closed
where is the tokenizer.model ? I did not find the tokenizer.model in the Meta-Llama-3-8B.