File size: 482 Bytes
5fa1a76 |
1 2 3 4 5 6 7 8 9 |
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
tokenizer.pad_token = tokenizer.eos_token # Most LLMs don't have a pad token by default
model = AutoModelForCausalLM.from_pretrained(
"mistralai/Mistral-7B-v0.1", device_map="auto", load_in_4bit=True
)
Generated output is too short/long
If not specified in the [~generation.GenerationConfig] file, generate returns up to 20 tokens by default. |