How do I access this model in google colab?
I am trying to use mistral. I am using the examples mentioned in the documentation. I don't know what a gated model is and I looked up in the net and it told me to get permission and I did that. I also did the hugging face_hub login thing.
and did the login(token='hf...')
now to the example part
I got 4 types of error back. HTTP Error, HfHubHTTP error, OS error and Local Entry Not Found Error. The final statement mentioned in the error states that
I have used BERT before via hugging face and I didn't face this issue then. Why is it different now? What am I missing? Please enlighten me.
nevermind,. i cant read.
Odd. i cant delete a post?
@Nurb4000 I didn't get you?
I got but still get access error @VikramanHF
I HOPE THIS MAY WORK
!pip install transformers datasets trl peft bitsandbytes accelerate
from kaggle_secrets import UserSecretsClient
from huggingface_hub import login
user_secrets = UserSecretsClient()
secret_value_0 = user_secrets.get_secret("HUGGINGFACE_API")
login(secret_value_0)
!huggingface-cli whoami
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
model_name = "mistralai/Mistral-7B-Instruct-v0.3"
model_name = "/kaggle/working/mistral_sdr_finetune"
Define quantization config
quantization_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_compute_dtype=torch.float16, # Use float16 for better performance
bnb_4bit_quant_type="nf4", # Normal Float 4 (nf4) is optimized for LLMs
bnb_4bit_use_double_quant=True # Further reduces memory footprint
)
Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name, token=True)
tokenizer.pad_token = tokenizer.eos_token
Load model with quantization
model = AutoModelForCausalLM.from_pretrained(
model_name,
token=True,
quantization_config=quantization_config,
device_map="auto" # Automatically distributes model across available GPUs/CPUs
)
print("Model loaded successfully!")