YAML Metadata
Warning:
The pipeline tag "conversational" is not in the official list: text-classification, token-classification, table-question-answering, question-answering, zero-shot-classification, translation, summarization, feature-extraction, text-generation, text2text-generation, fill-mask, sentence-similarity, text-to-speech, text-to-audio, automatic-speech-recognition, audio-to-audio, audio-classification, audio-text-to-text, voice-activity-detection, depth-estimation, image-classification, object-detection, image-segmentation, text-to-image, image-to-text, image-to-image, image-to-video, unconditional-image-generation, video-classification, reinforcement-learning, robotics, tabular-classification, tabular-regression, tabular-to-text, table-to-text, multiple-choice, text-retrieval, time-series-forecasting, text-to-video, image-text-to-text, visual-question-answering, document-question-answering, zero-shot-image-classification, graph-ml, mask-generation, zero-shot-object-detection, text-to-3d, image-to-3d, image-feature-extraction, video-text-to-text, keypoint-detection, any-to-any, other
This is a chitchat qlora model for Gaivoronsky/ruGPT-3.5-13B-8bit
Examples of usage
from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM, get_gptq_peft_model
from auto_gptq.utils.peft_utils import GPTQLoraConfig
device = 'cuda:0'
model_name = 'Gaivoronsky/ruGPT-3.5-13B-8bit'
model_basename = 'gptq_model-8bit-128g'
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)
model = AutoGPTQForCausalLM.from_quantized(
'Gaivoronsky/ruGPT-3.5-13B-8bit',
model_basename='gptq_model-8bit-128g',
variant='bin',
trust_remote_code=True,
device=device,
use_triton=False,
quantize_config=None
)
peft_config = GPTQLoraConfig(
inference_mode=True,
)
model = get_gptq_peft_model(model, peft_config, 'sadzip/SiberianPersona-ruGPT-3.5-qlora')
prompt = """
Ты девушка Саша, художница. Увлекаешься нейросетевым искусством. Умеешь программировать. Любишь рисовать. Продолжи диалог:
Собеседник: Привет
Ты: Привет
Собеседник: Как зовут?
Ты:
""".strip()
encoded_input = tokenizer(prompt, return_tensors='pt').to(device)
output = model.generate(
**encoded_input,
max_new_tokens=100,
do_sample=True,
temperature=1,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Inference Providers
NEW
This model is not currently available via any of the supported Inference Providers.
The model cannot be deployed to the HF Inference API:
The model authors have turned it off explicitly.