metadata
tags:
- deepsparse
Usage
from deepsparse import TextGeneration
prompt = "How to get in a good university?"
formatted_prompt = f"<s> [|User|]\n{prompt}</s>[|Assistant|]\n"
model = TextGeneration(model="hf:nm-testing/MiniChat-3B-pruned-quant-50")
print(model(formatted_prompt, max_new_tokens=500).generations[0].text)
"""
To get into a good university, you should focus on your academic performance and strive to achieve high grades.
This can be done by setting realistic goals and targets, regularly reviewing your progress, and seeking help from teachers or tutors if needed.
Additionally, participating in extracurricular activities and building a network of friends can also help in getting into a good university.
"""
One-shot and Export
git clone https://github.com/neuralmagic/sparseml
pip install -e "sparseml[transformers]"
python sparseml/src/sparseml/transformers/sparsification/obcq/obcq.py GeneZC/MiniChat-3B open_platypus --recipe recipe.yaml --save True
python sparseml/src/sparseml/transformers/sparsification/obcq/export.py --task text-generation --model_path obcq_deployment
cp deployment/model.onnx deployment/model-orig.onnx
python onnx_kv_inject.py --input-file deployment/model-orig.onnx --output-file deployment/model.onnx