File size: 1,291 Bytes
5d510b4 ba6cc08 507bf37 ba6cc08 5d510b4 e356521 5d510b4 2ee5da2 5d510b4 a608ae8 5d510b4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
---
tags:
- deepsparse
---
## Usage
```python
from deepsparse import TextGeneration
prompt = "How to get in a good university?"
formatted_prompt = f"<s> [|User|]\n{prompt}</s>[|Assistant|]\n"
model = TextGeneration(model="hf:nm-testing/MiniChat-3B-pruned-quant-50")
print(model(formatted_prompt, max_new_tokens=500).generations[0].text)
"""
To get into a good university, you should focus on your academic performance and strive to achieve high grades.
This can be done by setting realistic goals and targets, regularly reviewing your progress, and seeking help from teachers or tutors if needed.
Additionally, participating in extracurricular activities and building a network of friends can also help in getting into a good university.
"""
```
## One-shot and Export
```bash
git clone https://github.com/neuralmagic/sparseml
pip install -e "sparseml[transformers]"
python sparseml/src/sparseml/transformers/sparsification/obcq/obcq.py GeneZC/MiniChat-3B open_platypus --recipe recipe.yaml --save True
python sparseml/src/sparseml/transformers/sparsification/obcq/export.py --task text-generation --model_path obcq_deployment
cp deployment/model.onnx deployment/model-orig.onnx
python onnx_kv_inject.py --input-file deployment/model-orig.onnx --output-file deployment/model.onnx
``` |