|
--- |
|
tags: |
|
- deepsparse |
|
--- |
|
## Usage |
|
|
|
```python |
|
from deepsparse import TextGeneration |
|
|
|
prompt = "How to get in a good university?" |
|
formatted_prompt = f"<s> [|User|]\n{prompt}</s>[|Assistant|]\n" |
|
|
|
model = TextGeneration(model="hf:nm-testing/MiniChat-3B-pruned-quant-50") |
|
|
|
print(model(formatted_prompt, max_new_tokens=500).generations[0].text) |
|
""" |
|
To get into a good university, you should focus on your academic performance and strive to achieve high grades. |
|
This can be done by setting realistic goals and targets, regularly reviewing your progress, and seeking help from teachers or tutors if needed. |
|
Additionally, participating in extracurricular activities and building a network of friends can also help in getting into a good university. |
|
""" |
|
``` |
|
|
|
## One-shot and Export |
|
|
|
```bash |
|
git clone https://github.com/neuralmagic/sparseml |
|
pip install -e "sparseml[transformers]" |
|
python sparseml/src/sparseml/transformers/sparsification/obcq/obcq.py GeneZC/MiniChat-3B open_platypus --recipe recipe.yaml --save True |
|
python sparseml/src/sparseml/transformers/sparsification/obcq/export.py --task text-generation --model_path obcq_deployment |
|
cp deployment/model.onnx deployment/model-orig.onnx |
|
python onnx_kv_inject.py --input-file deployment/model-orig.onnx --output-file deployment/model.onnx |
|
``` |
