File size: 2,427 Bytes
fc786e8 9bd78e5 eddf6bf fc786e8 eddf6bf fc786e8 2f49d44 a730b8d 2f49d44 fc786e8 dd400dc fc786e8 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 |
---
tags:
- deepsparse
---
## Usage
```python
from deepsparse import TextGeneration
prompt = "How to get in a good university?"
formatted_prompt = f"<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n"
model = TextGeneration(model_path="hf:nm-testing/TinyLlama-1.1B-Chat-v0.4-pruned60-quant")
print(model(formatted_prompt, max_new_tokens=200).generations[0].text)
"""
There are several factors to consider when choosing a university:
1. Location: The university should be located in a region with a high number of students. This will ensure that there are enough students to ensure that there are enough professors.
2. Tuition: The tuition of the university should be low. This will ensure that students have enough money to attend the university.
3. Academic: The university should have a good academic program. This will ensure that students have knowledge of the subject.
4. Faculty: The faculty of the university should be good. This will ensure that professors have knowledge of the subject.
5. Faculty: The faculty of the university should be good. This will ensure that professors have knowledge of the subject.
6. Faculty: The faculty of the university should be good. This will ensure that professors have knowledge of the subject.
"""
```
## With Repetition Penalty
```python
from deepsparse import TextGeneration
generation_config = {
"repetition_penalty": 1.1,
"do_sample": True,
"max_new_tokens": 500,
}
prompt = "How to get in a good university?"
formatted_prompt = f"<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n"
model = TextGeneration(model="hf:nm-testing/TinyLlama-1.1B-Chat-v0.4-pruned60-quant")
print(model(formatted_prompt, generation_config=generation_config,).generations[0].text)
"""
The university is one of the best options for students.
It provides the right atmosphere for studying.
The
""""
```
## One-shot and Export
```bash
git clone https://github.com/neuralmagic/sparseml
pip install -e "sparseml[transformers]"
python sparseml/src/sparseml/transformers/sparsification/obcq/obcq.py TinyLlama/TinyLlama-1.1B-Chat-v0.4 open_platypus --recipe recipe.yaml --save True
python sparseml/src/sparseml/transformers/sparsification/obcq/export.py --task text-generation --model_path obcq_deployment
cp deployment/model.onnx deployment/model-orig.onnx
python onnx_kv_inject.py --input-file deployment/model-orig.onnx --output-file deployment/model.onnx
``` |