Commit
·
3409c51
1
Parent(s):
21a7271
Create README.md
Browse files
README.md
ADDED
@@ -0,0 +1,61 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
tags:
|
3 |
+
- deepsparse
|
4 |
+
---
|
5 |
+
## Usage
|
6 |
+
|
7 |
+
```python
|
8 |
+
from deepsparse import TextGeneration
|
9 |
+
|
10 |
+
prompt = "How to make banana bread?"
|
11 |
+
formatted_prompt = f"<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n"
|
12 |
+
|
13 |
+
model = TextGeneration(model="hf:nm-testing/TinyLlama-1.1B-Chat-v0.4-pruned50-quant")
|
14 |
+
print(model(formatted_prompt, max_new_tokens=300).generations[0].text)
|
15 |
+
|
16 |
+
"""
|
17 |
+
Banana bread is a delicious and healthy recipe that is easy to make. Here is a recipe for banana bread:
|
18 |
+
|
19 |
+
Ingredients:
|
20 |
+
|
21 |
+
1. 15 bananas
|
22 |
+
2. 1 cup of mashed bananas
|
23 |
+
3. 1 cup of milk
|
24 |
+
4. 1 cup of eggs
|
25 |
+
5. 1 cup of sugar
|
26 |
+
6. 1 cup of flour
|
27 |
+
7. 1 teaspoon of baking powder
|
28 |
+
8. 1 teaspoon of baking sugar
|
29 |
+
9. 1 cup of whole wheat flour
|
30 |
+
10. 1 cup of whole wheat sugar
|
31 |
+
11. 1 cup of whole wheat flour
|
32 |
+
12. 1 cup of whole wheat flour
|
33 |
+
13. 1 cup of whole wheat flour
|
34 |
+
14. 1 cup of whole wheat flour
|
35 |
+
15. 1 cup of whole wheat flour
|
36 |
+
|
37 |
+
Step 1: Preheat the oven to 325°F (164°C).
|
38 |
+
|
39 |
+
Step 2: Grease a 10- to 12-inch (25- to 35-centimeter) round baking pan.
|
40 |
+
|
41 |
+
Step 3: Add the ingredients for the batter.
|
42 |
+
|
43 |
+
Step 4: Pour the batter into the prepared pan.
|
44 |
+
|
45 |
+
Step 5
|
46 |
+
"""
|
47 |
+
```
|
48 |
+
|
49 |
+
## One-shot and Export
|
50 |
+
|
51 |
+
```
|
52 |
+
git clone https://github.com/neuralmagic/sparseml
|
53 |
+
pip install -e "sparseml[transformers]" "torch<2"
|
54 |
+
cd sparseml
|
55 |
+
git checkout update/onnx_export/duplicate
|
56 |
+
python src/sparseml/transformers/sparsification/obcq/obcq.py TinyLlama/TinyLlama-1.1B-Chat-v0.4 open_platypus --recipe recipe.yaml --save True
|
57 |
+
python src/sparseml/transformers/sparsification/obcq/export.py --task text-generation --model_path obcq_deployment
|
58 |
+
cp deployment/model.onnx deployment/model-orig.onnx
|
59 |
+
python onnx_kv_inject.py --input-file deployment/model-orig.onnx --output-file deployment/model.onnx
|
60 |
+
```
|
61 |
+
```
|