mwitiderrick commited on
Commit
3409c51
·
1 Parent(s): 21a7271

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +61 -0
README.md ADDED
@@ -0,0 +1,61 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ tags:
3
+ - deepsparse
4
+ ---
5
+ ## Usage
6
+
7
+ ```python
8
+ from deepsparse import TextGeneration
9
+
10
+ prompt = "How to make banana bread?"
11
+ formatted_prompt = f"<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n"
12
+
13
+ model = TextGeneration(model="hf:nm-testing/TinyLlama-1.1B-Chat-v0.4-pruned50-quant")
14
+ print(model(formatted_prompt, max_new_tokens=300).generations[0].text)
15
+
16
+ """
17
+ Banana bread is a delicious and healthy recipe that is easy to make. Here is a recipe for banana bread:
18
+
19
+ Ingredients:
20
+
21
+ 1. 15 bananas
22
+ 2. 1 cup of mashed bananas
23
+ 3. 1 cup of milk
24
+ 4. 1 cup of eggs
25
+ 5. 1 cup of sugar
26
+ 6. 1 cup of flour
27
+ 7. 1 teaspoon of baking powder
28
+ 8. 1 teaspoon of baking sugar
29
+ 9. 1 cup of whole wheat flour
30
+ 10. 1 cup of whole wheat sugar
31
+ 11. 1 cup of whole wheat flour
32
+ 12. 1 cup of whole wheat flour
33
+ 13. 1 cup of whole wheat flour
34
+ 14. 1 cup of whole wheat flour
35
+ 15. 1 cup of whole wheat flour
36
+
37
+ Step 1: Preheat the oven to 325°F (164°C).
38
+
39
+ Step 2: Grease a 10- to 12-inch (25- to 35-centimeter) round baking pan.
40
+
41
+ Step 3: Add the ingredients for the batter.
42
+
43
+ Step 4: Pour the batter into the prepared pan.
44
+
45
+ Step 5
46
+ """
47
+ ```
48
+
49
+ ## One-shot and Export
50
+
51
+ ```
52
+ git clone https://github.com/neuralmagic/sparseml
53
+ pip install -e "sparseml[transformers]" "torch<2"
54
+ cd sparseml
55
+ git checkout update/onnx_export/duplicate
56
+ python src/sparseml/transformers/sparsification/obcq/obcq.py TinyLlama/TinyLlama-1.1B-Chat-v0.4 open_platypus --recipe recipe.yaml --save True
57
+ python src/sparseml/transformers/sparsification/obcq/export.py --task text-generation --model_path obcq_deployment
58
+ cp deployment/model.onnx deployment/model-orig.onnx
59
+ python onnx_kv_inject.py --input-file deployment/model-orig.onnx --output-file deployment/model.onnx
60
+ ```
61
+ ```