File size: 3,469 Bytes
01b06a0 05f798b 01b06a0 05f798b 01b06a0 05f798b 01b06a0 05f798b 01b06a0 bf62f76 05f798b 01b06a0 05f798b 01b06a0 5d772d6 01b06a0 cbaeee2 01b06a0 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 |
---
base_model: openlm-research/open_llama_3b
inference: false
model_type: llama
prompt_template: |
Q:
{prompt}
\nA
quantized_by: mwitiderrick
tags:
- deepsparse
---
## OpenLLaMA 3B - DeepSparse
This repo contains model files for [OpenLLaMA](https://huggingface.co/openlm-research/open_llama_3b) optimized for [DeepSparse](https://github.com/neuralmagic/deepsparse), a CPU inference runtime for sparse models.
This model was quantized and pruned with [SparseGPT](https://arxiv.org/abs/2301.00774), using [SparseML](https://github.com/neuralmagic/sparseml).
## Inference
Install [DeepSparse LLM](https://github.com/neuralmagic/deepsparse) for fast inference on CPUs:
```bash
pip install deepsparse-nightly[llm]
```
Run in a [Python pipeline](https://github.com/neuralmagic/deepsparse/blob/main/docs/llms/text-generation-pipeline.md):
```python
from deepsparse import TextGeneration
prompt = 'Q: What is the largest animal?\nA:'
formatted_prompt = f"Q: {prompt}\nA:"
model = TextGeneration(model_path="hf:nm-testing/open_llama_3b-pruned50-quant-ds")
print(model(formatted_prompt, max_new_tokens=200).generations[0].text)
"""
the in the in in the in in in in in the in the the in in the in in the the the the the in the the in the the in the the in the in the the in the the in the the in in the the the the in in in the the the the in the in in the the the the in the the in the the in the the the the the the the in the the the in the the the the in the the the the in in the the the the the the the in the the the the in the the the in the the in the the the in the the the the the in the the the the the the the the the in the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the
"""
```
## Prompt template
```
Q:
{prompt}
\nA:
```
## Sparsification
For details on how this model was sparsified, see the `recipe.yaml` in this repo and follow the instructions below.
```bash
git clone https://github.com/neuralmagic/sparseml
pip install -e "sparseml[transformers]"
python sparseml/src/sparseml/transformers/sparsification/obcq/obcq.py openlm-research/open_llama_3b open_platypus --recipe recipe.yaml --save True
python sparseml/src/sparseml/transformers/sparsification/obcq/export.py --task text-generation --model_path obcq_deployment
cp deployment/model.onnx deployment/model-orig.onnx
```
Run this kv-cache injection to speed up the model at inference by caching the Key and Value states:
```python
import os
import onnx
from sparseml.exporters.kv_cache_injector import KeyValueCacheInjector
input_file = "deployment/model-orig.onnx"
output_file = "deployment/model.onnx"
model = onnx.load(input_file, load_external_data=False)
model = KeyValueCacheInjector(model_path=os.path.dirname(input_file)).apply(model)
onnx.save(model, output_file)
print(f"Modified model saved to: {output_file}")
```
Follow the instructions on our [One Shot With SparseML](https://github.com/neuralmagic/sparseml/tree/main/src/sparseml/transformers/sparsification/obcq) page for a step-by-step guide for performing one-shot quantization of large language models.
## Slack
For further support, and discussions on these models and AI in general, join [Neural Magic's Slack Community](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ) |