README.md

---
base_model: GeneZC/MiniChat-2-3B
inference: false
model_type: llama
prompt_template: |
  <s> [|User|]\n
  {prompt}</s>
  [|Assistant|]\n
quantized_by: mwitiderrick
tags:
- deepsparse
---
# MiniChat-2-3B - DeepSparse
This repo contains model files for [MiniChat-2-3B](https://huggingface.co/GeneZC/MiniChat-2-3B) optimized for [DeepSparse](https://github.com/neuralmagic/deepsparse), a CPU inference runtime for sparse models.

This model was quantized and pruned with [SparseGPT](https://arxiv.org/abs/2301.00774), using [SparseML](https://github.com/neuralmagic/sparseml).
## Inference
Install [DeepSparse LLM](https://github.com/neuralmagic/deepsparse) for fast inference on CPUs: 
```bash
pip install deepsparse-nightly[llm]
```
Run in a [Python pipeline](https://github.com/neuralmagic/deepsparse/blob/main/docs/llms/text-generation-pipeline.md):
```python
from deepsparse import TextGeneration

prompt = "How to get in a good university?"
formatted_prompt =  f"<s> [|User|]\n{prompt}</s>[|Assistant|]\n"

model = TextGeneration(model_path="hf:nm-testing/MiniChat-2-3B-pruned50-quant-ds")

print(model(formatted_prompt, max_new_tokens=200).generations[0].text)
"""
Getting into a good university involves several steps and considerations. Here's a general outline to help you navigate the process effectively:

1. Assess your strengths and weaknesses:
   - Evaluate your academic strengths and areas where you excel. This will help you identify which fields you are particularly good at and which universities may be more likely to accept you.
   - Consider your weak areas as well. This will help you understand which fields you may struggle in, and you can focus on improving your skills in those areas.

2. Research universities:
   - Look into universities that align with your strengths and weaknesses. You can research universities based on factors such as academic reputation, location, campus culture, and tuition fees.
   - Look into universities that offer scholarships or financial aid for students from your area.

3. Prepare for entrance exams:
"""
```

## Prompt template
```

  <s> [|User|]\n
  {prompt}
  </s>[|Assistant|]\n
```
## Sparsification
For details on how this model was sparsified, see the `recipe.yaml` in this repo and follow the instructions below.

```bash
git clone https://github.com/neuralmagic/sparseml
pip install -e "sparseml[transformers]"
python sparseml/src/sparseml/transformers/sparsification/obcq/obcq.py GeneZC/MiniChat-2-3B open_platypus --recipe recipe.yaml --save True
python sparseml/src/sparseml/transformers/sparsification/obcq/export.py --task text-generation --model_path obcq_deployment 
cp deployment/model.onnx deployment/model-orig.onnx
```
Run this kv-cache injection to speed up the model at inference by caching the Key and Value states:
```python
import os
import onnx
from sparseml.exporters.kv_cache_injector import KeyValueCacheInjector
input_file = "deployment/model-orig.onnx"
output_file = "deployment/model.onnx"
model = onnx.load(input_file, load_external_data=False)
model = KeyValueCacheInjector(model_path=os.path.dirname(input_file)).apply(model)
onnx.save(model, output_file)
print(f"Modified model saved to: {output_file}")
```
Follow the instructions on our [One Shot With SparseML](https://github.com/neuralmagic/sparseml/tree/main/src/sparseml/transformers/sparsification/obcq) page for a step-by-step guide for performing one-shot quantization of large language models. 
## Slack

For further support, and discussions on these models and AI in general, join [Neural Magic's Slack Community](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ)