Create README.md

README.md

@@ -0,0 +1,139 @@
+---
+language:
+- en
+license: llama3
+tags:
+- Llama-3
+- instruct
+- finetune
+- chatml
+- gpt4
+- synthetic data
+- distillation
+- function calling
+- json mode
+- axolotl
+- roleplaying
+- chat
+- quantization
+- AWQ
+base_model: meta-llama/Meta-Llama-3.2-3B
+widget:
+- example_title: Hermes 3 AWQ 4-bit
+  messages:
+  - role: system
+    content: >-
+      You are a sentient, superintelligent artificial general intelligence, here
+      to teach and assist me.
+  - role: user
+    content: >-
+      Write a short story about Goku discovering Kirby has teamed up with Majin
+      Buu to destroy the world.
+model-index:
+- name: Hermes-3-Llama-3.2-3B-AWQ-4bit
+  results: []
+library_name: transformers
+---
+
+# Hermes 3 - Llama-3.2 3B (AWQ 4-bit)
+
+![image/jpeg](https://cdn-uploads.huggingface.co/production/uploads/6317aade83d8d2fd903192d9/-kj_KflXsdpcZoTQsvx7W.jpeg)
+
+## Model Description
+
+This is a 4-bit AWQ (Activation-aware Weight Quantization) quantized version of **Hermes 3 - Llama-3.2 3B**, a fine-tuned LLM developed by Nous Research. The quantization was performed to improve efficiency while maintaining strong performance, making the model suitable for low-memory devices and inference acceleration.
+
+For details on the original model, please see the [**Hermes 3 Technical Report**](https://arxiv.org/abs/2408.11857).
+
+### What is AWQ 4-bit Quantization?
+
+AWQ (Activation-aware Weight Quantization) is a quantization technique designed to optimize large language models for efficient inference while minimizing performance loss. The **4-bit AWQ** version of this model:
+
+- **Reduces memory footprint**, enabling deployment on lower-end hardware (e.g., consumer GPUs and edge devices).
+- **Speeds up inference**, making response times faster while maintaining accuracy.
+- **Preserves performance**, as AWQ selectively quantizes weights based on activation sensitivity, ensuring minimal loss in capability.
+
+## Base Model Information
+
+Hermes 3 3B is a generalist language model fine-tuned from **Llama-3.2 3B**, with improvements in:
+- Reasoning
+- Roleplaying
+- Function calling & structured outputs
+- Multi-turn conversation
+- Long-context coherence
+
+This quantized version retains these enhancements while offering better efficiency.
+
+## Performance Benchmarks
+
+The original **Hermes 3 3B** model achieved strong performance on various benchmarks. While the AWQ quantized version maintains high accuracy, minor variations may occur due to the quantization process. For benchmarking, refer to the original model's results.
+
+## Prompt Format
+
+This model follows **ChatML formatting**, similar to OpenAI's API prompt structure. Example:
+
+```python
+messages = [
+    {"role": "system", "content": "You are Hermes 3."},
+    {"role": "user", "content": "Hello, who are you?"}
+]
+gen_input = tokenizer.apply_chat_template(messages, return_tensors="pt")
+model.generate(**gen_input)
+```
+
+For more details, see the [Hermes 3 documentation](https://huggingface.co/NousResearch/Hermes-3-Llama-3.2-3B).
+
+## Inference with AWQ 4-bit Model
+
+To use this quantized model efficiently, load it with **AutoAWQ** or **transformers**:
+
+```python
+from transformers import AutoTokenizer
+from autoawq import AutoAWQForCausalLM
+
+model_path = "your_model_path"
+tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
+model = AutoAWQForCausalLM.from_quantized(model_path, device="cuda")
+
+prompt = "<|im_start|>user\nHello! How are you?<|im_end|>\n<|im_start|>assistant"
+input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to("cuda")
+output = model.generate(input_ids, max_new_tokens=256)
+response = tokenizer.decode(output[0], skip_special_tokens=True)
+print(response)
+```
+
+## Quantized Model Use Cases
+
+- Running LLMs on lower-end **consumer GPUs** (e.g., RTX 3060, 4060, etc.)
+- **Faster inference** with minimal degradation in quality
+- **Edge computing & on-device AI** with constrained resources
+- **Cloud inference** with optimized performance/cost ratio
+
+## Limitations & Considerations
+
+- **Minor accuracy loss** due to 4-bit quantization (slightly less precise responses in rare cases)
+- **Lower computational overhead** at the expense of some fine-grained details
+- **Best suited for inference**, rather than fine-tuning or continued training
+
+## Citation
+
+If you use this model, please cite the original **Hermes 3 Technical Report**:
+
+```bibtex
+@misc{teknium2024hermes3technicalreport,
+      title={Hermes 3 Technical Report},
+      author={Ryan Teknium and Jeffrey Quesnelle and Chen Guang},
+      year={2024},
+      eprint={2408.11857},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2408.11857},
+}
+```
+
+## Acknowledgments
+
+This quantization was performed using the **AWQ** method for LLM optimization. The base model was developed by Nous Research, and quantization was applied to enhance deployment efficiency while preserving model quality.
+
+For further details, refer to [Nous Research](https://huggingface.co/NousResearch) and [Hermes 3 models](https://huggingface.co/NousResearch/Hermes-3-Llama-3.2-3B).
+