README.md

---
title: README
emoji: 💻
colorFrom: purple
colorTo: blue
sdk: static
pinned: false
---

# Software-Delivered AI Inference

Neural Magic helps developers in accelerating deep learning performance using automated model compression technologies and inference engines.
Download our compression-aware inference engines and open source tools for fast model inference. 
* [nm-vllm](https://neuralmagic.com/nm-vllm/): A high-throughput and memory-efficient inference engine for LLMs, our supported enterprise distribution of [vLLM](https://github.com/vllm-project/vllm).
* [DeepSparse](https://github.com/neuralmagic/deepsparse): Inference runtime offering accelerated performance on CPUs and APIs to integrate ML into your application
* [LLM Compressor](https://github.com/vllm-project/llm-compressor/): Libraries for applying sparsification recipes to neural networks with a few lines of code, enabling faster and smaller models

![image/png](https://cdn-uploads.huggingface.co/production/uploads/60466e4b4f40b01b66151416/2IDqpxbtCtw_ilOZbTSj0.png)

In this profile we provide accurate model checkpoints compressed with SOTA methods ready to run in vLLM such as W4A16, W8A16, W8A8 (int8 and fp8), and many more!