Spaces:
Running
Running
File size: 1,263 Bytes
c3d28e6 0f34e54 c3d28e6 0f34e54 c3d28e6 a2c8f17 5a9070c f9ac629 15cda38 a2c8f17 f9ac629 c9b8405 f9ac629 c3ddd3d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
---
title: README
emoji: 💻
colorFrom: purple
colorTo: blue
sdk: static
pinned: false
---
# The Future of AI is Open
Neural Magic helps developers in accelerating deep learning performance using automated model compression technologies and inference engines.
Download our compression-aware inference engines and open source tools for fast model inference.
* [vLLM](https://github.com/vllm-project/vllm): A high-throughput and memory-efficient inference engine for LLMs
* [llm-compressor](https://github.com/vllm-project/llm-compressor/): HF-native library for applying quantization and sparsity algorithms to llms for optimized deployment with vLLM
* [DeepSparse](https://github.com/neuralmagic/deepsparse): Inference runtime offering accelerated performance on CPUs and APIs to integrate ML into your application
In this profile we provide accurate model checkpoints compressed with SOTA methods ready to run in vLLM such as W4A16, W8A16, W8A8 (int8 and fp8), and many more! If you would like help quantizing a model or have a request for us to add a checkpoint, please open an issue in https://github.com/vllm-project/llm-compressor. |