Create README.md

README.md

@@ -0,0 +1,125 @@
+---
+license: other
+license_name: flux-1-dev-non-commercial-license
+tags:
+- image-to-image
+- SVDQuant
+- INT4
+- FLUX.1
+- Diffusion
+- Quantization
+- ControlNet
+- depth-to-image
+- image-generation
+- text-to-image
+- ICLR2025
+- FLUX.1-Canny-dev
+language:
+- en
+base_model:
+- black-forest-labs/FLUX.1-Canny-dev
+base_model_relation: quantized
+pipeline_tag: image-to-image
+datasets:
+- mit-han-lab/svdquant-datasets
+library_name: diffusers
+---
+
+<p align="center" style="border-radius: 10px">
+  <img src="https://github.com/mit-han-lab/nunchaku/raw/refs/heads/main/assets/logo.svg" width="50%" alt="logo"/>
+</p>
+<h4 style="display: flex; justify-content: center; align-items: center; text-align: center;">Quantization Library:&nbsp;<a href='https://github.com/mit-han-lab/deepcompressor'>DeepCompressor</a> &ensp; Inference Engine:&nbsp;<a href='https://github.com/mit-han-lab/nunchaku'>Nunchaku</a>
+</h4>
+
+
+<div style="display: flex; justify-content: center; align-items: center; text-align: center;">
+  <a href="https://arxiv.org/abs/2411.05007">[Paper]</a>&ensp;
+  <a href='https://github.com/mit-han-lab/nunchaku'>[Code]</a>&ensp;
+  <a href='https://svdquant.mit.edu'>[Demo]</a>&ensp;
+  <a href='https://hanlab.mit.edu/projects/svdquant'>[Website]</a>&ensp;
+  <a href='https://hanlab.mit.edu/blog/svdquant'>[Blog]</a>
+</div>
+
+![teaser](https://huggingface.co/mit-han-lab/svdq-int4-flux.1-depth-dev/resolve/main/demo.jpg)
+`svdq-int4-flux.1-canny-dev` is an INT4-quantized version of [`FLUX.1-Canny-dev`](https://huggingface.co/black-forest-labs/FLUX.1-Canny-dev), which can generate an image based on a text description while following the Canny edge of a given input image. It offers approximately 4× memory savings while also running 2–3× faster than the original BF16 model.
+
+## Method
+#### Quantization Method -- SVDQuant
+
+![intuition](https://github.com/mit-han-lab/nunchaku/raw/refs/heads/main/assets/intuition.gif)
+Overview of SVDQuant. Stage1: Originally, both the activation ***X*** and weights ***W*** contain outliers, making 4-bit quantization challenging.  Stage 2: We migrate the outliers from activations to weights, resulting in the updated activation and weight. While the activation becomes easier to quantize, the weight now becomes more difficult. Stage 3: SVDQuant further decomposes the weight into a low-rank component and a residual with SVD. Thus, the quantization difficulty is alleviated by the low-rank branch, which runs at 16-bit precision. 
+
+#### Nunchaku Engine Design
+
+![engine](https://github.com/mit-han-lab/nunchaku/raw/refs/heads/main/assets/engine.jpg) (a) Naïvely running low-rank branch with rank 32 will introduce 57% latency overhead due to extra read of 16-bit inputs in *Down Projection* and extra write of 16-bit outputs in *Up Projection*. Nunchaku optimizes this overhead with kernel fusion. (b) *Down Projection* and *Quantize* kernels use the same input, while *Up Projection* and *4-Bit Compute* kernels share the same output. To reduce data movement overhead, we fuse the first two and the latter two kernels together.
+
+## Model Description
+
+- **Developed by:** MIT, NVIDIA, CMU, Princeton, UC Berkeley, SJTU and Pika Labs
+- **Model type:** INT W4A4 model
+- **Model size:** 6.64GB
+- **Model resolution:** The number of pixels need to be a multiple of 65,536.
+- **License:** Apache-2.0
+
+## Usage
+
+### Diffusers
+
+Please follow the instructions in [mit-han-lab/nunchaku](https://github.com/mit-han-lab/nunchaku) to set up the environment. Also, install some ControlNet dependencies:
+
+```shell
+pip install git+https://github.com/asomoza/image_gen_aux.git
+pip install controlnet_aux mediapipe
+```
+
+Then you can run the model with
+
+```python
+import torch
+from controlnet_aux import CannyDetector
+from diffusers import FluxControlPipeline
+from diffusers.utils import load_image
+
+from nunchaku.models.transformer_flux import NunchakuFluxTransformer2dModel
+
+transformer = NunchakuFluxTransformer2dModel.from_pretrained("mit-han-lab/svdq-int4-flux.1-canny-dev")
+pipe = FluxControlPipeline.from_pretrained(
+    "black-forest-labs/FLUX.1-Canny-dev", transformer=transformer, torch_dtype=torch.bfloat16
+).to("cuda")
+
+prompt = "A robot made of exotic candies and chocolates of different kinds. The background is filled with confetti and celebratory gifts."
+control_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/robot.png")
+
+processor = CannyDetector()
+control_image = processor(
+    control_image, low_threshold=50, high_threshold=200, detect_resolution=1024, image_resolution=1024
+)
+
+image = pipe(
+    prompt=prompt, control_image=control_image, height=1024, width=1024, num_inference_steps=50, guidance_scale=30.0
+).images[0]
+image.save("flux.1-canny-dev.png")
+```
+
+### Comfy UI
+
+Work in progress. Stay tuned!
+
+## Limitations
+
+- The model is only runnable on NVIDIA GPUs with architectures sm_86 (Ampere: RTX 3090, A6000), sm_89 (Ada: RTX 4090), and sm_80 (A100). See this [issue](https://github.com/mit-han-lab/nunchaku/issues/1) for more details.
+- You may observe some slight differences from the BF16 models in detail.
+
+### Citation
+
+If you find this model useful or relevant to your research, please cite
+
+```bibtex
+@inproceedings{
+  li2024svdquant,
+  title={SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models},
+  author={Li*, Muyang and Lin*, Yujun and Zhang*, Zhekai and Cai, Tianle and Li, Xiuyu and Guo, Junxian and Xie, Enze and Meng, Chenlin and Zhu, Jun-Yan and Han, Song},
+  booktitle={The Thirteenth International Conference on Learning Representations},
+  year={2025}
+}
+```