mit-han-lab
/

svdq-int4-flux.1-canny-dev

@@ -1,118 +1,28 @@
 ---
 license: other
 license_name: flux-1-dev-non-commercial-license
 tags:
 - image-to-image
 - SVDQuant
-- INT4
 - FLUX.1
 - Diffusion
 - Quantization
-- ControlNet
-- depth-to-image
-- image-generation
-- text-to-image
 - ICLR2025
-- FLUX.1-Canny-dev
-language:
-- en
-base_model:
-- black-forest-labs/FLUX.1-Canny-dev
-base_model_relation: quantized
-pipeline_tag: image-to-image
-datasets:
-- mit-han-lab/svdquant-datasets
-library_name: diffusers
----
-<p align="center" style="border-radius: 10px">
-  <img src="https://github.com/mit-han-lab/nunchaku/raw/refs/heads/main/assets/logo.svg" width="50%" alt="logo"/>
-</p>
-<h4 style="display: flex; justify-content: center; align-items: center; text-align: center;">Quantization Library:&nbsp;<a href='https://github.com/mit-han-lab/deepcompressor'>DeepCompressor</a> &ensp; Inference Engine:&nbsp;<a href='https://github.com/mit-han-lab/nunchaku'>Nunchaku</a>
-</h4>
-<div style="display: flex; justify-content: center; align-items: center; text-align: center;">
-  <a href="https://arxiv.org/abs/2411.05007">[Paper]</a>&ensp;
-  <a href='https://github.com/mit-han-lab/nunchaku'>[Code]</a>&ensp;
-  <a href='https://svdquant.mit.edu'>[Demo]</a>&ensp;
-  <a href='https://hanlab.mit.edu/projects/svdquant'>[Website]</a>&ensp;
-  <a href='https://hanlab.mit.edu/blog/svdquant'>[Blog]</a>
-</div>
-![teaser](https://huggingface.co/mit-han-lab/svdq-int4-flux.1-canny-dev/resolve/main/demo.jpg)
-`svdq-int4-flux.1-canny-dev` is an INT4-quantized version of [`FLUX.1-Canny-dev`](https://huggingface.co/black-forest-labs/FLUX.1-Canny-dev), which can generate an image based on a text description while following the Canny edge of a given input image. It offers approximately 4× memory savings while also running 2–3× faster than the original BF16 model.
-## Method
-#### Quantization Method -- SVDQuant
-![intuition](https://github.com/mit-han-lab/nunchaku/raw/refs/heads/main/assets/intuition.gif)
-Overview of SVDQuant. Stage1: Originally, both the activation ***X*** and weights ***W*** contain outliers, making 4-bit quantization challenging.  Stage 2: We migrate the outliers from activations to weights, resulting in the updated activation and weight. While the activation becomes easier to quantize, the weight now becomes more difficult. Stage 3: SVDQuant further decomposes the weight into a low-rank component and a residual with SVD. Thus, the quantization difficulty is alleviated by the low-rank branch, which runs at 16-bit precision.
-#### Nunchaku Engine Design
-![engine](https://github.com/mit-han-lab/nunchaku/raw/refs/heads/main/assets/engine.jpg) (a) Naïvely running low-rank branch with rank 32 will introduce 57% latency overhead due to extra read of 16-bit inputs in *Down Projection* and extra write of 16-bit outputs in *Up Projection*. Nunchaku optimizes this overhead with kernel fusion. (b) *Down Projection* and *Quantize* kernels use the same input, while *Up Projection* and *4-Bit Compute* kernels share the same output. To reduce data movement overhead, we fuse the first two and the latter two kernels together.
-## Model Description
-- **Developed by:** MIT, NVIDIA, CMU, Princeton, UC Berkeley, SJTU and Pika Labs
-- **Model type:** INT W4A4 model
-- **Model size:** 6.64GB
-- **Model resolution:** The number of pixels need to be a multiple of 65,536.
-- **License:** Apache-2.0
-## Usage
-### Diffusers
-Please follow the instructions in [mit-han-lab/nunchaku](https://github.com/mit-han-lab/nunchaku) to set up the environment. Also, install some ControlNet dependencies:
-```shell
-pip install git+https://github.com/asomoza/image_gen_aux.git
-pip install controlnet_aux mediapipe
-```
-Then you can run the model with
-```python
-import torch
-from controlnet_aux import CannyDetector
-from diffusers import FluxControlPipeline
-from diffusers.utils import load_image
-from nunchaku.models.transformer_flux import NunchakuFluxTransformer2dModel
-transformer = NunchakuFluxTransformer2dModel.from_pretrained("mit-han-lab/svdq-int4-flux.1-canny-dev")
-pipe = FluxControlPipeline.from_pretrained(
-    "black-forest-labs/FLUX.1-Canny-dev", transformer=transformer, torch_dtype=torch.bfloat16
-).to("cuda")
-prompt = "A robot made of exotic candies and chocolates of different kinds. The background is filled with confetti and celebratory gifts."
-control_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/robot.png")
-processor = CannyDetector()
-control_image = processor(
-    control_image, low_threshold=50, high_threshold=200, detect_resolution=1024, image_resolution=1024
-)
-image = pipe(
-    prompt=prompt, control_image=control_image, height=1024, width=1024, num_inference_steps=50, guidance_scale=30.0
-).images[0]
-image.save("flux.1-canny-dev.png")
-```
-### Comfy UI
-Work in progress. Stay tuned!
-## Limitations
-- The model is only runnable on NVIDIA GPUs with architectures sm_86 (Ampere: RTX 3090, A6000), sm_89 (Ada: RTX 4090), and sm_80 (A100). See this [issue](https://github.com/mit-han-lab/nunchaku/issues/1) for more details.
-- You may observe some slight differences from the BF16 models in detail.
-### Citation
-If you find this model useful or relevant to your research, please cite
 ```bibtex
 @inproceedings{
@@ -122,4 +32,8 @@ If you find this model useful or relevant to your research, please cite
   booktitle={The Thirteenth International Conference on Learning Representations},
   year={2025}
 }
-```

 ---
+base_model: black-forest-labs/FLUX.1-Canny-dev
+base_model_relation: quantized
+datasets:
+- mit-han-lab/svdquant-datasets
+language:
+- en
+library_name: diffusers
 license: other
+license_link: https://huggingface.co/black-forest-labs/FLUX.1-Canny-dev/blob/main/LICENSE.md
 license_name: flux-1-dev-non-commercial-license
+pipeline_tag: image-to-image
 tags:
 - image-to-image
 - SVDQuant
+- FLUX.1-Canny-dev
 - FLUX.1
 - Diffusion
 - Quantization
 - ICLR2025
+---
+**This repository has been deprecated and will be hidden in December 2025. Please use https://huggingface.co/nunchaku-tech/nunchaku-flux.1-canny-dev.**
+## Citation
 ```bibtex
 @inproceedings{
   booktitle={The Thirteenth International Conference on Learning Representations},
   year={2025}
 }
+```
+## Attribution Notice
+The FLUX.1 [dev] Model is licensed by Black Forest Labs Inc. under the FLUX.1 [dev] Non-Commercial License. Copyright Black Forest Labs Inc. IN NO EVENT SHALL BLACK FOREST LABS INC. BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH USE OF THIS MODEL.