Update README.md

README.md

@@ -1,5 +1,81 @@
----
-license: other
-license_name: flux-1-dev-non-commercial-license
-license_link: https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md
----
+---
+license: other
+license_name: flux-1-dev-non-commercial-license
+license_link: https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md
+
+language:
+  - en
+library_name: diffusers
+pipeline_tag: text-to-image
+
+tags:
+- Text-to-Image
+- ControlNet
+- Diffusers
+- Flux.1-dev
+- image-generation
+- Stable Diffusion
+base_model: black-forest-labs/FLUX.1-dev
+---
+
+# FLUX.1-dev-ControlNet-Union-Pro-2.0
+
+This repository contains an unified ControlNet for FLUX.1-dev model released by [Shakker Labs](https://huggingface.co/Shakker-Labs).
+
+# Keynotes
+In comparison with [Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro](https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro),
+- Remove mode embedding. Smaller model size (6.6GB -> 4.0GB).
+- Improve on canny and pose, better control and aesthetics.
+- Add support for soft edge. Remove support for tile.
+
+
+# Model Cards
+- This ControlNet consists of 6 double blocks and 0 single block as the same as [Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro](https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro). Mode embedding is removed.
+- We train the model from scratch for 300k steps using a dataset of 20M high-quality general and human images. We train at 512x512 resolution in BFloat16, batch size = 128, learning rate = 2e-5, the guidance is uniformly sampled from [1, 7]. We set the text drop ratio to 0.20.
+- This model supports multiple control modes, including canny, soft edge, depth, pose, gray.
+- This model can be jointly used with other ControlNets.
+
+# Inference
+```python
+import torch
+from diffusers.utils import load_image
+from diffusers import FluxControlNetPipeline, FluxControlNetModel
+
+base_model = 'black-forest-labs/FLUX.1-dev'
+controlnet_model_union = 'Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0'
+
+controlnet = FluxControlNetModel.from_pretrained(controlnet_model_union, torch_dtype=torch.bfloat16)
+pipe = FluxControlNetPipeline.from_pretrained(base_model, controlnet=controlnet, torch_dtype=torch.bfloat16)
+pipe.to("cuda")
+
+width, height = control_image_depth.size
+
+image = pipe(
+    prompt, 
+    control_image=control_image,
+    width=width,
+    height=height,
+    controlnet_conditioning_scale=0.7,
+    control_guidance_end=0.8,
+    num_inference_steps=24, 
+    guidance_scale=3.5,
+    generator=torch.manual_seed(42),
+).images[0]
+```
+
+# Recommended Parameters
+You can adjust controlnet_conditioning_scale and control_guidance_end for stronger control and better detail preservation.
+- Canny: use cv2.Canny, controlnet_conditioning_scale=0.7, control_guidance_end=0.8.
+- Soft Edge: use [AnylineDetector](https://github.com/huggingface/controlnet_aux), controlnet_conditioning_scale=0.7, control_guidance_end=0.8.
+- Depth: use [depth-anything](https://github.com/DepthAnything/Depth-Anything-V2), controlnet_conditioning_scale=0.8, control_guidance_end=0.8.
+- Pose: use [DWPose](https://github.com/IDEA-Research/DWPose/tree/onnx), controlnet_conditioning_scale=0.9, control_guidance_end=0.65.
+- Gray: use cv2.cvtColor, controlnet_conditioning_scale=0.9, control_guidance_end=0.8.
+
+# Resources
+- [InstantX/FLUX.1-dev-IP-Adapter](https://huggingface.co/InstantX/FLUX.1-dev-IP-Adapter)
+- [InstantX/FLUX.1-dev-Controlnet-Canny](https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Canny)
+- [Shakker-Labs/FLUX.1-dev-ControlNet-Depth](https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Depth)
+- [Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro](https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro)
+
+# Acknowledgements
+This model is developed by [Shakker Labs](https://huggingface.co/Shakker-Labs). The original idea is inspired by [xinsir/controlnet-union-sdxl-1.0](https://huggingface.co/xinsir/controlnet-union-sdxl-1.0). All copyright reserved.