Update README.md

README.md

@@ -3,7 +3,7 @@ sdk: gradio
 sdk_version: 4.42.0
 ---
 
-# [YOLOE: Real-Time Seeing Anything]()
+# [YOLOE: Real-Time Seeing Anything](https://arxiv.org/abs/2503.07465)
 
 Official PyTorch implementation of **YOLOE**.
 
@@ -12,8 +12,9 @@ Official PyTorch implementation of **YOLOE**.
   Comparison of performance, training cost, and inference efficiency between YOLOE (Ours) and YOLO-Worldv2 in terms of open text prompts.
 </p>
 
-[YOLOE: Real-Time Seeing Anything]().\
-Ao Wang*, Lihao Liu*, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding
+[YOLOE: Real-Time Seeing Anything](https://arxiv.org/abs/2503.07465).\
+Ao Wang*, Lihao Liu*, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding\
+[![arXiv](https://img.shields.io/badge/arXiv-2503.07465-b31b1b.svg)](https://arxiv.org/abs/2503.07465) [![Hugging Face Models](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Models-blue)](https://huggingface.co/jameslahm/yoloe/tree/main) [![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/jameslahm/yoloe)
 
 
 We introduce **YOLOE(ye)**, a highly **efficient**, **unified**, and **open** object detection and segmentation model, like human eye, under different prompt mechanisms, like *texts*, *visual inputs*, and *prompt-free paradigm*.
@@ -31,12 +32,12 @@ We introduce **YOLOE(ye)**, a highly **efficient**, **unified**, and **open** ob
   <summary>
   <font size="+1">Abstract</font>
   </summary>
-Object detection and segmentation are widely employed in computer vision applications, yet conventional models like YOLO series, while efficient and accurate, are limited by predefined categories, hindering adaptability in open scenarios. Recent open-set methods leverage text prompts, visual cues, or prompt-free paradigm to overcome this, but often compromise between performance and efficiency due to high computational demands or deployment complexity. In this work, we introduce YOLOE, which integrates detection and segmentation across diverse open prompt mechanisms within a single highly efficient model, achieving real-time seeing anything. For text prompts, we propose Re-parameterizable Region-Text Alignment (RepRTA) strategy. It refines pretrained textual embeddings via a re-parameterizable lightweight auxiliary network and enhances visual-textual alignment with zero inference and transferring overhead. For visual prompts, we present Semantic-Activated Visual Prompt Encoder (SAVPE). It employs decoupled semantic and activation branches to bring improved visual embedding and accuracy with minimal complexity. For prompt-free scenario, we introduce Lazy Region-Prompt Contrast (LRPC) strategy. It utilizes a built-in large vocabulary and specialized embedding to identify all objects, avoiding costly language model dependency. Extensive experiments show YOLOE's exceptional zero-shot performance and transferability with high inference efficiency and low training cost. Notably, on LVIS, with $3\times$ less training cost and $1.4\times$ inference speedup, YOLOE-v8-S surpasses YOLO-Worldv2-S by 3.5 AP. When transferring to COCO, YOLOE-v8-L achieves 0.6 $AP^b$ and 0.4 $AP^m$ gains over closed-set YOLOv8-L with nearly $4\times$ less training time. Code and models will be publicly available.
+Object detection and segmentation are widely employed in computer vision applications, yet conventional models like YOLO series, while efficient and accurate, are limited by predefined categories, hindering adaptability in open scenarios. Recent open-set methods leverage text prompts, visual cues, or prompt-free paradigm to overcome this, but often compromise between performance and efficiency due to high computational demands or deployment complexity. In this work, we introduce YOLOE, which integrates detection and segmentation across diverse open prompt mechanisms within a single highly efficient model, achieving real-time seeing anything. For text prompts, we propose Re-parameterizable Region-Text Alignment (RepRTA) strategy. It refines pretrained textual embeddings via a re-parameterizable lightweight auxiliary network and enhances visual-textual alignment with zero inference and transferring overhead. For visual prompts, we present Semantic-Activated Visual Prompt Encoder (SAVPE). It employs decoupled semantic and activation branches to bring improved visual embedding and accuracy with minimal complexity. For prompt-free scenario, we introduce Lazy Region-Prompt Contrast (LRPC) strategy. It utilizes a built-in large vocabulary and specialized embedding to identify all objects, avoiding costly language model dependency. Extensive experiments show YOLOE's exceptional zero-shot performance and transferability with high inference efficiency and low training cost. Notably, on LVIS, with $3\times$ less training cost and $1.4\times$ inference speedup, YOLOE-v8-S surpasses YOLO-Worldv2-S by 3.5 AP. When transferring to COCO, YOLOE-v8-L achieves 0.6 $AP^b$ and 0.4 $AP^m$ gains over closed-set YOLOv8-L with nearly $4\times$ less training time.
+</details>
 <p></p>
 <p align="center">
   <img src="figures/pipeline.svg" width=96%> <br>
 </p>
-</details>
 
 ## Performance
 
@@ -122,6 +123,7 @@ pip install -e CLIP
 ```
 
 ## Demo
+If desired objects are not identified, pleaset set a **smaller** confidence threshold, e.g., for visual prompts with handcrafted shape or cross-image prompts.
 ```bash
 # Optional for mirror: export HF_ENDPOINT=https://hf-mirror.com
 pip install gradio==4.42.0 gradio_image_prompter==0.1.0 fastapi==0.112.2
@@ -235,6 +237,8 @@ python train_seg.py
 python tools/convert_segm2det.py
 # Then, train the SAVPE module
 python train_vp.py
+# After training, please use tools/get_vp_segm.py to add the segmentation head
+# python tools/get_vp_segm.py
 ```
 
 ### Prompt free
@@ -245,6 +249,8 @@ python tools/generate_lvis_sc.py
 # Similar to visual prompt, because only the specialized prompt embedding is trained, we can adopt the detection pipleline with less training time
 python tools/convert_segm2det.py
 python train_pe_free.py
+# After training, please use tools/get_pf_free_segm.py to add the segmentation head
+# python tools/get_pf_free_segm.py
 ```
 
 ## Transferring
@@ -285,5 +291,13 @@ Thanks for the great implementations!
 
 If our code or models help your work, please cite our paper:
 ```BibTeX
-
-```
+@misc{wang2025yoloerealtimeseeing,
+      title={YOLOE: Real-Time Seeing Anything}, 
+      author={Ao Wang and Lihao Liu and Hui Chen and Zijia Lin and Jungong Han and Guiguang Ding},
+      year={2025},
+      eprint={2503.07465},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV},
+      url={https://arxiv.org/abs/2503.07465}, 
+}
+```