LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Models for Referring Expression Comprehension
Large Language Models;Vision-Language Models;Black-Box Adaptation;Referring Expression Comprehension
Yes
No ethics review needed.
- Can you explain in more detail how GroundingDino (GD) is used as zero-shot method for REC task, given that GD outputs bounding boxes that cover various objects in an image but cannot directly perform the reasoning needed to decide which box matches a complex query?

- Is the number of fine-tuned parameters an important aspect in achieving better performance? How does Llama with 0.42% parameters tuned performs in comparison with the Llama version in the paper (with 4.2% parameters fine-tuned)?

- Does the LLM always adhere to the generated bbox index?
- The paper introduces a novel method, LLM-wrapper, for adapting VLMs in a black-box manner, which is an interesting way to sidestep the need for model-specific fine-tuning which involves access to model weights or architectures.
- The paper compares LLM-wrapper with baseline zero-shot and fine-tuned VLMs across 3 different REC dataset variants.
- LLM-wrapper can ensemble and transfer across different VLMs, which is an interesting future area of research.
-  LLM-wrapper requires modest resources (e.g., a single 40GB-A100 GPU)
- The paper includes thorough ablation studies on the number of training samples used for fine-tuning, comparing zero-shot LLM-wrapper with fine-tuned LLM-wrapper, showing the transferability of LLM-wrapper trained on one VLM's outputs to another VLM or ensembling VLMs, etc.
- The paper shows improvements over zero-shot, but marginally closes some gap from finetuning. Still, there is significant room for improvement which is a good direction for future work.
The paper presents LLM-wrapper, a method for black-box adaptation of VLMs for the Referring Expression Comprehension (REC) task. It leverages the reasoning capabilities of LLMs to process and select the best-matching object proposals generated by VLMs like Grounding-DINO (GD) by converting VLM outputs (bounding boxes, labels, scores) into natural language prompts.
- The method requires constructing new training data to fine-tune the LLM, which partially defeats the purpose of a black-box approach that aims to minimize resource and data preparation efforts. 
- The paper lacks comparisons with existing black-box adaptation methods or prompting strategies from the literature, e.g., prompting methods that were discussed in the related work section. 
- There is noticeable performance variability depending on the choice of LLM or more likely the number of parameters finetuned (e.g., Llama 3 4.2% vs. Mixtral 0.24%). A deeper analysis of why these differences occur is needed, e.g., by fine-tuning fewer parameters in Llama and comparing Llama 4.2% and Llama 0.42% fine-tuned parameters. 
- There is limited discussion on how well the LLM adheres to producing only the best box as the output and whether there are cases where the LLM generates unexpected or incorrect responses. More generally, the paper does not provide enough detail on failure cases where the LLM-wrapper may not perform well.
- How far are results from SOTA performance on RefCOCO REC?

Minor comment:
- The paper is too verbose in many places, e.g., section 3.4 seems unnecessary as the benefits have been explained well in the previous sections. Instead, it would be nice to provide more analysis with SoTA baselines, more diverse qualitative examples, limitations, etc.
Yes
No ethics review needed.
Are the results reported on different benchmarks coming from LLMs fine-tuned on corresponding training data? For example, in table 1, when the LLM-wrapper gets 78.12 on RefCOCOg val-umd, is the Llama3 fine-tuned using only the bounding boxes and images from RefCOCOg or is the Llama3 fine-tuned using samples coming from all three benchmarks?
1. This approach does not need to access the original parameters of VLMs, but can still get benefits from them. In the case that the VLMs are closed-source, LLM-wrapper can be useful.
2. Finetuning LLMs using LoRA reduces the number of training parameters. 
3. LLM-wrapper trained using one specific VLM can learn general spatial and semantic knowledge, and therefore can be used during inference with another VLM. For some expensive VLMs, with such knowledge transferring, it is not necessary to re-train another LLM-wrapper.
Fine-tuning VLMs requires users to access the architecture, weights, and gradients of models, which can be difficult due to model privacy. This paper focuses on exploring the "black-box" adaptation of VLMs on referring expression comprehension (REC), which means only calling VLMs through forward functions. They propose LLM-wrapper, which uses the reasoning ability of LLMs to select the best matching box generated by VLMs. 
In LLM-wrapper, the VLMs first generate bounding box candidates and these boxes are transferred into text prompts including the box coordinates, labels and/or prediction scores. Next, these prompts are provided to fine-tune the LLMs using LoRA. Finally, these LLMs are capable of reasoning about the image using these prompts and selecting the best matching box from the prompts. In the experimental section, they conduct experiments on three REC benchmarks. For VLMs, they choose Grounding-Dino and Florence-2, and for LLMs they choose Mixtral and Llama3. With LLM-wrapper, they show the LLMs can outperform the zero-shot VLMs by a large margin. Moreover, they find that REC performances can be further boosted by ensembling outputs from different VLMs. They also show that LLM-wrapper can be fine-tuned on one VLM and transferred to another VLM.
1. In table 3, for Grounding-Dino, white-box experiments are conducted using SwinB, but both zero-shot baselines and LLM-wrapper experiments are conducted using SwimT. It would be better if the authors could provide White-box results for SwinT as well.
2. In table 5, the last two rows show the results on 300 samples, but for the first four rows, the results are reported on the whole val/test sets. I would like to see the improvements when transferring from GDrec to Flo2 and from Flo2 to GDrec on the same 300 samples. 
3. LLM-wrapper still uses the bounding box annotations. For REC, models with a similar number of parameters fine-tuned using the same amount of bounding boxes can even get better results.
Yes
No ethics review needed.
Please refer to weaknesses 4 and 5.
1. The paper is well-written, making the proposed method easy to understand with clear motivation and steps.

2. LLM-wrapper only needs to fine-tune the LLM once, allowing it to work with newer, more powerful VLMs without retraining, making it adaptable as VLM technology advances.

3. By combining outputs from multiple VLMs, LLM-wrapper reaches performance levels that compete with some fully fine-tuned VLMs, showing its robustness.

4. The method requires only a single GPU for fine-tuning, keeping training costs low and making it accessible.

5. Black-box approach is ideal for closed-source VLMs, adding value in scenarios where model internals aren’t accessible.
This paper introduces LLM-wrapper, a novel method for enhancing the Referring Expression Comprehension (REC) capabilities of black-box VLMs without directly fine-tuning them. LLM-wrapper leverages a fine-tuned LLM to refine the bounding box proposals generated by a frozen, zero-shot VLM. Specifically, LLM-wrapper takes in a natural language description of each proposal along with spatial coordinates and the original referring expression, then outputs the proposal most relevant to the given expression as the final REC prediction. This black-box adaptation approach allows closed-source VLMs to be adapted without access to their internal structure or gradients, making it adaptable to any VLM and capable of integrating proposals from multiple VLMs to improve performance through ensembling.
1. The LLM-wrapper approach doubles the inference cost compared to existing VLMs.

2. The effectiveness of the LLM-wrapper is highly dependent on the performance of the chosen VLM.

3. Several recent VLMs, such as Ferret [1], KOSMOS-2 [2], and Qwen-VL [3], support both referring expression comprehension (REC) and grounding. The paper could be strengthened by including more baseline experiments that compare the original REC performance of these VLMs with their LLM-wrapped versions.

4. In the LLM-wrapper paradigm, the number of entities in the referring expression likely influences the final performance. If the VLM can accurately predict all entities in a referring expression, further analysis is warranted on whether an increased number of entities aids or hinders the LLM's ability to ground the correct one. An ablation study examining the effect of entity count on performance would be valuable.

5. Extending point 4, the experiments are conducted on the RefCOCO series, which contains relatively simple referring expressions (with a length of 3.6–8.4 words, as reported in the paper). This limitation suggests a confined number of entities per sentence. It would be insightful to evaluate the model’s performance on longer expressions with more entities, such as HC-RefLoCo benchmark, to determine how the LLM performs when faced with more complex captions and a greater number of entities.

[1] H. You, H. Zhang, Z. Gan, X. Du, B. Zhang, Z. Wang, L. Cao, S.-F. Chang, and Y. Yang. Ferret: Refer
and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704, 2023.

[2] Z. Peng, W. Wang, L. Dong, Y. Hao, S. Huang, S. Ma, and F. Wei. Kosmos-2: Grounding multimodal
large language models to the world. arXiv preprint arXiv:2306.14824, 2023.

[3] Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023.
Yes
No ethics review needed.
please refer to above-mentioned.
1. The idea is impressive.
2. The paper is a well-structured format.
This paper proposes a method called LLM-wrapper, aiming at ‘black-box’ adaptation of VLMs for the REC task using Large Language Models.
1. The performances of this method is limited. The performances of many REC methods such as [1], [2], [3] don’t rely on the LLM are better than the performances of this method.
2. The experimental evaluation of this method is constrained. Additional dataset comparisons, such as Flickr30K entities and RefClef, should be taken into account. 

[1] Qiu, Heqian and Li, Hongliang and Zhao, Taijin and Wang, Lanxiao and Wu, Qingbo and Meng, Fanman, RefCrowd: Grounding the Target in Crowd with Referring Expressions, MM2022.

[2] Zhao, Peizhi and Zheng, Shiyi and Zhao, Wenye and Xu, Dongsheng and Li, Pijian and Cai, Yi and Huang, Qingbao, Rethinking Two-Stage Referring Expression Comprehension: A Novel Grounding and Segmentation Method Modulated by Point, AAAI2024

[3] Jing, Chenchen and Wu, Yuwei and Pei, Mingtao and Hu, Yao and Jia, Yunde and Wu, Qi, Visual-Semantic Graph Matching for Visual Grounding, MM2020.
Fine-tuning an LLM to reason on VLM outputs offers a way to adapt said VLM for the Referring Expression Comprehension task in a black-box, model agnostic and efficient manner.
@inproceedings{
anonymous2024llmwrapper,
title={{LLM}-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Models for Referring Expression Comprehension},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=PgXpOOqtyd},
note={under review}
}
Vision Language Models (VLMs) have demonstrated remarkable capabilities in various open-vocabulary tasks, yet their zero-shot performance lags behind task-specific fine-tuned models, particularly in complex tasks like Referring Expression Comprehension (REC). Fine-tuning usually requires ‘white-box’ access to the model’s architecture and weights, which is not always feasible due to proprietary or privacy concerns. In this work, we propose LLM-wrapper, a method for ‘black-box’ adaptation of VLMs for the REC task using Large Language Models (LLMs). LLM-wrapper capitalizes on the reasoning abilities of LLMs, improved with a light fine-tuning, to select the most relevant bounding box to match the referring expression, from candidates generated by a zero-shot black-box VLM. Our approach offers several advantages: it enables the adaptation of closed-source models without needing access to their internal workings, it is versatile and works with any VLM, transfers to new VLMs, and it allows for the adaptation of an ensemble of VLMs. We evaluate LLM-wrapper on multiple datasets using different VLMs and LLMs, demonstrating significant performance improvements and highlighting the versatility of our method. While LLM-wrapper is not meant to directly compete with standard white-box fine-tuning, it offers a practical and effective alternative for black-box VLM adaptation. The code will be open-sourced.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large Language Models Vision-Language Models Black-Box Adaptation Referring Expression Comprehension
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/4b1192960c9e070d091063dbd010659a445b53e7.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Models for Referring Expression Comprehension
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission