GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language Models
llms;vlms;prompt optimization
Yes
No ethics review needed.
* **Clarity of Algorithm 1**: At lines 9, 12, 28, 29, it's unclear what is the meaning of the square brackets, given that $K$ is an integer according to the input. It's also not clear how the top-3 prompts used for ensemble are selected: Are they from the last step, a single best step, or all steps through out the search process?

* **Sensitivity to the hyper-parameters**: The LLM steering part introduces two hyper-parameters, layer $l$ and scaling factor $\alpha$. Are these hyper-parameters searched on one dataset and kept the same for the others, or searched independently on each dataset? How different could the optimal choices be over different datasets?

* **Additional placeholders in prompts**: Some searched prompts (e.g., at Ln 1187-1192) seem to contain additional placeholders (in angle brackets). Are they from additional metadata of the dataset? Is the search process altered in any way to account for these additional information?

* **Average accuracy in main tables** (e.g., Table 1, 2) would make it easier to cross-reference the results with the ablation studies (e.g., Table 4)
* The motivation is sound and clear.

* The experimental results are transparent, with search outcomes in the appendix and code release promised.

* The prompts generated by search are shown to generalize within the same category (dual encoder) of VLMs.
The paper proposes GLOV, an LLM-assisted framework for automatic optimization of VLM prompts for specific downstream tasks. Specifically, an LLM is meta-prompted to generate downstream VLM prompts based on task description and in-context examples from previously generated prompts as feedbacks. On top of the meta prompt, the paper also applies feature-level guidance, i.e., the difference of sentence embedding from bad prompts to good prompts, as a second measure to push the LLM output towards the more effective direction. The proposed method is evaluated mainly on 16 few-shot classification tasks and shows improvement over baselines, while preliminary results on VQA are also provided.
* **Feature-level guidance poses white-box LLM constraint**: Despite the feature guidance being novel for VLM prompt optimization, it requires access to LLM intermediate features, which could be hard to obtain given that many strong LLMs are either closed-source or too large to easily run locally. This could be a hedge against the advantages of black-box methods, as the LLM intermediate features could be even harder to get than parameters or gradients of VLMs in many cases.

* **Sensitivity to LLM choices is not clear**: While the proposed method shows clear improvements, it would make the argument stronger if more evidence could be given showing that the reasoning of the LLM indeed plays an important role, especially with the fluctuation (e.g., Fig 1, 3, 6) of the results and the general impression that LLMs at 7B-level are not very good at reasoning or agent-like applications. One way to show this is higher accuracy or less steps to convergence with a stronger LLM.
Yes
No ethics review needed.
* In the context of encoder-decoder architectures, is there a potential for the emergence of lengthy and ambiguous symbolic representations during the optimization process? Furthermore, what measures can be implemented to ensure the efficacy of sentence transformers under such circumstances?

* The reviewer expresses concern that the adoption of top-k and bottom-k approaches for in-context examples may result in a significant disparity between positive and negative samples in the later stages of training, potentially hindering the model to learn subtle prompt refinements akin to the challenges posed by consistently employing a large learning rate in gradient-based optimization. Consequently, the reviewer prefers implementing a dynamic selection threshold as a more reasonable choice. Any insights regarding the current strategy would enhance the understanding of the paper.

* Similarly, in the steering of LLM, would it be more judicious to dynamically modify the rank interval between the positive (P+) and negative (P-)?
* The introduction of steering LLM response during the prompt optimization process presents a novel and effective methodology.
* The steering strategy designed by analogy with the gradient update process, while lacking a rigorous theoretical basis, conforms well to engineering intuition.
* The article is highly readable, featuring a well-defined and clear structure.
This paper proposes a novel framework GLOV, which enables LLMs to act as implicit optimizers for VLMs to enhance downstream vision tasks. Experiments highlight the effectiveness of GLOV for both dual-encoder and encoder-decoder architectures.
* The applicability of GLOV optimization to a given task is constrained by the existence of an objective fitness function for the task.
* For encoder-decoder models such as LLaVA, it seems the VLM response has to be relatively concise in form. When dealing with complex responses (such as responses for image captioning tasks), the reliability of the sentence embeddings computed via Equation 3 remains unverified.
Yes
No ethics review needed.
Please kindly answer the question in the weakness section.
- This paper introduces an interesting framework, GLOV, to enhance VLMs for image classification.
- The use of meta-prompts and LLM steering provides fresh insights in this field.
- Experimental results demonstrate the effectiveness of the proposed methods compared to  baselines.
This paper introduces GLOV, a method that uses Large Language Models (LLMs) as implicit optimizers to improve Vision-Language Models (VLMs) for vision tasks. GLOV employs meta-prompts to help the LLM generate effective prompts tailored to specific tasks, which are ranked for their effectiveness. It incorporates feedback from previous optimization steps through an offset vector that guides the LLM in generating better prompts. Experiments on 16 datasets demonstrate the effectiveness of proposed methods.
- Lack of comparison. While GLOV approaches image classification from a novel perspective, previous works [1,2,3] in this area already achieved promising results with lower costs. Could authors provide a comparison of GLOV with these baselines?
- The generalization ability of GLOV is not clear. The authors demonstrated the effectiveness of the proposed methods on VLM image classification and visual question answers under the same topic. However, if the GLOV is not competitive compared with other works focused on VLM image classification[1,2,3]. Could authors prove the generalization ability of GLOV on visual tasks beyond image classification?
- Clarity of Figure 2: The method overview in Figure 2 is difficult to understand. If the authors could clearly show the flow of iterative optimization, the methods would be easier to follow.
- Lack of inference cost comparison: Could the authors show the curve of iteration steps and inference time to illustrate the trade-off between performance and cost in GLOV?


Reference:
[1]AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation
[2]Sus-x: Training-free name-only transfer of vision-language models
[3]Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling
Yes
No ethics review needed.
For few-shot learning, the impact of uniquely labeled data on performance is significant. In this paper, how to select this sample to ensure that the reported results are statistically significant rather than random? What is the variance of performance if five times of experiments are conducted?
[+] Improving the generalization of VLM in downstream tasks with low cost (parameters, fine-tuning data, training speed) is one practical topic that deserves further explorations.

[+] The paper is easy to follow and understand, having clear logic.

[+] Some experiments are conducted to demonstrate the idea’s performance, as well as the values of optimal prompt search.
This paper aims to improve the VLMs’ performance in downstream tasks. One prompt optimization method, namely GLOV, is proposed by meta-prompting LLM to choose the type of language structure preferred by the downstream VLM. At each optimization step, one embedding space steering methodology is used to bound the outputs more strictly. Empirical experiments with different VLM architectures on multiple datasets show the GLOV’s effectiveness.
[-] Novelty. Meta-prompts have been introduced by [1], while this paper expands the idea to a few-shot training data, which is rather trivial and brings minor technological contributions to the community. For the designs of meta-prompts, how to verify that this solution is optimal?

[-] Impracticality. As we all know, due to the autoregressive paradigm, the LLM inference requires a significant amount of cost compared to encoder-based single-forward models. Thus, employing LLMs in an iterative workflow, to find optimal language prompts seems unrealistic for real-world applications. 

[-] Unknown efficacy. In the main manuscript, only the performance of downstream tasks is reported, without any computational/time complexity. The reviewer suggests to provide the inference time (in seconds) and the required GPU memory (in GB) for all methods in Table 1-2 to clarify its practical value.

[-] Incomplete comparisons. To improve the model performance on downstream tasks, one popular and effective idea is parameter efficient fine-tuning (PET), such as prompt-tuning, LoRA, and adapter, which has shown impressive few-shot capability. In Table 5 of the supplementary materials, CoOp performs even worse than CLIP, which is surprisingly and suspiciously. It is necessary to compare PET with the proposed method, in terms of performance, parameters, training time, and inference speed of under the same settings. 


[1] Meta-Prompting for Automating Zero-shot Visual Recognition with LLMs.
GLOV uses LLMs as implicit optimizers for VLMs, generating and refining task-specific prompts, leading to up to 57.5% improved performance on diverse vision tasks.
@inproceedings{
anonymous2024glov,
title={{GLOV}: Guided Large Language Models as Implicit Optimizers for Vision Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=3Mia9aFpgo},
note={under review}
}
In this work, we propose a novel method (GLOV) enabling Large Language Models (LLMs) to act as implicit Optimizers for Vision-Langugage Models (VLMs) to enhance downstream vision tasks. 
Our GLOV meta-prompts an LLM with the downstream task description, querying it for suitable VLM prompts (e.g., for zero-shot classification with CLIP). 
These prompts are ranked according to a purity measure obtained through a fitness function. 
In each respective optimization step, the ranked prompts are fed as in-context examples (with their accuracies) to equip the LLM with the knowledge of the type of text prompts preferred by the downstream VLM. 
Furthermore, we also explicitly steer the LLM generation process in each optimization step 
by specifically adding an offset difference vector of the embeddings from the \textit{positive} and \textit{negative} solutions found by the LLM, in previous optimization steps, to the intermediate layer of the network for the next generation step.
This offset vector steers the LLM generation toward the type of language preferred by the downstream VLM, resulting in enhanced performance on the downstream vision tasks. 
We comprehensively evaluate our GLOV on 16 diverse datasets using two families of VLMs, i.e., dual-encoder (e.g., CLIP) and encoder-decoder (e.g., LLaVa) models --
showing that the discovered solutions can enhance the recognition performance by up to $15.0$% and $57.5$% ($3.8$% and $21.6$% on average) for these models.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
llms vlms prompt optimization
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/d6b0cf06c989d94e541b9289413bf045dd58ce97.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/cb12045f2dbf2fc4cca8fa8ba58d82592c99b2f5.zip
GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission