RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognition
Multimodal Large Language Models
Yes
No ethics review needed.
The questions are listed in the “weakness” section.
1.The proposed RAR framework is simple yet effective, making it easy to understand and implement.

2.The authors conducted extensive experiments. For each of their statements, motivations, and methods, they provided thorough experimental support. 

3.The authors expanded their method to include object detection tasks (it can be regarded as a form of post-processing), not limited solely to fine-grained object recognition. This provides valuable insights for future research.

4.The model demonstrates improved performance on fine-grained visual recognition tasks and object detection tasks.
This paper introduces RAR framework to address limitations in fine-grained recognition by combining CLIP’s class candidates recognition with the fine-grained classification abilities of MLLMs. The authors claim that CLIP struggles with distinguishing subtle differences, while MLLMs, despite extensive pre-training, face limitations with increasing category complexity and context window size. Therefore, RAR uses a CLIP-based retriever to store memory of categories and retrieve top-k class candidates for MLLMs to rank and predict. RAR enhances MLLMs’ recognition capability, achieving improvements across fine-grained benchmarks and object detection tasks.
1. Relatively Limited Novelty. 
The authors use off-the-shelf models, such as CLIP and LLaVA, for fine-grained vision recognition and object detection. However, “multi-query re-ranking techniques” in RAG have already been widely adopted, for example, in Re2G (Retrieve, rerank, generate) [1] and RankRAG [2]. I did not observe any specific improvements in the Retrieving or Ranking strategy tailored to the fine-grained recognition task. This limits the novelty of the proposed framework.

2. Concern of Practical Application. 
Although FineR previously demonstrated the use of large models for fine-grained vision recognition, I still question the necessity of using large models. Why not use specialized/expert models with much smaller scale to accomplish this task? To my knowledge, these expert models already perform well on the tasks evaluated by the authors. Based on this concern, I believe that using MLLM for this task should allow for open-set responses (such as providing interpretability) rather than simply using MLLM for re-ranking final predictions.

3. Need more discussion of fine-tuning for re-ranking.
I noticed that the re-ranking operation requires fine-tuning the MLLM to achieve satisfactory performance. Although the authors claim that "RAR is not sensitive to changes in the fine-tuning dataset for ranking," they only conducted experiments on FGVC-Aircraft and Stanford-Cars. I believe this may be insufficient. If using a dataset with severe domain gap with the test datasets, catastrophic forgetting might occur, and I suggest that the authors discuss this issue.

[1] Glass, Michael, et al. "Re2G: Retrieve, rerank, generate." arXiv preprint arXiv:2207.06300 (2022).
[2] Yu, Yue, et al. "Rankrag: Unifying context ranking with retrieval-augmented generation in llms." arXiv preprint arXiv:2407.02485 (2024).
Yes
No ethics review needed.
The paper mentions, "Although the brute force method is inherently straightforward, its efficiency markedly diminishes as the dataset escalates to the magnitude of millions of embeddings." The HNSW method can significantly improve retrieval speed. So, how can we quantify the complexity of brute force retrieval and HNSW when the dataset scales to millions of embeddings? Please provide specific comparative results.
RAR’s design, particularly the use of CLIP-based retrieval augmented by MLLM ranking, is well-founded and thoughtfully justified. The use of retrieval augmentation with a multimodal memory structure effectively reduces the dependency on extensive context windows, which is a known limitation in handling large vocabularies.
This paper introduces RAR (Retrieving And Ranking), an approach to enhance Multimodal Large Language Models (MLLMs) with a retrieving and ranking augmentation. The proposed technique aims to address the challenges faced by models like CLIP and MLLMs when applied to fine-grained visual recognition tasks and datasets with large vocabularies. By integrating a multimodal retriever and leveraging a ranking mechanism, RAR seeks to improve zero-shot and few-shot recognition accuracy across diverse datasets.
Incorporating a detailed error analysis on retrieval failures or ranking misclassifications would provide insights into areas where RAR may need refinement, especially regarding failure cases in subtle category differentiation.
Yes
No ethics review needed.
When the K of K-NN is determined, the upper limit of the method's capability is determined by CLIP (whether the prediction result is in topk), and MLLM only helps the method approach this upper limit. In some cases, even the performance of CLIP will limit the play of MLLM (if the correct category is not in topk). Also, poor MLLM performance can also undermine CLIP's capabilities (see Table 2 Flower102 dataset).

Therefore, I want to see the ablation experiments under different k settings. In-context learning can be used if SFT is too time-consuming.
1.The paper is well-written, and the method's details are well-explained. The drawings are appealing and intuitive. 
2.The experiments are comprehensive and conducted on a sufficient number of datasets, and improvements have been achieved on many datasets.
3.The paper includes a complete evaluation and ablation study to understand the impact of the propose components.
The paper addresses the issues of CLIP's weak fine-grained category classification capabilities and MLLM's limitations when dealing with extensive vocabularies and fined-grained categorizations, by proposing a method named RAR. Specifically, it involves pre-storing features information about each category's images or labels in the dataset through CLIP's encoder. This stored information is then matched with the images’ features that need to be classified using either image-image kNN or image-text kNN methods to select the top-k targets. These top-k candidate targets are subsequently fed into MLLM for ranking, ultimately yielding the final prediction results.
The method does not fundamentally address the fine-grained categorization and extensive vocabulary classification issues of CLIP and MLLM. Essentially, it is more like a simple combination of the two, which lacks novelty and not suitable for ICLR.

1. Although the paper’s title is about augmenting MLLM, it is mostly just a simple application of MLLM. The SFT and in-context learning techniques used in the thesis only improve the MLLM's ability to maintain output formats, but do not actually enhance its classification ability. This is why the SFT effects on different datasets in Table 5 are not significantly different.

2. The motivation was the improvement in fine-grained categories and vast vocabulary classification for CLIP and MLLM, but since no substantial optimization was done for CLIP and MLLM, and their classification abilities for this scenario are relatively weak, so the improvement of their combination is not significant, as shown in Table 1 against FineR and Table 5. The reason why the improvement was significant in Table 4 was because the object categories in LVIS were 1,000, which was not enough, and could not effectively prove the motivation.

3. The performance increase in Table 2 is all compared with CLIP+KNN, which makes people feel that RAR improves CLIP rather than MLLM, and the performance increase here is suggested to be worse than the sub-optimal result.
Yes
No ethics review needed.
- Could you clarify why vanilla LLaVa results are not included in Table 1? Additionally, for zero-shot image classification, is there a comparison to vanilla CLIP using only category names?
- Well-written and straightforward
- Easy to understand with a clear methodology
- Achieves strong results on the proposed benchmark
- Thoroughly tested across multiple visual benchmarks, including fine-grained classification, few-shot recognition, and zero-shot object detection
This paper introduces RAR (Retrieving and Ranking), a method to enhance multimodal large language models (MLLMs) in visual recognition tasks. RAR combines CLIP’s broad retrieval ability with MLLMs' fine-grained differentiation abilities. It retrieves candidate categories from external memory based on the input image, which the MLLM then ranks to make a final prediction. Extensive experiments show that RAR significantly improves performance across various visual benchmarks, including fine-grained classification, few-shot recognition, and zero-shot object detection.
- Limited novelty in the proposed method.
@inproceedings{
anonymous2024rar,
title={{RAR}: Retrieving And Ranking Augmented {MLLM}s for Visual Recognition},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=ayg1PztmXP},
note={under review}
}
CLlP (Contrastive Language lmage Pre-training) uses contrastive learning from noise image-text pairs to excel at recognizing a wide array of candidates, yet its focus on broad associations hinders the precision in distinguishing subtle differences among fine-grained items.Conversely, Multimodal Large Language Models (MLLMs) excel at classifying fine-grained categories, thanks to their substantial knowledge from pre-training on web-level corpora. However, the performance of MLLMs declines with an increase in category numbers, primarily due to growing complexity and constraints of limited context window size.To synergize the strengths of both approaches and enhance the few-shot/zero-shot recognition abilities for datasets characterized by extensive and fine-grained vocabularies, this paper introduces RAR, a Retrieving And Ranking augmented method for MLLMs. We initially establish a multi-modal retriever based on CLIP to create and store explicit memory for different categories beyond the immediate context window. During inference, RAR retrieves the top-k similar results from the memory and uses MLLMs to rank and make the final predictions. Our proposed approach not only addresses the inherent limitations in fine-grained recognition but also preserves the model's comprehensive knowledge base,significantly boosting accuracy across a range of vision-language recognition tasks. Notably, our approach demonstrates a significant improvement in performance on 5 fine-grained visual recognition benchmarks, 11 few-shot image recognition datasets, and the 2 object detection datasets under the zero-shot recognition setting.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Multimodal Large Language Models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/5b5fd15732f4ee617e2de8710419d20897160832.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognition
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission