Prompt as Knowledge Bank: Boost Vision-language model via Structural Representation for zero-shot medical detection
Vision-language model;  zero-shot enhancement; Structural Representation
Thanks for your careful review! 

We will try our best to demonstrate and improve our method soon!
Thanks for your careful review! We will try our best to reply to your concern regarding our method soon!
Thanks for your detailed review! We will organize our feedback very soon!!
Thanks for your detailed review! We will organize our feedback very soon!!
Yes
No ethics review needed.
(1)	To what extent can the findings be generalized to other medical imaging modalities or less common diseases? Are there plans to evaluate the model on broader datasets?
(2)	Besides Average Precision, what other metrics were considered for evaluating model performance? Are there plans to incorporate user feedback or clinical outcomes in future evaluations?
(3)	This paper focuses on zero-shot medical detection, whereas GLIP was initially developed for natural images. Can the proposed method also be applied effectively to natural image datasets?
(1)	The paper introduces an effective structural representation by encoding prompts into a knowledge bank and utilizing a dual-branch structure. This approach enables adaptive and context-aware alignment, which is particularly advantageous for complex medical detection tasks.
(2)	StructuralGLIP outperforms traditional zero-shot models by effectively handling both instance-level and category-level prompts, achieving significant improvements across various benchmarks in endoscopy, microscopy, radiology, and more.
(3)	By allowing for zero-shot enhancement, the model can be fine-tuned and then further improved with category-level prompts, a feature well-suited for dynamic medical settings where data annotation is scarce.
In this study, the authors introduce StructuralGLIP, a novel approach to zero-shot medical detection using vision-language models (VLMs). This method leverages structured representations within a dual-branch architecture that enables nuanced alignment between images and textual prompts, significantly enhancing the model's adaptability to new medical scenarios without needing annotated data. StructuralGLIP uses category-level prompts, maintained in a latent knowledge bank, and a mutual selection mechanism for precise cross-modal fusion, thus improving accuracy across diverse medical imaging datasets.
(1)	The proposed dual-branch structure with a knowledge bank requires complex engineering and computational resources, potentially limiting its accessibility for practitioners in less resource-rich environments.
(2)	The paper may not adequately address the potential data imbalance present in the datasets used for evaluation. Some diseases or conditions may have significantly fewer examples, which could impact the model's performance and generalizability.
(3)	The model's inner workings, particularly regarding how it selects and utilizes prompts, may be difficult for practitioners to interpret, limiting trust in its decisions and making it harder to diagnose potential failures.
(4)	Despite improvements in alignment, there may still be instances of misalignment between visual features and prompts, especially in cases of atypical presentations, which could lead to missed detections.
Yes
NA
No ethics review needed.
1. How to evaluate the quality of the generated prompts by VQA method. As some previous work pointed out, the VLM without medical domain adaptation perform poorly on some medical datasets, especially for radiology datasets.
2. For Prompt as Knowledge Bank ablation study seciton, I would like to see an experiment on the whether adding noisy knowledge (for example, knowledge for another target) would sharply downgrade the StructureGLIP performance. This experiment would the the robustness of the mutual-selection process.
1. This work is aiming for a vital issue in medical image understanding field, which is the generalization capability of foundation models with limit data access. The motivation of improving the existing work is clear and strong, which is the lack of object level prompt and fail to capture the various feature of images during prompt desgin.
2. This work present a novel but efficient method, called StructualGLIP, to increase the model's zero-shot/few-shot performance on various datasets. The desgin of StructualGLIP introduce the knowledge bank and mutual selection process to help prompt design process. This method address several shortcomings of current method and is novela and effective.
3. This method largely increased the zero-shot performance on different medical image datasets across different modalities.
This paper introduce a new zero-shot method for medical Vision-Language Models(VLMs) for detecting unknown targets. To address the prompt umatched with the variations in the medcial images, the authors propose a StructualGLIP desgin with a main and an auxiliary branch encoders for text input and introduce a mutual selcetion mechanism. 
The author explain that the auxiliary branch would work as a knowledge bank where the main branch can extract latent prompt tokens, while the tokens in the knowledge banked are filtered by the mutual selection process.
Overall, the motivation, method, and performance of this work is good enough, but I still need some explaination for some detail, please refer to the weakness and question section. I will consider adjust my rating based on the authors response.
1. Line 234, the sentence seems not finished. 
2. One of the major problem of the proposed method is not trainable as the Top-K selection operation is non-differentiable, while previous work is differentiable and thus finetuning would result in better performance. I would suggest include Reparameterization Trick for Gumbel-Softmax to improve your method. Though this work is good enough as a stand-alone method for zero-shot detection. But I see a potential to achieve better performance.
Yes
No ethics review needed.
1. How does the prompt generation module ensure the creation of valuable prompts for unseen classes without introducing distracting information?
2. How is the accuracy of instance-level contextual prompts generated from visual input for VQA ensured, as this seems crucial to the final experimental performance?
3. How does using alternative LLMs affect the experimental results?
1. By introducing a dynamic knowledge bank that selects the most relevant prompts, the model achieves more flexible and accurate vision-language alignment.
2. This method performs exceptionally well in unsupervised and few-shot scenarios, maintaining high detection accuracy even with limited labeled data.
3. The prompt generation module extracts information from prior knowledge of unseen classes, enhancing the model’s adaptability and making it suitable for tasks across various domains.
This paper presents Prompt as Knowledge Bank, a method that encodes prompts into an adjustable knowledge bank, enhancing multimodal models’ accuracy in zero shot tasks by dynamically selecting prompts from prior knowledge. This approach achieves robust performance in high-precision fields like medical imaging, even with limited data.
1. The introduction of the knowledge bank and prompt generation module increases computational costs, raising demands on hardware resources.
2. The model heavily relies on prompt quality, and low-quality prompts may negatively impact its performance.
3. The impact of the prompt’s LLM on performance is apparent; however, the paper does not analyze how the choice of LLM (e.g., LLaMA, Gemini) affects the results.
4. The complex prompt selection and knowledge bank structure reduce the transparency of the model's decision-making process, posing challenges for clinical applications.
5. There are formatting errors in the paper, such as an unexpected horizontal line near the number 15 in the table 5.
Yes
No ethics review needed.
Please address Weaknesses #1,5,6.
1. the paper demonstrates a comprehensive evaluation of four medical image modalities.
2. the paper's representation and structure are mostly clear and easy to follow (although with some language and word choice issues, which will be discussed below).
This paper introduces a zero-shot object detection model, StructrualGLIP, for medical image-based object detection tasks. The method uses a knowledge bank that stores encoded text prompts, which are later used to select and match relevant image features to form fine-grained structural representations to allow better alignment of image features with the prompt information, achieving accurate, context-aware detection.
1. The claimed novelty appears to be on the latent knowledge bank and its function to store encoded prompt tokens and later be used at each encoder layer as a vision token selector and vice versa. This is coined as a mutual selection process. The selected information is then merged into the original image and text encoder back at each layer. The entire process is different to a standard contextual prompt method (Fig. 1(a)) but it feels like a quite incremental difference which I don't see much novelty.
2. The word choice of "medical detection" bugs me, in medical science detection refers to "detecting diseases" whereas here this is object detection in medical images so this may cause confusion to certain readers.
3. Line 234 looks unfinished.
4. Eq. 6&7, perhaps the Top-P/Q^{max} function can be simplified by using argmax/argsort function?
5. L201/534: "... fine-grained alignment between target descriptions and medical images", L:256 "forming fine-grained structural representations", can the authors clarify what "fine-grained" refers to in those places?
6. L294: "...like BLIP Li et al. (2022a)", is the VQA model BLIP or not? Have you considered other VQA models and would the performance of other VQA models fluctuate your detection results?
7. L177: I don't entirely agree that zero-shot detection has a real-world clinical need as the clinicians I've encountered would not trust zero-shot settings, in the medical domain, accurate detection/segmentation/diagnosis is the the most important thing.
@inproceedings{
anonymous2024prompt,
title={Prompt as Knowledge Bank: Boost Vision-language model via Structural Representation for  zero-shot medical detection},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=l0t2rumAvR},
note={under review}
}
Zero-shot medical detection can further improve detection performance without relying on annotated medical images even upon the fine-tuned model, showing great clinical value. Recent studies leverage grounded vision-language models (GLIP) to achieve this by using detailed disease descriptions as prompts for the target disease name during the inference phase.  
However, these methods typically treat prompts as equivalent context to the target name, making it difficult to assign specific disease knowledge based on visual information, leading to a coarse alignment between images and target descriptions. In this paper, we propose StructuralGLIP, which introduces an auxiliary branch to encode prompts into a latent knowledge bank layer-by-layer, enabling more context-aware and fine-grained alignment. Specifically, in each layer, we select highly similar features from both the image representation and the knowledge bank, forming structural representations that capture nuanced relationships between image patches and target descriptions. These features are then fused across modalities to further enhance detection performance.
Extensive experiments demonstrate that StructuralGLIP achieves a +4.1\% AP improvement over prior state-of-the-art methods across seven zero-shot medical detection benchmarks, and consistently improves fine-tuned models by +3.2\% AP on endoscopy image datasets.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Vision-language model;  zero-shot enhancement; Structural Representation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/a6437e9296645470c7780702d7700a8aacd8288b.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Prompt as Knowledge Bank: Boost Vision-language model via Structural Representation for  zero-shot medical detection
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission