Unleashing the Potential of Vision-Language Pre-Training for 3D Zero-Shot Lesion Segmentation via Mask-Attribute Alignment
Medical Image Segmentation;Vision-Language Pre-Training;Zero-Shot Segmentation
Yes
No ethics review needed.
Considering the practicality issue, would replacing GPT-4 with an open-source LLM for attribute construction (probably also without human annotators?) reaching a similar level of performance gain (as mentioned above)?
The idea of decomposing free-text reports into attributes for fine-grain training and inference is intuitive and feasible. It can significantly reduce the noise and ambiguity associated with unstructured free-text reports and yields improved performance on unseen lesions.

The key components and the detailed implementations are described in good detail. 

The proposed approach demonstrates performance improvements compared with previous zero-shot image segmentation methods.

Detailed ablation studies are presented to verify the effectiveness of key components.
This work presents a zero-shot medical image semantic segmentation framework that is based on maskformer and attribute-based visual-language feature fusion. To achieve fine-grain alignment between visual and textual concepts, the authors propose to decompose radiological reports into attributes describing lesions of interest, assisted by LLMs and human annotators. The obtained attributes allows fine-grain fusion and alignment between lesion features and textual concepts, yielding improved generalization performance upon unseen lesions. The proposed framework is evaluated on tumor segmentation tasks, and demonstrates improved segmentation results compared with existing zero-shot segmentation approaches on unseen lesions.
Despite stronger segmentation performance, the major framework looks similar to ZePT (Jiang et al., 2024): maskformer backbone + interacting visual and textual features that are obtained from detailed description of the lesions. Therefore, the key take-home message for readers may be a bit unclear: should the readers interpret the key technical contribution as explicitly decomposing textual descriptions/reports into categorized attributes for the ease of learning and inference?

Readers may argue that using GPT-4 for attribute construction from radiological reports is often infeasible in practice due to the privacy and legal concerns. The authors are encouraged to argue if switching to local open-source LLMs would yield a similar level of performance gain. 

The authors may want to decompose some of technical details in Figure 1 into separate figures and move them closer to corresponding paragraphs.
Yes
This paper utilizes private clinical data, which may require an ethics review to ensure compliance with privacy, security, and safety standards.
Yes, Privacy, security and safety
please refer to Weaknesses
1. The topic on zero-shot lesion segmentation is valuable, as clinical settings often involve diverse, emerging anomalies, and data collection is challenging, increasing the need for models that handle unseen diseases in an open-set context.
2. The authors strengthen their claims with both qualitative and quantitative results, enhancing the credibility of their findings and providing a well-rounded evaluation of the proposed approach.
3. The paper is well-organized, with a clear and logical structure that makes complex concepts accessible and easy to follow for readers.
This paper introduces Malenia, a framework for zero-shot lesion segmentation in 3D medical images, aimed at addressing the challenge of transferring image-level knowledge to pixel-level segmentation tasks. Building on advancements in medical vision-language pre-training, Malenia integrates a multi-scale mask-attribute alignment framework and a Cross-Modal Knowledge Injection module to link visual and textual features, enabling the model to handle previously unseen lesions. The authors evaluate Malenia's performance through experiments on three datasets (two public and one private) across 12 lesion categories.
1. The Zero-Shot Inference section lacks sufficient detail on how the model handles test CT images containing unseen tumor types. For instance, if a test image includes both a kidney tumor and a gallbladder tumor, it is unclear how the model predicts these different tumors within the same image. Additionally, the phrase “we obtain the class information for each predicted lesion mask by referencing a clinical knowledge table” is ambiguous. Further clarification is needed on what this clinical knowledge table entails and how it is used in the inference process.
2. The comparisons in Table 1 and Table 2 may not be entirely fair. The authors introduce additional ATTRIBUTE DESCRIPTIONS annotations, which provide extra information not available to baseline methods like TransUNet, nnUNet, and Swin UNETR. These baselines rely solely on image data, while the proposed approach leverages attribute annotations, giving it an advantage in performance comparisons.
3. The fully-supervised performance of baseline methods reported in Table 2 appears unusually low. For instance, in the official nnUNet paper, liver tumor segmentation achieved a Dice score of 76 on the test set, whereas the presented result here is only 61.33. Similarly, lung tumor segmentation originally reported a Dice score of 74, but this paper reports 54.44. This discrepancy raises concerns about the reproducibility and fairness of the comparisons.
4. In Table 7, the authors compare their method to existing vision-language pretraining strategies, but it is unclear how these methods were reproduced. Most vision-language pretraining approaches require access to diagnostic reports, which are not included in the public MSD and KiTS23 datasets. It would be helpful to understand how the authors conducted these comparisons without complete diagnostic reports. Additionally, the absence of comparisons with important pretraining models like CT-CLIP weakens the evaluation.
5. Scalability of the proposed method is questionable due to its reliance on ATTRIBUTE DESCRIPTIONS. For practical clinical applications, this approach would require physicians to provide additional lesion descriptions across eight visual attribute aspects, which may not be feasible in all settings. This dependency could limit the method’s applicability in diverse clinical environments.
Yes
No ethics review needed.
1. Could the authors provide insights on the computational efficiency of their framework?
2. This work and its motivation are similar to the previous works [1-2], please describe the differences between them.
[1] Li, Z., Li, Y., Li, Q., Wang, P., Guo, D., Lu, L., ... & Hong, Q. (2023). Lvit: language meets vision transformer in medical image segmentation. IEEE transactions on medical imaging.
[2] Huang, X., Li, H., Cao, M., Chen, L., You, C., & An, D. (2024). Cross-Modal Conditioned Reconstruction for Language-guided Medical Image Segmentation. arXiv preprint arXiv:2404.02845.
1. This study bridges the gap between mask representations and textual disease attributes for zero-shot lesion segmentation.
2. The empirical validation includes comprehensive experiments across multiple datasets and lesion types, demonstrating the robustness and effectiveness of Malenia.
The paper introduces "Malenia," a framework for zero-shot 3D lesion segmentation using vision-language pre-training methods. This framework aligns multi-scale mask representations of lesions with textual embeddings of disease attributes, which facilitates the segmentation of unseen lesion types in medical imaging.
1. While the paper effectively addresses 3D CT scans, it does not discuss the applicability of the Malenia framework to other imaging modalities like X-ray.
2. The paper does not discuss how the framework handles ambiguous or borderline cases where lesion boundaries are not well-defined or where lesions exhibit highly atypical appearances.
3. There is no thorough discussion on the limitations or failures of the proposed model.
Yes
No ethics review needed.
1. In Section 3.1, the authors employ the Multi-Positive InfoNCE (MP-NCE) loss for aligning mask embeddings with multiple attribute embeddings. Could the authors provide more details on how the MP-NCE loss is computed and how it facilitates learning with multiple positive pairs? Specifically, how are the positive and negative pairs defined in the context of multiple attributes?
   Could the authors explain how this formulation effectively handles multiple positive pairs for each mask token?
2. The model uses a fixed number of mask tokens ($N=16$) and defines $R=8$ attribute aspects. Have the authors conducted any experiments to assess how varying $N$ or the number of attributes affects the model's performance? An ablation study on these hyperparameters could provide insights into the model's sensitivity and optimal settings.
3. As noted in Appendix D, the model performs less well on attributes like "Surface Characteristics" and "Specific Features". What are the potential reasons for this lower performance? Could incorporating additional contextual information or advanced features help the model better capture complex visual semantics?
1. The proposed Malenia framework is novel, particularly in its use of multi-scale mask-attribute alignment for 3D lesion segmentation. This is a meaningful extension of zero-shot segmentation methods to the medical domain, where unseen lesion categories are common (e.g., "hepatocellular carcinoma," mentioned in Sec. 1).
2. The CMKI module is a strong contribution because it combines visual and textual embeddings, enriching the segmentation prediction with complementary information from both modalities. This is a novel approach that enhances the zero-shot capability of the model.
3. The paper introduces a multi-positive contrastive loss to fine-tune the alignment between lesion masks and textual attributes. This approach allows the model to generalize better to unseen lesions by learning from shared visual attributes, which distinguishes it from prior works like SAM (Kirillov et al., 2023).
The paper introduces Malenia, a novel framework specifically designed for 3D zero-shot lesion segmentation in medical images. The authors aim to address the challenges of transferring image-level knowledge to pixel-level tasks—such as lesion segmentation—by proposing a multi-scale lesion-level mask-attribute alignment approach. Malenia improves the alignment between the visual features of unseen lesions with textual representations, leveraging Cross-Modal Knowledge Injection (CMKI) to enhance both visual and textual embeddings. Experimental results across three datasets demonstrate that Malenia outperforms state-of-the-art (SOTA) methods in zero-shot lesion segmentation.
1. The method relies on transforming patient reports into structured descriptions of eight visual attributes, which requires expert radiologist involvement. This process may not be scalable or practical in real-world applications where such annotations are not readily available.
2. As noted in Appendix D, the model performs less effectively on attributes requiring complex visual semantic understanding, such as "Surface Characteristics" and "Specific Features". This suggests a potential limitation in capturing intricate visual patterns.
3. While the model shows strong performance on 12 lesion categories, the diversity of lesions in clinical practice is vast. The ability of Malenia to generalize to a wider range of unseen lesions without additional training remains uncertain.
We introduce Malenia, a novel multi-scale lesion-level mask-attribute alignment framework designed for superior zero-shot lesion segmentation in 3D CT scans.
@inproceedings{
anonymous2024unleashing,
title={Unleashing the Potential of Vision-Language Pre-Training for 3D Zero-Shot Lesion Segmentation via Mask-Attribute Alignment},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=QG31By6S6w},
note={under review}
}
Recent advancements in medical vision-language pre-training models have driven significant progress in zero-shot disease recognition. However, transferring image-level knowledge to pixel-level tasks, such as lesion segmentation in 3D CT scans, remains a critical challenge. Due to the complexity and variability of pathological visual characteristics, existing methods struggle to align fine-grained lesion features not encountered during training with disease-related textual representations. In this paper, we present Malenia, a novel multi-scale lesion-level mask-attribute alignment framework, specifically designed for 3D zero-shot lesion segmentation. Malenia improves the compatibility between mask representations and their associated elemental attributes, explicitly linking the visual features of unseen lesions with the extensible knowledge learned from previously seen ones. Furthermore, we design a Cross-Modal Knowledge Injection module to enhance both visual and textual features with mutually beneficial information, effectively guiding the generation of segmentation results. Comprehensive experiments across three datasets and 12 lesion categories validate the superior performance of Malenia. Codes will be publicly available.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Medical Image Segmentation Vision-Language Pre-Training Zero-Shot Segmentation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/3ec0b779ebf66ec83a20d937424c6f95371259bc.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Unleashing the Potential of Vision-Language Pre-Training for 3D Zero-Shot Lesion Segmentation via Mask-Attribute Alignment
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission