LoGra-Med: Long-Context Multi-Graph Alignment for Medical Visual-Language Models
multi-modal LLM;AI for Healthcare;multi-modal learning
Yes
No ethics review needed.
1. I’m actually a bit confused on why the graph perspective is needed here. It seems the paper is solving the exactly the same problem as optimal transport, but all these optimal papers do not talk about graphs? 
2. I wonder, without using the verbose version, how’s this paper different from the PLOT paper, since it’s essentially solving a matching problem between 2 domains?
3. I find counterintuitive that DCI provides mixed results. What’s the explanation here? To be honest, I would be totally fine if those “+DCI” are not there. But if the authors decide to add these experiments, then they need to explain why utilizing multi-scale features are harmful for some cases.
I like how the paper motivates the necessity of the proposed algorithm from the perspective of training efficiency, which is quite refreshing. The experiments also validate that the proposed algorithm indeed results in better learning outcomes at low data regimes and even does so when using all the pretraining data. 

The presentation of the core methodology flows very well, with tight logical connections. The gradient-based method to by-pass the high time complexity of the solver also seems reasonable.

The paper also seems to be quite comprehensive in terms of baselines. The ablation studies are satisfactory, especially the one without “long-context”.

The experiment details in the supplemental material is very comprehensive, including important details such as system prompt and evaluation protocols.
The paper proposes a multi-graph matching objective to enhance the vision-language alignment in generative VLMs for medical tasks. The authors motivate the need for such an auxiliary learning objective from a training efficiency point of view, showing that the next token prediction loss function demands a large amount of visual instruction tuning data that's not easy to acquire. To reduce the sample complexity, this paper proposed their method of treating visual features, text features, and augmented text features (based on a more verbose version of the ground truth answer) as graphs and letting the VLM learn to match between the triplets. Since solving a multi-domain graph alignment problem is computationally expensive, the authors propose a more scalable solution by separately aligning each domains with a barycenter graph. The experiments in the paper show that multi-graph alignment significantly improves upon LlaVA-Med, at both low pretraining data (10%) and full data regimes across a wide range of tasks.
I have concerns regarding the core methodology. 

1. The paper seems to have skipped simpler solutions that might as well worked and jumped right into a complex proposal. In section 3.4, the authors argue the necessity of their scalable multi-graph alignment algorithm by saying pairwise graph alignment requires K choose 2 pairs. But in this specific case, K is 3 and K choose 2 is really only 3 pairs. From my point of view, this really should be the method to be tried first before moving on to more complex formulations. 
2. I am too sure what leads to LoGRA’s success due to how the visual & text representations are obtained. In its essence, the representations from both modalities are sets with varying number of elements. For images, this will be different number of patches due to different resolutions; for text, it’s whatever the number of tokens the ground truth answer corresponds to. The authors are simply taking a naive average over the elements. I really doubt the average of word embeddings / visual tokens will end up giving any meaningful representations. I would like to see some analysis on this design choice, for example comparing the distribution of cosine similarity (for example) between positive and negative pairs, so that I know the proposed objective is really forcing the alignment instead of picking up some other signals. 
3. As far as I know, most medical datasets are highly imbalanced, but the paper seems only to report the overall accuracy instead of per-disease macro scores. This is a bit concerning since if all the alignment algorithm does is to guild the model to favor the majority class, we will get a model with high accuracy but no practical use. I think for classification tasks, macro F1 scores should be reported. For VQA tasks, per-disease scores should be reported for datasets that include a class-label for the image-question-answer pair.
4. I think the word “long-context” is misused: from the examples, it looks like the so called “long-context” texts are no more than a few hundred words. Based on the context of the paper, the authors really meant a more verbose / extended rewrite of the original ground truth answers as an augmentation for the triplet learning. A better terminology should be used to distinguish this paper from works dealing with long-context tasks, which usually consumes at least tens of thousands of tokens for each query.
Yes
NA
No ethics review needed.
1. Why extending the short answer to longer-context, why would that beneficial the pre-training? 
2. Have you run any qualification test on the GPT generated text?
3. Would you mind to clarify, when you mentioned "10% data", you mean, "10% datasize for the stage2 instruction-tuning stage, but the datasize for the stage1 pre-trianing is still fullzie". Is that correct?
1. This model propose a method to convert auto-gressive finetuning to Graph-alignment task, a fresh perspective. This seems interesting but lack of motivation.
2. The results showing that the proposed method is definetly increase the performance on different tasks with limited datasize. This is quite important and might showing a new way except data curation.
3. Some of the detail are not well-explained in the paper, please refer to the weakness and quesiton section. I will consider adjust my scores based on the authors response.
This work propose a method that aiming for decrease the data needed for medical Multi-Modal large langauge model pre-training by enforcing triplet correlations on the latent embedding among image, descriptions, and captions. How to efficiently pre-train large models with limited data in medical domain is a quite important topic, and thus the motivation is good enough. However, the reason behind applying multi-gprah alignment is not well-explained. And this process required retrain the LLM model, which is more costful compared other methods.
1. The motivation of adding graph alignment task in the pre-training stage is not explained.
2. This paper also cite the work of MedTrinity(Xie et al.2024), but you didn't include their results in your comparison tables for your experiments. Their model's performance on some dataset are better than yours, you should include them to give readers a fair comparision. You may argue that their training data is larger than yours. But your method also introduce augmented description data generated by GPT-4.
3. SInce you introduced more data by extending the short answer to long answers. So, when you finetuning the LLavaMed basedline with 10% data, do you also include the extended version answer?
Yes
No ethics review needed.
1. Can you provide the hyperparameters used in the paper, including the batch size B and the value of k used in the k-NN?

2. Are there any experiment results presented in the paper that can prove the performance gain compared to LLaVA-Med is due to the graph alignment algorithm itself rather than additional data or extra computing?
1. The empirical results demonstrate that the proposed approach can reduce the demand for pre-training data for MLLM alignment, outperforming previous approaches.

2. The authors conducted a set of ablation studies, which demonstrated the contribution of each component.

3. The visualization in the manuscripts helped me to better understand the proposed approach and experiments.
In this paper, the authors introduced a self-supervised multi-graph alignment objective strategy for pre-training medical MLLM. Specifically, the proposed method aligns the triplet consisting of the input image, its instruction data, and its extended long context generated by GPT. The graphs of images, the instruction data, and the long context are trained to align with the barycenter graph. The authors perform experiments on multiple downstream tasks, and the proposed method outperforms baseline methods. The proposed method achieved better performance with the limited amount of pre-training dataset.
1. From Table 5, it seems that the generated long context data plays an important role in LoGra-Med. Without it, the performance is slightly lower than the baseline LLaVA-Med. Since LLaVA-Med doesn't use the generated long context data, these results cast doubt on the contribution of the graph alignment algorithm. For a fair comparison to validate that the performance gain is not mainly due to the extra data, the author could also pre-train LLaVA-Med with the long context data and compare the performance.

2. The proposed method's performance gain may come from the extra computing in training (the authors mention that the training time is longer than the baseline method by 1 hour) rather than the method itself. For a fair comparison, the authors may pre-train the LLaVA-Med baseline method using some self-supervised objectives (e.g., InfoNCE) or additional long context data for 1 hour.

3. The authors need to mention some of the important hyperparameters used. For example, what's the batch size B? I would assume the larger the batch size, the larger the graph. And what's the value of k used in the k-NN algorithm for building edges?
Yes
No ethics review needed.
The sensitivity of LOGRA-MED to the quality and consistency of GPT-4 generated contexts remains unclear？ 

how would the method perform with simpler context extension approaches or with different language models? 

The paper could elaborate on the interplay between the traditional autoregressive loss and the graph alignment objective, particularly regarding their relative contributions to the final performance？

The choice of hyperparameters in the graph construction phase, especially the selection of k in k-nearest neighbors, appears critical but lacks detailed justification？
The technical innovation of LOGRA-MED lies in its formulation of vision-language alignment as a multi-graph matching problem, generalizing beyond traditional pairwise contrastive learning approaches. The theoretical foundation is particularly strong, proving both metric and geodesic properties of the graph alignment distance. The implementation cleverly circumvents the NP-hard nature of multi-graph matching using Lagrange decomposition and efficient heuristic solvers. The empirical results are comprehensive, demonstrating that with only 10% of training data, LOGRA-MED achieves 72.52% accuracy on VQA-RAD, nearly matching LLAVA-Med's 72.64% with full data. The method's effectiveness extends across various tasks including medical visual chatbot capabilities (44.82% overall score) and zero-shot classification on 23 datasets spanning microscopy, CT, and CXR modalities.
This paper introduces LOGRA-MED, a novel approach addressing the data efficiency challenge in medical multi-modal large language models (med-MLLM) through multi-graph alignment. The work reveals that current autoregressive training methods like LLAVA-Med require extensive instruction-following data, showing a significant performance drop from 72.64% to 52.39% on VQA-RAD when reducing training data to 10%. To address this, LOGRA-MED enforces triplet correlations in the latent space among image features, instruction data, and GPT-4 generated extended contextual captions using a structure-aware graph matching framework. The method efficiently handles the combinatorial complexity of multi-graph alignment through an implicit maximum likelihood estimation approach with black-box gradient computation.
The reliance on GPT-4 for generating extended contexts raises questions about potential biases and the method's generalizability. While the paper demonstrates improved performance, the computational complexity analysis could be more thorough - the reported one-hour additional training time compared to LLAVA-Med merits deeper discussion given the sophisticated graph matching operations. The variable performance across different medical domains (particularly in PathVQA and some zero-shot classification tasks) suggests potential limitations in handling diverse medical contexts. The ablation studies, while informative, could benefit from exploring different graph construction strategies beyond k-nearest neighbors and analyzing the impact of various distance metrics in the graph matching formulation.
multi-graph alignment algorithm to train medical multi-modal LLM
@inproceedings{
anonymous2024logramed,
title={LoGra-Med: Long-Context Multi-Graph Alignment for Medical Visual-Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=SOsotxYtPC},
note={under review}
}
State-of-the-art medical multi-modal large language models (med-MLLM), such as LLAVA-MED or BIOMEDGPT, leverage instruction-following data in their pre-training stages. However, those models primarily focus on scaling the model size and data volume to boost performance while mainly relying on the autoregressive learning objectives. Surprisingly, we reveal that such learning schemes might result in a weak alignment between vision and language modalities, making these models highly reliant on extensive pre-training datasets — a significant challenge in medical domains due to the expensive and time-consuming nature of curating high-quality instruction-following instances. We address this challenge with a new multi-graph alignment algorithm, namely LOGRA-MED, which enforces triplet correlations on the latent embedding space among image modalities, conversation-based descriptions, and extended contextual captions. Owing to this technique, the model is encouraged to capture the semantic meaning of the context, handle linguistic variability where the captions or questions may differ from training instances, and learn cross-modal associations, linking visual elements with various textual interpretations. To scale our algorithm to the med-MLLM setting, we also design an efficient end-to-end learning scheme based on advanced black-box gradient-estimation techniques that permit fast forward and backward steps through the LLM model (LLaMa 7B). Empirical results show
that we can match the performance of LLAVA-Med pre-trained on 600K image-text pairs from PMC-15M for Medical VQA tasks and significantly outperform it when trained on only 10% of the data. For instance, on VQA-RAD, we exceed LLAVA-Med (both trained on 10%) by 20.13% and achieve near parity with the 100% pre-training setting (72.52% vs. 72.64%). Additionally, we also surpass other SOTA pre-training methods and med-MLLM such as BIOMEDGPT on visual chatbot or RADFM on zero-shot image classification with VQA, showcasing the power of multi-graph alignment in improving vision-language integration for medical-MLLM.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
multi-modal LLM AI for Healthcare multi-modal learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/e53b5c915b3400aca5f0685a69f96186db3ae350.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
LoGra-Med: Long-Context Multi-Graph Alignment for Medical Visual-Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission