CycleAug: Cycle-Consistent Visual Augmentation for Large Multimodal Models
large multimodal models; synthetic data; data augmentation
Yes
No ethics review needed.
1. Besides using detailed LLM-generated captions to prompt SD, have you tried generating an image with SD where the starting point is a noisy anchor image? I believe this could also help maintain consistency with the QA pairs.
1. The paper is well written and organized.
2. The method is simple and easy to follow and replicate.
The paper presents a data augmentation framework called CYCLEAUG, which generates multiple images paired with existing QA pairs using Stable Diffusion. Several post-processing mechanisms are employed to ensure consistency between the generated images and the anchor QA pairs. Experimental results demonstrate slight improved performance of MLLMs when incorporating this augmented data for VQA tasks.
1. The main issue is that the paper aims to address the limitations of images generated by SD, specifically "homogeneous content" and the "can not handle negation word" (lines 253-257), but the work does not have any concrete efforts to resolve these problems.
2. The method is not well-suited for text-rich VQA, as the current SD model struggles with generating characters.
3. The performance gain is quite limited; for example, in Table 3, the augmented images even have a negative impact on the VQAT dataset.
4. The paper would benefit from reporting the cost (e.g., FLOPS, latency) required to produce these images, as well as the proportion of images filtered out by cycle consistency sampling.
Yes
No ethics review needed.
Refer to Weakness.
1. The idea of using cycle-consistency for MLLMs for data augmentation is novel.  While data augmentation and synthetic data generation are established fields, CYCLEAUG advances these techniques by ensuring semantic alignment between generated images and their question-answer pairs, effectively addressing quality control and enhancing robustness in synthetic data generation.

2. The paper is well-organized and easy to follow.

3. The proposed method can be seamlessly integrated with existing visual instruction tuning frameworks, and its modular design holds potential for adaptation and extension to a broader range of multimodal tasks.
This paper presents CYCLEAUG, a data augmentation framework for MLLMs that generates diverse, cycle-consistent synthetic image-question-answer triplets. By ensuring alignment between generated images and their corresponding QA pairs, CYCLEAUG enhances model robustness and generalization without requiring additional real-world data. Experiments verify the effectiveness of the proposed methods.
1. The author does not provide a detailed discussion on the specific computational costs and resource consumption of the proposed method.  This could pose challenges for actual application environments with limited resources.

2. The paper presents a method for creating diverse synthetic images from multi-turn question-answer dialogues but lacks an in-depth exploration of alternative dialogue generation strategies, which could restrict the analysis of image diversity and relevance, impacting the overall model performance.

3. In addition to the direct sampling method, there are numerous data augmentation techniques applicable to the field of multimodal learning.  Incorporating comparative experiments would aid in a more comprehensive assessment of CYCLEAUG's performance and potentially reveal its advantages.
Yes
No ethics review needed.
1. When generating image captions,  integrating the question-answer pair into the prompt  may improve the proportion of consistent triplets in all generated images. I hope the authors can analyze the potential trade-offs of this suggestion, such as its impact on image diversity or whether it introduces other effects into the framework.
2. Related to weakness (1). What is the impact of different MLLM and diffusion models on experimental results? I hope the author can provide additional experiments and analysis on how various MLLM and diffusion models will affect the final performance on the benchmark, and whether non-diffusion generative model architectures have any impact on the effectiveness of the framework.
(1) This paper propsed a novel data augmentation framework for visual instruction tuning. By leveraging cycle-consistency sampling, the framework can generate diverse synthetic images based on existing IQA anchor triplets.
(2) The proposed method is simple and intuitive. The authors validate their method across various benchmarks, demonstrating the effectiveness of the framework.
(3) The paper is generally well written.
This paper propose a novel data augmentation framework for visual instruction tuning.  The framework uses MLLM to generate detailed captions for the original images, and then generate synthetic image based on these captions. Question-answer pairs of the origin images are re-verified on the synthetic image to select the well-aligned synthetic image and question-answer triplets, thus achieving cycle-consistent visual augmentation. Experiments shows the framework can improve model performance on multiple visual question answering benchmarks without additional real data, confirming the effectiveness of the augmentation .
(1) Performance should be evaluated using more diverse backbones, which can better assess the robustness and effectiveness of the proposed framework.
(2) There are some works that have explored the cycle consistency in the VQA area. The approach presented in [1] involves Cycle-Consistency for Robust Visual Question Answering, wherein the model is trained not only to generate answers to given questions but also to formulate questions conditioned on specific answers. This methodology ensures that the answers predicted for the generated questions are consistent with the true answers corresponding to the original questions. This paper should clarify the relevance to the existing work [1].
[1] (CVPR 2019) Cycle-Consistency for Robust Visual Question Answering
Yes
No ethics review needed.
1. What is the approximate computational cost for the data augmentation by CYCLEAUG in the main experiment?
2. Since the authors use better MLLM in the CYCLEAUG framework, is the performance gain from the CYCLEAUG framework itself, or from distilling knowledge from the better MLLM (LLaVA-NeXT) to the MLLM we are training (LLaVA-1.5-7B)?
Data augmentation/generation for MLLM training is an important problem. The experiment results prove the effectiveness of the augmented data output by the proposed CYCLEAUG.
This paper propose CYCLEAUG, a data augmentation framework for MLLM training. CYCLEAUG first generate new images based on the image caption of the source image-text data, then use an MLLM to check the consistency of the generated images with the original image, and filter the inconsistent ones. The authors use the filtered augmented data to train a LLaVA1.5 and show the effectiveness of the generated data in MLLM training.
1. The computational cost of the whole pipeline might be too heavy. 
2. The main experiment is not convincing. The authors use LLama3-LLaVA-NeXT-8B for image caption (line 388), and LLaVA-NeXT for for consistency judge (according to the citation in line 308). However, the augmented data are used to train a LLaVA-1.5-7B, which uses a way worse base LLM (Vicuna-7B) and a weaker structure than LLaVA-NeXT. The potential assumption is that there exists a better MLLM of similar scale compared to the MLLM we are going to train. If so, why can't we use the better MLLM directly?
Yes
No ethics review needed.
1. The novelty is weak. The consistency sampling is directly inspired by the existing cross-modal consistency alignment framework. Although the idea is straightforward, there is no special or novel technical design, weakening the novelty of the proposed method.

2. The implicit Bayesian interpretation is good, however, it is not the main contribution of this paper in my opinion. The authors just did a straightforward thing to generate synthetic data to boost the MLLMs, which is reasonable and easy to understand. Utilizing technique-irrelevant formulations and proof is not necessary.

3. The authors only consider the visual augmentation via the loop of text-image-text. What about generating synthetic texts via the loop of image-text-image to provide new aspects of the same image?

4. Experiments are unconvincing. Firstly, the authors only conduct the experiments on a single MLLM model. At least 3-4 MLLM models should be performed to evaluate the proposed method fairly. Secondly, as shown in Table 1, the proposed method fails to achieve the SOTA on FIVE important benchmarks. Thirdly, the authors should provide a detailed complexity and efficiency analysis of the cycle-consistency visual augmentation process.

5. The authors should provide discussions about the failure cases and their limitations. Not all generated images are positive to the MLLM training. A more in-depth analysis is also required to be provided.
1. The paper is easy to follow.

2. The motivation is reasonable.
This paper proposes a new visual augmentation strategy for improving the MLLM's generalization ability. Specifically, it introduces a cycle-consistency sampling process, which utilizes a three-step process to collect new images. In the first step, it generates images based on the caption/question; then in the second step, it measures the consistency between the new image and the original caption/question. At last, the positive samples will be added to the datasets for training. The authors also exploit implicit Bayesian to interpret the generation process. Experiments are conducted on thirteen benchmarks.
1. The novelty is weak. The consistency sampling is directly inspired by the existing cross-modal consistency alignment framework. Although the idea is straightforward, there is no special or novel technical design, weakening the novelty of the proposed method.

2. The implicit Bayesian interpretation is good, however, it is not the main contribution of this paper in my opinion. The authors just did a straightforward thing to generate synthetic data to boost the MLLMs, which is reasonable and easy to understand. Utilizing technique-irrelevant formulations and proof is not necessary.

3. The authors only consider the visual augmentation via the loop of text-image-text. What about generating synthetic texts via the loop of image-text-image to provide new aspects of the same image?

4. Experiments are unconvincing. Firstly, the authors only conduct the experiments on a single MLLM model. At least 3-4 MLLM models should be performed to evaluate the proposed method fairly. Secondly, as shown in Table 1, the proposed method fails to achieve the SOTA on FIVE important benchmarks. Thirdly, the authors should provide a detailed complexity and efficiency analysis of the cycle-consistency visual augmentation process.

5. The authors should provide discussions about the failure cases and their limitations. Not all generated images are positive to the MLLM training. A more in-depth analysis is also required to be provided.
We introduce CycleAug, a novel framework that efficiently generates diverse synthetic images from existing IQA triplets through cycle-consistency, augmenting the visual instruction tuning of MLLMs.
@inproceedings{
anonymous2024cycleaug,
title={CycleAug: Cycle-Consistent Visual Augmentation for Large Multimodal Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=IiRlImvLQI},
note={under review}
}
Training multimodal large language models (MLLMs) requires high-quality image-question-answer (IQA) triplets, which are labour-intensive to curate and often lack diversity. We propose a novel data augmentation framework for visual instruction tuning that efficiently generates diverse synthetic images based on existing IQA anchor triplets. To ensure that the generated images align with their associated QA pairs, we propose CycleAug --- cycle-consistency visual augmentation which involves synthesizing images from text (text $\rightarrow$ image) and then performing a verification step to confirm that the answers derived from the synthetic images match the original answers (image $\rightarrow$ text), ensuring consistency across images, questions, and answers. By combining synthetic images with high-quality real data in the training phase, we demonstrate these synthetic triplets act as an implicit regularization, which improves the robustness of MLLMs and enables analogical reasoning. Extensive experiments show that our approach improves model performance on multiple visual question-answering benchmarks without additional real-world data. This work highlights the potential of leveraging visual foundational models to enhance visual instruction tuning in MLLMs.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
large multimodal models; synthetic data; data augmentation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/0045695feec03fe0b3cf1553b2861913bdda6687.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
CycleAug: Cycle-Consistent Visual Augmentation for Large Multimodal Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission