Interleaved Scene Graph for Interleaved Text-and-Image Generation Assessment
Interleaved Text-and-Image Generation;Generative Models;Multimodal Large Language Model;Scene Graphs;Automatic Evaluation;Benchmark
Yes
No ethics review needed.
Please refer to weakness.
1. Propose a novel ISG framework and ISG-BENCH benchmark dataset that fill a notable gap in the evaluation of multimodal interleaved generation. 
2. The multi-level evaluation in ISG, combining visual question answering (VQA) with reasoning-based questions, offers a fine-grained approach to assess the structure and quality of generated content in complex multimodal tasks.
3. The creation of ISG-BENCH, covering 21 multimodal generation tasks, provides a standardized dataset for researchers.
This paper presents a multi-level evaluation framework, INTERLEAVED SCENE GRAPH (ISG), along with a benchmark dataset, ISG-BENCH, to address the challenges of evaluating multimodal interleaved text-and-image generation tasks. Additionally, the authors propose ISG-AGENT, a compositional generation framework designed to explore the upper limits of interleaved generation with a structured agent-based workflow.
1. The description of the methodology could be more detailed, for example, in terms of the experimental setup.
2. While ISG-AGENT performs well across tasks, it still exhibits notable shortcomings in specific areas. A more detailed discussion of these limitations and suggestions for future improvement would be beneficial in the conclusion.
Yes
In section 3.2, the papers say the data of the benchmarks were collected from existing datasets and further annotated by human experts. However, the paper does not mark the source of these datasets and it's a potential concern if there are licence issues of the source datasets.

Besides, as a benchmark, the release of the dataset may also undergo the ethics review process for the safety reason.
Yes, Legal compliance (e.g., GDPR, copyright, terms of use) Yes, Responsible research practice (e.g., human subjects, data release)
1. What's the cost of evaluating a new model on the ISG-Bench since evaluation heavily relies on the GPT-4 in the process
2. While you recognize the potential bias of using MLLM as a judge for evaluating image quality, did you figure out any ways to mitigate the bias? Have you conducted any human studies to understand how will these kinds of biases affect the evaluation of the benchmark?
3. How many resources does ISG-Bench need to finish a query in the ISG-Bench? Is it comparable with using other models like show-o to directly generate the outputs?
4. There are many samples in your datasets that have images in the query. Have you ever conducted any study that removes the image in the query and keeps only the text part, then conduct the evaluation in a text-input-only setting? This is to see the gap between the normal setting and make sure that these images in the query are really important to generate a good response, which is now a pretty common analysis for the vision understanding benchmarks.
1. This paper presents first benchmark to evaluate the models that can generate interleaved text-and-image models, which differs fundamentally from previous vision understanding benchmarks like MMMU. The benchmarks feature an important usage of multimodal models for content generation, and can greatly impact the community if well-maintained and updated.
2. The design of ISG-Agent is reasonable and novel, achieving state-of-the-art performance on the ISG-Bench. The introduce of Planning, Tool-usage, and refinement works well in practice.
3.  The analysis part brings many insights such as MLLM-as-a-Judge cannon evaluate well due biases like "image-qualtiy bias", etc.
This paper introduces a new benchmark for evaluating the interleaved text-and-image generation tasks. While previous multimodal benchmarks only focus on evaluating vision understanding ability where only text outputs are required, this benchmark designed 21 text-image generation tasks with 1150 samples in total. The benchmark have multiple-level of evaluation including structure, block, image, and holistic levels. They also propose a ISG-Agent, which utilize external tools to fulfill the requirements of the benchmarks. Results show that existing models fall short of the benchmark and ISO-Agent performs significantly better than baseline models.
1. The evaluation still uses GPT-4o as a judge, which can bring a lot of bias, as section 4.2 states. The 
2. It might be a bit unfair to compare ISG-Agent with another model that can generate interleaved text and images in a whole, as the agent still uses external tools and additional inference tokens for planning, etc. I am not saying ISG-Agent is not good. But it should also be compared with other agents for a fair comparison.
Yes
No ethics review needed.
- I believe the intent of this benchmark should be to establish samples that are well-suited to tasks requiring both text and image generation for optimal performance, correct? If the agentic approach can yield good results, might this indicate that the benchmark isn't ideally suited for evaluating a unified model? If the benchmark contains some tasks that only a unified model could perform well, that would be even better.
- This is a timely work that studies the critical and underexplored challenge of evaluating the recent popular unified models.

- ISG-BENCH provides a comprehensive benchmark with 1,150 carefully designed samples across 21 tasks, supporting standardized evaluation in multimodal generation.

- The paper conducts extensive multi-granular evaluations, offering a thorough assessment that reveals limitations in current models.

- The proposed ISG-AGENT explores the upper bounds of interleaved generation, highlighting strengths and limitations within the framework.

- The paper is very well written and organized.
This paper presents a unified approach for evaluating interleaved multimodal text-and-image generation using a novel framework called INTERLEAVED SCENE GRAPH (ISG), which assesses generation accuracy at four granular levels through atomic question-based visual verification. Additionally, it introduces ISG-BENCH, a benchmark of 1,150 samples across 21 generation tasks, and ISG-AGENT, a compositional agent framework designed to explore the limits of interleaved generation. The study’s experiments reveal that, while ISG enables precise evaluation aligned with ground truth and human preferences, existing models often fail in accurate instruction-following, particularly in vision-dominated tasks, highlighting areas for improvement in multimodal generation research.
- The model evaluation is relatively complex, as many steps involve MLLMs and LLMs to assess results. Given that the author's proposed task is more challenging, how reliable are current MLLMs and LLMs in this evaluation?

- Some tasks seem overly simplistic and not directly relevant to multimodal generation—for instance, Image Decomposition, which resembles a basic computer vision task. Has there been any explanation of the criteria used to filter these task categories?
Yes
No ethics review needed.
see weakness.
the proposed benchmark is a multi-granular evaluation set for interleaved text-and-image generation. this provide options to study the model performance through various perspectives. 

the proposed benchmark scale is the largest by far, as shown in the paper Tab 1, consisting of 1k+ samples, in 21 categories. this could be the primary contribution to the community.

the agent-based approach utilizes a "Plan-Execute-Refine" structure, with tool use. This compositional approach shows promise in generating high-quality outputs and provides a strong baseline for further research.
the paper proposed one multimodal benchmark, named ISG-BENCH, and one agent-based approach to tackle the problem. the focus is the interleaved image-text data as the model output, which is a promising direction in the field. in the benchmark, there are 4-level evaluation methods, which is structure, block, image, and holistic. these provide multiple perspectives to study the model performance. the proposed agent-based approach also achieves the best among all the existing and the baseline approaches.
the primary contribution could be the benchmark with multi-granularity evaluation. on one hand, this is great to provide more options. on the other hand, this may suggest this work is a combination of multiple existing approaches, e.g. by combining openleaf and interleavedbench. although the work provides some unique features, e.g. structure-level evaluation, however, the most important evaluation metric could be the holistic evaluation, which has already been studied in the literature. 

the multi-granularity evaluation is performed by different approaches. would be it possible to use only one approach (e.g. MLLM) to evaluate all these perspectives in a more unified, simpler, more elegant way?  

considering the holistic evaluation may be the most important, it is better to show more study on why the proposed MLLM-as-the-judge is a good judge, e.g, how the evaluation is aligned with the human judgement. if approach A is better than approach B from MLLM-as-the-judge, is it also true for human judgement? what kind of failure cases could be with the MLLM-as-the-judge.
Yes
No ethics review needed.
1. In Table 1, for OpenLeaf (An et al., 2023), why is the number of samples 30 instead of 660 listed in the paper?

2. It would be helpful to discuss the plan on setting up and maintaining the automatic evaluation benchmark for easier and wider use.

3. More examples on the evaluation questions, and category examples are helpful.
1. The proposed evaluation aims to address the problem that MLLM evaluation alone is not reliable. This is a valid question, and the proposed improvements: (1) multi-granular evaluation; (2) converting to question answering, do serve as an effective add-on for evaluation.

2. The benchmark covers a wide range of evaluation scenarios for the interleaved image-text generation.

3. The paper is presented well, with sufficient discussion on evaluation pipeline, related works, and evaluation categories.
The paper presents a new evaluation method and benchmark for the interleaved image-text generation task, which aims to generate coherent image text contents for queried use cases, such as instruction generation, visual storytelling, and others (Table 2, Figure 3). For evaluation, the study presents converting multimodal queries into atomic questions for a better scoring than MLLM-based evaluations. The constructed benchmark contains 1150 samples covering 21 generation scenarios. In addition to existing methods, an agent framework named ISG-AGENT is also proposed to enhance the performance.
1. One concern is on the definition of the task “interleaved image-text generation,” related to the discussion in the first paragraph of introduction. The current definition of the task is based on the application scenarios, which are valid and of good application value. However, it fails to provide extra insights to the task, including what are the unique challenges, and how to balance the categories and samples in the evaluation benchmark.

For the categories and samples percentage, the benchmark already covers a wide range of scenarios (Table 2, Figure 3). But the motivation of the percentage and construction remains unclear: why certain scenarios are selected, how to decide between 50/100 samples, and whether there is an “ideal” distribution to follow for the task. 

2. The experiment evaluates various interleaved generative frameworks.

Related to the task’s unique challenge discussions, it would be helpful to discuss whether the image and text generation can be disentangled: providing insights on whether agent systems or naturally multimodal (unified) models (e.g., GPT4o and others) are more promising in the future. The experiments discuss the performance of existing models in these two lines, but fail to suggest future directions on them.

The current observation is that the open-sourced “unified model” is much worse compared with agent systems. This is partially because some of the better unified models are not open-sourced. It would be interesting to add a few examples compared with the demos shown in their paper/webpage, to provide insights on “future better unified models.” (e.g., https://openai.com/index/hello-gpt-4o/ Explorations of capabilities: Visual narratives - Sally the mailwoman) And when both directions have comparable base capacity, what are there relative strengths and weaknesses.

3. The “GT” in abstract and Table 1 still works on pre-selected evaluated models, instead of extending to arbitrary new models to evaluate. Therefore, the evaluation could still remain noisy similar to MLLM-scoring.
This paper introduces a fine-grained automatic evaluation framework and a new benchmark for interleaved text-and-image generation, offering valuable insights for future research in accurate interleaved generation.
@inproceedings{
anonymous2024interleaved,
title={Interleaved Scene Graph for Interleaved Text-and-Image Generation Assessment},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=rDLgnYLM5b},
note={under review}
}
Unified Transformer-based models have enabled simultaneous multimodal understanding and generation, showing promise in unifying both vision and language tasks with interleaved text-and-image generation. However, assessing the performance of multimodal interleaved generation remains unexplored and challenging due to the complexity of interleaved content.  In this paper, we design an automatic multi-granular evaluation framework Interleaved Scene Graph (ISG) with four levels to evaluate accurate interleaved generation tasks, which converts multimodal queries into atomic questions, then perform visual question answering for precise verification. Moreover, we propose ISG-Bench, the first multimodal interleaved benchmark with concrete generation requirement consisting of 1,150 samples across 21 text-image generation tasks. Additionally, we pioneer in a compositional agent framework ISG-Agent to explore the upper bound of interleaved generation with agent workflow. In our experiments, we conduct a multi-granular evaluation of ISG, demonstrating its potential for automatically evaluating interleaved generation consistent with ground truth and human preferences. Furthermore, comprehensive assessments of 10 interleaved generative frameworks reveal that unified models still lack basic accurate instruction-following capabilities, falling short even in structural requirements. Additionally, our ISG-Agent outperforms other compositional frameworks in interleaved generation at various levels but still struggles with vision-dominated tasks. Our work offers valuable insights for advancing future research in interleaved text-and-image generation.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Interleaved Text-and-Image Generation Generative Models Multimodal Large Language Model Scene Graphs Automatic Evaluation Benchmark
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/5591b6f4458dfb5f5d2465c3a1ef9bdfde0f5ae2.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/40fbb648b84a75109416c0b3344678bc9fcf462d.zip
Interleaved Scene Graph for Interleaved Text-and-Image Generation Assessment
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission