Visual Transformation Telling
visual reasoning;transformation;captioning
Yes
N/A.
No ethics review needed.
Please see the weaknesses section.
Significance: This work has its significance since it focuses on spatiotemporal processing, the authors discuss that some advanced MLLMs struggle to describe the given transformations (that being said, the evaluation procedure should still be revised).
This work creates a new benchmark which tests how well the current multimodal LLMs (MLLMs) process visual transformations (i.e. change of states) between two adjacent visual scenes, like opening/closing a door, or breaking some glass. In this benchmark, a set of open-source and closed-source models are evaluated via both human validators and automated metrics.
The paper has a poor presentation.
- Fig. 1 could be improved by showing explicitly what are the inputs and the outputs in this task.
- Sec 3.1 should be linked to Fig. 1.
- The most importantly, the main text lacks details about TTNet, which makes following the paper very difficult.
- Fig. 2(d) completely could be put into the appendix (that could save some space for the TTNet).

The evaluation procedure.
- Some models achieve very high CIDEr scores. Could the authors please double check?
- While I appreciate the number of different evaulation metrics, I am skeptical about a couple of them.
- Why do we have two task-related human evaluation criterias (relevance/logical soundness) where they could be combined? If some generation is irrelevant, could the logical soundness matter for that generation? (could you please provide a counter-example if there is one?)
- Why do the authors evaluate fluency? The most of the transformation descriptions are relatively short (Fig 2.(c)) and the literature already evaluated the fluency enough previously.
- Is there a reason to avoid some evaluation procedure similar to LLM-as-judge in this work?
- It is not clear to me, how they evaluate the models. Is the evaluation per-transformation or per-scene?
- How are the images being processed by the models? For instance, as far as I know some models could process input images subsequently, are all of the tested models capable of doing the same? If not, how do the authors process input images? Do they concatenate input images?

The models.
- The number of open-source models could be increased (e.g. PaliGemma, OpenFlamingo, Idefics, InstructBLIP there are some suggestions, a newer suggestion could be Pixtral).
- Some video models could be added. Actually some of these models are pretrained using both video and image data, so they could be adapted easily (e.g. [Frozen-in-Time](https://arxiv.org/abs/2104.00650), [Video-LLaMA](https://aclanthology.org/2023.emnlp-demo.49/) and many others).

Missing some relevant literature.
- Video-Text models should be included.
- [The ViLMA benchmark's change-of-state task](https://openreview.net/forum?id=liuqDwmbQJ) is quite related to the proposed benchmark. It should be mentioned and discussed.
Yes
No ethics review needed.
1. How did you input multiple images if the model only supports single-image input?

2. What's the performance difference between single image and multiple image inputs? For example, will models' output differ when giving $s_i$ and $s_{i+1}$ versus only using $s_{i+1}$?

3. In table 2, why Gemini gets worse performance when using multi-turn?

4. What's the fine-tuned performance of the baselines?
The experiments are comprehensive. The presentation of the paper is good.
This paper proposes a new task called Visual Transformation Telling (VTT), which requires the model to generate a caption between two images in an instructional video. This paper evaluated the performance of existing models on this new task, which shows
1. **Motivation.** The motivation to create this task is vague. It is unclear why describing the transformation between two images is more important than a single image. From the qualitative examples in the paper, it seems that only using one image is enough to generate the caption. The evidence for real-world or downstream applications is not clearly justified in the paper. Here are some questions that authors can think about when evaluating the motivation of this work: Will this new task benefit models on other tasks? What are the real-world use cases of such a task/model? Do I need a robot to understand the state transformation in **natural language format** to perform daily tasks? If the answer is only "current models cannot perform well on our task", then you need to really think about whether to do such research.

2. **Distinction with other video-language tasks.** Basically, this paper converts the existing video action recognition tasks like COIN into a video captioning task. The concept of state transformation is ambiguous, and it is hard to justify this in a better format. Furthermore, compared to the original format of COIN, the generation format of VTT poses additional difficulties in evaluation, as indicated by the bad performance of humans when using automatic metrics (Table 2).

3. **Limited Contribution.** Overall, this paper creates a task without strong motivation and claims the existing model cannot perform well on this new task. There is not much technical content in the paper, and even the data is reused from previous work. The overall contribution of this work is below the standard of ICLR.
Yes
No ethics review needed.
1. How do you ensure whether a change between two images is explainable and deterministic? How do you evaluate and address the multiple possibilities of changes between two images?
2. Could more models trained on image-text interleaved data, such as InternVL-2, be evaluated?
1. The writing of this paper is clear.

2. The experiments in this paper are comprehensive, including various quantitative tests and visual result analyses.
This paper introduces a task called Visual Transformation Telling (VTT), which aims to evaluate a model’s reasoning ability about the cause of state changes between two images, especially in natural language. The authors collect a large-scale dataset related to this task from instructional video datasets, conduct human verification, and create a new benchmark. This paper also tests the performance of existing open-source and commercial models and proposes a new model to address the task, which outperforms existing methods.
1. The difference between VTT and the existing Procedure Planning task seem not significant. Typically, Procedure Planning tasks are done on CrossTask and COIN datasets, and there is already a lot of related work on this. Moreover, both tasks are based on pre-collected temporal segment annotations from CrossTask and COIN, so the textual content should be the same. It seems like this paper simply transforms a classification problem into an open-ended text generation problem.
2. If the focus of VTT is on explanation, why not use two images instead of a reasoning chain? How does the previous reasoning process affect subsequent reasoning? When the model makes errors, are they due to the accumulation of prior mistakes or the inability to infer the current two states?
3. Specific details regarding testing models like GPT-4o need to be provided. For example, GPT-4 is a purely text-based model—how did you test it, or did you test GPT-4V? Additionally, the specific prompts and the details such as model version should be thoroughly explained.



**It is recommended to add a figure that clearly describes the differences between the task proposed in this paper and all previously related tasks.**
Visual Transformation Telling: a new dataset and benchmark that requires to reason and describe transformations from a series of states.
@inproceedings{
anonymous2024visual,
title={Visual Transformation Telling},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=qu6UMVT4k1},
note={under review}
}
Humans can naturally reason from superficial state differences (e.g. ground wetness) to transformations descriptions (e.g. raining) according to their life experience. In this paper, we propose a new visual reasoning task to test this transformation reasoning ability in real-world scenarios, called **V**sual **T**ransformation **T**elling (VTT). Given a series of states (i.e., images), VTT requires to describe the transformation occurring between every two adjacent states. Different from existing visual reasoning tasks that focus on surface state reasoning, the advantage of VTT is that it captures the underlying causes, e.g. actions or events, behind the differences among states. We collect a novel dataset which comprise 13,547 samples to support the study of transformation reasoning. Each sample involves several key state images along with their transformation descriptions. Our dataset spans diverse real-world activities, providing a rich resource for training and evaluation with automated, human, and LLM assessments. To construct an initial benchmark for VTT, we test models including traditional visual storytelling (CST, GLACNet) or dense video captioning methods (Densecap) and advanced multimodal large language models (LLaVA v1.5-7B, Qwen-VL-chat, Gemini-1.5, GPT-4o, and GPT-4), as well as their upgraded versions based on our learning on human reasoning. Experimental results reveal that even state-of-the-art models still have a significant gap with human performance in VTT, highlighting substantial areas for improvement.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
visual reasoning transformation captioning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/b5248367557ca04b6bdf1304d742672e0c7764c8.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/0541876017749cc7cbeadd3e509ae3abd00bcf03.zip
Visual Transformation Telling
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission