Video Action Differencing
Video;Actions;Differencing;Zero-shot;benchmark
Yes
No ethics review needed.
see above
1. The proposed task has not been explored, which has key applications for some scenarios in real life.
2. The construction process for the dataset is technically sound with different splits.
3. The visulizations are clear and interesting.
This paper introduces Video Action Differencing, a novel task of identifying subtle differences between videos of the same action. It also introduces a new benchmark sourced from mutliple video datasets with new annatations. A new method is proposed for this new task with state-of-the-art performance.
Weakness and questions:
1. Do the authors consider factors like fps for videos, which may impact the restults of answering questions like "the speed of the arms is faster" for distinguishing videos A and B.
2. For the open-set benchmark, have the authors analyzed the reasons for why QWen2-VL performs so worse?
3. Have the authors visualized the selected frames by the frame localizer compared to the ground-truth frames? What's about the effects of frame localizer compared to the ground-truth frames?
Yes
No ethics review needed.
Please refer to weakness for more details.
- The method is shown to outperform baseline large multimodal models by systematically isolating action differences through a three-stage approach, excelling in both closed and open settings.
- The introduction of benchmark, with extensive annotations across varied domains (e.g., fitness, surgery, sports), provides a unique and structured dataset for fine-grained video action comparison.
- Evaluations and ablation studies demonstrate the robustness and effectiveness of the method, especially in tasks that require precise frame alignment and action differentiation.
- The proposed task and methods address real-world challenges in skill-based learning environments.
The authors introduce a method and dataset designed to compare subtle action differences across video pairs. Their method, VidDiff, uses a three-stage process to generate, localize, and verify these differences using multimodal models.

Main Contributions:

- A Dataset includes 557 video pairs across domains like sports, fitness, and surgery, annotated with over 4,700 fine-grained action differences. The dataset is designed to help models learn and evaluate nuanced action comparisons.
- A Framework uses a three-step pipeline to identify differences: (1) generating potential differences with a language model, (2) aligning frames between videos using CLIP and Viterbi algorithms, and (3) verifying differences with a vision-language model.
- A method is compared with leading multimodal models (e.g., GPT-4o and Gemini-1.5 Pro), showing improved performance in closed-set settings. This comparison also highlights challenges in current models for frame alignment and action detail recognition.
- The performance of the leading multimodal models on the dataset is not clearly demonstrated, and examples comparing success and failure cases across models would enhance understanding of their effectiveness.
- The Difference Proposer stage’s reliance on large language models (LLMs) may introduce biases or inaccuracies, especially when generating action differences for complex or nuanced tasks. Providing more details on the generated proposer queries and their corresponding ground truth labels would enhance clarity.
- Although the multi-stage approach is well-structured, it presents a risk of error propagation. Inaccuracies in early stages could impact the final outputs, potentially reducing overall reliability, particularly in the open-set task, where the Difference Proposer’s effectiveness is not fully evaluated.
- While the paper introduces a detailed taxonomy for annotations, the reasonableness of some annotations remains unclear. For example, the “surgery_0” annotation includes the description "the movements in video A are more efficient than in video B," which lacks a concrete definition and could be interpreted inconsistently even by human evaluators. Scaling this annotation approach to larger datasets or adapting it to new domains could also present significant challenges.
- Minor issue: The table mentioned as Table 7 in Section 5.3 is missing.
Yes
No ethics review needed.
1. Does the sampling of pairs of videos mean that these might not contain the same action (see above)? Or the actions are sampled first to ensure a wide range of comparison difficulty over actions before video pairs are sampled within action?
2. Were the annotators skilled/experts/knowledgeable in the actions that they were annotating? Or was this found to not be that important given the annotation task? Additionally, how many total annotators were used and were they renumerated for their time?
3. Has the potential bias of the video pairs in the closed task been checked to ensure that naive performance should be 50% instead of video A (or B) occurring as the answer more than 50% of the time. Additionally, I would be interested to know if candidates which could be categorised as C (for insignificant differences) can be understood by the model as this would also increase the naive difficulty to 33% before taking into account a non-uniform distribution.
4. The evaluation protocol for the open-set task seems like it could include errors/inconsistencies depending on the LLM output. Has there been any investigation into this and how much it differs per run and how much it aligns with a human? Currently, the prompts are also given in the appendix with little to no discussion as to why these prompts were chosen, if they were developed over multiple iterations to find the best performing prompt, etc.
5. Did the easy/medium/hard classifications align with experts' opinions for each of the actions? It would be good to know the types of actions that are classed as easy/medium/hard as these are not present within the paper as far as I could tell. It's not clear why an LLM was chosen to do this task.
6. Could more qualitative results and statistics be provided about the dataset? For example, there is very little in the paper regarding the retrieval task of localising the differences: How much of the video does the method need to localise? Are there any temporal biases regarding the timestamps from the videos? Additionally, under the closed set task, more statistics over the number of As, Bs, and Cs that have been annotated and included for each action would be interesting to see. Other statistics that feel like they are missing are the average length of each video (potentially broken down per category) as well as the total number of hours within the dataset.
7. As a thought, has an experiment where the same video is duplicated and provided into the methods, would the output predictions (esp. for the closed set task) give a 50% response rate? Ideally, this is where a method could predict the difference is negligible also.
* The new task of Video Action Differencing is an interesting new task for video understanding, forcing models to recognise and understand fine-grained differences between two very similar videos.
* The collected dataset combines four datasets with 5 different categories of video, providing a varied test bed for this new task.
* The proposed method performs well on the dataset, outperforming off the shelf LMMs on the task yet still showcase that there is a lot still to work on in this area for future work.
In this paper, a new task called video action differencing is proposed which aims for models to be able to understand fine-grained differences between multiple videos of people performing the same action. A new benchmark dataset is collected, named VidDiffBench, which includes 5 categories from 4 different existing datasets. Annotations are collected from pairs of videos with statements given per pair of video based on the action (for example video A includes someone jumping higher than Video B for a lay-up shot). There are two main evaluation protocols for this task, a closed set setting, in which the model must predict A or B for each possible description, and a closed set setting in which the method must generate the description. A new method which combines stages named VidDiff is proposed which outperforms standard LMMs on the dataset.
# Weaknesses

* There are some missing references for skill determination within the related work [a, b, c, d] as another example of fine-grained differences between videos containing the same action.
* Line 196: It is mentioned here in the text that *"Video pairs are randomly sampled within each dataset to ensure a wide range of comparison difficulty, from simple actions to more advanced tasks requiring fine-grained understanding"* This implies that videos of differing actions are compared against one another. 
* Section 3.3.2: There are some missing information about the annotators, regarding skill level, total number, renumeration etc.
* For the closed set, a binary classification setup was used as all candidate difference statements which is mentioned to be unbiased on Line 298. However, has this been checked? If videos are not randomly swapped at inference/training time there could have been a bias towards one video or another.
* The open set evaluation seems like it could be prone to some errors/inconsistencies depending on the LLM chosen and how much it could hallucinate/not understand the task and doesn't represent a potentially sound evaluation protocol.
* It is not clear within the paper as to why an LLM was used to choose the easy/medium/hard splits for each of the actions.
* This paper did not feel like an easy read, whilst the grammar/sentence clarity was good. There was a lot of information that is split across the main paper and the appendix which necessitates jumping between them. The structure of the paper could also be improved, the method details occur within the experiments yet are given as a main contribution within the introduction with only a small amount of space given to explain the model. Another major factor for this is that details of the dataset are given before the task is formally defined, which given this is a new task, makes it harder to read than it should be.

# Additional Comments
Line 158 is referring to the wrong table, this should be Table 1
Line 1040 (in supp.) vary -> very
Section D.1 in the appendix is empty, maybe D.2 is meant to be a subheading of D.1?
For results tables, it would be good to include a random performance row.

# References
[a] Doughty, Hazel, Dima Damen, and Walterio Mayol-Cuevas. "Who's better? who's best? pairwise deep ranking for skill determination." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

[b] Doughty, Hazel, Walterio Mayol-Cuevas, and Dima Damen. "The pros and cons: Rank-aware temporal attention for skill determination in long videos." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

[c] Pan, Jia-Hui, Jibin Gao, and Wei-Shi Zheng. "Adaptive action assessment." IEEE Transactions on Pattern Analysis and Machine Intelligence 44.12 (2021): 8779-8795.

[d] Zhang, Shao-Jie, et al. "Adaptive stage-aware assessment skill transfer for skill determination." IEEE Transactions on Multimedia (2023)
Yes
No ethics review needed.
See weaknesses, plus the following:

- Which LLMs/VLMs are used for the *Difference Proposer* and *Action Differencer*?
- How does the benchmark handle cases of inverse correlation? For example, would *lower squat in video A* be equivalent to *higher squat in video B*?
- Since the videos are not curated, factors such as different camera angles, varying FPS, or differences in the actor's height could introduce biases in the annotations and results. How do the authors address these potential biases?
### Originality
- **A novel task**: This paper introduces the new task of video action differencing with natural language. While related tasks, such as difference captioning, have been explored to provide a coarse comparison between videos, no prior work has tackled video action differencing in the same way—focusing on fine-grained differences described in natural language.
- **A challenging benchmark**: The proposed benchmark, VidDiffBench, is comprehensive, covering five categories of instructional videos. It has proven to be highly challenging, even for top-performing closed-source vision-language models (VLMs).
- **An agent-based system**: The paper presents an agent-based system that decomposes the task, achieving better performance than existing VLMs.
### Clarity
The flow of ideas is straightforward, making the paper easy to follow and understand.

### Significance
The paper convincingly demonstrates the importance of video action differencing, and the introduction of the new benchmark is likely to inspire further research in this area.
This paper introduces the first large-scale video action differencing dataset, presenting a novel task of identifying differences between videos depicting the same action. The authors compile over 500 video pairs from existing datasets across five categories: Fitness, Ball Sports, Diving, Music, and Surgery. These videos are then assigned to annotators along with 147 distinct descriptions. Annotators must indicate which video (A or B) most closely aligns with each description. For example, given two videos of different actors performing a squat, a description might read "deeper squat," and the annotator would select A or B based on which video demonstrates the deeper squat. To ensure dataset quality, 25% of the initial annotations undergo re-annotation, revealing a very low discrepancy rate. The dataset also includes action localization (pinpointing where the action occurs in the video) and specific key points for each action (e.g., when knees start to bend).

The authors also develop an agentic model called VidDiff to address the action differencing challenge. VidDiff employs several Large Language Models (LLMs) and Vision Language Models (VLMs) as agents to solve specific aspects of the problem: proposing potential differences based on the action description, localizing frames where such actions might occur, and finally specifying which video (A or B) corresponds to the observed difference. VidDiff outperforms other zero-shot VLMs in this task.

Lastly, the authors provide ablation experiments that highlight the challenges presented by their new benchmark.
### Unproven claims
- In the introduction, the authors claim they will address the challenges of *precise temporal alignment and the need for fine-grained understanding of action dynamics*. However, it remains unclear how they specifically solve the issue of temporal alignment. Could you elaborate on how you solve this issue or point us to the location where it is addressed?

### Benchmark and results
- Similar datasets are presented in the related work section; however, since this work is primarily a benchmark paper, more comparisons with existing benchmarks would be make the differences clearer (e.g., similar to Table 1 but with other datasets in the first column). Consider adding what is unique about each dataset and how the current dataset differs.
- As a benchmark paper, we would expect more results from other open-source VLMs (especially those addressing video data such as LLaVA-video) to better understand their limitations and make it easier for other researchers to work with this benchmark. 

### Clarity
- 557 or 656 video pairs? In the abstract, the authors state that the dataset contains *557 video pairs...  4,719 fine-grained action differences* (line 013-014), but on line 260, they mention *656 video pairs, 5,580 annotated differences*. Clarification needed on which is correct. 
- Figure 1: The distinction between the first and second row is unclear, yet the caption claims these represent two different challenges. These two challenges are not discussed elsewhere in the paper and don't seem to be related to the dataset splits. Please clarify this.
A new task and benchmark for comparing how an action is performed between two videos, with a zero-shot method
@inproceedings{
anonymous2024video,
title={Video Action Differencing},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=3bcN6xlO6f},
note={under review}
}
How do two individuals differ when performing the same action? In this work, we introduce Video Action Differencing, the novel task of identifying subtle differences between videos of the same action, which has numerous applications, such as coaching and skill acquisition. To enable development on this new task, we first create VidDiffBench, a benchmark dataset containing 557 video pairs, with human annotations of 4,719 fine-grained action differences and 2,075 timestamps indicating where these differences occur. Our experiments demonstrate that VidDiffBench poses a significant challenge for state-of-the-art large multimodal models (LMMs), such as GPT-4o, Gemini 1.5 Pro, and Qwen2-VL. By analyzing the failure cases of LMMs on VidDiffBench, we highlight two key challenges for this task: frame-by-frame alignment and fine-grained frame comparison. To overcome these, we propose VidDiff, an agent-based system that breaks the task into three stages: action difference proposal, keyframe localization, and difference verification, each stage utilizing specialized foundation models. The VidDiff method outperforms these baseline LMMs. We release both the dataset and code to encourage and support future research in this domain.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Video Actions Differencing Zero-shot benchmark
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/9ef4acd4c5de63ad17d8b82c81ee3b0c46cb5ffa.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/d4363054841c9ff375be851451deab36c7f3a3b9.pdf
Video Action Differencing
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission