InterMask: 3D Human Interaction Generation via Collaborative Masked Modelling
Motion Synthesis;Human Interaction Generation;Masked Generative Transformer;Text-driven Generation;Vector Quantized VAE
Yes
No ethics review needed.
1. I noticed that in the second video of the visualization "Everyday Actions," the character on the right makes a sudden arm-raising motion, which seems somewhat disconnected. Additionally, is there a 10-second video available for reference? I saw that the authors mentioned the ability to generate interactive results up to 10 seconds long.

2. In terms of quantitative experiments, it seems there is no comparison with PriorMDM, which also produces results for two-person interactions. Similarly, there doesn't appear to be a comparison with other VQ-VAE methods like T2M-GPT. I'm curious about their results, especially since they perform very well on single-person datasets. If T2M-GPT were given a reference action sequence for one individual as an additional condition, how would it perform in generating interactive individuals, particularly on interactive datasets?

3. Regarding the Inter-M Transformer structure, although there is some evidence in the ablation studies concerning the relative importance of different attention mechanisms, it may not fully clarify their specific roles and trade-offs in different interaction scenarios and data distributions.
1. A Transformer structure combined with VQ-VAE has been proposed for generating 3D human interaction actions. 
2. It utilizes a 2D discrete motion token map to represent motion, distinguishing itself from traditional 1D representation methods. This representation better preserves spatiotemporal details and spatial awareness, which is crucial for accurately capturing complex motion relationships in human interactions. 
3. The framework has achieved state-of-the-art results on the InterHuman and InterX datasets, performing exceptionally well on various metrics. 
4. The average inference time is only 0.77 seconds, significantly lower than InterGen's 1.63 seconds, providing a considerable advantage in practical applications by enabling faster result generation.
This article introduces a new framework called InterMask for generating 3D human interaction actions from text descriptions. The proposed framework appears to be based on the T2M-GPT framework, modeling in discrete space using generative masks, while also incorporating the concept of interhand to collaborate within the mask modeling framework by utilizing tokens from two interacting individuals. During the inference process, it starts with a fully masked sequence and gradually fills in the tokens for both individuals.
1. In the visualization results, issues such as body penetration between interacting individuals and unnatural sliding movements still occur.
2. The article also mentions the masking strategy, but I believe its effectiveness may be influenced by various factors, such as the setting of the mask ratio and the dynamic adjustment of masks at different stages. The article does not delve deeply into how these factors affect model performance or how to optimally set these parameters.
Yes
No ethics review needed.
1. Could the authors comment on the suitability of the current framework for multi-person interactions? Given the focus on two-person setups, it would be useful to understand any potential modifications or challenges in adapting this approach for more complex, multi-person interaction scenarios.

2. How does the model handle longer, more complex instructions or instructions that are less structured (in-the-wild)? 

3. Do the authors see potential for combining InterMask with language models to improve interpretability or to handle a wider range of textual prompts?
1. The use of 2D token maps and spatial-temporal attention layers in the Inter-M transformer enables more nuanced modeling of human interactions, demonstrating improved fidelity in motion generation.

2. InterMask achieves state-of-the-art results on key metrics in standard datasets, indicating its effectiveness in generating realistic, text-aligned interactions.

3. The model is relatively lightweight and demonstrates reduced inference time.
The paper presents InterMask, a framework for generating 3D human interactions using masked modeling in discrete space. By encoding motion as 2D token maps and modeling intra- and inter-person dependencies with a custom transformer, InterMask achieves impressive results on the InterHuman and InterX datasets. The model shows strong performance in terms of realism and alignment with text prompts, outperforming existing methods in controlled settings.
1. The paper’s innovative contribution is not well-defined. Although the authors introduce a transformer module to model intra- and inter-person spatial-temporal dependencies, this approach is widely used in various generative tasks. The ablation studies, which compare only with/without this module, do not sufficiently demonstrate the model’s unique contribution. If this module is intended as a core innovation, the authors should focus on explaining why its internal structure, arrangement, or sequence specifically suits the 3D interaction modeling problem. Merely showing the presence or absence of the module does not justify its relevance, as it is a fairly standard approach.

2. The process of converting each motion sequence into a 2D discrete motion token map lacks detailed description. The main innovation of this paper appears to lie in the combination of the 2D discrete motion token map with MAE; however, there is insufficient explanation of how the 2D token map is constructed. Providing a more thorough description, ideally with a detailed diagram, would enhance clarity and better communicate this aspect of the method.

3. User Study. The user study includes only three participants, which is insufficient to draw robust conclusions on user preference. A larger sample size and detailed descriptions of the study’s methodology would add reliability and provide stronger validation for claims of user preference for InterMask-generated interactions.

4. Upon reviewing the supplementary videos, the generated motions appear to lack the fluidity seen in InterGen’s outputs. A discussion on this aspect would be valuable, particularly to understand any limitations in InterMask’s approach to generating smooth, natural motion.
Yes
No ethics review needed.
I have no other questions.
1. This paper introduces a novel 2D VQ-VAE that quantizes motion into a 2D discrete map, which can more effectively preserve fine-grained spatio-temporal details than the 1D version.

2. The proposed method achieves SOTA performance on the InterHuman and InterX benchmarks.
This paper addresses the problem of generating human-human interaction from textual descriptions. The authors introduce InterMask, a novel framework that employs 2D VQ-VAE to encode interactions into a discrete space and generative masked modeling to capture complex spatio-temporal interdependencies.
1. The method does not appear substantially different from prior work. Overall, the paper feels somewhat underwhelming, lacking distinctive insights, though there are no major issues with the experiments or writing. I would rate it as borderline.

2. The proposed approach faces challenges in generating human reactions in an online manner (although this isn’t the primary focus of the current setting). This capability would be more practical for some virtual reality applications.
A novel framework for human interaction generation using collaborative masked modeling in the discrete space, which explicitly models spatio-temporal dependencies within and between the interacting individuals.
@inproceedings{
anonymous2024intermask,
title={InterMask: 3D Human Interaction Generation via Collaborative Masked Modelling},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=ZAyuwJYN8N},
note={under review}
}
Generating realistic 3D human-human interactions from textual descriptions remains a challenging task. Existing approaches, typically based on diffusion models, often generate unnatural and unrealistic results. In this work, we introduce *InterMask*, a novel framework for generating human interactions using collaborative masked modeling in discrete space. InterMask first employs a VQ-VAE to transform each motion sequence into a 2D discrete motion token map. Unlike traditional 1D VQ token maps, it better preserves fine-grained spatio-temporal details and promotes *spatial awareness* within each token. Building on this representation, InterMask utilizes a generative masked modeling framework to collaboratively model the tokens of two interacting individuals. This is achieved by employing a transformer architecture specifically designed to capture complex spatio-temporal interdependencies. During training, it randomly masks the motion tokens of both individuals and learns to predict them. In inference, starting from fully masked sequences, it progressively fills in the tokens for both individuals. With its enhanced motion representation, dedicated architecture, and effective learning strategy, InterMask achieves state-of-the-art results, producing high-fidelity and diverse human interactions. It outperforms previous methods, achieving an FID of $5.154$ (vs $5.535$ for in2IN) on the InterHuman dataset and $0.399$ (vs $5.207$ for InterGen) on the InterX dataset. Additionally, InterMask seamlessly supports reaction generation without the need for model redesign or fine-tuning.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Motion Synthesis Human Interaction Generation Masked Generative Transformer Text-driven Generation Vector Quantized VAE
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/600db12ed9f6547ed9684d931dba6d10af207ece.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/f676b6c643071af10fd2123288ff181a89c05685.zip
InterMask: 3D Human Interaction Generation via Collaborative Masked Modelling
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission