Visual Representation Learning for World Models by Predicting Fine-Grained Motion
world models;model-based reinforcement learning;visual representation learning
Yes
No ethics review needed.
Please see weaknesses above.
1. The attention that the authors give to relevant foreground information and moving tiny objects is meaningful for constructing world models.

2. The performance gain of the proposed method is considerable on the Atari 100K benchmark and DeepMind Control Suite.
This paper presents the Motion-Aware World Model. It integrates a fine-grained motion predictor with an adaptive motion-aware scheduler and involves action-conditional video prediction to filter out backgrounds and track object motion at the pixel level.
1. The author elaborates on some details in the method section; however, many technical aspects, like the Convolutional Block Attention Module, were not employed in previous approaches. Nevertheless, the relevant experiments have not undergone ablation, which could readily prompt people to doubt the fairness of the experiments.

2. The results of the ablation experiments are rather confusing. It appears that MAWM is not consistently the best choice. Moreover, for the majority of the experiments, it doesn't seem that they have reached convergence.

3. The motivation of this paper is to solve the problem of struggling with irrelevant background information and omitting moving tiny objects. Is there any corresponding visualization to verify this?

4. Many experimental details are not clearly stated. For example, what are the values of $\alpha$ and $r_{dizzy}$?  

5. Discussions on limitations are necessary.
Yes
No ethics review needed.
Please see the weakness
The paper presents several strengths:
1. The experimental validation of the proposed model on two standard datasets, Atari 100k and DMC, demonstrates its effectiveness and generalizability.
2. The writing of the paper is clear, making it easy for readers to understand the proposed method and its experimental validation.
3. The details of the model in the appendix are valuable contributions to the community, as it enables other researchers to reproduce the results and build upon the proposed method.
The paper presents a novel approach in terms of model-based reinforcement learning (MBRL), where an image-based world model is utilized to improve sample efficiency by mimicking a scalable digital copy of the environment, such as DreamerV3. The authors claims that traditional world models tend to ignore moving tiny objects and their connections with tasks, thereby they propose a Motion-Aware World Model (MBRL) to account for them. Specifically, MAWM 1. focuses on small objects via pixel-level attention mechanisms and 2. deals with rapid changes of them via an adaptive control scheduler. The results on Atari 100k and DeepMind Control Suite (DMC) depict the superiority of the proposed method against DreamV3.
1. One concern is regarding the novelty of the proposed two techniques. The motion-aware auxiliary loss is not a novel topic [1]. Additionally, the technical contributions of the proposed pixel-level attention and scheduler are not clear enough to me.
[1] 3D Motion Decomposition for RGBD Future Dynamic Scene Synthesis, CVPR 2019. 

2. Is it possible to compare MAWM with pretrained video generation models like stable video diffusion, to figure out whether they can capture the patterns of moving targets or not?

3. I also have some concerns about the experimental results. As an example, Table 1 is confusing. In the last row, the Median score of a counterpart SimPLe is 1.34, which seems to obtain the best results without being marked bold. Besides, the different components proposed in this paper don’t demonstrate convincing improvement in Figure 4. A more comprehensive and significant ablation study could help address this.
Yes
No ethics review needed.
See the weakness section.
The proposed motion awareness in model learning is intuitive and the proposed method achieves comparative results with state-of-the-art methods.
This paper introduces a new method for learning the model in model-based RL with special considerations in motion prediction modeling. The authors introduces motion awareness by adding foreground motion prediction and adaptive motion-blur losses over traditional video generative pipelines. The resulting model achieves comparable results against existing MBRL methods.
One concern about this paper is the significance of the proposed method. First, except in curves and Fig.3 from the ablation studies, the authors might want to provide more visualizations on the effect of the motion awareness introduced, especially when considering the current experiment setting is only in atari and dm-control-suite where data are all synthetic and should potentially be simpler compared to real-world videos. Second, I wonder if the same pipeline could be applied to real-world videos as currently there is increasing trend in leveraging video generative models as "world models" to facilitate various tasks. It would be better if the authors could find a proper way to compare or address such line of methods. Lastly, the paper organization could be improved for better clarity.
Yes
No ethics review needed.
1. In ablation study (L486-L491), why video prediction loss instead of image reconstruction loss in MAWM framework is important for DM control experiments but not Atari experiments?
2. What is the intuition and rationale to use an adaptive Gaussian mixture model to extract ground truth label of whether each pixel belongs to foreground or background?
3. How does MAWM compare to MWM [3] that uses MAE objective on convolution features to better learn representations, which is also shown specifically more suitable for small objects?

[3] Seo, Younggyo, et al. "Masked world models for visual control." Conference on Robot Learning. PMLR, 2023.
1. The proposed approach shows improvements over baselines in benchmark evaluations
2. Authors perform various ablations to demonstrate effectiveness of each component
3. Paper is well structured and experiments are well organized
This paper tackles the problem that model-based visual RL methods tend to ignore small objects that are essential for tasks while learning irrelevant background information. The proposed approach MAWM introduces a pixel-level loss function based on motion information together with adaptive motion-aware scheduler based on timesteps and video prediction loss instead of image reconstruction as representation learning objective. Authors perform experiments on Atari and DM control benchmarks. Authors compared MAWM to several model-based and model-free RL baselines, and demonstrated improvements averaging across multiple tasks. Ablation studies show contributions of each component in this framework.
1. The main difference from existing model-based RL methods is an auxiliary objective based on pixel-wise motion information and replacing image reconstruction loss with video prediction objective, both of are not significant changes.
2. Since the motivation of MAWM is to avoid distractions from irrelevant background information, in addition to standard control benchmarks, environments such as distracted DM control [1] and DM control generalization [2] can better demonstrate benefits of the proposed approach than baselines.
3. Current experiments do not quantitatively establish correlation whether proposed approach has more benefits in environments or tasks where objects tend to be smaller, which is a main motivation in method design.

[1] Zhang, Amy, et al. "Learning invariant representations for reinforcement learning without reconstruction." arXiv preprint arXiv:2006.10742 (2020).

[2] Hansen, Nicklas, and Xiaolong Wang. "Generalization in reinforcement learning by soft data augmentation." 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.
@inproceedings{
anonymous2024visual,
title={Visual Representation Learning for World Models by Predicting Fine-Grained Motion},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=8BJl6LQgW5},
note={under review}
}
Originating from model-based reinforcement learning (MBRL) methods, algorithms based on world models have been widely applied to boost sample efficiency in visual environments. However, existing world models often struggle with irrelevant background information and omit moving tiny objects that can be essential to tasks. To solve this problem, we introduce the Motion-Aware World Model (MAWM), which incorporates a fine-grained motion predictor and entails action-conditional video prediction. The combination yields compact and robust representations of environments, filters out extraneous backgrounds, and keeps track of the pixel-level motion of objects. Moreover, we demonstrate that a world model with action-conditional video prediction can be interpreted as a variational autoencoder (VAE) for the whole video. Experiments on the Atari 100k benchmark show that the proposed MAWM outperforms current prevailing MBRL methods. We further show its state-of-the-art performance across challenging tasks from the DeepMind Control Suite.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
world models model-based reinforcement learning visual representation learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/fd195726918897cc075c3d050b5f7acd1b10b390.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Visual Representation Learning for World Models by Predicting Fine-Grained Motion
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission