Inter-Environmental World Modeling for Continuous and Compositional Dynamics
world model;state space model;object centric learning;symmetry
Yes
No ethics review needed.
- In Eq. (1), does the function Ctrl return an observation or a transition operator?
- In Eq. (3), shouldn't one take the limit of delta to 0 instead of t to infinity?
- Are there any limitations arising from the restriction of transitions to Lie groups?
- Since the latent space is divided into different slots for different objects and the slots forward dynamics are independent of each other, how can the model represent interactions between objects? 
- It seems that the size of the latent space is a crucial hyperparameter; how was it determined (both N and J)?
- The paper is clearly structured and well written
- The use of the Lie group theory and NFT to model the dynamics of interactive environments seems to be both novel and interesting
- The method is well motivated and theoretically grounded
- Good reproducibility (code provided)
The paper proposes a method akin to state-space models for simulating environments by applying Neural Fourier Transform. It uses Lie group theory to obtain a continuous representation of the environment dynamics.
- No runtime analysis
- No discussion of hyperparameter selection
- No significance tests
- No error bars
- No scaling laws
- No quantitative results on Phyre

Unfortunately, the empirical evaluation does not meet scientific standards and, therefore, I do not deem the paper ready for publication yet. Once these problems are solved, I think this paper will constitute a nice contribution to the field.
Yes
No ethics review needed.
- How are the parameters $N$ and $J$ chosen?  Do they need to scale with the complexity of the environment?  If so, how do we expect computation costs to scale? 
- Line 192 suggests an assumption about infinitesimal changes to observations between timesteps.  This assumption seems very strong, how would the method handle larger changes?
- How similar do the environments need to be for the inter-environment simulator to model them jointly?  For example, could you mix the Phyre and ProcGen environments?
- Does it matter how the dataset is sampled from each environment?
- The paper presents an interesting idea, and a clever method for exploiting similarities between several environments at once.
- Learning reliable world models from observation trajectories without paired actions would be a dramatic step forward, and this paper presents a novel angle on that problem.
- Evaluation shows improvements over a strong baseline.
The paper proposes a method for enforcing a Lie group structure on the latent space of a multi-environment world model.  This structure imposes compositionality and continuity in the latent space, and allows the dynamics in the latent space to be linear.  The latent space is further structured by imposing object-specific dynamics (through the use of slot attention and assignment of "close" slots in nearby time steps).  This method allows to learn a common dynamic model from a sequence of observations sampled from a set of environments, and the paper shows that this model can generate observations that better reconstruct the ground truth observation compared to a strong baseline.
- In places the presentation can be a bit unclear.  This is mostly in the description of the method.  I would have benefitted from a diagram of the steps required to train WLA and then use that to solve the CIP.  The current Figure 1 may not be necessary; the inter-environment aspect of WLA is probably the most straightforward step in the process.  
- Although not direct alternatives to the latent space presented here, there is some previous work in structured latent spaces that it may be interesting to compare against in the related work section.  eg.
    - Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images.  Manuel Watter, Jost Tobias Springenberg, Joschka Boedecker, Martin Riedmiller
    - Object Files and Schemata: Factorizing Declarative and Procedural Knowledge in Dynamical Systems.  Anirudh Goyal, Alex Lamb, Phanideep Gampa, Philippe Beaudoin, Sergey Levine, Charles Blundell, Yoshua Bengio, Michael Mozer
- Not a weakness, but Line 111 contains a reference to OCCAM, which I presume is a previous name for WLA?
Yes
No ethics review needed.
1. Was the encoder and decoder trained from scratch?
2. Did you do multiple runs with different random initializations, or are the results from just one experimental run?
1. The paper is clearly structured, and the individual components of the model with accompanying formulas, as well as the setting is well introduced. The authors carefully set the scope of the work and how it compares to similar approaches.
2. Benchmarks with different test settings (e.g. different FPS compared to train) shows the robustness of the model on this benchmark and the ability to infer continuous dynamics.
3. Given a main focus of the framework is the Lie group theory, I appreciate the ablation study that reveals the relevance of it (Rotation) to the predictive performance of the model
In this paper, the authors present World modeling through Lie Action (WLA), a generative state-space-modeling framework that can be trained in an unsupervised fashion on interactive environments, and that allows controllable generation of future frames of a given environment. One main goal of this framework is learning continuous and compositional action representations, similar to what humans are able to do. It is a step towards an interactive world model that generalizes across environments, given these environments have a common basic rules of composition and continuity.
The paper presents a model built using this framework that is benchmarked on two datasets (ProcGen and Phyre). In general, the model can be trained using no or only few action labels, making it more versatile and faster to adapt to new settings with different action labels.
My main concern is the limited amount of comparability to previous works and approaches. The model is only benchmarked on two datasets, and one is mainly used as a (successful) sanity check. The second dataset (with 16 quite diverse environments) is benchmarked against only one other model, though using several metrics. Even if it is a ‘first of its kind’ framework as stated by the authors, more benchmarks to confirm its proper function would increase trust in the method/framework. It can be acknowledged though that the number of available datasets/benchmarks for this specific scope is low, as also stated in a recent survey [1].
Other things that are not clear to me at the moment are in the question section.


[1] McCarthy, Robert, et al. "Towards Generalist Robot Learning from Internet Video: A Survey." arXiv preprint arXiv:2404.19664 (2024).
Yes
No ethics review needed.
Revise the math definitions and formulations significantly and I will reconsider my score.

See weakness for all the math confusions.
The originality seems novel. The universal interface for control is an important problem to study. The emprical evidence seems to be strong, it can outperform the baseline significantly. I think there could be some contributions in the paper, but the presentation needs to be improved a lot.
The paper introduces World Modeling through Lie Action (WLA), a generative framework that learns continuous, compositional actions for controlling agents across diverse environments using Lie group theory. Claimed contributions include:

Unified Simulator: A shared, environment-agnostic simulator leverages continuous actions, unlike models with discrete, environment-specific setups.
Controller Interface Problem (CIP): WLA solves CIP with adaptable controllers, even under minimal labeling.
Object-centric Modeling: Slot-based object-centric modeling supports interactions across multiple objects and actions.
Performance: WLA seems to outperform on benchmarks (Phyre and ProcGen), proving its generalizability and adaptability.
This approach enhances state-space modeling with flexible, multi-environment simulations
The paper’s ambitious framework for inter-environmental modeling has notable weaknesses, especially in mathematical clarity and formulation, which hinder its accessibility and rigor.
 Key Issues:
1. **Ambiguity in Transition Operator (\( g_{t, \delta} \))**: \( g_{t, \delta} \) maps specific observations \( x(t) \) to \( x(t+\delta) \), making it appear as a trivial one point to one point mapping. This definition does not capture the desired dynamic evolution. It does not make any sense in the form written in the paper. The paper acknowledges that \( g_{t, \delta} \) depends on individual trajectories but simplifies it as a generic operator, risking confusion. This omission detracts from the model’s mathematical precision, especially for multi-environment dynamics.

3. **Underdefined Action Space \( A \)**: The action space \( A \) and action \( a(t, \delta) \) lack clarity on whether actions are fixed or variable over intervals. This ambiguity in structure reduces the framework's comprehensibility in continuous control scenarios.

4. **Non-Compositional Transition Operators**: Due to the triviality of \( g_{t, \delta} \), composing transitions over time is not feasible, so why does it form a Lie group? The paper introduces Lie groups without a clear justification for their relevance to the specific environments. Additional reasoning would strengthen its argument for using Lie groups to model compositional and continuous dynamics.

To improve clarity, the paper would benefit from revisiting the definitions and dependencies in its transition operators, better defining the action space, and providing justification for its mathematical choices. More precise formulation and notation would significantly enhance its accessibility and application.
@inproceedings{
anonymous2024interenvironmental,
title={Inter-Environmental World Modeling for Continuous and Compositional Dynamics},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=cojJ2s1e35},
note={under review}
}
Recent advancements in generative models have showcased their effectiveness across various domains such as language, chemistry, and genomics. However, these models often rely on discrete action representations or language-based interactions, limiting their ability to generalize across different environments without explicit labels. In this paper, we introduce the Object-Centric Algebraic Mental Model (OCAM), an unsupervised framework that learns a continuous latent action representation by combining object-centric representations with Lie group theory. OCAM treats actions as continuous operators acting linearly on a partitioned latent vector space corresponding to different objects and basic action axes. This approach allows OCAM to generalize beyond the environments it was trained on, enabling the simulation of new environments and efficient solving of inverse dynamics problems. We demonstrate that OCAM can learn interactive generative models from unlabeled video data, interpolate between frames, extrapolate long-term dynamics, and quickly adapt to new environments with minimal or no action labels. Experiments on benchmark datasets like PHYRE and ProcGen validate the effectiveness of OCAM in advancing the capabilities of scalable artificial general intelligence.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
world model state space model object centric learning symmetry
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/bbde2df13bffc1f413033bb07707a99271fb78e1.pdf
learning on time series and dynamical systems
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/9c5e4bce69cafbb6ef8e3371ae120ce2df67e410.zip
Inter-Environmental World Modeling for Continuous and Compositional Dynamics
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission