On Rollouts in Model-Based Reinforcement Learning
Model-Based Reinforcement Learning;Model Rollouts;Uncertainty Quantification
Yes
No ethics review needed.
1. Is the model error come solely from epistemic noise?
2. What is the advantage of using AES models, and why do you believe Infoprop does not impact their effectiveness?
1. **Significance**: Model error remains a significant challenge in MBRL, and Infoprop addresses it through the rollout process.
2. **Extensive Experimentation**: The authors provide comprehensive experiments and analysis, including detailed evaluations of algorithm trajectories and environment buffers, which validate the effectiveness of the Infoprop algorithm.
This paper proposes Infoprop, a rollout mechanism designed to improve sample efficiency and asymptotic performance in model-based reinforcement learning (MBRL). Infoprop addresses the issue of epistemic model uncertainty typically introduced by ensemble methods or dropouts. It projects sampled next states into the maximum likelihood distribution of the predicted next state distribution and uses an information-theoretic criterion to determine the termination point of rollouts. When applied to Dyna-style MBRL, Infoprop demonstrates state-of-the-art performance, as shown in experiment results.
1. **Assumptions**: The approach relies on two key assumptions. It is important to empirically assess the extent to which these assumptions hold in experimental environments.
2. **Organization**: The theoretical section could benefit from clearer structure. Presenting the main results in the form of definitions or theorems could improve readability. Simplifying technical details and focusing on the implementation—particularly regarding Equation (14)—would make the logical flow smoother.
Yes
No ethics review needed.
The main limitations of the work, as identified above, are in the quality and significance as discussed above. These could be alleviated with a more in-depth experimentation which would lead to a more mature solution, and a more in-depth discussion of related work. All of them require space, and a such my questions relate to the guidance for the discussion section.

Specifically:
1. where is overfitting most prominent with Infoprop?
2. where there other information theoretic metrics (e.g redundancy) considered in the Infoprop design? If no, why not? if yes, why were they discarded?
Soundness
======
The paper uses the idea of marginal entropy and mutual information to determine data corruption and stop a rollout in MBRL. The idea is presented carefully with a derivation for the main result.
  
Significance & Related work
=========
The paper has a brief related work section, as part of the main paper, which is appreciated. However, there is a need for a deeper analysis of existing work, including  more in-depth comparisons that show the benefit of Infoprop to others, specifically including work in using simulated rollouts to guarantee performance, robust MBRL (Sung 2024, Kuo 2021), risk sensitivity (e.g. Webster 2021) and other uses of information theory in MBRL.

Presentation
=========
The paper is well written with few typos, usually around empty spaces in the text.

Quality
======
My main concerns are around the quality of the experimentation (see below) and on the need for authors to explore the limitations of their approach a bit deeper before publication. For example, the authors mention that the rollouts exhibit excellent data quality and yet there are instabilities: is this a function of the integration mechanism or of overfitting?
This paper presents a rollout mechanism that uses information theory to reduce the impact of epistemic uncertainty in MBRL. The paper presents some experimental analysis and a derivation of a result related to information loss.  The paper uses the idea of marginal entropy and mutual information to determine data corruption and stop a rollout.
Experimentation
=========
The experimental analysis show the benefits of Infoprop when it runs in simple environments, however deeper experimentation is needed. For example, I would have expected to see some ablation studies showing the impact of the choice of lambda, instability, overfitting and other issues. 

Overall the paper presents a very promising initial idea with several avenues of experimentation that need to be explored to ensure its maximum potential is reached.
Yes
No ethics review needed.
* How does eq 14 actually follow from eq 4 and 9? 
* When the authors say eliminate epistemic uncertainty, does that mean setting the last term in eq 9 to zero? If so is the rest of the derivations simply accounting for $\mu^{\Delta}$?
The proposed method is original and sound. The experiment hypotheses are very clear the the results clearly demonstrated the advantage of the proposed method.
This paper proposes a model rollout method for model-based reinforcement learning called infoprop. The main idea is to control for epistemic uncertainty using model rollout to reduce model error accumulation and in turn improve value or policy optimization. The concrete method is a re-weighting of ensemble predictions and an entropy-based criteria to stop rollout. The Mujoco experiments demonstrated substantial improvement in reducing error accumulation as well as convergence speed.
The paper could benefit a lot from intuitive explanations of the proposed approach. For example, what's the intuition of the covariance intersection fusion result for readers that are not familiar with this literature.
Yes
No ethics review needed.
In Figure 6, INFOPROP Dyna does not outperform MACURA. However, if I understand correctly, the proposed method can be used as a plugin within MACURA, which makes it puzzling that INFOPROP Dyna does not surpass MACURA.
The author developed a new method, Infoprop, that enhances the performance of Dyna-style model-based RL:

*Versatility*: The Infoprop method can be integrated as a plugin for Dyna-style model-based RL, making it a flexible and adaptable enhancement for existing frameworks.

*Improved Data Quality*: By increasing the quality of data rollouts from the estimated model, Infoprop enhances the overall performance of Dyna-style RL approaches.

*Experimental Results*: The experiments conducted in Mujoco environments demonstrate state-of-the-art (SOTA) performance, showcasing the effectiveness of Infoprop in most environments, with the exception of the humanoid scenario.

*Theoretical Foundation*: The method is backed by robust theoretical analysis, explaining how the plugin improves rollout quality from the estimated model, which adds credibility to the approach.
The author presents a new method called *Infoprop*, which can be used as a plugin for Dyna-style model-based RL. The plugin increases the quality of data rollouts from the estimated model, thus enhancing the Dyna-style model-based RL. The experiment in Mujoco showed the SOTA performance except for the humanoid environment. The author also provides a solid theoretical analysis of why the plugin can increase the quality of rollouts based on the estimated model.
1. The theoretical results in Section 4 are difficult to follow. A summary of the main result should be given at the start of the section. Also, I suggest the author clearly present the main result as a theorem. 
2. Too many notations have been introduced, which could be simplified. For example, Section 2.4 could be condensed into a single sentence, accompanied by equation (5) and relevant citations since it represents a well-known formulation in model-based RL.
3. The enhancement applies specifically to Dyna-style RL, not all model-based RL approaches. Making the title misleading.
4. The Assumptions at the end of Section 4.2 are stated without any explainations.
We propose Infoprop a novel rollout mechanism for model-based reinforcement learning yielding substantially improved data consistency and long-term planning capabilities.
@inproceedings{
anonymous2024on,
title={On Rollouts in Model-Based Reinforcement Learning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Uh5GRmLlvt},
note={under review}
}
Model-based reinforcement learning (MBRL) seeks to enhance data efficiency by learning a model of the environment and generating synthetic rollouts from it. However, accumulated model errors during these rollouts can distort the data distribution, negatively impacting policy learning and hindering long-term planning. Thus, the accumulation of model errors is a key bottleneck in current MBRL methods. We propose Infoprop, a model-based rollout mechanism that separates aleatoric from epistemic model uncertainty and reduces the influence of the latter on the data distribution. Further, Infoprop keeps track of accumulated model errors along a model rollout and provides termination criteria to limit data corruption. We demonstrate the capabilities of Infoprop in the Infoprop-Dyna algorithm, reporting state-of-the-art performance in Dyna-style MBRL on common MuJoCo benchmark tasks while substantially increasing rollout length and data quality.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Model-Based Reinforcement Learning Model Rollouts Uncertainty Quantification
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/ad69712a87260f7ed83cc74bf5b5736f5af365f1.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/1700530ff1518d021069fa2e8caf530d25f0c64a.zip
On Rollouts in Model-Based Reinforcement Learning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission