A Temporally Correlated Latent Exploration for Reinforcement Learning
Colored Noise;Intrinsic Reward;Latent State Representation;Reinforcement Learning;Temporal Correlation
Yes
No ethics review needed.
- In Ablation Study 1, the authors show that the CVAE's action conditioning helps one of the considered problems but not the other. In lines 503-506, the authors mention that this is because of better self-supervised learning. It is not clear to me precisely what that statement means. Could the authors elaborate on why action conditioning is helpful?  

- Is there a reason to use the Number of Updates as the x-axis in Figures 5/7 and the Number of Steps in Figures 6/8? Using the Number of Steps in all cases could be better, or specifying the correspondence between updates and steps (and frames in Atari) in the main text.
S1. Using temporally correlated noise on a latent representation of inverse dynamics model features appears novel (but also see W2).

S2. The approach outperforms popular intrinsic bonuses like RND and ICM in Minigrid environments with noisy TVs and also in many of the considered atari environments.
This paper aims to develop an approach to providing intrinsic rewards more robust to stochasticity (and noisy-TV scenarios) than previous approaches. The proposed approach–TeCLE–uses inverse dynamics modeling to learn a state representation and uses the action-conditioned state reconstruction loss (through a conditional VAE) as an intrinsic reward. While sampling the latent representation from the encoder’s output, temporally correlated noise is added (modulated by $\beta$). Empirical analysis shows that TeCLE outperforms bonuses such as RND and ICM in Minigrid environments with noisy-TV and some stochastic Atari environments. The authors also illustrate qualitative differences in exploration behavior based on the color of noise added.
W1. From the aggregated and normalized results in Figure 2, all noise levels seem to perform similarly (high overlap in standard errors), even white noise with $\beta=0$. This seems to indicate that colored/temporally correlated noise may not be so important to the performance of the proposed approach, which is a major theme of the paper.

W2. While the paper describes approaches that add noise in the action space, it misses comparisons with approaches that add noise in parameter space [1, 2], which would be closer to adding noise in some latent space. Similarly, another work adds temporally correlated noise to the latent state for exploration [3]. It would be helpful to compare the differences with the approach introduced here. Other forms of temporally correlated exploration, like temporally extended epsilon-greedy [4] could also be interesting to consider in the atari evaluation.

W3. The clarity of the paper requires improvement. Section 4A describes the inverse dynamics modeling task used for feature embedding. As far as I can tell, this is the same idea of feature embedding used in ICM. While the authors describe the ICM paper in the Background section, the current presentation of feature embedding in Section 4A is more likely to be incorrectly interpreted as a novel contribution of this paper. It should be clearly mentioned that the representation/feature learning part is the same as that of ICM. If there is a difference, please clarify what is new in that section. 

Further, since ICM and TeCLE use the same feature embedding (which captures the same information about states), the difference in robustness to stochasticity between the two approaches must arise from the tasks used to generate intrinsic rewards (model error vs. reconstruction error), and it is not clear why reconstruction error would work better. Perhaps the authors could comment and elaborate on why TeCLE works better than ICM.

Please also see the “Questions” section for additional details that weren’t clear to me. I remain open to increasing the score should the weaknesses and questions be adequately addressed/clarified.

### References

[1] Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al. Noisy networks for exploration. arXiv preprint arXiv:1706.10295, 2017


[2] Rückstiess, T., Sehnke, F., Schaul, T., Wierstra, D., Sun, Y., & Schmidhuber, J. (2010). Exploring parameter space in reinforcement learning. Paladyn, 1, 14-24.

[3] Chiappa, A. S., Marin Vargas, A., Huang, A., & Mathis, A. (2023). Latent exploration for reinforcement learning. Advances in Neural Information Processing Systems, 36.

[4] Dabney, W., Ostrovski, G., & Barreto, A (2021). Temporally-Extended ε-Greedy Exploration. In International Conference on Learning Representations.
Yes
No ethics review needed.
N/A
The connection between temporal correlation and the NoisyTV problem could be interesting.
This paper proposes to add temporal correlation to intrinsic rewards computation by learning an action conditional C-VAE for regularizing the representation of the state representation. The experiments in a few MiniGrid and Atari games show some improvements.
## Writing
Overall, this paper only states how they implement the method but barely talks about the motivation behind them, which will make the reader lose. For example, in Line 48, the author is supposed to say how they address the noisy tv problem mentioned in the previous paragraph, but the author directly jumps in the implementation details of their method and never mentions why temporal correlation can mitigate NoisyTV problems. There are more examples like this, but it will take forever to enumerate all of them.

## Experiments
This paper lacks a lot of stronger baselines in the chosen environments (e.g., [1]). RND and ICM are not state-of-the-art in those environments. Moreover, only a fraction of Atari games are chosen, and the most classic hard-exploration problem, Montezuma Revenge, is not studied, which makes the results questionable.


[1] Zhang, Tianjun, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E. Gonzalez, and Yuandong Tian. "Noveld: A simple yet effective exploration criterion." Advances in Neural Information Processing Systems 34 (2021): 25217-25230.
Yes
No ethics review needed.
See Weaknesses.
- The proposed method is a novel combination of action-conditioned latent space and temporally correlated signals for addressing the noisy-TV problem.
- The empirical evaluations is comprehensive, with extensive experiments and ablation studies across multiple environments.
- The propose method empirically exhibits robustness with respect to noise.
This paper proposes Temporally Correlated Latent Exploration (TeCLE), a novel curiosity-driven exploration method for reinforcement learning that uses action-conditioned latent spaces and temporal correlation, for addressing the noisy-TV issue, a long-standing problem in efficient exploration in RL. The authors propose to use action-conditioned latent space to model state distributions and compute intrinsic rewards based on reconstruction error. Temporal correlation is induced through colored noise in the latent sampling process, and different noise profiles lead to different exploratory behaviours. The method is evaluated on Minigrid and Stochastic Atari environments, showing improved robustness to the Noisy TV problem and stochasticity compared to baselines like ICM and RND.
- Empirical evaluations on hard exploration tasks (e.g., Montezuma's Revenge) show modest improvement.
- The theoretical grounding between the injected temporal correlations and resulting exploratory behaviour requires further justification.
- The proposed method induces significant computational overhead compared to simpler methods.
Yes
No ethics review needed.
Could you extend on your motivation for colored noise? I understand that temporal correlation and anti-correlation will affect the smoothness of the intrinsic reward. But how does it relate to the noisy TV problem?
- The description of the approach is clear.
- While colored noise was previously used for sampling temporal correlated actions for better exploration, injecting temporal correlations while sampling the latent of the VAE is novel.
The paper proposes a method for improving exploration in RL using intrinsic motivation. Similar to ICM, a state embedding is learned with respect to an inverse dynamics loss and the error between a predicted embedding of the next state and the actual embedding of the next state serves an auxiliary reward. However, unlike ICM, the predicted embedding of the next state is obtained by a forward model but by reconstructing the next state embedding using a VAE. The paper proposes to use an action-conditioned C-VAE and to use colored noise while sampling the latent variable of the C-VAE to induce temporal correlation (or anti-correlation). The method is evaluated on suitable gridworlds (DoorKey) and Atari environments, where it achieved better results than the baselines (ICM and RND).
- The contributions are quite limited. The main contribution seems to be sampling the VAE latent with colored noise.
- The use of colored noise while sampling the VAE latent is not well motivated.
- Related work is insufficient discussed. For example, Kubovcik et. al used the reconstruction of an auto-encoder as intrinsic reward, Klissarov et al. also used a variational autoencoder, but used the KL between the posterior and prior for intrinsic reward, and, recently, Yan et al. also proposed to use the reconstruction error of a VAE for intrinsic reward but extended it with adaptive exploration mechanism, based on a discriminator.
- The empirical evaluation is insufficient. Table 1 does not show confidence intervals or baseline results. It is not clear how important hyperparameters (e.g. weight of the intrinsic reward) for the baseline were selected and which values were used, and, furthermore, important baselines are missing, e.g. the above mentioned related works.
- The writing could be improved and is often inaccurate. For example the abstract states that the latent space "models" the probability distribution of states, and that the environment where the agent performs best depends on the amount of temporal correlation. Some claims are also questionable, e.g. that the existing curiosity-driven methods are vulnerable to Noisy TV problems (some are, but not all of them)  

References:

Martin Kubovcik, Iveta Dirgova Luptakova and Jiri Pospichal. Signal Novelty Detection as an Intrinsic Reward for Robotics. Sensors 2023.

Martin Klissarov, Riashat Islam, Khimya Khetarpal and Doina Precup. Variational State Encoding as Intrinsic Motivation in Reinforcement Learning. Task-Agnostic Reinforcement Learning Workshop at ICLR. 2019.

Renye Yan, Yaozhang Gan, You Wu, Ling Liang, Junliang Xing, Yimao Cai and Ru Huang. The Exploration-Exploitation Dilemma Revisited: An Entropy Perspective. ArXiV 2024.
A novel intrinsic reward formulation that employs an action-conditioned latent space and temporal correlation
@inproceedings{
anonymous2024a,
title={A Temporally Correlated Latent Exploration for Reinforcement Learning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=uGka5qOsop},
note={under review}
}
Efficient exploration remains one of the longstanding problems of deep reinforcement learning. Instead of depending solely on extrinsic rewards from the environments, existing methods use intrinsic rewards to enhance exploration. However, we demonstrate that these methods are vulnerable to Noisy TV and stochasticity. To tackle this problem, we propose Temporally Correlated Latent Exploration (TeCLE), which is a novel intrinsic reward formulation that employs an action-conditioned latent space and temporal correlation. The action-conditioned latent space models the probability distribution of states, thereby avoiding the assignment of excessive intrinsic rewards to unpredictable states and effectively addressing both problems. Whereas previous works inject temporal correlation for action selection, the proposed method injects it for intrinsic reward computation. We find that the injected temporal correlation determines the exploratory behaviors of agents. Various experiments show that the environment where the agent performs well depends on the amount of temporal correlation. To the best of our knowledge, the proposed TeCLE is the first approach to consider the action-conditioned latent space and temporal correlation for curiosity-driven exploration. We prove that the proposed TeCLE can be robust to the Noisy TV and stochasticity in benchmark environments, including Minigrid and Stochastic Atari.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Colored Noise Intrinsic Reward Latent State Representation Reinforcement Learning Temporal Correlation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/b93c9d25fcc67e870eddbd2f159380e9c117f0b2.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/9dcc572a4362c63501d862ce2822ac67435e2523.zip
A Temporally Correlated Latent Exploration for Reinforcement Learning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission