Episodic Novelty Through Temporal Distance
Reinforcement Learning
Yes
NA
No ethics review needed.
1. Could the authors shed light on why PPO, an on-policy algorithm, was chosen to be combined with ETD instead of off-policy algorithms such as DQN or SAC? Prior work has used off-policy algorithms extensively for sparse-reward tasks [2] [3].
2. Is ETD straightforward to be extended to continuous action spaces? If so, did the authors run any experiments? If not, what could be possible bottlenecks?
3. Did the authors consider combining temporal distance with reward-based similarity, or explore weighting the temporal distance based on extrinsic rewards? Are there any potential trade-offs or challenges in incorporating extrinsic rewards into the similarity metric that the authors anticipate?

[2] Andrychowicz, Marcin, et al. "Hindsight experience replay." Advances in neural information processing systems 30 (2017).

[3] Co-Reyes, John D., et al. "Evolving reinforcement learning algorithms." arXiv preprint arXiv:2101.03958 (2021).
1. The paper introduces a novel method to use temporal distance as a (quasi)metric for state similarity.
2. The paper conducts extensive experiments across multiple CMDP environments, comparing ETD to several baseline methods.
3. ETD + PPO demonstrates robust performance improvements, especially in challenging sparse reward scenarios.
4. Results on extensive experiments across multiple CMDP environments, comparing ETD to several baseline methods have been reported.
5. The paper is well-structured, with clear explanations of the motivation, methods, and results.
This paper introduces Episodic Novelty through Temporal Distance (ETD), a new approach for exploration in Contextual Markov Decision Processes (CMDPs) with sparse rewards. ETD uses temporal distance as a metric for state similarity to compute intrinsic rewards, encouraging agents to explore states that are temporally distant from their episodic history. The authors employ contrastive learning to design a quasimetric that estimates episodic temporal distances. They evaluate  ETD when combined with PPO on various CMDP benchmarks, i.e., MiniGrid, and Crafter (pixel-based observations), and MiniWorld (pixel-based observations).
1. The proposed ETD doesn’t take into consideration extrinsic rewards to compute similarity. Intuitively, states with similar rewards could be considered similar in terms of the task objective [1].
2. The approach has been primarily tested on discrete action spaces, and its effectiveness in continuous action domains such as MuJoCo [2], DeepMind Control Suite [3], or Fetch [4] environments remains unexplored.

[1] Agarwal, Rishabh, et. al. “Contrastive behavioral similarity embeddings for generalization in reinforcement learning.” arXiv preprint arXiv:2101.05265 (2021).

[2] https://gymnasium.farama.org/environments/mujoco/

[3] Tassa, Yuval, et al. "Deepmind control suite." arXiv preprint arXiv:1801.00690 (2018).

[4] Plappert, Matthias, et al. "Multi-goal reinforcement learning: Challenging robotics environments and request for research." arXiv preprint arXiv:1802.09464 (2018).
Yes
No ethics review needed.
Please address the issues listed in the weaknesses section.
- The paper is well-written, with a clear and cohesive narrative. Most technical details are effectively conveyed through illustrative figures and results from intuitive toy tasks.
- The experimental tasks are appropriately chosen, providing sufficient complexity to evaluate the approach.
- The experimental results, along with ablation studies, clearly demonstrate the advantages of the proposed method over other baselines.
- The paper offers a comprehensive analysis and comparison of different technical approaches used by the proposed method and baseline methods, providing valuable insights for readers.
This paper presents a novel reinforcement learning approach to measure similarities between visited states, using the resulting state distances as a reward bonus to encourage exploration in tasks with sparse rewards. The methodology is technically sound, and the results are promising.
## MDP assumption
The definition of the intrinsic bonus reward violates the MDP assumption. In section 2, the total reward $r(s_t, a_t, s_{t+1})$ is a decomposed as the environment reward $r_t^e$ plus the weighted bonus $\beta b_t$. However, $b_t$ is a function that depends on the visited states within the episode, which disrupts the definition of total reward and violates the MDP assumption. In this case, the visited states influences the intrinsic reward, potentially harming the policy built upon MDP, as the actions are solely selected by $\pi(a_t|s_t)$ without knowledge of the states' visiting history. Consequently, the same action $a_t$ may yield completely different rewards $r_{t+1}$, depending on whether the $s_{t+1}$ has been visited in the history or not. I believe this is a critical technical issue in this paper, and should be explained thoroughly. At the same time, do other related works using reward bonuses also encounter the same issue?

## Unclear content
The content between lines 186 and 208 is challenging to follow. I suggest the authors restructure this section, providing additional explanations and intuitions. Specifically:
- The meaning of the sentence, "Contrastive learning can estimate the successor distance when positive samples are drawn from the discounted state occupancy measure," is unclear. What does it mean to draw samples from a measure?
- The motivation for using the energy function and its decomposition into the potential net and quasimetric net in practice is unclear. What is this energy function, and why is it applied here?
- The rationale behind using the infoNCE loss also needs further explanation.

## Minor issues
- The SpiralMaze in Figure 2 appears to have a linear structure, which may not effectively demonstrate ETD's advantage in distance learning. A more complex example could better illustrate the method's strengths.
- What is the best practice of choosing the $\beta$? I think it depends on the specific task and is therefore a HP.
- Line 188, there should be a $ds$ in the integral.
Yes
No ethics review needed.
- a more detailed discussion on the successor distance and its assumptions would be insightful. E.g., what happens if the MDP does not allow for loops (which is the case we have to deal with real dynamics, e.g. velocities). Then this distance would be infinite, wouldnt it?
- More insights on the parametrization of the energy function. Why do we need separate networks for  the potential and the assymetric distance? Could we also ablate removing the potential here?
- Why do we need to "symmetrize" the infoNCE loss? Does this imply that transitions are "invertible"? Also here, a ablation and more discussion would be nice.
- A well motivated new intrinsic reward signal for RL
- The concept is simple and provides a strong performance
- Exhaustive evaluations, good comparisons the baseline methods as well as nice ablations
- Good insights why baseline methods fail
The paper presents a new intrinsic reward for novelty-based reinforcement learning. The intrinsic reward is based on the estimated temporal distance between two states, which estimates the probability that one state can be reached from another state in multiple time steps. Here, discounting is used to give preference for shorter transitions. The temporal distance is learned by a contrastive loss based on the infoNCE loss. The algorithm is compared against several baseline methods on a wide variety of discrete action exploration tasks and shows promising performance.
the paper is already of high quality and I could not find major weaknesses. Minor weaknesses are given in the questions section.
Yes
No ethics review needed.
1. Please test the performance of ETD on continuous control tasks.
To evaluate the ETD method in continuous control tasks, the authors could consider tasks in environments such as Meta-World (or Panda-Gym) and Mujoco. Examples of basic tasks include "Push," "Reach," and "Pick-Place" in Meta-World (or Panda-Gym), which offer high environmental complexity and sparse reward settings, making them suitable for assessing ETD's exploratory capabilities. Additionally, Mujoco environments like "Cheetah-Vel-Sparse," which have high-dimensional state spaces, could further validate ETD’s performance in high-dimensional continuous spaces. By plotting and comparing learning curves across environments—such as average episode rewards or success rates over training steps—the authors could demonstrate whether ETD converges faster or more stably in sparse reward continuous control tasks compared to other methods.

2. Please provide code.
The paper compares the proposed method with various existing approaches (including DEIR, NovelD, E3B, EC, NGU, RND, and the Count-based method), covering many classical and common algorithms in the exploration field. By testing these methods across multiple benchmark environments, the paper demonstrates the advantages of the ETD method in sparse reward and high-dimensional pixel environments, validating its applicability across different exploration demands and tasks.
This paper addresses the challenge of exploration in sparse reward environments within Contextual Markov Decision Processes (CMDPs), where environments vary between episodes. The authors propose Episodic Novelty Through Temporal Distance (ETD), a method that leverages temporal distance as a metric for state similarity and intrinsic reward computation. Using contrastive learning, ETD estimates temporal distances to assign intrinsic rewards based on the novelty of states within an episode. Experimental results on diverse benchmark tasks show that ETD outperforms state-of-the-art methods, enhancing exploration in sparse reward CMDPs. The paper is well-structured and detailed, and the ETD method is well-described.
1. The core approach of ETD primarily builds upon the main idea of "Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making," using contrastive learning to estimate temporal distance. It lacks substantial algorithmic innovation.
The innovation of the ETD (Episodic Temporal Distance) method primarily lies in its approach to learning temporal distances. However, this method closely follows key aspects of the referenced work, "Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making," in several crucial areas:
1) Discounted State Occupancy Measure: In the ETD method, the discounted state occupancy measure p^\pi_\gamma(s_f = y | s_0 = x) is used to estimate the transition probability from state x to y . The formula is as follows: p^\pi_\gamma(s_f = y | s_0 = x) = (1 - \gamma) \sum_{k=0}^{\infty} \gamma^k p^\pi(s_k = y | s_0 = x) This formula originates from "Learning Temporal Distances," where the discounted state occupancy measure p^{\pi}_{\gamma} is defined as the discounted distribution over future states s^{+} : p^{\pi}_{\gamma}(s^{+} = s' | s_0 = s) = (1 - \gamma) \sum_{k=0}^{\infty} \gamma^k p^{\pi}(s_k = s' | s_0 = s)
2) Successor Distance: The successor distance formula used in ETD is also directly derived from the referenced work, "Learning Temporal Distances."
3) Loss Function: Both works adopt a symmetrized InfoNCE loss function to train the model through contrastive learning, which is a critical component of the approach.
4) Energy Function: The energy function utilized in ETD, f_{\theta = (\phi, \psi)}(x, y) = c_{\psi}(y) - d_{\phi}(x, y) , is also sourced from "Learning Temporal Distances," where it is presented as: f_{\phi, \psi}(s, a, g) = c_{\psi}(g) - d_{\phi}(s, a, g)


2. Experiments are conducted solely on discrete control tasks, with no tests on continuous control tasks (e.g., robot control tasks in Meta-World and Mujoco). State spaces in discrete control tasks are relatively small, allowing agents to explore enough states within a shorter timeframe. In contrast, continuous control tasks present larger state spaces and greater exploration challenges, making it essential to validate the method’s effectiveness in these scenarios. Given that many real-world tasks involve continuous control, this omission limits the method's persuasiveness for broader application scenarios.
3. The absence of publicly available code restricts the ability to independently verify the method's effectiveness.
@inproceedings{
anonymous2024episodic,
title={Episodic Novelty Through Temporal Distance},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=I7DeajDEx7},
note={under review}
}
Exploration in sparse reward environments remains a significant challenge in reinforcement learning, particularly in Contextual Markov Decision Processes (CMDPs), where environments differ across episodes. Existing episodic intrinsic motivation methods for CMDPs primarily rely on count-based approaches, which are ineffective in large state spaces, or on similarity-based methods that lack appropriate metrics for state comparison. To address these shortcomings, we propose Episodic Novelty Through Temporal Distance (ETD), a novel approach that introduces temporal distance as a robust metric for state similarity and intrinsic reward computation. By employing contrastive learning, ETD accurately estimates temporal distances and derives intrinsic rewards based on the novelty of states within the current episode. Extensive experiments on various benchmark tasks demonstrate that ETD significantly outperforms state-of-the-art methods, highlighting its effectiveness in enhancing exploration in sparse reward CMDPs.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Reinforcement Learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/e77ab832297bdebd15f9637a1682be5d474aa4da.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Episodic Novelty Through Temporal Distance
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission