PASRL: Stabilising Reinforcement Learning with Past Action-State Representation Learning
Reinforcement learning;action smoothness;recurrent neural networks
Yes
No ethics review needed.
-
The problem itself is important. When training for real world deployment, the constraints of the real systems need to be taken into account.
The paper tackles the problem of action smoothness, which is especially crucial for robotic systems that are incapable of instant changes of the motor torques.
They propose augmenting the state with the prior actions, and perform an ablation to showcase the performance differences.
I'm not sure what the authors try to convey in terms of success/failure.
Their method is better at predicting smoother actions, this is not surprising. But also not surprising is the degradation in performance. This is a result of over-regularization.

Real systems don't necessarily need the "smoothest" policy possible, but rather have some constraints as to how reactive they can be.
I think this work should be extended to the constrained RL setting. Providing the required smoothness parameter as a part of the optimization process.

Then, the authors can compare with prior methods with differing hyper-parameters.
In the constrained RL setting, you want to show that your work is (1) easier to tune or requires no tuning, whereas other works need delicate reward design. (2) for a given constraint, out of all methods that meet the constraints, your controller reaches the best performance on the task itself.
Yes
Simulation based paper, no need to have ethic review
No ethics review needed.
1) Auther memtioned "The reliance on ONLY the current time step information ... causes instability and oscillations", for off-policy method such as TD3 and TD7, the data comes from reply buffer can be past time step information. Can you make the point more clear?
2) For Figure 7, how can the plot has shaded area for single seed line? when you do the experiment, I assume you change the environment seed. Do you fix the random seed, torch seed and/or numpy seed?
3) For Figure 5, the red, blue, and brown line, espicially red line has lowest reward but result in the best action smoothness. Can the auther explan why the better action smoothness will hurt the reward performance?
1) PASRL introduce an approach to enhance action smoothness by incorporating past state-action representations through a RNN.
2) PASRL use pink noise  rather than standard white noise.
This paper proposed a mathod, PASRL, as an extension toTD7 which address instability and oscillations in the smoothness of concurrent actions.
1) the main weakness is the optimality issue. Action plays a significant role in RL (exploration). Ideally, in order to reach optimal Q value ($Q^*(s,a)$) and optimal policy ($\pi^*$), you need to visite all state action pair (s,a) since  ($Q^{\pi^*}(s,a) \geq Q^\pi(s,a)$). During training, when you smooth the action, you will take the risk of limited exploration which affect the optimality, especially in high-dimensional environments such as humanoid.
2) Limitation: smooth action is not always best. Consider a well-known control example, racing car, the objective is only minimizing time. In this case bang-bang control is the optimal solution however it is not smooth.
3) Simulation examples are very limited only ant and hopper.
Yes
No ethics review needed.
1. As shown in Fig.5,  the action smoothness results of each method during training are given. Are these results evaluated in training mode (the agent takes actions with exploration noises) or in evaluation mode (without exploration noises)? 
   Please clarify this issue in the paper, which has a significant influence on the experiment result.
1. Improving action smoothness is a meaningful research topic in DRL domains, especially in continuous control tasks such as robotics control and auto driving. 
2. The key idea of the new method PASRL is easy to follow and implement.
This paper proposes a new DRL method called PASRL based on TD7 method, extended with past action-state representation learning. PASRL can achieve smoother action sequences compared to existing DRL methods through end-to-end training without heuristics design. The experiment on the Ant control task demonstrates the effectiveness of the new method.
1. More related works need to be introduced in this paper. Improving action smoothness is the key contribution of this paper, which has been widely researched recently in DRL domain. The following are some related works:

   1. LipsNet[1]: reduce action fluctuation of DRL methods in continuous control tasks through constraining the Lipschitz constant of policy networks utilizing Multi-dimensional Gradient Normalization (MGN).
   2. TAAC[2]:  introduce a switch policy to make binary choices on whether to use the current new action or the last action at each step in continuous control tasks.
   3. PIC[3]: introduce action inertia into discrete DRL methods, where the policy tends to take the same actions at the previous step.

   Please find more related works and give a more detailed description of this research filed in the manuscript.

2. The experiment is insufficient and cannot demonstrate the effectiveness of the new method:

   1. The experiment result of PASRL in the Ant environment is not satisfactory, with unignorable reward gap compared to TD7  and other baselines (Fig. 5). Although the action smoothness of PASRL is better than other methods, the sacrifice on reward performance is unacceptable in application scenarios.
   2. The experiment is only conducted in the Ant task. It is necessary to evaluate the performance of each method in more environments, such as Hopper, HalfCheetah, Humanoid, etc.
   3. More baseline methods are necessary to demonstrate the performance of PASRL. As described above, some DRL methods have been proposed to improve action smoothness, such as LipsNet, TAAC, and some regularization methods introduced in the paper (SR^2L[4], L2C2[5], etc.). Please conduct experiments with additional baseline methods to demonstrate the superior performance of PASRL compared to these existing methods.

3. The expression of this paper needs to be revised for readiability,:

   1. Add punctuations, especially commas in some sentences. This is significant to remove ambiguity and  improve readiability.
   2. Some notations and terms are used without clear definitions. For example, in line 214 section 4.2, the readers cannot find the definition of $nh_t$. Some terms including LPF and ERE are used without claiming their full names. Please revise the manuscript accordingly.
   3. Some latex typos, such as ``representational drift'' in Line 105.

[1] LipsNet: A Smooth and Robust Neural Network with Adaptive Lipschitz Constant for High Accuracy Optimal Control

[2] TAAC: Temporally Abstract Actor-Critic for Continuous Control

[3] Addressing Action Oscillations through Learning Policy Inertia

[4] Deep Reinforcement Learning with Robust and Smooth Policy

[5] L2C2: Locally Lipschitz Continuous Constraint towards Stable and Smooth Reinforcement Learning
Improving action smoothness in reinforcement learning controllers through past action-state representation learning, without relying on traditional control algorithms or heuristics.
@inproceedings{
anonymous2024pasrl,
title={{PASRL}: Stabilising Reinforcement Learning with Past Action-State Representation Learning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=IcHHjgdb0o},
note={under review}
}
Although deep reinforcement learning (DRL) deals with sequential decision making problems, temporal information representation is absent from state of the art actor-critic algorithms. The reliance on only the current time step information and densely connected neural networks causes instability and oscillations in the smoothness of concurrent actions. Therefore many applied DRL robotics control methods employ various reward shaping, low-pass filter and traditional controller based methods to mitigate this effect. However the interactions of these different parts hinders the performance of the original goal for the RL algorithm. In this paper we present a reinforcement learning algorithm extended with past action-state representation learning, which allows for the end-to-end training of RL based control methods without the need for common heuristics. Our end-to-end training approach produces the smoothest actions while achieving performance scores comparable to the top mixed traditional control and reinforcement learning algorithms.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Reinforcement learning action smoothness recurrent neural networks
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/1f42c0089446cf32e895813741b3775631144855.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/bb15cb54e13083ce39532f4280c9b06074ef118b.zip
PASRL: Stabilising Reinforcement Learning with Past Action-State Representation Learning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission