Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning
Actor-Critic;Exploration;Reinforcement Learning;Thompson Sampling;Langevin Monte Carlo;Deep Reinforcement learning;Continuous Control
Yes
No ethics review needed.
1. Is $Q_{\psi}(s,a)$ deterministic or stochastic? How is $Q_{\psi}(s,a)$ sampled in Equation (11)? Given that $Z_{\psi}(\cdot| s, a) = N(Q_{\psi}(s, a), \sigma_{\psi}(s, a))$ is defined as the distribution of the $Q(s,a)$ value, I would expect the $Q_{\psi} + \text{noise} \cdot \sqrt{\sigma}$ term to represent the Q-value.
2. I’m unclear about the role of LMC in this paper. In lines 231 and 232, the authors claim that $Q_{\psi}$ has a posterior distribution in exponential form, suggesting that LMC is used to sample $Q_{\psi}$. However, in Algorithm 1, lines 12 and 13, $\psi$ appears to be uniformly sampled from a set, and the update rule, instead of a sampling scheme has a form similar to LMC.
3. Since the Q-value distribution is defined as a Gaussian with mean $Q_{\psi}$, is there any connection between the distributional Bellman operator loss in Eq. (6) and the standard TD_c loss? I noticed that Eq. (15) presents an equivalent form of Eq. (6), where the new TD loss is expressed as $L(\psi) = \frac{TD_c(\psi)}{\sigma_{\psi}} + \log(\sigma_{\psi})$. Does this imply a trade-off between the standard TD loss and variance?
1. They propose a new method that models each Q(s, a) value as a distribution. Then, an LMC-based sampling algorithm is used to sample the posterior of the Q value, which encourages exploration. The accordingly distributional Bellman operator is used.
2. They propose using a diffusion model to generate more state-action pairs to update the critic function, where each pair's action is replaced with its Q action gradient updated ones. The use of generative models helps increase the sample efficiency.
3. Thorough experiments are conducted, with the algorithm implemented on two benchmarks, demonstrating improved or comparable empirical performance.
This paper aims to address inefficient exploration and bad sample usage in actor-critic algorithms for continuous control problems. They propose Langevin Soft Actor-Critic (LSAC) which uses adaptive Langevin Monte Carlo (aSGLD) to sample from a Q-value distribution, improving exploration by capturing uncertainty. A diffusion model is also applied to generate diverse synthetic state-action pairs, refined by Q-action gradients to focus on high-reward areas, thereby enhancing sample efficiency. The algorithm demonstrates improved or comparable empirical performance on MuJoCo and DeepMind Control Suite benchmarks compared to standard RL algorithms. An ablation study is also provided.
1. The presentation lacks clarity. Many techniques or tricks are applied without sufficient interpretation or motivation. For example, in Equation (8) and Equation (9), what is the role of the adaptive preconditioner $\xi_k$, and why is it modeled as in Equation (9)? Similarly, what is the rationale behind the entropy weight $\alpha$ update rule in Equation (12)?
2. The overall pipeline is complex, involving a diffusion model for generating additional state-action pairs and a parallel Langevin Monte Carlo for sampling from the Q(s, a) distribution. Could you provide a comparison of the training time against other baseline algorithms?
3. The insights and analysis presented are limited and could be further developed.
Yes
No ethics review needed.
As all the techniques introduced and incorporated into LSAC and complexity of the implementation, any analysis why LSAC is not working better than DSAC in the humanoid environment (which is also regarded as the one of the most complex environments among the benchmarks) ? 
What is the impact of each key component (LMC updates, parallel tempering, diffusion models) on the performance of LSAC? Have you tried to remove one of these components and see the affect the overall effectiveness?
Novel ideas of combining LMC updates, parallel tempering, and diffusion-synthesized action refinement in the LSAC algorithm.
By efficiently exploring state-action spaces through LMC based Q updates and multimodal Q posteriors with parallel tempering, LSAC has the potential to learn a better policy in complex environments.
LSAC demonstrates competitive or superior performance against several established baselines on standard continuous control benchmarks.
The methodological choices in LSAC are grounded in theoretical analysis presented in the paper.
Authors of this paper introduces a model-free online RL algorithm named Langevin Soft Actor-Critic (LSAC). The algorithm incorporates  Langevin Monte Carlo (LMC) updates for critic learning with parallel tempering and action refinement on diffusion-synthesized trajectories. it also utilizes distributional Q objective, allowing diverse sampling from multimodal Q posteriors. Through experiments on MuJoCo benchmarks, the algorithm is shown to outperform/match the performance of SOTA baselines.
The implementation seems very complex even by reading through the pseudo code. 
The computational cost of LSAC lacks discussion in the paper.
Additional parameters and hyper-parameters are introduced, including 10 parallel critics, additional diffusion buffer, inverse temperature, etc, and ablation study shows that LSAC is sensitive to some of the parameters, which may limit its scalability to different applications.
Yes
No ethics review needed.
- It seems to me a main part of the strong performance comes from a better exploration technique. In the baselines you compared to, are some of them focused on exploration techniques? If we replace the exploration in proposed method with some other recent exploration technique, will we obtain a similar strong performance?
**originality**

- authors study and show how to do Langevin Monte Carlo (LMC) based Thompson sampling in RL continuous control setting properly and achieve strong performance, this is a novel contribution. 

**quality**

- overall good quality of presentation and writing. 
- fairness of comparison: I appreciate that the authors discuss comparison fairness and provide full details of the hyperparameter setting in the appendix and give source code. 

**clarity**
- overall clear 

**significance**
- strong results: Figure 1, table 1 show that proposed method has fairly consistent stronger results compared to other baselines. 
- results show how to properly use Langevin Monte Carlo (LMC) based Thompson sampling in DMC and MuJoCo. 
- ablations: it is good to have ablations to understand the effect of different design choices.
This paper introduces a new model-free RL algorithm Langevin Soft Actor Critic (LSAC). It uses Langevin Monte Carlo (LMC) for updating the Q function to help exploration and improve performance, the authors pointed out several key challenges in taking this approach and propose solutions that show benefit to performance in the ablations. When compared to a number of other baseline RL algorithms on MuJoCo and DMC benchmarks, LSAC shows stronger overall performance. This is the the first successful application of a Langevin Monte Carlo (LMC) based Thompson sampling in RL continuous control setting.
Some of the arguments made in the paper seems not adequately supported, some more explanations or experiments will help:

- Line 346: the authors argue LSAC has good performance while being simple in implementation. Why is it simple in implementation? It seems to me it has a number of additional components/hyperparameters compared to a baseline such as SAC, and these extra components seem not trivial to implement correctly. The complexity of the method seems a weakness to me. 
- Line 460: The authors argue that Figure 5 shows LSAC as more stable. I don't feel convinced. The fluctuation in its performance seems similar to DSAC-T, and even when compared to DIPO, LSAC has some wild changes in Figure 5 (c). 
- Is there a comparison on computation efficiency? With the additionl components in the proposed method, how much computation/wall clock time does it take to finish a run compared to other algorithms? 
- Although a lot of details and discussion are given to show an effort to compare to alternative algorithms in a fair manner, it is a bit unclear to me how 2 things are made fair: (1) network capacity, the proposed approach seems to have more critics and additional networks (such as in the diffusion part), how does it compare to other baselines in terms of e.g. number of parameters? And (2) UTD ratio, if the proposed approach is doing more updates, is it still fair to compare it to baselines that take limited updates? I understand that some of them might not work well with more updates due to instability, but in that case, one might argue it would be good to compare to algorithms that do benefit from ensemble of networks and higher UTD such as REDQ?
Yes
N/A
No ethics review needed.
- In line 9 of Algorithm1, you directly replaced $a_M$ with $a_{M'}$, which destroyed the Markov chain. What problems exist in fusing such data with $B_D$? Please analyze it theoretically.

- Algorithm1 Why doesn't the strategy also choose to update on $B_i$?

- Line341 Is the frequency of updating the diffusion model too low? Does the data generated in this way really have many benefits for sample distribution? Please analyze the sample distribution, as well as the performance and sample distribution results corresponding to different update frequencies.

- Why do other algorithms in Table1 have no variance? It seems that you didn't say how you calculated the data in the table.

- Do you also sample 20 samples like DSAC-T for each interaction with the environment?

- In the high-dimensional action space task Humanoid-v3, why is your performance much lower than DSAC-T?
- This paper uses LMC combined with distributed Q to learn critic, and also uses the action gradient method to improve the diversity of samples in the buffer. It is an interesting work;
- The experiments in this paper are very comprehensive, and it has achieved very good performance in many environments, and the curve rises very quickly;
- Compared with diffusion as a policy, this paper does not have the risk of too long sampling time in actual use.
This paper presents a novel model-free RL algorithm, Langevin Soft Actor Critic (LSAC), which enhances critic learning through uncertainty estimation rather than just optimizing the policy. LSAC features three innovations: approximate Thompson sampling via distributional Langevin Q updates, parallel tempering for exploring multiple posterior modes, and diffusion-synthesized state-action samples regularized with Q action gradients. Experiments show that LSAC outperforms or matches mainstream model-free RL algorithms in continuous control tasks and is the first successful application of Langevin Monte Carlo-based Thompson sampling in continuous action spaces.
- In the related work, I strongly suggest you add the work on combining diffusion with online reinforcement learning. Although they focus on the policy, they are also directly related to your paper. Eg. Policy Representation via Diffusion Probability Model for Reinforcement Learning (Yang et al.), Learning a Diffusion Model Policy from Rewards via Q-Score Matching (Psenka et al.), Diffusion Actor-Critic with Entropy Regulator (Wang et al.), etc. Please write a paragraph in the paper to summarize.

- I don't agree with the paragraph around line 68. Methods like SAC and DSAC-T are soft policies. They balance exploration and utilization by introducing policy entropy. How can this be classified as a trick? I think their core is also heuristic, which is essentially the same category as your method.

- In Algorithm 1, there are 4 for loops for each policy update, and the time-consuming action gradient is used, but the time efficiency is not analyzed, and the performance difference brought by different action gradient steps is not analyzed. Please add time efficiency analysis: for example, when training for 8 hours at a time, compare the performance of the algorithms (draw a curve with time as the horizontal axis and TAR as the vertical axis).

- You don’t seem to have experimentally proved some of the problems caused by high-dimensional action space. Compare your current method with the previous method to see why this problem disappears (analyze the experimental results).

- Based on the citations of DSAC-T, I found a work Diffusion Actor-Critic with Entropy Regulator (Wang et al.). Please compare their performance numerically.

- I think the core work in Q-update is to combine Provable and practical: Efficient exploration in reinforcement learning via Langevin Monte Carlo with DSAC-T: Distributional Soft Actor-Critic with Three Refinements. This way of updating Q can indeed alleviate overestimation while bringing better performance, but the novelty may be less.

I'll consider score changes based on your response.
We propose a principled exploration approach for actor-critic algorithms through critic learning using approximate Thompson sampling
@inproceedings{
anonymous2024langevin,
title={Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=FvQsk3la17},
note={under review}
}
Existing actor-critic algorithms, which are popular for continuous control reinforcement learning (RL) tasks, suffer from poor sample efficiency due to lack of principled exploration mechanism within them. Motivated by the success of Thompson sampling for efficient exploration in RL, we propose a novel model-free RL algorithm, \emph{Langevin Soft Actor Critic} (LSAC), which prioritizes enhancing critic learning through uncertainty estimation over policy optimization. LSAC employs three key innovations: approximate Thompson sampling through distributional Langevin $Q$ updates, parallel tempering for exploring multiple modes of the posterior of the $Q$ function and diffusion synthesized state-action samples regularized with $Q$ action gradients. Our extensive experiments demonstrate that LSAC outperforms or matches the performance of mainstream model-free RL algorithms for continuous control tasks.
Notably, LSAC marks the first successful application of a Langevin Monte Carlo (LMC) based Thompson sampling in continuous control tasks with continuous action spaces, setting a new benchmark for future research in the field.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Actor-Critic Exploration Reinforcement Learning Thompson Sampling Langevin Monte Carlo Deep Reinforcement learning Continuous Control
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/4897fb8ccfd45d145965276673bca0bc0c3175de.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission