AFU: Actor-Free critic Updates in off-policy RL for continuous control
Nicolas Perrin-Gilbert


Interpretable Machine Learning Models for Predicting the Next Targets of Activist Funds
    [
    April 29, 2024
=======================================================================================




This paper presents AFU, an off-policy deep RL algorithm addressing in a new way the challenging “max-Q problem” in Q-learning for continuous action spaces, with a solution based on regression and conditional gradient scaling. AFU has an actor but its critic updates are entirely independent from it. As a consequence, the actor can be chosen freely. In the initial version, AFU-alpha, we employ the same stochastic actor as in Soft Actor-Critic (SAC), but we then study a simple failure mode of SAC and show how AFU can be modified to make actor updates less likely to become trapped in local optima, resulting in a second version of the algorithm, AFU-beta. Experimental results demonstrate the sample efficiency of both versions of AFU, marking it as the first model-free off-policy algorithm competitive with state-of-the-art actor-critic methods while departing from the actor-critic perspective.







§ INTRODUCTION



Q-learning <cit.> stands as a fundamental algorithm in the realm of model-free reinforcement learning. As mentioned in <cit.>, “it provides agents with the capability of learning to act optimally in Markovian domains by experiencing the consequences of actions, without requiring them to build maps of the domains”. It is centered around the Bellman optimality equation and leverages dynamic programming to compute or approximate a function known as the optimal  
Q-function Q^*. The integration of deep neural networks to approximate Q-functions and the efficient computation of gradient-based updates has led to the successful development of the Deep Q-Network (DQN) algorithm <cit.>, catalyzing important advancements in reinforcement learning. However, Q-learning requires computing the maximum of the Q-function over the action space, which can be difficult if it is continuous and multi-dimensional. To circumvent this “max-Q problem”, Q-learning can be combined with an actor-critic perspective, which involves training the actor to generate actions maximizing the Q-function, thereby solving the max-Q problem indirectly. This approach gave rise to DDPG <cit.>, a seminal actor-critic algorithm benefiting from the off-policy nature of Q-learning and consequently from a high sample-efficiency. Yet, in DDPG and its derivatives like TD3 <cit.>, the actor may become trapped in local optima <cit.>. Other approaches have attempted to face the max-Q problem head-on, like CAQL <cit.> or Implicit Q-Learning (IQL, <cit.>), but the former does not scale well to high-dimensional action spaces, and requires adaptations such as constraining the action range for complex problems, whereas in the latter, the expectile loss becomes unbalanced when trying to produce estimates that are very close to the true maxima, which has so far restricted the application of IQL and similar methods to offline reinforcement learning.

In this paper, we propose a novel way to solve the max-Q problem, using regression and conditional gradient scaling (see Section <ref>), resulting in a new algorithm that adapts Q-learning to continuous action spaces. The algorithm still has an actor to select actions and produce episodes, but unlike state-of-the-art model-free off-policy algorithms, most of which are derived from TD3 or SAC <cit.>, its critic updates are entirely independent from the actor. We call the algorithm AFU for “Actor-Free Updates”. In its first version, AFU-alpha (see Sections <ref> and <ref>), we use a stochastic actor and train it like the actor in SAC. We then study in Section <ref> a simple failure mode of SAC (and AFU-alpha), and show that the value function trained by regression in AFU can help improve the actor update and make it less prone to local optima, resulting in a new version of the algorithm, AFU-beta (see Section <ref>), which does not fail in the same way. 

Our experiments show that AFU-alpha and AFU-beta are competitive in sample-efficiency with TD3 and SAC without being more computationally expensive. To the best of our knowledge, AFU is the first model-free off-policy RL algorithm that is competitive with the state-of-the-art and truly departs from the actor-critic perspective. Furthermore, by reducing the risk of being ensnared in local optima, we believe that AFU-beta   represents an advancement in establishing solid foundations for a purely off-policy RL algorithm suited to continuous control problems.






§ RELATED WORK

In domains where high sample efficiency is crucial, such as robotics, the off-policy nature of reinforcement learning algorithms becomes paramount. This allows training on samples obtained from different policies or older versions of the current policy, facilitating faster learning and compatibility with various exploration strategies.
One way to obtain an off-policy algorithm is to adapt Q-learning, but as previously mentioned, in continuous action spaces, direct approaches attempting to solve the max-Q problem of Q-learning have faced limitations. Besides CAQL, which formulates the max-Q problem as a mixed-integer program, and IQL, which treats Q-functions as state-dependent random variables and relies on expectile regression to estimate their maxima, we can cite NAF <cit.> and ICNN <cit.>, which impose action-convex Q-functions making the max-Q problem tractable, QT-Opt <cit.>, which uses a stochastic optimizer to tackle non-convex max-Q problems, or approaches based on a discretization of the action space, such as SMC-learning <cit.> and SDQN <cit.>. However, these methods often struggle with complex, high-dimensional continuous control tasks, either due to a lack of expressiveness or to prohibitive computational costs.
A unique off-policy algorithm, AWR <cit.>, employs regression to train a value function and a policy but falls short of the sample efficiency achieved by state-of-the-art off-policy algorithms. Presently, the most successful approaches in model-free off-policy RL for continuous control are actor-critic algorithms with interwoven actor and critic updates. 
The first off-policy actor-critic algorithm was introduced in <cit.>, and the most recent ones are typically based on TD3, an improvement of DDPG, or on SAC, which relies on an entropy maximization framework that led to various off-policy algorithms by creating connections between policy gradients and Q-learning updates (see <cit.>).
 Among the algorithms improving upon TD3 and SAC, we can mention TQC <cit.>, a distributional approach to control the overestimation bias, REDQ <cit.> or AQE <cit.> which employ critic ensembles, DroQ <cit.> which uses dropout and layer normalization in the critic networks, and BAC <cit.> which merges Q-function updates from SAC and IQL. While these ideas could be incorporated into our proposed algorithm AFU[In the case of BAC, our proposed method to solve the max-Q problem could replace the IQL updates.], we leave this for future work and focus on comparing AFU to SAC and TD3. In contrast to methods building upon SAC and TD3, AFU is structurally distinct because the critic updates remain unaffected by the actor. Notably, the critic is never trained with out-of-distribution actions, yet AFU achieves a level of sample efficiency competitive with SAC and TD3. Furthermore, unlike other direct adaptations of Q-learning to continuous control, AFU does not fail on the most complex tasks. On the contrary, the most challenging MuJoCo task Humanoid is one of the environments in which AFU performs the best comparatively to SAC and TD3.







§ PRELIMINARIES

We consider a discounted infinite horizon Markov Decision Problem (MDP) <S, A, T, R, γ>, where S is a state space, A a continuous action space, T a stochastic transition function, R : S × A →ℝ a reward function, and 0 ≤γ < 1 a discount factor. 
We denote by s' (resp. s_t+1) a state obtained after performing an action a (resp. a_t) in state s (resp. s_t). Transitions are tuples (s, a, r, s') with r = R(s,a). The optimal Q-function Q^* is defined by:
Q^*(s, a) = 𝔼[∑_t=0^∞γ^t R(s_t, a_t) | s_0 = s, a_0 = a, π^* ],
where the policy used from t=1 onwards is π^*, which selects actions optimally in every state. 
The optimal value function V^* verifies
V^*(s) = max_a ∈ A (Q^*(s, a)).
Let V_φ_1 and V_φ_2 denote two function approximators for the value function, and Q_ψ a function approximator for the Q-function (the critic). 
We use feed forward neural networks for all the function approximators. For the value function, we also consider target networks (see <cit.>), more specifically parameter vectors φ^target_1 and φ^target_2 updated with the rule  for some target smoothing coefficient 0 < τ < 1.
We wish to train the critic on mini-batches B of transitions (s, a, r, s') taken from an experience replay buffer, with the following loss derived from the clipped Double Q-learning loss of TD3 <cit.>:

    L_Q(ψ) = (s, a, r, s') ∈ BMean[ ( Q_ψ(s, a) - r - γmin_i ∈{1, 2}V_φ^target_i(s') )^2 ]


The use of two function approximators V_φ_1 and V_φ_2 aims at avoiding the overestimation bias that can make Q-learning based approaches diverge (see <cit.>). In practice, transitions can be terminal, which requires a simple modification of the loss ignored here for the sake of clarity (γ is set to 0 for terminal transitions).
Provided that V_φ_1(s) and V_φ_2(s) return good estimates of the maximum of Q_ψ(s, ·), a maximization sometimes referred to as the max-Q problem (see <cit.>), Equation (<ref>) amounts to the mean squared Bellman error that should drive Q_ψ toward the fixpoint of the Bellman optimality operator, i.e. Q^* <cit.>.






§ SOLVING THE MAX-Q PROBLEM WITHOUT REGULARIZATION

The main remaining problem is: how to efficiently train V_φ_1 and V_φ_2? For the learning to be successful, V_φ_1 and V_φ_2 should both converge to precise solutions of the max-Q problem, and the convergence should be fast, because if changes in Q_ψ are not tracked promptly, errors such as overestimation of Q-values could lead to failures.



 §.§ Method


We introduce two new function approximators A_ξ_1 and A_ξ_2, for the optimal advantage function defined by
A^*(s,a) = Q^*(s,a) - V^*(s).
For any state-action pair (s, a), A^*(s,a)≤ 0. Preliminarily, we assume that outputs of A_ξ_i can only be non-positive. Assuming that Q_ψ is fixed, training V_φ_i and A_ξ_i can be done by minimizing the following regression loss on mini-batches B: 

    l_V, A(φ_i, ξ_i) = (s, a, _, _) ∈ BMean[ ( V_φ_i(s) + A_ξ_i(s,a) - Q_ψ(s, a) )^2 ].

This loss causes the values V_φ_i(s) to become upper bounds of Q_ψ(s, ·), but not tight ones.
A regularization term can be added to penalizes large outputs of V_φ_i, but the issue is that the resulting convergence is either slow (for very small regularization coefficients) or significantly biased (for larger coefficients). For some problems, finding the right coefficient is possible, but in general regularization does not lead to satisfactory results. We propose a different approach, noticing that during the minimization of the loss l_V, A(φ_i, ξ_i), the values of V_φ_i(s) + A_ξ_i(s,a) typically quickly end up oscillating around Q_ψ(s, a). When V_φ_i(s) + A_ξ_i(s,a) is greater than its target Q_ψ(s, a), by gradient descent both V_φ_i(s) and A_ξ_i(s,a) are decreased, and conversely when V_φ_i(s) + A_ξ_i(s,a) is smaller than the target, values are increased. Without regularization, all the upper bounds of Q_ψ(s, ·) are equally good values for 
V_φ_i(s), but we can put a downward pressure on V_φ_i(s) to make it progressively decrease during the oscillations. To this end, we introduce an asymmetry by applying only a fraction of the gradient descent update on φ_i when V_φ_i(s) is increased, and the whole gradient descent update when V_φ_i(s) is decreased. It can be done by defining, for 0 < ϱ < 1:

    Υ_i^a(s) = (1 - ϱ I_i^s,a) V_φ_i(s) + ϱ I_i^s,a V_φ^no_grad_i(s),

where φ^no_grad_i is a copy of the parameters φ_i, and I_i^s,a is the following index:

    I_i^s,a ={[ 1,  .; 0,  . ].

Replacing V_φ_i(s) by Υ_i^a(s) in (<ref>) yields a new version of the loss:

    Λ_V, A(φ_i, ξ_i) = (s, a, _, _) ∈ BMean[ (Υ_i^a(s) + A_ξ_i(s,a) - Q_ψ(s, a) )^2 ].


We effectively constrain the sign of A_ξ_i(s,a) in a soft way, as using for example an absolute value in output would lead to multiple solutions and potentially jittering gradients. So A_ξ_i can return positive outputs, but we modify the regression loss to have only non-positive targets for A_ξ_i(s,a). In Equation (<ref>), we call Q_ψ(s, a) - Υ_i^a(s) the target of A_ξ_i(s,a), as it is the value of A_ξ_i(s,a) minimizing (Υ_i^a(s) + A_ξ_i(s,a) - Q_ψ(s, a) )^2. If , the best non-positive target for A_ξ_i(s,a) is 0, in which case the target for  should also be 0. In this situation, we replace  by . To do so, we introduce Z:

    Z(x,y) ={[ (x+y)^2,        .; x^2+y^2,        . ].

The loss is updated as follows:

    Λ'_V, A(φ_i, ξ_i) = (s, a, _, _) ∈ BMean[  Z(Υ_i^a(s) - Q_ψ(s, a), A_ξ_i(s,a) ) ].




 §.§ Experiments




A mathematical proof of the convergence of V_φ_i(s) toward 
max_a ∈ A (Q_ψ(s, a)) is outside the scope of this paper, but we empirically compare our method to 3 baselines on a toy problem.
We define the function Q_toy(s,a) = sin(4s) + 0.7cos(4a)
for s ∈ [-1, 1] and a ∈ [-1, 1].
We use a single feedforward neural network for V: V_φ, and a single feedforward neural network for A: A_ξ. Both networks have two hidden layers of size 256 and ReLU activations in the hidden layers. Our method trains both V_φ and A_ξ, while the 3 baselines IQL, SQL and EQL directly train V_φ. All 3 baselines have been successfully applied to offline reinforcement learning.
IQL is Implicit Q-Learning <cit.>, and SQL and EQL are respectively Sparse Q-Learning and Exponential Q-Learning, both introduced in <cit.>. For a fixed s, IQL treats Q_toy(s,a) as a random variable (the randomness being determined by the action) and uses an expectile regression loss to train V_φ(s) to estimate a state conditional upper expectile of this random variable. The expectile is determined by the parameter 0 < τ < 1, and the closer τ is to 1, the closer V_φ(s) gets to max_a ∈ A (Q_toy(s, a)). 
Unfortunately, if τ is very close to 1 (e.g. τ = 0.99), the loss becomes unbalanced, with elements weighted hundreds of times more than others, so in practice the values used in <cit.> are 0.7, 0.8 and 0.9. SQL and EQL are derived in <cit.> from a general method called Implicit Value Regularization. It relies on a behavior-regularized MDP with a term that penalizes policies diverging from the underlying behavior policy of the training dataset. Various f-divergences can be used to measure the difference between the policy and the behavior policy, resulting in distinct algorithms, including SQL and EQL which are special cases. They have distinct losses for the training of V_φ(s), both depending on a parameter α, and in both cases, for α→ 0, V_φ(s) is trained to approximate the maximum operator over in-support values, i.e. max_a ∈ A (Q_toy(s, a)).
However, similarly to IQL, very small values of α result in unbalanced losses, so in practice the values leading to the best results on the benchmarks tested in <cit.> are α=0.1, α=0.5, α=1 and α=3 for SQL, and α=0.5, α=2.0 and α=5 for EQL.

Figure <ref> shows results for different hyperparameters after performing 3000 gradient descent steps with Adam <cit.> on uniformly drawn mini-batches of size 256. For IQL, SQL and EQL, since unbalanced losses are not an issue on this simple toy problem, we include parameters leading to a better resolution of the max-Q problem, but that are not representative of the parameters working well in actual offline reinforcement learning experiments. We observe that, with our method, although ϱ=0.05 leads to overestimations, for a wide range of parameter values (from ϱ=0.2 to ϱ=0.7), we obtain more accurate results than with all the other baselines, even when considering parameter values that are inapplicable to offline reinforcement learning. Besides, with our proposed approach, the different values of ϱ that perform well do not result in unbalanced losses.







§ TRAINING THE ACTOR

First, we remove the dependency to Q_ψ in loss (<ref>) by replacing Q_ψ(s, a) by the targets used to train it in (<ref>). We obtain the following loss (for i ∈{1, 2}):

    L_V, A(φ_i, ξ_i) = (s, a, r, s') ∈ BMean[  Z(Υ_i^a(s) -  r - γmin_i ∈{1, 2}V_φ^target_i(s'), A_ξ_i(s,a) ) ].

With the losses L_Q(ψ) (<ref>) and L_V, A(φ_i, ξ_i) (<ref>), we can train Q_ψ, V_φ_i and A_ξ_i without needing an actor.
Compared to methods derived from DDPG (like TD3), solving directly the max-Q problem has an advantage over first using an actor to solve the argmax-Q problem, i.e. to approximate argmax_a ∈ A (Q_ψ(s, a)). The reason is that continuous changes in Q_ψ result in continuous changes of its state conditioned maxima, while it can result in discontinuous changes of its state conditioned argmax. So, in an off-policy setting, if the exploration policy discovers better results with very different actions, the maximum of Q_ψ(s,a) can be tracked smoothly, while the tracking of the argmax can be much more difficult, with the potential arising of deceptive value landscapes in which the actor can get stuck (see <cit.>). This theoretical advantage, as well as the
actor-free Q_ψ, V_φ_i and A_ξ_i updates are all important aspects of our approach. However, since we are interested in online reinforcement learning, we still need an actor to select actions and produce episodes. To train this actor, if we would use the same gradient ascent over a ↦ Q_ψ(s,a) as in DDPG, our global method would be prone to the same failure modes as DDPG, and most of the advantages of the max-Q based training of V_φ_i would be lost.
One thing we can notice is that, since we do not need the actor to return 
argmax_a ∈ A (Q_ψ(s, a)), we also do not need the actor to be deterministic.
To benefit from a better exploration, we opt for a stochastic actor and follow the approach proposed in SAC <cit.>. 

Let π_θ denote the actor. We follow a common implementation in which its backbone is a feedforward neural network returning action distributions as state-dependent Gaussians with diagonal covariance matrices. Since actions are usually constrained between -1 and 1, we apply a tanh transformation to its outputs. Given a state s, the resulting probability density function is π_θ(· | s). The actor π_θ can transform input noise vectors sampled from a fixed distribution into action samples. Again, we train π_θ on mini-batches of transitions. We use the actor to resample an action a_s for each state s of a mini-batch B. The actor loss L_π(θ) is based on the average Kullback-Leibler divergence between the actor's output distributions and targeted Boltzmann policy distributions. It is defined as follows:

    L_π(θ) = (s, _, _, _) ∈ B 
     a_s ∼π_θ(· | s)Mean[ αlog(π_θ(a_s | s)) - Q_ψ(s, a_s) ],

where α is a temperature parameter. As in SAC, we adjust this temperature via gradient descent on a loss aiming at keeping the average entropy of action distributions close to a target entropy ℋ̅:

    L_temp(α) = (s, _, _, _) ∈ B 
     a_s ∼π_θ(· | s)Mean[ -αlog(π_θ(a_s | s)) - αℋ̅].







§ AFU-ALPHA

We combine the losses L_Q (<ref>), L_V,A (<ref>), L_π (<ref>), and L_temp (<ref>) to devise a new off-policy reinforcement learning algorithm. 
It has a critic (Q_ψ) and an actor (π_θ), but the critic updates are derived from our novel adaptation of Q-learning to continuous action spaces (obtained with our new method to solve the max-Q problem), therefore they are independent from the actor. Hence the name AFU for the algorithm, for “Actor-Free Updates”. 
We specifically call it AFU-alpha to contrast it with AFU-beta introduced in Section <ref>. AFU-alpha, described in Algorithm <ref>, alternates between environments steps that gather experience in a replay buffer and gradient steps that draw batches from the replay buffer to compute loss gradients and update all parameters of the function approximators. In our implementation, an iteration consists of a single environment step followed by a single gradient step.



 §.§ Algorithm


0.49


0.49





 §.§ Experiments


We test AFU-alpha on a classical benchmark of 7 MuJoCo <cit.> tasks from the Gymnasium library <cit.>. We compare it to SAC and TD3, and to variants of AFU-alpha in which the loss L_V,A aiming at solving the max-Q problem is replaced by the corresponding loss taken from IQL, SQL or EQL. The results are shown in Figure <ref>.
For both SAC and AFU-alpha, we use the same heuristic for the definition of ℋ̅: we set it to -d, where d is the dimension of the action space.
We use the same value of τ and the same learning rate for the actor and critic updates in AFU-alpha, SAC and TD3, and the same learning rate for the temperature updates in AFU-alpha and SAC.
For each algorithm, for each value of the hyperparameter (ϱ for AFU-alpha, τ for IQL, α for SQL and EQL), and for each of the 7 MuJoCo tasks, we perform 10 runs initialized with different random seeds, and evaluate the performance of the policy every 10,000 steps on 10 rollouts. The first 10,000 steps of each run are performed with actions that are uniformly randomly drawn,  and without gradient steps. For each run, the learning curve is smoothed with a moving average window of size 10. The raw score of the run is the last average return, i.e. the average return over the last 10 evaluations.
For each task, we linearly rescale the scores based on two reference points: (1) the maximum evaluation seen across all algorithms and all runs corresponds to a score of 100, and (2) the mean episode return across all algorithms and runs corresponds to a score of 0.
Following the recommendations of <cit.>, we compute with the rliable library the performance profiles for each algorithm across the 7 tasks: Ant-v4, HalfCheetah-v4, Hopper-v4, Humanoid-v4, InvertedDoublePendulum-v4, Reacher-v4 and Walker2d-v4. The length of the runs is 1 million steps for InvertedDoublePendulum and Reacher, 3 million steps for Ant, Hopper, Humanoid and Walker2d, and 5 million steps for HalfCheetah.

In Figure <ref>, we see that our proposed method for the max-Q problem yields significantly better results than the IQL, SQL and EQL baselines. The best results are obtained with ϱ∈{0.2, 0.3}.
Figure <ref> shows that AFU-alpha is competitive with SAC and TD3.









§ A SIMPLE FAILURE MODE OF SAC

With a deterministic actor trained by stochastic gradient ascent over the Q-function landscape, DDPG, TD3 and similarly structured deterministic actor-critic algorithms can easily get stuck in local optima (see <cit.>). 
With a stochastic actor and updates based on the Kullback-Leibler (KL) divergence between output distributions and target distributions of the form exp(1/αQ(s, ·))/z(s), algorithms like SAC are less prone to deadlocks.
For instance, in areas where the gradient of the Q-function is close to zero, exploiting the KL loss results in an increase of the variance of the action distribution, which eventually helps find larger gradients and escape from the flat region.
Yet, the policy networks used in practice mostly output unimodal action distributions[This is starting to change, thanks to the influence of recent methods such as diffusion policies <cit.>, but such expressive and multimodal stochastic policies are still more cumbersome than unimodal policies.], and with this restriction even the KL loss generates undesirable local optima. We illustrate this with a trivial environment which we call SFM (for “SAC Failure Mode”). It consists of a single state s_0, and unidimensional actions in [-1, 1]. The reward of an action is given by the function:

    R_SFM(s_0, a) ={[ 5 - 100 (a - 0.1)^2,                    ,;                   0,                    . ].

All transitions are terminal, so all episodes stop after one step. The optimal policy selects a=0.1 and yields a return of 5. We train SAC on SMF with the same hyperparameters as in our other experiments. We start by performing 1000 steps with random actions, which helps the critic Q_SAC quickly converge toward the optimal Q-function, Q^*, which is simply equal to R_SFM. Figure <ref> shows Q_SAC after 20,000 steps. Although Q_SAC converges toward a very precise approximation of Q^*, the actor policy converges toward a suboptimal solution, as shown in Figure <ref>. If we just modify SAC by locking the mode of the policy distribution at 0, we can see in Figure <ref> that the actor loss becomes much smaller, even after convergence of the actor entropy, which indicates that the policy of the default SAC algorithm gets stuck in a local optimum. There are two phases in the failure mode: at the beginning, when the entropy is relatively large, 
the asymmetry of R_SFM makes the actor shift toward -1. As seen in Figure <ref>, Q_SAC approximates the discontinuity in R_SFM with a steep slope, and when the policy distribution becomes concentrated on the left of this slope, it acts as a barrier that traps the actor. Later in the training, when the entropy becomes smaller and converges to the target entropy (-1), it would be much preferable for the mode of the policy to converge back toward 0.1, but the steep slope results in a deceptive gradient in the KL loss that prevents it from happening, and SAC remains stuck in the local optimum.












§ AFU-BETA

With the same actor loss as SAC, AFU-alpha fails similarly on SFM. We propose to improve the actor loss to make it theoretically much less likely to get stuck in local optima.
The first idea is to train by regression an estimate of where the mode of the actor should be.
If the learning progresses well, Q_ψ(s, a) > min_i ∈{1,2}(V_φ_i(s)) should only be possibly true in the vicinity of the argmax (argmax_a ∈ A(Q(s,a))). Furthermore, mini-batches regularly contain pairs (s,a) verifying this inequality, otherwise our method would make values of V_φ_i(s) decrease until it would become the case. We use actions a with a Q-value greater than min_i ∈{1,2}(V_φ_i(s)) as targets. To find such actions we use both actions in the mini-batches and actions resampled with the actor on those mini-batches. Let us consider a mini-batch B of transitions (s, a, r, s'), and actions a_s resampled with the actor. We denote by ℳ(B) the set of state-action pairs (s, a_∙)
such that a_∙ = a or  and Q_ψ(s, a_∙) > min_i ∈{1,2}(V_φ_i(s)).
We introduce a new deterministic function   with parameters ζ and train it with the following loss:

    L_μ(ζ) = (s, a_∙) ∈ℳ(B)Mean[ (μ_ζ(s) - a_∙)^2 ].

In our implementation, most of the parameters between ζ and θ are shared: we simply modify the output dimension of π_θ to make it also return μ_ζ(s).
It does not change the approach in any way, but when computing the gradient of the loss (<ref>), one must carefully ignore the influence of the parameters ζ on resampled actions a_s.

Let us reconsider the actor loss from Equation (<ref>).
It balances two terms, the first one (αlog(π_θ(a_s | s))) that maximizes the entropy, and the second one (-Q_ψ(s, a_s)) that encourages π_θ to output
actions maximizing Q_ψ(s, ·). In the gradient ∇_θ L_π(θ), which can be expressed by making explicit the relationship between sampled actions a_s and the input noise (see <cit.>), the second term results in small modifications of θ that attempt to change the actions a_s in the direction of ∇_a Q_ψ(s, a), where a is evaluated in a_s, and which we write by abuse of notation ∇_a_s Q_ψ(s, a_s). If ∇_a_s Q_ψ(s, a_s) points away from the global optimum, it can contribute to the creation of a local minimum in the actor loss. 
We want to edit ∇_a_s Q_ψ(s, a_s) in order to avoid deceptive gradients. To do so, we compute the dot product between ∇_a_s Q_ψ(s, a_s) and μ_ζ(s) - a_s, which is an estimate of a direction toward argmax_a ∈ A(Q(s,a)). If the dot product is positive or zero, the gradient does not point away from μ_ζ(s), so we can keep it unchanged. However, if ∇_a_s Q_ψ(s, a_s) · (μ_ζ(s) - a_s) < 0, then we project ∇_a_s Q_ψ(s, a_s) onto (μ_ζ(s) - a_s)^⊥ to anneal the dot product. We do it only if we estimate that a_s is not already in the vicinity of the argmax, i.e. if Q_ψ(s, a_s) < min_i ∈{1,2}(V_φ_i(s)). We introduce the following operator:

    𝒢^s, a_s(v) ={[ proj_(μ_ζ(s) - a_s)^⊥(v),                         .;                        v,                         . ].

When computing the gradient ∇_θ L_π(θ), we replace the terms ∇_a_s Q_ψ(s, a_s) (resulting from the chain rule) by 𝒢^s, a_s(∇_a_s Q_ψ(s, a_s)). It leads to a modified gradient which we denote by
∇^MODIF_θ L_π(θ).



This process is illustrated in Figure <ref>. It can be understood as an artificial modification of the landscape of Q_ψ(s, ·) so that, outside the region defined by Q_ψ(s, ·) ≥min_i ∈{1,2}(V_φ_i(s)), its gradient never points away from μ_ζ(s). 
μ_ζ(s) has the advantage of being trained by regression, and its training includes actions coming directly from the replay buffer, not only ones resampled by the actor. It means that, in a very off-policy setting, if a new peak of Q_ψ(s, a) appears far from the actions currently likely to be sampled, the update of μ_ζ(s) can occur first and then guide the update of π_θ by removing all deceptive gradients that would need to be crossed to reach the new peak.
More generally, the use of μ_ζ prevents the actor from being trapped in local optima, as long as the training of the critic is doing well. Since training the critic is independent from the actor, we believe that our proposed approach goes one step further in the development of sound foundations for a purely off-policy reinforcement learning algorithm performing well in continuous action spaces.

We call AFU-beta the updated algorithm. It works like AFU-alpha, with the additional training of μ_ζ and the replacement of ∇_θ L_π(θ) by ∇^MODIF_θ L_π(θ), as described in Algorithm <ref>.
Figure <ref> shows that, in the SFM environment, unlike SAC, AFU-beta quickly converges to the optimal solution.

We evaluate AFU-beta on the MuJoCo benchmark in the same way as AFU-alpha and show results in Figure <ref>.
Again, AFU-beta is competitive with SAC and TD3. The differences between AFU-beta and AFU-alpha are not very significant on the MuJoCo benchmark, possibly because issues with local optima are rarely encountered in these environments. We leave for future work the search for meaningful and complex environments in which AFU-beta has a notable advantage over AFU-alpha. 









§ CONCLUSION

We presented AFU, an off-policy RL algorithm with critic updates independent from the actor. 
At its core is a novel way to solve the continuous action Q-function maximization (max-Q) problem using regression and conditional gradient scaling, which we believe could have applications outside the RL field. We plan to work on a mathematical proof of the convergence of this new method and analyze whether connections with it can be established in the vast landscape of convex and non-convex optimization, for instance with penalty and multiplier methods <cit.> or Thikonov regularization <cit.>.

The first version of AFU (AFU-alpha) has a stochastic actor trained as in SAC <cit.>. We provide a simple example of failure mode for SAC, and show how the value function trained in AFU can help improve the actor loss and make it less prone to local optima, resulting in a second version of AFU (AFU-beta) which does not exhibit the same failure mode as SAC.

Our experimental results on a classical benchmark show that both versions of AFU are competitive with SAC and TD3, two state-of-the-art off-policy model-free RL algorithms. As far as we know, AFU is the first off-policy RL algorithm that is competitive in sample-efficiency with the state-of-the-art and truly departs from the actor-critic perspective. We believe that it could open up new avenues for off-policy RL algorithms applied to continuous control problems.






§ ACKNOWLEDGEMENTS

This work was supported by the French National Research Agency (ANR), Project ANR-18-CE33-0005 HUSKI,
and has received funding from the European Commission’s Horizon Europe Framework Program under
grant agreement No 101070381 (PILLAR-Robots project). For the numerical experiments, this work 
was granted access to the HPC resources of IDRIS under the 
allocations 2022-A0131013011 and 2023-A0151013011 made by GENCI.









rlc












§ HYPERPARAMETERS


The following hyperparameters were used in all our experiments.

We did not do any reward scaling.





§ CODE



The code used for the experiments is based on a fork of RLJax (<https://github.com/toshikwa/rljax>). 



§ LEARNING CURVES



The plots below show learning curves for AFU-alpha and AFU-beta for all the values of the hyperparameter ϱ (in {0.1, 0.2, 0.3, 0.4, 0.5}).

All learning curves are averaged over 10 runs with different random seeds, and the shaded areas range from the 25th to the 75th percentile. For each run, evaluations are done over 10 rollouts every 10,000 steps, and each run is smoothed with a moving average window of size 10. The first 10,000 steps are always done without gradient steps and with uniformly randomly drawn actions.















