Diagnosing Bottlenecks in Deep Q-learning Algorithms
Justin Fu Aviral Kumar Matthew Soh Sergey Levine


[
Diagnosing Bottlenecks in Deep Q-learning Algorithms














equal*


Justin Fuequal,to
Aviral Kumarequal,to
Matthew Sohto
Sergey Levineto










toBerkeley AI Research, University of California, Berkeley



Justin Fu
Aviral Kumar






0.3in
]









  




Q-learning methods represent a commonly used class of algorithms in reinforcement learning: they are generally efficient and simple, and can be combined readily with function approximators for deep reinforcement learning (RL). However, the behavior of Q-learning methods with function approximation is poorly understood, both theoretically and empirically. In this work, we aim to experimentally investigate potential issues in Q-learning, by means of a "unit testing" framework where we can utilize oracles to disentangle sources of error. 
Specifically, we investigate questions related to function approximation, sampling error and nonstationarity, and where available, verify if trends found in oracle settings hold true with modern deep RL methods.
We find that large neural network architectures have many benefits with regards to learning stability; offer several practical compensations for overfitting; and develop a novel sampling method based on explicitly compensating for function approximation error that yields fair improvement on high-dimensional continuous control domains. 




§ INTRODUCTION

Q-learning algorithms, which are based on approximating state-action value functions, are an efficient and commonly used class of RL methods. In recent years, such methods have been applied to great effect in domains such as playing video games from raw pixels <cit.> and continuous control in robotics <cit.>. Methods based on approximate dynamic programming and Q-function estimation have several very appealing properties: they are generally moderately sample-efficient, when compared to policy gradient methods, they are simple to use, and they allow for off-policy learning. This makes them an appealing choice for a wide range of tasks, from robotic control <cit.> to off-policy learning from historical data for recommender <cit.> systems and other applications. However, although the basic tabular Q-learning algorithm is convergent and admits theoretical analysis <cit.>, its non-linear counterpart with function approximation (such as with deep neural networks) is poorly understood theoretically. 
In this paper, we aim to investigate the degree to which the theoretical issues with Q-learning actually manifest in practice. 
Thus, we empirically analyze aspects of the Q-learning method in a unit testing framework, where we can employ oracle solvers to obtain ground truth Q-functions and distributions for exact analysis. We investigate the following questions:

1) What is the effect of function approximation on convergence?
Most practical reinforcement learning problems, such as robotic control, require function approximation to handle large or continuous state spaces. However, the behavior of Q-learning methods under function approximation is not well understood. There are known counterexamples where the method diverges <cit.>, and there are no known convergence guarantees <cit.>. 
To investigate these problems, we study the convergence behavior of Q-learning methods with function approximation, parametrically varying the function approximator power and analyzing the quality of the solution as compared to the optimal Q-function and the optimal projected Q-function under that function approximator. 
We find, somewhat surprisingly, that function approximation error is not a major problem in Q-learning algorithms, but only when the representational capacity of the function approximator is high. This makes sense in light of the theory: a high-capacity function approximator can perform a nearly perfect projection of the backed up Q-function, thus mitigating potentially convergence issues due to an imperfect ℓ_2 norm projection. We also find that divergence rarely occurs, for example, we observed divergence in only 0.9% of our experiments. We discuss this further in Section <ref>.

2) What is the effect of sampling error and overfitting?
Q-learning is used to solve problems where we do not have access to the transition function of the MDP. Thus, Q-learning methods need to learn by collecting samples in the environment, and training on these samples incurs sampling error, potentially leading to overfitting. This causes errors in the computation of the Bellman backup, which degrades the quality of the solution. We experimentally show that overfitting exists in practice by performing ablation studies on the number of gradient steps, and by demonstrating that oracle based early stopping techniques can be used to improve performance of Q-learning algorithms.  (Section <ref>).
Thus, in our experiments we quantify the amount of overfitting which happens in practice, incorporating a variety of metrics, an performing a number of ablations and investigate methods to mitigate its effects.

3) What is the effect of distribution shift and a moving target?
The standard formulation of Q-learning prescribes an update rule, with no corresponding objective function <cit.>. This results in a process which optimizes an objective that is non-stationary in two ways: the target values are updated during training, and the distribution under which the Bellman error is optimized changes, as samples are drawn from different policies. We refer to these problems as the moving target and distribution shift problems, respectively. These properties can make convergence behavior difficult to understand, and prior works have hypothesized that nonstationarity is a source of instability <cit.>. In our experiments, we develop metrics to quantify the amount of distribution shift and performance change due to non-stationary targets. Surprisingly, we find that in a controlled experiment, distributional shift and non-stationary targets do not in fact correlate with reduction in performance. In fact, sampling strategies with large distributional shift often perform very well.

4) What is the best sampling or weighting distribution?
Deeply tied to the distribution shift problem is the choice of which distribution to sample from. Do moving distributions cause instability, as Q-values trained on one distribution are evaluated under another in subsequent iterations?
Researchers have often noted that on-policy samples are typically superior to off-policy samples <cit.>, and there are several theoretical results that highlight favorable convergence properties under on-policy samples. However, there is little theoretical guidance on how to pick distributions so as to maximize learning rate. To this end, we investigate several choices for the sampling distribution. Surprisingly, we find that on-policy training distributions are not always preferable, and that a clear pattern in performance with respect to training distribution is that broader, higher-entropy distributions perform better, regardless of distributional shift. Motivated by our findings, we propose a novel weighting distribution, adversarial feature matching (AFM), which is explicitly compensates for function approximator error, while still producing high-entropy sampling distributions.

Our contributions are as follows:
We introduce a unit testing framework for Q-learning to disentangle issues related to function approximation, sampling, and distributional shift where approximate components are replaced by oracles. This allows for controlled analysis of different sources of error. We perform a detailed experimental analysis of many hypothesized sources of instability, error, and slow training in Q-learning algorithms on tabular domains, and show that many of these trends hold true in high dimensional domains. We propose novel choices of sampling distributions which lead to improved performance even on high-dimensional tasks. Our overall aim is to offer practical guidance for designing RL algorithms, as well as to identify important issues to solve in future research.




§ PRELIMINARIES


Q-learning algorithms aim to solve a Markov decision process (MDP) by learning the optimal state-action value function, or Q-function. We define an MDP as a tuple (𝒮, 𝒜, , R, γ). 𝒮, 𝒜 represent the state and action spaces, respectively. (s' | s, a) and R(s,a) represent the dynamics (transition distribution) and reward function, and γ∈ (0,1) represents the discount factor. The goal in RL is to find a policy π(a|s) that maximizes the expected cumulative discounted rewards, known as the returns:

    π^* = π E_s_t+1∼(·|s_t, a_t), a_t ∼π(·|s_t)[∑_t=0^∞γ^t R(s_t, a_t)]

The quantity of interest in Q-learning methods are state-action value functions, which give the expected future return starting from a particular state-action tuple, denoted Q^π(s,a). The state value function can also be denoted as V^π(s,a). Q-learning algorithms are based on iterating the Bellman backup operator , defined as

    ( Q)(s, a) = R(s, a) + γ E_s' ∼[V(s')]


    V(s) = max_a' Q(s, a')

The (tabular) Q-iteration algorithm is a dynamic programming algorithm that iterates the Bellman backup Q^t+1← Q^t. Because the Bellman backup is a γ-contraction in the L-∞ norm, and Q^* (the Q-values of π^*) is its fixed point, Q-iteration can be shown to converge to Q^* <cit.>. A deterministic optimal policy can then be obtained as π^*(s) = a Q^*(s,a).

When state spaces cannot be enumerated in a tabular format, function approximators can be used to represent the Q-values. An important class of such Q-learning methods are fitted Q-iteration (FQI) <cit.>, or approximate dynamic programming (ADP) methods, which form the basis of modern deep RL methods such as DQN <cit.>.
FQI projects the values of the Bellman backup onto a family of Q-function approximators :

    Q^t+1←( Q^t) .

Here,  denotes a μ-weighted L2 projection, which minimizes the Bellman error via supervised learning:

    (Q) Q' ∈ E_s,a ∼μ[(Q'(s,a) - Q(s,a))^2]
     .

The values produced by the Bellman backup, ( Q^t)(s,a) are commonly referred to as target values, and when neural networks are used for function approximation, the previous Q-function Q^t(s,a) is referred to as the target network. In this work, we distinguish between the cases when the Bellman error is estimated with Monte-Carlo sampling or computed exactly (see Section <ref>). The sampled variant corresponds to FQI as described in the literature <cit.>, while the exact variant is analogous to conventional ADP <cit.>. 

Convergence guarantees for Q-iteration do not cleanly translate to FQI.  is an  projection, but  is a contraction in the  norm – this norm mistmatch means the composition of the backup and projection is no longer guaranteed to be a contraction under any norm <cit.>, and hence the convergence is not guaranteed.

A related branch of Q-learning methods are online Q-learning methods,
in which Q-values are updated while samples are being collected in the MDP. This includes classic algorithms such as Watkin's Q-learning <cit.>. Online Q-learning methods can be viewed as a form of stochastic approximation (such as Robbins-Monro) applied to Q-iteration and FQI <cit.>, and share many of its theoretical properties <cit.>.
Modern deep RL algorithms such as DQN <cit.> have characteristics of both online Q-learning and FQI – using replay buffers means the sampling distribution μ changes very little between target updates (see Section <ref>), and target networks are justified from the viewpoint of FQI. Because FQI corresponds to the case when the sampling distribution is static between target updates, the behavior of modern deep RL methods more closely resembles FQI than a true online method without target networks.




§ EXPERIMENTAL SETUP


Our experimental setup is centered around unit-testing. We first introduce a spectrum of Q-learning algorithms, starting with exact approximate dynamic programming and gradually replacing oracle components, such as knowledge of dynamics, until the algorithm resembles modern deep Q-learning methods. We then introduce a suite of tabular environments where oracle solutions can be computed and compared against, to aid in diagnosis, as well as testing in high-dimensional environments to verify our hypotheses.

In order to provide consistent metrics across domains, we normalize returns and errors involving Q-functions (such as Bellman error) by the returns of the expert policy π^* on each environment.



 §.§ Algorithms


In the analysis presented in Section <ref>, <ref>, <ref> and <ref>, we will use three different Q-learning variants, each of which remove some of the approximations in the standard Q-learning method used in the literature – 
Exact-FQI, Sampling-FQI, and Replay-FQI. Although FQI is not exactly identical to commonly used deep RL methods, such as DQN <cit.>, DDPG <cit.>, and SAC <cit.>, it is structurally similar and, when the replay buffer for the commonly used methods becomes large, the difference becomes negligible, since the sampling distribution changes very little between target network updates. However, FQI methods are much more amenable for controlled analysis, since we can separately isolate target values, update rates, and the number of samples used for each iteration. We therefore use variants of FQI as the basis for our analysis, but we also confirm that similar trends hold with more commonly used algorithms on standard benchmark problems.




Exact-FQI (Algorithm <ref>): Exact-FQI computes the backup and projection on all state-action tuples without any sampling error.
It also assumes knowledge of dynamics and reward function to compute Bellman backups exactly. We use Exact-FQI to study convergence, distribution shift (by varying weighting distributions on transitions), and function approximation in the absence of sampling error. Exact-FQI eliminates errors due to sampling states, and computing inexact, sampled backups.

Sampled-FQI (Algorithm <ref>): Sampled-FQI is a special case of Exact-FQI,
where the Bellman error is approximated with Monte-Carlo estimates from a sampling distribution μ, and the Bellman backup is approximated with samples from the dynamics as r(s,a) + γmax_a'Q(s', a'). We use Sampled-FQI to study effects of overfitting. Sampled-FQI incorporates all sources of error – arising from function approximation, sampling and also distribution shift.

Replay-FQI (Algorithm <ref>): Replay-FQI is a special case of Sampled-FQI that uses a replay buffer <cit.>,
that saves past transition samples (s, a, s', r), which are used for computing Bellman error. Replay-FQI strongle resembles DQN <cit.>, lacking the online updates that allow μ to change within an FQI iteration. 
With large replay buffers, we expect the difference between Replay-FQI and DQN to be minimal as μ changes slowly.

We additionally investigate the following choices of weighting distributions (μ) for the Bellman error. When sampling the Bellman error, these can be implemented by sampling directly from the distribution, or via importance sampling.

Unif(s,a): Uniform weights over state-action space. This is the weighting distribution typically used by dynamic programming algorithms, such as FQI.

π(s,a): The on-policy state-action marginal induced by π.

π^*(s,a): The state-action marginal induced by π^*.

Random(s,a): State-action marginal induced by executing uniformly random actions.

Prioritized(s,a): Weights Bellman errors proportional to |Q(s,a)- Q(s,a)|. This is similar to prioritized replay <cit.> without importance sampling.

Replay(s,a) and Replay10(s,a): Averaged state-action marginal of all policies (or the previous 10) produced during training. This simulates sampling uniformly from a replay buffer where infinite samples are collected from each policy. 



 §.§ Domains


We evaluate our methods on suite of tabular environments where we can compute oracle values. This will help us compare, analyze and fix various sources of error by means of comparing the learned Q-functions to the true, oracle-compute Q-functions.
We selected 8 tabular domains, each with different qualitative attributes, including: gridworlds of varying sizes and observations, blind Cliffwalk <cit.>,
discretized Pendulum and Mountain Car based on implementations in OpenAI Gym <cit.>,
and a random sparsely connected graph. We give full details of these environments in Appendix <ref>, as well as their motivation for inclusion.



 §.§ Function Approximators

Throughout our experiments, we use 2-layer ReLU networks, denoted by a tuple (N, N) where N represents the number of units in a layer. The “Tabular” architecture refers to the case when no function approximation is used. 



 §.§ High-Dimensional Testing


In addition to diagnostic experiments on tabular domains, we also wish to see if the observed trends hold true on high-dimensional environments. To this end, we include experiments on continuous control tasks in the OpenAI Gym benchmark <cit.> (HalfCheetah-v2, Hopper-v2, Ant-v2, Walker2d-v2). In continuous domains, computing the maximum over actions of the Q-value is difficult (max_a Q(s,a)). A common choice in this case is to use a second “actor” neural network to approximate max_a Q(s,a) <cit.>. This approach most closely resembles Replay-FQI, but using the actor network in place of the max.



§ FUNCTION APPROXIMATION AND CONVERGENCE



The first issue we investigate is the connection between function approximation and convergence properties.



 §.§ Technical Background

As discussed in Section <ref>, when function approximation is introduced to Q-learning, convergence guarantees are lost. This interaction between approximation and convergence has been a long-studied topic in reinforcement learning. In the control literature, it is closely related to the problems of state-aliasing or interference <cit.>. <cit.> introduces a simple counterexample in which Watkin's Q-learning with linear approximators can cause unbounded divergence. In the policy evaluation scenario, <cit.> prove that on-policy TD-learning with linear function approximators converge, and methods such as GTD <cit.> and ETD <cit.> have extended results to off-policy cases.
In the control scenario, convergent algorithms such as SBEED <cit.> and Greedy-GQ <cit.> have been developed. However, several works have noted that divergence need not occur. <cit.> theoretically addresses the norm-mismatch problem, which show that unbounded divergence is impossible provided μ has adequate support and projections are non-expansive in p-norms. Concurrently to us, <cit.> experimentally find that unbounded divergence rarely occurs with DQN variants on Atari games. 


 §.§ How does function approximation affect convergence properties and suboptimality of solutions?

The crucial quantities we wish to measure are a trend between function approximation and performance, and a measure for the bias in the learning procedure introduced by function approximation.
Thus, using Exact-FQI with uniform weighting (to remove sampling error), we measure the returns of the learning policy, and the  error between Q^* and the solution found by Exact-FQI (lim_t →∞ ()^t Q^0) or the projection of the optimal solution  (Q^*).
Q^* represents the best solution inside the model class, in absence of error from the bootstrapping process of FQI. Thus, the difference between FQI error and projection error represents the bias introduced by the bootstrapping procedure, while controlling for bias that is simply due to function approximation – this quantity is roughly the inherent Bellman error of the function class <cit.>. This is the gap which can possibly be improved upon via better Q-learning algorithm design. We plot our results in Fig. <ref>.



We first note the obvious trend that smaller architectures produce lower returns, and converge to more suboptimal solutions. 
However, we also find that smaller architectures introduce significant bias in the learning process, and there is often a significant gap between the solution found by Exact-FQI and the best solution within the model class. 
This gap may be due to the fact that when the target is bootstrapped, we must be able to represent all Q-function along the path to the solution, and not just the final result <cit.>.
This observation implies that using large architectures is crucial not only because they have capacity to represent a better solution, but also because they are significantly easier to train using bootstrapping, and suffer less from nonconvergence issues. 
We also note that divergence rarely happens in practice. We observed divergence in 0.9% of our experiments using function approximation, measured by the largest Q-value growing larger than 10 times that of Q^*. 

For high-dimensional problems, we present experiments on varying the architecture of the Q-network in SAC <cit.> in Appendix Fig. <ref>. We still observe that large networks have the best performance, and that divergence rarely happens even in high-dimensional continuous spaces. We briefly discuss theoretical intuitions on apparent discrepancy between the lack of unbounded divergence in relation known counterexamples in Appendix <ref>.



§ SAMPLING ERROR AND OVERFITTING



A second source of error in minimizing the Bellman error, orthogonal to function approximation, is that of sampling or generalization error. The next issue we investigate is the effect of sampling error on Q-learning methods.



 §.§ Technical Background

r0.6



    < g r a p h i c s >

 Samples plotted with returns for a 256x256 network. More samples yields better performance.






Approximate dynamic programming assumes that the projection of the Bellman backup (Eqn. <ref>) is computed exactly, but in reinforcement learning we can normally only compute the empirical Bellman error over a finite set of samples. In the PAC framework, overfitting can be quantified by a bounded error in between the empirical and expected loss with high probability, which decays with sample size <cit.>. <cit.> provide such PAC-bounds which account for sampling error in the context of Q-learning and value-based methods, and quantify the quality of the final solution in terms of sample complexity.

We analyze several key points that relate to sampling error. First, we show that Q-learning is prone to overfitting, and that this overfitting has a real impact on performance, in both tabular and high-dimensional settings. We also show that the replay buffer is in fact a very effective technique in addressing this issue, and discuss several methods to migitate the effects of overfitting in practice.





 §.§ Quantifying Overfitting

We first quantify the amount of overfitting that happens during training, by varying the number of samples. In order provide comparable validation errors across different experiments, we fix a reference sequence of Q-functions, Q^1, ... , Q^N, obtained during a normal training run. We then retrace the training sequence, and minimize the projection error (Q^t) for each training iteration, using varying amounts of on-policy data or sampling from a replay buffer. We measure the exact validation error (the expected Bellman error) at each iteration under the on-policy distribution, plotted in Fig. <ref>. We note the obvious trend that more samples leads to lower validation loss, confirming that overfitting can in fact occur. A more interesting observation is that sampling from the replay buffer results in the lowest on-policy validation loss, despite bias due to distribution mismatch from sampling off-policy data. As we discuss in Section <ref>, we believe that replay buffers are mainly effective because they greatly reduce the effect of overfitting and create relatively good coverage over the state space, not necessarily due to reducing the effects of distribution shift.

Next, Fig. <ref> shows the relationship between number of samples and returns. We see a clear trend that higher sample count leads to improved learning speed and a better final solution, confirming our hypothesis that overfitting has a significant effect on the performance of Q-learning. A full sweep including architectures is presented in Appendix Fig. <ref>. We observe that despite overfitting being an issue, larger architectures still perform better because the bias introduced by smaller architectures dominates.







 §.§ What methods can be used to compensate for overfitting?


Finally, we discuss methods to compensate for overfitting. One common method for reducing overfitting is to regularize the function approximator to reduce its capacity. However, as we have seen before that weaker architectures can give rise to suboptimal convergence, we instead study early stopping methods to mitigate overfitting without reducing model size.
First, we observe that the number of gradient steps taken per sample in the projection step has an important effect on performance – too few steps and the algorithm learns slowly, but too many steps and the algorithm may initially learn quickly but overfit. To show this, we run a hyperparameter sweep over the number of gradient steps taken per environment step in Replay-FQI and TD3 (TD3 uses 1 by default). Results for FQI are shown in Fig. <ref>, and for TD3 in Appendix Fig. <ref>.

In order to understand whether better early stopping criteria can possibly help with overfitting, we employ oracle early stopping rules. While neither of these rules can be used to solve overfitting in practice, these experiments can provide guidance for future methods and an “upper bound” on the best improvement that can be obtained from optimal stopping. We investigate two oracle early stopping criteria for setting the number of gradient steps: using the expected Bellman error and the expected returns of the greedy policy w.r.t. the current Q-function (oracle returns). We implement both methods by running the projection step of Replay-FQI to convergence using gradient descent, and afterwards selecting the intermediate Q-function which is judged best by the evaluation metric (lowest Bellman error or highest returns). Using such oracle stopping metrics results in a modest boost in performance in tabular domains (Fig. <ref>). Thus, we believe that there is promise in further improving such early-stopping methods for reducing overfitting in deep RL algorithms.

We might draw a few actionable conclusions from these experiments. First, overfitting is indeed a serious issue with Q-learning, and too many gradient steps or too few samples can lead to poor performance. Second, replay buffers and early stopping can be used to mitigate the effects of overfitting. Third, although overfitting is a problem, large architectures are still preferred, because the harm from function approximation bias outweighs the harm from increased overfitting with large models.


§ NON-STATIONARITY



In this section, we discuss issues related to the non-stationarity of the Q-learning process (relating to the Bellman backup and Bellman error minimization).



 §.§ Technical Background

Instability in Q-learning methods is often attributed to the nonstationarity of the regression objective <cit.>. 
Nonstationarity occurs in two places: in the changing target values Q, and in a changing weighting distribution μ (“distribution shift”) (i.e., due to samples being taken from different policies). Note that a non-stationary objective, by itself, is not indicative of instability. For example, gradient descent can be viewed as successively minimizing linear approximations to a function: for gradient descent on f with parameter θ and learning rate α, we have the “moving” objective θ^t+1 = θ{θ^T ∇_θ f(θ^t) - 1/2αθ - θ^t} = θ^t - α∇_θ f(θ^t). 
However, the fact that the Q-learning algorithm prescribes an update rule and not a stationary objective complicates analysis. Indeed, the motivation behind algorithms such as GTD <cit.> and residual methods <cit.> can be seen as introducing a stationary objective that can be optimized with standard procedures such as gradient descent for increased stability.
Therefore, a key question to investigate is whether these non-stationarities are detrimental to the learning process.



 §.§ Does a moving target cause instability in the absence of a moving distribution?


To study the moving target problem, we must first isolate the effects of a moving target, and study how the rate at which the target changes impacts performance. To control the rate at which the target changes, we introduce an additional smoothing parameter α to Q-iteration, where the target values are now computed as an α-moving average over previous targets. We define the α-smoothed Bellman backup, ^α, as follows: 

    ^αQ = α Q + (1-α)Q

This scheme is inspired by the soft target update used in algorithms such as DDPG <cit.> and SAC <cit.> to improve the stability of learning. Standard Q-iteration uses a “hard” update where α=1. A soft target update weakens the contraction of Q-iteration from γ to 1-α+αγ (See Appendix <ref>),
so we expect slower convergence, but perhaps it is more stable under heavy function approximation error. We performed experiments with this modified backup using Exact-FQI under the Unif(s,a) weighting distribution.

Our results are presented in Appendix Fig. <ref>.
We find that the most cases, the hard update with α=1 results in the fastest convergence and highest asymptotic performance. However, for the smallest two architectures we used, 4 × 4 and 16 × 16, lower values of α (such as 0.1) achieve slightly higher asymptotic performance. Thus, while more expressive architectures are still stable under fast-changing targets, we believe that a slowly moving target may have benefits under heavy approximation error. This evidence points to either using large function approximators, in line with the conclusions drawn in the previous sections, or adaptively slowing the target updates when the architecture is weak (relative to the problem difficulty) and the projected Bellman error is therefore high.



 §.§ Does distribution shift impact performance?






To study the distribution shift problem, we exactly compute the amount of distribution shift between iterations in total-variation distance, D_TV(μ^t+1 || μ^t) and the “loss shift”:

    𝔼_μ^t+1[ (Q^t -  Q^t)^2] - 𝔼_μ^t[ (Q^t -  Q^t)^2] .
 
The loss shift quantifies the Bellman error objective when evaluated under a new distribution - if the distribution shifts to previously unseen or low support states, we would expect a highly inaccurate Q-value in such states, and a correspondingly high loss shift.




We run our experiments using Exact-FQI with a 256x256 layer architecture, and plot the distribution discrepancy and the loss discrepancy in Fig. <ref>. 
We find that Prioritized(s,a) has the greatest shift, followed by on-policy variants. Replay buffers greatly reduce distribution shift compared to on-policy learning, which is similar to the de-correlation argument cited for its use by <cit.>.
However, we find that this metric correlates very little with the actual performance of the algorithm (Fig. <ref>). For example, prioritized weighting performs well yet has high distribution shift.

Overall, our experiments indicate that nonstationarities in both distributions and target values, when isolated, do not cause significant stability issues. Instead, other factors such as sampling error and function approximation appear to have more significant effects on performance. In the light of these findings, we might therefore ask: can we design a better sampling distribution, without regard for distributional shift and with regard for high-entropy, that results in better final performance, and is realizable in practice? We investigate this in the following section.



§ SAMPLING DISTRIBUTIONS



As alluded to in Section <ref>, the choice of sampling distribution μ is an important design decision can have a large impact on performance. Indeed, it is not immediately clear which distribution is ideal for Q-learning. In this section, we hope to shed some light on this issue.



 §.§ Technical Background

Off-policy data has been cited as one of the “deadly triads” for Q-learning <cit.>, which has potential to cause instabilities in learning. On-policy distributions <cit.> and fixed behavior distributions <cit.> have often been targeted for theoretical convergence analysis, and many works use importance sampling to correct for off-policyness <cit.>
However, to our knowledge, there is relatively little guidance which compares how different weighting distributions compare in terms of convergence rate and final solutions.

Nevertheless, several works give hypotheses on good choices for weighting distributions. <cit.> provides an error bound which suggests that “more uniform” weighting distributions can guarantee better worst-case performance. <cit.> suggests that when the state-distribution is fixed, the action distribution should be weighted by the optimal policy for residual Bellman errors. In deep RL, several methods have been developed to prevent instabilities in Q-Learning, such as prioritized replay <cit.>, and mixing replay buffer with on-policy data <cit.> have been found to be beneficial. In our experiments, we aim to empirically analyze multiple choices for weighting distributions to determine which are the most effective.



 §.§ What Are the Best Weighting Distributions in Absence of Sampling Error?






We begin by studying the effect of weighting distributions when disentangled from sampling error. We run Exact-FQI with varying choices of architectures and weighting distributions and report our results in Fig. <ref>. Unif(s,a) Replay(s,a), and Prioritized(s,a) consistently result in the highest returns across all architectures.
We believe that these results are in favor of the uniformity hypothesis: the top performing distributions spread weight across larger support of the state-action space. For example, a replay buffer contains state-action tuples from many policies, and therefore would be expected to have wider support than the state-action distribution of a single policy. We can see this general trend in Fig. <ref>. These distributions generally result in the tightest contraction rates, and allow the Q-function to focus on locations where the error is high. In the sampled setting, this observation motivates exploration algorithms that maximize state coverage (for example,  <cit.> solve an exploration objective which maximizes state-space entropy).
However, note that in this particular experiment, there is no sampling. All states are observed, just with different weights, thus isolating the issue of distributions from the issue of sampling.



 §.§ Designing a Better Off-Policy Distribution: Adversarial Feature Matching



In our final study, we attempt to design a better weighting distribution using insights from previous sections that can be easily integrated into deep RL methods. We refer to this method as adversarial feature-matching (AFM). We draw upon three specific insights outlined in previous analysis. First, the function approximator should be incentivized to maximize its ability to distinguish states to minimize function approximation bias (Section <ref>). Second, the weighting distribution should emphasize areas where the Q-function incurs high Bellman error, in order to minimize the discrepancy between  norm error and  norm error. Third, more-uniform weighting distributions tend to be higher performant. The first insight was also demonstrated in <cit.> where enforcing sparsity in the Q-function was found to provide locality in the Q-function which prevented catastrophic interference and provided better values for bootstrapping.

We propose to model our problem as a minimax game, where the weighting distribution is a parameterized adversary p_ϕ(s, a) which tries to maximize the Bellman error, while the Q-function (Q_θ(s, a)) tries to minimize it. 
Note that in the unconstrained setting, this game is equivalent to minimizing the  norm error in its dual-norm representation. However, in practical settings where minimizing stochastic approximations of the  norm can be difficult for neural networks (also noticed when using PER <cit.>), it is crucial to introduce constraints to limit the power of the adversary. These constraints also make the adversary closer to the uniform distribution while still allowing it to be sufficiently different at specific state-action pairs.

We elect to use a feature matching constraint which enforces the expected feature vectors, 𝔼[Φ(s)], under p_ϕ(s, a) to roughly match the expected feature vector under uniform sampling from the replay buffer. We can express the output of a neural network Q-function as Q_θ(s, a) = w_a^T Φ_θ(s) or, in the continuous case, as Q_θ(s, a) = w^T Φ_θ(s, a),  where the feature vector Φ_θ(s), Φ_θ(s, a) represent the the output of all but the final layer.
Intuitively, this constraint restricts the adversarial sampler to distributing probability mass among states (or state-action pairs) that are perceptually similar to the Q-function, which in turn forces the Q-function to reduce state-aliasing by learning features that are more separable. Note that, in our case, ϕ(s,a) = ∇_w Q_w, θ(s, a). This also provides a natural extension of our method by performing expected gradient matching over all parameters (𝔼_p_ϕ(s, a)[∇_w, θ Q_w, θ(s, a)]), instead of matching only Φ (we leave it to future work to explore this direction). Formally, this objective is given as follows:

    min_θ, wmax_ϕ𝔼_p_ϕ(s, a) [(Q_w, θ (s, a) - y(s, a))^2]
    
    s.t.  ||𝔼_p_ϕ(s, a)[Φ(s)] - ∑_i Φ(s_i)/N||≤ε

Note that Φ(s) is a function of θ but, while solving the maximization, θ is assumed to be a constant. This is equivalent to solving only the inner maximization with a constraint, and empirically provides better stability. Implementation details for AFM are provided in Appendix <ref>. The ∑_iΦ(s_i)/N denotes an estimator for the true expectation under some sampling distribution, such as a uniform distribution over all states and actions (in exact FQI) or the replay buffer distribution. So, ∑_iΦ(s_i)/N≈ E_p_rb[Φ] holds when using a replay buffer.

While both AFM and PER tend to upweight samples in the buffer with a high Bellman error, PER explicitly attempts to reduce distribution shift via importance sampling. As we observed in Section <ref>, distributional shift is not actually harmful in practice, and AFM dispenses with this goal, instead explicitly aiming to rebalance the buffer to attain better coverage via adversarial optimization. In our experiments, this results in substantially better performance, consistent with the hypothesis that coverage, rather than reduction of distributional shift, is the most important property in a sampling distribution.

In tabular domains with Exact-FQI, we find that AFM performs at par with the top performing weighting distributions, such as Unif(s, a) and better than Prioritized(s, a) (Fig. <ref>). This confirms that adaptive prioritization works better than Prioritized(s, a). Another benefit of AFM is its robustness to function approximation and the performance gains in the case of small architectures (say, (4, 4)) are particularly noticeable. (Fig. <ref>)

In tabular domains with Replay-FQI (Table <ref>), we also compare AFM to prioritized replay (PER) <cit.>, where AFM and PER perform similarly in terms of normalized returns. Note that AFM reweights samples drawn uniformly from the buffer, whereas PER changes which samples are actually drawn. We also evaluate a variant of AFM (AFM+Sampling in Table <ref>) which changes which samples instead of reweighting.
Essentially, in this version we sample from the replay buffer using probabilities determined by the AFM optimization, rather than using importance sampling while making bellman updates. We note that, in Table <ref>, AFM+Sampling performs strictly better than AFM and PER.

We further evaluate AFM on MuJoCo tasks with the TD3 algorithm <cit.> and the entropy constrained SAC algorithm <cit.>. We find that in all 3 tested domains (Half-Cheetah, Hopper and Ant), AFM yields substantial empirical improvement in the case of TD3 (Fig. <ref>) and performs slightly better than entropy constrained SAC (Fig. <ref>). Surprisingly, we found PER to not work very well in these domains. In light of these results, we conclude that: (1) the choice of sampling distribution is very  important for performance, and (2) considerations such as incorporating knowledge about the function approximator (for example, through Φ) into the choice of μ (the sampling/weighting distribution) can be very effective.










§ CONCLUSIONS AND DISCUSSION


From our analysis, we have several broad takeaways for the design of deep Q-learning algorithms. 

Potential convergence issues with Q-learning do not seem to be endemic empirically, but function approximation still has a strong impact on the solution to which these methods converge. This impact goes beyond just approximation error, suggesting that Q-learning methods do find suboptimal solutions (within the given function class) with smaller function approximators. However, expressive architectures largely mitigate this problem, suffer less from bootstrapping error, converge faster, and more stable with moving targets.

Sampling error can cause substantial overfitting problems with Q-learning. However, replay buffers and early stopping can mitigate this problem, and the biases incurred from small function approximators outweigh any benefits they may have in terms of overfitting. We believe the best strategy is to keep large architectures but carefully select the number of gradient steps used per sample. We showed that employing oracle early stopping techniques can provide huge benefits in the performance in Q-learning. This motivates the future research direction of devising early stopping techniques to dynamically control the number of gradient steps in Q-learning, rather than setting it as a hyperparameter as this can give rise to big difference in performance.

The choice of sampling or weighting distribution has significant effect on solution quality, even in the absence of sampling error. Surprisingly, we do not find on-policy distributions to be the most performant, but rather methods which have high state-entropy and spread mass uniformly among state-action pairs, seem to be highly effective for training. Based on these insights, we propose a new weighting distribution which balances high-entropy and state aliasing, AFM, that yields fair improvements in both tabular and continuous domains with state-of-the-art off-policy RL algorithms.


Finally, we note that there are many other topics in Q-learning that we did not investigate, such as overestimation bias and multi-step returns. We believe that these issues too could be studied in future work with our oracle-based analysis framework.




§ ACKNOWLEDGEMENTS

We thank Vitchyr Pong and Kristian Hartikainen for providing us with implementations of RL algorithms. We thank Chelsea Finn for comments on an earlier draft of this paper. SL thanks George Tucker for helpful discussion. We thank Google, NVIDIA, and Amazon
for providing computational resources. This research was supported by Berkeley DeepDrive, NSF IIS-1651843 and IIS-1614653, the DARPA Assured Autonomy program, and ARL DCIST CRA W911NF-17-2-0181.

















icml2019









§ APPENDICES




§ BENCHMARK TABULAR DOMAINS



We evaluate on a benchmark of 8 tabular domains, selected for qualitative differences.

4 Gridworlds. The Gridworld environment is an NxN grid with randomly placed walls. The reward is proportional to Manhattan distance to a goal state (1 at the goal, 0 at the initial position), and there is a 5% chance the agent travels in a different direction than commanded. We vary two parameters: the size (16 × 16 and 64 × 64), and the state representations. We use a “one-hot” representation, an (X, Y) coordinate tuple (represented as two one-hot vectors), and a “random” representation, a vector drawn from 𝒩(0, 1)^N, where N is the width or height of the Gridworld. The random observation significantly increases the challenge of function approximation, as significant state aliasing occurs.

Cliffwalk: Cliffwalk is a toy example from <cit.>. It consists of a sequence of states, where each state has two allowed actions: advance to the next state or return to the initial state. A reward of 1.0 is obtained when the agent reaches the final state. Observations consist of vectors drawn from 𝒩(0, 1)^16.

InvertedPendulum and MountainCar: InvertedPendulum and MountainCar are discretized versions of continuous control tasks found in OpenAI gym <cit.>, and are based on problems from classical RL literature. 
In the InvertedPendulum task, an agent must swing up an pendulum and hold it in its upright position. The state consists of the angle and angular velocity of the pendulum. Maximum reward is given when the pendulum is upright. The observation consists of the sin and cos of the pendulum angle, and the angular velocity.
In the MountainCar task, the agent must push a vehicle up a hill, but the hill is steep enough that the agent must gather momentum by swinging back and forth within a valley in order to reach the top. The state consists of the position and velocity of the vehicle.

SparseGraph: The SparseGraph environment is a 256-state graph with randomly drawn edges. Each state has two edges, each corresponding to an action. One state is chosen as the goal state, where the agent receives a reward of one.



§ FITTED Q-ITERATION WITH BOUNDED PROJECTION ERROR



When function approximation is introduced to Q-iteration, we lose guarantees that our solution will converge to the optimal solution Q^*, because the composition of projection and backup is no longer guaranteed to be a contraction under any norm. However, this does not imply divergence, and in most cases it merely degrades the quality of solution found.

This can be seen by recalling the following result from <cit.>, that describes the quality of the solution obtained by fitted Q-iteration (FQI) when the projection error at each step is bounded. The conclusion is that FQI converges to an  ball around the optimal solution which scales proportionally with the projection error. While this statement does not claim that divergence cannot occur in general (this theorem can only be applied in retrospect, since we cannot always uniformly bound the projection error at each iteration), it nevertheless offers important intuitions on the behavior of FQI under approximation error. For similar results concerning μ-weighted  norms, see <cit.>.



Let the projection or Bellman error at each iteration of FQI be uniformly bounded by δ, i.e. Q̂_i+1 - Q̂_i_∞≤δ ∀ i. Then, the error in the final solution is bounded as


    lim_i →∞Q̂_i - Q^*_∞≤δ/1-γ



See of Chapter 6 of <cit.>.

















We can use this statement to provide a bound on the performance of the final policy. 


Suppose we run fitted Q-iteration, and let the projection error at each iteration be uniformly bounded by δ, i.e. Q̂_i+1 - Q̂_i_∞≤δ ∀ i. Letting η(π) denote the returns of a policy π, the the performance of the final policy is bounded as:


    lim_i →∞ |η(π^i) - η(π^*)| ≤2γδ/(1-γ)^2



This result is obtained by substituting Thm. <ref> into Propositon 6.1 of  <cit.>.




 §.§ Unbounded divergence in FQI


Because  norms are bounded by the  norm, Thm. <ref> implies that unbounded divergence is impossible when weighting distribution has positive support at all states and actions (i.e. μ(s,a) > 0 ∀ (s, a) ∈ (𝒮, 𝒜)), and the projection is non-expansive in the  norm (such as when using linear approximators). 

We can bound the μ-weighted  in terms of the  as follows: ·≤1/minμ(s,a)·≤√(|𝒮||𝒜|)/minμ(s,a)·_∞. Thus, we can apply Thm. <ref> with δ = √(|𝒮||𝒜|)/minμ(s,a) to show that unbounded divergence is impossible. Note that because this bound scales with the size of the state and action spaces, it is fairly loose in many practical cases, and practitioners may nevertheless see Q-values grow to large values (tighter bounds concerning L2 norms can be found in <cit.>, which depend on the transition distribution). It also suggests that distributions which are fairly uniform (so as to maximize the denominator) can perform well.

When the weighting distribution μ does not have support over all states and actions, divergence can still occur, as noted in the counterexamples such as Section 11.2 of  <cit.>. In this case, we consider two states (state 1 and 2) with feature vectors 1 and 2, respectively, and a linear approximator with parameter w. There exists a single action with a deterministic transition from state 1 to state 2, and we only sample the transition from state 1 to state 2 (i.e. μ(s,a) is 1 for state 1 and 0 for state 2). All rewards are 0. In this case, the projected Bellman backup takes the form:

    w^t+1 = w (w - 2γ w^t)^2

Which will cause unbounded growth lim_t →∞ w_t = ∞ when iterated, provided γ > 0.5. However, if we add a transition from state 2 back to itself or to state 1, and place nonzero probability on sampling these transitions, divergence can be avoided.



§ Α-SMOOTHED Q-ITERATION



In this section we show that the α-smoothed Bellman backup introduced in Section <ref> is still a valid Q-iteration method, in that it is a contraction (for 1 ≥α > 0) and thus converges to Q^*.

We define the α-smoothed Bellman backup as:

    ^αQ = α Q + (1-α)Q




^α is a 1-α+γα-contraction:

    ^αQ_1 - ^αQ_2_∞≤ (1-α+γα)Q_1 - Q_2_∞



This statement follows from straightforward application of the triangle rule and the fact that  is a γ-contraction:

    ^αQ_1 - ^αQ_2_∞
       = (α Q_1 - (1-α)Q_1) - (α Q_2 - (1-α)Q_2 _∞
       = α ( Q_1 -  Q_2) + (1-α)(Q_1-Q_2) _∞
       ≤α Q_1 -  Q_2_∞ + (1-α)Q_1-Q_2 _∞
       ≤αγQ_1 - Q_2_∞ + (1-α)Q_1-Q_2 _∞
       =  (1-α + αγ) Q_1 - Q_2_∞








§ ADVERSARIAL FEATURE MATCHING (AFM): DETAILED EXPLANATION AND PRACTICAL IMPLEMENTATION


As described in section <ref>, we devise a novel weighting scheme for the Bellman error objective based on an adversarial minimax game. The adversary computes weights p_ϕ(s, a) (representing the weighting distribution μ), for the Bellman error: (Q_θ, w(s, a) - y(s, a))^2. Recalling from Section <ref>, the optimization problem is given by:

    min_θ, wmax_ϕ𝔼_p_ϕ(s, a) [(Q_w, θ (s, a) - y(s, a))^2]
    
    s.t.  ||𝔼_p_ϕ(s, a)[Φ(s)] - ∑_i Φ(s_i)/N||≤ε

where Φ(s) are the state features learned by the Q-function approximator. Φ(s) is easy to extract out of the multiheaded Q(s,a) (Q(s, a) = w_a^TΦ(s)) model typically used for discrete action control, as one choice is to let Φ(s) be the output of the penultimate layer of the Q-network. For continuous control tasks, however, we model Φ(s, a) (which is a function of the actions as well) as state-only features are unavailable, unless separately modeled. This can also be interpreted as modelling a feature matching constraint on the gradient of Q(s, a) with respect to the last linear parameters w_a. A possible extension is to take into account the entire gradient as the features in the feature matching constraint, that is, ∇_w, θ Q_w, θ(s, a).

This choice of the constraint is suitable and can be interpreted in two ways. First, an adversary constrained in this manner has enough power to exploit the Q-network at states which get aliased under the chosen function class, thereby promoting more separable feature learning and reducing some negative aspects of function approximation that can arise in Q-learning. This is also similar in motivation to <cit.>. Second, this feature constraint also bears a similarity the Maximum Mean Discrepancy (MMD) distance between two distributions P(x) and Q(x) that can be written as MMD^2(P, Q) := ||𝔼_P[Φ] - 𝔼_Q[Φ]||_ℋ, where the set of functions Φ is the canonical feature map, Φ: ℝ^n →ℋ (from real space to the RKHS). In our context, this is analogous to optimizing a distance between the adversarial distribution p_ϕ(s, a) and the replay buffer distribution p_rb(s, a) (as the average is a Monte-Carlo estimator of the expected Φ under the replay buffer distribution p_rb(s, a)). In the light of these arguments, AFM, and other associated methods that take into account the properties of the function approximator into account (for example, Φ here), can greatly reduce the bias incurred due to function approximation in the due course of Q-learning/FQI, as depicted in <ref>.



  
Solving the optimization We solve this saddle point problem using alternating dual gradient descent. We first solve the inner maximization problem, and then use its solution to then solve the outer minimization problem. We first compute the Lagrangian for the maximization, ℒ_inner(ϕ; λ, θ) by introducing a dual variable λ,

    ℒ_inner(ϕ; λ, θ) =  -𝔼_p_ϕ(s, a) [(Q_θ (s, a) - y(s, a))^2] +
    λ( ||𝔼_p_ϕ(s, a)[Φ(s)] - ∑Φ(s)/N|| - ε)

(Note that this Lagrangian is flipped in sign because we first convert the maximization problem to standard minimization form.) We now solve the inner problem using dual gradient descent. We then plug in the solutions (approximate solutions obtained after gradient descent), (p^*, λ^*) into the Lagrangian, to then solve the outside minimization over θ. Note that while Φ  depends on θ (as it is the feature layer of the Q-network), we don not backpropagate through Φ while solving the minimization. This improves stability of the Q-network training in practice and to makes sure that Q-function is only affected by FQI updates. In practice, we take up to 10 gradient steps for the inner problem every 1 gradient step of the outer problem. The algorithm is summarized in Algorithm <ref>. Our results provided in the main paper and here don't particularly assume any other tricks like Optimistic Gradient <cit.>, using exponential moving average of the parameters <cit.>. Our tabular experiments seemed to benefit some what using these tricks.   





  
Practical implementation with replay buffers We incorporate this weighting/sampling distribution into Q-learning in the setting with replay buffers and with state-action sampling. We evaluate the weighting version of our method, AFM, where, we usually sample a large batch B of state-action pairs from a usual replay buffer used in Q-learning, but use importance weights to then match p_ϕ(s,a) in expectation. Thus, we use a parametric function approximator to model p_ϕ(s, a)/p_rb(s, a) – that is, the importance weights of the adversarial distribution with respect to the replay buffer distribution p_rb(s, a). Mathematically, we estimate: E_p_ϕ(s, a)[δ(s, a)] := E_p_rb(s, a)[p_ϕ(s, a)/p_rb(s, a)δ(s, a)], where δ(s, a) = (Q_θ, w (s, a) - y(s, a))^2. The latter expectation is then approximated using a set of finite samples. It has been noted in literature that importance sampling (IS) suffers from high variance especially if the number of samples is small. Hence, we use the self-normalized importance sampling estimator, which averages the importance weights in a set of samples or a large number of samples. That is, let w_p/p_rb = p_ϕ(s, a)/p_rb(s, a)), then instead of using w_p/p_rb as the importance weights, we use w_p/p_rb(x) = w_p/p_rb(x)/∑_y ∈ B w_p/p_rb(y) (where x and y represent state-action tuples; concisely mentioned for visual clarity) as the importance weights. We also regularize the second-order Renyi Divergence between p_rb and p_ϕ for stability. Mathematically, it can be shown that this is a lower bound on the true expectation of δ under p_ϕ, which is being estimated using importance sampling. This result has also been shown in <cit.> (Theorem 4.1), where the authors use this lower bound in policy optimization via importance sampling. We state the theorem below for completeness.


<cit.> Let P and Q be two probability measures on the measurable space (X , F) such that
P << Q and d_2(P ||Q) < +∞. Let x_1, x_2, ⋯ , x_N be i.i.d. random variables sampled from Q, and f : X →ℝ be a bounded function. Then, for any 0 < δ≤ 1 and N > 0 with probability at least 1 - δ it holds that:

    𝔼_x ∼ Q[f(x)] ≥1/N∑ w_P/Q(x_i) f(x_i) -
     ||f||_∞√((1 - δ) d_2(P||Q)/Nδ)

where d_2(P||Q) ∝𝔼_Q[ (P(x)/Q(x))^2 ] is the exponentiated second-order Renyi Divergence between P and Q.


Hence, our objective for the inner loop now becomes: max_ϕ𝔼_p_ϕ(s, a) [δ(s, a)] = max_ϕ𝔼_p_rb[p_ϕ(s, a)/p_rb(s, a)δ(s, a)] is now computed using samples with an additional renyi regularisation term. Since, we end up modeling this ratio,  f_ϕ(s, a) := p_θ(s, a)/p_rb(s, a) through out parameteric model, we can hence easily compute an estimator for the Renyi divergence term. The overall lower bound inner maximization problem is:

    max_ϕ1/N∑_(s, a) ∼ p_rb[f_ϕ(s, a) (Q_θ, w (s, a) - y(s, a))^2] -
     C √((1 - δ) (∑ f_ϕ(s, a))^2/N)/Nδ)
    s.t.  ||∑_s, a ∈ p_rb[f_ϕ(s, a) Φ(s)]/N - ∑_s,a ∈ p_rbΦ(s)/N||≤ε


We found that this Renyi penalty helped stabilize training. In practice, we model the importance weights: f_ϕ(s, a) as a parametric model with an identical architecture to the Q-network. We use parameter clipping for f_ϕ(s, a), where the parameter are clipped to [-0.1, 0.1], analogous to Wasserstein GANs <cit.>. We also found that self-normalization during importance sampling has a huge practical benefit. Note that as the true  norm of the Bellman error is not known, for computing C in the Renyi Divergence term, and hence we either replace it by constant, or compute a stochastic approximation to the  norm over the current batch. We found the former to be more stable, and hence, used that in all our experiments. This coefficient of the Renyi divergence penalty is tuned uniformly between [0.0, 0.25]. The learning rate for the adversary was chosen to be 1e-4 for the tabular environments, and 5e-4 for TD3. The batch size for our algorithm was chosen to be 128 for the tabular environments and 500 for TD3/SAC. Note that a larger batch size ensures smoothness in the minmax optimization problem. We also found that instead of having a 1D Lagrange multiplier for the feature matching constraint, having d Lagrange multipliers for constraining each of the individual dimensions of the features Φ∈ℝ^d also helps very much. This is to ensure that the hyperparameters remain the same across different architectures regardless of the dimension of the penultimate layer of the Q-network. The algorithm in this case is exactly the same as the algorithm before with a vector valued dual variable λ. We used TD3 and SAC implementations from rlkit (<https://github.com/vitchyr/rlkit/tree/master/rlkit>)



§ FUNCTION APPROXIMATION ANALYSIS ON MUJOCO TASKS


As discussed in Section <ref>, we validate our findings on the effect of function approximation on 3 MuJoCo tasks from OpenAI Gym with the SAC algorithm from the author's implementation at <cit.>. We observe that bigger networks learn faster and better in general.





§ ADDITIONAL PLOTS











