Published as a conference paper at ICLR 2018

PARAMETER S PACE N OISE FOR E XPLORATION

arXiv:1706.01905v2 [cs.LG] 31 Jan 2018

Matthias Plappert†‡ , Rein Houthooft† , Prafulla Dhariwal† , Szymon Sidor† ,
Richard Y. Chen† , Xi Chen†† , Tamim Asfour‡ , Pieter Abbeel†† , and Marcin Andrychowicz†
†
OpenAI
‡
Karlsruhe Institute of Technology (KIT)
††
University of California, Berkeley
Correspondence to matthias@openai.com

A BSTRACT
Deep reinforcement learning (RL) methods generally engage in exploratory behavior through noise injection in the action space. An alternative is to add noise
directly to the agent’s parameters, which can lead to more consistent exploration
and a richer set of behaviors. Methods such as evolutionary strategies use parameter
perturbations, but discard all temporal structure in the process and require significantly more samples. Combining parameter noise with traditional RL methods
allows to combine the best of both worlds. We demonstrate that both off- and
on-policy methods benefit from this approach through experimental comparison
of DQN, DDPG, and TRPO on high-dimensional discrete action environments as
well as continuous control tasks.

1

I NTRODUCTION

Exploration remains a key challenge in contemporary deep reinforcement learning (RL). Its main
purpose is to ensure that the agent’s behavior does not converge prematurely to a local optimum.
Enabling efficient and effective exploration is, however, not trivial since it is not directed by the
reward function of the underlying Markov decision process (MDP). Although a plethora of methods
have been proposed to tackle this challenge in high-dimensional and/or continuous-action MDPs,
they often rely on complex additional structures such as counting tables (Tang et al., 2016), density
modeling of the state space (Ostrovski et al., 2017), learned dynamics models (Houthooft et al., 2016;
Achiam & Sastry, 2017; Stadie et al., 2015), or self-supervised curiosity (Pathak et al., 2017).
An orthogonal way of increasing the exploratory nature of these algorithms is through the addition
of temporally-correlated noise, for example as done in bootstrapped DQN (Osband et al., 2016a).
Along the same lines, it was shown that the addition of parameter noise leads to better exploration by
obtaining a policy that exhibits a larger variety of behaviors (Sun et al., 2009b; Salimans et al., 2017).
We discuss these related approaches in greater detail in Section 5. Their main limitation, however,
is that they are either only proposed and evaluated for the on-policy setting with relatively small
and shallow function approximators (Rückstieß et al., 2008) or disregard all temporal structure and
gradient information (Salimans et al., 2017; Kober & Peters, 2008; Sehnke et al., 2010).
This paper investigates how parameter space noise can be effectively combined with off-the-shelf deep
RL algorithms such as DQN (Mnih et al., 2015), DDPG (Lillicrap et al., 2015), and TRPO (Schulman
et al., 2015b) to improve their exploratory behavior. Experiments show that this form of exploration
is applicable to both high-dimensional discrete environments and continuous control tasks, using
on- and off-policy methods. Our results indicate that parameter noise outperforms traditional action
space noise-based baselines, especially in tasks where the reward signal is extremely sparse.

2

BACKGROUND

We consider the standard RL framework consisting of an agent interacting with an environment.
To simplify the exposition we assume that the environment is fully observable. An environment is
modeled as a Markov decision process (MDP) and is defined by a set of states S, a set of actions
A, a distribution over initial states p(s0 ), a reward function r : S × A 7→ R, transition probabilities
1

Published as a conference paper at ICLR 2018

p(st+1 |st , at ), a time horizon T , and a discount factor γ ∈ [0, 1). We denote by πθ a policy
parametrized by θ, which can be either deterministic, π : S 7→ A, or stochastic, π : S 7→ P(A). The
PT
agent’s goal is to maximize the expected discounted return η(πθ ) = Eτ [ t=0 γ t r(st , at )], where τ =
(s0 , a0 , . . . , sT ) denotes a trajectory with s0 ∼ p(s0 ), at ∼ πθ (at |st ), and st+1 ∼ p(st+1 |st , at ).
PT
Experimental evaluation is based on the undiscounted return Eτ [ t=0 r(st , at )].1
2.1

O FF - POLICY M ETHODS

Off-policy RL methods allow learning based on data captured by arbitrary policies. This paper
considers two popular off-policy algorithms, namely Deep Q-Networks (DQN, Mnih et al. (2015))
and Deep Deterministic Policy Gradients (DDPG, Lillicrap et al. (2015)).
Deep Q-Networks (DQN) DQN uses a deep neural network as a function approximator to estimate
the optimal Q-value function, which conforms to the Bellman optimality equation:
Q(st , at ) = r(st , at ) + γ max
Q(st+1 , a0 ).
0
a ∈A

The policy is implicitly defined by Q as π(st ) = argmaxa0 ∈A Q(st , a0 ). Typically, a stochastic greedy or Boltzmann policy (Sutton & Barto, 1998) is derived from the Q-value function to encourage
exploration, which relies on sampling noise in the action space. The Q-network predicts a Q-value
for each action and is updated using off-policy data from a replay buffer.
Deep Deterministic Policy Gradients (DDPG) DDPG is an actor-critic algorithm, applicable to
continuous action spaces. Similar to DQN, the critic estimates the Q-value function using off-policy
data and the recursive Bellman equation:
Q(st , at ) = r(st , at ) + γQ (st+1 , πθ (st+1 )),
where πθ is the actor or policy. The actor is trained to maximize the critic’s estimated Q-values by
back-propagating through both networks. For exploration, DDPG uses a stochastic policy of the
form π
cθ (st ) = πθ (st ) + w, where w is either w ∼ N (0, σ 2 I) (uncorrelated) or w ∼ OU(0, σ 2 )
(correlated).2 Again, exploration is realized through action space noise.
2.2

O N - POLICY M ETHODS

In contrast to off-policy algorithms, on-policy methods require updating function approximators
according to the currently followed policy. In particular, we will consider Trust Region Policy
Optimization (TRPO, Schulman et al. (2015a)), an extension of traditional policy gradient methods
(Williams, 1992b) using the natural gradient direction (Peters & Schaal, 2008; Kakade, 2001).
Trust Region Policy Optimization (TRPO) TRPO improves upon REINFORCE (Williams,
1992b) by computing an ascent direction that ensures a small change in the policy distribution.
More specifically, TRPO solves the following constrained optimization problem:


πθ (a|s)
maximizeθ Es∼ρθ0 ,a∼πθ0
A(s,
a)
πθ0 (a|s)
s.t.
Es∼ρθ0 [DKL (πθ0 (·|s)kπθ (·|s))] ≤ δKL
where ρθ = ρπθ is the discounted state-visitation frequencies induced by πθ , A(s, a) denotes the
advantage function estimated by the empirical return minus the baseline, and δKL is a step size
parameter which controls how much the policy is allowed to change per iteration.

3

PARAMETER S PACE N OISE FOR E XPLORATION

This work considers policies that are realized as parameterized functions, which we denote as πθ ,
with θ being the parameter vector. We represent policies as neural networks but our technique can
1
If t = T , we write r(sT , aT ) to denote the terminal reward, even though it has has no dependence on aT ,
to simplify notation.
2
OU(·, ·) denotes the Ornstein-Uhlenbeck process (Uhlenbeck & Ornstein, 1930).

2

Published as a conference paper at ICLR 2018

be applied to arbitrary parametric models. To achieve structured exploration, we sample from a
set of policies by applying additive Gaussian noise to the parameter vector of the current policy:
θe = θ + N (0, σ 2 I). Importantly, the perturbed policy is sampled at the beginning of each episode
and kept fixed for the entire rollout. For convenience and readability, we denote this perturbed policy
as π
e := πθe and analogously define π := πθ .
State-dependent exploration As pointed out by Rückstieß et al. (2008), there is a crucial difference
between action space noise and parameter space noise. Consider the continuous action space
case. When using Gaussian action noise, actions are sampled according to some stochastic policy,
generating at = π(st ) + N (0, σ 2 I). Therefore, even for a fixed state s, we will almost certainly
obtain a different action whenever that state is sampled again in the rollout, since action space noise
is completely independent of the current state st (notice that this is equally true for correlated action
space noise). In contrast, if the parameters of the policy are perturbed at the beginning of each
episode, we get at = π
e(st ). In this case, the same action will be taken every time the same state st is
sampled in the rollout. This ensures consistency in actions, and directly introduces a dependence
between the state and the exploratory action taken.
Perturbing deep neural networks It is not immediately obvious that deep neural networks, with
potentially millions of parameters and complicated nonlinear interactions, can be perturbed in
meaningful ways by applying spherical Gaussian noise. However, as recently shown by Salimans
et al. (2017), a simple reparameterization of the network achieves exactly this. More concretely, we
use layer normalization (Ba et al., 2016) between perturbed layers.3 Due to this normalizing across
activations within a layer, the same perturbation scale can be used across all layers, even though
different layers may exhibit different sensitivities to noise.
Adaptive noise scaling Parameter space noise requires us to pick a suitable scale σ. This can be
problematic since the scale will strongly depend on the specific network architecture, and is likely to
vary over time as parameters become more sensitive to noise as learning progresses. Additionally,
while it is easy to intuitively grasp the scale of action space noise, it is far harder to understand the
scale in parameter space. We propose a simple solution that resolves all aforementioned limitations
in an easy and straightforward way. This is achieved by adapting the scale of the parameter space
noise over time and relating it to the variance in action space that it induces. More concretely, we can
define a distance measure between perturbed and non-perturbed policy in action space and adaptively
increase or decrease the parameter space noise depending on whether it is below or above a certain
threshold:

ασ
if d(π, π
e) ≤ δ,
σk+1 = 1 k
(1)
otherwise,
α σk
where α ∈ R>0 is a scaling factor and δ ∈ R>0 a threshold value. The concrete realization of d(·, ·)
depends on the algorithm at hand and we describe appropriate distance measures for DQN, DDPG,
and TRPO in Appendix C.
Parameter space noise for off-policy methods In the off-policy case, parameter space noise can
be applied straightforwardly since, by definition, data that was collected off-policy can be used. More
concretely, we only perturb the policy for exploration and train the non-perturbed network on this
data by replaying it.
Parameter space noise for on-policy methods Parameter noise can be incorporated in an onpolicy setting, using an adapted policy gradient, as set forth by Rückstieß et al. (2008). Policy
gradient methods optimize Eτ ∼(π,p) [R(τ )]. Given a stochastic policy πθ (a|s) with θ ∼ N (φ, Σ),
the expected return can be expanded using likelihood ratios and the re-parametrization trick (Kingma
& Welling, 2013) as
"T −1
#
1 X X
i 12
i
∇φ,Σ Eτ [R(τ )] ≈
∇φ,Σ log π(at |st ; φ +  Σ )Rt (τ )
(2)
N i i t=0
 ,τ

3
This is in contrast to Salimans et al. (2017), who use virtual batch normalization, which we found to perform
less consistently

3

Published as a conference paper at ICLR 2018

for N samples i ∼ N (0, I) and τ i ∼ (π

1

φ+i Σ 2

, p) (see Appendix B for a full derivation). Rather

than updating Σ according to the previously derived policy gradient, we fix its value to σ 2 I and scale
it adaptively as described in Appendix C.

4

E XPERIMENTS

This section answers the following questions:
(i) Do existing state-of-the-art RL algorithms benefit from incorporating parameter space noise?
(ii) Does parameter space noise aid in exploring sparse reward environments more effectively?
(iii) How does parameter space noise exploration compare against evolution strategies for deep
policies (Salimans et al., 2017) with respect to sample efficiency?
Reference implementations of DQN and DDPG with adaptive parameter space noise are available
online.4
4.1

C OMPARING PARAMETER S PACE N OISE TO ACTION S PACE N OISE

The added value of parameter space noise over action space noise is measured on both highdimensional discrete-action environments and continuous control tasks. For the discrete environments,
comparisons are made using DQN, while DDPG and TRPO are used on the continuous control tasks.
Discrete-action environments For discrete-action environments, we use the Arcade Learning
Environment (ALE, Bellemare et al. (2013)) benchmark along with a standard DQN implementation.
We compare a baseline DQN agent with -greedy action noise against a version of DQN with
parameter noise. We linearly anneal  from 1.0 to 0.1 over the first 1 million timesteps. For parameter
noise, we adapt the scale using a simple heuristic that increases the scale if the KL divergence between
perturbed and non-perturbed policy is less than the KL divergence between greedy and -greedy
policy and decreases it otherwise (see Section C.1 for details). By using this approach, we achieve a
fair comparison between action space noise and parameter space noise since the magnitude of the
noise is similar and also avoid the introduction of an additional hyperparameter.
For parameter perturbation, we found it useful to reparametrize the network in terms of an explicit
policy that represents the greedy policy π implied by the Q-values, rather than perturbing the Qfunction directly. To represent the policy π(a|s), we add a single fully connected layer after the
convolutional part of the network, followed by a softmax output layer. Thus, π predicts a discrete
probability distribution over actions, given a state. We find that perturbing π instead of Q results
in more meaningful changes since we now define an explicit behavioral policy. In this setting, the
Q-network is trained according to standard DQN practices. The policy π is trained by maximizing
the probability of outputting the greedy action accordingly to the current Q-network. Essentially, the
policy is trained to exhibit the same behavior as running greedy DQN. To rule out this double-headed
version of DQN alone exhibits significantly different behavior, we always compare our parameter
space noise approach against two baselines, regular DQN and two-headed DQN, both with -greedy
exploration.
We furthermore randomly sample actions for the first 50 thousand timesteps in all cases to fill the
replay buffer before starting training. Moreover, we found that parameter space noise performs better
if it is combined with a bit of action space noise (we use a -greedy behavioral policy with  = 0.01
for the parameter space noise experiments). Full experimental details are described in Section A.1.
We chose 21 games of varying complexity, according to the taxonomy presented by (Bellemare et al.,
2016). The learning curves are shown in Figure 1 for a selection of games (see Appendix D for full
results). Each agent is trained for 40 M frames. The overall performance is estimated by running
each configuration with three different random seeds, and we plot the median return (line) as well as
the interquartile range (shaded area). Note that performance is evaluated on the exploratory policy
since we are interested in its behavior especially.
4

https://github.com/openai/baselines

4

Published as a conference paper at ICLR 2018

parameter noise, separate policy head

Alien

Amidar

-greedy, separate policy head

BeamRider
7500

300

1000

200

400

5000

200

500

100

200

2500

100

0

0

0

1500

return

Breakout

10000

400
300

Enduro

600

Freeway

30

Pong

1000

10

0

0

250

WizardOfWor

return

4000
2000
0
0

Zaxxon

1

2

steps

3

4
1e7

0

1

2

steps

3

4
1e7

8000

150

2000

6000

100

4000
1000

50
0

0
10
20

Tutankham

200

6000

10

500

500

Qbert

20

750

20

1000

0

Frostbite

1500

return

-greedy

BankHeist

0

1

2

steps

3

4
1e7

0

2000
0
0

1

2

steps

3

4
1e7

0

1

2

steps

3

4
1e7

Figure 1: Median DQN returns for several ALE environment plotted over training steps.

Overall, our results show that parameter space noise often outperforms action space noise, especially
on games that require consistency (e.g. Enduro, Freeway) and performs comparably on the remaining
ones. Additionally, learning progress usually starts much sooner when using parameter space noise.
Finally, we also compare against a double-headed version of DQN with -greedy exploration to
ensure that this change in architecture is not responsible for improved exploration, which our results
confirm. Full results are available in Appendix D.
That being said, parameter space noise is unable to sufficiently explore in extremely challenging
games like Montezuma’s Revenge. More sophisticated exploration methods like Bellemare et al.
(2016) are likely necessary to successfully learn these games. However, such methods often rely on
some form of “inner” exploration method, which is usually traditional action space noise. It would be
interesting to evaluate the effect of parameter space noise when combined with exploration methods.
On a final note, proposed improvements to DQN like double DQN (Hasselt, 2010), prioritized
experience replay (Schaul et al., 2015), and dueling networks (Wang et al., 2015) are orthogonal to
our improvements and would therefore likely improve results further. We leave the experimental
validation of this theory to future work.
Continuous control environments We now compare parameter noise with action noise on the
continuous control environments implemented in OpenAI Gym (Brockman et al., 2016). We use
DDPG (Lillicrap et al., 2015) as the RL algorithm for all environments with similar hyperparameters
as outlined in the original paper except for the fact that layer normalization (Ba et al., 2016) is applied
after each layer before the nonlinearity, which we found to be useful in either case and especially
important for parameter space noise.
We compare the performance of the following configurations: (a) no noise at all, (b) uncorrelated
additive Gaussian action space noise (σ = 0.2), (c) correlated additive Gaussian action space
noise (Ornstein–Uhlenbeck process (Uhlenbeck & Ornstein, 1930) with σ = 0.2), and (d) adaptive
parameter space noise. In the case of parameter space noise, we adapt the scale so that the resulting
change in action space is comparable to our baselines with uncorrelated Gaussian action space noise
(see Section C.2 for full details).
We evaluate the performance on several continuous control tasks. Figure 2 depicts the results for
three exemplary environments. Each agent is trained for 1 M timesteps, where 1 epoch consists of
10 thousand timesteps. In order to make results comparable between configurations, we evaluate the
performance of the agent every 10 thousand steps by using no noise for 20 episodes.
On HalfCheetah, parameter space noise achieves significantly higher returns than all other configurations. We find that, in this environment, all other exploration schemes quickly converge to a local
optimum (in which the agent learns to flip on its back and then “wiggles” its way forward). Parameter
5

Published as a conference paper at ICLR 2018

adaptive parameter noise

correlated action noise

HalfCheetah

uncorrelated action noise

no noise

Hopper

Walker2d

1500
5000

1250

4000

return

2500
2000

1000

3000

750

1500

1000

500

1000

0

250

500

2000

20

40

60

80

100

20

40

60

epoch

80

100

20

40

60

epoch

80

100

epoch

Figure 2: Median DDPG returns for continuous control environments plotted over epochs.
space noise behaves similarly initially but still explores other options and quickly learns to break out
of this sub-optimal behavior. Also notice that parameter space noise vastly outperforms correlated
action space noise on this environment, clearly indicating that there is a significant difference between
the two. On the remaining two environments, parameter space noise performs on par with other
exploration strategies. Notice, however, that even if no noise is present, DDPG is capable of learning
good policies. We find that this is representative for the remaining environments (see Appendix E for
full results), which indicates that these environments do not require a lot of exploration to begin with
due to their well-shaped reward function.
TRPO with parameter noise ( = 0.01)

TRPO with parameter noise ( = 0.1)

HalfCheetah
5000

3000
2500

2000

3000

2000

1500

2000
1000
0

1500

1000

1000

500

500
0

0
0

2000

4000

6000

epoch

8000

10000

TRPO

Walker2D

2500

4000

return

TRPO with parameter noise ( = 1.0)

Hopper

0

2000

4000

6000

8000

10000

epoch

0

2000

4000

6000

8000

10000

epoch

Figure 3: Median TRPO returns for continuous control environments plotted over epochs.
The results for TRPO are depicted in Figure 3. Interestingly, in the Walker2D environment, we see
that adding parameter noise decreases the performance variance between seeds. This indicates that
parameter noise aids in escaping local optima.
4.2

D OES PARAMETER S PACE N OISE E XPLORE E FFICIENTLY ?

The environments in the previous section required relatively little exploration. In this section, we
evaluate whether parameter noise enables existing RL algorithms to learn on environments with very
sparse rewards, where uncorrelated action noise generally fails (Osband et al., 2016a; Achiam &
Sastry, 2017).
A scalable toy example We first evaluate parameter noise on a well-known toy problem, following
the setup described by Osband et al. (2016a) as closely as possible. The environment consists of a
chain of N states and the agent always starts in state s2 , from where it can either move left or right.
In state s1 , the agent receives a small reward of r = 0.001 and a larger reward r = 1 in state sN .
Obviously, it is much easier to discover the small reward in s1 than the large reward in sN , with
increasing difficulty as N grows. The environment is described in greater detail in Section A.3.
We compare adaptive parameter space noise DQN, bootstrapped DQN, and -greedy DQN. The
chain length N is varied and for each N three different seeds are trained and evaluated. After each
episode, we evaluate the performance of the current policy by performing a rollout with all noise
disabled (in the case of bootstrapped DQN, we perform majority voting over all heads). The problem
is considered solved if one hundred subsequent rollouts achieve the optimal return. We plot the
median number of episodes before the problem is considered solved (we abort if the problem is still
unsolved after 2 thousand episodes). Full experimental details are available in Section A.3.
6

Published as a conference paper at ICLR 2018

Parameter space noise DQN

Bootstrapped DQN

-greedy DQN

number of episodes

2000
1500
1000
500
0

20

40

60

80

100

20

40

chain length

60

80

100

20

40

chain length

60

80

100

chain length

Figure 4: Median number of episodes before considered solved for DQN with different exploration
strategies. Green indicates that the problem was solved whereas blue indicates that no solution was
found within 2 K episodes. Note that less number of episodes before solved is better.
Figure 4 shows that parameter space noise clearly outperforms action space noise (which completely
fails for moderately large N ) and even outperforms the more computational expensive bootstrapped
DQN. However, it is important to note that this environment is extremely simple in the sense that the
optimal strategy is to always go right. In a case where the agent needs to select a different optimal
action depending on the current state, parameter space noise would likely work less well since weight
randomization of the policy is less likely to yield this behavior. Our results thus only highlight the
difference in exploration behavior compared to action space noise in this specific case. In the general
case, parameter space noise does not guarantee optimal exploration.
Continuous control with sparse rewards We now make the continuous control environments
more challenging for exploration. Instead of providing a reward at every timestep, we use environments that only yield a non-zero reward after significant progress towards a goal. More concretely,
we consider the following environments from rllab5 (Duan et al., 2016), modified according to
Houthooft et al. (2016): (a) SparseCartpoleSwingup, which only yields a reward if the paddle is
raised above a given threshold, (b) SparseDoublePendulum, which only yields a reward if the agent
reaches the upright position, and (c) SparseHalfCheetah, which only yields a reward if the agent
crosses a target distance, (d) SparseMountainCar, which only yields a reward if the agent drives up
the hill, (e) SwimmerGather, yields a positive or negative reward upon reaching targets. For all tasks,
we use a time horizon of T = 500 steps before resetting.
adaptive parameter noise

correlated action noise

SparseCartpoleSwingup

uncorrelated action noise

no noise

SparseDoublePendulum

SparseHalfCheetah

80

0.6

300

return

60
40

200

20

100

0.4
0.2

0

0.0
20

SparseMountainCar

40

60

80

100

epoch

SparseSwimmerGather

1.0
0.04
0.8

return

0.02
0.6
0.00
0.4
0.02
0.2
0.04
0.0
20

40

60

80

100

20

epoch

40

60

80

100

epoch

Figure 5: Median DDPG returns for environments with sparse rewards plotted over epochs.
We consider both DDPG and TRPO to solve these environments (the exact experimental setup is
described in Section A.2). Figure 5 shows the performance of DDPG, while the results for TRPO have
been moved to Appendix F. The overall performance is estimated by running each configuration with
5

https://github.com/openai/rllab

7

Published as a conference paper at ICLR 2018

five different random seeds, after which we plot the median return (line) as well as the interquartile
range (shaded area).
For DDPG, SparseDoublePendulum seems to be easy to solve in general, with even no noise finding a
successful policy relatively quickly. The results for SparseCartpoleSwingup and SparseMountainCar
are more interesting: Here, only parameter space noise is capable of learning successful policies
since all other forms of noise, including correlated action space noise, never find states with nonzero rewards. For SparseHalfCheetah, DDPG at least finds the non-zero reward but never learns a
successful policy from that signal. On the challenging SwimmerGather task, all configurations of
DDPG fail.
Our results clearly show that parameter space noise can be used to improve the exploration behavior
of these off-the-shelf algorithms. However, it is important to note that improvements in exploration
are not guaranteed for the general case. It is therefore necessary to evaluate the potential benefit of
parameter space noise on a case-by-case basis.
4.3

I S RL WITH PARAMETER S PACE N OISE MORE S AMPLE - EFFICIENT THAN ES?

Evolution strategies (ES) are closely related to our approach since both explore by introducing
noise in the parameter space, which can lead to improved exploration behavior (Salimans et al.,
2017).6 However, ES disregards temporal information and uses black-box optimization to train the
neural network. By combining parameter space noise with traditional RL algorithms, we can include
temporal information as well rely on gradients computed by back-propagation for optimization while
still benefiting from improved exploratory behavior. We now compare ES and traditional RL with
parameter space noise directly.
We compare performance on the 21 ALE games that were used in Section 4.1. The performance
is estimated by running 10 episodes for each seed using the final policy with exploration disabled
and computing the median returns. For ES, we use the results obtained by Salimans et al. (2017),
which were obtained after training on 1 000 M frames. For DQN, we use the same parameter space
noise for exploration that was previously described and train on 40 M frames. Even though DQN
with parameter space noise has been exposed to 25 times less data, it outperforms ES on 15 out of
21 Atari games (full results are available in Appendix D). Combined with the previously described
results, this demonstrates that parameter space noise combines the desirable exploration properties of
ES with the sample efficiency of traditional RL.

5

R ELATED W ORK

The problem of exploration in reinforcement has been studied extensively. A range of algorithms (Kearns & Singh, 2002; Brafman & Tennenholtz, 2002; Auer et al., 2008) have been proposed
that guarantee near-optimal solutions after a number of steps that are polynomial in the number
of states, number of actions, and the horizon time. However, in many real-world reinforcements
learning problems both the state and action space are continuous and high dimensional so that,
even with discretization, these algorithms become impractical. In the context of deep reinforcement
learning, a large variety of techniques have been proposed to improve exploration (Stadie et al., 2015;
Houthooft et al., 2016; Tang et al., 2016; Osband et al., 2016a; Ostrovski et al., 2017; Sukhbaatar et al.,
2017; Osband et al., 2016b). However, all are non-trivial to implement and are often computational
expensive.
The idea of perturbing the parameters of a policy has been proposed by Rückstieß et al. (2008) for
policy gradient methods. The authors show that this form of perturbation generally outperforms random exploration and evaluate their exploration strategy with the REINFORCE (Williams, 1992a) and
Natural Actor-Critic (Peters & Schaal, 2008) algorithms. However, their policies are relatively lowdimensional compared to modern deep architectures, they use environments with low-dimensional
state spaces, and their contribution is strictly limited to the policy gradient case. In contrast, our
6
To clarify, when we refer to ES in this context, we refer to the recent work by Salimans et al. (2017), which
demonstrates that deep policy networks that learn from pixels can be trained using ES. We understand that there
is a vast body of other work in this field (compare section 5).

8

Published as a conference paper at ICLR 2018

method is applied and evaluated for both on and off-policy setting, we use high-dimensional policies,
and environments with large state spaces.
Our work is also closely related to evolution strategies (ES, Rechenberg & Eigen (1973); Schwefel
(1977)), and especially neural evolution strategies (NES, Sun et al. (2009a;b); Glasmachers et al.
(2010a;b); Schaul et al. (2011); Wierstra et al. (2014)). In the context of policy optimization, our
work is closely related to Kober & Peters (2008) and Sehnke et al. (2010). More recently, Salimans
et al. (2017) showed that ES can work for high-dimensional environments like Atari and OpenAI
Gym continuous control problems. However, ES generally disregards any temporal structure that
may be present in trajectories and typically suffers from sample inefficiency.
Bootstrapped DQN (Osband et al., 2016a) has been proposed to aid with more directed and consistent
exploration by using a network with multiple heads, where one specific head is selected at the
beginning of each episode. In contrast, our approach perturbs the parameters of the network directly,
thus achieving similar yet simpler (and as shown in Section 4.2, sometimes superior) exploration
behavior. Concurrently to our work, Fortunato et al. (2017) have proposed a similar approach that
utilizes parameter perturbations for more efficient exploration.

6

C ONCLUSION

In this work, we propose parameter space noise as a conceptually simple yet effective replacement
for traditional action space noise like -greedy and additive Gaussian noise. This work shows that
parameter perturbations can successfully be combined with contemporary on- and off-policy deep RL
algorithms such as DQN, DDPG, and TRPO and often results in improved performance compared
to action noise. Experimental results further demonstrate that using parameter noise allows solving
environments with very sparse rewards, in which action noise is unlikely to succeed. Our results
indicate that parameter space noise is a viable and interesting alternative to action space noise, which
is still the de facto standard in most reinforcement learning applications.

R EFERENCES
Joshua Achiam and Shankar Sastry. Surprise-based intrinsic motivation for deep reinforcement learning. arXiv
preprint arXiv:1703.01732, 2017.
Peter Auer, Thomas Jaksch, and Ronald Ortner.
Near-optimal regret bounds for
reinforcement learning.
In Advances in Neural Information Processing Systems 21 (NIPS), pp. 89–96,
2008.
URL http://papers.nips.cc/paper/
3401-near-optimal-regret-bounds-for-reinforcement-learning.
Lei Jimmy Ba, Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, abs/1607.06450, 2016. URL
http://arxiv.org/abs/1607.06450.
Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An
evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, 2013. doi:
10.1613/jair.3912. URL http://dx.doi.org/10.1613/jair.3912.
Marc G Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying
count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems 29
(NIPS), pp. 1471–1479, 2016.
Ronen I. Brafman and Moshe Tennenholtz. R-MAX - A general polynomial time algorithm for near-optimal
reinforcement learning. Journal of Machine Learning Research, 3:213–231, 2002. URL http://www.
jmlr.org/papers/v3/brafman02a.html.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech
Zaremba. OpenAI gym. arXiv preprint arXiv:1606.01540, 2016. URL http://arxiv.org/abs/1606.
01540.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement
learning for continous control. In Proceedings of the 33rd International Conference on Machine Learning
(ICML), pp. 1329–1338, 2016.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad
Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al. Noisy networks for exploration. arXiv preprint
arXiv:1706.10295, 2017.

9

Published as a conference paper at ICLR 2018

Tobias Glasmachers, Tom Schaul, and Jürgen Schmidhuber. A natural evolution strategy for multi-objective
optimization. In Parallel Problem Solving from Nature - PPSN XI, 11th International Conference, Kraków,
Poland, September 11-15, 2010, Proceedings, Part I, pp. 627–636, 2010a. doi: 10.1007/978-3-642-15844-5_
63. URL https://doi.org/10.1007/978-3-642-15844-5_63.
Tobias Glasmachers, Tom Schaul, Yi Sun, Daan Wierstra, and Jürgen Schmidhuber. Exponential natural
evolution strategies. In Genetic and Evolutionary Computation Conference, GECCO 2010, Proceedings,
Portland, Oregon, USA, July 7-11, 2010, pp. 393–400, 2010b. doi: 10.1145/1830483.1830557. URL
http://doi.acm.org/10.1145/1830483.1830557.
Hado V Hasselt. Double Q-learning. In Advances in Neural Information Processing Systems 23 (NIPS), pp.
2613–2621, 2010.
Rein Houthooft, Xi Chen, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel.
VIME: Variational information maximizing exploration. In Advances in Neural Information Processing Systems 29 (NIPS), pp. 1109–1117, 2016. URL http://papers.nips.cc/paper/
6591-vime-variational-information-maximizing-exploration.
Sham Kakade. A natural policy gradient. Advances in neural information processing systems, 14:1531–1538,
2001.
Michael J. Kearns and Satinder P. Singh. Near-optimal reinforcement learning in polynomial time. Machine
Learning, 49(2-3):209–232, 2002. doi: 10.1023/A:1017984413808. URL http://dx.doi.org/10.
1023/A:1017984413808.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the
International Conference on Learning Representations (ICLR), 2015.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Jens Kober and Jan Peters. Policy search for motor primitives in robotics. In Advances in Neural Information Processing Systems 21 (NIPS), pp. 849–856, 2008. URL http://papers.nips.cc/paper/
3545-policy-search-for-motor-primitives-in-robotics.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver,
and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, abs/1509.02971, 2015.
URL http://arxiv.org/abs/1509.02971.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex
Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir
Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis
Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015. doi:
10.1038/nature14236. URL http://dx.doi.org/10.1038/nature14236.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped
DQN. In Advances in Neural Information Processing Systems 29 (NIPS), pp. 4026–4034, 2016a. URL
http://papers.nips.cc/paper/6501-deep-exploration-via-bootstrapped-dqn.
Ian Osband, Benjamin Van Roy, and Zheng Wen. Generalization and exploration via randomized value functions.
In Proceedings of the 33nd International Conference on Machine Learning, ICML, pp. 2377–2386, 2016b.
URL http://jmlr.org/proceedings/papers/v48/osband16.html.
Georg Ostrovski, Marc G. Bellemare, Aäron van den Oord, and Rémi Munos. Count-based exploration with
neural density models. arXiv preprint arXiv:1703.01310, 2017. URL http://arxiv.org/abs/1703.
01310.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by selfsupervised prediction. In ICML, 2017.
Jan Peters and Stefan Schaal. Natural actor-critic. Neurocomputing, 71(7-9):1180–1190, 2008. doi: 10.1016/j.
neucom.2007.11.026. URL http://dx.doi.org/10.1016/j.neucom.2007.11.026.
Ananth Ranganathan. The Levenberg-Marquardt algorithm. Tutoral on LM algorithm, pp. 1–5, 2004.
Ingo Rechenberg and Manfred Eigen. Evolutionsstrategie: Optimierung technischer Systeme nach Prinzipien
der biologishen Evolution. Frommann-Holzboog Stuttgart, 1973.

10

Published as a conference paper at ICLR 2018

Thomas Rückstieß, Martin Felder, and Jürgen Schmidhuber. State-dependent exploration for policy gradient
methods. In Proceedings of the European Conference on Machine Learning and Knowledge Discovery
in Databases ECML/PKDD, pp. 234–249, 2008. doi: 10.1007/978-3-540-87481-2_16. URL http:
//dx.doi.org/10.1007/978-3-540-87481-2_16.
Thomas Rückstieß, Martin Felder, and Jürgen Schmidhuber. State-dependent exploration for policy gradient
methods. Machine Learning and Knowledge Discovery in Databases, pp. 234–249, 2008.
Tim Salimans, Jonathan Ho, Xi Chen, and Ilya Sutskever. Evolution strategies as a scalable alternative to
reinforcement learning. arXiv preprint arXiv:1703.03864, 2017. URL http://arxiv.org/abs/1703.
03864.
Tom Schaul, Tobias Glasmachers, and Jürgen Schmidhuber. High dimensions and heavy tails for natural
evolution strategies. In 13th Annual Genetic and Evolutionary Computation Conference, GECCO 2011,
Proceedings, Dublin, Ireland, July 12-16, 2011, pp. 845–852, 2011. doi: 10.1145/2001576.2001692. URL
http://doi.acm.org/10.1145/2001576.2001692.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint
arXiv:1511.05952, 2015.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy
optimization. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pp.
1889–1897, 2015a.
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy
optimization. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille,
France, 6-11 July 2015, pp. 1889–1897, 2015b. URL http://jmlr.org/proceedings/papers/
v37/schulman15.html.
Hans-Paul Schwefel. Numerische Optimierung von Computermodellen mittels der Evolutionsstrategie, volume 1.
Birkhäuser, Basel Switzerland, 1977.
Frank Sehnke, Christian Osendorfer, Thomas Rückstieß, Alex Graves, Jan Peters, and Jürgen Schmidhuber.
Parameter-exploring policy gradients. Neural Networks, 23(4):551–559, 2010. doi: 10.1016/j.neunet.2009.12.
004. URL http://dx.doi.org/10.1016/j.neunet.2009.12.004.
Bradly C. Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with
deep predictive models. arXiv preprint arXiv:1507.00814, 2015. URL http://arxiv.org/abs/1507.
00814.
Sainbayar Sukhbaatar, Ilya Kostrikov, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic
curricula via asymmetric self-play. arXiv preprint arXiv:1703.05407, 2017. URL http://arxiv.org/
abs/1703.05407.
Yi Sun, Daan Wierstra, Tom Schaul, and Jürgen Schmidhuber. Stochastic search using the natural gradient.
In Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009, Montreal,
Quebec, Canada, June 14-18, 2009, pp. 1161–1168, 2009a. doi: 10.1145/1553374.1553522. URL http:
//doi.acm.org/10.1145/1553374.1553522.
Yi Sun, Daan Wierstra, Tom Schaul, and Jürgen Schmidhuber. Efficient natural evolution strategies. In Genetic
and Evolutionary Computation Conference, GECCO 2009, Proceedings, Montreal, Québec, Canada, July
8-12, 2009, pp. 539–546, 2009b. doi: 10.1145/1569901.1569976. URL http://doi.acm.org/10.
1145/1569901.1569976.
Richard S Sutton and Andrew G Barto. Introduction to reinforcement learning, volume 135. MIT Press
Cambridge, 1998.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck,
and Pieter Abbeel. #Exploration: A study of count-based exploration for deep reinforcement learning. arXiv
preprint arXiv:1611.04717, 2016.
George E Uhlenbeck and Leonard S Ornstein. On the theory of the brownian motion. Physical review, 36(5):
823, 1930.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Marc Lanctot, and Nando de Freitas. Dueling
network architectures for deep reinforcement learning. arXiv preprint arXiv:1511.06581, 2015.

11

Published as a conference paper at ICLR 2018

Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. Natural evolution
strategies. Journal of Machine Learning Research, 15(1):949–980, 2014. URL http://dl.acm.org/
citation.cfm?id=2638566.
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning.
Machine Learning, 8:229–256, 1992a. doi: 10.1007/BF00992696. URL http://dx.doi.org/10.
1007/BF00992696.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning.
Machine learning, 8(3-4):229–256, 1992b.

A
A.1

E XPERIMENTAL S ETUP
A RCADE L EARNING E NVIRONMENT (ALE)

For ALE (Bellemare et al., 2013), the network architecture as described in Mnih et al. (2015) is used.
This consists of 3 convolutional layers (32 filters of size 8 × 8 and stride 4, 64 filters of size 4 × 4
and stride 2, 64 filters of size 3 × 3 and stride 1) followed by 1 hidden layer with 512 units followed
by a linear output layer with one unit for each action. ReLUs are used in each layer, while layer
normalization (Ba et al., 2016) is used in the fully connected part of the network. For parameter space
noise, we also include a second head after the convolutional stack of layers. This head determines
a policy network with the same architecture as the Q-value network, except for a softmax output
layer. The target networks are updated every 10 K timesteps. The Q-value network is trained using
the Adam optimizer (Kingma & Ba, 2015) with a learning rate of 10−4 and a batch size of 32. The
replay buffer can hold 1 M state transitions. For the -greedy baseline, we linearly anneal  from 1 to
0.1 over the first 1 M timesteps. For parameter space noise, we adaptively scale the noise to have a
similar effect in action space (see Section C.1 for details), effectively ensuring that the maximum KL
divergence between perturbed and non-perturbed π is softly enforced. The policy is perturbed at the
beginning of each episode and the standard deviation is adapted as described in Appendix C every 50
timesteps. Notice that we only perturb the policy head after the convolutional part of the network
(i.e. the fully connected part, which is also why we only include layer normalization in this part of
the network). To avoid getting stuck (which can potentially happen for a perturbed policy), we also
use -greedy action selection with  = 0.01. In all cases, we perform 50 K random actions to collect
initial data for the replay buffer before training starts. We set γ = 0.99, clip rewards to be in [−1, 1],
and clip gradients for the output layer of Q to be within [−1, 1]. For observations, each frame is
down-sampled to 84 × 84 pixels, after which it is converted to grayscale. The actual observation to
the network consists of a concatenation of 4 subsequent frames. Additionally, we use up to 30 noop
actions at the beginning of the episode. This setup is identical to what is described by Mnih et al.
(2015).
A.2

C ONTINUOUS C ONTROL

For DDPG, we use a similar network architecture as described by Lillicrap et al. (2015): both the
actor and critic use 2 hidden layers with 64 ReLU units each. For the critic, actions are not included
until the second hidden layer. Layer normalization (Ba et al., 2016) is applied to all layers. The
target networks are soft-updated with τ = 0.001. The critic is trained with a learning rate of 10−3
while the actor uses a learning rate of 10−4 . Both actor and critic are updated using the Adam
optimizer (Kingma & Ba, 2015) with batch sizes of 128. The critic is regularized using an L2 penalty
with 10−2 . The replay buffer holds 100 K state transitions and γ = 0.99 is used. Each observation
dimension is normalized by an online estimate of the mean and variance. For parameter space noise
with DDPG, we adaptively scale the noise to be comparable to the respective action space noise (see
Section C.2). For dense environments, we use action space noise with σ = 0.2 (and a comparable
adaptive noise scale). Sparse environments use an action space noise with σ = 0.6 (and a comparable
adaptive noise scale).
TRPO uses a step size of δKL = 0.01, a policy network of 2 hidden layers with 32 tanh units for the
nonlocomotion tasks, and 2 hidden layers of 64 tanh units for the locomotion tasks. The Hessian
calculation is subsampled with a factor of 0.1, γ = 0.99, and the batch size per epoch is set to
5 K timesteps. The baseline is a learned linear transformation of the observations.
12

Published as a conference paper at ICLR 2018

The following environments from OpenAI Gym7 (Brockman et al., 2016) are used:
• HalfCheetah (S ⊂ R17 , A ⊂ R6 ),
• Hopper (S ⊂ R11 , A ⊂ R3 ),
• InvertedDoublePendulum (S ⊂ R11 , A ⊂ R),
• InvertedPendulum (S ⊂ R4 , A ⊂ R),
• Reacher (S ⊂ R11 , A ⊂ R2 ),
• Swimmer (S ⊂ R8 , A ⊂ R2 ), and
• Walker2D (S ⊂ R17 , A ⊂ R6 ).
For the sparse tasks, we use the following environments from rllab8 (Duan et al., 2016), modified as
described by Houthooft et al. (2016):
• SparseCartpoleSwingup (S ⊂ R4 , A ⊂ R), which only yields a reward if the paddle is
raised above a given threshold,
• SparseHalfCheetah (S ⊂ R17 , A ⊂ R6 ), which only yields a reward if the agent crosses a
distance threshold,
• SparseMountainCar (S ⊂ R2 , A ⊂ R), which only yields a reward if the agent drives up
the hill,
• SparseDoublePendulum (S ⊂ R6 , A ⊂ R), which only yields a reward if the agent reaches
the upright position, and
• SwimmerGather (S ⊂ R33 , A ⊂ R2 ), which yields a positive or negative reward upon
reaching targets.
A.3

C HAIN E NVIRONMENT

We follow the state encoding proposed by Osband et al. (2016a) and use φ(st ) = (1{x ≤ st }) as
the observation, where 1 denotes the indicator function. DQN is used with a very simple network
to approximate the Q-value function that consists of 2 hidden layers with 16 ReLU units. Layer
normalization (Ba et al., 2016) is used for all hidden layers before applying the nonlinearity. Each
agent is then trained for up to 2 K episodes. The chain length N is varied and for each N three
different seeds are trained and evaluated. After each episode, the performance of the current policy is
evaluated by sampling a trajectory with noise disabled (in the case of bootstrapped DQN, majority
voting over all heads is performed). The problem is considered solved if one hundred subsequent
trajectories achieve the optimal episode return. Figure 6 depicts the environment.

r = 0.001

s1

s2

...

sN −1

sN

r=1

Figure 6: Simple and scalable environment to test for exploratory behavior (Osband et al., 2016a).
We compare adaptive parameter space noise DQN, bootstrapped DQN (Osband et al., 2016a) (with
K = 20 heads and Bernoulli masking with p = 0.5), and -greedy DQN (with  linearly annealed
from 1.0 to 0.1 over the first one hundred episodes). For adaptive parameter space noise, we only
use a single head and perturb Q directly, which works well in this setting. Parameter space noise
is adaptively scaled so that δ ≈ 0.05. In all cases, γ = 0.999, the replay buffer holds 100 K state
transitions, learning starts after 5 initial episodes, the target network is updated every 100 timesteps,
and the network is trained using the Adam optimizer (Kingma & Ba, 2015) with a learning rate of
10−3 and a batch size of 32.
7
8

https://github.com/openai/gym
https://github.com/openai/rllab

13

Published as a conference paper at ICLR 2018

B

PARAMETER S PACE N OISE FOR O N - POLICY M ETHODS

Policy gradient methods optimize Eτ ∼(π,p) [R(τ )]. Given a stochastic policy πθ (a|s) with θ ∼
N (φ, Σ), the expected return can be expanded using likelihood ratios and the reparametrization
trick (Kingma & Welling, 2013) as
#
"
X
∇φ,Σ Eτ [R(τ )] = ∇φ,Σ Eθ∼N (φ,Σ)
p(τ |θ)R(τ )
(3)
τ

"

#
X

= E∼N (0,I) ∇φ,Σ

1
2

p(τ |φ + Σ )R(τ )

(4)

τ

= E∼N (0,I),τ

"T −1
X

#
1
2

∇φ,Σ log π(at |st ; φ + Σ )Rt (τ )

(5)

t=0

#
"T −1
1
1 X X
≈
∇φ,Σ log π(at |st ; φ + i Σ 2 )Rt (τ i )
N i i t=0

(6)

 ,τ

for N samples i ∼ N (0, I) and τ i ∼ (π

1

φ+i Σ 2

, p), with Rt (τ i ) =

PT

t0 =t

0

γ t −t rti0 . This also

allows us to subtract a variance-reducing baseline bit , leading to
#
"T −1
1
1 X X
∇φ,Σ Eτ [R(τ )] ≈
∇φ,Σ log π(at |st ; φ + i Σ 2 )(Rt (τ i ) − bit ) .
N i i t=0

(7)

 ,τ

In our case, we set Σ := σ 2 I and use our proposed adaption method to re-scale as appropriate.

C

A DAPTIVE S CALING

Parameter space noise requires us to pick a suitable scale σ. This can be problematic since the scale
will highly depend on the specific network architecture, and is likely to vary over time as parameters
become more sensitive as learning progresses. Additionally, while it is easy to intuitively grasp the
scale of action space noise, it is far harder to understand the scale in parameter space.
We propose a simple solution that resolves all aforementioned limitations in an easy and straightforward way. This is achieved by adapting the scale of the parameter space noise over time, thus
using a time-varying scale σk . Furthermore, σk is related to the action space variance that it induces,
and updated accordingly. Concretely, we use the following simple heuristic to update σk every K
timesteps:

ασ , if d(π, π
e) < δ
σk+1 = 1 k
(8)
α σk , otherwise,
where d(·, ·) denotes some distance between the non-perturbed and perturbed policy (thus measuring
in action space), α ∈ R>0 is used to rescale σk , and δ ∈ R>0 denotes some threshold value. This idea
is based on the Levenberg-Marquardt heuristic (Ranganathan, 2004). The concrete distance measure
and appropriate choice of δ depends on the policy representation. In the following sections, we
outline our choice of d(·, ·) for methods that do (DDPG and TRPO) and do not (DQN) use behavioral
policies. In our experiments, we always use α = 1.01.
C.1

A D ISTANCE M EASURE FOR DQN

For DQN, the policy is defined implicitly by the Q-value function. Unfortunately, this means that a
e has pitfalls. For example, assume that the perturbed policy
naïve distance measure between Q and Q
has only changed the bias of the final layer, thus adding a constant value to each action’s Q-value. In
e 2 would be nonzero, although the policies
this case, a naïve distance measure like the norm kQ − Qk
e respectively) are exactly equal. This equally applies to the case where
π and π
e (implied by Q and Q,
DQN as two heads, one for Q and one for π.
14

Published as a conference paper at ICLR 2018

We therefore use a probabilistic formulation9 for both the non-perturbed and perturbed
policies: π, π
e:SP
× A 7→ [0, 1] by applying the softmax function over predicted Q values:
π(s) = exp Qi (s)/ i exp Qi (s), where Qi (·) denotes the Q-value of the i-th action. π
e is defined
e instead (or the perturbed head for π). Using this probabilistic
analogously but uses the perturbed Q
formulation of the policies, we can now measure the distance in action space:
d(π, π
e) = DKL (π k π
e),

(9)

where DKL (· k ·) denotes the Kullback-Leibler (KL) divergence. This formulation effectively
normalizes the Q-values and therefore does not suffer from the problem previously outlined.
We can further relate this distance measure to -greedy action space noise, which allows us to fairly
compare the two approaches and also avoids the need to pick an additional hyperparameter δ. More
concretely, the KL divergence between a greedy policy π(s, a) = 1 for a = argmaxa0 Q(s, a0 ) and

π(s, a) = 0 otherwise and an -greedy policy π
b(s, a) = 1 −  + |A|
for a = argmaxa0 Q(s, a0 ) and


π
b(s, a) = |A|
otherwise is DKL (π k π
b) = − log (1 −  + |A|
), where |A| denotes the number of
actions (this follows immediately from the definition of the KL divergence for discrete probability
distributions). We can use this distance measure to relate action space noise and parameter space
noise to have similar distances, by adaptively scaling σ so that it matches the KL divergence between

greedy and -greedy policy, thus setting δ := − log (1 −  + |A|
).
C.2

A D ISTANCE M EASURE FOR DDPG

For DDPG, we relate noise induced by parameter space perturbations to noise induced by additive
Gaussian noise. To do so, we use the following distance measure between the non-perturbed and
perturbed policy:
v
u
N
h
i
u1 X
2
Es (π(s)i − π
d(π, π
e) = t
e(s)i ) ,
(10)
N i=1
where Es [·] is estimated from a batch of states from the replay buffer and N denotes the dimension of
the action space (i.e. A ⊂ RN ). It is easy to show that d(π, π + N (0, σ 2 I)) = σ. Setting δ := σ as
the adaptive parameter space threshold thus results in effective action space noise that has the same
standard deviation as regular Gaussian action space noise.
C.3

A D ISTANCE M EASURE FOR TRPO

In order to scale the noise for TRPO, we adapt the sampled noise vectors σ by computing a natural
step H −1 σ. We essentially compute a trust region around the noise direction to ensure that the
perturbed policy π
e remains sufficiently close to the non-perturbed version via
Es∼ρθe [DKL (πθe(·|s)kπθ (·|s))] ≤ δKL .
Concretely, this is computed through the conjugate gradient algorithm, combined with a line search
along the noise direction to ensure constraint conformation, as described in Appendix C of Schulman
et al. (2015b).

D

A DDITIONAL R ESULTS ON ALE

Figure 7 provide the learning curves for all 21 Atari games.
Table 1 compares the final performance of ES after 1 000 M frames to the final performance of DQN
with -greedy exploration and parameter space noise exploration after 40 M frames. In all cases, the
performance is estimated by running 10 episodes with exploration disabled. We use the numbers
reported by Salimans et al. (2017) for ES and report the median return across three seeds for DQN.

9
It is important to note that we use this probabilistic formulation only for the sake of defining a well-behaved
distance measure. The actual policy used for rollouts is still deterministic.

15

Published as a conference paper at ICLR 2018

parameter noise, separate policy head

Alien

Amidar

-greedy, separate policy head

BeamRider
7500

300

1000

200

400

5000

200

500

100

200

2500

100

0

0

0

return

300

Enduro

600

Freeway

30

Frostbite

return

750

20

1000
500

10

0

0

500

MontezumaRevenge

250

0.6

200

0.4

150

250

0.2

100
0.0

Pitfall

0

0

Gravitar

300

1000

1500

Pong

PrivateEye

Qbert

400
600

10

1000

6000

0

500

4000

5000

10

0

2000

2500

20

SpaceInvaders

2000

Tutankham

200

Venture

WizardOfWor

30

150

1000

1500

7500

0

0

Solaris

2500

Seaquest

10000

20

200

return

Breakout

10000

400
1500

return

-greedy

BankHeist

2000

20

100

1000

500
0

Zaxxon

1

2

steps

3

4
1e7

0

1000

10

50

500

0
0

1

2

steps

3

4
1e7

0

1

2

steps

3

4
1e7

0

0

1

2

steps

3

4
1e7

return

8000
6000
4000
2000
0
0

1

2

steps

3

4
1e7

Figure 7: Median DQN returns for all ALE environment plotted over training steps.

Table 1: Performance comparison between Evolution Strategies (ES) as reported by Salimans et al.
(2017), DQN with -greedy, and DQN with parameter space noise (this paper). ES was trained on
1 000 M, while DQN was trained on only 40 M frames.
Game
ES DQN w/ -greedy DQN w/ param noise
Alien
Amidar
BankHeist
BeamRider
Breakout
Enduro
Freeway
Frostbite
Gravitar
MontezumaRevenge
Pitfall
Pong
PrivateEye
Qbert
Seaquest
Solaris
SpaceInvaders
Tutankham
Venture
WizardOfWor
Zaxxon

994.0
112.0
225.0
744.0
9.5
95.0
31.0
370.0
805.0
0.0
0.0
21.0
100.0
147.5
1390.0
2090.0
678.5
130.3
760.0
3480.0
6380.0

1535.0
281.0
510.0
8184.0
406.0
1094
32.0
250.0
300.0
0.0
-73.0
21.0
133.0
7625.0
8335.0
720.0
1000.0
109.5
0
2350.0
8100.0

16

2070.0
403.5
805.0
7884.0
390.5
1672.5
31.5
1310.0
250.0
0.0
-100.0
20.0
100.0
7525.0
8920.0
400.0
1205.0
181.0
0
1850.0
8050.0

Published as a conference paper at ICLR 2018

E

A DDITIONAL R ESULTS ON C ONTINUOUS C ONTROL WITH S HAPED
R EWARDS

For completeness, we provide the plots for all evaluated environments with dense rewards. The
results are depicted in Figure 8.
adaptive parameter noise

correlated action noise

HalfCheetah

no noise

Hopper

InvertedDoublePendulum

1400

5000

8000

1200

4000

return

uncorrelated action noise

1000

3000

6000

800

2000

600

4000

1000

400

2000

0

200
0

InvertedPendulum

Reacher

Swimmer

5

1000

50

6

800

40

return

7
600

8

30

400

9

20

10

10

200

11

0

0
20

40

60

epoch

Walker2d

80

100

20

40

60

80

100

epoch

2500

return

2000
1500
1000
500
20

40

60

80

100

epoch

Figure 8: Median DDPG returns for all evaluated environments with dense rewards plotted over
epochs.
The results for InvertedPendulum and InvertedDoublePendulum are very noisy due to the fact
that a small change in policy can easily degrade performance significantly, and thus hard to read.
Interestingly, adaptive parameter space noise achieves the most stable performance on InvertedDoublePendulum. Overall, performance is comparable to other exploration approaches. Again, no
noise in either the action nor the parameter space achieves comparable results, indicating that these
environments combined with DDPG are not well-suited to test for exploration.

F

A DDITIONAL R ESULTS ON C ONTINUOUS C ONTROL WITH S PARSE
R EWARDS

The performance of TRPO with noise scaled according to the parameter curvature, as defined in
Section C.3 is shown in Figure 9. The TRPO baseline uses only action noise by using a policy
network that outputs the mean of a Gaussian distribution, while the variance is learned. These results
show that adding parameter space noise aids in either learning much more consistently on these
challenging sparse environments.

17

Published as a conference paper at ICLR 2018

TRPO with adaptive parameter noise ( = 0.01)

SparseCartPoleSwingup
400

TRPO with adaptive parameter noise ( = 1.0)

TRPO

SparseHalfCheetah

SparseMountainCar

0.05

1.0

200

0.04

0.8

150

0.03

0.6

200

100

0.02

0.4

100

50

0.01

300

return

TRPO with adaptive parameter noise ( = 0.1)

SparseDoublePendulum
250

0

0
0

500

1000

1500

epoch

2000

2500

3000

0.2

0.00
0

500

1000

1500

2000

2500

3000

epoch

0.0
0

500

1000

1500

epoch

2000

2500

3000

0

500

1000

1500

2000

2500

3000

epoch

Figure 9: Median TRPO returns with three different environments with sparse rewards plotted over
epochs.

18

