Probabilistically Safe Policy Transfer

arXiv:1705.05394v1 [cs.RO] 15 May 2017

David Held1 , Zoe McCarthy1 , Michael Zhang1 , Fred Shentu1 , and Pieter Abbeel1,2,3

Abstract— Although learning-based methods have great potential for robotics, one concern is that a robot that updates
its parameters might cause large amounts of damage before it
learns the optimal policy. We formalize the idea of safe learning
in a probabilistic sense by defining an optimization problem:
we desire to maximize the expected return while keeping the
expected damage below a given safety limit. We study this
optimization for the case of a robot manipulator with safetybased torque limits. We would like to ensure that the damage
constraint is maintained at every step of the optimization and
not just at convergence. To achieve this aim, we introduce a
novel method which predicts how modifying the torque limit,
as well as how updating the policy parameters, might affect the
robot’s safety. We show through a number of experiments that
our approach allows the robot to improve its performance while
ensuring that the expected damage constraint is not violated
during the learning process.

Safe Policy
Transfer

Simula'on

Reality

Fig. 1. After training a robot to perform a task in simulation, we adapt
the learned policies to the real world. However, as the robot is adapting,
we enforce damage constraints via safety-based torque limits to prevent the
robot from applying high torques (and risk breaking something) when the
performance is poor.

I. I NTRODUCTION
Reinforcement learning has shown to be a powerful technique leading to impressive performance across a number of
domains [31]. For example, reinforcement learning methods
have been used to train a computer to outperform human performance in 49 Atari games [14], [18]. Recently, a learningbased method was used to train a computer (AlphaGo) to
beat a champion Go player [27]. Similar methods have also
been used to train robots to perform a number of difficult
tasks in simulation [10], [26] and in the real world [8], [22].
However, when applied to robotics tasks in the real world,
learning-based methods that adapt their parameters online
have the potential to be dangerous. For example, a selfdriving car or quadrotor that performs online learning might
suddenly adapt its parameters in such a way that causes it
to crash into a pedestrian or another obstacle.
Although there are many ways in which a robot can be
dangerous, for this work, we focus on the dangers caused
by a robot manipulator applying high torques. Such a robot
might break the object that it is interacting with, break a
nearby object, or damage itself.
One approach that is often used in practice is to place the
robot in an isolated environment while training, where it cannot break anything of importance. In such an environment,
the robot can perform learning with minimal risk. However,
if the isolated training environment is different from the test
environment, then this difference can lead to unexpected and
possibly dangerous behavior. For example, suppose we want
a robot to operate around or collaboratively with people;
1 Department of Electrical Engineering and Computer Science, University
of California, Berkeley, Berkeley, CA 94720
2 International Computer Science Institute (ICSI), University of California, Berkeley, Berkeley, CA 94704
3 OpenAI, San Francisco, CA 94110

the isolated training environment would likely not contain
any people, and thus the training environment would not be
representative of the test environment in which the robot
must operate.
Another solution is to train the robot in simulation. Efforts
can be made to make the simulation mimic the test environment as much as possible; however, despite one’s best
efforts, there will likely be differences between simulation
and reality, leading to policies that do not work correctly
when brought into the real world. If the robot must re-learn
how to behave in the real world, there is a risk of the robot
operating dangerously while it is adapting its behavior.
We propose that, when a robot is placed in a new environment, it should initially operate at low torques. Only
once the robot has demonstrated sufficient safety in its new
environment do we allow it to operate at higher torques. Our
approach thus makes the assumption that, if the robot violates
a safety constraint (which we will define), less damage will
be caused if the robot is operating at lower torques.
However, imposing safety-based torque limits during training brings up new questions that have not been sufficiently
explored in the robot learning literature: How do we define
the safety of the robot? How do we decide whether to
increase the torque limits? How do we ensure that the robot
always performs at a safe level of operation even while
learning to improve its performance?
We propose an approach to dealing with these challenges
for the case of a robot manipulator (shown in Figure 1).
We evaluate our approach quantitatively, and videos of our
results are also available online1 . We note that our method is
not a complete solution to the robot safety problem, and there
1 https://youtu.be/fprZHyP_50o

are still many safety-related challenges that our approach
does not address. Still, we believe our work provides a
contribution towards confronting these issues surrounding
learning and safe robot operation.
II. R ELATED W ORK
Policy Transfer. A number of recent papers have explored
how to transfer policies from one domain to another [5], [6],
[7], [11], [20], [24], [28], [29]. The goal of these papers is
typically to minimize the number of samples required in each
new environment by transferring knowledge from simulation
or from previous environments. However, these approaches
do not investigate the potential safety issues that can occur
when a policy is first run in a new environment, before any
adaptation has occurred.
AI Safety. Safety has recently become a popular topic
within the AI community, as indicated by several recent survey papers on the topic [2], [12], [21]. Amodei, Olah, et al.
organize problems of AI safety into five major categories [2].
One of their categories is safe exploration: “how to ensure
that exploratory actions in RL agents don’t lead to negative
or irrecoverable consequences that outweigh the long-term
value of exploration.” A related problem is “robustness to
distributional shift . . . how to avoid having ML systems make
bad decisions . . . when given inputs that are potentially very
different than what was seen during training.” Both of these
issues are relevant to the problem discussed in this paper.
Safe exploration. A review of past work on safe exploration is given in [2], [12], [21]. Our method falls into the
category of “constrained criterion . . . in which we want to
maximize the expectation of the return while keeping other
types of expected utilities lower than some given bounds” [1],
[12]. Typical constraints involve ensuring that either the expectation of the return exceeds a threshold [9], [13], [23], the
variance of the return remains below a threshold [4], or that
any state is reachable from any other state (ergodicity) [15],
[19]. However, these approaches only guarantee that the
constraints will hold at the optimal solution; in contrast to
our approach, they do not analyze whether the constraints
will hold at every step of the optimization.
A recent trajectory optimization algorithm (T-CHOMP)
can add a cost that keeps the velocity low near obstacles [3].
However, this method does not learn to adapt its parameters
based on errors in the model. In contrast, our approach performs online learning while maintaining damage constraints.
III. OVERVIEW
In order to obtain safe learning, we pose an optimization
problem: we want to train a robot to achieve a task as
optimally as possible while maintaining an expected safety
constraint. In order to maintain the expected safety constraint, we impose safety-based torque limits on the robot. In
Section IV, we will define our optimization problem more
precisely, and in Section V, we will describe our method for
maintaining the expected safety constraint while optimizing
the policy.

IV. P ROBLEM S ETUP
A. Minimizing Expected Damage
Suppose that a robot behaves according to some policy
π(a | s) which maps from states s to actions a (we will
abbreviate the policy as π). At each timestep t, the robot
visits a state st and takes action at , leading to a new state
st+1 and accrueing a reward, rt = r(st , at , st+1 ). Reinforcement learning problems normally aim to maximize the
expected
P return, defined as the discounted sum of rewards,
R = t γ t rt , for some constant γ.
However, operating the robot also has a risk of causing
damage to nearby objects or damaging the robot itself. On
any timestep, the agent might cause some amount of damage
d to occur. Our goal is to maximize the expected return while
keeping the expected damage below a given threshold:
maximize

E[R | π]

(1)

subject to

E[d | π] ≤ Dsafe

(2)

π

where E denotes an expectation that incorporates the stochasticity from the environment, the policy, or any other sources
of variation. The variable Dsafe is a constant that specifies
how much expected damage we are willing to tolerate.
For example, in the autonomous vehicle domain, we might
specify that we want the expected number of major accidents
per 100 million miles traveled to be less than 0.1. Note
that the damage d is assumed to be a non-negative quantity
(i.e. ∀π, d ≥ 0), so we can set Dsafe arbitrarily close to 0
to minimize the amount of expected damage that the robot
might cause. We refer to equation 2 as the damage constraint.
The problem that we wish to deal with is that, when
we first operate the robot using our initial policy, the robot
might violate the damage constraint. Further, as we optimize
the policy, the policy may change in ways that also violate
the constraint. Our goal is then to limit the amount of
expected damage caused by the robot while the policy is
being adapted.
B. Task Safety
In order to minimize expected damage, the robot must
have some notion of what might cause damage. We define a
safety constraint:
u(s) ≤ ulim

(3)

where u(s) is some function over states. For example, we
can specify that an autonomous vehicle must remain some
minimum distance from any pedestrian. We assume that
the robot might cause damage only when the constraint is
violated (i.e. when u(s) > ulim ). To discourage the robot
from violating this constraint, we use a penalty method for
optimization, with the reward received at time-step t as
rt0 = rt − λ max(0, u(st ) − u0lim )2

(4)

where u0lim ≤ ulim . By increasing u0lim over time until
u0lim = ulim and by increasing λ over time, we can ensure
that our final policy, at convergence, will maximize the
expected return without violating the safety constraint.

However, the problem is that penalty methods only guarantee that the safety constraint (equation 3) will not be
violated at convergence. On the other hand, for robotics, any
unsafe action can be dangerous or harmful, including actions
taken before convergence. We thus design a method that
ensures that, even if the robot violates the safety constraint
at intermediate steps of the optimization, the robot will not
cause a significant amount of damage.
We now define the expected damage in terms of safety
violations. We assume that, due to differences between simulation and reality and due to uncertainty in the environment,
avoiding violating the safety constraint entirely is impossible.
Let U be the event of violating the safety constraint from
equation 3 (and possibly causing damage) when acting under
policy π, and let pu (π) = p(U ). We refer to pu (π) as the
“unsafety rate,” which we abbreviate as pu . Note that no
damage can occur unless the event U has occurred.
Further, let dmax (π) be the maximum amount of damage
that can be caused by policy π if it violates the safety
constraint (as explained below, different policies can cause
different amounts of damage during constraint violations).
Then we can upper-bound the expected damage by
Z
E[d | π] = p(d(π)) d(π) dd
(5)
Z
= p(U ) p(d(π) | U ) d(π) dd
(6)
Z
≤ p(U ) p(d(π) | U ) dmax (π) dd
(7)
Z
= p(U ) dmax (π) p(d(π) | U ) dd
(8)
= p(U ) dmax (π)

(9)

where equation 6 comes from the total probability theorem
combined with the fact that p(d(π) | ¬U ) = 0 because we
assume that damage can only occur when the constraint is
violated. Thus our damage constraint (equation 2) becomes
pu (π)dmax (π) ≤ Dsafe .

(10)

Our goal is to impose constraints on the policy such that,
even if the policy violates the safety constraint (i.e. even
if pu (π) > 0), the amount of expected damage will never
exceed the threshold Dsafe .
C. Torque Limits
To concretize our problem, we assume that the actions
chosen by the robot are torques that will be applied to its
various motors. In order to limit the damage dmax (π) that
can be caused by the robot, we introduce torque limits into
our robot operation. At each time step, we limit the torque (in
absolute value) that the robot can output to some threshold
Tlim . Specifically, if our policy π outputs a torque T , then
we instead apply a torque
T 0 = max(min(T, Tlim ), −Tlim ).

(11)

We use the same torque limit for all joints.
By applying torque limits, we limit the amount of damage
that can be caused by the robot acting under any policy. For

example, if an autonomous vehicle is driving slowly, then
even if it moves unsafely (due to, for example, a slippery
road or poor visibility conditions), its slow speed will result
in less damage if the vehicle gets into an accident. Similarly,
if a robot manipulator bumps into a fragile object, it will have
a lower chance of breaking the object if it is moving at lower
torques, and if it bumps into a person, then the person will
be less likely to become injured.
The damage that can be caused by the robot is now a
function of both the policy π and the torque limit Tlim , which
we write as dmax (π, Tlim ). Analyzing the damage that can
be caused at different torque levels is outside of the scope
of this paper, so we assume that the maximum amount of
damage that can be caused in an unsafe state is linear in the
torque limit, i.e.
dmax (π, Tlim ) = α Tlim

(12)

for some constant α > 0. Our method does not depend on the
specific form of this function, and other forms can be easily
0
used. We can fold α into the safety limit Dsafe
= Dsafe /α,
so without loss of generality, we assume that α = 1.
Our damage constraint now becomes
pu (π, Tlim ) Tlim ≤ Dsafe

(13)

where the probability of violating the safety constraint pu is
now a function of both the policy π as well as the torque limit
Tlim , since both affect the robot’s operation. This equation
is fairly intuitive: if the robot is operating at a low torque
limit which is unlikely to cause much damage, we accept a
greater probability of violating the safety constraint; if the
robot is operating at a larger (and more dangerous) torque
limit, then we insist on a lower probability of violating the
safety constraint. Our approach is not restricted to torque
limits but can be used with any type of limit that controls
the amount of damage that can be caused by an unsafe policy,
such as velocity constraints.
Our initial policy might be unsafe, so we initially force
the robot to operate at a low torque limit. Although a robot
operating at a lower torque limit can cause less damage,
limiting the torques that the robot can apply also might
limit the rewards that can be obtained by the robot. A robot
that can only apply low torques will take a longer time to
achieve each task, and thus it will typically accrue lower
rewards (for many typical reward functions). We define the
Probabilistically Safe Policy Transfer problem as that of
deciding how to modify the policy and adjust the torque
limit Tlim in order to maximize the expected return while
maintaining the expected damage constraint.
One potential side-effect of this framework is that, in some
cases, varying the torque limits can increase the convergence
time of the policy: for each new value of the torque limit,
the policy might need to adapt its behavior, leading to slower
convergence. However, we accept this cost in convergence
time due to the benefit of safer robot behavior.

V. M ETHOD
A. Overall Algorithm
Now that we have defined what it means for a robot
to operate safely (in a probabilistic sense), we describe a
method for safely optimizing a policy. The overall algorithm
for Probabilistically Safe Policy Transfer is shown in Algorithm 1. We begin with an initial policy π and an initial
torque limit of Tlim = Tmin where Tmin < Dsafe , which
ensures that, even if pu = 1, our expected damage pu Tlim
will not exceed the damage limit. Using our initial policy
and torque limit, we perform a set of rollouts. Based on
these rollouts, we update the policy, as is normally done in
reinforcement learning.
Next, we adjust our torque limit. We would like to increase
our torque limit as much as possible, but we need to ensure
that the damage constraint will continue to hold for the next
iteration with the updated policy. We achieve this as follows:
we first emperically compute the current unsafety rate pu
from the rollouts. We then anticipate how much the unsafety
rate might increase in the next iteration, given by ∆pu . Let
∆pu1 be the amount that the unsafety rate might increase as
a result of changing the torque limit (described in Section VB), and let ∆pu2 be the amount that the unsafety rate might
increase as a result of changing our policy (described in
Section V-C). The probability of the union of these cases
is bounded by the sum of the individual probabilities. Using
this bound gives us the worst-case estimate of:
p0u = pu + ∆pu
≤ pu + ∆pu1 + ∆pu2 .

where Fµ,σ (T ) is the cumulative distribution function of the
Gaussian with mean µ and variance σ 2 (we have omitted
the dependency of the the mean and variance on the state s).
Because truncation is applied after sampling, Tlim contains
the probability mass of all torques T ≥ Tlim ; similarly, −Tlim
contains the probability mass for all torques T ≤ −Tlim . This
distribution is shown in Figure 2 (top).
1 - F(Tlim)

F(-Tlim)

(14)
(15)

Then, we adjust (increase or decrease) the torque limit Tlim
such that, if the unsafety rate increases to p0u , the damage
constraint would hold with equality, i.e. we set
Tlim = Dsafe /p0u .

To ensure that the damage constraint is not violated,
we impose safety-based torque limits on our robot. The
torque T is sampled from the distribution N (µ, σ 2 ) and then
truncated down to a torque T 0 based on the torque limit
(see Equation 11). The distribution for the applied torque
(i.e. after truncation) can thus be represented by a truncated
Gaussian:



−(T 0 −µ)2

√1
exp
, if − Tlim < T 0 < Tlim

2
2σ
 σ 2π


Fµ,σ (−Tlim ),
if T 0 = −Tlim
p(T 0 |s) =

1 − Fµ,σ (Tlim ),
if T 0 = Tlim




0
otherwise
(17)

(16)

We then perform new rollouts using our updated policy and
new torque limit, and we repeat until convergence.
Algorithm 1 Probabilistically Safe Policy Transfer
Input: Initial policy π, limit Dsafe , initial torque limit Tmin
1: Tlim ← Tmin
. Initialize the torque limit
2: while not converged do
3:
τ ← Run rollouts of policy π with torque limit Tlim
4:
Update the policy π
5:
Compute safety-related quantities:
6:
Compute unsafety rate pu from rollouts τ
7:
p0u ← pu + ∆pu . Predict the next unsafety rate
8:
Tlim ← Dsafe /p0u
. Update the torque limit
B. Changing the Torque Limit
We assume that our policy maps states to a Gaussian
distribution with mean µ(s) and variance σ 2 (s), and to
choose an action, we sample from this distribution. This is
a common representation in deep reinforcement learning for
continuous control problems [16], [17], [25].

Fig. 2. Top: Probability density function over the old torque limit (after
truncation). There is an additional discrete probability mass at the (positive
and negative) torque limits. Bottom: Probability density function from the
new torque limit (after truncation).

The probability density resulting from an increase in the
0
torque limit from Tlim to Tlim
is illustrated in Figure 2
(bottom). The probability density remains the same for all
torques for which −Tlim < T < Tlim , and the probability
density changes for torques in the range
TD = (T ≤ −Tlim ) ∪ (T ≥ Tlim ).

(18)

The probability that a torque would previously have been
sampled from this range is given by
p(T ∈ TD |s) = 1 − Fµ,σ (Tlim ) + Fµ,σ (−Tlim ).

(19)

This distribution is conditioned on the state s. In order to
compute the unconditional probability over torques, we must
integrate over the different states:
Z
p(T ∈ TD ) = p(s) p(T ∈ TD |s) ds.
(20)
s

The distribution over states p(s) that are visited depends on
the actions that are taken; thus, p(s) depends on the policy
π as well as the torque limit Tlim . However, estimating

p(T ∈ TD ) = Es∼p(s) [p(T ∈ TD |s)]

(21)

= Es∼p(s) [1 − Fµ,σ (Tlim ) + Fµ,σ (−Tlim )].
(22)
This quantity is the probability that a different torque will be
sampled due to changing the torque limit. If all such cases
lead to an unsafe action by our policy, then by increasing
the torque limit, the unsafety rate might increase by
∆pu1 = p(T ∈ TD ).

(23)

This quantity is used by our algorithm in Equations 15 and 16
to determine the new torque limit.
Note that our estimate of ∆pu1 does not depend on
the amount by which the torque limit was increased. Our
approach is fairly conservative: we assume that if any torque
T is sampled that is greater than our previous torque limit,
then this new torque might lead to an unsafe action, regardless of how close that torque is to the previous torque
limit. Although such an approach might in some cases be
conservative, our method errs on the side of ensuring safe
robot operation.
The above analysis holds only for torque limit increases
and not decreases. Nonetheless, in the case of torque limit
decreases, we still use the above formula to predict changes
to the unsafety rate. Based on the above analysis, such an
approach is conservative, because decreasing the torque limit
(i.e. making the torque limit tighter) will cause no new
torques to be sampled that could not have been sampled
on the previous iteration. Thus, using the same formula in
both cases results in a method that is both simpler and more
conservative than handling each case separately.
C. Updating the policy
In order to find the optimal policy in the test environment,
we must update the policy parameters. However, updating the
policy parameters can increase the unsafety rate, possibly
leading to a violation of the damage constraint. To examine
the effect of changing the policy parameters on the unsafety
rate, we compute the amount that the policy distribution can
change. An example is shown in Figure 3. The degree to
which the policy remains the same is given by the area of
intersection, AI , between the new policy and the old policy
(shown in green). The amount that the policy distribution has
changed is then given by 1 − AI , which is equal to the area
of the blue section in Figure 3. This probability mass has
shifted to the red area of this figure, which must also have
an area of 1 − AI in order for the new policy distribution to
integrate to 1. As in the previous section, if we conservatively
assume that changes in the policy distribution all lead to
unsafe states, then the unsafety rate might increase by
∆pu2 = (1 − AI ).

(24)

This quantity is used by our algorithm in Equations 15 and 16
to determine the new torque limit.
Old policy
New policy
Intersection

0.45
0.4
0.35
0.3

PDF

the effect of the torque limit on p(s) is complex, so we
approximate p(s) using the distribution from the previous
iteration (a similar approximation is often made for policy
gradient methods; see, e.g., [25]). In practice, we replace this
integral with the Monte-Carlo estimate, i.e.

0.25
0.2
0.15
0.1
0.05
0
-6

-4

-2

0

2

4

6

8

Torque

Fig. 3. Our policy is represented by a Gaussian, shown in blue and green.
After updating the policy parameters, the new policy is shown in red and
green. The green area is the probability mass that is consistent between the
new and old policies, whereas the blue mass from the old policy was shifted
to the red mass of the new policy. Both distributions integrate to 1, so the
blue and red areas must be equal. (Best viewed in color)

If the policy is represented by a complex function such
as a neural network, it is typically difficult to predict how
updating the parameters will affect the policy distribution.
However, if we are using Trust Region Policy Optimization
(TRPO) [25] to update the policy parameters, then we will
have a good estimate on how the distribution can change
between iterations. In TRPO, we update the policy according
to the following optimization:
h π (a|s)
i
θ
maximize
E
Aθold (s, a)
(25)
θ
πθold (a|s)
subject to
E[DKL (πθold (·|s) || πθ (·|s))] ≤ δKL (26)
where θold are the old policy parameters, θ are the new policy
parameters, DKL is the KL-divergence, δKL is a constant,
and Aθold (s, a) is the advantage function, computed as the
empirical return minus a baseline [10]. The constant δKL
limits the amount by which the KL-divergence of the policy
can change on each iteration. Thus, although we do not know
exactly how the policy has changed from one iteration to the
next, when using TRPO we know that the KL-divergence of
the policy cannot change more than δKL .
We would like to use this information to estimate (1−AI ).
The KL-divergence between two Gaussian distributions is
given by
KL(p1 ||p2 ) = log

1
σ 2 + (µ1 − µ2 )2
σ2
−
+ 1
2
σ1
2σ2
2

(27)

where p1 = N (µ1 , σ12 ) and p2 = N (µ2 , σ22 ). If we make the
simplifying assumption that the variance stays roughly the
same between each iteration (i.e. σ1 = σ2 = σ), and we set
the KL divergence equal to the constraint δKL , then we get
p
|µ1 − µ2 | = σ 2 δKL .
(28)
As illustrated in Figure 4, the area of intersection between
the two Gaussians of equal variance would then be given by

µ1 − µ2 
AI (µ1 , σ) = 2Fµ1 ,σ µ1 −
(29)
p2
= 2 Fµ1 ,σ (µ1 − σ δKL /2)
(30)

where Fµ1 ,σ (·) is the cumulative distribution function for a
normal distribution of mean µ1 and variance σ 2 . The mean
and variance are conditioned on the state, so as before, we
can use an expectation to remove the dependency on the
state:
p
(31)
AI = 2 Es∼p(s) [Fµ1 ,σ (µ1 − σ δKL /2)].
We can then use this estimate in Equation 24 to predict how
much the unsafety rate might increase as a result of updating
our policy.
0.45

0.45

0.4

0.4

0.35

0.35
0.3

PDF

PDF

0.3
0.25

0.25

0.2

0.2

0.15

0.15

0.1

0.1

0.05

0.05

0

Old policy Old policy
New policy New policy
IntersectionIntersection

0
-2

Safe behavior (Our method)

Unsafe behavior (baseline)

Fig. 5. Safe behavior (our method, left) vs unsafe behavior (baseline, right)
for the cup swirl task. The goal is to swirl a full cup of almonds without
spilling.
-2 0

0 2

2 4

4 6

6

Torque Torque

Fig. 4. Our old policy is represented by a Gaussian, shown in blue and
green. After updating the policy parameters, the new policy is shown in
red and green. The variances of the two distributions are assumed to be
approximately equal. (Best viewed in color)

VI. R ESULTS
A. Experimental Setup
We test our method both with a simulated PR2 (Section VI-B) and with a real PR2 (Section VI-C); pictures of
each robot can be seen in Figure 1. To illustrate our method,
we use as an example task that of swirling a cup in a circle
while keeping the cup oriented vertically (so that the contents
do not spill). Examples of safe and unsafe behavior for this
task can be seen in Figure 5, and videos of our results are
available online2 .
Our safety limit is defined with respect to the orientation
of the end-effector; in our experiments, the end-effector must
point straight down and maintain a fixed vertical orientation.
If θ is the angle of the end-effector with respect to the
downward direction, then our safety constraint is defined as
θ ≤ θlim , where θlim is defined as the maximum angle that
the robot can hold the cup without a risk of spilling. The
unsafety rate pu then measures the fraction of timesteps for
which the θ > θlim . We assume that the amount that is spilled
(i.e. the damage) will be greater if the constraint is violated
at high torques (i.e. if the robot tips the cup and applies a
large torque, moving quickly). Additionally, our experiments
demonstrate that, at higher torques, the robot will have a
harder time recovering from small errors, leading to larger
errors and more damage.
The reward is designed to encourage the robot to swirl the
cup in a horizontal circle. The modified reward used by our
2 https://youtu.be/fprZHyP_50o

reinforcement learning algorithm is defined using the penalty
method approach of equation 4. In our experiments, we set
u0lim = 0 and λ = 0.001. Our state space is defined by the left
arm’s joint-angles, the joint velocities, and the vector z which
measures the downward orientation of the end-effector; for
optimal safety, we should have z = (0, 0, −1). The actions
of our policy are the 7 joint torques for the left arm of the
PR2.
We pre-train the policy in simulation using the Mujoco [30] simulator. Our policy is represented as a neural network with 3 hidden layers of 64 nodes each, with
tanh non-linearities after each hidden layer. The policy is
optimized using TRPO [25] and Generalized Advantage
Estimation [26] with a linear baseline, as implemented by
rllab [10]. Each episode consists of 200 timesteps, where
each timestep is 50 milliseconds, and we pre-train the policy
with 50 episodes per batch. We use a discount factor of γ
= 0.95, λ = 0.98 from Generalized Advantage Estimation,
and a KL-divergence constraint of δKL = 0.01. During pretraining, we re-initialize the robot arm to a random initial
configuration before each episode, so that the policy will be
robust to the initial position of the robot.
After pre-training, we test our policy on a robot that may
have different model parameters from those that were used
during pre-training. For example, if we are pre-training in
simulation and then transferring our policy to the real world,
the real robot may have different values for friction, damping,
or other model parameters.
Our policy is fine-tuned in the test environment in order
to learn a safe policy that will correctly implement the
desired behavior. Because we desire for our policy to quickly
converge during fine-tuning, we increase our KL-divergence
constraint to δKL = 0.05 and we use only 5 episodes per
batch. Other parameters are kept the same as during pretraining, except as specified below. In order to ensure that

B. Simulation Results

Torque Limit (N·m)

Our simulated test environment allows us to test the
robustness of our algorithm under controlled environmental
variations; we will show results on a real PR2 in Section VIC. We test our algorithm for Probabilistically Safe Policy
Transfer in a simulated test environment that differs from
our training environment in that all masses and damping
parameters are reduced by a factor of 10-100 and moments
of inertia are reduced by a factor of 10.
We initialize the torque limit Tlim to 0.1 N·m and increase
it as specified by our method up to a maximum value of 3
N·m. We use Dsafe = 0.5, except where indicated otherwise.
For each method that we evaluate below, we run the method
with 5 random seeds, in order to test the robustness of each
approach.
Adaptive vs Fixed Torque Limit. First, we show the
benefit of using our adaptive torque limit method compared
to a baseline of using a fixed torque limit. Shown in Figure 6,
our method starts with a low torque limit (0.1 N·m) and
adjusts the torque limit over time. In contrast, the baseline
uses a fixed torque limit of 3 N·m. Note that our method
does not always increase the torque limit; sometimes it may
decrease it based on the unsafety rate.

Number of Itera-ons

Fig. 6. Our method (shown in blue) adjusts the torque limit over time,
based on the safety of the policy. We compare to a baseline (shown in
orange) which has a fixed torque limit, as is commonly done. The lines
show the mean and variance of the torque limit over 5 runs with different
random seeds. (Best viewed in color)

Figure 7 shows the expected damage from each of these

approaches. During the initial iterations of the policy, both
methods perform relatively poorly, leading to a large number
of constraint violations and hence a high unsafety rate (pu ).
The fixed torque limit approach operates at large torques even
during these initial stages, leading to a large expected damage
(pu Tlim ), as shown. In contrast, our adaptive torque limit
approach initially operates at a low torque limit (0.1 N·m),
and thus the initial expected damage is kept low. Our method
adapts the torque limit appropriately to keep the expected
damage below Dsafe as the policy adapts.
Expected Damage

the state-visitation distribution stays relatively fixed from one
time-step to the next, we ensure that the torque limit Tlim
does not increase by more than 5% between each iteration
(for our method as well as for the baselines). This is similar
to the KL-divergence constraint that is placed on the policy,
which prevents the policy from changing too dramatically in
each iteration.
At test time, we will be varying the torque limit Tlim to enforce the expected damage limit, using our method described
in Section V. In order to make the training environment
match to the test environment as much as possible, we also
vary the torque limit during training. We additionally input
the torque limit into the network as an additional input; thus,
our policy can learn a range of behaviors for varying torque
limits. Although this is not necessary for our method to work
(our method will still keep the expected damage below the
limit even without this modification), our experiments show
that this change is helpful for fast convergence of the policy
during test time.

Dsafe

Number of Itera4ons

Fig. 7. By adapting the torque limit based on the performance, our method
(shown in blue) is able to keep the expected damage below the safety limit
(shown in red). In contrast, using a fixed torque limit leads to a large
expected damage, especially during the initial iterations while the policy
is still adapting. The plot shows the mean and variance of the expected
damage over 5 runs with different random seeds. (Best viewed in color)

Ablation Analysis. Next, we perform an ablation analysis
to demonstrate the importance of each component of our
method. In our full method, we predict that the probability
of unsafety pu might increase from its current value due to
two factors:
• Changing the torque limit might cause pu to increase
by ∆pu1 (Section V-B)
• Updating the policy might cause pu to increase by ∆pu2
(Section V-C)
Thus, our full method predicts that, as a result of both of
these factors, the probability of unsafety pu might increase
to
p0u = pu + ∆pu1 + ∆pu2

(32)

We then set the torque limit to a value of Tlim = Dsafe /p0u ,
ensuring that the expected damage constraint will continue
to be satisfied.
In this section, we explore the effect of removing different
components of our method; specifically, we explore the
following variants for the predicted probability of unsafety:
v2) Not predicting effect of the torque limit increase:
p0u = pu + ∆pu2
v3) Not predicting effect of the policy update:
p0u = pu + ∆pu1
v4) Not predicting either effect:
p0u = pu
In each case, we set the torque limit to a value of Tlim =
Dsafe /p0u as before. We run each method with 5 random
seeds.
The result of our ablation analysis is shown in Figure 8. As
can be seen in this figure, removing any part of our method
results in a violation of the expected damage limit Dsafe .
Thus, in order to observe the expected damage limit, it is

Our full method

Expected Damage

important to predict both how changing the torque limit and
how updating the policy parameters will affect the probability
of unsafety.

Dsafe

that of our simulation; to overcome the static friction, we
initialized the torque limit Tlim to 1.8 N·m, and we increased
all torques output by the policy by 30%. We set Dsafe = 1.3.
To avoid unsafe actions due to the stochasticity of the policy,
we reduced the policy variance by a factor of two for realworld testing. We run our method three times, with different
random seeds. As shown in Figure 10, our method succeeds
in keeping the expected damage below the damage limit.

Dsafe

Expected Damage

Number of itera7ons

V3: Not
predic7ng eﬀect
of policy update

Dsafe

Dsafe
Expected Damage

Expected Damage

Number of itera7ons

V2: Not
predic7ng eﬀect
of torque limit
increase

V4: Not
predic7ng either
eﬀect

Expected Damage

Number of itera7ons

Dsafe

Number of itera7ons

Fig. 8. Ablation analysis: We compare our full method (top) to various
modified versions of our method, removing different components of our
approach (rows 2 through 4). Our full method is the only version that does
not violate the expected damage limit Dsafe .

Varying the expected damage limit. In Figure 9, we
show the effect on our policy of varying Dsafe , the expected
damage limit. We test our method with four different values
of Dsafe . In each case, our method adjusts the torque limit
accordingly to try to keep the expected damage below Dsafe .
In some trials we observed the emperical expected damage
exceed the safety threshold due to the stochasticity of our
policy in small batch sizes; with larger batch sizes, our
method observed the safety limit in all of our experiments.

Expected Damage

Dsafe

Number of Itera5ons
Fig. 10. By adapting the torque limit based on the performance, our method
is able to keep the expected damage below the limit Dsafe . Each line shows
our method with a different random seed.

We also tested whether our method can be used to successfully train a robot to swirl a full cup without spilling.
In order to verify this, we filled a cup with almonds and
tested both our method as well as the baseline (with no
torque limits). For each method, we counted the number
of almonds that were spilled after 10,000 timesteps of our
policy (8.3 minutes). Based on 5 repeated experiments, our
method drops an average of 7.8 almonds, compared to 18.2
almonds for the baseline (this result is statistically significant
with p < 0.05). Videos of our results are available online3
VII. C ONCLUSION
We present a probabilistic framework for defining robot
safety in terms of maximizing the expected return while
keeping the expected damage below a given limit. As the
policy adapts to its new environment, our method adjust the
torque limit accordingly to ensure that the robot remains safe.
Importantly, our method predicts how changes to the policy
or changes to the torque limit might affect the unsafety rate,
in order to keep the expected damage below the limit at every
iteration.
VIII. ACKNOWLEDGMENTS

Number of Itera5ons

Fig. 9. We show the effect on our algorithm of varying the expected damage
limit, Dsafe . Each line represents the expected damage for a different value
of Dsafe . The goal of our method is to ensure that, in each case, the expected
damage limit stays below Dsafe . (Best viewed in color)

C. Real-world Results
To test whether our method works in the real world, we
also fine-tune our pre-trained policy using the real PR2.
Our real robot appeared to have much greater friction than

This work was partially supported by an NSF CAREER
award (#1351028) and by Berkeley Deep Drive.
R EFERENCES
[1] Altman, E.: Asymptotic properties of constrained markov decision
processes. Zeitschrift für Operations Research 37(2), 151–170 (1993)
[2] Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman,
J., Mané, D.: Concrete problems in ai safety. arXiv preprint
arXiv:1606.06565 (2016)
[3] Byravan, A., Boots, B., Srinivasa, S.S., Fox, D.: Space-time functional
gradient optimization for motion planning. In: IEEE International
Conference on Robotics and Automation (ICRA). IEEE (2014)
3 https://youtu.be/fprZHyP_50o

[4] Castro, D.D., Tamar, A., Mannor, S.: Policy gradients with variance
related risk criteria. In: ICML (2012)
[5] Cutler, M., How, J.P.: Efficient reinforcement learning for robots using
informative simulated priors. In: 2015 IEEE International Conference
on Robotics and Automation (ICRA). pp. 2605–2612. IEEE (2015)
[6] Cutler, M., How, J.P.: Autonomous drifting using simulation-aided
reinforcement learning. In: ICRA. IEEE (2016)
[7] Cutler, M., Walsh, T.J., How, J.P.: Reinforcement learning with
multi-fidelity simulators. In: 2014 IEEE International Conference on
Robotics and Automation (ICRA). pp. 3888–3895. IEEE (2014)
[8] Deisenroth, M., Rasmussen, C.E.: Pilco: A model-based and dataefficient approach to policy search. In: 28th International Conference
on Machine Learning (ICML). pp. 465–472 (2011)
[9] Delage, E., Mannor, S.: Percentile optimization for markov decision
processes with parameter uncertainty. Operations research (2010)
[10] Duan, Y., Chen, X., Houthooft, R., Schulman, J., Abbeel, P.: Benchmarking deep reinforcement learning for continuous control. International Conference on Machine Learning (ICML) (2016)
[11] Fu, J., Levine, S., Abbeel, P.: One-shot learning of manipulation skills
with online dynamics adaptation and neural network priors. Proceedings of the 29th IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS) (2015)
[12] Garcıa, J., Fernández, F.: A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research (2015)
[13] Geibel, P.: Reinforcement learning for mdps with constraints. In:
European Conference on Machine Learning. Springer (2006)
[14] Guo, X., Singh, S., Lewis, R., Lee, H.: Deep learning for reward
design to improve monte carlo tree search in atari games. Conference
on Uncertainty in Artificial Intelligence (UAI) (2016)
[15] Hutter, M.: Self-optimizing and pareto-optimal policies in general
environments based on bayes-mixtures. In: International Conference
on Computational Learning Theory. pp. 364–379. Springer (2002)
[16] Levine, S., Finn, C., Darrell, T., Abbeel, P.: End-to-end training of deep
visuomotor policies. Journal of Machine Learning Research (2016)
[17] Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T.P., Harley,
T., Silver, D., Kavukcuoglu, K.: Asynchronous methods for deep
reinforcement learning. ICML (2016)
[18] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski,
G., et al.: Human-level control through deep reinforcement learning.
Nature 518(7540), 529–533 (2015)
[19] Moldovan, T.M., Abbeel, P.: Safe exploration in markov decision
processes. In: Proceedings of the 29th International Conference on
Machine Learning (ICML). pp. 1711–1718 (2012)
[20] Mordatch, I., Mishra, N., Eppner, C., Abbeel, P.: Combining modelbased policy search with online model learning for control of physical
humanoids. ICRA (2016)
[21] Pecka, M., Svoboda, T.: Safe exploration techniques for reinforcement
learning–an overview. In: International Workshop on Modelling and
Simulation for Autonomous Systems. pp. 357–375. Springer (2014)
[22] Peters, J., Schaal, S.: Reinforcement learning of motor skills with
policy gradients. Neural networks 21(4), 682–697 (2008)
[23] Ponda, S.S., Johnson, L.B., How, J.P.: Risk allocation strategies for
distributed chance-constrained task allocation. In: 2013 American
Control Conference. pp. 3230–3236. IEEE (2013)
[24] Rusu, A.A., Rabinowitz, N.C., Desjardins, G., Soyer, H., Kirkpatrick,
J., Kavukcuoglu, K., Pascanu, R., Hadsell, R.: Progressive neural
networks. arXiv preprint arXiv:1606.04671 (2016)
[25] Schulman, J., Levine, S., Abbeel, P., Jordan, M., Moritz, P.: Trust
region policy optimization. In: Proceedings of The 32nd International
Conference on Machine Learning. pp. 1889–1897 (2015)
[26] Schulman, J., Moritz, P., Levine, S., Jordan, M., Abbeel, P.: Highdimensional continuous control using generalized advantage estimation. ICLR (2016)
[27] Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., Van
Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam,
V., Lanctot, M., et al.: Mastering the game of go with deep neural
networks and tree search. Nature 529(7587), 484–489 (2016)
[28] Taylor, M.E., Stone, P.: An introduction to intertask transfer for
reinforcement learning. AI Magazine 32(1), 15 (2011)
[29] Tessler, C., Givony, S., Zahavy, T., Mankowitz, D.J., Mannor, S.: A
deep hierarchical approach to lifelong learning in minecraft. arXiv
preprint arXiv:1604.07255 (2016)

[30] Todorov, E., Erez, T., Tassa, Y.: Mujoco: A physics engine for
model-based control. In: 2012 IEEE/RSJ International Conference on
Intelligent Robots and Systems. pp. 5026–5033. IEEE (2012)
[31] Zhong, M., Johnson, M., Tassa, Y., Erez, T., Todorov, E.: Value function approximation and model predictive control. In: IEEE Symposium
on Adaptive Dynamic Programming and Reinforcement Learning
(ADPRL). pp. 100–107. IEEE (2013)

