Soft-Robust Actor-Critic Policy-Gradient

Esther Derman 1 , Daniel J. Mankowitz1 , Timothy A. Mann2 and Shie Mannor1

arXiv:1803.04848v1 [cs.LG] 11 Mar 2018

1

Technion Israel Institute of Technology, Haifa, Israel
2
Google Deepmind, London, UK

estherderman@technion.ac.il, danielm@campus.technion.ac.il, timothymann@google.com, shie@ee.technion.ac.il

Abstract
Robust Reinforcement Learning aims to derive
an optimal behavior that accounts for model uncertainty in dynamical systems. However, previous studies have shown that by considering
the worst case scenario, robust policies can be
overly conservative. Our soft-robust framework
is an attempt to overcome this issue. In this
paper, we present a novel Soft-Robust ActorCritic algorithm (SR-AC). It learns an optimal
policy with respect to a distribution over an
uncertainty set and stays robust to model uncertainty but avoids the conservativeness of robust
strategies. We show convergence of the SR-AC
and test the efficiency of our approach on different domains by comparing it against regular
learning methods and their robust formulations.

1

Introduction

Markov Decision Processes (MDPs) are commonly used
to model sequential decision making in stochastic environments. A strategy that maximizes the accumulated
expected reward is then considered as optimal and can be
learned from sampling. However, besides the uncertainty
that results from stochasticity of the environment, model
parameters are often estimated from noisy data or can
change during testing [Mannor et al., 2007; Roy et al.,
2017]. This second type of uncertainty can significantly
degrade the performance of the optimal strategy from the
model’s prediction.
Robust MDPs were proposed to address this problem
[Iyengar, 2005; Nilim and El Ghaoui, 2005; Tamar et al.,
2014]. In this framework, a transition model is assumed to
belong to a known uncertainty set and an optimal strategy
is learned under the worst parameter realizations. Although the robust approach is computationally efficient

when the uncertainty set is state-wise independent, compact and convex, it can lead to overly conservative results
[Mannor et al., 2012, 2016; Xu and Mannor, 2012; Yu
and Xu, 2016].
For example, consider a business scenario where an
agent’s goal is to make as much money as possible. It can
either create a startup which may make a fortune but may
also result in bankruptcy. Alternatively, it can choose to
live off school teaching and have almost no risk but low
reward. By choosing the teaching strategy, the agent may
be overly conservative and not account for opportunities
to invest in his own promising projects. Our claim is
that one could relax this conservativeness and construct
a softer behavior that interpolates between being aggressive and robust. Ideally, the soft-robust agent should stay
agnostic to outside financing uncertainty but still be able
to take advantage of the startup experience.
This type of dilemma can be found in various domains.
In the financial market, investors seek a good trade-off
between low risk and high returns regarding portfolio
management [Mitchell and Smetters, 2013]. In strategic
management, product firms must choose the amount of resources they put into innovation. A conservative strategy
would then consist of innovating only under necessary
conditions [Miller and Friesen, 1982].
In this paper, we focus on learning a soft-robust policy
(defined below) by incorporating soft-robustness into an
online actor-critic algorithm and show its convergence
properties. Existing works mitigate conservativeness of
robust MDP either by introducing coupled uncertainties
[Mannor et al., 2012, 2016] or by assuming prior information on the uncertainty set [Xu and Mannor, 2012; Yu and
Xu, 2016]. They use dynamic programming techniques to
estimate a robust policy. However, these methods present
some limiting restrictions such as non-scalability and offline estimation. Besides being computationally more efficient than batch learning [Wiering and van Otterlo, 2012],
the use of an online algorithm is of significant interest in

robust MDPs because it can detect non-adversarial stateactions pairs along a trajectory and result in less conservative results, something which cannot be performed when
solving the planning problem [Lim et al., 2016]. Other
works have attempted to incorporate robustness into an online algorithm for policy optimization [Mankowitz et al.,
2018; Tamar et al., 2015]. Although these approaches can
deal with large domains, a sampling procedure is required
for each critic estimate in Tamar et al. [2015], which differs from the strictly-speaking actor-critic. In Mankowitz
et al. [2018], the authors introduce a robust version of
actor-critic policy-gradient but its convergence results are
only shown for the actor updates. Moreover, these works
target the robust solution which may be too conservative.
We review all existing methods in Section 7 and compare
them to our approach.
To the best of our knowledge, our proposed work is the
first attempt to incorporate a soft form of robustness into
an online algorithm that has convergence guarantees besides being computationally scalable. We deal with the
curse of dimensionality by using function approximation
that parameterizes the expected value within a space of
much smaller dimension than the state space. By fixing
a distribution over the uncertainty set, the induced softrobust actor-critic learns a locally optimal policy in an
online manner. Under mild assumptions on the set of
distributions and uncertainty set, we show that our novel
Soft-Robust Actor-Critic (SR-AC) algorithm converges.
Our specific contributions are: (1) A soft-robust derivation of the objective function for policy-gradient; (2) An
SR-AC algorithm that uses stochastic approximation to
learn a variant of distributionally robust policy in an online manner; (3) Convergence proofs of SR-AC; (4) An
experiment of our framework to different domains that
shows the efficiency of soft-robustness. All proofs can be
found in the Appendix.

2

Background

In this section, we introduce the background material
related to our soft-robust approach.
Robust MDP A robust MDP is a tuple hX , A, r, Pi
where X is a finite state-space, A is a finite set of actions, r : X × A → R is the immediate reward function
which is deterministic and bounded and P is a set of
transition matrices. N
We assume that P is structured as
a cartesian product x∈X Px , which is known as the
rectangularity assumption [Nilim and El Ghaoui, 2005].
Given a state x ∈ X , the uncertainty set Px is a family
of transition models px ∈ Px we represent as vectors
in which the transition probabilities of each action are
arranged in the same block. For x, y ∈ X and a ∈ A,

denote by p(x, a, y) the probability of getting from state
x to state y given action a.
At timestep t, the agent is in state xt and chooses an
action at according to a stochastic policy π : X → M(A)
that maps each state to a probability distribution over the
action space, M(A) denoting the set of distributions over
A. It then gets a reward rt+1 and is brought to state xt+1
with probability p(xt , at , xt+1 ).
Policy-Gradient Policy-gradient methods are commonly
used to learn an agent policy. A policy π is parametrized
by θ and estimated by optimizing an objective function
using stochastic gradient descent. A typical objective to
be considered is the average reward function
T −1

1 X
rt+1 | π]
T →+∞
T t=0
X
X
=
dπp (x)
π(x, a)r(x, a)

Jp (π) = lim Ep [

x∈X

a∈A

where rt is the reward at time t, p an aperiodic and irreducible transition model under which the agent operates
and dπp is the stationary distribution of the Markov process
induced by p under policy π. The gradient objective has
previously been shown to be
X
X
∇θ Jp (π) =
dπp (x)
∇θ π(x, a)Qπp (x, a)
x∈X

a∈A

where Qπp (x, a) is the expected differential reward associated with state-action pair (x, a). This gradient is
then used to update the policy parameters according to:
θt+1 = θt + βt ∇θ Jp (π), with βt a positive step-size
[Sutton et al., 2000].
Actor-Critic Algorithm Theoretical analysis and empirical experiments have shown that regular policy-gradient
methods present a major issue namely high variance in the
gradient estimates that results in slow convergence and
inefficient sampling [Grondman et al., 2012]. First proposed by Barto et al. [1983], actor-critic methods attempt
to reduce the variance by using a critic that estimates the
value function. They borrow elements from both value
function and policy-based methods. The value function
estimate plays the role of a critic that helps evaluating the
performance of the policy. As in policy-based methods,
the actor then uses this signal to update policy parameters
in the direction of a gradient estimate of a performance
measure. Under appropriate conditions, the resulting algorithm is tractable and converges to a locally optimal policy
[Bhatnagar et al., 2009; Konda and Tsitsiklis, 2000].
Deep Q-networks Deep Q-Networks (DQNs) have
proven their capability of solving complex learning tasks
such as Atari video games [Mnih et al., 2013]. The Qlearning of Watkins and Dayan [1992] typically learns

a greedy or -greedy policy by updating the Q-function
based on a TD-error. In Deep Q-learning [Mnih et al.,
2013, 2015], a non-linear function such as a neural network is used as an approximator of the Q-function. It
is referred to as a Q-network. The agent is then trained
by optimizing the induced TD loss function thanks to
stochastic gradient descent. Like actor-critic, DQN is an
online algorithm that aims at finding an optimal policy.
The main difference with actor-critic is that it is off-policy:
it learns a greedy strategy while following an arbitrary
behavior [Mnih et al., 2013].

3

Soft-Robustness

Soft-Robust Framework Unlike robust MDPs that maximize the worst-case performance, we fix a prior on how
transition models are distributed over the uncertainty set.
A distribution over N
P is denoted by ω and is structured as
a cartesian product x∈X ωx . We find the same structure
in Xu and Mannor [2012]; Yu and Xu [2016]. Intuitively,
ω can be thought as the way the adversary distributes over
different transition models. The product structure then
means that this adversarial distribution only depends on
the current state of the agent without taking into account
its whole trajectory. This defines a probability distribution
ωx over Px independently for each state.
We further assume that ω is non-diffuse. This implies that
the uncertainty set is non-trivial with respect to ω in a
sense that the distribution does not affect zero mass to all
of the models.
Soft-Robust Objective Throughout this paper, we make
the following assumption:
Assumption 3.1. Under any policy π, the Markov chains
resulting from any of the MDPs with transition laws p ∈
P are irreducible and aperiodic.
Define dπp as the stationary distribution of the Markov
chain that results from following policy π under transition
model p ∈ P.
Definition 3.1. We call soft-robust objective or
¯
soft-robust average reward the function J(π)
:=
Ep∼ω [Jp (π)].
The distribution ω introduces a softer form of robustness
in the objective function because it averages over the
uncertainty set instead of considering the worst-case scenario. It also gives flexibility over the level of robustness
one would like to keep. A robust strategy would then
consist of putting more mass on pessimistic transition
models. Likewise, a distribution that puts all of its mass
on one target model would lead to an aggressive behavior
and result in model misspecification.

The soft-robust differential reward is given by
Q̄π (x, a) := Ep∼ω Qπp (x, a) where
Qπp (x, a)

p

:= E

X
+∞


rt+1 − Jp (π)|x0 = x, a0 = a, π .

t=0

Similarly, we introduce the quantity
V̄ π (x) :=

X



π(x, a)Q̄π (x, a) = Ep∼ω Vpπ (x)

a∈A

P
π
with Vpπ (x) :=
a∈A π(x, a)Qp (x, a). We will interchangeably term it as soft-robust expected differential
reward or soft-robust value function.
Soft-Robust Stationary Distribution The above perfor¯ cannot as yet be written as an expecmance objective J(π)
tation of the reward over a stationary distribution because
of the added measure ω on transition models. Define the
average transition model as p̄ := Ep∼ω [p]. It corresponds
to the transition probability that results from distributing
all transition models according to ω. In analogy to the
transition probability that minimizes the reward for each
given state and action in the robust transition function
[Mankowitz et al., 2018], our average model rather selects the expected distribution over all the uncertainty set
for each state and action. Under Assumption 3.1, we
can show that the transition p̄ as defined is irreducible
and aperiodic, which ensures the existence of a unique
stationary law we will denote by d¯π .
Proposition 3.1 (Stationary distribution in the average
transition model). Under Assumption 3.1, the average
transition matrix p̄ := Ep∼ω [p] is irreducible and aperiodic. In particular, it admits a unique stationary distribution.
As in regular MDPs, the soft-robust average reward satisfies a Poisson equation, as it was first stated in the discounted reward case in Lemma 3.1 of Xu and Mannor
[2012]. The following proposition reformulates this result
for the average reward.
Proposition 3.2 (Soft-Robust Poisson equation).
¯
J(π)
+ V̄ π (x)
!
=

X
a∈A

π(x, a) r(x, a) +

X

0

π

0

p̄(x, a, x )V̄ (x )

x0 ∈X

This Poisson equation enables us to establish an equivalence between the expectation of the stationary distributions over the uncertainty set and the stationary distribution of the average transition model, naming d¯π (x) =
Ep∼ω [dπp (x)] with x ∈ X . Indeed, we have the following:

Corollary 3.1. Recall d¯π the stationary distribution for
the average transition model p̄. Then
X
X
¯
J(π)
=
d¯π (x)
π(x, a)r(x, a).
x∈X

a∈A

The goal is to learn a policy that maximizes the soft-robust
¯ We use a policy-gradient method for
average reward J.
that purpose.

4

approximator of the soft-robust differential reward Q̄π . If
fw minimizes the mean squared error

2
X
X
π
π
π
¯
E (w) :=
d (x)
π(x, a) Q̄ (x, a)−fw (x, a)
x∈X

and is compatible in a sense that ∇w fw (x, a) =
∇θ log π(x, a), then
¯
∇θ J(π)
=

Soft-Robust Policy-Gradient

In policy-gradient methods, we consider a class of
parametrized stochastic policies πθ : X → M(A) with
θ ∈ Rd1 and estimate the gradient of the objective function J¯ with respect to policy parameters in order to update
¯
the policy in the direction of the estimated gradient of J.
The optimal set of parameters thus obtained is denoted by
¯ θ ).
θ∗ := arg max J(π
θ

When clear in the context, we will omit the subscript θ in
πθ for notation ease. We further make the following assumption, which is standard in policy-gradient litterature:
Assumption 4.1. For any (x, a) ∈ X × A, the mapping
θ 7→ πθ (x, a) is continuously differentiable with respect
to θ.
Using the same method as in Sutton et al. [2000], we
can derive the gradient of the soft-robust average reward
thanks to the previous results.
Theorem 4.1 (Soft-Robust Policy Gradient). For any
MDP satisfying previous assumptions, we have
X
X
¯
∇θ J(π)
=
d¯π (x)
∇θ π(x, a)Q̄π (x, a).
x∈X

a∈A

In order to manage with large state spaces, we also
introduce a linear approximation of Q̄π we define as
fw (x, a) := wT ψxa . Sutton et al. [2000] showed that
if the features ψxa satisfy a compatibility condition and
the approximation is locally optimal, then we can use it
in place of Q̄π and still point roughly in the direction of
the true gradient.

a∈A

X

d¯π (x)

x∈X

X

∇θ π(x, a)fw (x, a)

a∈A

We can further improve our gradient estimate by reducing
its variance. One direct method to do so is to subtract a
baseline b(x) from the previous gradient update. It is easy
to show that this will not affect the gradient derivation. In
particular, Bhatnagar et al. [2009] proved that the value
function minimizes the variance. It is therefore a proper
baseline to choose. We can thus write the following:


X
X
¯
∇θ J(π)
=
d¯π (x)
∇θ π(x, a) Q̄π (x, a) − V̄ π (x)
x∈X

=

X
x∈X

a∈A

¯π

d (x)

X

π(x, a)ψxa Āπ (x, a),

a∈A

(1)
where Āπ (x, a) is the soft-robust advantage function defined by Āπ (x, a) := Q̄π (x, a) − V̄ π (x).

5

Soft-Robust Actor-Critic Algorithm

In this section, we present our SR-AC algorithm which
is defined as Algorithm 1. This novel approach incorporates a variation of distributional robustness into an
online algorithm that effectively learns an optimal policy
in a scalable manner. Under mild assumptions, the resulting two-timescale stochastic approximation algorithm
converges to a locally optimal policy.
5.1

SR-AC algorithm

In the case of soft-robust average reward, this defines
a soft-robust gradient update that possesses the ability
to incorporate function approximation, as stated in the
following result. The main difference with that of Sutton
et al. [2000] is that we combine the dynamics of the
system with distributed transitions over the uncertainty
set.

An uncertainty set and a nominal model without uncertainty are provided as inputs. In practice, the nominal
model and the uncertainty set can respectively be an estimate of the transition model resulting from data sampling and its corresponding confidence interval. A distribution ω over the uncertainty set is also provided. It
corresponds to our prior information on the uncertainty
set. The step-size sequences (αt , βt , ξt ; t ≥ 0) consist of
small non-negative numbers properly chosen by the user
(see Appendix for more details).

Theorem 4.2 (Soft-Robust Policy Gradient with Function
Approximation). Let fw : X × A → R be a linear

At each iteration, samples are generated using the nominal model and the current policy. These are utilized to

update the soft-robust average reward (Line 5) and the
critic (Line 7) based on an estimate of a soft-robust TDerror we detail further. In our setting, the soft-robust value
function plays the role of the critic according to which
the actor parameters are updated. We then exploit the
critic to improve our policy by updating the policy parameters in the direction of a gradient estimate for the
soft-robust objective (Line 8). This process is repeated
until convergence.
5.2

Algorithm 1 SR-AC
1: Input: P - An uncertainty set; p̂ ∈ P - A nominal

2:

3:
4:

Convergence Analysis
5:

We establish convergence of SR-AC to a local maxima of
the soft-robust objective function by following an ODE
approach [Kushner and Yin, 1997].
Consider V̂ and Jˆ as unbiased estimates of V̄ and J¯ respectively. Calculating δt (Line 6 in Algorithm 1) requires
an estimate of the soft-robust average-reward that can be
obtained by averaging over samples given immediate reward r and distribution ω (Line 5). In order to get an
estimate of the soft-robust differential value V̂ , we use
linear function approximation. Considering ϕ as a d2 dimensional feature extractor over the state space X , we
may then approximate V̄ π (x) as v T ϕx , where v is a d2 dimensional parameter vector that we tune using linear
TD. This results in the following soft-robust TD-error:
X
δt := rt+1 − Jˆt+1 +
p̄(xt , at , x0 )vtT ϕx0 − vtT ϕxt ,

6:
7:
8:
9:
10:

model; ω - A distribution over P; fx - A feature
extractor for the SR value function;
Initialize: θ = θ0 - An arbitrary policy parameter;
v = v0 - An arbitrary set of value function parameters; α0 , β0 , ξ0 - Initial learning-rates; x0 - Initial
state
repeat
Act under at ∼ πθt (xt , at )
Observe next state xt+1 and reward rt+1
SR Average Reward Update:
Jˆt+1 = (1 − ξt )Jˆt + ξt rt+1
SR TD-Error:
P
δt = rt+1 − Jˆt+1 + x0 ∈X p̄(xt , at , x0 )V̂x0 − V̂xt
Critic Update: vt+1 = vt + αt δt ϕxt
Actor Update: θt+1 = θt + βt δt ψxt at
until convergence
Return: SR policy parameters θ and SR valuefunction parameters v

We used the existing structure of OpenAI Gym environments to run our experiments [Brockman et al., 2016].
6.1
6.1.1

Domains
A single-step MDP

x0 ∈X

where vt corresponds to the current estimate of the softrobust value function parameter.
As in regular MDPs, when doing linear TD learning, the
function approximation of the value function introduces
a bias in the gradient estimate [Bhatnagar et al., 2009].
d
¯
Denoting it as eπ , we have E[∇
θ J(π) | θ] = ∇θ J(π) +
π
e (see Appendix). This bias term then needs to be small
enough in order to ensure convergence.
Convergence of Algorithm 1 can be established by applying Theorem 2 from Bhatnagar et al. [2009] which exploits Borkar’s work on two-timescale algorithms [1997].
The convergence result is presented as Theorem 5.1.
Theorem 5.1. Under all the previous assumptions, given
 > 0, there exists δ > 0 such that for a parameter vector
θt , t ≥ 0 obtained using the algorithm, if supπt keπt k < δ,
then the SR-AC algorithm converges almost surely to an
¯
-neighborhood of a local maximum of J.

6

Numerical Experiments

We demonstrate the performance of soft-robustness on
two domains of finite as well as continuous state spaces.

We consider a simplified formulation of the startup vs
teaching dilemma described in Section 1. The problem
is modeled as a 7-state MDP in which one action corresponds to one strategy. An illustration of this construction
is given graphically in Figure 1. At the starting state s0 ,
the agent chooses one of three actions. Action a1 [corresponds to the startup adventure] may lead it to a very
high reward in case of success but can be catastrophic
in case of failure. Action a2 [corresponds to the school
teaching carrier] leads it to low positive reward in case
of success with no possibility of negative reward. Action
a3 [corresponds to an intermediate strategy between entrepreneurship and teaching] can lead to an intermediate
positive reward with a slight risk of negative reward. Depending on the action it chose and if it succeeded or not,
the agent is brought to one of the six right-hand states and
receives the corresponding reward. It is brought back to
s0 at the end of each episode. We assume the probability
of success to be the same for all three actions.
6.1.2

Cart-Pole

In the Cart-Pole system, the agent’s goal consists of balancing a pole atop a cart in a vertical position. It is modeled as a continuous MDP in which each state consists

F1
a1 , R = -105
S1
a1 , R = 105
a2 ,R = 0

F2

a2 ,R = 2000
S2
s0
a3 ,R = −100
a3 ,R = 5000

F3

S3

Figure 1: Illustration of the MDP with initial state s0 .
States F1 , F2 , F3 correspond to failing scenarios for each
action. The succeeding states are represented by states
S1 , S2 , S3 .

6.3

Learning Algorithms

We trained the agent on the nominal model in each experiment. The soft-robust agent was learned using SR-AC in
the single-step MDP. In Cart-Pole, we run a soft-robust
version of a DQN algorithm.
Soft-Robust AC We analyze the performance of SR-AC
by training a soft-robust agent on the single-step MDP.
We run a regular AC algorithm to derive an aggressive
policy and learn a robust behavior by using a robust formulation of AC which consists in replacing the TD-error
with a robust TD-error, as implemented in Mankowitz
et al. [2018]. The derived soft-robust agent is then compared with the resulting aggressive and robust strategies
respectively.
Soft-Robust DQN Robustness has already been incorporated in DQN [Di-Castro Shashua and Mannor, 2017].
The Q-network addressed there performs an online estimation of the Q-function by minimizing at each timestep
t the following robust TD-error:
X
rob
δdqn,t
: = r(xt , at ) + γ inf
p(xt , at , x0 ) max
Q(x0 , a0 )
0
p∈P

x0 ∈X

a ∈A

− Q(xt , at ),
where γ is a discount factor.

of a 4-tuple hx, ẋ, θ, θ̇i which represents the cart position,
the cart speed, the pole angle with respect to the vertical
and its angular speed respectively. The agent can make
two possible actions: apply a constant force either to the
right or to the left of the pole. It gets a positive reward
of 1 if the pole has not fallen down and if it stayed in the
boundary sides of the screen. If it terminates, the agent
receives a reward of 0. Since each episode lasts for 200
timesteps, the maximal reward an agent can get is 200
over one episode.

In our experiments, we incorporate a soft-robust TD-error
inside a DQN that trains a soft-robust agent according to
the induced loss function. The soft-robust TD-error for
DQN is given by:
X
srob
δdqn,t
: = r(xt , at ) + γ
p̄(xt , at , x0 ) max
Q(x0 , a0 )
0
x0 ∈X

− Q(xt , at )
We use the Cart-Pole domain to compare the resulting
policy with the aggressive and robust strategies that were
obtained from a regular and a robust DQN respectively.
6.4

6.2

Uncertainty Sets

For each experiment, we generate an uncertainty set P
before training. In the single-step MDP, we sample from
5 different probabilities of success using a uniform distribution over [0, 1]. In Cart-Pole, we sample 5 different
lengths from a normal distribution centered at the nominal length of the pole which we fix at 0.3. Each length
thus generates a different transition function. We then
sample the average model by fixing ω as a realization of
a Dirichlet distribution. A soft-robust update is applied
by taking the optimal action according to this average
transition function.

a ∈A

Implementation

For each experiment, we train the agent on the nominal
model but incorporate soft-robustness during learning.
A soft-robust policy is learned thanks to SR-AC in the
single-step MDP. We use a linear function approximation
with 5 features to estimate the value function. For CartPole, we run a DQN using a neural network of 3 fullyconnected hidden layers with 128 weights per layer and
ReLu activations. We chose the ADAM optimizer to
minimize all the induced loss functions. We used constant
learning rates which worked well in practice. Each agent
was trained over Mtrain = 3000 episodes. For testing,
we averaged the performance over Mtest = 600 episodes
per parameter setting. Other hyper-parameter values can
be found in the Appendix.

6.5

Results

Shashua and Mannor [2017]; Mankowitz et al. [2018].
However, it is outperformed by the soft-robust agent
around the nominal model. Furthermore, the soft-robust
strategy shows an equilibrium between aggressiveness
and robustness thus leading to better performance than
the non-robust agent on larger pole lengths. We trained
a soft-robust agent on other weighting distributions and
remarked that depending on its structure, soft-robustness
interpolates between aggressive and robust behaviors (see
Appendix).

Single-step MDP

5

1e3

Average Reward

4
3
2

2.0 1e4

1

Aggressive
0

5

Soft-Robust
10
Training Epoch

Robust

1.5

15

Figure 2: Comparison of robust, soft-robust and aggressive agents during training. One training epoch corresponds to 300 episodes. The soft-robust strategy is not as
conservative as the robust one.

1.0
Average reward

0

0.5
0.0
0.5
1.0

Aggressive Policy
Soft-Robust Policy
Robust Policy

1.5

The evaluation of each strategy is represented in Figure 3.
As the probability of success gets low, the performance
of the aggressive agent drops down below the robust and
the soft-robust agents, although it performs best when
the probability of success gets close to 1. The robust
agent stays stable independently of the parameters but
underperforms soft-robust agent which presents the best
balance between high reward and low risk. We remarked
that depending on the weighting distribution initially set,
soft-robustness tends to being more or less aggressive. In
fact, high mass put on low probabilities of success leads to
less aggressive strategies (see Appendix). Incorporating a
distribution over the uncertainty set thus gives significant
flexibility on the level of aggressiveness to be assigned to
the soft-robust agent.
Cart-Pole In Figure 4, we show the performance of all
three strategies over different values of pole length. Similarly to our previous example, the non-robust agent performs well around the nominal model but its reward degrades on more extreme values of pole length. The robust agent keeps a stable reward under model uncertainty
which is consistent with the results obtained in Di-Castro

2.0
0.0

0.2

0.4
0.6
Probability of Success

0.8

1.0

Figure 3: Average reward for different probabilities of
success. Soft-robust policy is less sensitive to uncertainty
than aggressive policy and performs better than robustness
under optimistic scenarios.

250
200
Average reward

Figure 2 shows the evolution of the performance for all
three agents during training. It becomes more stable along
training time, which confirms convergence of SR-AC in
particular. We can see that the aggressive agent performs
best due to the highest reward it can reach on the nominal
model. The soft-robust agent gets rewards in between the
aggressive and the robust agent which performs the worst
due to its pessimistic learning method.

150
100
50
0

DQN
Robust DQN
Soft-Robust DQN
1

2

3
Pole Length

4

5

Figure 4: Average reward performance for DQN, robust
DQN and soft-robust DQN

7

Related work

This paper is related to several domains in RL such as
robust and distributionally robust MDPs, actor-critic methods and online learning via stochastic approximation algorithms. Our work solves the problem of conservativeness
encountered in robust MDPs by incorporating a variational form of distributional robustness. The SR-AC algorithm combines scalability to large scale state-spaces and
online estimation of the optimal policy in an actor-critic
algorithm. Table 1 compares our proposed algorithm with
previous approaches.
Table 1: Comparison of previous approaches with SR-AC
Reference
SR-AC (this paper)
Mankowitz et al.
[2018]
Lim et al. [2016]
Yu and Xu [2016]
Mannor et al. [2012,
2016]
Tamar et al. [2015]
Xu and Mannor
[2012]
Bhatnagar et al.
[2009]

Scalable
3
3

ActorCritic
3
7

SoftlyRobust
3
7

7
7
7

7
7
7

7
3
7

3
7

7
7

7
3

3

3

7

Many solutions have been addressed to mitigate conservativeness of robust MDP. Mannor et al. [2012, 2016] relax
the state-wise independence property of the uncertainty
set and assume it to be coupled in a way such that the planning problem stays tracktable. Another approach tends to
assume a priori information on the parameter set. These
methods include distributionally robust MDPs [Xu and
Mannor, 2012; Yu and Xu, 2016] in which the optimal
strategy maximizes the expected reward under the most
adversarial distribution over the uncertainty set. For finite
and known MDPs, under some structural assumptions on
the considered set of distributions, this max-min problem reduces to classical robust MDPs and can be solved
efficiently by dynamic programming [Puterman, 2009].
However, besides becoming untracktable under largesized MDPs, these methods use an offline learning approach which cannot adapt its level of protection against
model uncertainty and may lead to overly conservative
results [Lim et al., 2016]. The work of Lim et al. [2016]
solutions this issue and addresses an online algorithm that
learns the transitions that are purely stochastic and those
that are adversarial. Although it ensures less conservative results as well as low regret, this method sticks to

the robust objective while strongly relying on the finite
structure of the state-space. To alleviate the curse of dimensionality, we incorporate function approximation of
the objective value and define it as a linear functional of
features.
First introduced in Barto et al. [1983] and later addressed
by Bhatnagar et al. [2009], actor-critic algorithms are online learning methods that aim at finding an optimal policy.
We used the formulation of Bhatnagar et al. [2009] as a
baseline for the algorithm we proposed. The key difference between their work and ours is that we incorporate
soft-robustness. This relates in a sense to the Bayesian
Actor-Critic setup in which the critic returns a complete
posterior distribution of value functions using Bayes’ rule
[Ghavamzadeh and Engel, 2007; Ghavamzadeh et al.,
2015, 2016]. Our study keeps a frequentist approach,
meaning that our algorithm updates return point estimates
of the average value-function which prevents from tracktability issues besides enabling the distribution to be more
flexible. Another major distinction is that the Bayesian
approach incorporates a prior distribution on one model
parameters whereas our soft-robust method considers a
prior on different transition models over an uncertainty
set.
In Mankowitz et al. [2018]; Tamar et al. [2015], the authors incorporate robustness into policy-gradient methods.
A sampling procedure is required for each critic estimate
in Tamar et al. [2015], which differs from the strictlyspeaking actor-critic. A robust version of actor-critic
policy-gradient is introduced in Mankowitz et al. [2018]
but its convergence guarantees are only shown for robust
policy-gradient ascent. Both of these methods target the
robust strategy whereas we seek a soft-robust policy that
is less conservative while protecting itself against model
uncertainty.

8

Discussion

We have presented the SR-AC framework that is able to
learn policies which keep a balance between aggressive
and robust behaviors. SR-AC requires a stationary distribution under the average transition model and compatibility conditions for deriving a soft-robust policy-gradient.
We have shown that this ensures convergence of SR-AC.
This is the first work that has attempted to incorporate a
soft form of robustness into an online actor-critic method.
Our approach has been shown to be computationally scalable to large domains because of its low computational
price. In our experiments, we have also shown that the
soft-robust agent interpolates between aggressive and robust strategies without being overly conservative which
leads it to outperform robust policies under model uncertainty. Subsequent experiments should test the efficiency

of soft-robustness on more complex domains with continuous action spaces.

Garud N. Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2):257–280, 2005.

The chosen weighting over the uncertainty set can be
thought as the way the adversary distributes over different
transition laws. In our current setting, this adversarial
distribution stays constant without accounting for the rewards obtained by the agent. Future work should address
the problem of learning the sequential game induced by
an evolving adversarial distribution to derive an optimal
soft-robust policy. Other extensions of our work may also
consider non-linear objective functions such as higher order moments with respect to the adversarial distribution.

Vijay R. Konda and John N. Tsitsiklis. Actor-critic algorithms. In Advances in Neural Information Processing
Systems, volume 12, 2000.

References
Andrew G. Barto, Richard S. Sutton, and Charles W. Anderson. Neuronlike adaptive elements that can solve
difficult learning control problems. IEEE Transactions
on Systems, Man, and Cybernetics, SMC-13(5):834 –
846, 1983.
Shalabh Bhatnagar, Richard Sutton, Mohammad
Ghavamzadeh, and Mark Lee. Natural Actor-Critic
Algorithms. Automatica, elsevier edition, 2009.
Vivek S. Borkar. Stochastic approximation with two
timescales. Systems and Control Letters, 29:291–294,
1997.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas
Schneider, John Schulman, Jie Tang, and Wojciech
Zaremba. Open{AI} {G}ym. arXiv:1606.01540v1,
2016.
Shirli Di-Castro Shashua and Shie Mannor. Deep robust
{K}alman filter. arXiv preprint arXiv:1703.02310v1,
2017.
Mohammad Ghavamzadeh and Yaakov Engel. Bayesian
actor-critic algorithms. Proceedings of the 24th international conference on Machine learning, pages 297–304,
2007.
Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau,
and Aviv Tamar. Bayesian reinforcement learning: A
survey. Foundations and Trends in Machine Learning,
8(5-6):359–492, 2015.
Mohammad Ghavamzadeh, Yaakov Engel, and Michal
Valko. Bayesian policy gradient and actor-critic algorithms. Journal of Machine Learning Research, 17(66):
1–53, 2016.
Ivo Grondman, Lucian Busoniu, Gabriel A.D. Lopes, and
Robert Babuska. A survey of actor-critic reinforcement
learning: Standard and natural policy gradients. IEEE
Transactions on Systems, Man, and Cybernetics—Part
C: Applications and Reviews, 42(1291-1307), 2012.

Harold J. Kushner and Dean S. Clark. Stochastic Approximation Methods for Constrained and Unconstrained
Systems. Springer Verlag, 1978.
Harold J. Kushner and G. George Yin. Stochastic Approximation Algorithms and Applications. Springer Verlag,
New York, 1997.
Shiau Hong Lim, Huan Xu, and Shie Mannor. Reinforcement learning in robust markov decision processes. Mathematics of Operations Research, 41(4):
1325–1353, 2016.
Daniel J Mankowitz, Timothy A Mann, Shie Mannor,
Doina Precup, and Pierre-Luc Bacon. Learning robust
options. In AAAI, 2018.
Shie Mannor, Duncan Simester, Peng Sun, and John N.
Tsitsiklis. Bias and variance approximation in value
function estimates. Management Science, 53(2):308–
322, 2007.
Shie Mannor, Ofir Mebel, and Huan Xu. Lightning does
not strike twice: Robust {MDP}s with coupled uncertainty. In ICML, 2012.
Shie Mannor, Ofir Mebel, and Huan Xu. Robust {MDP}s
with k-rectangular uncertainty. Mathematics of Operations Research, 41(4):1484–1509, 2016.
Danny Miller and Peter H. Friesen. Innovation in conservative and entrepreneurial firms: Two models of
strategic momentum. Strategic Management Journal,
1982.
Olivia S. Mitchell and Kent Smetters. The Market for
Retirement Financial Advice. Oxford University Press,
first edition edition, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver,
Alex Graves, Ioannis Antonoglou, Daan Wierstra, and
Martin Riedmiller. Playing {A}tari with deep reinforcement learning: Technical report. DeepMind Technologies, arXiv:1312.5602, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver,
Andrei A. Rusu, Joel Veness, Marc G. Bellemare,
Alex Graves, Martin Riedmiller, Andreas K. Fidjeland,
Georg Ostrovski, Stig Petersen, Charles Beattie, Amir
Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement
learning. Nature, 518:529–533, 2015.
Arnab Nilim and Laurent El Ghaoui. Robust control of
{M}arkov decision processes with uncertain transition
matrices. Operations Research, 53(5):783–798, 2005.

Martin L. Puterman. Markov decision processes: Discrete
stochastic dynamic programming, volume 414. Wiley
Series in Probability and Statistics, 2009.
Aurko Roy, Huan Xu, and Sebastian Pokutta. Reinforcement learning under model mismatch. 31st Conference
on Neural Information Processing Systems, 2017.
Richard S. Sutton, David McAllester, Satinger Singh, and
Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In
Advances in Neural Information Processing Systems,
volume 12, pages 1057–1063, 2000.
Aviv Tamar, Shie Mannor, and Huan Xu. Scaling up
robust {MDP}s using function approximation. ICML,
32:1401–1415, 2014.
Aviv Tamar, Yinlam Chow, Mohammad Ghavamzadeh,
and Shie Mannor. Policy gradient for coherent risk measures. In Advances in Neural Information Processing
Systems, pages 1468–1476, 2015.
Christopher J. C. H. Watkins and Peter Dayan. Q-learning.
Machine Learning, 8:279–292, 1992.
Marco Wiering and Martijn van Otterlo. Reinforcement
Learning: State-of-the-Art. 12. Springer-Verlag Berlin
Heidelberg, 2012.
Huan Xu and Shie Mannor. Distributionally robust
{M}arkov decision processes. Mathematics of Operations Research, 37(2):288–300, 2012.
Pengqian Yu and Huan Xu. Distributionally robust counterpart in {M}arkov decision processes. IEEE Transactions on Automatic Control, 61(9):2538 – 2543, 2016.

Appendix
A
A.1

Proofs
Proposition 3.1

Proof. Fix x, y ∈ X . For any policy π, we denote by p(x, y) the probability of getting from state x to state y, which
can be written as Ea∼π(x) [p(x, a, y)]. Since ω is non-diffuse, there exists p0 such that ω(p0 ) > 0. Also, by Assumption
3.1, there exists an integer n such that pn0 (x, y) > 0. We thus have
n
p̄ (x, y) = Ep∼ω [p] (x, y)

n
n
p̄ (x, y) ≥ p0 ω(p0 ) (x, y)
n



p̄n (x, y) ≥ pn0 (x, y)ω(p0 )n > 0
which shows that p̄ is irreducible. Using the same reasoning, we show {n ∈ N : pn0 (x, x) > 0} ⊂ {n ∈ N : p̄n (x, x) >
0} and then use the fact that p0 is aperiodic to conclude that p̄ is aperiodic too.
A.2

Proposition 3.2

This recursive equation comes from the same reasoning as in Lemma 3.1 of Xu and Mannor [2012]. We apply it to the
average reward criterion.
Proof. For every p ∈ P, we can apply the Poisson equation to the corresponding model:
!
Jp (π) +

Vpπ (x)

=

X

X

π(x, a) r(x, a) +

p(x, a, x

0

)Vpπ (x0 )

x0 ∈X

a∈A

By integrating with respect to ω we obtain:
!
¯
J(π)
+ V̄ π (x) =

X

π(x, a) r(x, a) +

X

Ep∼ω [p(x, a, x

0

)Vpπ (x0 )]

x0 ∈X

a∈A

We then use the statewise independence assumption on ω to make the recursion explicit. We thus have
(1)
¯
J(π)
+ V̄ π (x) =

X
a∈A

(2)

=



X Z
0
π 0
0
0
π(x, a) r(x, a)+
p(x, a, x )Vp (x )dωx (px )dωx (px )
x0 ∈X

!
X

π(x, a) r(x, a) +

X

Epx ∼ωx [p(x, a, x

a∈A

x0 ∈X

X

X

0

)]Epx0 ∼ωx0 [Vpπ (x0 )]

!
=

a∈A

π(x, a) r(x, a) +

0

π

0

p̄(x, a, x )V̄ (x ) ,

x0 ∈X

where (1) results from the rectangularity assumption on ω. (2) Since p(x, a, x0 ) is an element of vector px that only
depends on the uncertainty set at state x and Vpπ (x0 ) depends on the uncertainty set at state x0 , we can split the integrals.
We slightly abuse notation here because a state can be visited multiple times. In fact, we implicitly introduce
dummy
N+∞
states and treat multiple visits to a state as visiting different states. More explicitely, we write ω as ω = t=0 ωx,t
where ωx,t = ωx , ωx being the distribution at state x. This representation is termed as the stationary model in Xu and
Mannor [2012].

A.3

Corollary 3.1

Proof. According to Proposition 3.2 and summing up both sides of the equality with respect to the stationary distribution
d¯π , we have
!
X
X
X
X
π
π
π
0
π
0
¯
J(π)
+
d¯ (x)V̄ (x) =
d¯ (x)
π(x, a) r(x, a) +
Ep ∼ω [p(x, a, x )]V̄ (x )
x

x∈X

x∈X

a∈A

x0 ∈X

X

X

x

!
X

=

d¯π (x)

x∈X

π(x, a) r(x, a) +

0

π

0

p̄(x, a, x )V̄ (x )

x0 ∈X

a∈A

Since d¯π is stationary with respect to p̄, we can then write
X

¯
J(π)
+

X

d¯π (x)V̄ π (x) =

x∈X

d¯π (x)

x∈X

X

π(x, a)r(x, a) +

X

d¯π (x0 )V̄ π (x0 ).

x0 ∈X

a∈A

It remains to simplify both sides of the equality in order to get the result.
A.4

Theorem 4.1

We use the same technique as in Mankowitz et al. [2018]; Sutton et al. [2000] in order to prove a soft-robust version of
policy-gradient theorem.
Proof.
X

∇θ V̄ π (x) = ∇θ

π(x, a)Q̄π (x, a)

a∈A


X
π
π
π
∇θ V̄ (x) =
∇θ π(x, a)Q̄ (x, a) + π(x, a)∇θ Q̄ (x, a)
a∈A



X
X
π
0
π 0
¯
∇θ V̄ (x) =
∇θ π(x, a)Q̄ (x, a) + π(x, a)∇θ r(x, a) − J(π) +
p̄(x, a, x )V̄ (x )
π

(1)

x0 ∈X

a∈A



X
X
π
π
0
π 0
¯
∇θ V̄ (x) =
∇θ π(x, a)Q̄ (x, a) + π(x, a) −∇θ J(π) +
p̄(x, a, x )∇θ V̄ (x )
x0 ∈X

a∈A

X

X
π
0
π 0
¯
∇θ J(π) =
∇θ π(x, a)Q̄ (x, a) + π(x, a)
p̄(x, a, x )∇θ V̄ (x ) − ∇θ V̄ π (x)
x0 ∈X

a∈A

X

(2)
¯
d¯π (x)∇θ J(π)
=

X

d¯π (x)

X

x∈X

x∈X

a∈A

X

X

X

¯
d¯π (x)∇θ J(π)
=

x∈X

d¯π (x)

x∈X

−

∇θ π(x, a)Q̄ (x, a) +

X

π(x, a)

X

π

∇θ π(x, a)Q̄ (x, a) +

d¯π (x)

x∈X

X

 X
p̄(x, a, x )∇θ V̄ (x ) −
d¯π (x)∇θ V̄ π (x)
0

0

π

x0 ∈X

a∈A

a∈A

X

π

X
a∈A

π(x, a)

x∈X

X

0

p̄(x, a, x )∇θ V̄ (x0 )
π

x0 ∈X

d¯π (x)∇θ V̄ π (x)

x∈X

X

(3)

¯
d (x)∇θ J(π)
=

X

¯
∇θ J(π)
=

X

¯π

x∈X

x∈X

d¯π (x)

X

∇θ π(x, a)Q̄π (x, a) +

X
x0 ∈X

a∈A

d¯π (x0 )∇θ V̄ (x0 ) −

X

d¯π (x)∇θ V̄ π (x)

x∈X

π

∇θ π(x, a)Q̄ (x, a)

a∈A

P
¯π
where (1) occurs thanks to the soft-robust Poisson equation.
(2) Multiply
both sides of the Equation by
x∈X d (x). (3)
P
P
P
π
π
0
π 0
¯
¯
¯
Since d (x) is stationary with respect to p̄, we have that x∈X d (x) a∈A π(x, a)p̄(x, a, x ) = x0 ∈X d (x ).

A.5

Theorem 4.2

Proof. Recall the mean squared error:
X

π

E (w) :=

x∈X

d¯π (x)

X


2
π
π(x, a) Q̄ (x, a) − fw (x, a)

a∈A

with respect to the soft-robust state distribution d¯π (x). If we derive this distribution with respect to the parameters w
and analyze it when the process has converged to a local optimum as in Sutton et al. [2000], then we get:


X
X
d¯π (x)
π(x, a) Q̄π (x, a) − fw (x, a) ∇w fw (x, a) = 0
x∈X

a∈A

Additionally, the compatibility condition ∇w fw (x, a) = ∇θ log π(x, a) yields:


X
X
d¯π (x)
π(x, a) Q̄π (x, a) − fw (x, a) ∇θ π(x, a)

1
π(x, a)
a∈A


X
X
d¯π (x)
∇θ π(x, a) Q̄π (x, a) − fw (x, a)

x∈X

x∈X

=

0

=

0

a∈A

Subtract this quantity from the soft-robust policy gradient (Theorem 4.1). We then have:


X
X
X
X
π
π
π
π
¯
¯
¯
∇θ J(π) =
d (x)
∇θ π(x, a)Q̄ (x, a) −
d (x)
∇θ π(x, a) Q̄ (x, a) − fw (x, a)
x∈X

=

X
x∈X

A.6

a∈A

d¯π (x)

X

x∈X

a∈A

∇θ π(x, a)fw (x, a).

a∈A

Convergence Proof for SR-AC

We define as soft-robust TD-error at time t the following random quantity:
X
δt := rt+1 − Jˆt+1 +
p̄(xt , at , x0 )V̂x0 − V̂xt
x0 ∈X

¯
where V̂xt and Jˆt are unbiased estimates that satisfy E[V̂xt | xt , π] = V̄ π (xt ) and E[Jˆt+1 | xt , π] = J(π)
respectively.
π
We can easily show that this defines an unbiased estimate of the soft-robust advantage function Ā [Bhatnagar et al.,
¯
2009]. Thus, using equation (1), an unbiased estimate of the gradient ∇θ J(π)
can be obtained by taking
d
∇
θ J(π) := δt ψxt at .
Similarly, recall the soft-robust TD-error with linear function approximation at time t:
X
δt := rt+1 − Jˆt+1 +
p̄(xt , at , x0 )vtT ϕx0 − vtT ϕxt ,
x0 ∈X

where vt corresponds to the current estimate of the soft-robust value function parameter.
As in regular MDPs, when doing linear TD learning, the function approximation of the value function introduces a bias
in the gradient estimate Bhatnagar et al. [2009].
Define the quantity
Ve π (x) =

X
a∈A



X
¯
π(x, a) r(x, a) − J(π)
+
p̄(x, a, x0 )vπT ϕx0
x0 ∈X

where vπT ϕx0 is an estimate of the value function upon convergence of a TD recursion, that is vπ = limt→∞ vt . Also,
define as δtπ the associated error upon convergence:
X
δtπ := rt+1 − Jˆt+1 +
p̄(xt , at , x0 )vπT ϕx0 − vπT ϕxt .
x0 ∈X

Similarly to Lemma 4 of Bhatnagar et al. [2009], the bias of the soft-robust gradient estimate is given by
eπ :=

X



d¯π (x) ∇θ Ve π (x) − ∇θ vπT ϕx ,

x∈X
π
d
¯
that is E[∇
θ J(π) | θ] = ∇θ J(π) + e . This error term then needs to be small enough in order to ensure convergence
of the algorithm.

Let denote as V̄ (v) := Φv the linear approximation to the soft-robust differential value function defined earlier, where
Φ ∈ Rn×d2 is a matrix and each feature vector ϕx (k) corresponds to the k th column in Φ. We make the following
assumption:
Assumption A.1. The basis functions ϕx ∈ Rd2 are linearly independent. In particular, Φ has full rank. We also have
Φv 6= e for all value function parameters v ∈ Rd2 where e is a vector of all ones.
The learning rates αt and
1)
Pβt (Lines
P7 and 8 in Algorithm
P
P are established such that αt → 0 slower than βt → 0
as t → ∞. In addition, t αt = t βt = ∞ and t αt2 , t βt2 < ∞. We also set the soft-robust average reward
step-size ξt = cαt for a positive constant c. The soft-robust average reward, TD-error and critic will all operate on
the faster timescale αt and therefore converge faster. Eventually, define a diagonal matrix D where the steady-state
distribution d¯π forms the diagonal of this matrix. We write the soft-robust transition probability matrix as:
X
P̄ π (x, x0 ) =
π(x, a)p̄(x, a, x0 ),
a∈A
π
n
where x, P
x0 ∈ X and p̄ designates the
P average transition model. By denoting R ∈ R the column vector of
rewards ( a∈A π(x1 , a)r(x1 , a), · · · a∈A π(xn , a)r(xn , a))T where (x1 , . . . , xn ) is a numbered representation of
the state-space and using the following operator T : Rn → Rn , we can express the soft-robust Poisson equation as:

¯
T (J) = Rπ − J(π)e
+ P̄ π J
The soft-robust average reward iterates (Line 5) and the critic iterates (Line 7) defined in Algorithm 1 converge almost
surely, as stated in the following Lemma which is a straightforward application of Lemma 5 from Bhatnagar et al.
[2009].
Lemma A.1. For any given policy π and {Jˆt }, {vt } as in the soft-robust average reward and critic updates, we have
¯
Jˆt → J(π)
and vt → v π almost surely, where
X
X
¯
J(π)
=
d¯π (x)
π(x, a)r(x, a)
x∈X

a∈A

π

is the average reward under policy π and v is the unique solution to
ΦT DΦv π = ΦT DT (Φv π )
Thanks to all the previous results, convergence of Algorithm 1 can be established by applying Theorem 2 from Bhatnagar
et al. [2009] which exploits Borkar’s work on two-timescale algorithms [1997]. For simplicity, we assume that the
iterates resulting from the actor update (Line 10 of Algorithm 1) in SR-AC remain bounded, although one could prove
convergence without such an assumption by incorporating an operator that projects any policy parameter to a compact
set, as described in Kushner and Clark [1978]. The resulting actor update would then be the projected value of the
predefined iterate. The convergence result is presented as Theorem A.1.
Theorem A.1. Under all the previous assumptions, given  > 0, there exists δ > 0 such that for a parameter vector
θt , t ≥ 0 obtained using the algorithm, if supπt keπt k < δ, then the SR-AC algorithm converges almost surely to an
¯
-neighborhood of a local maximum of J.

B
B.1

Experiments
One-step MDP
Model Parameters
Nominal probability of success
Uncertainty set for probabilities of success
Weighting Distribution 1
Weighting Distribution 2
Aggressive rewards
Soft robust rewards
Robust rewards

Value
0.8
[0.1, 0.7, 0.8, 0.3, 0.5]
[0.47, 0.22, 0.1, 0.09, 0.12]
[0.63, 0.04, 0.05, 0.02, 0.26]
105 ; −105
5000; −100
2000; 0

Hyperparameters
Critic Learning rate α
Actor Learning rate β
Step size ξ
Number of linear features
Number of episodes for training Mtrain
Number of episodes for testing Mtest

Value
5e-3
5e-5
3α
5
3000
600

2.0 1e4
1.5
Average reward

1.0
0.5
0.0
0.5
1.0

Aggressive Policy
Soft-Robust Policy 2
Robust Policy

1.5
2.0
0.0

0.2

0.4
0.6
Probability of Success

0.8

1.0

Figure 5: Average reward for different probabilities of success (distribution 2). Soft-robust policy interpolates between
aggressive and robust strategies.
B.2

Cart-Pole example
Hyperparameters
Discount factor γ
Learning rate
Mini-batch size
Final epsilon
Target update interval
Max number of episodes for training Mtrain
Number of episodes for testing Mtest

Value
0.9
1e-4
256
1e-5
10
3000
600

We trained a soft-robust agent on a different weighting over the uncertainty set. Figure 6 shows the performance of the
resulting strategy that presents a similar performance as the robust agent. This stronger form of robustness demonstrates

the flexibility we have on the way we fix the weights, which leads to more or less aggressive behaviors.

250

Average reward

200
150
100
50
0

DQN
Robust DQN
Soft-Robust DQN 2
1

2

3
Pole Length

4

5

Figure 6: Average reward performance for DQN, robust DQN and soft-robust DQN (distribution 2). Soft-robust policy
interpolates between aggressive and robust strategies.

