arXiv:1705.07269v1 [cs.LG] 20 May 2017

Learning to Factor Policies and Action-Value
Functions: Factored Action Space Representations
for Deep Reinforcement learning

Sahil Sharma
Department of Computer Science
Indian Institute of Technology, Madras

Aravind Suresh*
Department of Electrical Engineering
Indian Institute of Technology, Madras

Rahul Ramesh*
Department of Computer Science
Indian Institute of Technology, Madras

Balaraman Ravindran
Department of Computer Science
Indian Institute of Technology, Madras

Abstract
Deep Reinforcement Learning (DRL) methods have performed well in an increasing numbering of high-dimensional visual decision making domains. Among all
such visual decision making problems, those with discrete action spaces often tend
to have underlying compositional structure in the said action space. Such action
spaces often contain actions such as go left, go up as well as go diagonally up
and left (which is a composition of the former two actions). The representations
of control policies in such domains have traditionally been modeled without exploiting this inherent compositional structure in the action spaces. We propose a
new learning paradigm, Factored Action space Representations (FAR) wherein we
decompose a control policy learned using a Deep Reinforcement Learning Algorithm into independent components, analogous to decomposing a vector in terms of
some orthogonal basis vectors. This architectural modification of the control policy
representation allows the agent to learn about multiple actions simultaneously,
while executing only one of them. We demonstrate that FAR yields considerable
improvements on top of two DRL algorithms in Atari 2600: FARA3C outperforms
A3C (Asynchronous Advantage Actor Critic) in 9 out of 14 tasks and FARAQL
outperforms AQL (Asynchronous n-step Q-Learning) in 9 out of 13 tasks.

1

Introduction

Traditional Reinforcement Learning (RL) [Sutton & Barto, 1998] algorithms have worked with
relatively simple environments (such as grid worlds) wherein policy estimates can be constructed
using tabular methods or simple linear parameterizations. The state representation in such problems
often consists of hand-crafted features. Recent advances in Deep Learning (DL) [Bengio et al., 2009;
LeCun et al., 2015] have enabled RL methods to scale to problems with exponentially larger state
spaces and even continuous action domains. This combination of RL based cost functions and DL
based compositional and hierarchical representations for the state, policies and value functions has
resulted in the field of Deep Reinforcement Learning (DRL). Such DRL methods [Guo et al., 2014;
Mnih et al., 2015; Schulman et al., 2015; Lillicrap et al., 2015; Schaul et al., 2015; Mnih et al., 2016b;
Van Hasselt et al., 2016; Mnih et al., 2016a; Bacon et al., 2017; Jaderberg et al., 2017] perform
impressively in many challenging high-dimensional sequential decision making problem such as
Atari 2600 [Bellemare et al., 2013], MuJoCo [Todorov et al., 2012], TORCS Wymann et al. [2000]
and the board game of Go [Silver et al., 2016].
* These authors contributed equally

Figure 1: Visualization of the full action space for Atari 2600

Many such DRL algorithms operate on discrete action space domains such as Atari 2600. The total
number of actions in this domain is eighteen. These eighteen actions have been visualized in Figure
1. Observe that these eighteen actions, although presented to the DRL algorithms as the smallest
indivisible units of action, are in fact not indivisible at all. There exists an inherent underlying
compositional structure in this action space which a DRL algorithm can potentially take advantage
of, while training. Consider a DRL agent that executes the action go diagonally up and left and gets
some reward corresponding to this action. The key insight in this work is that this feedback can be
used to learn not only about the go diagonally up and left action but also the actions go up and go
left. Hence, every time a diagonal step is executed, it is possible to learn about the individual action
factors as well. In the Atari domain, the action space can be decomposed along three independent
dimensions: vertical motion ({go up, go down, don’t move vertically}), horizontal motion ({go left,
go right, don’t move horizontally}) and firing ({fire, don’t fire}). This work explicitly factors policies
and action-value functions along these dimensions by building the factoring into the DRL agent’s
network architecture. The notion of factoring is not new to the DRL setup. Sharma et al. [2017] have
demonstrated remarkable improvements in a variety of domains using a framework that factors the
policy into one for choosing actions and another for choosing repetitions.
The key motivations for our work are principles underlying biological systems, that have shown
to learn representations in an independent and orthogonal manner [Baumann et al., 2011; Bell &
Sejnowski, 1997]. At an abstract level, our work is also similar to intra-option learning [Sutton
et al., 1998] frameworks, which revolves around the idea of learning about an option while executing
another. Arguably, humans also make decisions in a similar manner, with the fundamental unit of
decisions being the orthogonal directions along which the decision space can be decomposed. While
our factoring scheme can be used to extend any DRL algorithm which operates with compositional
discrete action spaces, in this paper we extend the Asynchronous Advantage Actor Critic (A3C) and
the Asynchronous n-step Q-Learning [Mnih et al., 2016b] algorithms. We demonstrate empirically
that our proposed paradigm , Factored Action space Representations (FAR), provides considerable
improvements over the A3C and AQL algorithms. We also provide an analysis of the factored policy
learned using FARA3C that demonstrates its robustness compared to A3C policies.

2

Preliminaries

2.1

Q-learning

Some control algorithms estimate the optimal action-value function Q∗ (s, a) which guides the policy
executed by the RL agent. One such off-policy TD-learning algorithm [Sutton, 1988] is Q-learning
[Watkins & Dayan, 1992]. Q-learning results in the convergence of an estimated Q-function to the
optimal Q-function. The Q-learning updates are give by the equation:


Q(st , at ) ← Q(st , at ) + α rt+1 + γ max
Q(st+1 , a0 ) − Q(st , at )
0
a

A policy can be derived from the Q-function estimate by selecting any action a such that a ∈
arg max Q(s, a0 ).
a0

2

2.2

Advantage Actor-Critic

Actor-Critic algorithms [Konda & Tsitsiklis, 2000] are policy-gradient methods [Sutton &
Barto, 1998]. Most Actor Critic algorithms use parametric representations for the actor
(π(s;
θ)) and the critic

 V (s; w). A biased sample-estimate for the policy gradient is given by
∇θ log(π(at |st ; θt ) Q(st , at ). To lower variance in the updates, a baseline term is introduced and



the sample-estimate of policy gradient becomes ∇θt log(π(at |st ; θt ) Q(st , at ) − V (st ) . The
second multiplicand in the sample-estimate is the known as the advantage function A(st , at ):
A(st , at ) = Q(st , at ) − V (st ) = rt+1 + γV (st+1 ) − V (st )
where rt+1 + γV (st+1 ) is used as an estimate for Q(st , at ). This estimation for Q(st , at ) allows the
agent to model the advantage function by modeling only the value function V (s).
2.3

Asynchronous Advantage Actor-Critic (A3C)

Extending the Advantage Actor-Critic algorithms naively in the DRL setup fails to work because
the stochastic gradient descent algorithms commonly used with such DRL setups assume that the
input samples are independent and are identically distributed. An on-policy Actor-Critic algorithm
parametrized by neural networks has highly correlated inputs and therefore performs poorly when
gradient based methods are used. Asynchronous Advantage Actor-Critic (A3C) methods [Mnih et al.,
2016b] overcome this problem by using asynchronous parallel actor-learners which simultaneously
explore different parts of state space. Each learner maintains its own set of parameters which
are routinely synchronized with the other learners. Using parallel actor learners ensures that their
exploration of different parts of the state space translates into updates for the neural network that are
relatively uncorrelated.
2.4

Asynchronous N-step Q-learning (AQL)

A modified version of Q-learning, uses an n-step return based TD-target [Peng & Williams, 1996;
Watkins & Dayan, 1992] in order to achieve faster reward propagation and trade-off between bias and
variance in the estimation of the action-value function. The modified update equation is given by:
n
X

Q(st , at ) ← Q(st , at ) + α
Rt+i + γ max
Q(st+n , a0 ) − Q(st , at )
0
a

i=1

Similar to the Advantage Actor-Critic, n-Step Q-learning, has been extended to work in the DRL
setup by coming up with asynchronous n-step Q learning [Mnih et al., 2016b].

3

Factored Action Representations for Deep Reinforcement Learning

The Factored Action Representations (FAR) framework can extend any DRL algorithm that operates
on problems with compositional discrete action spaces. This includes algorithms which model
Q-functions (like DQN [Mnih et al., 2015] or asynchronous n-step Q-learning [Mnih et al., 2016b])
and actor-critic algorithms (like A3C [Mnih et al., 2016b]). In the actor critic methods, FAR extends
only the policy modeled by the actor; the critic is not modified. In the deep Q-learning methods,
FAR modifies the representation for the action-value function that the DRL algorithm models. Let X
represent either the policy of an actor-critic DRL algorithm or the Q-function of a deep Q-learning
algorithm. FAR represents X (Xi corresponds to the action ai ) using a factored representation. Each
of the factors represents a different dimension of the composite action space. We explain FAR with
the Atari domain action space as an example. Figure 2 visualizes a factoring of X over the complete
Atari action space. Any action in the complete Atari action space can be represented as a tuple
(hi , vi , fi ) with vi ∈ {go up, go down, don’t move vertically}, hi ∈ {go left, go right, don’t move
horizontally} and fi ∈ {fire, don’t fire}. The choice of the factors over which X is decomposed
depends on the set of possible actions, for a given task. This decomposed representation of X allows
the DRL agent to learn X corresponding to multiple actions simultaneously, while executing a single
action. When the action a = up-right-fire is executed, the parameters corresponding to the individual
factors of Xa : up, right and fire get updated. Hence Xup-fire , Xright-fire and Xup-left are also adjusted,
and not just the Xa .
3

Let S denote set of all states in an MDP and A denote the discrete action set for a DRL agent. We
claim that often, action spaces A are compositional and thus allow the decomposition of any action
a ∈ A into n independent action-factors [a1 , a2 , · · · , an ] such that ai ∈ Ai , where Ai is the set
of values that factor i can take. We claim that instead of modeling X over A, the agent would be
better off, modeling the individual components of X over the factor-spaces Ai . These individual
components of X are realized using independent output layers (referred to as factor-layers hereafter)
of a neural network f1 , f2 ,..., fn where fi corresponds to Ai and has size |Ai |. X can be written in
terms of the factor-layer outputs as: X(s, a) = M (f1 (a1 |s), f2 (a2 |s), ..., fn (an |s)) where s ∈ S. In
this equation, the combination
Pfunction M can have any suitable parameterized or non-parametrized
form. If X is a policy, then a∈A X(a|s) = 1 must be enforced. Detailed training algorithms for
training FAR variants proposed in this paper can be found in Appendix B.
3.1

Visualization of Action Factoring for Atari

The full action space for Atari has 18 basis actions. These basis actions can be decomposed into
three independent action factors. The first factor encodes horizontal movement (ah = left/right/no
horizontal movement). The second factor encodes vertical movement (av = up/down/no vertical
movement). The third factor encodes whether to fire or not (af = fire/don’t fire). A visualization of
this action space decomposition for the Atari domain is shown in Figure 2. This composite action
space can be visualized as a 3-dimensional cuboid of dimensions 3 × 3 × 2 where each 1 × 1 × 1
cell represents a composite action. The axes of the cuboid represent the action-factors.

Figure 2: Visualization of action factors for Atari 2600 Domain

3.2

FARA3C

We describe the instantiation of FAR for the A3C algorithm in this subsection. X is the policy
of the actor part of A3C. Let ph , pv and pf denote the action-factors corresponding to horizontal,
vertical and the firing dimensions. The action-factors are combined using a combination function
M = softmax ◦ m. The action policy is given by:
π(a|s) = softmax(m(ph (ah |s), pv (av |s), pf (af |s)))
In the equation, a = [ah , av , af ] ∈ A, ah ∈ Ah , av ∈ Av and af ∈ Af . Figure 3 contains a visual
depiction of FARA3C’s architecture.
3.3

The Additive Combination Function

The additive function is one non-parametric choice for m in the definition of M . For FARA3C, the
additive combination function has the form:
π(a|s) = softmax(ph (ah |s) + pv (av |s) + pf (af |s)))
4

Figure 3: FARA3C’s network architecture with Factored Action Representations
This policy can now be re-written as a product of three probability distributions:
π(a|s) = softmax(ph (ah |s)) × softmax(pv (av |s)) × softmax(pf (af |s))
Consider an action a = [ah , av , af ] ∈ A, such that ah ∈ Ah , av ∈ Av and af ∈ Af .
Sampling a composite action a from π(a|s) is equivalent to independently sampling ai from
πi = softmax(pi (ai |s)) where i ∈ {h, v, f }. This sub-section demonstrates that the choice of
additive combination function for A3C gives the action policy a nice alternate interpretation in terms
a product of constituent factors’ policies (such representations have been explored in other works
such as Sharma et al. [2017]).
3.4

FARAQL

We describe the instantiation of our framework for the AQL algorithm in this subsection. X is the
Q-function modeled by AQL. Let Qh , Qv and Qf denote the set of action-factors corresponding to
horizontal, vertical and the firing dimension. The action-factors are combined using a combination
function M . In this case:
Q(s, a) = M (Qh (ah |s), Qv (av |s), Qf (af |s))
where, a = [ah , av , aw ] ∈ A, ah ∈ Ah , av ∈ Av and af ∈ Af .

4

Experiments and results

Figure 4: Comparison of the different non-parametric choices for M
Our experiments are geared towards answering the following questions:
5

Figure 5: Comparison of the performances of FARA3C and A3C

1. What kind of combination functions (M) work well for action space factoring?
2. Does action factoring improve the performance of DRL algorithms (A3C & AQL)?
3. Does FAR learn fundamentally robust policy representations, compared to the baselines?
4.1

Choice of the Combination Function (M)

In this subsection we answer the first question. For chosing a good M , many competing
FARA3C networks were trained, each implementing a different non-parametric choice of
M on the Alien task in the Atari domain. The networks were trained for 50 million steps
starting from the same random initialization. The combination functions M that we experimented with were of the form softmax(m(f1 (a1 |s), f2 (a2 |s), ..., fn (an |s))), where m ∈
{Summation, Multiplication, Arithmetic Mean, Harmonic Mean, Geometric Mean, Minimum}.
Note that the Arithmetic Mean and the Summation functions are identical barring a scaling factor of
n (number of action factors). This subtle difference could however lead to different policies being
learned.

Figure 6: Training curves for FARA3C and A3C
6

Figure 7: Training curves for AQL and FARAQL

We observed that choosing m = Summation resulted in the highest performance for FARA3C as
indicated by Figure 4. We decided to stick to a similar functional form for M for the AQL case as
well, to demonstrate the robustness of our choice of the combination function. Hence the Q-values
are computed using the summation of the factor-layer outputs as:
Q(s, a) = Qh (ah |s) + Qv (av |s) + Qf (af |s))
4.2

(1)

Gameplay Experiments and Results: FARA3C

We trained FARA3C networks as depicted in Figure 3 on 14 tasks in the Atari domain. For each of
the tasks, we trained a network with three different random seeds and averaged the results to estimate
the performance of the algorithm. A baseline A3C agent was also trained with the same random
seeds. A comparison of FARA3C and A3C agents’ performance is in Figure 5. All the results of our
experiments are tabulated in Table 1 in Appendix C. The evolution of the game-play performance on
each of the tasks was plotted versus the training time. The training curves have been averaged over
the three random seeds. Figure 6 contains the training curves for six of the games. Training curves
for all the games can be found in Appendix D. Detailed explanations about the training procedure,
the evaluation procedure and the hyper-parameters can be found in Appendix A.

Figure 8: Comparison of the performances of FARAQL and AQL
7

4.3

Gameplay Experiments and Results: FARAQL

Similar to Figure 3, a modified network was constructed for FARAQL using equation 1 and trained
for thirteen tasks in Atari domain. Performance was estimated by averaging the performances across
three random seeds. A baseline AQL agent was trained using the same random seeds. A visual
comparison of the AQL and the FARAQL agent is presented in Figure 8. Training curves for six
of the games have been presented in Figure 7. Training curves for all the games can be found in
Appendix D.

5

Analysis: Test for robustness of policies learned by FARA3C & A3C

We claim that policies learned by FARA3C are more robust compared to those learned by A3C. This
is because using a factored policy representation enables the FARA3C agents to learn about multiple
actions while executing one. As a result, one would expect FARA3C to have a better ordering
of actions in the policy than A3C. In order to validate this hypothesis empirically, we conducted
experiments comparing the robustness of the A3C and FARA3C policies.
5.1

Uniformly Random from best-K analysis

The experiment is as follows: A trained agent is taken. With probability 1 − , the agent samples
actions from the learned policy. With probability , it samples actions uniformly at random from
the best k actions. The use of  introduces noise in the learned policy and the agents with the more
robust policies would demonstrate a lower drop in performance, as  increases. The value for  was
varied from 0 to 0.5 in steps of 0.05 and the corresponding normalized performances were plotted.
The individual curves have been normalized using the maximum score to ensure a fair comparison
between the FARA3C and the A3C agents by comparing only the relative changes. The experiments
in this sub-section were conducted with k = 2. From Figure 10, observe that the FARA3C agent is
comparatively more robust against policy corruption based on .
5.2

Noise Injection analysis

In this analysis, a trained agent for a particular task is taken. The policy of this agent is then
corrupted by injecting noise based on using the temperature (Z). If the policy of the agent is
written as π(a|s) = softmax(G(s)), the corrupted policy πcor (a|s) of the agent can be represented
as πcor (a|s) = softmax(G(s)/Z). Increasing Z results in the injection of more noise into the policy.
Hence a more robust policy is expected to have a smaller drop in performance on increasing Z. The
hyper-parameter Z was varied from 1.0 to 3.0 in steps of 0.25 and the performance was plotted after
normalizing it against the maximum score obtained by the agent. From Figure 9 it is clear that for
most tasks, FARA3C agents are more robust to noise injection.

Figure 9: Variation in relative score plotted against temperature Z for the FAR and FARA3C agents
8

Figure 10: Variation in relative score plotted against  for the FAR and FARA3C agents

6

Conclusion and Future Work

We propose a novel framework (FAR) for the decomposition of policies and action-value functions
over a discrete action space. FAR factors a policy/value function into independent components and
models those using independent output layers of a neural network. The FAR framework allows DRL
agents to exploit the underlying compositional structure in discrete action spaces found in common
RL domains to learn better policies and action-value functions. We empirically demonstrate the
superiority of FAR to the baseline methods considered. A possible extension of this framework would
be to combine action-space factoring with the concept of action repetition as discussed by Sharma
et al. [2017]. Action repetition could act as yet another factor of the action space and this extension
could allow one to capture a large set of macro actions.

9

References
Bacon, Pierre-Luc, Harb, Jean, and Precup, Doina. The option-critic architecture. AAAI, 2017.
Baumann, Simon, Griffiths, Timothy D, Sun, Li, Petkov, Christopher I, Thiele, Alexander, and
Rees, Adrian. Orthogonal representation of sound dimensions in the primate midbrain. Nature
neuroscience, 14(4):423–425, 2011.
Bell, Anthony J and Sejnowski, Terrence J. The “independent components” of natural scenes are
edge filters. Vision research, 37(23):3327–3338, 1997.
Bellemare, Marc G., Naddaf, Yavar, Veness, Joel, and Bowling, Michael. The arcade learning
environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research,
pp. 253–279, June 2013.
Bengio, Yoshua et al. Learning deep architectures for ai. Foundations and trends R in Machine
Learning, 2(1):1–127, 2009.
Guo, Xiaoxiao, Singh, Satinder, Lee, Honglak, Lewis, Richard L, and Wang, Xiaoshi. Deep learning
for real-time atari game play using offline monte-carlo tree search planning. In Advances in neural
information processing systems, pp. 3338–3346, 2014.
Jaderberg, Max, Mnih, Volodymyr, Czarnecki, Wojciech Marian, Schaul, Tom, Leibo, Joel Z, Silver,
David, and Kavukcuoglu, Koray. Reinforcement learning with unsupervised auxiliary tasks. To
appear in 5th International Conference on Learning Representations, 2017.
Konda, Vijay R and Tsitsiklis, John N. Actor-critic algorithms. In Advances in neural information
processing systems, pp. 1008–1014, 2000.
LeCun, Yann, Bengio, Yoshua, and Hinton, Geoffrey. Deep learning. Nature, 521(7553):436–444,
2015.
Lillicrap, Timothy P, Hunt, Jonathan J, Pritzel, Alexander, Heess, Nicolas, Erez, Tom, Tassa, Yuval,
Silver, David, and Wierstra, Daan. Continuous control with deep reinforcement learning. arXiv
preprint arXiv:1509.02971, 2015.
Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Graves, Alex, Antonoglou, Ioannis, Wierstra,
Daan, and Riedmiller, Martin. Playing atari with deep reinforcement learning. arXiv preprint
arXiv:1312.5602, 2013.
Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A., Veness, Joel, Bellemare,
Marc G., Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K., Ostrovski, Georg, Petersen, Stig,
Beattie, Charles, Sadik, Amir, Antonoglou, Ioannis, King, Helen, Kumaran, Dharshan, Wierstra,
Daan, Legg, Shane, and Hassabis, Demis. Human-level control through deep reinforcement
learning. Nature, February 2015.
Mnih, Volodymyr, Agapiou, John, Osindero, Simon, Graves, Alex, Vinyals, Oriol, Kavukcuoglu,
Koray, et al. Strategic attentive writer for learning macro-actions. arXiv preprint arXiv:1606.04695,
2016a.
Mnih, Volodymyr, Badia, Adria Puigdomenech, Mirza, Mehdi, Graves, Alex, Lillicrap, Timothy P,
Harley, Tim, Silver, David, and Kavukcuoglu, Koray. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, 2016b.
Peng, Jing and Williams, Ronald J. Incremental multi-step q-learning. Machine learning, 22(1):
283–290, 1996.
Schaul, Tom, Quan, John, Antonoglou, Ioannis, and Silver, David. Prioritized experience replay. 4th
International Conference on Learning Representations, 2015.
Schulman, John, Levine, Sergey, Moritz, Philipp, Jordan, Michael I, and Abbeel, Pieter. Trust region
policy optimization. CoRR, abs/1502.05477, 2015.
10

Sharma, Sahil, Lakshminarayanan, Aravind S., and Ravindran, Balaraman. Learning to repeat:
Fine grained action repetition for deep reinforcement learning. To appear in 5th International
Conference on Learning Representations, 2017.
Silver, David, Huang, Aja, Maddison, Chris J., Guez, Arthur, Sifre, Laurent, van den Driessche,
George, Schrittwieser, Julian, Antonoglou, Ioannis, Panneershelvam, Vedavyas, Lanctot, Marc,
Dieleman, Sander, Grewe, Dominik, Nham, John, Kalchbrenner, Nal, Sutskever, Ilya, Lillicrap,
Timothy P., Leach, Madeleine, Kavukcuoglu, Koray, Graepel, Thore, and Hassabis, Demis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489,
2016. doi: 10.1038/nature16961. URL http://dx.doi.org/10.1038/nature16961.
Sutton, Richard S. Learning to predict by the methods of temporal differences. Machine learning, 3
(1):9–44, 1988.
Sutton, Richard S. and Barto, Andrew G. Introduction to reinforcement learning. MIT Press, 1998.
Sutton, Richard S, Precup, Doina, and Singh, Satinder P. Intra-option learning about temporally
abstract actions. In ICML, volume 98, pp. 556–564, 1998.
Todorov, Emanuel, Erez, Tom, and Tassa, Yuval. Mujoco: A physics engine for model-based
control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS
2012, Vilamoura, Algarve, Portugal, October 7-12, 2012, pp. 5026–5033, 2012. doi: 10.1109/
IROS.2012.6386109. URL http://dx.doi.org/10.1109/IROS.2012.6386109.
Van Hasselt, Hado, Guez, Arthur, and Silver, David. Deep reinforcement learning with double
q-learning. In AAAI, pp. 2094–2100, 2016.
Watkins, Christopher J. C. H. and Dayan, Peter. Technical note: Q-learning. Mach. Learn., 8(3-4):
279–292, May 1992.
Wymann, Bernhard, Espié, Eric, Guionneau, Christophe, Dimitrakakis, Christos, Coulom, Rémi,
and Sumner, Andrew. Torcs, the open racing car simulator. Software available at http://torcs.
sourceforge. net, 2000.

11

Appendix A: Experimental Details
In this appendix, we document the experimental details for all of our experiments. Note that all the
reported game-play performances and graphs are averages across 3 random seeds to ensure that the
comparisons are robust to random starting points for the parameter vectors.
FARA3C and A3C
The human-starts evaluation paradigm proposed by Mnih et al. [2016b] is hard to replicate in the
absence of the same human-tester trajectories. Hence, we use the same training and evaluation
procedure as Sharma et al. [2017].
On hyper-parameters
We used the LSTM-variant of A3C [Mnih et al. [2016b]] algorithm for the both the FARA3C and
the A3C experiments. The async-rmsprop algorithm [Mnih et al. [2016b]] was used for updating
parameters with the same hyper-parameters as in Mnih et al. [2016b]. The initial learning rate used
was 10−3 and it was linearly annealed to 0 over 100 million steps. The n used in n-step returns was
20. Entropy regularization was used to encourage exploration, similar to Mnih et al. [2016b]. The β
for entropy regularization was found to be 0.01 after hyper-parameter tuning, both for FARA3C and
A3C. The β was tuned in the set {0.01, 0.02}. The optimal learning rate was found to be 7 × 10−4
for both FARA3C and A3C separately. The learning rate was tuned over the set {7 × 10−4 , 10−3 }.
The discounting factor for rewards was retained at 0.99 since it seems to work well for a large number
of methods [Mnih et al., 2016b; Sharma et al., 2017; Jaderberg et al., 2017].
All the models were trained for 100 million time steps. Evaluation was done after every 1 million
steps of training and followed the strategy described in Sharma et al. [2017]. This evaluation was
done for 100 episodes, with each episode’s length capped at 20000 steps, to arrive at an average score.
The evolution of this average game-play performance with training progress has been demonstrated
for a few games in Figure 6. An expanded version of the figure for all the games can be found in
Appendix D.
Table 1 in Appendix C contains the raw scores obtained by FARA3C and A3C agents on 15 Atari
2600 games. The evaluation was done using the best agent obtained after training for 100 million
steps (where the best agent was chosen according to the model selection paradigm outlines in citedqn).
Architecture details
We used a low level architecture similar to Mnih et al. [2016b]; Sharma et al. [2017] which in turn
uses the same low level architecture as Mnih et al. [2015]. Figure 3 contains a visual depiction of the
network used for FARA3C. The same architecture was used for FARA3C and A3C agents and has
been described below:
The first three layers are convolutional layers with same filter sizes, strides, padding and number of
filters as Mnih et al. [2015, 2016b]; Sharma et al. [2017]. These convolutional layers are followed by
two fully connected (FC) layers and an LSTM layer. A policy and a value function are derived from
the LSTM outputs using two different output heads. The number of neurons in each of the FC layers
and the LSTM layers is 256.
Similar to Mnih et al. [2016b] the Actor and Critic networks share all but the final output layer. Each
of the two functions: policy and value function are realized with a different final output layer, with
the value function outputs having no non-linearity and with the policy and having a softmax-non
linearity as output non-linearity, to model the multinomial distribution.

12

FARAQL and AQL
We used a highly tuned open-source implementation of asynchronous Q-learning found at:
https://github.com/Kaixhin/Atari.
AQL
The learning rate was set to be 7 × 10−4 . The value of n for n-step returns was set to be 5. In keeping
with the -greedy strategy highlighted for training asynchronous n-step Q-learning in Mnih et al.
[2016b] we randomly sampled  in each thread and the  was decayed to various different values
from the set {0.05, 0.1, 0.5}. The training period lasted 80 million time steps. The starting value of
epsilon was 1 and it was linearly decayed over 64 million steps to its final value. The same network
architecture as Mnih et al. [2013] was used for all the experiments.
FARAQL
To better understand the generalization capabilities of our FAR framework in general and FARQL
algorithm in particular, we decided to use the exact same hyper-parameter choices as AQL for
FARAQL. None of the hyper-parameters for FARAQL were tuned specifically for this algorithm.
This represents a hyper-parameter setting which is not very favorable to FARAQL. Even in this
setting, we observe that FARAQL significantly outperforms AQL. Tuning the hyper-parameters
specifically for FARAQL could result in a further improvement in performance.

13

Appendix B: The FARA3C Algorithm
In this appendix, we present pseudo-code versions of the training algorithm for FARA3C and
FARAQL.
FARA3C
Algorithm 1 FARA3C
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:

// Assume global shared parameter vectors θ and w
// Assume global step counter (shared) T = 0
n ← Number of independent factors
Tmax ← Total number of training steps for the FARA3C agent
π ← Policy of the agent
f ← List of factor heads for independent factors
Initialize local thread’s step counter t ← 1

repeat
tinit = t
dθ ← 0
dw ← 0
// Assume local thread’s parameters as θ0 and w0
Synchronize local thread parameters θ0 = θ and w0 = w
Obtain state stinit
repeat
π(a|st ) = M (f1 (a1 |st ), f2 (a2 |st ), · · · , fn (an |st ))
Sample at ∼ π(a|st ; θ0 )
Execute action at to obtain reward rt and observe next state st+1
t←t+1
T ←T +1
Obtain state st
until st is terminal or t == tinit + tmax
if st is terminal then
R ← 0
else
R ← V (st , w0 )
for i ∈ {t − 1, . . . tinit } do
R ← ri + γR
Accumulate gradients for θ0 : dθ ← dθ + ∇θ0 log(π(ai |si ; θ0 )(R − V (si ; w0 ))
(si ;w0 ))2
32:
Accumulate gradients for w0 : dw ← dw + ∂(R−V∂w
0
33:
Perform asynchronous update of θ using dθ and w using dw
34: until T > Tmax

14

FARAQL
Algorithm 2 FARAQL
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:

// Assume global shared parameter vector θ
// Assume global shared parameter vector for target network θ−
// Assume global step counter (shared) T = 0
n ←
Tmax
Q ←
q ←

Number of independent factors
← Total number of training steps for the FARAQL agent
Action-value function of the agent
List of factor heads for independent factors

Initialize local thread’s step counter t ← 1
Initialize target network parameters θ− ← θ
Initialize local thread’s parameter θ0 = θ
repeat
tinit = t
dθ ← 0
dw ← 0
Synchronize local thread parameters θ0 = θ
Obtain state stinit
repeat
Q(st , a) = M (Q1 (a1 |st ), Q2 (a2 |st ), · · · , Qn (an |st ))
at ∼ the  − greedy policy derived from Q(st , a; θ0 )
Execute action at to obtain reward rt and observe next state st+1
t←t+1
T ←T +1
Obtain state st
until st is terminal or t == tinit + tmax
if st is terminal then
R ← 0
else
R ← maxa (st , a; θ− )
for i ∈ {t − 1, . . . tinit } do
R ← ri + γR
Accumulate gradients for θ0 : dθ ← dθ + ∇θ0 ((R − Q(si , ai ; θ0 ))2
Perform asynchronous update of θ using dθ
until T > Tmax

15

Appendix C: Raw Performance tables
In this appendix, we document the raw performances of all our agents and the corresponding baseline
performances. All the performances have been obtained by averaging across 3 random seeds to
ensure a robust comparison.
FARA3C and A3C
Table 1 demonstrates that FARA3C outperforms A3C on 9 out of 14 tasks that we experimented with.
Name

FARA3C

A3C

Alien
Bank Heist
Boxing
Frostbite
Gravitar
Hero
James Bond
Krull
Roadrunner
Seaquest
Solaris
Space Invaders
Star Gunner
Zaxxon

3033.1
695.9
66.00
1300.70
178.17
15574.37
373.17
4550.23
59025.00
8159.90
4074.4
2056.7833
10357.33
952.67

1128.73
1769.43
-26.27
558.87
152.00
24623.97
273.33
5752.17
57585.33
2804.73
3192.53
1716.1666
35725.33
1286.67

Table 1: A3C/FARA3C Experiments on Atari 2600 domain

FARAQL and AQL
Table 2 demonstrates that FARAQL outperforms AQL on 9 out of 13 tasks that we experimented
with.
Name

FARAQL

AQL

Alien
Bank Heist
Boxing
Frostbite
Gravitar
Hero
James Bond
Krull
Roadrunner
Seaquest
Solaris
Star Gunner
Zaxxon

2361.69
878.61
95.47
1656.34
403.81
13404.56
646.54
10370.19
29656.46
4258.37
2233.29
37314.48
5122.07

2070.97
824.28
94.98
2069.62
253.64
12244.68
523.45
10112.62
36391.57
4141.18
2739.61
36265.23
3650.64

Table 2: AQL/FARAQL Experiments on Atari 2600 domain

16

Appendix D: Training curves
This appendix contains all the training curves for all the algorithms presented in this work. All
training curves have been averaged across three random seeds to ensure a robust comparison.

Figure 11: Training curves for the A3C and FARA3C agents

17

Figure 12: Training curves for the AQL and FARAQL agents

18

