Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning

Junhyuk Oh 1 Satinder Singh 1 Honglak Lee 1 2 Pushmeet Kohli 3

arXiv:1706.05064v2 [] 7 Nov 2017

Abstract
As a step towards developing zero-shot task generalization capabilities in reinforcement learning
(RL), we introduce a new RL problem where the
agent should learn to execute sequences of instructions after learning useful skills that solve
subtasks. In this problem, we consider two types
of generalizations: to previously unseen instructions and to longer sequences of instructions. For
generalization over unseen instructions, we propose a new objective which encourages learning correspondences between similar subtasks by
making analogies. For generalization over sequential instructions, we present a hierarchical
architecture where a meta controller learns to use
the acquired skills for executing the instructions.
To deal with delayed reward, we propose a new
neural architecture in the meta controller that
learns when to update the subtask, which makes
learning more efficient. Experimental results on
a stochastic 3D domain show that the proposed
ideas are crucial for generalization to longer instructions as well as unseen instructions.

1. Introduction
The ability to understand and follow instructions allows
us to perform a large number of new complex sequential tasks even without additional learning. For example,
we can make a new dish following a recipe, and explore
a new city following a guidebook. Developing the ability to execute instructions can potentially allow reinforcement learning (RL) agents to generalize quickly over tasks
for which such instructions are available. For example,
factory-trained household robots could execute novel tasks
in a new house following a human user’s instructions (e.g.,
tasks involving household chores, going to a new place,
picking up/manipulating new objects, etc.). In addition to
generalization over instructions, an intelligent agent should
also be able to handle unexpected events (e.g., low bat1
University of Michigan 2 Google Brain 3 Microsoft Research.
Correspondence to: Junhyuk Oh <junhyuk@umich.edu>.
th

Proceedings of the 34 International Conference on Machine
Learning, Sydney, Australia, PMLR 70, 2017. Copyright 2017
by the author(s).

Figure 1: Example of 3D world and instructions. The agent is
tasked to execute longer sequences of instructions in the correct
order after training on short sequences of instructions; in addition, previously unseen instructions can be given during evaluation (blue text). Additional reward is available from randomly
appearing boxes regardless of instructions (green circle).

tery, arrivals of reward sources) while executing instructions. Thus, the agent should not blindly execute instructions sequentially but sometimes deviate from instructions
depending on circumstances, which requires balancing between two different objectives.
Problem. To develop such capabilities, this paper introduces the instruction execution problem where the agent’s
overall task is to execute a given list of instructions described by a simple form of natural language while dealing
with unexpected events, as illustrated in Figure 1. More
specifically, we assume that each instruction can be executed by performing one or more high-level subtasks in sequence. Even though the agent can pre-learn skills to perform such subtasks (e.g., [Pick up, Pig] in Figure 1), and the
instructions can be easily translated to subtasks, our problem is difficult due to the following challenges.
• Generalization: Pre-training of skills can only be done
on a subset of subtasks, but the agent is required to perform previously unseen subtasks (e.g., going to a new
place) in order to execute unseen instructions during testing. Thus, the agent should learn to generalize to new
subtasks in the skill learning stage. Furthermore, the
agent is required to execute previously unseen and longer
sequences of instructions during evaluation.
• Delayed reward: The agent is not told which instruction
to execute at any point of time from the environment but
just given the full list of instructions as input. In addition,
the agent does not receive any signal on completing in-

Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning

dividual instructions from the environment, i.e., successreward is provided only when all instructions are executed correctly. Therefore, the agent should keep track
of which instruction it is executing and decide when to
move on to the next instruction.
• Interruption: As described in Figure 1, there can be unexpected events in uncertain environments, such as opportunities to earn bonuses (e.g., windfalls), or emergencies (e.g., low battery). It can be better for the agent to interrupt the ongoing subtask before it is finished, perform
a different subtask to deal with such events, and resume
executing the interrupted subtask in the instructions after
that. Thus, the agent should achieve a balance between
executing instructions and dealing with such events.
• Memory: There are loop instructions (e.g., “Pick up 3
pig”) which require the agent to perform the same subtask ([Pick up, Pig]) multiple times and take into account
the history of observations and subtasks in order to decide when to move on to the next instruction correctly.
Due to these challenges, the agent should be able to execute
a novel subtask, keep track of what it has done, monitor
observations to interrupt ongoing subtasks depending on
circumstances, and switch to the next instruction precisely
when the current instruction is finished.
Our Approach and Technical Contributions. To address the aforementioned challenges, we divide the learning problem into two stages: 1) learning skills to perform
a set of subtasks and generalizing to unseen subtasks, and
2) learning to execute instructions using the learned skills.
Specifically, we assume that subtasks are defined by several
disentangled parameters. Thus, in the first stage our architecture learns a parameterized skill (da Silva et al., 2012) to
perform different subtasks depending on input parameters.
In order to generalize over unseen parameters, we propose
a new objective function that encourages making analogies
between similar subtasks so that the underlying manifold of
the entire subtask space can be learned without experiencing all subtasks. In the second stage, our architecture learns
a meta controller on top of the parameterized skill so that
it can read instructions and decide which subtask to perform. The overall hierarchical RL architecture is shown in
Figure 3. To deal with delayed reward as well as interruption, we propose a novel neural network (see Figure 4) that
learns when to update the subtask in the meta controller.
This not only allows learning to be more efficient under delayed reward by operating at a larger time-scale but also allows interruptions of ongoing subtasks when an unexpected
event is observed.
Main Results. We developed a 3D visual environment
using Minecraft based on Oh et al. (2016) where the
agent can interact with many objects. Our results on multiple sets of parameterized subtasks show that our pro-

posed analogy-making objective can generalize successfully. Our results on multiple instruction execution problems show that our meta controller’s ability to learn when
to update the subtask plays a key role in solving the overall problem and outperforms several hierarchical baselines. The demo videos are available at the following
website: https://sites.google.com/a/umich.
edu/junhyuk-oh/task-generalization.
The rest of the sections are organized as follows. Section 2
presents related work. Section 3 presents our analogymaking objective for generalization to parameterized tasks
and demonstrates its application to different generalization
scenarios. Section 4 presents our hierarchical architecture
for the instruction execution problem with our new neural
network that learns to operate at a large time-scale. In addition, we demonstrate our agent’s ability to generalize over
sequences of instructions, as well as provide a comparison
to several alternative approaches.

2. Related Work
Hierarchical RL. A number of hierarchical RL approaches are designed to deal with sequential tasks. Typically these have the form of a meta controller and a set of
lower-level controllers for subtasks (Sutton et al., 1999; Dietterich, 2000; Parr and Russell, 1997; Ghavamzadeh and
Mahadevan, 2003; Konidaris et al., 2012; Konidaris and
Barto, 2007). However, much of the previous work assumes that the overall task is fixed (e.g., Taxi domain (Dietterich, 2000)). In other words, the optimal sequence of
subtasks is fixed during evaluation (e.g., picking up a passenger followed by navigating to a destination in the Taxi
domain). This makes it hard to evaluate the agent’s ability
to compose pre-learned policies to solve previously unseen
sequential tasks in a zero-shot fashion unless we re-train the
agent on the new tasks in a transfer learning setting (Singh,
1991; 1992; McGovern and Barto, 2002). Our work is also
closely related to Programmable HAMs (PHAMs) (Andre
and Russell, 2000; 2002) in that a PHAM is designed to
execute a given program. However, the program explicitly
specifies the policy in PHAMs which effectively reduces
the state-action search space. In contrast, instructions are a
description of the task in our work, which means that the
agent should learn to use the instructions to maximize its
reward.
Hierarchical Deep RL. Hierarhical RL has been recently combined with deep learning. Kulkarni et al. (2016)
proposed hierarchical Deep Q-Learning and demonstrated
improved exploration in a challenging Atari game. Tessler
et al. (2017) proposed a similar architecture, but the highlevel controller is allowed to choose primitive actions directly. Bacon et al. (2017) proposed the option-critic architecture, which learns options without pseudo reward
and demonstrated that it can learn distinct options in Atari

Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
Action
Observation
Task
parameters

Termination?
Task
embedding

Figure 2: Architecture of parameterized skill. See text for details.

games. Heess et al. (2016) formulated the actions of the
meta controller as continuous variables that are used to
modulate the behavior of the low-level controller. Florensa
et al. (2017) trained a stochastic neural network with mutual information regularization to discover skills. Most of
these approaches build an open-loop policy at the highlevel controller that waits until the previous subtask is finished once it is chosen. This approach is not able to interrupt ongoing subtasks in principle, while our architecture
can switch its subtask at any time.
Zero-Shot Task Generalization. There have been a few
papers on zero-shot generalization to new tasks. For example, da Silva et al. (2012) introduced parameterized skills
that map sets of task descriptions to policies. Isele et al.
(2016) achieved zero-shot task generalization through dictionary learning with sparsity constraints. Schaul et al.
(2015) proposed universal value function approximators
(UVFAs) that learn value functions for state and goal pairs.
Devin et al. (2017) proposed composing sub-networks that
are shared across tasks and robots in order to achieve generalization to unseen configurations of them. Unlike the
above prior work, we propose a flexible metric learning
method which can be applied to various generalization scenarios. Andreas et al. (2016) proposed a framework to learn
the underlying subtasks from a policy sketch which specifies a sequence of subtasks, and the agent can generalize
over new sequences of them in principle. In contrast, our
work aims to generalize over unseen subtasks as well as unseen sequences of them. In addition, the agent should handle unexpected events in our problem that are not described
by the instructions by interrupting subtasks appropriately.
Instruction Execution. There has been a line of work for
building agents that can execute natural language instructions: Tellex et al. (2011; 2014) for robotics and MacMahon et al. (2006); Chen and Mooney (2011); Mei et al.
(2015) for a simulated environment. However, these approaches focus on natural language understanding to map
instructions to actions or groundings in a supervised setting. In contrast, we focus on generalization to sequences
of instructions without any supervision for language understanding or for actions. Although Branavan et al. (2009)
also tackle a similar problem, the agent is given a single
instruction at a time, while our agent needs to learn how to
align instructions and state given a full list of instructions.

3. Learning a Parameterized Skill
In this paper, a parameterized skill is a multi-task policy
corresponding to multiple tasks defined by categorical input task parameters, e.g., [Pick up, X]. More formally, we
define a parameterized skill as a mapping O × G → A × B,
where O is a set of observations, G is a set of task parameters, A is a set of primitive actions, and B = {0, 1}
indicates whether the task is finished or not. A space of
tasks is defined using the Cartesian product of task parameters: G = G (1) × ... × G (n) , where G (i) is a set of the
i-th parameters (e.g., G = {Visit, Pick up}×{X, Y, Z}).
Given
 an observation
 xt ∈ O at time t and task parameters
g = g (1) , ..., g (n) ∈ G, where g (i) is a one-hot vector, the
parameterized skill is the following functions:
Policy: πφ (at |xt , g)
Termination: βφ (bt |xt , g),
where πφ is the policy optimized for the task g, and βφ
is a termination function (Sutton et al., 1999) which is the
probability that the state is terminal at time t for the given
task g. The parameterized skill is represented by a nonlinear function approximator φ(·), a neural network in this
paper. The neural network architecture of our parameterized skill is illustrated in Figure 2. The network maps input
task parameters into a task embedding space ϕ(g), which
is combined with the observation followed by the output
layers. More details are described in the Appendix.
3.1. Learning to Generalize by Analogy-Making
Only a subset of tasks (G 0 ⊂ G) are available during training, and so in order to generalize to unseen tasks during
evaluation the network needs to learn knowledge about the
relationship between different task parameters when learning the task embedding ϕ(g).
To this end, we propose an analogy-making objective inspired by Reed et al. (2015). The main idea is to learn
correspondences between tasks. For example, if target objects and ‘Visit/Pick up’ actions are independent (i.e., each
action can be applied to any target object), we can enforce
the analogy [Visit, X] : [Visit, Y] :: [Pick up, X] : [Pick up,
Y] for any X and Y in the embedding space, which means
that the difference between ‘Visit’ and ‘Pick up’ is consistent regardless of target objects and vice versa. This allows
the agent to generalize to unseen combinations of actions
and target objects, such as performing [Pick up, Y] after it
has learned to perform [Pick up, X] and [Visit, Y].
More specifically, we define several constraints as follows:
k∆ (gA , gB ) − ∆ (gC , gD )k ≈ 0

if gA : gB :: gC : gD

k∆ (gA , gB ) − ∆ (gC , gD )k ≥ τdis

if gA : gB 6= gC : gD

k∆ (gA , gB )k ≥ τdif f
if gA 6= gB ,
h
i
(1) (2)
(n)
where gk = gk , gk , ..., gk
∈ G are task parameters,

Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
Scenario
Analogy
Train
Unseen
∆ (gA , gB ) = ϕ(gA ) − ϕ(gB ) is the difference vector be×
0.3
(99.8%)
-3.7
(34.8%)
tween two tasks in the embedding space, and τdis and τdif f
Independent
X
0.3 (99.8%) 0.3 (99.5%)
are constant threshold distances. Intuitively, the first con×
0.3 (99.7%) -5.0 (2.2%)
Object-dependent
straint enforces the analogy (i.e., parallelogram structure
X
0.3 (99.8%) 0.3 (99.7%)
in the embedding space; see Mikolov et al. (2013); Reed
×
-0.7 (97.5%) -2.2 (24.9%)
Inter/Extrapolation
X
-0.7 (97.5%) -1.7 (94.5%)
et al. (2015)), while the other constraints prevent trivial solutions. We incorporate these constraints into the following
Table 1: Performance on parameterized tasks. Each entry shows
‘Average reward (Success rate)’. We assume an episode is sucobjectives based on contrastive loss (Hadsell et al., 2006):
cessful only if the agent successfully finishes the task and its ter

2
Lsim = EgA...D ∼Gsim k∆ (gA , gB ) − ∆ (gC , gD ) k
mination predictions are correct throughout the whole episode.
h
i
2
Ldis = EgA...D ∼Gdis (τdis − k∆ (gA , gB ) − ∆ (gC , gD ) k)+
i
h
Results. To see how useful analogy-making is for gen2
Ldif f = EgA,B ∼Gdif f (τdif f − k∆ (gA , gB ) k)+ ,
eralization to unseen parameterized tasks, we trained and
evaluated the parameterized skill on three different sets of
where (·)+ = max(0, ·) and Gsim , Gdis , Gdif f are sets of
parameterized tasks defined below1 .
task parameters that satisfy corresponding conditions in the

above three constraints. The final analogy-making objective is the weighted sum of the above three objectives.
3.2. Training
The parameterized skill is trained on a set of tasks (G 0 ⊂ G)
through the actor-critic method with generalized advantage
estimation (Schulman et al., 2016). We also found that
pre-training through policy distillation (Rusu et al., 2016;
Parisotto et al., 2016) gives slightly better results as discussed in Tessler et al. (2017). Throughout training, the
parameterized skill is also made to predict whether the current state is terminal or not through a binary classification
objective, and the analogy-making objective is applied to
the task embedding separately. The full details of the learning objectives are described in the Appendix.
3.3. Experiments
Environment. We developed a 3D visual environment
using Minecraft based on Oh et al. (2016) as shown in
Figure 1. An observation is represented as a 64 × 64
pixel RGB image. There are 7 different types of objects:
Pig, Sheep, Greenbot, Horse, Cat, Box, and Ice. The
topology of the world and the objects are randomly generated for every episode. The agent has 9 actions: Look
(Left/Right/Up/Down), Move (Forward/Backward), Pick
up, Transform, and No operation. Pick up removes the object in front of the agent, and Transform changes the object
in front of the agent to ice (a special object).
Implementation Details. The network architecture of
the parameterized skill consists of 4 convolution layers
and one LSTM (Hochreiter and Schmidhuber, 1997) layer.
We conducted curriculum training by changing the size
of the world, the density of object and walls according
to the agent’s success rate. We implemented actor-critic
method with 16 CPU threads based on Sukhbaatar et al.
(2015). The parameters are updated after 8 episodes for
each thread. The details of architectures and hyperparameters are described in the Appendix.

• Independent: The task space is defined as G = T × X ,
where T = {Visit, Pick up, Transform} and X is the set
of object types. The agent should move on top of the
target object given ‘Visit’ task and perform the corresponding actions in front of the target given ‘Pick up’
and ‘Transform’ tasks. Only a subset of tasks are encountered during training, so the agent should generalize
over unseen configurations of task parameters.
• Object-dependent: The task space is defined as G =
T 0 × X , where T 0 = T ∪ {Interact with}. We divided
objects into two groups, each of which should be either
picked up or transformed given ‘Interact with’ task. Only
a subset of target object types are encountered during
training, so there is no chance for the agent to generalize
without knowledge of the group of each object. We applied analogy-making so that analogies can be made only
within the same group. This allows the agent to perform
object-dependent actions even for unseen objects.
• Interpolation/Extrapolation: The task space is defined
as G = T × X × C, where C = {1, 2, ..., 7}. The
agent should perform a task for a given number of times
(c ∈ C). Only {1, 3, 5} ⊂ C is given during training,
and the agent should generalize over unseen numbers
{2, 4, 6, 7}. Note that the optimal policy for a task can be
derived from T × X , but predicting termination requires
generalization to unseen numbers. We applied analogymaking based on arithmetic (e.g., [Pick up, X, 2] : [Pick
up, X, 5] :: [Transform, Y, 3] : [Transform, Y, 6]).
As summarized in Table 1, the parameterized skill with
our analogy-making objective can successfully generalize
to unseen tasks in all generalization scenarios. This suggests that when learning a representation of task parameters, it is possible to inject prior knowledge in the form of
the analogy-making objective so that the agent can learn to
1
The sets of subtasks used for training and evaluation are described in the Appendix.

Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
Instructions

Instruction
memory

Observation

Context

Subtask
updater

Retrieved
instruction

Update

Recurrence

Yes
No

Subtask
parameters

Figure 3: Overview of our hierarchical architecture.

Subtask
parameters

Subtask
termination?

Figure 4: Neural network architecture of meta controller.

generalize over unseen tasks in various ways depending on
semantics or context without needing to experience them.

4. Learning to Execute Instructions using
Parameterized Skill
We now consider the instruction execution problem where
the agent is given a sequence of simple natural language
instructions, as illustrated in Figure 1. We assume an already trained parameterized skill, as described in Section 3.
Thus, the main remaining problem is how to use the parameterized skill to execute instructions. Although the requirement that instructions be executed sequentially makes
the problem easier (than, e.g., conditional-instructions), the
agent still needs to make complex decisions because it
should deviate from instructions to deal with unexpected
events (e.g., low battery) and remember what it has done to
deal with loop instructions, as discussed in Section 1.
To address the above challenges, our hierarchical RL architecture (see Figure 3) consists of two modules: meta controller and parameterized skill. Specifically, a meta controller reads the instructions and passes subtask parameters
to a parameterized skill which executes the given subtask
and provides its termination signal back to the meta controller. Section 4.1 describes the overall architecture of the
meta controller for dealing with instructions. Section 4.2
describes a novel neural architecture that learns when to
update the subtask in order to better deal with delayed reward signal as well as unexpected events.
4.1. Meta Controller Architecture
As illustrated in Figure 4, the meta controller is a mapping
O × M × G × B → G, where M is a list of instructions.
Intuitively, the meta controller decides subtask parameters
gt ∈ G conditioned on the observation xt ∈ O, the list of
instructions M ∈ M, the previously selected subtask gt−1 ,
and its termination signal (b ∼ βφ ).
In contrast to recent hierarchical deep RL approaches
where the meta controller can update its subtask (or option)
only when the previous one terminates or only after a fixed
number of steps, our meta controller can update the subtask
at any time and takes the termination signal as additional
input. This gives more flexibility to the meta controller and

enables interrupting ongoing tasks before termination.
In order to keep track of the agent’s progress on instruction
execution, the meta controller maintains its internal state by
computing a context vector (Section 4.1.1) and determines
which subtask to execute by focusing on one instruction at
a time from the list of instructions (Section 4.1.2).
4.1.1. C ONTEXT
Given the sentence embedding rt−1 retrieved at the previous time-step from the instructions (described in Section 4.1.2), the previously selected subtask
 gt−1 , and the
subtask termination bt ∼ βφ bt |st , gt−1 , the meta controller computes the context vector (ht ) as follows:
ht = LSTM (st , ht−1 )

st = f xt , rt−1 , gt−1 , bt ,
where f is a neural network. Intuitively, gt−1 and bt provide information about which subtask was being solved by
the parameterized skill and whether it has finished or not.
Thus, st is a summary of the current observation and the
ongoing subtask. ht takes the history of st into account
through the LSTM, which is used by the subtask updater.
4.1.2. S UBTASK U PDATER
The subtask updater constructs a memory structure from
the list of instructions, retrieves an instruction by maintaining a pointer into the memory, and computes the subtask
parameters.
Instruction Memory. Given instructions as a list of sentences M = (m1 , m2 , ..., mK ), where each sentence consists of a list of words, mi = w1 , ..., w|mi | , the subtask
updater constructs memory blocks M ∈ RE×K (i.e., each
column is an E-dimensional embedding of a sentence).
The subtask updater maintains an instruction pointer (pt ∈
RK ) which is non-negative and sums up to 1 indicating
which instruction the meta controller is executing. Memory construction and retrieval can be written as:
Memory: M = [ϕw (m1 ) , ϕw (m2 ) , ..., ϕw (mK )] (1)
Retrieval: rt = Mpt ,

(2)

Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
Update
Subtask

Update
Subtask

Copy

Copy

Copy

meta-controller policy that is not able to interrupt ongoing subtasks before termination, which is necessary to deal
with unexpected events not specified in the instructions.

Update

Figure 5: Unrolled illustration of the meta controller with a
learned time-scale. The internal states (p, r, h) and the subtask
(g) are updated only when c = 1. If c = 0, the meta controller
continues the previous subtask without updating its internal states.

where ϕw (mi ) ∈ RE is the embedding of the i-th sentence
(e.g., Bag-of-words), and rt ∈ RE is the retrieved sentence
embedding which is used for computing the subtask parameters. Intuitively, if pt is a one-hot vector, rt indicates
a single instruction from the whole list of instructions. The
meta controller should learn to manage pt so that it can
focus on the correct instruction at each time-step.
Since instructions should be executed sequentially, we use
a location-based memory addressing mechanism (Zaremba
and Sutskever, 2015; Graves et al., 2014) to manage the
instruction pointer. Specifically, the subtask updater shifts
the instruction pointer by [−1, 1] as follows:

pt = lt ∗ pt−1 where lt = Softmax ϕshif t (ht ) ,

(3)

where ∗ is a convolution operator, ϕshif t is a neural network, and lt ∈ R3 is a soft-attention vector over the three
shift operations {−1, 0, +1}. The optimal policy should
keep the instruction pointer unchanged while executing an
instruction and increase the pointer by +1 precisely when
the current instruction is finished.
Subtask Parameters. The subtask updater takes the context (ht ), updates the instruction pointer (pt ), retrieves an
instruction (rt ), and computes subtask parameters as:

Y  (i)
πθ (gt |ht , rt ) =
πθ gt |ht , rt ,
(4)
i





(i)
where πθ gt |ht , rt ∝ exp ϕgoal
(ht , rt ) , and ϕgoal
i
i
is a neural network for the i-th subtask parameter.
4.2. Learning to Operate at a Large Time-Scale
Although the meta controller can learn an optimal policy by
updating the subtask at each time-step in principle, making
a decision at every time-step can be inefficient because subtasks do not change frequently. Instead, having temporallyextended actions can be useful for dealing with delayed
reward by operating at a larger time-scale (Sutton et al.,
1999). While it is reasonable to use the subtask termination
signal to define the temporal scale of the meta controller as
in many recent hierarchical deep RL approaches (see Section 2), this approach would result in a mostly open-loop

To address this dilemma, we propose to learn the time-scale
of the meta controller by introducing an internal binary decision which indicates whether to invoke the subtask updater to update the subtask or not, as illustrated in Figure 5.
This decision is defined as: ct ∼ σ ϕupdate (st , ht−1 )
where σ is a sigmoid function. If ct = 0, the meta controller continues the current subtask without updating the
subtask updater. Otherwise, if ct = 1, the subtask updater
updates its internal states (e.g., instruction pointer) and the
subtask parameters. This allows the subtask updater to operate at a large time-scale because one decision made by
the subtask updater results in multiple actions depending
on c values. The overall meta controller architecture with
this update scheme is illustrated in Figure 4.
Soft-Update. To ease optimization of the nondifferentiable variable (ct ), we propose a soft-update
rule by using ct = σ ϕupdate (st , ht−1 ) instead of
sampling it. The key idea is to take the weighted sum of
both ‘update’ and ‘copy’ scenarios using ct as the weight.
This method is described in Algorithm 1. We found that
training the meta controller using soft-update followed
by fine-tuning by sampling ct is crucial for training the
meta controller. Note that the soft-update rule reduces to
the original formulation if we sample ct and lt from the
Bernoulli and multinomial distributions, which justifies
our initialization trick.
Algorithm 1 Subtask update (Soft)
Input: st , ht−1 , pt−1 , rt−1 , gt−1
Output: ht , pt , rt , gt

ct ← σ ϕupdate (st , ht−1 )
# Decide update weight
h̃t ← LSTM (st , ht−1 ) 
# Update the context
lt ← Softmax ϕshif t h̃t
# Decide shift operation
p̃t ← lt ∗ pt−1
# Shift the instruction pointer
r̃t ← Mp̃t
# Retrieve instruction
# Merge two scenarios (update/copy) using ct as weight
[pt , rt , ht ] ← ct [p̃t , r̃t , h̃t ] + (1 − ct ) [pt−1 , rt−1 , ht−1 ]


(i)
(i)
(i)
gt ∼ ct πθ gt |h̃t , r̃t + (1 − ct ) gt−1 ∀i

Integrating with Hierarchical RNN. The idea of learning the time-scale of a recurrent neural network is closely
related to hierarchical RNN approaches (Koutnik et al.,
2014; Chung et al., 2017) where different groups of recurrent hidden units operate at different time-scales to capture
both long-term and short-term temporal information. Our
idea can be naturally integrated with hierarchical RNNs by
applying the update decision (c value) only for a subset of
recurrent units instead of all the units. Specifically,
we dih
i
(l)
(h)
vide the context vector into two groups: ht = ht , ht .
(l)

The low-level units (ht ) are updated at every time-step,

Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
(h)

while the high-level units (ht ) are updated depending on
the value of c. This simple modification leads to a form of
hierarchical RNN where the low-level units focus on shortterm temporal information while the high-level units capture long-term dependencies.
4.3. Training
The meta controller is trained on a training set of lists of
instructions. Given a pre-trained and fixed parameterized
skill, the actor-critic method is used to update the parameters of the meta controller. Since the meta controller also
learns a subtask embedding ϕ(gt−1 ) and has to deal with
unseen subtasks during evaluation, analogy-making objective is also applied. The details of the objective function
are provided in the Appendix.
4.4. Experiments
The experiments are designed to explore the following
questions: (1) Will the proposed hierarchical architecture
outperform a non-hierarchical baseline? (2) How beneficial is the meta controller’s ability to learn when to update
the subtask? We are also interested in understanding the
qualitative properties of our agent’s behavior.2
Environment. We used the same Minecraft domain used
in Section 3.3. The agent receives a time penalty (−0.1)
for each step and receives +1 reward when it finishes the
entire list of instructions in the correct order. Throughout
an episode, a box (including treasures) randomly appears
with probability of 0.03 and transforming a box gives +0.9
reward.
The subtask space is defined as G = T ×X , and the semantics of each subtask are the same as the ‘Independent’ case
in Section 3.3. We used the best-performing parameterized
skill throughout this experiment.
There are 7 types of instructions: {Visit X, Pick up X,
Transform X, Pick up 2 X, Transform 2 X, Pick up 3 X,
Transform 3 X} where ‘X’ is the target object type. Note
that the parameterized skill used in this experiment was
not trained on loop instructions (e.g., Pick up 3 X), so the
last four instructions require the meta controller to learn to
repeat the corresponding subtask for the given number of
times. To see how the agent generalizes to previously unseen instructions, only a subset of instructions and subtasks
was presented during training.
Implementation Details. The meta controller consists of
3 convolution layers and one LSTM layer. We also conducted curriculum training by changing the size of the
world, the density of object and walls, and the number of
instructions according to the agent’s success rate. We used
2

For further analysis, we also conducted comprehensive experiments on a 2D grid-world domain. However, due to space
limits, those results are provided in the Appendix.

Length of instructions
Flat
Hierarchical-Long
Hierarchical-Short
Hierarchical-Dynamic

Train
4
-7.1 (1%)
-5.8 (31%)
-3.3 (83%)
-3.1 (95%)

Test (Seen) Test (Unseen)
20
20
-63.6 (0%) -62.0 (0%)
-59.2 (0%) -59.2 (0%)
-53.4 (23%) -53.6 (18%)
-30.3 (75%) -38.0 (56%)

Table 2: Performance on instruction execution. Each entry shows
average reward and success rate. ‘Hierarchical-Dynamic’ is our
approach that learns when to update the subtask. An episode is
successful only when the agent solves all instructions correctly.

the actor-critic implementation described in Section 3.3.
Baselines. To understand the advantage of using the hierarchical structure and the benefit of our meta controller’s
ability to learn when to update the subtask, we trained three
baselines as follows.
• Flat: identical to our meta controller except that it directly chooses primitive actions without using the parameterized skill. It is also pre-trained on the training
set of subtasks.
• Hierarchical-Long: identical to our architecture except
that the meta controller can update the subtask only when
the current subtask is finished. This approach is similar
to recent hierarchical deep RL methods (Kulkarni et al.,
2016; Tessler et al., 2017).
• Hierarchical-Short: identical to our architecture except that the meta controller updates the subtask at every
time-step.
Overall Performance. The results on the instruction execution are summarized in Table 7 and Figure 10. It shows
that our architecture (‘Hierarchical-Dynamic’) can handle a relatively long list of seen and unseen instructions of
length 20 with reasonably high success rates, even though
it is trained on short instructions of length 4. Although
the performance degrades as the number of instructions
increases, our architecture finishes 18 out of 20 seen instructions and 14 out of 20 unseen instructions on average. These results show that our agent is able to generalize
to longer compositions of seen/unseen instructions by just
learning to solve short sequences of a subset of instructions.
Flat vs. Hierarchy. Table 7 shows that the flat baseline
completely fails even on training instructions. The flat controller tends to struggle with loop instructions (e.g., Pick up
3 pig) so that it learned a sub-optimal policy which moves
to the next instruction with a small probability at each step
regardless of its progress. This implies that it is hard for the
flat controller to detect precisely when a subtask is finished,
whereas hierarchical architectures can easily detect when a
subtask is done, because the parameterized skill provides a
termination signal to the meta controller.
Effect of Learned Time-Scale. As shown in Table 7 and
Figure 10, ‘Hierarchical-Long’ baseline performs significantly worse than our architecture. We found that whenever

Success rate

Reward

−20
−30
−40
−50
−60
−70

5
10
15
20
Num of instructions

1.0

600

0.8

500

0.6

#steps

0
−10

0.4
0.2
0.0

#instructions completed

Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning

400
300
200
100

5
10
15
20
Num of instructions

0

5
10
15
20
Num of instructions

18
16
14
12
10
8
6
4
2
0

Flat (Seen)
Flat (Unseen)
Hierarchical-Long (Seen)
Hierarchical-Long (Unseen)
Hierarchical-Short (Seen)
Hierarchical-Short (Unseen)
Hierarchical-Dynamic (Seen)
Hierarchical-Dynamic (Unseen)

5
10
15
Num of instructions

20

Figure 6: Performance per number of instructions. From left to right, the plots show reward, success rate, the number of steps, and the
average number of instructions completed respectively.
Subtask
Update

AB

C

D
-1
0
+1

Shift
Instruction

A

B

C

D

Figure 7: Analysis of the learned policy. ‘Update’ shows our agent’s internal update decision. ‘Shift’ shows our agent’s instruction-shift
decision (-1, 0, and +1 from top to bottom). The bottom text shows the instruction indicated by the instruction pointer, while the top text
shows the subtask chosen by the meta controller. (A) the agent picks up the pig to finish the instruction and moves to the next instruction.
(B) When the agent observes a box that randomly appeared while executing ‘Pick up 2 pig’ instruction, it immediately changes its subtask
to [Transform, Box]. (C) After dealing with the event (transforming a box), the agent resumes executing the instruction (‘Pick up 2 pig’).
(D) The agent finishes the final instruction.

a subtask is finished, this baseline puts a high probability
to switch to [Transform, Box] regardless of the existence
of box because transforming a box gives a bonus reward if
a box exists by chance. However, this leads to wasting too
much time finding a box until it appears and results in a
poor success rate due to the time limit. This result implies
that an open-loop policy that has to wait until a subtask finishes can be confused by such an uncertain event because
it cannot interrupt ongoing subtasks before termination.
On the other hand, we observed that ‘Hierarchical-Short’
often fails on loop instructions by moving on to the next
instruction before it finishes such instructions. This baseline should repeat the same subtask while not changing the
instruction pointer for a long time and the reward is even
more delayed given loop instructions. In contrast, the subtask updater in our architecture makes fewer decisions by
operating at a large time-scale so that it can get more direct
feedback from the long-term future. We conjecture that this
is why our architecture performs better than this baseline.
This result shows that learning when to update the subtask
using the neural network is beneficial for dealing with delayed reward without compromising the ability to interrupt.
Analysis of The Learned Policy. We visualized our
agent’s behavior given a long list of instructions in Figure 7. Interestingly, when the agent sees a box, the meta
controller immediately changes its subtask to [Transform,
Box] to get a positive reward even though its instruction

pointer is indicating ‘Pick up 2 pig’ and resumes executing the instruction after dealing with the box. Throughout this event and the loop instruction, the meta controller
keeps the instruction pointer unchanged as illustrated in
(B-C) in Figure 7. In addition, the agent learned to update the instruction pointer and the subtask almost only
when it is needed, which provides the subtask updater with
temporally-extended actions. This is not only computationally efficient but also useful for learning a better policy.

5. Conclusion
In this paper, we explored a type of zero-shot task generalization in RL with a new problem where the agent is required to execute and generalize over sequences of instructions. We proposed an analogy-making objective which
enables generalization over unseen parameterized tasks in
various scenarios. We also proposed a novel way to learn
the time-scale of the meta controller that proved to be more
efficient and flexible than alternative approaches for interrupting subtasks and for dealing with delayed sequential
decision problems. Our empirical results on a stochastic
3D domain showed that our architecture generalizes well
to longer sequences of instructions as well as unseen instructions. Although our hierarchical RL architecture was
demonstrated in the simple setting where the set of instructions should be executed sequentially, we believe that our
key ideas are not limited to this setting but can be extended
to richer forms of instructions.

Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning

Acknowledgement
This work was supported by NSF grant IIS-1526059. Any
opinions, findings, conclusions, or recommendations expressed here are those of the authors and do not necessarily
reflect the views of the sponsor.

R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR,
2006.

References

N. Heess, G. Wayne, Y. Tassa, T. P. Lillicrap, M. A. Riedmiller, and D. Silver. Learning and transfer of modulated
locomotor controllers. arXiv preprint arXiv:1610.05182,
2016.

D. Andre and S. J. Russell. Programmable reinforcement
learning agents. In NIPS, 2000.

S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.

D. Andre and S. J. Russell.
State abstraction for
programmable reinforcement learning agents.
In
AAAI/IAAI, 2002.

D. Isele, M. Rostami, and E. Eaton. Using task features
for zero-shot knowledge transfer in lifelong learning. In
IJCAI, 2016.

J. Andreas, D. Klein, and S. Levine. Modular multitask
reinforcement learning with policy sketches. CoRR,
abs/1611.01796, 2016.

V. R. Konda and J. N. Tsitsiklis. Actor-critic algorithms. In
NIPS, 1999.

P.-L. Bacon, J. Harb, and D. Precup. The option-critic architecture. In AAAI, 2017.
L. Bertinetto, J. F. Henriques, J. Valmadre, P. H. Torr, and
A. Vedaldi. Learning feed-forward one-shot learners.
arXiv preprint arXiv:1606.05233, 2016.
S. R. K. Branavan, H. Chen, L. S. Zettlemoyer, and
R. Barzilay. Reinforcement learning for mapping instructions to actions. In ACL/IJCNLP, 2009.
D. L. Chen and R. J. Mooney. Learning to interpret natural
language navigation instructions from observations. In
AAAI, 2011.
J. Chung, S. Ahn, and Y. Bengio. Hierarchical multiscale
recurrent neural networks. In ICLR, 2017.
B. C. da Silva, G. Konidaris, and A. G. Barto. Learning
parameterized skills. In ICML, 2012.
C. Devin, A. Gupta, T. Darrell, P. Abbeel, and S. Levine.
Learning modular neural network policies for multi-task
and multi-robot transfer. In ICRA, 2017.
T. G. Dietterich. Hierarchical reinforcement learning with
the maxq value function decomposition. Journal of Artificial Intelligence Research, 13:227–303, 2000.
C. Florensa, Y. Duan, and P. Abbeel. Stochastic neural networks for hierarchical reinforcement learning. In ICLR,
2017.
M. Ghavamzadeh and S. Mahadevan. Hierarchical policy
gradient algorithms. In ICML, 2003.
A. Graves, G. Wayne, and I. Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014.

G. Konidaris and A. G. Barto. Building portable options:
Skill transfer in reinforcement learning. In IJCAI, 2007.
G. Konidaris, I. Scheidwasser, and A. G. Barto. Transfer
in reinforcement learning via shared features. Journal of
Machine Learning Research, 13:1333–1371, 2012.
J. Koutnik, K. Greff, F. Gomez, and J. Schmidhuber. A
clockwork rnn. In ICML, 2014.
T. D. Kulkarni, K. R. Narasimhan, A. Saeedi, and J. B.
Tenenbaum. Hierarchical deep reinforcement learning:
Integrating temporal abstraction and intrinsic motivation. arXiv preprint arXiv:1604.06057, 2016.
J. Lei Ba, K. Swersky, S. Fidler, et al. Predicting deep
zero-shot convolutional neural networks using textual
descriptions. In ICCV, 2015.
M. MacMahon, B. Stankiewicz, and B. Kuipers. Walk
the talk: Connecting language, knowledge, and action
in route instructions. In AAAI, 2006.
A. McGovern and A. G. Barto. Autonomous discovery of
temporal abstractions from interaction with an environment. PhD thesis, University of Massachusetts, 2002.
H. Mei, M. Bansal, and M. R. Walter. Listen, attend, and
walk: Neural mapping of navigational instructions to action sequences. arXiv preprint arXiv:1506.04089, 2015.
R. Memisevic and G. E. Hinton. Learning to represent spatial transformations with factored higher-order
boltzmann machines. Neural Computation, 22(6):1473–
1492, 2010.
T. Mikolov, Q. V. Le, and I. Sutskever. Exploiting similarities among languages for machine translation. arXiv
preprint arXiv:1309.4168, 2013.

Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning

J. Oh, X. Guo, H. Lee, R. L. Lewis, and S. Singh. Actionconditional video prediction using deep networks in atari
games. In NIPS, 2015.
J. Oh, V. Chockalingam, S. P. Singh, and H. Lee. Control
of memory, active perception, and action in minecraft. In
ICML, 2016.
E. Parisotto, J. L. Ba, and R. Salakhutdinov. Actor-mimic:
Deep multitask and transfer reinforcement learning. In
ICLR, 2016.
R. Parr and S. J. Russell. Reinforcement learning with hierarchies of machines. In NIPS, 1997.
S. E. Reed, Y. Zhang, Y. Zhang, and H. Lee. Deep visual
analogy-making. In NIPS, 2015.
A. A. Rusu, S. G. Colmenarejo, C. Gulcehre, G. Desjardins, J. Kirkpatrick, R. Pascanu, V. Mnih,
K. Kavukcuoglu, and R. Hadsell.
Policy distillation. In ICLR, 2016.
T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal
value function approximators. In ICML, 2015.
J. Schulman, P. Moritz, S. Levine, M. Jordan, and
P. Abbeel. High-dimensional continuous control using
generalized advantage estimation. In ICLR, 2016.
S. P. Singh. The efficient learning of multiple task sequences. In NIPS, 1991.
S. P. Singh. Transfer of learning by composing solutions of
elemental sequential tasks. Machine Learning, 8(3-4):
323–339, 1992.
S. Sukhbaatar, A. Szlam, G. Synnaeve, S. Chintala, and
R. Fergus. Mazebase: A sandbox for learning from
games. arXiv preprint arXiv:1511.07401, 2015.
R. S. Sutton, D. Precup, and S. Singh. Between mdps
and semi-mdps: A framework for temporal abstraction
in reinforcement learning. Artificial intelligence, 112(1):
181–211, 1999.
S. Tellex, T. Kollar, S. Dickerson, M. R. Walter, A. G.
Banerjee, S. J. Teller, and N. Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In AAAI, 2011.
S. Tellex, R. A. Knepper, A. Li, D. Rus, and N. Roy. Asking
for help using inverse semantics. In RSS, 2014.
C. Tessler, S. Givony, T. Zahavy, D. J. Mankowitz, and
S. Mannor. A deep hierarchical approach to lifelong
learning in minecraft. In AAAI, 2017.
W. Zaremba and I. Sutskever. Reinforcement learning neural turing machines. arXiv preprint arXiv:1505.00521,
2015.

Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning

A. Experimental Setting for Parameterized Tasks
The episode terminates after 50 steps for ‘Independent’ and ‘Object-dependent’ cases and 70 steps for ‘Inter/Extrapolation’
case. The agent receives a positive reward (+1) when it successfully finishes the given task and receives a time penalty of
−0.1 for each step. The details of each generalization scenario are described below.
Independent. The semantics of the tasks are consistent across all types of target objects. The training set of tasks is
shown in Table 3. Examples of analogies used in this experiment are:
• [Visit, X] : [Visit, Y] :: [Transform, X] : [Transform, Y]
• [Visit, X] : [Visit, Y] 6= [Transform, X] : [Pick up, Y]
• [Visit, X] 6= [Visit, Y],
where X and Y can be any object type (X 6= Y).
Object-dependent. We divided objects into two groups: Group A and Group B. Given ‘Interact with’ action, Group A
should be picked up, whereas Group B should be transformed by the agent. The training set of tasks with groups for each
object is shown in Table 4. Examples of analogies used in this experiment are:
• [Visit, Sheep] : [Visit, Cat] :: [Interact with, Sheep] : [Interact with, Cat]
• [Visit, Sheep] : [Visit, Greenbot] 6= [Interact with, Sheep] : [Interact with, Greenbot]
• [Visit, Horse] : [Visit, Greenbot] :: [Interact with, Horse] : [Interact with, Greenbot]
Note that the second example implies that Sheep and Greenbot should be treated in a different way because they belong to
different groups. Given such analogies, the agent can learn to interact with Cat as it interacts with Sheep and interact with
Greenbot as it interacts with Horse.
Inter/Extrapolation. In this experiment, a task is defined by three parameters: action, object, and number. The agent
should repeat the same subtask for a given number of times. The agent is trained on all configurations of actions and target
objects. However, only a subset of numbers is used during training. In order to interpolate and extrapolate, we define
analogies based on simple arithmetic such as:
• [Pick up, X, 1] : [Pick up, X, 6] :: [Pick up, X, 2] : [Pick up, X, 7]
• [Pick up, X, 3] : [Pick up, X, 6] :: [Transform, Y, 4] : [Transform, Y, 7]
• [Pick up, X, 1] : [Pick up, X, 3] 6= [Transform, Y, 2] : [Transform, Y, 3]

Sheep
Horse
Pig
Box
Cat
Greenbot

Visit
X
X
X
X

Pick up
X
X
X
X

Transform
X
X
X
X

Table 3: Training set of tasks for the ‘Independent’ case.

Sheep (A)
Horse (B)
Pig (A)
Box (B)
Cat (A)
Greenbot (B)

Visit
X
X
X
X
X
X

Pick up
X
X
X
X
X
X

Transform
X
X
X
X
X
X

Interact with
X
X
X
X

Table 4: Training set of tasks for the ‘Object-dependent’ case.

Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning

B. Experimental Setting for Instruction Execution
The episode terminates after 80 steps during training of the meta controller. A box appears with probability of 0.03 and
disappears after 20 steps. During evaluation on longer instructions, the episode terminates after 600 steps. We constructed
a training set of instructions and an unseen set of instructions as described in Table 5. We generated different sequences of
instructions for training and evaluation by sampling instructions from such sets of instructions.

Sheep
Horse
Pig
Cat
Greenbot

Visit
X
X
X

Pick up

Pick up 2

Pick up 3

X
X
X

X
X
X

X
X
X

Transform
X
X

Transform 2
X
X

Transform 3
X
X

X
X

X
X

X
X

Table 5: Training set of instructions.

C. Experiment on 2D Grid-World
Environment. To see how our approach can be generally applied to different domains, we developed a 2D
grid-world based on MazeBase (Sukhbaatar et al., 2015)
where the agent can interact with many objects, as illustrated in Figure 8. Unlike the original MazeBase,
an observation is represented as a binary 3D tensor:
xt ∈ R18×10×10 where 18 is the number of object types
and 10 × 10 is the size of the grid world. Each channel
is a binary mask indicating the presence of each object
type. There are agent, blocks, water, and 15 types of objects with which the agent can interact, and all of them
are randomly placed for each episode.

Block

Cow

Milk

Water

Pig

Meat

Agent

Rock

Stone

Tree

Wood

Box

Diamond

Duck

Egg

Enemy
Candy

Heart

Figure 8: Example of 2D grid-world with object specification. The
arrows represent the outcome of object transformation. Objects
without arrows disappear when transformed. The agent is not allowed to go through blocks and gets a penalty for going through
water.

The agent has 13 primitive actions: No-operation, Move
(North/South/West/East, referred to as “NSWE”), Pick
up (NSWE), and Transform (NSWE). Move actions move the agent by one cell in the specified direction. Pick up actions
remove the adjacent object in the corresponding relative position, and Transform actions either remove it or transform it to
another object depending on the object type, as shown in Figure 8.
The agent receives a time penalty (−0.1) for each time-step. Water cells act as obstacles which give −0.3 when the agent
visits them. The agent receives +1 reward when it finishes all instructions in the correct order. Throughout the episode, an
enemy randomly appears, moves, and disappears after 10 steps. Transforming an enemy gives +0.9 reward.
Evaluation on Parameterized Tasks. As in the main experiment, we evaluated the parameterized skill on seen and
unseen parameterized tasks separately using the same generalization scenarios. As shown in Table 6, the parameterized
skill with analogy-making can successfully generalize to unseen tasks both in independent and object-dependent scenarios.
Scenario

Analogy
×
Independent
X
×
Object-dependent
X

Train
0.56 (99.9%)
0.56 (99.9%)
0.55 (99.9%)
0.55 (99.9%)

Unseen
-1.88 (49.6%)
0.55 (99.6%)
-3.23 (43.2%)
0.55 (99.5%)

Table 6: Performance on parameterized tasks. Each entry shows ‘Average reward (Success
rate)’. We assume an episode is successful only if the agent successfully finishes the task and
its termination predictions are correct throughout the whole episode.

We visualized the value function learned by the critic network of the parameterized skill in Figure 9. As expected from its
generalization performance, our parameterized skill trained with analogy-making objective learned high values around the
target objects given unseen tasks.

Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning

(a) Observation

(b) Visit egg

(c) Pick up cow

(d) Transform meat

Figure 9: Value function visualization given unseen tasks. (b-d) visualizes learned values for each position of the agent in a grid world
(a). The agent estimates high values around the target object in the world.

Evaluation on Instruction Execution. There are 5 types of instructions: Visit X, Pick up X, Transform X, Pick up all
X, and Transform all X, where ‘X’ is the target object type. While the first three instructions require the agent to perform
the corresponding subtask, the last two instructions require the agent to repeat the same subtask until the target objects
completely disappear from the world.

250

0.8

200

0.6

150

−5
−10
−15
−20

5
10
15
20
Num of instructions

0.4
0.2
0.0

#instructions completed

1.0

0

#steps

5
Success rate

Reward

The overall result is consistent with the result on 3D environment as shown in Figure 10 and Table 7. The flat baseline
learned a sub-optimal policy that transforms or picks up all target objects in the world even if there is no ‘all’ adverb in
the instructions. This sub-optimal policy unnecessarily removes objects that can be potentially target objects in the future
instructions. This is why the performance of the flat baseline drastically decreases as the number of instructions increases
in Figure 10. Our architecture with learned time-scale (‘Hierarchical-Dynamic’) performs much better than ‘HierarchicalShort’ baseline which updates the subtask at every time-step. This also suggests that learning to operate in a large-time
scale is crucial for dealing with delayed reward. We also observed that our agent learned to deal with enemies whenever
they appear, and thus it outperforms the ‘Shortest Path’ method which is near-optimal in executing instructions while
ignoring enemies.

100
50

5
10
15
20
Num of instructions

0

5
10
15
20
Num of instructions

20
Near-Optimal
Flat (Seen)
Flat (Unseen)
Hierarchical-Short (Seen)
Hierarchical-Short (Unseen)
Hierarchical-Dynamic (Seen)
Hierarchical-Dynamic (Unseen)

15
10
5
0

5
10
15
Num of instructions

20

Figure 10: Performance per number of instructions.

#Instructions
Shortest Path
Near-Optimal
Flat
Hierarchical-Short
Hierarchical-Dynamic

Train
Test (Seen)
Test (Unseen)
4
20
20
-1.62 (99.7%)
-11.94 (99.4%)
-1.34 (99.5%)
-10.30 (99.3%)
-2.38 (76.0%) -18.83 (0.1%) -18.92 (0.0%)
-1.74 (81.0%) -15.89 (28.0%) -17.23 (11.3%)
-1.26 (95.5%) -11.30 (81.3%) -14.75 (40.3%)

Table 7: Performance of meta controller. Each entry in the table represents reward with success rate in parentheses averaged over 10-best
runs among 20 independent runs. ‘Shortest Path’ is a hand-designed policy which executes instructions optimally based on the shortest
path but ignores enemies. ‘Near-Optimal’ is a near-optimal policy that executes instructions based the shortest path and transforms
enemies when they are close to the agent.

Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning

D. Details of Learning Objectives
D.1. Parameterized Skill
The parameterized skill is first trained through policy distillation (Rusu et al., 2016; Parisotto et al., 2016) and fine-tuned using actor-critic method (Konda and Tsitsiklis, 1999) with generalized advantage estimation (GAE) (Schulman et al., 2016).
The parameterized skill is also trained to predict whether the current state is terminal or not through binary classification
objective.
The idea of policy distillation is to first train separate teacher policies (πTg (a|s)) for each task (g) through reinforcement
learning and train a multi-task policy (πφg (a|s)) to mimic teachers’ behavior by minimizing KL divergence between them
as follows:
h
h


ii
∇φ LRL = Eg∼U Es∼πφg ∇φ DKL πTg ||πφg + α∇φ Lterm ,
(5)


P
π g (a|s)
where DKL πTg ||πφg = a πTg (a|s) log πTg (a|s) and U ⊂ G is the training set of tasks. Lterm = − log βφ (st , g) =
φ

− log Pφ (st ∈ Tg ) is the cross-entropy loss for termination prediction. Intuitively, we sample a mini-batch of tasks (g),
use the parameterized skill to generate episodes, and train it to predict teachers’ actions. This method has been shown to
be efficient for multi-task learning.
After policy distillation, the parameterized skill is fine-tuned through actor-critic with generalized advantage estimation
(GAE) (Schulman et al., 2016) as follows:
h
h
ii
(γ,λ)
∇φ LRL = Eg∼U Es∼πφg −∇φ log πφ (at |st , g) Ât
+ α∇φ Lterm ,
(6)
P∞
(γ,λ)
V
where Ât
=
(γλ)l δt+l
and δtV = rt + γV π (st+1 ; φ0 ) − V π (st ; φ0 ). φ0 is optimized to minimize
h
il=0
2
E (Rt − V π (st ; φ0 )) . γ, λ ∈ [0, 1] are a discount factor and a weight for balancing between bias and variance of
the advantage estimation.
The final update rule for the parameterized skill is:
∆φ ∝ − (∇φ LRL + ξ∇φ LAM ) ,

(7)

where LAM = Lsim +ρ1 Ldis +ρ2 Ldif f is the analogy-making regularizer defined as the weighted sum of three objectives
described in the main text. ρ1 , ρ2 , ξ are hyperparameters for each objective.
D.2. Meta Controller
Actor-critic method with GAE is used to update the parameter of the meta controller as follows:
 h P



(i)
(γ,λ)

E ct

i ∇θ log πθ gt |ht , rt + ∇θ log P (lt |ht ) Ât


 i
(γ,λ)
∇θ LRL = − + ∇θ log P (ct |st , ht−1 ) Ât
+ η∇θ σ ϕupdate (st , ht−1 ) 1

h
h


i
i


E P ∇θ log ct πθ gt(i) |h̃t , r̃t + (1 − ct )g (i) Ât(γ,λ)
t−1
i

(Hard)
(8)
(Soft),



where ct ∼ P (ct |st , ht−1 ) ∝ σ ϕupdate (st , ht−1 ) , and P (lt |ht ) ∝ Softmax ϕshif t (ht ) . We applied L1-penalty to the
probability of update to penalize too frequent updates, and η is a weight for the update penalty.
The final update rule for the meta controller is:
∆θ ∝ − (∇θ LRL + ξ∇θ LAM ) ,
where LAM is the analogy-making objective.

(9)

Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning

E. Architectures and Hyperparameters
Background: Multiplicative Interaction For combining condition variables into a neural network (e.g., combining
task embedding into the convolutional network in the parameterized skill), we used a form of multiplicative interaction
instead of concatenating such variables as suggested by (Memisevic and Hinton, 2010; Oh et al., 2015). This is also
related to parameter prediction approaches where the parameters of the neural network is produced by condition variables
(e.g., exempler, class embedding). This approach has been shown to be effective for achieving zero-shot and one-shot
generalization in image classification problems (Lei Ba et al., 2015; Bertinetto et al., 2016). More formally, given an input
(x), the output (y) of a convolution and a fully-connected layer with parameters predicted by a condition variable (g) can
be written as:
Convolution: y = ϕ (g) ∗ x + b
Fully-connected: y = W0 diag (ϕ (g)) Wx + b,
where ϕ is the embedding of the condition variable learned by a multi-layer perceptron (MLP). Note that we use matrix
factorization (similar to (Memisevic and Hinton, 2010)) to reduce the number of parameters for the fully-connected layer.
Intuitively, the condition variable is converted to the weight of the convolution or fully-connected layer through multiplicative interactions. We used this approach both in the parameterized skill and the meta controller. The details are described
below.
Parameterized skill. The teacher architecture used for policy distillation is Conv1(32x8x8-4)-Conv2(64x5x5-2)LSTM(64).3 The network has two fully-connected output layers for actions and value (baseline) respectively. The parameterized skill architecture consists of Conv1(16x8x8-4)-Conv2(32x1x1-1)-Conv3(32x1x1-1)-Conv4(32x5x5-2)-LSTM(64).


The parameterized skill takes two task parameters (g = g (1) , g (2) ) as additional input and computes ϕ(g) =
ReLU(W(1) g (1) W(2) g (2) ) to compute the subtask embedding. It is further linearly transformed into the weights of
Conv3 and the (factorized) weight of LSTM through multiplicative interaction as described above. Finally, the network
has three fully-connected output layers for actions, termination probability, and baseline, respectively.
We used RMSProp optimizer with the smoothing parameter of 0.97 and epsilon of 1e − 6. When training the teacher policy
through actor-critic, we used a learning rate of 2.5e − 4. For training the parameterized skill, we used a learning rate of
2.5e − 4 and 1e − 4 for policy distillation and actor-critic fine-tuning respectively. We used τdis = τdif f = 3, α = 0.1
for analogy-making regularization and the termination prediction objective respectively. γ = 0.99 and λ = 0.96 are used
as a discount factor and a balancing weight for GAE. 16 threads with batch size of 8 are used to run 16 × 8 episodes in
parallel, and the parameter is updated after each run (1 iteration = 16 × 8 episodes). For better exploration, we applied
entropy regularization with a weight of 0.1 and linearly decreased it to zero for the first 7500 iterations. The total number
of iterations was 15,000 for both policy distillation and actor-critic fine-tuning.
Meta Controller. The meta controller consists of Conv1(16x8x8-4)-Conv2(32x1x1-1)-Conv3(32x1x1-1)-Pool(5)LSTM(256). The embedding of previously selected subtask (ϕ(gt−1 )), the previously retrieved instruction (rt−1 ), and
the subtask termination (bt ) are concatenated and given as input for one-layer MLP to compute a 256-dimensional joint
embedding. This is further linearly transformed into the weights of Conv3 and LSTM through multiplicative interaction.
The output of FC1 is used as the context vector (ht ). We used the bag-of-words (BoW) representation as a sentence emP|mi | m
bedding which computes the sum of all word embeddings in a sentence: ϕw (mi ) = j=1
W wj where Wm is the word
embedding matrix, each of which is 256-dimensional. An MLP with one hidden layer with 256 units is for ϕshif t , a fullyconnected layer is used for ϕupdate . ϕgoal is an MLP with one hidden layer with 256 units that takes the concatenation of rt
and ht as an input and computes the probabilities over subtask parameters as the outputs. The baseline network is a linear
regression from the concatenation of the memory pointer pt , a binary mask indicating the presence of given instruction,
and the final hidden layer (256 hidden units in ϕgoal ).
We used the same hyperparameters used in the parameterized skill except that the batch size was 32 (1 iteration = 16 × 32
episodes). We trained the soft-architecture (with soft-update) with a learning rate of 2.5e − 4 using curriculum learning
for 15,000 iterations and a weight of 0.015 for entropy regularization, and fine-tuned it with a learning rate of 1e − 4
without curriculum learning for 5,000 iterations. Finally, we initialized the hard-architecture (with hard-update) to the
3
For convolution layers, NxKxK-S represents N kernels with size of KxK and stride of S. The number in LSTM represents the
number of hidden units.

Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning

soft-architecture and fine-tuned it using a learning rate of 1e − 4 for 5,000 iterations. η = 0.001 is used to penalize
frequent update decision in Eq (8).
Flat Controller. The flat controller architecture consists of the same layers used in the meta controller with the following
differences. The previously retrieved instruction (rt−1 ) is transformed through an MLP with two hidden layers to compute
the weight of Conv3 and LSTM. The output is probabilities over primitive actions.
Curriculum Learning. For training all architectures, we randomly sampled the size of the world from {5, 6, 7, 8}, the
density of walls are sampled from [0, 0.1], and the density of objects are sampled from [0.1, 0.8] during training of parameterized skill and [0, 0.15] during training of the meta controller. We sampled the number of instructions from {1, 2, 3, 4}
for training the meta controller. The sampling range was determined based on the success rate of the agent.

