DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
Irina Higgins Arka Pal Andrei A. Rusu Loic Matthey Christopher P Burgess Alexander Pritzel Matthew Botvinick Charles Blundell Alexander Lerchner
stat.ML cs.AI cs.LG


[
DARLA: Improving Zero-Shot Transfer in Reinforcement Learning














equal*


Irina Higginsequal,dm
Arka Palequal,dm
Andrei Rusudm
Loic Mattheydm
Christopher Burgessdm
Alexander Pritzeldm
Matthew Botvinickdm
Charles Blundelldm
Alexander Lerchnerdm


dmDeepMind, 6 Pancras Square, Kings Cross, London, N1C 4AG, UK

Irina Higginsirinah@google.com
Arka Palarkap@google.com




transfer learning, domain adaptation, reinforcement learning, disentangled representations, beta-VAE

0.3in
]










 



Domain adaptation is an important open problem in deep reinforcement learning (RL). In many scenarios of interest data is hard to obtain, so agents may learn a source policy in a setting where data is readily available, with the hope that it generalises well to the target domain. We propose a new multi-stage RL agent, DARLA (DisentAngled Representation Learning Agent), which learns to see before learning to act. DARLA's vision is based on learning a disentangled representation of the observed environment. Once DARLA can see, it is able to acquire source policies that are robust to many domain shifts - even with no access to the target domain. DARLA significantly outperforms conventional baselines in zero-shot domain adaptation scenarios, an effect that holds across a variety of RL environments (Jaco arm, DeepMind Lab) and base RL algorithms (DQN, A3C and EC).




§ INTRODUCTION


Autonomous agents can learn how to maximise future expected rewards by choosing how to act based on incoming sensory observations via reinforcement learning (RL). Early RL approaches did not scale well to environments with large state spaces and high-dimensional raw observations <cit.>. A commonly used workaround was to embed the observations in a lower-dimensional space, typically via hand-crafted and/or privileged-information features. Recently, the advent of deep learning and its successful combination with RL has enabled end-to-end learning of such embeddings directly from raw inputs, sparking success in a wide variety of previously challenging RL domains <cit.>. Despite the seemingly universal efficacy of deep RL, however, fundamental issues remain. These include data inefficiency, the reactive nature and general brittleness of learnt policies to changes in input data distribution, and lack of model interpretability <cit.>. This paper focuses on one of these outstanding issues: the ability of RL agents to deal with changes to the input distribution, a form of transfer learning known as domain adaptation <cit.>. In domain adaptation scenarios, an agent trained on a particular input distribution with a specified reward structure (termed the source domain) is placed in a setting where the input distribution is modified but the reward structure remains largely intact (the target domain). We aim to develop an agent that can learn a robust policy using observations and rewards obtained exclusively within the source domain. Here, a policy is considered as robust if it generalises with minimal drop in performance to the target domain without extra fine-tuning.

Past attempts to build RL agents with strong domain adaptation performance highlighted the importance of learning good internal representations of raw observations <cit.>. Typically, these approaches tried to align the source and target domain representations by utilising observation and reward signals from both domains <cit.>. In many scenarios, such as robotics, this reliance on target domain information can be problematic, as the data may be expensive or difficult to obtain <cit.>. Furthermore, the target domain may simply not be known in advance. On the other hand, policies learnt exclusively on the source domain using existing deep RL approaches that have few constraints on the nature of the learnt representations often overfit to the source input distribution, resulting in poor domain adaptation performance <cit.>.




We propose tackling both of these issues by focusing instead on learning representations which capture an underlying low-dimensional factorised representation of the world and are therefore not task or domain specific. Many naturalistic domains such as video game environments, simulations and our own world are well described in terms of such a structure. Examples of such factors of variation are object properties like colour, scale, or position; other examples correspond to general environmental factors, such as geometry and lighting. We think of these factors as a set of high-level parameters that can be used by a world graphics engine to generate a particular natural visual scene <cit.>. Learning how to project raw observations into such a factorised description of the world is addressed by the large body of literature on disentangled representation learning <cit.>. Disentangled representations are defined as interpretable, factorised latent representations where either a single latent or a group of latent units are sensitive to changes in single ground truth factors of variation used to generate the visual world, while being invariant to changes in other factors <cit.>. The theoretical utility of disentangled representations for supervised and reinforcement learning has been described before <cit.>; however, to our knowledge, it has not been empirically validated to date.


We demonstrate how disentangled representations can improve the robustness of RL algorithms in domain adaptation scenarios by introducing DARLA (DisentAngled Representation Learning Agent), a new RL agent capable of learning a robust policy on the source domain that achieves significantly better out-of-the-box performance in domain adaptation scenarios compared to various baselines. DARLA relies on learning a latent state representation that is shared between the source and target domains, by learning a disentangled representation of the environment's generative factors. Crucially, DARLA does not require target domain data to form its representations. Our approach utilises a three stage pipeline: 1) learning to see, 2) learning to act, 3) transfer. During the first stage, DARLA develops its vision, learning to parse the world in terms of basic visual concepts, such as objects, positions, colours, etc. by utilising a stream of raw unlabelled observations – not unlike human babies in their first few months of life <cit.>. In the second stage, the agent utilises this disentangled visual representation to learn a robust source policy. In stage three, we demonstrate that the DARLA source policy is more robust to domain shifts, leading to a significantly smaller drop in performance in the target domain even when no further policy finetuning is allowed (median 270.3% improvement). These effects hold consistently across a number of different RL environments <cit.> and algorithms <cit.>.














§ FRAMEWORK



 §.§ Domain adaptation in Reinforcement Learning

We now formalise domain adaptation scenarios in a reinforcement learning (RL) setting. We denote the source and target domains as D_S and D_T, respectively. Each domain corresponds to an MDP defined as a tuple D_S ≡ (𝒮_S, 𝒜_S, 𝒯_S, R_S) or D_T ≡ (𝒮_T, 𝒜_T, 𝒯_T, R_T) (we assume a shared fixed discount factor γ), each with its own state space 𝒮, action space 𝒜, transition function 𝒯 and reward function R.[For further background on the notation relating to the RL paradigm, see Section <ref> in the Supplementary Materials.] In domain adaptation scenarios the states 𝒮 of the source and the target domains can be quite different, while the action spaces 𝒜 are shared and the transitions 𝒯 and reward functions R have structural similarity. For example, consider a domain adaptation scenario for the Jaco robotic arm, where the MuJoCo <cit.> simulation of the arm is the source domain, and the real world setting is the target domain. The state spaces (raw pixels) of the source and the target domains differ significantly due to the perceptual-reality gap <cit.>; that is to say, 𝒮_S ≠𝒮_T. Both domains, however, share action spaces (𝒜_S = 𝒜_T), since the policy learns to control the same set of actuators within the arm. Finally, the source and target domain transition and reward functions share structural similarity (𝒯_S ≈𝒯_T and R_S ≈ R_T), since in both domains transitions between states are governed by the physics of the world and the performance on the task depends on the relative position of the arm's end effectors (i.e. fingertips) with respect to an object of interest.



 §.§ DARLA


In order to describe our proposed DARLA framework, we assume that there exists a set ℳ of MDPs that is the set of all natural world MDPs, and each MDP D_i is sampled from ℳ. We define ℳ in terms of the state space 𝒮̂ that contains all possible conjunctions of high-level factors of variation necessary to generate any naturalistic observation in any D_i ∈ℳ. A natural world MDP D_i is then one whose state space 𝒮 corresponds to some subset of 𝒮̂. In simple terms, we assume that there exists some shared underlying structure between the MDPs D_i sampled from ℳ. We contend that this is a reasonable assumption that permits inclusion of many interesting problems, including being able to characterise our own reality <cit.>.

We now introduce notation for two state space variables that may in principle be used interchangeably within the source and target domain MDPs D_S and D_T – the agent observation state space 𝒮^o, and the agent's internal latent state space 𝒮^z.[Note that we do not assume these to be Markovian i.e. it is not necessarily the case that p(s^o_t+1|s^o_t) = p(s^o_t+1|s^o_t, s^o_t-1, …, s^o_1), and similarly for s^z. Note the index t here corresponds to time.] 𝒮^o_i in D_i consists of raw (pixel) observations s^o_i generated by the true world simulator from a sampled set of data generative factors ŝ_i, i.e. s^o_i ∼Sim(ŝ_i). ŝ_i is sampled by some distribution or process 𝒢_i on 𝒮̂, ŝ_i ∼𝒢_i(𝒮̂). 

Using the newly introduced notation, domain adaptation scenarios can be described as having different sampling processes 𝒢_S and 𝒢_T such that ŝ_S ∼𝒢_S(𝒮̂) and ŝ_T ∼𝒢_T(𝒮̂) for the source and target domains respectively, and then using these to generate different agent observation states s^o_S ∼Sim(ŝ_S) and s^o_T ∼Sim(ŝ_T). Intuitively, consider a source domain where oranges appear in blue rooms and apples appear in red rooms, and a target domain where the object/room conjunctions are reversed and oranges appear in red rooms and apples appear in blue rooms. While the true data generative factors of variation 𝒮̂ remain the same - room colour (blue or red) and object type (apples and oranges) - the particular source and target distributions 𝒢_S and 𝒢_T differ. 


Typically deep RL agents <cit.> operating in an MDP D_i ∈ℳ learn an end-to-end mapping from raw (pixel) observations s^o_i ∈𝒮^o_i to actions a_i ∈𝒜_i (either directly or via a value function Q_i(s^o_i, a_i) from which actions can be derived). In the process of doing so, the agent implicitly learns a function ℱ: 𝒮^o_i →𝒮^z_i that maps the typically high-dimensional raw observations s^o_i to typically low-dimensional latent states s^z_i; followed by a policy function π_i: 𝒮^z_i →𝒜_i that maps the latent states s^z_i to actions a_i ∈𝒜_i. In the context of domain adaptation, if the agent learns a naive latent state mapping function ℱ_S:  𝒮^o_S →𝒮^z_S on the source domain using reward signals to shape the representation learning, it is likely that ℱ_S will overfit to the source domain and will not generalise well to the target domain. Returning to our intuitive example, imagine an agent that has learnt a policy to pick up oranges and avoid apples on the source domain. Such a source policy π_S is likely to be based on an entangled latent state space 𝒮^z_S of object/room conjunctions: oranges/blue → good, apples/red → bad, since this is arguably the most efficient representation for maximising expected rewards on the source task in the absence of extra supervision signals suggesting otherwise. A source policy π_S(a|s^z_S; θ) based on such an entangled latent representation s^z_S will not generalise well to the target domain without further fine-tuning, since ℱ_S(s^o_S) ≠ℱ_S(s^o_T) and therefore crucially S^z_S ≠ S^z_T.

On the other hand, since both ŝ_S ∼𝒢_S(𝒮̂) and ŝ_T ∼𝒢_T(𝒮̂) are sampled from the same natural world state space 𝒮̂ for the source and target domains respectively, it should be possible to learn a latent state mapping function 
ℱ̂: 𝒮^o →𝒮^z_𝒮̂, which projects the agent observation state space 𝒮^o to a latent state space 𝒮^z_𝒮̂ expressed in terms of factorised data generative factors that are representative of the natural world i.e. S^z_Ŝ≈Ŝ. Consider again our intuitive example, where ℱ̂ maps agent observations (s^o_S: orange in a blue room) to a factorised or disentangled representation expressed in terms of the data generative factors (s^z_𝒮̂: room type = blue; object type = orange). Such a disentangled latent state mapping function should then directly generalise to both the source and the target domains, so that ℱ̂(s^o_S) = ℱ̂(s^o_T) = s^z_𝒮̂. Since 𝒮^z_𝒮̂ is a disentangled representation of object and room attributes, the source policy π_S can learn a decision boundary that ignores the irrelevant room attributes: oranges → good, apples → bad. Such a policy would then generalise well to the target domain out of the box, since  π_S(a|ℱ̂(s^o_S); θ) = π_T(a|ℱ̂(s^o_T); θ) = π_T(a|s^z_𝒮̂; θ). Hence, DARLA is based on the idea that a good quality ℱ̂ learnt exclusively on the source domain D_S ∈ℳ will zero-shot-generalise to all target domains D_i ∈ℳ, and therefore the source policy π(a|𝒮^z_𝒮̂; θ) will also generalise to all target domains D_i ∈ℳ out of the box.

Next we describe each of the stages of the DARLA pipeline that allow it to learn source policies π_S that are robust to domain adaptation scenarios, despite being trained with no knowledge of the target domains (see Fig. <ref> for a graphical representation of these steps):

1) Learn to see (unsupervised learning of ℱ_U) – the task of inferring a factorised set of generative factors 𝒮^z_𝒮̂ = Ŝ from observations 𝒮^o is the goal of the extensive disentangled factor learning literature <cit.>. Hence, in stage one we learn a mapping ℱ_U: 𝒮^o_U →𝒮^z_U, where 𝒮^z_U ≈𝒮^z_𝒮̂ (U stands for `unsupervised') using an unsupervised model for learning disentangled factors that utilises observations collected by an agent with a random policy π_U from a visual pre-training MDP D_U ∈ℳ. Note that we require sufficient variability of factors and their conjunctions in D_U in order to have S^z_U ≈𝒮^z_𝒮̂;

2) Learn to act (reinforcement learning of π_S in the source domain D_S utilising previously learned ℱ_U) – an agent that has learnt to see the world in stage one in terms of the natural data generative factors is now exposed to a source domain D_S ∈ℳ. The agent is tasked with learning the source policy π_S(a|s^z_S; θ), where s^z_S = ℱ_U(s^o_S) ≈ s^z_𝒮̂, via a standard reinforcement learning algorithm. Crucially, we do not allow ℱ_U to be modified (e.g. by gradient updates) during this phase;

3) Transfer (to a target domain D_T) – in the final step, we test how well the policy π_S learnt on the source domain generalises to the target domain D_T ∈ℳ in a zero-shot domain adaptation setting, i.e. the agent is evaluated on the target domain without retraining. We compare the performance of policies learnt with a disentangled latent state 𝒮^z_𝒮̂ to various baselines where the latent state mapping function ℱ_U projects agent observations s^o to entangled latent state representations s^z.




 §.§ Learning disentangled representations


In order to learn ℱ_U, DARLA utilises  <cit.>, a state-of-the-art unsupervised model for automated discovery of factorised latent representations from raw image data.  is a modification of the variational autoencoder framework <cit.> that controls the nature of the learnt latent representations by introducing an adjustable hyperparameter β to balance reconstruction accuracy with latent channel capacity and independence constraints. It maximises the objective:

 
 


    ℒ(θ, ϕ; x̌, ž, β)  =   𝔼_q_ϕ(ž|x̌) [log p_θ(x̌ | ž)] 
        - β D_KL(q_ϕ(ž|x̌) || p(ž))

where ϕ, θ parametrise the distributions of the encoder and the decoder respectively. Well-chosen values of β - usually larger than one (β>1) - typically result in more disentangled latent representations ž by limiting the capacity of the latent information channel, and hence encouraging a more efficient factorised encoding through the increased pressure to match the isotropic unit Gaussian prior p(ž) <cit.>. 




  §.§.§ Perceptual Similarity Loss


The cost of increasing β is that crucial information about the scene may be discarded in the latent representation ž, particularly if that information takes up a small proportion of the observations x̌ in pixel space. We encountered this issue in some of our tasks, as discussed in Section <ref>. The shortcomings of calculating the log-likelihood term 𝔼_q_ϕ(ž|x̌)[log p_θ(x̌|ž)] on a per-pixel basis are known and have been addressed in the past by calculating the reconstruction cost in an abstract, high-level feature space given by another neural network model, such as a GAN <cit.> or a pre-trained AlexNet <cit.>. In practice we found that pre-training a denoising autoencoder <cit.> on data from the visual pre-training MDP D_U ∈ℳ worked best as the reconstruction targets for  to match (see Fig. <ref> for model architecture and Sec. <ref> in Supplementary Materials for implementation details). The new model was trained according to Eq. <ref>:


    ℒ(θ, ϕ; x̌, ž, β)  =     - 𝔼_q_ϕ(ž|x̌)J(x̂̌̂) - J(x̌)  _2^2   
       - β D_KL(q_ϕ(ž|x̌) || p(ž))


 
 

where x̂̌̂∼  p_θ(x̌ | ž) and J : ℝ^W × H × C→ℝ^N is the function that maps images from pixel space with dimensionality W × H × C to a high-level feature space with dimensionality N given by a stack of pre-trained DAE layers up to a certain layer depth. Note that by replacing the pixel based reconstruction loss in Eq. <ref> with high-level feature reconstruction loss in Eq. <ref> we are no longer optimising the variational lower bound, and with β=1 loses its equivalence to the Variational Autoencoder (VAE) framework as proposed by <cit.>. In this setting, the only way to interpret β is as a mixing coefficient that balances the capacity of the latent channel ž of against the pressure to match the high-level features within the pre-trained DAE.






 §.§ Reinforcement Learning Algorithms


We used various RL algorithms <cit.> to learn the source policy π^S during stage two of the pipeline using the latent states s^z acquired by  based models during stage one of the DARLA pipeline. 

Deep Q Network
(DQN) <cit.> is a variant of the Q-learning algorithm <cit.> that utilises deep learning. It uses a neural network to parametrise an approximation for the action-value function Q(s, a; θ) using parameters θ. 
 
Asynchronous Advantage Actor-Critic
(A3C) <cit.> is an asynchronous implementation of the advantage actor-critic paradigm <cit.>, where separate threads run in parallel and perform updates to shared parameters. The different threads each hold their own instance of the environment and have different exploration policies, thereby decorrelating parameter updates without the need for experience replay. Therefore, A3C is an online algorithm, whereas DQN learns its policy offline, resulting in different learning dynamics between the two algorithms.

Model-Free Episodic Control
(EC) <cit.> was proposed as a complementary learning system to the other RL algorithms described above. The EC algorithm relies on near-determinism of state transitions and rewards in RL environments; in settings where this holds, it can exploit these properties to memorise which action led to high returns in similar situations in the past. Since in its simplest form EC relies on a lookup table, it learns good policies much faster than value-function-approximation based deep RL algorithms like DQN trained via gradient descent - at the cost of generality (i.e. potentially poor performance in non-deterministic environments).

We also compared our approach to that of UNREAL <cit.>, a recently proposed RL algorithm which also attempts to utilise unsupervised data in the environment. The UNREAL agent takes as a base an LSTM A3C agent <cit.> and augments it with a number of unsupervised auxiliary tasks that make use of the rich perceptual data available to the agent besides the (sometimes very sparse) extrinsic reward signals. This auxiliary learning tends to improve the representation learnt by the agent. See Sec. <ref> in Supplementary Materials for further details of the algorithms above.





§ TASKS


We evaluate the performance of DARLA on different task and environment setups that probe subtly different aspects of domain adaptation. As a reminder, in Sec. <ref> we defined 𝒮̂ as a state space that contains all possible conjunctions of high-level factors of variation necessary to generate any naturalistic observation in any D_i ∈ℳ. During domain adaptation scenarios agent observation states are generated according to s^o_S ∼Sim_S(ŝ_S) and s^o_T ∼Sim_T(ŝ_T) for the source and target domains respectively, where ŝ_S and ŝ_T are sampled by some distributions or processes 𝒢_S and 𝒢_T according to  ŝ_S ∼𝒢_S(𝒮̂) and ŝ_T ∼𝒢_T(𝒮̂).

We use DeepMind Lab <cit.> to test a version of domain adaptation setup where the source and target domain observation simulators are equal (Sim_S = Sim_T), but the processes used to sample  ŝ_S and  ŝ_T are different (𝒢_S ≠𝒢_T). We use the Jaco arm with a matching MuJoCo simulation environment <cit.> in two domain adaptation scenarios: simulation to simulation (sim2sim) and simulation to reality (sim2real). The sim2sim domain adaptation setup is relatively similar to DeepMind Lab i.e. the source and target domains differ in terms of processes 𝒢_S and 𝒢_T. However, there is a significant point of difference. In DeepMind Lab, all values of factors in the target domain, ŝ_T, are previously seen in the source domain; however, ŝ_S ≠ŝ_T as the conjunctions of these factor values are different. In sim2sim, by contrast, novel factor values are experienced in the target domain (this accordingly also leads to novel factor conjunctions). Hence, DeepMind Lab may be considered to be assessing domain interpolation performance, whereas sim2sim tests domain extrapolation. 

The sim2real setup, on the other hand, is based on identical processes 𝒢_S = 𝒢_T, but different observation simulators Sim_S ≠Sim_T corresponding to the MuJoCo simulation and the real world, which results in the so-called `perceptual reality gap' <cit.>. More details of the tasks are given below.





 §.§ DeepMind Lab


DeepMind Lab is a first person 3D game environment with rich visuals and realistic physics. We used a standard seek-avoid object gathering setup, where a room is initialised with an equal number of randomly placed objects of two different types. One of the object varieties is `good' (its collection is rewarded +1), while the other is `bad' (its collection is punished -1). The full state space 𝒮̂ consisted of all conjunctions of two room types (pink and green based on the colour of the walls) and four object types (hat, can, cake and balloon) (see Fig. <ref>A). The source domain D_S contained environments with hats/cans presented in the green room, and balloons/cakes presented in either the green or the pink room. The target domain D_T contained hats/cans presented in the pink room. In both domains cans and balloons were the rewarded objects. 

















1) Learn to see: we used to learn the disentangled latent state representation s^z that includes both the room and the object generative factors of variation within DeepMind Lab. We had to use the high-level feature space of a pre-trained DAE within the framework (see Section <ref>), instead of the pixel space of vanilla , because we found that objects failed to reconstruct when using the values of β necessary to disentangle the generative factors of variation within DeepMind Lab (see Fig. <ref>B). 
















was trained on observations s^o_U collected by an RL agent with a simple wall-avoiding policy π_U (otherwise the training data was dominated by close up images of walls). In order to enable the model to learn ℱ(s^o_U) ≈𝒮̂, it is important to expose the agent to at least a minimal set of environments that span the range of values for each factor, and where no extraneous correlations are added between different factors[In our setup of DeepMind Lab domain adaptation task, the object and environment factors are supposed to be independent. In order to ensure that learns a factorised representation that reflects this ground truth independence, we present observations of all possible conjunctions of room and individual object types.](see Fig. <ref>A, yellow). See Section <ref> in Supplementary Materials for details of training.



2) Learn to act: the agent was trained with the algorithms detailed in Section <ref> on a seek-avoid task using the source domain (D_S) conjunctions of object/room shown in Fig. <ref>A (green). Pre-trained from stage one was used as the `vision' part of various RL algorithms <cit.> to learn a source policy π_S that picks up balloons and avoids cakes in both the green and the pink rooms, and picks up cans and avoids hats in the green rooms. See Section <ref> in Supplementary Materials for more details of the various versions of DARLA we have tried, each based on a different base RL algorithm.



3) Transfer: we tested the ability of DARLA to transfer the seek-avoid policy π_S it had learnt on the source domain in stage two using the domain adaptation condition D_T illustrated in Figure <ref>A (red). The agent had to continue picking up cans and avoid hats in the pink room, even though these objects had only been seen in the green room during source policy training. The optimal policy π_T is one that maintains the reward polarity from the source domain (cans are good and hats are bad). For further details, see Appendix <ref>.
 


 §.§ Jaco Arm and MuJoCo


We used frames from an RGB camera facing a robotic Jaco arm, or a matching rendered camera view from a MuJoCo physics simulation environment <cit.> to investigate the performance of DARLA in two domain adaptation scenarios: 1) simulation to simulation (sim2sim), and 2) simulation to reality (sim2real). The sim2real setup is of particular importance, since the progress that deep RL has brought to control tasks in simulation <cit.> has not yet translated as well to reality, despite various attempts <cit.>. Solving control problems in reality is hard due to sparse reward signals, expensive data acquisition and the attendant danger of breaking the robot (or its human minders) during exploration.

In both sim2sim and sim2real, we trained the agent to perform an object reaching policy where the goal is to place the end effector as close to the object as possible. While conceptually the reaching task is simple, it is a hard control problem since it requires correct inference of the arm and object positions and velocities from raw visual inputs.




1) Learn to see: was trained on observations collected in MuJoCo simulations with the same factors of variation as in D_S. In order to enable the model to learn ℱ(s^o_U) ≈ŝ, a reaching policy was applied to phantom objects placed in random positions - therefore ensuring that the agent learnt the independent nature of the arm position and object position (see Fig. <ref>C, left);  

2) Learn to act: a feedforward-A3C based agent with the vision module pre-trained in stage one was taught a source reaching policy π_S towards the real object in simulation (see Fig. <ref>C (left) for an example frame, and Sec. <ref> in Supplementary Materials for a fuller description of the agent). In the source domain D_S the agent was trained on a distribution of camera angles and positions. The colour of the tabletop on which the arm rests and the object colour were both sampled anew every episode.

3) Transfer: sim2sim: in the target domain, D_T, the agent was faced with a new distribution of camera angles and positions with little overlap with the source domain distributions, as well as a completely held out set of object colours (see Fig. <ref>C, middle). sim2real: in the target domain D_T the camera position and angle as well as the tabletop colour and object colour were sampled from the same distributions as seen in the source domain D_S, but the target domain D_T was now the real world. Many details present in the real world such as shadows, specularity, multiple light sources and so on are not modelled in the simulation; the physics engine is also not a perfect model of reality. Thus sim2real tests the ability of the agent to cross the perceptual-reality gap and generalise its source policy π_S to the real world (see Fig. <ref>C, right). For further details, see Appendix <ref>.














§ RESULTS









We evaluated the robustness of DARLA's policy π_S learnt on the source domain to various shifts in the input data distribution. In particular, we used domain adaptation scenarios based on the DeepMind Lab seek-avoid task and the Jaco arm reaching task described in Sec. <ref>. On each task we compared DARLA's performance to that of various baselines. We evaluated the importance of learning `good' vision during stage one of the pipeline, i.e one that maps the input observations s^o to disentangled representations s^z ≈ŝ. In order to do this, we ran the DARLA pipeline with different vision models: the encoders of a disentangled -0.5ex [In this section of the paper, we use the term to refer to a standard for the MuJoCo experiments, and a for the DeepMind Lab experiments (as described in stage 1 of Sec. <ref>).] (the original DARLA), an entangled (DARLA_ENT), and a denoising autoencoder (DARLA_DAE). Apart from the nature of the learnt representations s^z, DARLA and all versions of its baselines were equivalent throughout the three stages of our proposed pipeline in terms of architecture and the observed data distribution (see Sec. <ref> in Supplementary Materials for more details).


Figs. <ref>-<ref> display the degree of disentanglement learnt by the vision modules of DARLA and DARLA_ENT on DeepMind Lab and MuJoCo. DARLA's vision learnt to independently represent environment variables (such as room colour-scheme and geometry) and object-related variables (change of object type, size, rotation) on DeepMind Lab (Fig. <ref>, left). Disentangling was also evident in MuJoCo. Fig. <ref>, left, shows that DARLA's single latent units z_i learnt to represent different aspects of the Jaco arm, the object, and the camera. By contrast, in the representations learnt by DARLA_ENT, each latent is responsible for changes to both the environment and objects (Fig. <ref>, right) in DeepMind Lab, or a mixture of camera, object and/or arm movements (Fig. <ref>, right) in MuJoCo.




The table in Fig. <ref> shows the average performance (across different seeds) in terms of rewards per episode of the various agents on the target domain with no fine-tuning of the source policy π_S. It can be seen that DARLA is able to zero-shot-generalise significantly better than DARLA_ENT or DARLA_DAE, highlighting the importance of learning a disentangled representation s^z = s^z_𝒮̂ during the unsupervised stage one of the DARLA pipeline. In particular, this also demonstrates that the improved domain transfer performance is not simply a function of increased exposure to training observations, as both DARLA_ENT and DARLA_DAE were exposed to the same data. The results are mostly consistent across target domains and in most cases DARLA is significantly better than the second-best-performing agent. This holds in the sim2real task [See https://youtu.be/sZqrWFl0wQ4 for example sim2sim and sim2real zero-shot transfer policies of DARLA and baseline A3C agent.], where being able to perform zero-shot policy transfer is highly valuable due to the particular difficulties of gathering data in the real world.

DARLA's performance is particularly surprising as it actually preserves less information about the raw observations s^o than DARLA_ENT and DARLA_DAE. This is due to the nature of the and how it achieves disentangling; the disentangled model utilised a significantly higher value of the hyperparameter β than the entangled model (see Appendix <ref> for further details), which constrains the capacity of the latent channel. Indeed, DARLA's only utilises 8 of its possible 32 Gaussian latents to store observation-specific information for MuJoCo/Jaco (and 20 in DeepMind Lab), whereas DARLA_ENT utilises all 32 for both environments (as does DARLA_DAE).


Furthermore, we examined what happens if DARLA's vision (i.e. the encoder of the disentangled -0.5ex) is allowed to be fine-tuned via gradient updates while learning the source policy during stage two of the pipeline. This is denoted by DARLA_FT in the table in Fig. <ref>. We see that it exhibits significantly worse performance than that of DARLA in zero-shot domain adaptation using an A3C-based agent in all tasks. This suggests that a favourable initialisation does not make up for subsequent overfitting to the source domain for the on-policy A3C. However, the off-policy DQN-based fine-tuned agent performs very well. We leave further investigation of this curious effect for future work.

Finally, we compared the performance of DARLA to an UNREAL <cit.> agent with the same architecture. Despite also exploiting the unsupervised data available in the source domain, UNREAL performed worse than baseline A3C on the DeepMind Lab domain adaptation task. This further demonstrates that use of unsupervised data in itself is not a panacea for transfer performance; it must be utilised in a careful and structured manner conducive to learning disentangled latent states s^z = s^z_𝒮̂.

In order to quantitatively evaluate our hypothesis that disentangled representations are essential for DARLA's performance in domain adaptation scenarios, we trained various DARLAs with different degrees of learnt disentanglement in s^z by varying β (of -0.5ex) during stage one of the pipeline. We then calculated the correlation between the performance of the EC-based DARLA on the DeepMind Lab domain adaptation task and the transfer metric, which approximately measures the quality of disentanglement of DARLA's latent representations s^z (see Sec. <ref> in Supplementary Materials). This is shown in the chart in Fig. <ref>; as can be seen, there is a strong positive correlation between the level of disentanglement and DARLA's zero-shot domain transfer performance (r=0.6, p<0.001).











Having shown the robust utility of disentangled representations in agents for domain adaptation, we note that there is evidence that they can provide an important additional benefit. We found significantly improved speed of learning of π_S on the source domain itself, as a function of how disentangled the model was. The gain in data efficiency from disentangled representations for source policy learning is not the main focus of this paper so we leave it out of the main text; however, we provide results and discussion in Section <ref> in Supplementary Materials.









§ CONCLUSION

We have demonstrated the benefits of using disentangled representations in a deep RL setting for domain adaptation. In particular, we have proposed DARLA, a multi-stage RL agent. DARLA first learns a visual system that encodes the observations it receives from the environment as disentangled representations, in a completely unsupervised manner. It then uses these representations to learn a robust source policy that is capable of zero-shot domain adaptation. 

We have demonstrated the efficacy of this approach in a range of domains and task setups: a 3D naturalistic first-person environment (DeepMind Lab), a simulated graphics and physics engine (MuJoCo), and crossing the simulation to reality gap (MuJoCo to Jaco sim2real). We have also shown that the effect of disentangling is consistent across very different RL algorithms (DQN, A3C, EC), achieving significant improvements over the baseline algorithms (median 2.7 times improvement in zero-shot transfer across tasks and algorithms). To the best of our knowledge, this is the first comprehensive empirical demonstration of the strength of disentangled representations for domain adaptation in a deep RL setting.




icml2017




















§ SUPPLEMENTARY MATERIALS




 §.§ The Reinforcement Learning Paradigm


The reinforcement learning (RL) paradigm consists of an agent receiving a sequence of observations s^o_t which are some function of environment states s_t ∈𝒮 and may be accompanied by rewards r_t+1∈ R conditional on the actions a_t ∈𝒜, chosen at each time step t <cit.>. We assume that these interactions can be modelled as a Markov Decision Process (MDP) <cit.> defined as a tuple D ≡ (𝒮, 𝒜, 𝒯, R, γ ).  𝒯 = p(s | s_t, a_t) is a transition function that models the distribution of all possible next states given action a_t is taken in state s_t for all s_t ∈𝒮 and a_t ∈𝒜. Each transition s_t a_t→ s_t+1 may be accompanied by a reward signal r_t+1(s_t, a_t, s_t+1). The goal of the agent is to learn a policy π(a_t|s_t), a probability distribution over actions a_t ∈𝒜, that maximises the expected return i.e. the discounted sum of future rewards R_t = 𝔼[∑_τ=1^T-tγ^τ-1r_t+τ]. T is the time step at which each episode ends, and  γ∈ [0, 1) is the discount factor that progressively down-weights future rewards. Given a policy π(a|s), one can define the value function V_π(s) = 𝔼 [R_t|s_t=s, π], which is the expected return from state s following policy π. The action-value function Q_π(s, a) = 𝔼 [R_t|s_t=s, a_t=a, π] is the expected return for taking action a in state s at time t, and then following policy π from time t+1 onward.



 §.§ Further task details





  §.§.§ DeepMind Lab


As described in Sec <ref>, in each source episode of DeepMind Lab the agent was presented with one of three possible room/object type conjunctions, chosen at random. These are marked D_S in Fig <ref>. The setup was a seek-avoid style task, where one of the two object types in the room gave a reward of +1 and the other gave a reward of -1. The agent was allowed to pick up objects for 60 seconds after which the episode would terminate and a new one would begin; if the agent was able to pick up all the `good' objects in less than 60 seconds, a new episode was begun immediately. The agent was spawned in a random location in the room at the start of each new episode.

During transfer, the agent was placed into the held out conjunction of object types and room background; see D_T in Fig <ref>.

Visual pre-training was performed in other conjunctions of object type and room background denoted D_U in Fig <ref>.

The observation size of frames in the DeepMind Lab task was 84x84x3 (HxWxC).



  §.§.§ MuJoCo/Jaco Arm Experiments


As described in Sec <ref>, the source task consisted of an agent learning to control a simulated arm in order to reach toward an object. A shaping reward was used, with a maximum value of 1 when the centre of the object fell between the pinch and grip sites of the end effector, or within a 10cm distance of the two. Distances on the x and y dimensions counted double compared to distances on the z dimension.

During each episode the object was placed at a random drop point within a 40x40cm area, and the arm was set to a random initial start position high above the work-space, independent of the object's position. Each episode lasted for 150 steps, or 7.5 seconds, with a control step of 50ms. Observations s^o_U were sampled randomly across episodes. Overall, 4 million frames of dimensions 64x64x3 (HxWxC) were used for this stage of the curriculum. For each episode the camera position and orientation were randomly sampled from an isotropic normal distribution centred around the approximate position and orientation of the real camera, with standard deviation 0.01. No precise measurements were used to match the two. Work-space table colour was sampled uniformly between -5% and +5% around the midpoint, independently for each RGB channel; object colours were sampled uniformly at random in RGB space, rejecting colours which fell within a ball around 10 held-out intensities (radius 10% of range); the latter were only used for simulated transfer experiments, i.e. in D_T in the sim2sim experiments. Additionally, Gaussian noise with standard deviation 0.01 was added to the observations s^o_T in the sim2sim task.

For the real Jaco arm and its MuJoCo simulation counterpart, each of the nine joints could independently take 11 different actions (a linear discretisation of the continuous velocity action space).  In simulation Gaussian noise with standard deviation 0.1 was added to each discrete velocity output; delays in the real setup between observations and action execution were simulated by randomly mixing velocity outputs from two previous steps instead of emitting the last output directly. Speed ranges were between -50% and 50% of the Jaco arm's top speed on joints 1 through 6 starting at the base, while the fingers could use a full range. For safety reasons the speed ranges have been reduced by a factor of 0.3 while evaluating agents on the Jaco arm, without significant performance degradation.



 §.§ Vision model details





  §.§.§ Denoising Autoencoder for 



A denoising autoencoder (DAE) was used as a model to provide the feature space for the reconstruction loss to be computed over (for motivation, see Sec. <ref>). The DAE was trained with occlusion-style masking noise in the vein of <cit.>, with the aim for the DAE to learn a semantic representation of the input frames. Concretely, two values were independently sampled from U[0, W] and two from U[0, H] where W and H were the width and height of the input frames. These four values determined the corners of the rectangular mask applied; all pixels that fell within the mask were set to zero.

The DAE architecture consisted of four convolutional layers, each with kernel size 4 and stride 2 in both the height and width dimensions. The number of filters learnt for each layer was {32, 32, 64, 64} respectively. The bottleneck layer consisted of a fully connected layer of size 100 neurons. This was followed by four deconvolutional layers, again with kernel sizes 4, strides 2, and {64, 64, 32, 32} filters. The padding algorithm used was `SAME' in TensorFlow <cit.>. ReLU non-linearities were used throughout.

The model was trained with loss given by the L2 distance of the outputs from the original, un-noised inputs. The optimiser used was Adam <cit.> with a learning rate of 1e-3.



  §.§.§ with Perceptual Similarity Loss



After training a DAE, as detailed in the previous section[In principle, the could also have been trained end-to-end in one pass, but we did not experiment with this.], a was trained with perceptual similarity loss given by Eq. <ref>, repeated here:

    ℒ(θ, ϕ; x̌, ž, β)  =   𝔼_q_ϕ(ž|x̌)J(x̂̌̂) - J(x̌)  _2^2   
       - β D_KL(q_ϕ(ž|x̌) || p(ž))

Specifically, the input was passed through the and a sampled[It is more typical to use the mean of the reconstruction distribution, but this does not induce any pressure on the Gaussians parametrising the decoder to reduce their variances. Hence full samples were used instead.] reconstruction was passed through the pre-trained DAE up to a designated layer. The L2 distance of this representation from the representation of the original input passed through the same layers of the DAE was then computed, and this formed the training loss for the part of the [The representations were taken after passing through the layer but before passing through the following non-linearity. We also briefly experimented with taking the L2 loss post-activation but did not find a significant difference.]. The DAE weights remained frozen throughout.

The architecture consisted of an encoder of four convolutional layers, each with kernel size 4, and stride 2 in the height and width dimensions. The number of filters learnt for each layer was {32, 32, 64, 64} respectively. This was followed by a fully connected layer of size 256 neurons. The latent layer comprised 64 neurons parametrising 32 (marginally) independent Gaussian distributions. The decoder architecture was simply the reverse of the encoder, utilising deconvolutional layers. The decoder used was Gaussian, so that the number of output channels was 2C, where C was the number of channels that the input frames had. The padding algorithm used was `SAME' in TensorFlow. ReLU non-linearities were used throughout.

The model was trained with the loss given by Eq. <ref>. Specifically, the disentangled model used for DARLA was trained with a β hyperparameter value of 1 and the layer of the DAE used to compute the perceptual similarity loss was the last deconvolutional layer. The entangled model used for DARLA_ENT was trained with a β hyperparameter value of 0.1 with the last deconvolutional layer of the DAE was used to compute the perceptual similarity loss.

The optimiser used was Adam with a learning rate of 1e-4.



  §.§.§ 


For the MuJoCo/Jaco tasks, a standard was used rather than the used for DeepMind Lab. The architecture of the VAE encoder, decoder and the latent size were exactly as described in the previous section <ref>. β for the the disentangled in DARLA was 175. β for the entangled model DARLA_ENT was 1, corresponding to the standard VAE of <cit.>.

The optimizer used was Adam with a learning rate of 1e-4.



  §.§.§ Denoising Autoencoder for baseline

For the baseline model DARLA_DAE, we trained a denoising autoencoder with occlusion-style masking noise as described in Appendix Section <ref>. The architecture used matched that exactly of the described in Appendix Section <ref> - however, all stochastic nodes were replaced with deterministic neurons.

The optimizer used was Adam with a learning rate of 1e-4.



 §.§ Reinforcement Learning Algorithm Details





  §.§.§ DeepMind Lab

The action space in the DeepMind Lab task consisted of 8 discrete actions.

DQN: in DQN, the convolutional (or `vision') part of the Q-net was replaced with the encoder of the from stage 1 and frozen. DQN takes four consecutive frames as input in order to capture some aspect of environment dynamics in the agent's state. In order to match this in our setup with a pre-trained vision stack ℱ_U, we passed each observation frame s^o_{1..4} through the pre-trained model s^z_{1..4} = ℱ_U(s^o_{1..4})  and then concatenated the outputs together to form the k-dimensional (where k = 4|s^z|) input to the policy network. In this case the size of s^z was 64 for DARLA as well as DARLA_ENT, DARLA_DAE and DARLA_FT.

On top of the frozen convolutional stack, two `policy' layers of 512 neurons each were used, with a final linear layer of 8 neurons corresponding to the size of the action space in the DeepMind Lab task. ReLU non-linearities were used throughout. All other hyperparameters were as reported in <cit.>.

A3C: in A3C, as with DQN, the convolutional part of the network that is shared between the policy net and the value net was replaced with the encoder of the in DeepMind Lab tasks. All other hyperparameters were as reported in <cit.>.

Episodic Control: for the Episodic Controller-based DARLA we used mostly the same hyperparameters as in the original paper by <cit.>. We explored the following hyperparameter settings: number of nearest neighbours ∈{10, 50}, return horizon ∈{100, 400, 800, 1800, 500000}, kernel type ∈ {inverse, gaussian}, kernel width ∈{1e-6, 1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 0.5, 0.99} and we tried training EC with and without Peng's Q(λ) <cit.>. In practice we found that none of the explored hyperparameter choices significantly influenced the results of our experiments. The final hyperparameters used for all experiments reported in the paper were the following: number of nearest neighbours: 10, return horizon: 400, kernel type: inverse, kernel width: 1e-6 and no Peng's Q(λ) <cit.>.

UNREAL: We used a vanilla version of UNREAL, with parameters as reported in <cit.>.



  §.§.§ MuJoCo/Jaco Arm Experiments


For the real Jaco arm and its MuJoCo simulation, each of the nine joints could independently take 11 different actions (a linear discretisation of the continuous velocity action space). Therefore the action space size was 99.

DARLA for MuJoCo/Jaco was based on feedforward A3C <cit.>. We closely followed the simulation training setup of <cit.> for feed-forward networks using raw visual-input only. In place of the usual conv-stack, however, we used the encoder of the as described in Appendix <ref>. This was followed by a linear layer with 512 units, a ReLU non-linearity and a collection of 9 linear and softmax layers for the 9 independent policy outputs, as well as a single value output layer that outputted the value function.





 §.§ Disentanglement Evaluation



  §.§.§ Visual Heuristic Details


In order to choose the optimal value of β for the -DAE models and evaluate the fitness of the representations s^z_U learnt in stage 1 of our pipeline (in terms of disentanglement achieved), we used the visual inspection heuristic described in <cit.>. The heuristic involved clustering trained  based models based on the number of informative latents (estimated as the number of latents z_i with average inferred standard deviation below 0.75). For each cluster we examined the degree of learnt disentanglement by running inference on a number of seed images, then traversing each latent unit z_{ i } one at a time over three standard deviations away from its average inferred mean while keeping all other latents z_{∖ i } fixed to their inferred values. This allowed us to visually examine whether each individual latent unit z_i learnt to control a single interpretable factor of variation in the data. A similar heuristic has been the de rigueur method for exhibiting disentanglement in the disentanglement literature <cit.>.

 



  §.§.§ Transfer Metric Details


In the case of DeepMind Lab, we were able to use the ground truth labels corresponding to the two factors of variation of the object type and the background to design a proxy to the disentanglement metric proposed in <cit.>. The procedure used consisted of the following steps:
 
1) Train the model under consideration on observations s^o_U to learn ℱ_𝒰, as described in stage 1 of the DARLA pipeline.

2) Learn a linear model ℒ: S^z_V → M × N from the representations s^z_V = ℱ_V (s^o_V), where M ∈{0, 1} corresponds to the set of possible rooms and N ∈{0, 1, 2, 3} corresponds to the set of possible objects[For the purposes of this metric, we utilised rooms with only single objects, which we denote by the subscript V e.g. the observation set S^o_V.]. Therefore we are learning a low-VC dimension classifier to predict the room and the object class from the latent representation of the model. Crucially, the linear model ℒ is trained on only a subset of the Cartesian product M × N e.g. on {{0, 0}, {0, 3}, {1, 1}, {1, 2}}. In practice, we utilised a softmax classifier each for M and N and trained this using backpropagation with a cross-entropy loss, keeping the unsupervised model (and therefore ℱ_𝒰) fixed.

3) The trained linear model ℒ's accuracy is evaluated on the held out subset of the Cartesian product M × N.

Although the above procedure only measures disentangling up to linearity, and only does so for the latents of object type and room background, we nevertheless found that the metric was highly correlated with disentanglement as determined via visual inspection (see Fig. <ref>).




 §.§ Background on RL Algorithms


In this Appendix, we provide background on the different RL algorithms that the DARLA framework was tested on in this paper.



  §.§.§ DQN


(DQN) <cit.> is a variant of the Q-learning algorithm <cit.> that utilises deep learning. It uses a neural network to parametrise an approximation for the action-value function Q(s, a; θ) using parameters θ. These parameters are updated by minimising the mean-squared error of a 1-step lookahead loss ℒ_Q = 𝔼 [ (r_t + γ max_a' Q(s', a'; θ^-) - Q(s,a;θ))^2  ], where θ^- are parameters corresponding to a frozen network and optimisation is performed with respect to θ, with θ^- being synced to θ at regular intervals.



  §.§.§ A3C


Asynchronous Advantage Actor-Critic
(A3C) <cit.> is an asynchronous implementation of the advantage actor-critic paradigm <cit.>, where separate threads run in parallel and perform updates to shared parameters. The different threads each hold their own instance of the environment and have different exploration policies, thereby decorrelating parameter updates without the need for experience replay.

A3C uses neural networks to approximate both policy π(a|s; θ) and value V_π(s; θ) functions using parameters θ using n-step look-ahead loss <cit.>. The algorithm is trained using an advantage actor-critic loss function with an entropy regularisation penalty: ℒ_A3C≈ℒ_VR + ℒ_π - 𝔼_s∼π [ α H(π(a|s; θ))  ], where H is entropy. The parameter updates are performed after every t_max actions or when a terminal state is reached. ℒ_VR  = 𝔼_s∼π [ (R_t:t+n + γ ^n V(s_t+n+1; θ) - V(s_t;θ))^2  ] and ℒ_π  = 𝔼_s∼π [ log π(a|s; θ)(Q^π(s, a; θ) - V^π(s; θ))  ]. Unlike DQN, A3C uses an LSTM core to encode its history and therefore has a longer term memory permitting it to perform better in partially observed environments. In the version of A3C used in this paper for the DeepMind Lab task, the policy net additionally takes the last action a_t-1 and last reward r_t-1 as inputs along with the observation s^o_t, as introduced in <cit.>.



  §.§.§ UNREAL


The UNREAL agent <cit.> takes as a base an LSTM A3C agent <cit.> and augments it with a number of unsupervised auxiliary tasks that make use of the rich perceptual data available to the agent besides the (sometimes very sparse) extrinsic reward signals. This auxiliary learning tends to improve the representation learnt by the agent. While training the base agent, its observations, rewards, and actions are stored in a replay buffer, which is used by the auxiliary learning tasks. The tasks include: 1) pixel control – the agent learns how to control the environment by training auxiliary policies to maximally change pixel intensities in different parts of the input; 2) reward prediction - given a replay buffer of observations within a short time period of an extrinsic reward, the agent has to predict the reward obtained during the next unobserved timestep using a sequence of three preceding steps; 3) value function replay - extra training of the value function to promote faster value iteration.



  §.§.§ Episodic Control


In its simplest form EC is a lookup table of states and actions denoted as Q^EC(s, a). In each state EC picks the action with the highest Q_EC value. At the end of each episode Q^EC(s, a) is set to R_t if (s_t, a_t) ∉ Q^EC, where R_t is the discounted return. Otherwise Q^EC(s, a) = max{ Q^EC(s,a), R_t }. In order to generalise its policy to novel states that are not in Q^EC, EC uses a non-parametric nearest neighbours search Q^EC(s, a) = 1/k∑_i=1^k Q^EC(s^i, a), where s^i, i=1,...,k are k states with the smallest distance to the novel state s. Like DQN, EC takes a concatenation of four frames as input.

The EC algorithm is proposed as a model of fast hippocampal instance-based learning in the brain <cit.>, while the deep RL algorithms described above are more analogous to slow cortical learning that relies on generalised statistical summaries of the input distribution <cit.>.



 §.§ Source Task Performance Results



The focus of this paper is primarily on zero-shot domain adaptation performance. However, it is also interesting to analyse the effect of the DARLA approach on source domain policy performance. In order to compare the models' behaviour on the source task, we examined the training curves (see Figures <ref>-<ref>) and noted in particular their:

    
  * Asymptotic task performance, i.e. the rewards per episode at the point where π_S has converged for the agent under consideration.
    
  * Data efficiency, i.e. how quickly the training curve was able to achieve convergence.



 

 

 

 



We note the following consistent trends across the results:

    
  * Using DARLA provided an initial boost in learning performance, which depended on the degree of disentanglement of the representation. This was particularly observable in A3C, see Fig. <ref>.
    
  * Baseline algorithms where ℱ could be fine-tuned to the source task were able to achieve higher asymptotic performance. This was particularly notable on DQN and A3C (see Figs. <ref> and  <ref>) in DeepMind Lab. However, in both those cases, DARLA was able to learn very reasonable policies on the source task which were on the order of 20% lower than the fine-tuned models – arguably a worthwhile sacrifice for a subsequent median 270% improvement in target domain performance noted in the main text.
    
  * Allowing DARLA to fine-tune its vision module (DARLA_FT) boosted its source task learning speed, and allowed the agent to asymptote at the same level as the baseline algorithms. As discussed in the main text, this comes at the cost of significantly reduced domain transfer performance on A3C. For DQN, however, finetuning appears to offer the best of both worlds.
    
  * Perhaps most relevantly for this paper, even if solely examining source task performance, DARLA outperforms both DARLA_ENT and DARLA_DAE on both asymptotic performance and data efficiency – suggesting that disentangled representations have wider applicability in RL beyond the zero-shot domain adaptation that is the focus of this paper.


