Question: Represent the article for finding a single-sentence TLDR

Question: The ability of an agent to {\em discover} its own learning objectives has long been considered a key ingredient for artificial general intelligence. Breakthroughs in autonomous decision making and reinforcement learning have primarily been in domains where the agent's goal is outlined and clear: such as playing a game to win, or driving safely. Several studies have demonstrated that learning extramural sub-tasks and auxiliary predictions can improve single human-specified task learning, transfer of learning, and the agent's learned representation of the world. In all these examples, the agent was instructed what to learn about. We investigate a framework for discovery: curating a large collection of predictions, which are used to construct the agent's representation of the world. Specifically, our system maintains a large collection of predictions, continually pruning and replacing predictions. We highlight the importance of considering stability rather than convergence for such a system, and develop an adaptive, regularized algorithm towards that aim. We provide several experiments in computational micro-worlds demonstrating that this simple approach can be effective for discovering useful predictions autonomously. The idea that an agent's knowledge might be represented as predictions has a long history in machine learning. The first references to such a predictive approach can be found in the work of BID4, BID2 BID6 who hypothesized that agents would construct their understanding of the world from interaction, rather than human engineering. These ideas inspired work on predictive state representations (PSRs), as an approach to modeling dynamical systems. Simply put, a PSR can predict all possible interactions between an agent and it's environment by reweighting a minimal collection of core test (sequence of actions and observations) and their predictions, as an alternative to keeping a finite history or learning the one step latent dynamics of the world, as in a POMDP. Extensions to high-dimensional continuous tasks have demonstrated that the predictive approach to dynamical system modeling is competitive with state-of-the-art system identification methods BID9. One important limitation of the PSR formalism is that the agent's internal representation of state must be composed exclusively of predictions. Recently, BID23 introduced a formalism for specifying and learning large collections of predictions using value functions from reinforcement learning. These General Value Functions (GVFs), can represent a wide array of multi-step state contingent predictions BID19, while the predictions made by a collection of GVFs can be used to construct the agent's state representation (; BID29 . State representation's constructed from predictions have been shown to be useful for reward maximization tasks BID21), and transfer learning . One of the great innovation of GVFs is that we can clearly separate the desire to learn and make use of predictions in various ways, from the construction of the agent's internal state representation. For example, the UNREAL learning system BID10, learns many auxiliary tasks (formalized as GVFs) while using an actor-critic algorithm to maximize a single external reward signal-the score in an Atari game. The auxiliary GVFs and the primary task learner share the same deep convolutional network structure. Learning the auxiliary tasks in a better state representation than simply learning the main task alone. GVFs allow an agent to make use of both increased representational power of predictive representations, and the flexibility of state-of-the-art deep learning systems. In all the works described above, the GVFs were manually specified by the designer; an autonomous agent, however, must discover these GVFs. Most work on discovery has been on the related topics of temporal difference (TD) networks BID15 and options BID11 BID17 BID16 BID28 BID16 BID1. Discovery for options is more related than TD networks, because similarly to a GVF, an option specifies small sub-tasks within an environment. Option discovery, however, has been largely directed towards providing temporally abstract actions, towards solving the larger task, rather than providing a predictive representation. For example, BID1 formulated a gradient descent update on option parameters-policies and termination functions-using policy gradient objectives. The difficulty in extending such gradient-based approaches is in specifying a suitable objective for prediction accuracy, which is difficult to measure online. We take inspiration from ideas from representation search methods developed for neural networks, to tackle the daunting challenge of GVF discovery for predictive representations. Our approach is inspired by algorithms that search the topology space of neural networks. One of the first such approaches, called the cascade correlation network learning typifies this approach BID7 ). The idea is to continually propose new hidden units over time, incrementally growing the network to multiple levels of abstraction. To avoid the computation and memory required to pose units whose activation is de-correlated with the network activations, BID24 empirically demonstrated that simply generating large numbers of hidden units outperformed equal sized fixed networks in online supervised learning problems. Related approaches demonstrated that massive random representations can be highly effective (; BID0 BID8 . This randomized feature search can be improved with the addition of periodic pruning, particularly for an incremental learning setting. In this paper, we demonstrate such a curation framework for GVF discovery, with simple algorithms to propose and prune GVFs. To parallel representation search, we need both a basic functional form-a GVF primitive-for each unit in the network and an update to adjust the weights on these units. We propose a simple set of GVF primitives, from which to randomly generate candidate GVFs. We develop a regularized updating algorithm, to facilitate pruning less useful GVFs, with a stepsize adaptation approach that maintains stability in the representation. We demonstrate both the ability for the regularizer to prune less useful GVFs-and the corresponding predictive features-as well as utility of the GVF primitives as predictive features in several partially observable domains. Our approach provides a first investigation into a framework for curation of GVFs for predictive representations, with the aim to facilitate further development. The setting considered in this paper is an agent learning to predict long-term outcomes in a partially observable environment. The dynamics are driven by an underlying Markov decision process (MDP), with potentially (uncountably) infinite state-space S and action-space A, and transitions given by the density P: S × A × S → [0, ∞). The agent does not see the underlying states, but rather only observes observation vector o t ∈ O ⊂ R m for corresponding state s t ∈ S. The agent follows a fixed behaviour policy, µ: S × A → [0, ∞), taking actions according to µ(s t, ·). Though we write this policy as a function of state, the behaviour policy is restricted to being a function of input observations-which is itself a function of state-and whatever agent state the agent constructs. The agent aims to build up a predictive representation, p t = f (o t, p t−1) ∈ R n for some function f, as a part of its agent state to overcome the partial observability. GVF networks provide such a predictive representation, and are a generalization on PSRs and TD networks. A General Value Function (GVF) consists of a target policy π: S × A → [0, ∞), discount function γ: S × A × S → and cumulant r: S × A × S → R. The value function v π: S → R is defined as the expected return, G t, where DISPLAYFORM0 A GVF prediction, for example, could provide the probability of hitting a wall, if the agent goes forward, by selecting a policy that persistently takes the forward action, a cumulant that equals 1 when a wall is hit and 0 otherwise and a discount of 1, until termination. We direct the reader to prior work detailing the specification and expressiveness of GVFs BID23; we also provide further examples of GVFs throughout this paper. Figure 1: The inputs-observations o t+1 and GVF predictions p t from the last time step-pass through a nonlinear expansion, such as a fixed neural network or tile coding, producing the feature vector p t+1. The feature vector is weighted linearly to produce the next set of predictions p t+1. This approach decouples the specification of the representation for the learner, which consist of both observations and predictive features, and the updating algorithm. Such a framework could be modified to include a history of observations; for simplicity here, we only consider using predictions to overcome partial observability and do not maintain histories of observations. A GVF network consists of a set of GVFs DISPLAYFORM1 where the prediction vector p t ∈ R n consists of the approximate value functions DISPLAYFORM2 and is computed as a function of the current observation and predictions of the previous step p = f (o t, p t−1). This network is depicted in Figure 1. GVF networks can encode both PSRs and TD networks, providing a general formalism for predictive representations; for space, we more thoroughly describe the relationships to these approaches in Appendix A.There has been some work towards the goal of learning f to provide predictions p t, in the original papers on TD networks and TD networks with options and with a following algorithm using recurrent gradient updates . Additionally, there has been quite a bit of work in off-policy learning, with gradient temporal difference (GTD) algorithms BID22 BID13, which can be used to estimate the value functions for a GVF policy π, from experience generated by a different behaviour policy µ. We leverage these works, to develop a new updating approach for GVF networks in the next section. When learning in a partially observable domain, the agent inherently faces a non-stationary problem. From the agent's perspective, a similar observation is observed, but the target may vary significantly, because hidden variables are influencing the outcome. For such settings, tracking has been shown to be critical for accuracy BID25, even when the underlying distribution is in fact stationary. Tracking-continually updating the weights with recent experience-contrasts the typical goal of convergence; much of the previous algorithm development, however, has been towards the aim of convergence .We propose treating the learning system as a dynamical system-where the weight update is based on stochastic updates known to suitably track the targets-and consider the choice of stepsize as the inputs to the system to maintain stability. Such updates have been previously considered under adaptive gain for least-mean squares (LMS) BID3, Chapter 4) , where weights are treated as state following a random drift. These approaches, however, are designed particularly for the LMS update and so do not extend to the off-policy temporal difference learning algorithms needed to learn GVFs. To generalize this idea to other incremental algorithms, we propose a more general criteria based on the magnitude of the update. Consider a generic update DISPLAYFORM0 where ∆ t ∈ R d is the update for this step, for weights w t ∈ R d and constant stepsize α. Typically the update includes a normalization constant c t, dependent on the norm of the features and target. For example, for normalized LMS predicting target y t from observation vector o t, ∆ t = o t (y t − o t w t) and c t = o t o t 2 2 + an estimate of the variance of the noise in the targets. Such normalization ensures the update appropriately reflects descent direction, but is invariant to scale of the features and targets. The weights w t evolve as a function of the previous weights, with stepsize α acting as a control input for how this system evolves. A criteria for α to maintain stability in the system is to keep the norm of the update small DISPLAYFORM1 for a small > 0 that provides a minimum stepsize. The update ∆ t (α) on this time step is dependent on the stepsize α, because that stepsize influences w t and past updates. The expected value is over all possible update vectors ∆ t (α), for the given stepsize and assuming the system started in some w 0. If α is small enough to ensure updates are bounded, and policy π and MDP satisfy the standard requirements for ergodicity, a stationary distribution exists, with ∆ t (α) not dependent on the initial w 0 and instead only driven by the underlying state dynamics and target for the weights. In the next sections, we derive an algorithm to estimate α for this dynamical system, first for a general off-policy learning update and then when adding regularization. We call this algorithm AdaGain: Adaptive Gain for Stability. We generically consider an update ∆ t in that includes both TD and GTD. Before deriving the algorithm, we demonstrate concrete updates for the stepsize. For TD, the update is DISPLAYFORM0 where α 0 = 1.0, ψ 1 = 0,ᾱ is a meta-stepsize and β is a forgetting parameter to forget old gradient information (e.g., β = 0.01). The operator (·) thresholds any values below > 0 to (e.g., = 0.001), ensuring nonzero stepsizes. Another canonical algorithm for learning value functions is GTD(λ), with trace parameter DISPLAYFORM1 with importance sampling correction ρ t DISPLAYFORM2 For the auxiliary weights h t -which estimate a part of the GTD objective-we use a small, fixed stepsize α (h) t = 0.01, previously found to be effective BID30.We consider the derivation more generally for such temporal difference methods, where both TD and GTD arise as special cases. Consider any update of the form DISPLAYFORM3 for vectors e t, u t not dependent on w t. For GTD(λ), u t = −γ t+1 (1 − λ t+1)x t+1. We minimize using stochastic gradient descent, with gradient for one sample of the norm of the update DISPLAYFORM4 where d t def = γ t+1 x t+1 − x t and we can recursively define ∂wt ∂α as DISPLAYFORM5 We obtain a similar such recursive relationship for DISPLAYFORM6 where the last line follows from the fact that DISPLAYFORM7 This recursive form provides a mechanism to avoid storing all previous samples and eligibility traces, and still approximate the stochastic gradient update for the stepsize. Though the above updates are exact, when implementing such a recursive form in practice, we can only obtain an estimate of ψ t, if we want to avoid storing all past data. In particular, when using ψ t−1 computed on the last time step t − 1, this gradient estimate is in fact w.r.t. to the previous stepsize α t−2, rather than α t−1. Because these stepsizes are slowly changing, this gradient still provides a reasonable estimate of the actual ψ t−1 for the current stepsize. However, for many steps into the past, these accumulates gradients in ψ t andψ t are inaccurate. For example, even if the stepsize is nearing the optimal value, ψ t will include larger gradients from the first step when the stepsizes where inaccurate. To forget these outdated gradients, we maintain an exponential moving average, which focuses the accumulation of gradients in ψ t to a more recent window. The adjusted update with forgetting parameter 0 < β < 1 gives the recursive form for ψ t+1 andψ t+1 in. A regularized GTD update for the weights can both reduce variance from noisy predictions and reduce weight on less useful features to facilitate pruning. To add regularization to GTD, for regularizer R(w) and regularization parameter η ≥ 0, we can use proximal updates BID14, DISPLAYFORM0 where prox αηR is the proximal operator for function αηR. The proximal operator acts like a projection, first updating the weights according to the GTD objective and then projecting the weights back to a solution that appropriately reflects the properties encoded by the regularizer. A proximal operator exists for our proposed regularizer, the clipped 2 regularizer DISPLAYFORM1 where > 0 is the clipping threshold above which w i has a fixed regularization. Though other regularizers are possible, we select this clipped 2 regularizer for two reasons. The clipping ensures that high magnitude weights are not prevented for being learned, and reduces bias from shrinkage. Because the predictive representation requires accurate GVF predictions, we found the bias without clipping prevented learning. Additionally, we chose 2 in the clipping, rather than 1, because the clipping already facilitates pruning, and does not introduce additional non-differentiability. The regularization below still prefers to reduce the magnitude of less useful features. For example, if two features are repeated, such a regularizer will prefer to have a higher magnitude weight on one feature, and zero weight on the other; no regularizer, or 2 without clipping, will needlessly use both features. We provide the derivation-which closely parallels the one above-and updates in Appendix B. In this section we show that AdaGain with regularization (AdaGain-R) reduces the weights on less useful features. This investigation demonstrates the utility of the algorithm for pruning features, and so also for pruning proposed GVFs. We first test AdaGain-R on a robot platform-which is partially observable-with fixed image features. We provide additional experiments in two micro-worlds in Appendix C for pruning GVFs. The first experiment is learning a GVF prediction on a robot, using a nonlinear expansion on input pixels. The robotic platform is a Kabuki rolling robot with an added ASUS XtionPRO RGB and Depth sensor. The agent receives a new image every 0.05 seconds and 100 random pixels are sampled to construct the state. Each pixel's RGB values are tiled with 4 tiles and 4 tilings on each colour channel, ing in 4800 bit values. A bias bit is also included, with a value of 1 on each time step. The fixed behaviour policy is to move forward until a wall is hit and then turn for a random amount of time. The goal is to learn the value function for a policy that always goes forward; with a cumulant of 1 when the agent bumps a wall and otherwise 0; and a discount of 0.97 everywhere except when the agent bumps the wall, ing in a discount of 0 (termination). The GVF is learned off-policy, with GTD(λ) and AdaGain-R with the GTD(λ) updates. Both GTD(λ) and AdaGain-R receive the same experience from the behaviour policy. Results are averaged over 7 runs. Our goal is to ascertain if AdaGain-R can learn in this environment, and if the regularization enables it to reduce magnitude on features without affecting performance. FIG1 (a) depicts a sample trajectory of the predictions made by both algorithms, after about 40k learning steps; both are able to track the return accurately. This is further emphasized in FIG1 (b), where averaged error decreases over time. Additionally, though they reach similar performance, AdaGain-R only has significant magnitude on about half of the features. Our approach to generating a predictive representation is simple: we generate a large collection of GVFs and iteratively refine the representation through replacing the least used GVFs with new GVFs. In the previous section, we provided evidence that our AdaGain-R algorithm effectively prunes features; we now address the larger curation framework in this section. We provide a set of GVF primitives, that enable candidates to be generated for the GVF network. We demonstrate the utility of this set, and that iteratively pruning and generating GVFs in our GVF network builds up an effective predictive representation. To enable generation of GVFs for this discovery approach, we introduce GVF primitives. The goal is to provide modular components that can be combined to produce different structures. For example, within neural networks, it is common to modularly swap different activation functions, such as sigmoidal or tanh activations. For networks of GVFs, we similarly need these basic units to enable definition of the structure. We propose basic types for each component of the GVF: discount, cumulant and policy. For discounts, we consider myopic discounts (γ = 0), horizon discounts (γ ∈) and termination discounts (the discount is set to 1 everywhere, except for at an event, which consists of a transition (o, a, o)). For cumulants, we consider stimuli cumulants (the cumulant is one of the observations, or inverted, where the cumulant is zero until an observation goes above a threshold) and compositional cumulants (the cumulant is the prediction of another GVF). We also investigate random cumulants (the cumulant is a random number generated from a zero-mean Gaussian with a random variance sampled from a uniform distribution); we do not expect these to be useful, but they provide a baseline. For the policies, we propose random policies (an action is chosen at random) and persistent policies (always follows one action). For example, a GVF could consist of a myopic discount, with stimuli cumulant on observation bit one and a random policy. This would correspond to predicting the first component of the observation vector on the next step, assuming a random action is taken. As another example, a GVF could consist of a termination discount, an inverted stimuli cumulant for observation one and a persistent policy with action forward. If observation can only be'0' or'1', this GVF corresponds to predicting the probability of seeing observation one changing to'0' (inactive) from'1' (active), given the agent persistently drives forward. We conduct experiments on our discovery approach for GVF networks in Compass World BID20, a partially observable grid-world where the agent can only see the colour immediately in front of it. There are four walls, with different colours; the agent observes this colour if it takes the action forward in front of the wall. Otherwise, the agent just sees white. There are five colours in total, with one wall having two colours and so more difficult to predict. The observation vector is fivedimensional, consisting of an indicator bit if the colour is observed or not. We test the performance of the learned GVF network for answering five difficult GVF predictions about the environment, that cannot be learned using only the observations. Each difficult GVF prediction corresponds to a colour, with the goal to predict the probability of seeing that colour, if the agent always goes forward. These GVFs are not used as part of the representation. A priori, it is not clear that the GVF primitives are sufficient to enable prediction in the Compass World, particularly as using just the observations in this domain enables almost no learning of these five difficult GVFs. The GVFs for the network are generated uniformly randomly from the set of GVF primitives. Because the observations are all one bit (0 or 1), the stimuli cumulants are generated by selecting a bit index i (1 to 5) and then either setting the cumulant to that observation value, o i, or to the inverse of that value, 1 − o i. The events for termination are similarly randomly generated, with the event corresponding to a bit o i flipping. The nonlinear transformation used for this GVF network is the hyperbolic tangent. Every two million steps, the bottom 10% of the current GVFs are pruned and replaced with newly generated GVFs. Results are averaged over 10 runs. FIG2 demonstrates that AdaGain-R with randomly generated GVF primitives learns a GVF network-and corresponding predictive representation-that can accurately predict the five difficult GVFs. The show that with as few as 100 GVFs in the network, accurate predictions can be learned, though increasing to 200 is a noticeable improvement. The also indicate that random cumulants were of no benefit, as expected, and that our system appropriately pruned those GVFs. Finally, compositional GVFs were particularly beneficial in later learning, suggesting that the system started to make better use of these compositions once the GVFs became more accurate. In this paper, we proposed a discovery methodology for GVF networks, to learn of predictive representations. The strategy involves iteratively generating and pruning GVF primitives for the GVF network, with a new algorithm called AdaGain to promote stability and facilitate pruning. The demonstrate utility of this curation strategy for discovering GVF networks. There are many aspects of our system that could have been designed differently, namely in terms of the learning algorithm, generation approach and pruning approach; here, our goal was to provide a first such demonstration, with the aim to facilitate further development. We discuss the two key aspects in our system below, and potential avenues to expand along these dimensions. In the development of our learning strategy, we underline the importance of treating the predictive representation as a dynamical system. For a standard supervised learning setting, the representation is static: the network can be queried at any time with inputs. The predictive representations considered here cannot be turned off and on, because they progressively build up accurate predictions. This dynamic nature necessitates a different view of learning. We proposed a focus on stability of the predictive system, deriving an algorithm to learn a stepsize. The stepsize can be seen as a control input, to stabilize the system, and was obtained with a relatively straightforward descent algorithm. More complex control inputs, however, could be considered. For example, a control function outputting a stepsize based on the current agent state could be much more reactive. Such an extension would the necessitate a more complex stability analysis from control theory. Our discovery experiments reflect a life-long learning setting, where the predictive representation is slowly built up over millions of steps. This was slower than strictly necessary, because we wanted to enable convergence for each GVF network before culling. Further, the pruning strategy was simplistic, using a threshold of 10%; more compact GVF networks could likely be learned-and learned more quickly-with a more informed pruning approach. Nonetheless, even when designing learning with less conservative learning times, building such representations should be a long-term endeavour. A natural next step is to more explicitly explore scaffolding. For example, without compositional GVFs, myopic discounts were less frequently kept; this suggests initially preferring horizon and termination discounts, and increasing preference on myopic discounts once compositional GVFs are added. Further, to keep the system as simple as possible, we did not treat compositional GVFs differently when pruning. For example, there is a sudden rise in prediction error at about 80 million steps in FIG2 (b); this was likely caused by pruning a GVF whose prediction was the cumulant for a critical compositional GVF. Finally, we only considered a simple set of GVF primitives; though this simple set was quite effective, there is an opportunity to design other GVF primitives, and particularly those that might be amenable to composition. There have been several approaches proposed to deal with partial observability. A common approach has been to use history: the most recent p observations o t−1,..., o t−p. For example, a blind agent in the middle of an empty room can localize itself using a history of information. Once it reaches a wall, examining its history can determine how far away it is from a wall. This could clearly fail, however, if the history is too short. Predictive approaches, like PSRs and TD networks, have been shown to overcome this issue. Further, PSRs have been shown to more compactly represent state, than POMDPs BID12.A PSR is composed of a set of action-observations sequences and the corresponding probability of these observations occurring given the sequence of actions. The goal is to find a sufficient subset of such sequences (core tests) to determine the probability of all possible observations given any action sequence. PSRs have been extended with the use of options BID31, and discovery of the core tests BID18. A PSR can be represented as a GVF network by using myopic γ = 0 and compositional predictions. For a test a 1 o 1, for example, to compute the probability of seeing o 1, the cumulant is 1 if o 1 is observed and 0 otherwise. To get a longer test, say a 0 o 0 a 1 o 1, a second GVF can be added which predicts the output of the first GVF (i.e., the probability of seeing o 1 given a 1 is taken), with fixed action a 0. This equivalence is only for computing probabilities of sequences of observations, given sequences of actions. GVF networks specify the question, not the answer, and so GVF networks do not encompass the discovery methods or other nice mathematical properties of PSRs, such as can be obtained with linear PSRs. A TD network is similarly composed of n predictions on each time step, and more heavily uses compositional questions to obtain complex predictions. Similarly to GVF networks, on each step, the predictions from the previous step and the current observations are used for this step. The targets for the nodes can be a function of the observation, and/or a function of another node (compositional). TD networks are restricted to asking questions about the outcomes from particular actions, rather than about outcomes from policies. TD networks with options BID21 BID20 were introduced, to generalize to temporally extended actions. TD networks with options are almost equivalent to GVF networks, but have small differences due to generalizations to return specifications-in GVFs-since then. For example, options have terminating conditions, which corresponds to having a fixed discount during execution of the option and a termination discount of 0 at the end of the options. GVFs allow for more general discount functions. Additionally, TD networks, both with and without options, have a condition function. The generalization to policies, to allowing action-values to be learned rather than just value functions and the use of importance sampling corrections, encompasses these functions. The key differences, then, between GVF networks and TD networks is in how the question networks are expressed and subsequently how they can be answered. GVF networks are less cumbersome to specify, because they use the language of GVFs. Further, once in this language, it is more straightforward to apply algorithms designed for learning GVFs. There are some algorithmic extensions to TD networks that are not encompassed by GVFs, such as TD networks with traces BID27. A proximal operator for a function R with weighting αη is defined as DISPLAYFORM0 2 + αηR(u) Though proximal gradient algorithsm are typically considered for convex regularizers, the proximal gradient update can be applied for our nonconvex regularizer because our proximal operator has a unique solution BID32. The proximal operator for the clipped 2 regularizer is defined element-wise, for each entry in w: DISPLAYFORM1 The derivation of AdaGain with a proximal operator is similar to the derivation of AdaGain without regularization. The only difference is in the gradient of the weights, ψ t = ∂ ∂α w t, with no change in the gradients of δ t and of h t (i.e.,ψ t). Because the proximal operator has non-differentiable points, we can only obtain a subderivative of the proximal operator w.r.t. to the stepsize. For gradient descent, we do need to more carefully use this subderivative; in practice, however, using a subderivative within the stochastic gradient descent update seems to perform well. We similarly found this to be the case, and so simply use the subderivative in our stochastic gradient descent update. To derive the update, letw = w t + α∆ t be the weights before applying the proximal operator. The subderivative of the proximal operator w.r.t. α, which we call dprox αηR, is DISPLAYFORM2 The proximal operator usesw t,i andψ t,i for the element-wise update. The ing updates, including the exponential average with forgetting parameter β, is DISPLAYFORM3: otherwise with ψ 1 =ψ 1 =ψ 1 = 0. The initial step size for GTD(λ) and AdaGain-R was chosen to be 0.1, with the step size of GTD(λ) normalized over the number of active features (same as AdaGain-R's normalization factor). AdaGain-R has a regularization parameters τ = 0.001, η = 1 normalized over the number of active features, and a meta-stepsizeᾱ = 1.0. When generating questions on the fly it is hard to know if certain questions will be learnable, or if their answers are harmful to future learning. To investigate the utility of our system for ignoring dysfunctional predictive features within the network, we conducted a small experiment in a six state cycle world domain BID26. Cycle World consists of six states where the agent progresses through the cycle deterministically. Each state has a single observation bit set to zero except for a single state with an observation of one. We define seven GVFs for the GVF network to learn in Cycle World. Six of the GVFs correspond to the states of the cycle. The first of these GVFs has the observation bit as its cumulant, and a discount of zero. The goal for this GVF prediction is to predict the observation bit it expects to see on the next time step, which must be a'0' or a'1'. The second GVF predicts the prediction of this first GVF on the next steps: it's cumulant is the prediction from the first GVF on the next step. For example, imagine that on this step the first GVF accurately predicts a'1' will be observed on the next step. This means that on the next step, to be accurate the first GVF should predict that a'0' will be observed. Since the second GVF gets this prediction for the next time step, its target will be a'0' for this time step, and it will be attempting to predict the observation bit in two time steps. Similar, the cumulant for the third GVF is the second GVFs prediction on the next time step, and so it is aiming Figure 4: The progression of improved stability, with addition of components of our system. Even without a nonlinear transformation, AdaGain can maintain stability in the system (in (a)), though the meta-step size needs to be set more aggressively small. Interestingly, the addition of regularization significantly improved stability, and even convergence rate (in (b)). The seventh GVF is used more aggressively after Phase 1-once it becomes useful. The addition of a nonlinear transformation (in (c)) then finally lets the system react quickly once the seventh GVF becomes accurate. Again, without regularization, the magnitude of weights is more spread out, but otherwise the performance is almost exactly the same as (c). For both (b) and (c), the meta-parameters are η = 0.1, = 0.01 normalized with the number of features.to predict the observation bit in three time steps. The fourth, fifth and sixth GVFs correspondingly aim to predict the observation bit in four, five and six time steps respectively. These six GVFs all have a discount of 0, since they are predicting immediate next observations. Finally, the seventh GVF reflects the likelihood of seeing a'1' within a short horizon, where the cumulant is the observation bit on the next step and the discount is 0.9 when the observation is'0' and 0.0 when the observation is'1'. With only the first six GVFs in the GVF network, the network cannot learn to make accurate predictions BID26 ). By adding the seventh GVF, the whole network can reach zero error; we add this GVF, therefore, to make this domain suitable to test the algorithms. Though not the intended purpose of this experiment, it is illuminating to see the immediate benefits of the expanded flexibility of the language of GVFs beyond TD networks. In these experiments we want to measure our system's ability to stabilize learning in a situation where a GVF in the representation is dysfunctional. To simulate the circumstance of a harmful question, we replace the seventh GVF-the critical GVF for reducing error-with random noise sampled from a Gaussian of mean and variance 0.5. When this first phase is complete, after 50k time steps, we replace the noise with the unlearned critical GVF and measure the prediction error of the system. From Figure 4, we demonstrate that AdaGain enables stable learning under this perturbed learning setting. When the seventh GVF contained noise, AdaGain quickly dropped the step size for the others GVFs to the lower threshold. This also occurred when learning without the seventh GVF, and would prevent the instability seen in Cycle World in previous work BID26 ). After Phase 1, once the seventh GVF begins to learn, the step-sizes are increased. In this case, the addition of a regularizer also seems to improve stability. For this experiment, there is not as clear a separation between the adapting the step-size for stability and using regularization to prune features; in fact, they both seem to play this role to some extent. The overall system, though, effectively handles this dysfunctional feature. We demonstrate the ability of our system to put higher preference on more useful GVFs in the compass world domain, and the effect of pruning less used GVFs. We construct a network with 45 expert GVFs defined in BID20, and 155 GVFs which produce noise sampled from a gaussian of mean 0 and variance randomly select by a uniform distribution. We prune 20 GVFs every two million steps based on the average magnitude of the feature weights, all other parameters are the same for the experiments above. Because the handcrafted expert GVFs contain useful information they should be used more by our system. Similarly to the experiments in section 5.2, we use the five evaluation GVFs to measure the representation. As we can see in FIG5, AdaGain-R does mostly remove the dysfunctional GVFs first, and when the expert GVFs are pruned the representation isn't damaged until the penultimate prune. These also show how pruning dysfunctional or unused GVFs from a representation is not harmful to the learning task. The instability seen in the ends of learning can be overcome by allowing the system to generate new GVFs to replace those that were pruned and by pruning a small amount based on the size of network used as a representation.

represent the short summary for finding the corresponding scientific article (long) Given We propose a method of transferring knowledge between related RL tasks using visual mappings, and demonstrate its effectiveness on visual variants of the Atari Breakout game and different levels of Road Fighter, a Nintendo car driving game., a positive would be Deep Reinforcement Learning has managed to achieve state-of-the-art in learning control policies directly from raw pixels. However, despite its remarkable success, it fails to generalize, a fundamental component required in a stable Artificial Intelligence system. Using the Atari game Breakout, we demonstrate the difficulty of a trained agent in adjusting to simple modifications in the raw image, ones that a human could adapt to trivially. In transfer learning, the goal is to use the knowledge gained from the source task to make the training of the target task faster and better. We show that using various forms of fine-tuning, a common method for transfer learning, is not effective for adapting to such small visual changes. In fact, it is often easier to re-train the agent from scratch than to fine-tune a trained agent. We suggest that in some cases transfer learning can be improved by adding a dedicated component whose goal is to learn to visually map between the known domain and the new one. Concretely, we use Unaligned Generative Adversarial Networks (GANs) to create a mapping function to translate images in the target task to corresponding images in the source task. These mapping functions allow us to transform between various variations of the Breakout game, as well as between different levels of a Nintendo game, Road Fighter. We show that learning this mapping is substantially more efficient than re-training. A visualization of a trained agent playing Breakout and Road Fighter, with and without the GAN transfer, can be seen in \url{https://streamable.com/msgtm} and \url{https://streamable.com/5e2ka}. Transferring knowledge from previous occurrences to new circumstances is a fundamental human capability and is a major challenge for deep learning applications. A plausible requirement for artificial general intelligence is that a network trained on one task can reuse existing knowledge instead of learning from scratch for another task. For instance, consider the task of navigation during different hours of the day. A human that knows how to get from one point to another on daylight will quickly adjust itself to do the same task during night time, while for a machine learning system making a decision based on an input image it might be a harder task. That is because it is easier for us to make analogies between similar situations, especially in the things we see, as opposed to a robot that does not have this ability and its knowledge is based mainly on what it already saw. Deep reinforcement learning has caught the attention of researchers in the past years for its remarkable success in achieving human-level performance in a wide variety of tasks. One of the field's famous achievements was on the Atari 2600 games where an agent was trained to play video games directly from the screen pixels and information received from the game BID20. However, this approach depends on interacting with the environment a substantial number of times during training. Moreover, it struggles to generalize beyond its experience, the training process of a new task has to be performed from scratch even for a related one. Recent works have tried to overcome this inefficiency with different approaches such as, learning universal policies that can generalize between related tasks BID25, as well as other transfer approaches BID7 BID24. In this work, we first focus on the Atari game Breakout, in which the main concept is moving the paddle towards the ball in order to maximize the score of the game. We modify the game by introducing visual changes such as adding a rectangle in the middle of the image or diagonals in the . From a human perspective, it appears that making visual changes that are not significant to the game's dynamics should not influence the score of the game, a player who mastered the original game should be able to trivially adapt to such visual variants. We show that the agent fails to transfer. Furthermore, fine-tuning, the main transfer learning method used today in neural networks, also fails to adapt to the small visual change: the information learned in the source task does not benefit the learning process of the very related target task, and can even decelerate it. The algorithm behaves as if these are entirely new tasks. Our second focus is attempting to transfer agent behavior across different levels of a video game: can an agent trained on the first level of a game use this knowledge and perform adequately on subsequent levels? We explore the Nintendo game Road Fighter, a car racing game where the goal is to finish the track before the time runs out without crashing. The levels all share the same dynamics, but differ from each other visually and in aspects such as road width. Similar to the Breakout , an agent trained to play the first level fails to correctly adapt its past experience, causing the learned policy to completely fail on the new levels. To address the generalization problem, we propose a zero-shot generalization approach, in which the agent learns to transfer between related tasks by learning to visually map images from the target task back to familiar corresponding images from the source task. Such mapping is naturally achieved using Generative Adversarial Networks (GANs) BID9, one of the most popular methods for the image-to-image translation that is being used in computer vision tasks such as style transfer BID15, object transfiguration BID31, photo enhancement BID17 and more recently, video game level generation BID27. In our setup, it is not realistic to assume paired images in both domains, calling for the use of Unaligned GANs BID19 BID15. Using this approach we manage to transfer between similar tasks with no additional learning. Contributions This work presents three main contributions. First, in Section 2, we demonstrate how an agent trained with deep reinforcement learning algorithms fails to adapt to small visual changes, and that the common transfer method of fine-tuning fails as well. Second, in Section 3, we propose to separate the visual mapping from the game dynamics, ing in a new transfer learning approach for related tasks based on visual input mapping. We evaluate this approach on Breakout and Road Fighter, and present the comparing to different baselines. We show that our visual transfer approach is much more sample efficient then the alternatives. Third, in section 5, we suggest an evaluation setup for unaligned GAN architectures, based on their achieved performance on concrete down-stream tasks. Many Breakout variations can be constructed that involve the same dynamics. The main idea is to make modifications that are not critical for a human playing the game but are for the algorithm that relies on visual inputs. We demonstrate the difficulty of deep reinforcement learning to generalize using 4 types of modifications as presented in FIG0. For all the experiments in this section forward we use the Asynchronous Advantage Actor-Critic (A3C) algorithm BID21, taking advantage of being faster than Deep Q-Network (DQN) BID20. The A3C learns the policy and the state-value function using parallel actor-learners exploring different policies for the acceleration and stability of the training. We rescale the image to 80 × 80 and keep the RGB colors for more realistic images. We use 32 actor learners, a discount rate of 0.99, learning rate of 0.0001, 20-step returns, and entropy regularization weight of 0.01. The A3C variation we choose is the LSTM-A3C network. We use the standard high-performance architecture implemented in BID16. The setup mentioned in 2.1 successfully trains on Breakout, reaching a score of over 400 points. However, when a network trained on the original game is presented with the game variants, it fails completely, reaching to a maximum score of only 3 points. This shows that the network does not necessarily learn the game's concepts and heavily relies on the images it receives. The common approach for transferring knowledge across tasks is fine-tuning. We experiment with common techniques used in deep learning models. In each setting, we have a combination of frozen and fine-tuned layers (Partial/Full) as well as layers that are initialized with the target's parameters and layers that are initialized with random values (Random). Our settings are inspired by BID29. We train each one of the tasks (before and after the transformation) for 60 million frames, and our evaluation metric is the total reward the agents collect in an episode averaged by the number of episodes, where an episode ends when the game is terminated or when a number of maximum steps is reached. We periodically compute the average during training. We consider the following settings:• From-Scratch: The game is being trained from scratch on the target game.• Full-FT: All of the layers are initialized with the weights of the source task and are fine-tuned on the target task.• Random-Output: The convolutional layers and the LSTM layer are initialized with the weights of the source task and are fine-tuned on the target task. The output layers are initialized randomly.• Partial-FT: All of the layers are initialized with the weights of the source task. The three first convolutional layers are kept frozen, and the rest are fine-tuned on the target task.• Partial-Random-FT: The three first convolutional layers are initialized with the weights of the source task and are kept frozen, and the rest are initialized randomly. The presented in FIG1 show a complete failure of all the fine-tuning approaches to transfer to the target tasks. In the best scenarios the transfer takes just as many epochs as training from scratch, while in other cases starting from a trained network makes it harder for the network to learn the target task. As the graphs show, some of the modification influenced more than others. For example, FIG1 shows that adding a simple rectangle can be destructive for a trained agent: while training from scratch consistently and reliably achieves scores over 300, the settings starting from a trained agent struggle to pass the 200 points mark within the same number of iterations, and have a very high variance. We noticed that during training the agent learns a strategy to maximize the score with a minimum number of actions. None of the experiments we performed showed better when the layers in the network were fine-tuned, and some showed negative transfer which is a clear indication of an overfitting problem. The A3C model learned the detail and noise in the training data to the extent that it negatively impacted the performance of the model on new data. Our and drawn from them are consistent with the shown when a similar approach was used on Pong BID24. In addition to Breakout, we also experimented transfer between the first and advanced level of Road Fighter, where the s change but the dynamics remains the same. This experiments ed with 0 points on each of the levels, a complete failure of the agent to re-use the driving techniques learned on the first levels on the next ones. An agent capable of performing a task in a source domain is now presented with a new domain. Fine-tuning the agent on the target domain fails to transfer knowledge from the source domain. We propose to separate the visual transfer from the dynamics transfer. To perform well, the agent can try and make analogies from the new domain to the old one: after observing a set of states (images) in the new domain, the agent can learn to map them to similar, familiar states from the source domain, and act according to its source domain policy on the mapped state. More concretely, given a trained policy π(a|s; θ) with trained parameters θ proposing an action a for source domain states s ∈ S, we wish to learn a mapping function G: T → S from target domain states t ∈ T such that interacting with the environment T by applying the policy π(a|G(t); θ) will in a good distribution of actions for the states T, as indicated by high overall scores. In other words, we seek a mapping function G that allows us to re-use the same policy π θ learned for source environment S when interacting with the target environment T.As both the source and target domain items are images, we heuristically learn the function G by collecting sets of images from S and T and learning to visually map between them using Unaligned GAN BID19 BID15. We use the scores obtained from interacting with the environment via π(a|G(t); θ) for the GAN model selection and stopping criteria. In this work, we focus on learning setups that receive only raw image data, without additional domain knowledge about objects or game dynamics. This prohibits us from using supervised paired GANs BID13 for learning the mapping function G: we cannot collect the needed supervision of corresponding (s, t) pairs. Instead, we use unaligned GANs BID19 BID15, in which the learner observes two sets of images, one from each domain, with the goal of learning to translate images in one domain to images in another. All major approaches to the unaligned image-to-image translation use the Cycle-Consistency principle. We have two mapping (encoding) functions DISPLAYFORM0 is a set of images collected from the source task and T = {t j} M j=1 is a set of images collected from the target task. The goal is to generate an image s, for any given t ∈ T where G 1 (t) = s, that is indistinguishable from s ∈ S. The cycle consistency principle relies on the assumption that the two functions, G 1 and G 2 are inverses of each other. It encourages unsupervised mapping by forcing G 2 (G 1 (t)) = t and G 1 (G 2 (s)) = s where s and t are the input images. The second component of the GAN architecture are the discriminators D 1 and D 2 aiming to distinguish between images generated by G 1 and G 2 and the real images from the target and source distributions respectively. In the following experiments, we use the UNIT framework BID19, which we found to perform well for the Breakout tasks (in section 5 we explicitly compare the UNIT and CycleGAN approaches on both the Breakout and Road Fighter transfer tasks). A distinguishing element in the UNIT framework is the shared-latent space assumption, according to which there is a shared-latent space consisting a shared latent code z for any pair of images s and t that can be recovered from this code. This share-latent space is represented as the weights of the last few layers of the encoding network and the few first layers of the decoding networks, and is learned by using Variational Autoencoders (VAEs). This sharing strongly ties the images in the source and target domain to each other, encouraging mappings that preserve similarities across domains. In contrast, the CycleGAN architecture does not make the shared space assumption and instead the generators are trained independently with two separate networks. For further information of the unaligned GAN architectures, see the original papers. Datasets. The Unaligned GAN training dataset requires images from both domains. We collect images from the source domain by running an untrained agent and collecting the observed images, and we do similarly for the target domain. The number of collected images should balance between two objectives: On the one hand, we want to take a small number of images, and on the other hand, it is essential for us to have a diverse dataset. We repeat this procedure for every target task, and create a source-target dataset for each. During training, we further ensure the images pairs are not aligned by randomly picking an image from each set at each iteration. Setup and Analysis. For our experiments we use the same architecture and hyper-parameters proposed in the UNIT paper. We initialize the weights with Xavier initialization BID8, set the batch size to 1 and train the network for a different number of iterations on each task. Some tasks are harder than others, the more changes exist in the frames the harder it is for the GAN to learn the mapping between the domains. However, our evaluation metric, testing the agent with the generated images, is a clear indication of how hard each task is and the number of iterations needed is based on the of this evaluation. Evaluation. We use GAN training to learn a mapping function G. GAN training, and unaligned GANs training in particular, are unstable and it challenging to find a good loss-based stopping criteria for them. A major issue with GANs is the lack of an evaluation metric that works well for all models and architectures, and which can assist in model selection. Different works use different methods that were suitable for their types of data. Our setup suggests a natural evaluation criteria: we run the source agent without any further training while using the model to translate each image of the target task back to the source task and collect the rewards the agent receives during the game when presented with the translated image. We use the total accumulated rewards (the score) the agent collects during the game as our model accuracy. We examine how well the agent does when receiving translated frames generated by the generator trained with GANs. Since we initialized the layers with the values of the trained network, we assume that the success of the agent is dependent on the similarity between the generated and the source task's frames. We first test our approach on Breakout, evaluating its ability to remove the changes added in the images. Second, we challenge our method even more on Road Fighter, where the goal is to transfer between different environments. Our goal in Breakout is removing the modifications in each one of the target tasks and transfer between each to the original game. Although the variations share many similarities, some tasks were more challenging than others, e.g., the lines of the Green Lines variation hide parts of the ball in Table 1: The score and number of frames needed for it of: the source task (Source), target task when initialized with the source task' network parameters with no additional training (Target) and the target task when initialized with the source task' network parameters where every frame is translated to a frame from the source task (Target with GANs). Target Task Target Target DISPLAYFORM0 Figure 3: Illustration of a frame taken from the target task (left) and its matching frame of the source task generated with GANs (right) for each one of the Breakout variations. (a)-(d) demonstrate successes, while (e) and (f) show failure modes of the unaligned GAN. In (e) the ball in the input image is not generated in the output and in (f) not all bricks are generated, and some of the generated bricks appear smudged. some frames. On the opposite side, the Rectangle variation requires less training since the number of pixels changed in the image is small. During testing, we encountered problems with the images generation that we did not observe during the GAN training. The translation task we attempted to perform was supposedly simple -search for the differences between the domains shared by all images, change them to the way they are in the opposite domain and leave everything else the same. Unfortunately, since the network does not have any prior information about objects in the image, it struggles to generate them even if they were not changed. The most common problem we had was that the generator generated bricks that were supposed to be removed from the game, and in some cases, they were very noisy in the generated image (Fig. 3f). Another problem was the ball generation and more specifically, the location of the generated ball. Since the ball is small and changes its position often, it was hard for the generator, trained with unaligned pairs, to decide if and where to locate it in the generated image (Fig. 3e). These issues and others eventually caused the agent to fail in following the policies it learned on the source task. We found that more training leads to better for some of the variations and so the number of iterations needed was different for each variation. In Table 1 we show the of a test game played by the agent with and without GANs. We stop the training after reaching 300 points, which we consider to be a high score. As the show, the source game trained from scratch requires ten of millions of images to achieve such score comparing to the target task trained with GANs that only needs a few hundreds of thousands-a 100x fold increase in sample efficiency. Moreover, the frames the GAN was trained on were limited to the first games in which the A3C network was not trained, and yet it managed to generalize to more advanced stages of the game. While the Breakout variants work well to demonstrate transfer failure cases, they can be considered as "toy examples". We proceed to demonstrate the effectiveness of our transfer method on a "real" task: learning to transfer between different levels of the Road Fighter game. Road Fighter contains 4 different levels FIG2, each with a different where some are more difficult than others. The levels mostly differ visually and all have the same rules and require the same driving techniques. Thus, we believe that these techniques should sustain when playing a new level. We start by training an RL agent to play the first level of the game. To maximize the score, the agent has to acquire 3 main capabilities: driving fast, avoiding collision with obstacles, and if a car collision occurs reacting fast to avoid crashing. We use the A2C algorithm, the synchronous version of the Advantage Actor-Critic which performs better on this game than A3C, reaching over 10, 000 game points on level 1.For the GAN training, we collect 100k 84x84 frames from each of levels 2, 3 and 4 by running an untrained agent repeatedly until we have collected sufficient samples (training an RL agent to reach a score of 10,000 points on Level 1 required observing above 100M frames.). Using the collected images we train a mapping function for each task to map the new level (target task) to the first one (source task). We use the same GAN architecture used for Breakout, but initialize the weights with Orthogonal initialization. Compared to Breakout, these tasks introduce new challenges: rather than removing a mostly static element, the GAN has to be able to change the and road size while keeping the cars in the right position on the road. On the other hand, this setup may be closer to the one unaligned GANs are usually applied in. We restrict ourselves to collecting images from the beginning of the game, before the agent had any training. This restricts the phenomena the GAN can observe, and some target tasks' images do not have a clear corresponding situation in the first level, potentially causing unpredictable behaviors. For example, the generator matches the diagonal shaped roads to one of the first rare and unique images of level 1 (Fig. 5e).Our experiments presented in TAB1 demonstrate how an agent trained to master the first level of the game fails to follow the optimal policies on new levels, reaching 0 points. However, with the GAN-based visual analogies the agent is able to apply some of the abilities it gained when training on the first level, most notably driving fast, staying on the road, avoiding some cars, and, most importantly, recovering from car crashes. The ing agent achieves impressive scores on levels 2, 3 and 4 (5350, 5350 and 2050 points, respectively), with no additional RL training and while observing only a fraction of the frames required for training a corresponding RL agent from scratch for these levels. Limitations. While the GAN works well in generating objects it has seen before, such as the agent's car, it does have limitations on objects it has not seen. As a , it ends up generating differently colored cars all in red, or not generating them at all, as shown in Fig. 5a, 5d and 5f. Colorful cars can be "collected" by the agent and are worth 1000 points each. Generating them in red makes the agent avoid them, losing these extra points and achieving overall lower scores even if finishing the track. When cars are not fully generated, the agent is less likely to avoid them, and eventually crashes. Data Efficiency. We measure the number of frames of game-interaction needed for the analogytransfer method. We collect 100k frames, and then train the GAN for up to 500k iterations, evaluating it every 10, 000 iterations by running the game and observing the score, and pick the best scoring model. This amounts to 100k + 50 * F frames, where F = 3000 is roughly the average number of frames in a game. This amounts to about 250k frames of game interaction for each transfered level, Figure 5: Left: the original frame. Right: GAN generated. Upper row shows the success cases of the GAN while the lower row shows representative failures: in (d) and (f) the only object generated on the road is the player's car and in (e) the diagonal shaped road of level 2 in matched to the starting point of level 1.an order of magnitude fewer interaction frames than training an RL agent to achieve a comparable score. Discussion. The approach is successfully transferring to most tasks, achieving the best scores on levels 2 and 3. In general, as the game progresses, the more obstacles are presented making it harder to play. One of our best is achieved on level 2 where the road is identical to level 1's, reaching the best score after 320k GAN iterations. On level 3, the challenge increases as the road is very narrow, making it harder for a player to avoid crashing. However, the GAN manages to generate the road in the right shape in most frames and position the cars in the matching ratio. Moreover, despite the challenges, due to the agent's ability to avoid crashing when colliding with cars it gets over 5000 points after 450k GAN iterations. In level 4 we get the maximum after 270k iterations. This level is also the most challenging one to play, which might be the reason for the score being the lowest out of all tasks. The main obstacles are the sudden turns in the road causing the car to be very close to the sideways and the increasing dangers a player has to avoid. Theses difficulties make this level much harder than level 1 and might require more training even from a human player. Overall, the agent performs quite well by successfully applying 2 out of 3 main capabilities it gained during training. It is missing the third capability, avoiding collisions and collecting bonus cars, mainly because of bad generation. We believe that these tasks and demonstrate a success of the analogy transfer method for zero-short generalization across different levels of a video game. They also suggest a potential of performing well on additional real world tasks in which visual analogies can be made. Evaluating GAN models and comparing them to each other is a major challenge: working with images and without well-defined objectives, testing the quality of the generator is delegated to human judgment, often using crowdsourcing to evaluate the generated images BID13; BID5. This metric is not stable and can be unreliable due to changing human factors. Others use a linear classifier to evaluate the image representations the GAN learned on supervised datasets, e.g., MNIST and CIFAR-10 . These approaches and others may be good enough for a limited group of test images but do not necessarily reflect the performance of a model in the real world. We note that while unaligned GANs are known to achieve impressive with only few training examples, our seemingly trivial translation cases proved to be quite challenging for the unaligned GAN architectures: producing valid translations that work in the game require training on a substantial number of images. During training, the GAN sees images from the early stages of the game where only a few bricks are missing in Breakout and there are no obstacles in Road Fighter. Our task requires it to generalize by translating images where objects are removed, added or in a different locations. This requires levels of generalization that may not be reflected in existing test-sets. We argue that using simple, well defined yet diverse test cases, situated in the context of a concrete down-stream task like we do here, is an important step forward in the evaluation of GAN models. We propose to evaluate GAN by running a game in which the trained Deep RL network that learned policies based on images is now receiving images from the same domain generated with GAN, and equate a successful GAN model with one ing in a high game score. We use the approach to compare Cycle-GAN and UNIT-GAN BID19.We examine both methods by running the RL agent with each every 1000 GAN training iterations and considering the maximum score after 500k iterations. We present the in TAB2. The UNIT GAN performs better than Cycle-GAN in most Breakout tasks, while Cycle-GAN outperforms UNIT in most Road Fighter tasks while requiring fewer iterations. The main difference between the two methods is the weight-sharing constraint applied in UNIT, making the domains dependent on each other by sharing and updating the weights of one or several decoders and encoders layers. We hypothesize this constraint is an advantage in tasks where the representation of the images in the different domains are similar. Thus, in Breakout, where most pixels are identical in the source and target images, Cycle-GAN often fails where UNIT succeed. However, in tasks such as Road Fighter's where most pixels are different, the agent could benefit from architectures such as Cycle-GAN where the two domains are independent of each other. Transfer Learning (TL) is a machine learning technique used to improve the training speed of a target task with knowledge learned in a source task. Pretraining and fine-tuning was proposed in BID11 and applied to TL in BID0 and BID4. In this procedure, the approach is to train the base network and then copy its first n layers to the first n layers of a target network. One can choose to update the feature layers transferred to the new task with the error backpropagated from its output, or they can be left frozen, meaning that they do not change during training on the new task. Unfortunately, as we have shown, while fine-tuning might have the ability to accelerate the training process is some cases, it can also have a damaging impact on others. Generalization is a key element in training deep learning models with time or data size constraints. Recent discussions on overfitting in Deep RL algorithms BID30 encouraged better evaluation (e.g. OpenAI Retro Contest 1) and generalization methods. In Atari, there are many similarities between the goals and the mechanism of the games. For this reason, there have been many works attempting to transfer between games or between different variations of the same game, one approach trying to do both is the progressive networks BID24. A progressive network is constructed by successively training copies of A3C on each task of interest. In that work, they transferred between different games as well as from different variations of the game Pong. The drawback of this approach is the number of parameters growing quadratically with the number of tasks. However, even if this growth rate was improved, different tasks may require different adjustments and the predefinition of the number of layers and network representation is preventing it. Zero-shot generalization is a discussed and researched topic nowadays. One work transferring between modified versions of the same game using zero-shot transfer is the schema networks BID14. Like us, they also chose to demonstrate their method on the game Breakout, using Object Oriented Markov Decision Process. In contrast, we do not use the representation of the objects in the game, and we wish to preserve the accomplishments of DQN and transfer using only raw data. Other attempted to achieve robust policies using learned disentangled representation of the image BID10, analogies between sets of instructions BID22, interactive replay BID3 while training and learn general policies by training on multiple tasks in parallel BID6 BID26.Finally, the idea of using GANs for transfer learning and domain adaptation was explored for supervised image classification and robotics applications by several authors BID1 BID12 BID18 BID2. In these methods, there is supervised source domain data (x Our RL-based setup is different: first, our coverage of target-domain data is very limited (we can only observe states which are reachable by the un-adapted or untrained agent). Second, we do not have access to supervised gold labels on the source domain, but only to a learned policy network. Third, interactions with the game environment provide very indirect rewards, so using this reward signal to influence the GAN training will be very inefficient. We thus opt for a different strategy: rather than mapping the source to the target domain and training on the projected signal, which is unrealistic an costly in the RL setup, we instead take a pre-trained source model and train an unaligned GAN to map from the target domain back to the source domain, in order to re-use the source model's knowledge and apply it to the target domain data. We believe this form of usage of GAN for transfer learning is novel. We demonstrated the lack of generalization by looking at artificially constructed visual variants of a game (Breakout), and different levels of a game (Road Fighter). We further show that transfer learning by fine-tuning fails. The policies learned using model-free RL algorithms on the original game are not directly transferred to the modified games even when the changes are irrelevant to the game's dynamics. We present a new approach for transfer learning between related RL environments using GANs without the need for any additional training of the RL agent, and while requiring orders of magnitude less interactions with the environment. We further suggest this setup as a way to evaluate GAN architectures by observing their behavior on concrete tasks, revealing differences between the Cycle-GAN and UNIT-GAN architectures. We believe our approach is applicable to cases involving both direct and less direct mapping between environments, as long as an image-to-image translation exist. While we report a success in analogy transfer using Unaligned GANs, we also encountered limitations in the generation process that made it difficult for the agent to maximize the on the Road Fighter's tasks. In future work, we plan to explore a tighter integration between the analogy transfer method and the RL training process, to facilitate better performance where dynamic adjustments are needed in addition to the visual mapping.

We investigate a framework for discovery: curating a large collection of predictions, which are used to construct the agent’s representation in partially observable domains.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We isolate one factor of RL generalization by analyzing the case when the agent only overfits to the observations. We show that architectural implicit regularizations occur in this regime.