Question: represent the scientific article to find a short tldr

Question: Credit assignment in Meta-reinforcement learning (Meta-RL) is still poorly understood. Existing methods either neglect credit assignment to pre-adaptation behavior or implement it naively. This leads to poor sample-efficiency during meta-training as well as ineffective task identification strategies. This paper provides a theoretical analysis of credit assignment in gradient-based Meta-RL. Building on the gained insights we develop a novel meta-learning algorithm that overcomes both the issue of poor credit assignment and previous difficulties in estimating meta-policy gradients. By controlling the statistical distance of both pre-adaptation and adapted policies during meta-policy search, the proposed algorithm endows efficient and stable meta-learning. Our approach leads to superior pre-adaptation policy behavior and consistently outperforms previous Meta-RL algorithms in sample-efficiency, wall-clock time, and asymptotic performance. A remarkable trait of human intelligence is the ability to adapt to new situations in the face of limited experience. In contrast, our most successful artificial agents struggle in such scenarios. While achieving impressive , they suffer from high sample complexity in learning even a single task, fail to generalize to new situations, and require large amounts of additional data to successfully adapt to new environments. Meta-learning addresses these shortcomings by learning how to learn. Its objective is to learn an algorithm that allows the artificial agent to succeed in an unseen task when only limited experience is available, aiming to achieve the same fast adaptation that humans possess .Despite recent progress, deep reinforcement learning (RL) still relies heavily on hand-crafted features and reward functions as well as engineered problem specific inductive bias. Meta-RL aims to forego such reliance by acquiring inductive bias in a data-driven manner. Recent work proves this approach to be promising, demonstrating that Meta-RL allows agents to obtain a diverse set of skills, attain better exploration strategies, and learn faster through meta-learned dynamics models or synthetic returns BID8; BID14 ).Meta-RL is a multi-stage process in which the agent, after a few sampled environment interactions, adapts its behavior to the given task. Despite its wide utilization, little work has been done to promote theoretical understanding of this process, leaving Meta-RL grounded on unstable foundations. Although the behavior prior to the adaptation step is instrumental for task identification, the interplay between pre-adaptation sampling and posterior performance of the policy remains poorly understood. In fact, prior work in gradient-based Meta-RL has either entirely neglected credit assignment to the pre-update distribution BID9 or implemented such credit assignment in a naive way BID10 ).To our knowledge, we provide the first formal in-depth analysis of credit assignment w.r.t. preadaptation sampling distribution in Meta-RL. Based on our findings, we develop a novel Meta-RL algorithm. First, we analyze two distinct methods for assigning credit to pre-adaptation behavior.of MAML, was first introduced by BID9. We refer to it as formulation I which can be expressed as maximizing the objective DISPLAYFORM0 In that U denotes the update function which depends on the task T, and performs one VPG step towards maximizing the performance of the policy in T. For national brevity and conciseness we assume a single policy gradient adaptation step. Nonetheless, all presented concepts can easily be extended to multiple adaptation steps. Later work proposes a slightly different notion of gradient-based Meta-RL, also known as E-MAML, that attempts to circumvent issues with the meta-gradient estimation in MAML BID10 ): DISPLAYFORM1 R(τ) with θ:= U (θ, τ 1: DISPLAYFORM2 Formulation II views U as a deterministic function that depends on N sampled trajectories from a specific task. In contrast to formulation I, the expectation over pre-update trajectories τ is applied outside of the update function. Throughout this paper we refer to π θ as pre-update policy, and π θ as post-update policy. This section analyzes the two gradient-based Meta-RL formulations introduced in Section 3. The red arrows depict how credit assignment w.r.t the pre-update sampling distribution P T (τ |θ) is propagated. Formulation I (left) propagates the credit assignment through the update step, thereby exploiting the full problem structure. In contrast, formulation II (right) neglects the inherent structure, directly assigning credit from post-update return R to the pre-update policy π θ which leads to noisier, less effective credit assignment. Both formulations optimize for the same objective, and are equivalent at the 0 th order. However, because of the difference in their formulation and stochastic computation graph, their gradients and the ing optimization step differs. In the following, we shed light on how and where formulation II loses signal by analyzing the gradients of both formulations, which can be written as (see Appendix A for more details and derivations) ∇ θ J post (τ, τ) simply corresponds to a policy gradient step on the post-update policy π θ w.r.t θ, followed by a linear transformation from post-to pre-update parameters. It corresponds to increasing the likelihood of the trajectories τ that led to higher returns. However, this term does not optimize for the pre-update sampling distribution, i.e., which trajectories τ led to better adaptation steps. The credit assignment w.r.t. the pre-updated sampling distribution is carried out by the second term. In formulation II, ∇ θ J II pre can be viewed as standard reinforcement learning on π θ with R(τ) as reward signal, treating the update function U as part of the unknown dynamics of the system. This shifts the pre-update sampling distribution to better adaptation steps. Formulation I takes the causal dependence of P T (τ |θ) on P T (τ |θ) into account. It does so by maximizing the inner product of pre-update and post-update policy gradients (see Eq. 4). This steers the pre-update policy towards 1) larger post-updates returns 2) larger adaptation steps α∇ θ J inner, 3) better alignment of pre-and post-update policy gradients . When combined, these effects directly optimize for adaptation. As a , we expect the first meta-policy gradient formulation, J I, to yield superior learning properties. In the previous section we show that the formulation introduced by BID9 in superior meta-gradient updates, which should in principle lead to improved convergence properties. However, obtaining correct and low variance estimates of the respective meta-gradients proves challenging. As discussed by BID10, and shown in Appendix B.3, the score function surrogate objective approach is ill suited for calculating higher order derivatives via automatic differentiation toolboxes. This important fact was overlooked in the original RL-MAML implementation BID9 leading to incorrect meta-gradient estimates 1. As a , ∇ θ J pre does not appear in the gradients of the meta-objective (i.e. ∇ θ J = ∇ θ J post). Hence, MAML does not perform any credit assignment to pre-adaptation behavior. But, even when properly implemented, we show that the meta-gradients exhibit high variance. Specifically, the estimation of the hessian of the RL-objective, which is inherent in the metagradients, requires special consideration. In this section, we motivate and introduce the low variance curvature estimator (LVC): an improved estimator for the hessian of the RL-objective which promotes better meta-policy gradient updates. As we show in Appendix A.1, we can write the gradient of the meta-learning objective as DISPLAYFORM0 H−1 t =t r(s t, a t)|s t, a t denotes the expected state-action value function under policy π θ at time t. Computing the expectation of the RL-objective is in general intractable. Typically, its gradients are computed with a Monte Carlo estimate based on the policy gradient theorem (Eq. 82). In practical implementations, such an estimate is obtained by automatically differentiating a surrogate objective (b). However, this in a highly biased hessian estimate which just computes H 2, entirely dropping the terms H 1 and H 12 +H 12. In the notation of the previous section, it leads to neglecting the ∇ θ J pre term, ignoring the influence of the pre-update sampling distribution. The issue can be overcome using the DiCE formulation, which allows to compute unbiased higherorder Monte Carlos estimates of arbitrary stochastic computation graphs BID10. The DiCE-RL objective can be rewritten as follows DISPLAYFORM1 DISPLAYFORM2 In that, ⊥ denotes the "stop gradient" operator, i.e., DISPLAYFORM3 The sequential dependence of π θ (a t |s t) within the trajectory, manifesting itself through the product of importance weights in FORMULA4, in high variance estimates of the hessian DISPLAYFORM4 As noted by BID12, H 12 is particularly difficult to estimate, since it involves three nested sums along the trajectory. In section 7.2 we empirically show that the high variance estimates of the DiCE objective lead to noisy meta-policy gradients and poor learning performance. To facilitate a sample efficient meta-learning, we introduce the low variance curvature (LVC) estimator: DISPLAYFORM5 By removing the sequential dependence of π θ (a t |s t) within trajectories, the hessian estimate neglects the term H 12 + H 12 which leads to a variance reduction, but makes the estimate biased. The choice of this objective function is motivated by findings in BID12: under certain conditions the term H 12 + H 12 vanishes around local optima θ DISPLAYFORM6 Hence, the bias of the LVC estimator becomes negligible close to local optima. The experiments in section 7.2 underpin the theoretical findings, showing that the low variance hessian estimates obtained through J LVC improve the sample-efficiency of meta-learning by a significant margin when compared to J DiCE. We refer the interested reader to Appendix B for derivations and a more detailed discussion.6 PROMP: PROXIMAL META-POLICY SEARCH Building on the previous sections, we develop a novel meta-policy search method based on the low variance curvature objective which aims to solve the following optimization problem: DISPLAYFORM7 Prior work has optimized this objective using either vanilla policy gradient (VPG) or TRPO (a). TRPO holds the promise to be more data efficient and stable during the learning process when compared to VPG. However, it requires computing the Fisher information matrix (FIM). Estimating the FIM is particularly problematic in the meta-learning set up. The meta-policy gradients already involve second order derivatives; as a , the time complexity of the FIM estimate is cubic in the number of policy parameters. Typically, the problem is circumvented using finite difference methods, which introduce further approximation errors. The recently introduced PPO algorithm achieves comparable to TRPO with the advantage of being a first order method. PPO uses a surrogate clipping objective which allows it to safely take multiple gradient steps without re-sampling trajectories. for step n = 0,..., N − 1 do DISPLAYFORM8 if n = 0 then 6: DISPLAYFORM0 for all DISPLAYFORM1 Sample pre-update trajectories D i = {τ i} from T i using π θ 9:Compute adapted parameters DISPLAYFORM2 Sample post-update trajectories DISPLAYFORM3 11: DISPLAYFORM4 In case of Meta-RL, it does not suffice to just replace the post-update reward objective with J CLIP T. In order to safely perform multiple meta-gradient steps based on the same sampled data from a recent policy π θo, we also need to 1) account for changes in the pre-update action distribution π θ (a t |s t), and 2) bound changes in the pre-update state visitation distribution .We propose Proximal Meta-Policy Search (ProMP) which incorporates both the benefits of proximal policy optimization and the low variance curvature objective (see Alg. 1.) In order to comply with requirement 1), ProMP replaces the "stop gradient" importance weight DISPLAYFORM5, which in the following objective DISPLAYFORM6 An important feature of this objective is that its derivatives w.r.t θ evaluated at θ o are identical to those of the LVC objective, and it additionally accounts for changes in the pre-update action distribution. To satisfy condition 2) we extend the clipped meta-objective with a KL-penalty term between π θ and π θo. This KL-penalty term enforces a soft local "trust region" around π θo, preventing the shift in state visitation distribution to become large during optimization. This enables us to take multiple meta-policy gradient steps without re-sampling. Altogether, ProMP optimizes DISPLAYFORM7 ProMP consolidates the insights developed throughout the course of this paper, while at the same time making maximal use of recently developed policy gradients algorithms. First, its meta-learning formulation exploits the full structural knowledge of gradient-based meta-learning. Second, it incorporates a low variance estimate of the RL-objective hessian. Third, ProMP controls the statistical distance of both pre-and post-adaptation policies, promoting efficient and stable meta-learning. All in all, ProMP consistently outperforms previous gradient-based meta-RL algorithms in sample complexity, wall clock time, and asymptotic performance (see Section 7.1). In order to empirically validate the theoretical arguments outlined above, this section provides a detailed experimental analysis that aims to answer the following questions: (i) How does ProMP perform against previous Meta-RL algorithms? (ii) How do the lower variance but biased LVC gradient estimates compare to the high variance, unbiased DiCE estimates? (iii) Do the different formulations in different pre-update exploration properties? (iv) How do formulation I and formulation II differ in their meta-gradient estimates and convergence properties?To answer the posed questions, we evaluate our approach on six continuous control Meta-RL benchmark environments based on OpenAI Gym and the Mujoco simulator BID5 ). A description of the experimental setup is found in Appendix D. In all experiments, the reported curves are averaged over at least three random seeds. Returns are estimated based on sampled trajectories from the adapted post-update policies and averaged over sampled tasks. The source code and the experiment data are available on our supplementary website. We compare our method, ProMP, in sample complexity and asymptotic performance to the gradientbased meta-learning approaches MAML-TRPO BID9 ) and E-MAML-TRPO (see FIG2). Note that MAML corresponds to the original implementation of RL-MAML by BID9 where no credit assignment to the pre-adaptation policy is happening (see Appendix B.3 for details). Moreover, we provide a second study which focuses on the underlying meta-gradient estimator. Specifically, we compare the LVC, DiCE, MAML and E-MAML estimators while optimizing meta-learning objective with vanilla policy gradient (VPG) ascent. This can be viewed as an ablated version of the algorithms which tries to eliminate the influences of the outer optimizers on the learning performance (see Fig. 3).These algorithms are benchmarked on six different locomotion tasks that require adaptation: the half-cheetah and walker must switch between running forward and backward, the high-dimensional agents ant and humanoid must learn to adapt to run in different directions in the 2D-plane, and the hopper and walker have to adapt to different configuration of their dynamics. The in FIG2 highlight the strength of ProMP in terms of sample efficiency and asymptotic performance. In the meta-gradient estimator study in Fig. 3, we demonstrate the positive effect of the LVC objective, as it consistently outperforms the other estimators. In contrast, DiCE learns only slowly when compared to the other approaches. As we have motivated mathematically and substantiate empirically in the following experiment, the poor performance of DiCE may be ascribed to the high variance of its meta-gradient estimates. The fact that the of MAML and E-MAML are comparable underpins the ineffectiveness of the naive pre-update credit assignment (i.e. formulation II), as discussed in section 4.Results for four additional environments are displayed in Appendix D along with hyperparameter settings, environment specifications and a wall-clock time comparison of the algorithms. In Section 5 we discussed how the DiCE formulation yields unbiased but high variance estimates of the RL-objective hessian and served as motivation for the low variance curvature (LVC) estimator. Here we investigate the meta-gradient variance of both estimators as well as its implication on the learning performance. Specifically, we report the relative standard deviation of the metapolicy gradients as well as the average return throughout the learning process in three of the metaenvironments. The , depicted in Figure 4, highlight the advantage of the low variance curvature estimate. The trajectory level dependencies inherent in the DiCE estimator leads to a meta-gradient standard deviation that is on average 60% higher when compared to LVC. As the learning curves indicate, the noisy gradients may be a driving factor for the poor performance of DiCE, impeding sample efficient meta-learning. Meta-policy search based on the LVC estimator leads to substantially better sample-efficiency and asymptotic performance. In case of HalfCheetahFwdBack, we observe some unstable learning behavior of LVC-VPG which is most likely caused by the bias of LVC in combination with the naive VPG optimizer. However, the mechanisms in ProMP that ensure proximity w.r.t. to the policys KL-divergence seem to counteract these instabilities during training, giving us a stable and efficient meta-learning algorithm. Here we evaluate the effect of the different objectives on the learned pre-update sampling distribution. We compare the low variance curvature (LVC) estimator with TRPO (LVC-TRPO) against MAML BID9 ) and E-MAML-TRPO in a 2D environment on which the exploration behavior can be visualized. Each task of this environment corresponds to reaching a different corner location; however, the 2D agent only experiences reward when it is sufficiently close to the corner (translucent regions of FIG5). Thus, to successfully identify the task, the agent must explore the different regions. We perform three inner adaptation steps on each task, allowing the agent to fully change its behavior from exploration to exploitation. functions. Through its superior credit assignment, the LVC objective learns a pre-update policy that is able to identify the current task and respectively adapt its policy, successfully reaching the goal (dark green circle).The different exploration-exploitation strategies are displayed in FIG5. Since the MAML implementation does not assign credit to the pre-update sampling trajectory, it is unable to learn a sound exploration strategy for task identification and thus fails to accomplish the task. On the other hand, E-MAML, which corresponds to formulation II, learns to explore in long but random paths: because it can only assign credit to batches of pre-update trajectories, there is no notion of which actions in particular facilitate good task adaptation. As consequence the adapted policy slightly misses the task-specific target. The LVC estimator, instead, learns a consistent pattern of exploration, visiting each of the four regions, which it harnesses to fully solve the task. To shed more light on the differences of the gradients of formulation I and formulation II, we evaluate the meta-gradient updates and the corresponding convergence to the optimum of both formulations in a simple 1D environment. In this environment, the agent starts in a random position in the real line and has to reach a goal located at the position 1 or -1. In order to visualize the convergence, we parameterize the policy with only two parameters θ 0 and θ 1. We employ formulation I by optimizing the DiCE objective with VPG, and formulation II by optimizing its (E-MAML) objective with VPG. Figure 6 depicts meta-gradient updates of the parameters θ i for both formulations. Formulation I (red) exploits the internal structure of the adaptation update yielding faster and steadier convergence to the optimum. Due to its inferior credit assignment, formulation II (green) produces noisier gradient estimates leading to worse convergence properties. In this paper we propose a novel Meta-RL algorithm, proximal meta-policy search (ProMP), which fully optimizes for the pre-update sampling distribution leading to effective task identification. Our method is the of a theoretical analysis of gradient-based Meta-RL formulations, based on which we develop the low variance curvature (LVC) surrogate objective that produces low variance meta-policy gradient estimates. Experimental demonstrate that our approach surpasses previous meta-reinforcement learning approaches in a diverse set of continuous control tasks. Finally, we underpin our theoretical contributions with illustrative examples which further justify the soundness and effectiveness of our method. In this section we discuss two different gradient-based meta-learning formulations, derive their gradients and analyze the differences between them. The first meta-learning formulation, known as MAML BID9, views the inner update rule U (θ, T) as a mapping from the pre-update parameter θ and the task T to an adapted policy parameter θ. The update function can be viewed as stand-alone procedure that encapsulates sampling from the task-specific trajectory distribution P T (τ |π θ) and updating the policy parameters. Building on this concept, the meta-objective can be written as DISPLAYFORM0 The task-specific gradients follow as DISPLAYFORM1 DISPLAYFORM2 DISPLAYFORM3 In order to derive the gradients of the inner update ∇ θ θ = ∇ θ U (θ, T) it is necessary to know the structure of U. The main part of this paper assumes the inner update rule to be a policy gradient descent step DISPLAYFORM4 DISPLAYFORM5 Thereby the second term in is the local curvature (hessian) of the inner adaptation objective function. The correct hessian of the inner objective can be derived as follows: DISPLAYFORM6 The second meta-reinforcement learning formulation views the the inner update θ = U (θ, τ 1:N) as a deterministic function of the pre-update policy parameters θ and N trajectories τ 1:N ∼ P T (τ 1:N |θ) sampled from the pre-update trajectory distribution. This formulation was introduced in BID10 and further discussed with respect to its exploration properties in.Viewing U as a function that adapts the policy parameters θ to a specific task T given policy rollouts in this task, the corresponding meta-learning objective can be written as DISPLAYFORM0 Since the first part of the gradient derivation is agnostic to the inner update rule U (θ, τ 1:N), we only assume that the inner update function U is differentiable w.r.t. θ. First we rewrite the meta-objective J(θ) as expectation of task specific objectives J II T (θ) under the task distribution. This allows us to express the meta-policy gradients as expectation of task-specific gradients: DISPLAYFORM1 The task specific gradients can be calculated as follows DISPLAYFORM2 As in A.1 the structure of U (θ, τ 1:N) must be known in order to derive the gradient ∇ θ θ. Since we assume the inner update to be vanilla policy gradient, the respective gradient follows as DISPLAYFORM3 The respective gradient of U (θ, τ 1:N) follows as DISPLAYFORM4 DISPLAYFORM5 In the following we analyze the differences between the gradients derived for the two formulations. To do so, we begin with ∇ θ J I T (θ) by inserting the gradient of the inner adaptation step into FORMULA4: DISPLAYFORM0 We can substitute the hessian of the inner objective by its derived expression from FORMULA26 and then rearrange the terms. Also note that ∇ θ log P T (τ |θ) = ∇ θ log π θ (τ) = H−1 t=1 log π θ (a t |s t) where H is the MDP horizon. DISPLAYFORM1 Next, we rearrange the gradient of J II into a similar form as ∇ θ J I T (θ). For that, we start by inserting for ∇ θ θ and replacing the expectation over pre-update trajectories τ 1:N by the expectation over a single trajectory τ. DISPLAYFORM2 While the first part of the gradients match ( and ), the second part ( and FORMULA35 ) differs. Since the second gradient term can be viewed as responsible for shifting the pre-update sampling distribution P T (τ |θ) towards higher post-update returns, we refer to it as ∇ θ J pre (τ, τ). To further analyze the difference between ∇ θ J I pre and ∇ θ J II pre we slightly rearrange and put both gradient terms next to each other: DISPLAYFORM3 In the following we interpret and and compare of the derived gradient terms, aiming to provide intuition for the differences between the formulations:The first gradient term J post that matches in both formulations corresponds to a policy gradient step on the post-update policy π θ. Since θ itself is a function of θ, the term I + αR(τ)∇ 2 θ log π θ (τ)) can be seen as linear transformation of the policy gradient update R(τ)∇ θ log π θ (τ) from the post-update parameter θ into θ. Although J post takes into account the functional relationship between θ and θ, it does not take into account the pre-update sampling distribution P T (τ |θ).This is where ∇ θ J pre comes into play: ∇ θ J I pre can be viewed as policy gradient update of the preupdate policy π θ w.r.t. to the post-update return R(τ). Hence this gradient term aims to shift the pre-update sampling distribution so that higher post-update returns are achieved. However, ∇ θ J II pre does not take into account the causal dependence of the post-update policy on the pre-update policy. Thus a change in θ due to ∇ θ J II pre may counteract the change due to ∇ θ J II post. In contrast, ∇ θ J I pre takes the dependence of the the post-update policy on the pre-update sampling distribution into account. Instead of simply weighting the gradients of the pre-update policy ∇ θ log π θ (τ) with R(τ) as in ∇ θ J I post, ∇ θ J I post weights the gradients with inner product of the pre-update and post-update policy gradients. This inner product can be written as DISPLAYFORM4 wherein δ denotes the angle between the the inner and outer pre-update and post-update policy gradients. Hence, ∇ θ J I post steers the pre-update policy towards not only towards larger post-updates returns but also towards larger adaptation steps α∇ θ J inner, and better alignment of pre-and postupdate policy gradients. This directly optimizes for maximal improvement / adaptation for the respective task.; for a comparable analysis in case of domain generalization and supervised meta-learning. Also note that allows formulation I to perform credit assignment on the trajectory level whereas formulation II can only assign credit to entire batches of N pre-update trajectories τ 1:N.As a , we expect the first meta-policy gradient formulation to learn faster and more stably since the respective gradients take the dependence of the pre-update returns on the pre-update sampling distribution into account while this causal link is neglected in the second formulation. When employing formulation I for gradient-based meta-learning, we aim maximize the loss DISPLAYFORM0 by performing a form of gradient-descent on J(θ). Note that we, from now on, assume J:= J I and thus omit the superscript indicating the respective meta-learning formulation. As shown in A.2 the gradient can be derived as DISPLAYFORM1 where DISPLAYFORM2 ] denotes hessian of the inner adaptation objective w.r.t. θ. This section concerns the question of how to properly estimate this hessian. Since the expectation over the trajectory distribution P T (τ |θ) is in general intractable, the score function trick is typically used to used to produce a Monte Carlo estimate of the policy gradients. Although the gradient estimate can be directly defined, when using a automatic-differentiation toolbox it is usually more convenient to use an objective function whose gradients correspond to the policy gradient estimate. Due to the Policy Gradient Theorem (PGT) such a "surrogate" objective can be written as: DISPLAYFORM0 While and FORMULA41 are equivalent , the more popular formulation formulation can be seen as forward looking credit assignment while can be interpreted as backward looking credit assignment BID10. A generalized procedure for constructing "surrogate" objectives for arbitrary stochastic computation graphs can be found in Schulman et al. (2015a). Estimating the the hessian of the reinforcement learning objective has been discussed in BID12 and BID4 with focus on second order policy gradient methods. In the infinite horizon MDP case, BID4 derive a decomposition of the hessian. In the following, we extend their finding to the finite horizon case. Proof. As derived in, the hessian of J inner (θ) follows as: DISPLAYFORM0 DISPLAYFORM1 The term in is equal to H 2. We continue by showing that the remaining term in is equivalent to H 1 + H 12 + H 12. For that, we split the inner double sum in into three components: DISPLAYFORM2 DISPLAYFORM3 By changing the backward looking summation over outer products into a forward looking summation of rewards, can be shown to be equal to H 1: DISPLAYFORM4 DISPLAYFORM5 By simply exchanging the summation indices t and h in FORMULA45 it is straightforward to show that FORMULA45 is the transpose of. Hence it is sufficient to show that is equivalent to H 12. However, instead of following the direction of the previous proof we will now start with the definition of H 12 and derive the expression in. DISPLAYFORM6 The gradient of Q π θ t can be expressed recursively: DISPLAYFORM7 By induction, it follows that DISPLAYFORM8 When inserting FORMULA51 into FORMULA48 and swapping the summation, we are able to show that H 12 is equivalent to. DISPLAYFORM9 This concludes the proof that the hessian of the expected sum of rewards under policy π θ and an MDP with finite time horizon H can be decomposed into H 1 + H 2 + H 12 + H 12. As pointed out by BID10 and BID10, simply differentiating through the gradient of surrogate objective J PGT as done in the original MAML version BID9 leads to biased hessian estimates. Specifically, when compared with the unbiased estimate, as derived in FORMULA26 and decomposed in Appendix B.2, both H 1 and H 12 + H 12 are missing. Thus, ∇ θ J pre does not appear in the gradients of the meta-objective (i.e. ∇ θ J = ∇ θ J post). Only performing gradient descent with ∇ θ J post entirely neglects influences of the pre-update sampling distribution. This issue was overseen in the RL-MAML implementation of BID9. As discussed in this leads to poor performance in meta-learning problems that require exploration during the pre-update sampling. Addressing the issue of incorrect higher-order derivatives of monte-carlo estimators, BID10 propose DICE which mainly builds upon an newly introduced MagicBox operator. This operator allows to formulate monte-carlo estimators with correct higher-order derivatives. A DICE formulation of a policy gradient estimator reads as: DISPLAYFORM0 DISPLAYFORM1 In that, ⊥ denotes a "stop gradient" operator (i.e. DISPLAYFORM2 Note that → denotes a "evaluates to" and does not necessarily imply equality w.r.t. to gradients. Hence, J DICE (θ) evaluates to the sum of rewards at 0th order but produces the unbiased gradients ∇ n θ J DICE (θ) when differentiated n-times (see BID10 for proof). To shed more light on the maverick DICE formulation, we rewrite as follows: DISPLAYFORM3 Interpreting this novel formulation, the MagicBox operator θ ({a t ≤t}) can be understood as "dry" importance sampling weight. At 0th order it evaluates to 1 and leaves the objective function unaffected, but when differentiated once it yields an estimator for the marginal rate of return due to a change in the policy-implied trajectory distribution. In addition to the expected reward J(π) under policy π, we will use the state value function V π, the state-action value function Q π as well as the advantage function A π: DISPLAYFORM4 with a t ∼ π(a t |s t) and s t+1 ∼ p(s t+1 |s t, a t).The expected return under a policyπ can be expressed as the sum of the expected return of another policy π and the expected discounted advantage ofπ over π (see Schulman et al. (2015a) for proof). DISPLAYFORM5 Let d π denote the discounted state visitation frequency: DISPLAYFORM6 We can use d π to express the expectation over trajectories τ ∼ p π (τ) in terms of states and actions: DISPLAYFORM7 Local policy search aims to find a policy update π →π in the proximity of π so that J(π) is maximized. Since J(π) is not affected by the policy update π →π, it is sufficient to maximize the expected advantage underπ. However, the complex dependence of dπ(s) onπ makes it hard to directly maximize the objective in. Using a local approximation of where it is assumed that the state visitation frequencies d π and dπ are identical, the optimization can be phrased as DISPLAYFORM8 In the following we refer toJ(π) as surrogate objective. It can be shown that the surrogate objectivẽ J matches J to first order when π =π (see). If π θ is a parametric and differentiable function with parameter vector θ, this means that for any θ o: DISPLAYFORM9 When π =π, an approximation error of the surrogate objectiveJ w.r.t. to the true objective J is introduced. BID0 derive a lower bound for the true expected return ofπ: DISPLAYFORM10 DISPLAYFORM11 Trust region policy optimization (TPRO) (a) attempts to approximate the bound in by phrasing local policy search as a constrained optimization problem: DISPLAYFORM12 Thereby the KL-constraint δ induces a local trust region around the current policy π θo. A practical implementation of TPRO uses a quadratic approximation of the KL-constraint which leads to the following update rule: DISPLAYFORM13 with g:= ∇ θ E s∼dπ θo (s) DISPLAYFORM14 π θo (a|s) A π θo (s, a) being the gradient of the objective and F = ∇ 2 θD KL [π θo ||π θ] the Fisher information matrix of the current policy π θo. In order to avoid the cubic time complexity that arise when inverting F, the Conjugate Gradient (CG) algorithm is typically used to approximate the Hessian vector product F −1 g. While TPRO is framed as constrained optimization, the theory discussed in Appendix C.1 suggest to optimize the lower bound. Based on this insight, propose adding a KL penalty to the objective and solve the following unconstrained optimization problem: DISPLAYFORM0 However, they also show that it is not sufficient to set a fixed penalty coefficient β and propose two alternative methods, known as Proximal Policy Optimization (PPO) that aim towards alleviating this issue:1) Adapting the KL coefficient β so that a desired target KL-divergenceD KL [π θo ||π θ] between the policy before and after the parameter update is achieved 2) Clipping the likelihood ratio so that the optimization has no incentive to move the policy π θ too far away from the original policy π θo. A corresponding optimization objective reads as: DISPLAYFORM1 Empirical show that the latter approach leads to better learning performance .Since PPO objective keeps π θ in proximity of π θo, it allows to perform multiple gradient steps without re-sampling trajectories from the updated policy. This property substantially improves the data-efficiency of PPO over vanilla policy gradient methods which need to re-estimate the gradients after each step. The optimal hyperparameter for each algorithm was determined using parameter sweeps. Table 1 contains the hyperparameter settings used for the different algorithms. Any environment specific modifications are noted in the respective paragraph describing the environment. PointEnv (used in the experiment in 7.3)• Trajectory Length: 100• Num Adapt Steps: 3 In this environment, each task corresponds to one corner of the area. The point mass must reach the goal by applying directional forces. The agent only experiences a reward when within a certain radius of the goal, and the magnitude of the reward is equal to the distance to the goal. HalfCheetahFwdBack, AntFwdBack, WalkerFwdBack, HumanoidFwdBack• Trajectory Length: 100 (HalfCheetah, Ant); 200 (Humanoid, Walker)• Num Adapt Steps: 1The task is chosen between two directions -forward and backward. Each agent must run along the goal direction as far as possible, with reward equal to average velocity minus control costs. • Trajectory Length: 100 (Ant); 200 (Humanoid)• Num Adapt Steps: 1Each task corresponds to a random direction in the XY plane. As above, each agent must learn to run in that direction as far as possible, with reward equal to average velocity minus control costs. • Trajectory Length: 200• Num Adapt Steps: 2In this environment, each task is a location randomly chosen from a circle in the XY plane. The goal is not given to the agent -it must learn to locate, approach, and stop at the target. The agent receives a penalty equal to the distance from the goal. • Trajectory Length: 200• Inner LR: 0.05• Num Adapt Steps: 1The agent must move forward as quickly as it can. Each task is a different randomization of the simulation parameters, including friction, joint mass, and inertia. The agent receives a reward equal to its velocity. Published as a conference paper at ICLR 2019 In addition to the six environments displayed in 2, we ran experiments on the other four continuous control environments described above. The are displayed in 7. In addition to the improved sample complexity and better asymptotic performance, another advantage of ProMP is its computation time. FIG8 shows the average time spent per iteration throughout the learning process in the humanoid environment differences of ProMP, LVC-VPG, and MAML-TRPO. Due to the expensive conjugate gradient steps used in TRPO, MAML takes far longer than either first order method. Since ProMP takes multiple stochastic gradient descent steps per iteration, it leads to longer outer update times compared to VPG, but in both cases the update time is a fraction of the time spent sampling from the environment. The difference in sampling time is due to the reset process: resetting the environment when the agent "dies" is an expensive operation. ProMP acquires better performance quicker, and as a the agent experiences longer trajectories and the environment is reset less often. In our setup, instances of the environment are run in parallel and performing a reset blocks all environments.

represent the short summary for finding the corresponding scientific article (long)

A novel and theoretically grounded meta-reinforcement learning algorithm

Represent the short summary for finding the corresponding scientific article (long)!


Question: Represent the scientific article to find a short TLDR.

Question: When training a neural network for a desired task, one may prefer to adapt a pretrained network rather than start with a randomly initialized one -- due to lacking enough training data, performing lifelong learning where the system has to learn a new task while being previously trained for other tasks, or wishing to encode priors in the network via preset weights. The most commonly employed approaches for network adaptation are fine-tuning and using the pre-trained network as a fixed feature extractor, among others. In this paper we propose a straightforward alternative: Side-Tuning. Side-tuning adapts a pretrained network by training a lightweight "side" network that is fused with the (unchanged) pre-rained network using a simple additive process. This simple method works as well as or better than existing solutions while it resolves some of the basic issues with fine-tuning, fixed features, and several other common baselines. In particular, side-tuning is less prone to overfitting when little training data is available, yields better than using a fixed feature extractor, and doesn't suffer from catastrophic forgetting in lifelong learning. We demonstrate the performance of side-tuning under a diverse set of scenarios, including lifelong learning (iCIFAR, Taskonomy), reinforcement learning, imitation learning (visual navigation in Habitat), NLP question-answering (SQuAD v2), and single-task transfer learning (Taskonomy), with consistently promising . The goal of side-tuning is to capitalize on a pretrained model to better learn one or more novel tasks. By design, side-tuning does so without degrading performance of the base model. The framework is straightforward: it assumes access to the frozen base model B: X → Y that maps inputs into some representation space that is shared between the base task and the current (target) task. This representation space is flexible and could either be a latent space (e.g. in R N) or actual model predictions. Side-tuning then learns a side model S: X → Y, so that the representations for the target task are R(x) B(x) ⊕ S(x), fine-tuning adapts too easily and forgets old information. Side-tuning is a simple method to address these limitations. for some combining operation ⊕. We use a learned alpha-blending, a ⊕ b αa + (1 − α)b for this purpose (other options are discussed in Section 3.0.3). Certain pre-set curricula of α reduce the side-tuning framework to: fine-tuning, feature extration, and stage-wise training (see Fig. 3, right). Hence those can be viewed as special cases of the general side-tuning framework. Also, other curricula suggest (e.g.) a maximum a posteriori estimator that integrates the B(x) prior with the evidence from S(x). Side-tuning is an example of an additive learning approach as it adds (strategically placed) parameters for each new task. Fixed feature extraction would be a simple example of an additive approach with zero new parameters. As a , fixed features are don't adapt the base network over the lifetime of the agent. A number of existing approaches address this by learning new parameters (the number of which scales with the size of the base network) for each new task (e.g. . Unlike these approaches, side-tuning places no constraints on the structure of the side network, allowing the parameters to be strategically allocated. In particular, side-tuning can use tiny networks when the base requires only minor updates. By adding fewer parameters per task, side-tuning can learn more tasks before the model grows large enough to require parameter consolidation. These approaches stand in contrast to most existing methods for incremental learning, which do not increase the number of parameters over time and instead gradually fill up the capacity of a large base model. For example, fine-tuning updates all the parameters. A large body of constraint-based methods focus on how to regularize these updates in order to prevent inter-task interference . Side-tuning does not require such regularization since the additive structure means inter-task interference is not possible. We compare side-tuning to alternative approaches on both the iCIFAR and Taskonomy datasets. iCIFAR consists of ten distinct 10-class image classification problems. Taskonomy covers multiple tasks of varied complexity from across computer vision (surface normal and depth estimation, edge detection, image 1000-way classification, etc.). On these datasets, side-tuning uses side networks that are much smaller than the base. Consequently, even without consolidation, side-tuning uses fewer learnable parameters than the alternative methods. This remarkably simple approach deals with the key challenges of incremental learning. Namely, it does not suffer from either: • Catastrophic forgetting: which is the tendency of a network to abruptly lose previously learned knowledge upon learning new information. We show this in Section 4.2.1. • Rigidity: where networks become increasingly unable to adapt to new problems as they accrue constraints from previous problems. We explore this in Section 4.2.2. Side-tuning avoids these problems while remaining highly performant, which we demonstrate in Section 4.2.3. Broadly speaking, network adaptation methods either overwrite existing knowledge (substitutive methods) or save it and add new parameters (additive learning). In incremental (lifelong) learning, substitutive methods like fine-tuning are at risk of forgetting early tasks. To prevent forgetting, existing methods add non-interference constraints that eventually slow down learning or they force tasks to be independent which prevents reusing knowledge. Side-tuning is an additive approach that performs well and scales well, and, by design, does not suffer from the aforementioned problems. We show this experimentally on various tasks and datasets, including iCIFAR (b), Habitat , SQuAD v2 , and Taskonomy . In the remainder of this section we overview sidetuning's connection to related fields. Network Adaptation modifies an existing network to solve a single new task. The most common approach is to update some or all of the network weights (fine-tuning), possibly by adding constraints. Other approaches freeze the weights and modulate the output by learning additional task-specific parameters. An economical approach is to use off-the-shelf-features with one or more readout layers . Other approaches use custom connection schema. instead modulate the output by applying learned weight masks. These approaches, like side-tuning, are examples of additive learning. Incremental learning has the objective of learning a sequence of tasks T 1,..., T m and, at the end of training, performing well on the entire set. The sequential presentation creates two problems for neural networks. The first is catastrophic forgetting and the second is learning speed, which should not slow down as more tasks are added. Because of these issues (and scaling), not every network adaptation approach lends itself to incremental learning. Standard incremental learning approaches avoid catastrophic forgetting by imposing constraints how the parameters are updated. relegates each task to approximately orthogonal subspaces.;; add a parameter regularization term per task. Imposing constraints tends to slow down learning on later tasks (intransigence,) while making tasks independent ignores the possibility for useful transfer from relevant previous tasks. Additive methods in general, and side-tuning in particular, have the advantage that they do not suffer from catastrophic forgetting and are capable of transfer. However, additive methods have not been much explored because it is assumed that they either have poor performance or scale poorly. We show that side-tuning has good performance and scaling, and demonstrate the additive advantages experimentally; using the iCIFAR and Taskonomy datasets. Meta-learning seeks to create agents that rapidly adapt to new problems by first training on tasks sampled from a standing distribution of tasks. Side-tuning is fundamentally compatible with this formulation and with existing approaches (e.g.). Moreover, recent work suggests that these approaches work primarily by feature adaptation rather than rapid learning , and feature adaptation is also the motivation for our method. Residual Learning exploits the fact that it is sometimes easier to approximate a difference rather than the original function. This has been successfully used in ResNets and robotics, where residual RL learns a single task by first training a coarse policy (e.g. behavior cloning) and then training a residual network on top (using RL). Additive Learning in Other Literature. Concepts similar to additive learning have been studied in a number of fields. For instance, developing infants are hypothesized to learn separate, discontinuous, and context-dependent perception systems during development . Adults are able to rapidly learn new affordances, but only when those are minor updates to familiar, well-practiced systems . On a more fine-grained scale, there are areas of functional specificity within the brain , including wholly separate pathways where output is mutually conditioned on one another . Side-tuning learns a side model S(X) and combines this with a base model B(x) so that the representations for the target task are computed as R(x) B(x) ⊕ S(x). The base model B(x) provides some core cognition or perception, and we put no restrictions on how B(x) is computed. We never update B(x), and in our approach it has zero learnable parameters. In general B(x) could be nonparametric, and it might not be optimized for any particular task. We consider several choices for B(x) in Section 4.4, but the simplest choice is just a pretrained network. Unlike the base model, the side network S(x) is updated during training; learning a residual that we apply on top of the base encoding. Iteratively learning residuals for a single task is known as gradient boosting (see Section 4.4 for a comparison). Side-tuning is instead focused on learning multiple tasks. One crucial component of the framework is that the complexity of the side network can scale to the difficulty of the problem at hand. When the base is relevant and requires only a minor update, a very small network can suffice. Section 4.4 explores the effect of network size, how that changes with the choice of base and target tasks. While the side network can be initialized using a variety of methods, we initialize the side network with a copy of the base network. When the forms of the base and side networks differ, we initialize the side network with weights distilled from the base network using knowledge distillation . We test alternatives in Section 4.4. The final side-tuning representation is a combination, B(x) ⊕ S(x). What should ⊕ be? Side-tuning admits several options for this combination operator. Choosing max yields the subsumption architecture. Concatenation and summation are other viable choices. We observe that alpha blending, a ⊕ b αa + (1 − α)b, works well in practice. Alpha blending preserves the dimensions of the inputs and is simpler than concatenation. In fact, concatenation followed by a channel-collapsing operation (e.g. 1x1 convolution) is a strict generalization of alpha-blending. While simple, alpha blending is expressive enough that it encompasses several common transfer learning approaches. As shown in Figure 3 and when the side network is the same as the base, sidetuning is equivalent to feature extraction when α = 1. When α = 0, side-tuning is instead equivalent to fine-tuning. If we allow α to vary during training (which we generally do), then switching α from 1 to 0 is equivalent to the common (stage-wise) training curriculum in RL where a policy is trained on top of some fixed features that are unlocked partway through training. Another notable curriculum is α(N) = k k+N for k > 0 (hyperbolic decay). In this curriculum, α controls the weighting of the prior (B(x)) with the learned estimate (S(x)), and the weight of the evidence scales with the amount of data. This curriculum is suggestive of a maximum a posteriori estimate and, like the MAP estimate, it converges to the MLE (fine-tuning, α = 0). Finally, α can treated as a learnable free parameter that determines how heavily to weight the base model. In practice, the value of α correlates with task relevance (see Section 4.4). When minimizing estimation error there is often a tradeoff between the bias and variance contributions . Choosing between feature extraction or fine-tuning exemplifies this dilemma. Feature extraction (α = 0) locks the weights and corresponds to a point-mass prior that, unless the weights are already optimal, yields a very biased estimator. In fact, the estimator allows no adaptation to new evidence and is asymptotically inconsistent. On the other hand, fine-tuning (α = 1), is an uninformative prior yielding a low-bias high-variance estimator. With enough data, fine-tuning can produce better estimates, but this usually takes more data than feature extraction. Side-tuning addresses both the these problems. It reduces variance by including the fixed features in the representation, and it is consistent because it allows updating via the residual side network. While α provides a way to control the importance of the prior, another natural approach for enforcing a prior is to penalize deviations from the original feature representation. Typically, it is easier to specify meaningful explicit priors on outputs (e.g. L2 for pixels) than on the latent representations, which can be difficult if not impossible to interpret. As long as the decoder D: Y → A is differentiable, any distance measure on the outputs can be pulled back through the decoder and into the latent space. This induced distance d D on the latent representations is called the pullback metric in differential geometry, and in deep learning it is called the perceptual loss . This may be a useful method for knowledge transfer when (i) the previous task is relevant to the new task and (ii) there is limited training data. A recent successful application of this approach would be the auxiliary losses in GPT , though we did not find it effective. Perceptual regularization is often used to dampen catastrophic forgetting. For example, Elastic Weight Consolidation uses a diagonalized second-order Taylor expansion of the expectation of the pullback metric. Learning Without Forgetting uses a decoder-based approach that can be interpreted as jointly updating both the base network and the pullback metric. We show that such regularization does not fully address the problem of catastrophic forgetting (Section 4.2.1). Side-tuning avoids catastrophic forgetting by design (as the base network is never updated). Network adaptability is the sole criterion only if we care we solely about raw performance on a single target task. In reality we often care about the performance on both the current and previous tasks. This is the case for incremental learning, where we want an agent that can learn a sequence of tasks T 1,..., T m and, at the end, is capable of reasonable performance across the entire set. Thus, catastrophic forgetting becomes a major issue. In our experiments we dedicate one new side network to each new task and train it independently of the earlier side networks. In principle, learning of new tasks can benefit from all the side networks learned in previous tasks (i.e. the nth task can use all n − 1 previous tasks). Since we do not make use of this available information, our should be considered as a lower bound on side-tuning performance. We show that this simple approach provides a strong baseline for incremental learning-outperforming existing approaches in the literature while using fewer parameters on more tasks (in Section 4.2). Side-tuning takes an additive learning approach to incremental learning, which means that alreadylearned components are never updated and performance across the whole set can only increase as the agent sees more tasks. This monotonicity is the key property of the additive family of algorithms. It is worth repeating that there is No Catastrophic Forgetting in Additive Learning and a typical learning curve for one of the tasks is shown in Figure 4. Furthermore, because our implementation of side-tuning treats problems independently of their order in the sequence (always using the fixed base and one side network), side-tuning incurs no rigidity during training. We show this in Section 4.2.2. Side-tuning naturally handles other continuous learning scenarios besides incremental learning. A related problem is that of continuous adaptation, where the agent needs to perform well (e.g. minimizing regret) on a stream of tasks with undefined boundaries and where there might very little data per task and no task repeats. As we show in Section 4.2, inflexibility becomes a serious problem for constraint-based methods and task-specific performance declines after learning more than a handful of tasks. Moreover, continuous adaptation requires an online method as task boundaries must be detected and data cannot be replayed (e.g. to generate constraints for EWC). Side-tuning could be applied to continuous adaptation by keeping a small working memory of cheap side networks that constantly adapt the base network to the input task. These side networks are small, easy to train, and when one of the networks begins performing poorly (e.g. signaling a distribution shift) that network can simply be discarded. This is an online approach, and online adaptation with small cheap networks has found recent success in (e.g.). In the first section we show that side-tuning compares favorably to existing incremental learning approaches on both iCIFAR and the more challenging Taskonomy dataset. We then extend to multiple domains (computer vision, RL, imitation learning, NLP) in the simplified (transfer learning) scenario for N = 2 tasks. Finally, we interpret side-tuning in a series of analysis experiments. We provide comparisons of side-tuning against the following methods: The network is given a good random initialization and then trained normally. The base network is used as-is and is not updated during training. Fine-tuning: An umbrella term that encompasses a variety of techniques, we consider a more narrow definition where pretrained weights are used as initialization and then training proceeds as in scratch. A constraint-based incremental learning approach from. We use the formulation from which scales better-, giving an advantage to EWC since otherwise we could use a larger side-tune network and maintain parameter parity. Parameter Superposition (PSP): A parameter-masking approach from. Progressive Neural Network (PNN): A network adaptation approach from. Independent: Each task uses a network trained independently for the target task. This method uses far more learnable parameters than all the alternatives (e.g. saving a separate ResNet-50 for each task) and achieves very strong performance. Due to the scaling, it is generally not considered an incremental learning method. incremental learning experiments for three tasks on Taskonomy (left) and iCIFAR dataset (right). The fact that side-tuning losses are flat after training (as we go right) shows that it does not forget previously learned tasks. Similarly, the performance remains consistent even on later tasks (as we go down), showing that side-tuning does not become rigid. Alternative methods clearly forget (e.g. PSP) and/or become rigid (e.g. EWC). In Taskonomy, PNN and Independent are hidden under Sidetune. In iCIFAR, Sidetune (A) merges base and side information with a multilayer perceptron (adapter). On both the Taskonomy dataset and incremental CIFAR (iCIFAR, a), side-tuning outperforms existing incremental learning approaches while using fewer parameters 1. Moreover, the performance gap is larger on more challenging datasets. Taskonomy includes labels for multiple computer vision tasks including 2D (e.g. edge detection), 3D (e.g. surface normal estimation), and semantic (e.g. object classification) tasks. We first selected the twelve tasks that make predictions from a single RGB image, and then created an incremental learning setup by selecting a random order in which to learn these tasks (starting with curvature). As images are 256x256 we use a ResNet-50 for the base network and a 5-layer convolutional network for the side-tuning side network. The number of learnable network parameters used across all tasks is 24.6M for EWC and PSP, and 11.0M for side-tuning 2. iCIFAR. First, we pretrain the base network (ResNet-44) on CIFAR-10. Then the 10 subsequent tasks are formed by partitioning CIFAR-100 classes into 10 disjoint sets of 10-classes each. We train on each subtask for 20k steps before moving to the next one. Our state-of-the-art substitutive baselines (EWC and PSP) update the base network for each task (683K parameters), while sidetuning updates a four layer convolutional network per task (259K parameters after 10 tasks). As expected, there is no catastrophic forgetting in side-tuning. Figure 6 shows that the error for side-tuning does not increase after training (blue shaded region), while it increases sharply for the other methods on both Taskonomy and iCIFAR. The difference is meaningful, and Figure 5 shows sample predictions from side-tuning vs. EWC for a few tasks during and after training. As is evident from the bottom rows, EWC exhibits catastrophic forgetting on all tasks (worse image quality as we move right). In contrast, side-tuning (top) shows no forgetting and the final predictions are significantly closer to the ground-truth (boxed red). Side-tuning learns later tasks as easily as the first, while constraint-based methods such as EWC stagnate. The predictions for later tasks such as surface normals (in Figure 5) are significantly better using side-tuning-even immediately after training and before any forgetting can occur. Figure 7: Rigidity and average rank on Taskonomy and iCiFAR. From left: Side-tuning always learns new tasks easily; EWC becomes increasingly unable to learn new tasks as training progresses. Center: The same trend holds on iCIFAR, and the average rigidity is zero for side-tuning (and almost zero for PSP). Right: Side-tuning outperforms alternatives on both datasets, achieving a significantly better average rank on all tasks. Figure 7 quantifies this slowdown. We measure rigidity as the log-ratio of the actual loss or the ith task over the loss when that task is instead trained first in the sequence. As expected, side-tuning experiences zero slowdown on both datasets. For EWC, the increasing constraints make learning new tasks increasingly difficult-and the log-ratio increases with the number of tasks (Taskonomy, left). It is too rigid (log-ratio > 0) even in iCIFAR, where the later tasks are similar to earlier ones. Overall, side-tuning significantly outperforms the other methods while using fewer than half the number of trainable parameters of the other methods. When the other methods use smaller networks, their performance decreases further. On both iCIFAR and Taskonomy, side-tuning achieves the best average rank (1.12 of 4 on Taskonomy, while the next best is 2.33 (PSP)). This is a direct of the fact (shown above) that side-tuning does not suffer from catastrophic forgetting or rigidity. It is not due to the fact that the sidetuning structure is specially designed for these types of image tasks; it is not (we show in Sec. 4.3 that it performs well on other domains). In fact, the much larger networks used in EWC and PSP should achieve better performance on any single task. For example, EWC produces sharper images early on in training, before it has had a chance to accumulate too many constraints (e.g. reshading in Fig. 5). But this factor was outweighed by side-tuning's immunity from the effects of catastrophic forgetting and creeping rigidity. In order to address the possibility that side-tuning is somehow domain-or task-specific, we provide showing that it is well-behaved in other settings. As the concern with additive learning is mainly that it is too inflexible to learn new tasks, we compare with fine-tuning (which outperforms other lifelong learning tasks when forgetting is not an issue). For extremely limited amounts of data, feature extraction can outperform fine-tuning. We show that side-tuning generally performs as well as features or fine-tuning-whichever is better. We trained networks to perform one of three target tasks (object classification, surface normal estimation, and curvature estimation) on the Taskonomy dataset and varied the size of the training set N ∈ {100, 4 × 10 6}. In each scenario, the base network was trained (from scratch) to predict one of the non-target tasks. The side network was a copy of the original base network. We experimented with a version of fine-tuning that updated both the base and side networks; the were similar to standard fine-tuning 3. In all scenarios, side-tuning successfully matched the adaptiveness of fine-tuning, and significantly outperformed learning from scratch, as shown in Figure 4.3. The additional structure of the frozen base did not constrain performance with large amounts of data (4M images), and side-tuning performed as well as (and sometimes slightly better than) fine-tuning. We also evaluated side-tuning on a question-answering task (SQuAD v2 ) using a non-convolutional architecture. We use a pretrained BERT model for our base, and a second for the side network 3. Unlike in the previous experiments, BERT uses attention and no convolutions. Still, side-tuning adapts to the new task just as well as fine-tuning, outperforming features and scratch (Figure 4.3). We trained an agent to navigate to a target coordinate in the Habitat environment. The agent is provided with both RGB input image and also an occupancy map of previous locations. The map does not contain any information about the environment-just previous locations. In this section we use Behavior Cloning to train an agent to imitate experts following the shortest path on 49k trajectories in 72 buildings. The agents are evaluated in 14 held-out validation buildings. Depending on the what the base network was trained on, the source task might be useful (Curvature) or harmful (Denoising) for imitating the expert and this determines whether features or learning from scratch performs best. Figure 4.3 shows that regardless of the which approach worked best, side-tuning consistently matched or beat it. Reinforcement Learning for Navigation in Habitat Using a different learning algorithm (PPO) and using direct interaction instead of expert trajectories, we observe identical trends. We trained agents directly in Habitat (74 buildings) 3. Figure 4.3 shows performance in 16 held-out buildings after 10M frames of training. Side-tuning performs comparably to the max of competing approaches. Task relevance predicts alpha α. In our experiments, we treat α as a learnable parameter and find that the relative values of α are predictive of emprical performance. In imitation learning (Fig. 4.3), curvature (α = 0.557) outperformed denoising (α = 0.252). In Taskonomy, the α values from training on just 100 images predicted the actual transfer performance to normals in , (e.g. curvature (α = 0.56) outperformed object classification (α = 0.50)). For small datasets, usually α ≈ 0.5 and the relative order, rather than the actual value is important. We showed in the previous section that side-tuning performs like the best of {features, fine-tuning, scratch} in domains with abundant or scant data. In order to test whether side-tuning could profitably synthesize the features with intermediate amounts of data, we evaluated each approach's ability to learn to navigate using 49, 490, 4900, or 49k expert trajectories and pretrained denoising features. Side-tuning was always the best-performing approach and, on intermediate amounts of data (e.g. 4.9k trajectories), outperformed the other techniques (side-tune 9.3 vs. fine-tune: 7.5, features: 6.7, scratch: 6.6), Figure 9b ). Network size. Does network size matter? We find (i) If the target problem benefits from a large network (e.g. classification tasks), then performance is sensitive to side network size but not size of the base. (ii) The base network can usually be distilled to a smaller network and sidetuning will still offer advantages over alternatives. In the supplementary material we provide supporting experiments from Taskonomy using both high-and low-data settings (curvature → {obj. class, normals}, obj. class → normals), and in Habitat (RL using {curvature, denoise} → navigation). Not Boosting. Since the side network learns a residual on top of the base network, we ask: what benefits we could glean by extending side-tuning to do boosting? Although network boosting this does improve performance on iCIFAR (Figure 9a), if catastrophic forgetting is not a concern then the parameters would've been better used in a deeper network rather than many shallow networks. Initialization. A good side network initialization can yield a minor boost in performance. We found that initializing from the base network slightly outperforms a low-energy initialization 4, which slightly outperforms Xavier initialization. However, we found that these differences were not statistically significant across tasks (H 0 : pretrained = xavier; p = 0.07, Wilcoxon signed-rank test). We suspect that initialization might be more important on harder problems. We test this by repeating the analysis without the simple texture-based tasks (2D keypoint + edge detection and autoencoding) and find the difference in initialization is now significant (p = 0.01). More than just stable updates. In RL, fine-tuning often fails to improve performance. One common rationalization is that the early updates in RL are'high variance'. The usual solution is to first train using fixed features and then unfreeze the weights at some point in training (via a hyperparameter to be set). We found that this stage-wise approach performs as well (but no better than) keeping the features fixed-and side-tuning performed as well as both while being simpler than stage-wise (Fig. 9c). We tested the'high-variance update' theory by fine-tuning with both gradient clipping and an optimizer designed to prevent such high-variance updates by adaptively warming up the learning rate . This provided no benefits over vanilla fine-tuning, suggesting that the benefits of side-tuning are not solely due to gradient stabilization early in training. We have introduced the side-tuning framework, a simple yet effective approach for additive learning. Since it does not suffer from catastrophic forgetting or rigidity, it is naturally suited to incremental learning. The theoretical advantages are reflected in empirical , and side-tuning outperforms existing approaches in challenging contexts and with state-of-the-art neural networks. We further demonstrated that the approach is effective in multiple domains and with various network types. The naïve approach to incremental learning used in this paper made a number of design decisions. These decisions could be analyzed and subsequently relaxed. In particular: Flexible parameterizations for side networks: Our incremental learning experiments used the same side network architecture for all subtasks. A method for automatically adapting the networks to the subtask at hand could make more efficient use of the computation and supervision. Better forward transfer: Our experiments used only a single base and single side network. Leveraging the already previously trained side networks could yield better performance on later tasks. Learning when to deploy side networks: Like most incremental learning setups, we assume that the tasks are presented in a sequence and that task identities are known. Using several active side networks in tandem would provide a natural way to detecting distribution shift. Using side-tuning to measure task relevance: We noted that α tracked task relevance, but a more rigorous treatment of the interaction between the base network, side network, α and final performance could yield insight into how tasks relate to one another. We test the effect of base model architecture on performance and find that the small five layer convolutional network does comparable to the ResNet-50 when using features. Rectified Adam is a method introduced to deal with destructive high variance updates at the beginning of training. We tried using this for RL but found no improvements (shown in Figure 15). We ablate over different quantities of expert trajectories. We observe that when data is scarce, features is a powerful choice whereas when data is plentiful, fine-tuning performs well. In both scenarios, side-tuning is able to perform as well as the stronger approach. In domains with very few examples, we found that side-tuning is unable to match the performance of other methods. We evaluated our setup in vision transfer for 5 images from the same building, imitation learning given 5 expert trajectories. Taskonomy Our data is 4M images on 12 single image tasks. The tasks that we use are the following: curvature, semantic segmentation, reshading, keypoints3d, keypoints2d, texture edges, occlusion edges, distance, depth, surface normals, object classification and autoencoding. The tasks were chosen in no particular special order. Our base model and side model are ResNet-50s. We pretrain on curvatures. Then, we train each task for three epochs before moving on to the next task. We use cross entropy loss for classification tasks (semantic segmentation and object classification), L2 loss for curvature and L1 loss for the other tasks. We use Adam optimizer with an initial learning rate of 1e-4, weight decay coefficient of 2e-6, gradient clipping to 1.0, and batch size of 32. We evaluate our performance on a held out set of images, both immediately after training a specific task, and after training of all the tasks are complete. iCIFAR We start by pretraining a model on CIFAR 10 (from https://github.com/ akamaster/pytorch_resnet_cifar10). Then we partition CIFAR100 into 10 distinct sets of 10 classes. Then, we train for 4 epochs on these tasks using Adam optimizer, learning rate of 1e-3, batch size of 128. We train and test on the the question answering dataset SQuAD2.0, a reading comprehension dataset consisting of 100,000 questions with 50,000 unanswerable questions. Both our base encoding and side network is a BERT transformer pretrained on a larger corpus. Finetuning trains a single BERT transfer. We use the training setup found at https://github.com/huggingface/ pytorch-transformers (train for 2 epochs at a learning rate of 3e-5) wth one caveat -we use an effective batch size of 3 (vs. their 24) due to the We borrow the experimental setup from work to be published in October 2019: We use the Habitat environment with the Gibson dataset. The dataset virtualizes 572 actual buildings, reproducing the intrinsic visual and semantic complexity of real-world scenes. We train and test our agents in two disjoint sets of buildings (Fig. ??). During testing we use buildings that are different and completely unseen during training. We use up to 72 building for training and 14 test buildings for testing. The train and test spaces comprise 15678.4m 2 (square meters) and 1752.4m 2, respectively. The agent must direct itself to a given nonvisual target destination (specified using coordinates), avoiding obstacles and walls as it navigates to the target. The maximum episode length is 500 timesteps, and the target distance is between 1.4 and 15 meters from the start. This setup is shared between imitation learning and RL, which differ in the data, architecture and optimization process. Imitation Learning We collect 49,325 shortest path expert trajectories in Habitat, 2,813,750 state action pairs. We learn a neural network mapping from states to actions. Our base encoding is a ResNet-50 and the side network is a five layer convolutional network. The representation output is then fed into a neural network policy. We train the model for 10 epochs using cross entropy loss and Adam at an initial learning rate of 2e-4 and weight decay coefficient of 3.8e-7. We initialize alpha to 0.5. Finetuning uses the same model architecture but updates all the weights. Feature extraction only uses the ResNet-50 to collect features. RL Similarly, we borrow the RL setup from the same work. In all experiments we use the common Proximal Policy Optimization (PPO) algorithm with Generalized Advan-tage Estimation. Due to the computational load of ren-dering perceptually realistic images in Gibson we are only able to use a single rollout worker and we therefore decorre-late our batches using experience replay and off-policy vari-ant of PPO. The formulation is similar to Actor-Critic with Experience Replay (ACER) in that full trajectories are sampled from the replay buffer and reweighted using the first-order approximation for importance sampling. During training, the agent receives a large one-time reward for reaching the goal, a positive reward proportional to Euclidean distance toward the goal and a small negative reward each timestep. The maximum episode length is 500 timesteps, and the target distance is between 1.4 and 15 meters from the start. Due to this paradigms' compute and memory constraints, it would be difficult for us to use large architectures for this setting. Thus, our base encoding is a five layer convolutional network distilled from the trained ResNet-50. Our side network is also a five layer convoultional network. Finetuning is handled the same way -update all the weights in this setup. Feature extraction uses the five layer network to collect features. Low energy initialization In classical teacher student distillation, the student is trained to minimize the distance between its output and the teacher's output. In this setting, we minimize the distance between the teacher's output and the summation of the student's output and the teacher's output). The output space may have a different geometry than that of the input space and this would allow us to work with A.5 ADDITIONAL ANALYSIS We provide alternative perspectives and additional insights for our lifelong learning tasks. iCIFAR In Fig. 7 (right), we see that the average rank of side-tuning higher than that of PNN. We find that side-tuning can bridge this gap with a multilayer perceptron (adapter) to merge the base and side networks. This is a common practice in PNN. In Fig. 20, we see with the adapter network, the two methods are very similar when measuring classification error. I n d e p. P N N S i d e t u n e (A) S i d e t u n e E W C P S P F i n e t u n e In d e p. S id e t u n e P N N 3 P N N F in e t u n e E W C P S P In d e p. S id e t u n e P N N 3 P N N F in e t u n e E W C P S P In d e p. S id e t u n e P N N 3 P N N F in e t u n e E W C P S P Taskonomy In Fig. 7 (right), we found that the ranking of our method is better than all other methods, including PNN. By altering the connections in the PNN, we found an alternate (PNN3) that has comparable performance to side-tuning. In 21, we show all the losses normalized by single task loss (independent) as presented in. The quantitative performance of our method outperforms all other methods and matches closely with that of PNN. We qualitatively show in Figures 22, 23, 24, that these methods are comparable in performance. An alternative perspective views these methods as various fusions between some base information and new side information. In this framework, side-tuning is a late-fusion approach whereas PNN is a distributed-fusion approach. In Fig. 25, we compare various fusion methods in iCIFAR and find that late fusion performs better than early fusion and comparable to if not better than distributed fusion. We run this analysis in Taskonomy as well -while the loss values differ somewhat, we find that the qualitative seen in Figures 26, 27, 28 are rather similar. Thus, we conclude that the methods do not vary much. Distributed and early fusion require knowledge about the structure of how the information is computed. This is something late fusion is agnostic to and it can consider each information column a black box -this is useful in the case that your base information is NOT a neural network, perhaps non-parametric. In Fig. A.5.2, we show that side-tuning can effectively use ground truth curvature as a base for lifelong learning whereas all the methods we compare against cannot use this information. Specifically, we downsample the curvature image and stack it into the same dimensions as the side output. Side-tuning with ground truth curvature achieves a better rank on the Taskonomy dataset than all other methods and comparable performance. GT Late Dist Early Figure 29: Sidetuning can be used successfully even with black-box side information. When the base information comes from a black-box process for which we have no other information, sidetuning can still be used (and performance improves vis-a-vis not using the inputs, and vs using inputs generated from a neural network). Existing lifelong learning approaches have no standard way to make use of this type of information.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Side-tuning adapts a pre-trained network by training a lightweight "side" network that is fused with the (unchanged) pre-trained network using a simple additive process.

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.