Question: represent the text to find a very short summary (1-2 sentences max)

Question: While momentum-based methods, in conjunction with the stochastic gradient descent, are widely used when training machine learning models, there is little theoretical understanding on the generalization error of such methods. In practice, the momentum parameter is often chosen in a heuristic fashion with little theoretical guidance. In this work, we use the framework of algorithmic stability to provide an upper-bound on the generalization error for the class of strongly convex loss functions, under mild technical assumptions. Our bound decays to zero inversely with the size of the training set, and increases as the momentum parameter is increased. We also develop an upper-bound on the expected true risk, in terms of the number of training steps, the size of the training set, and the momentum parameter. A fundamental issue for any machine learning algorithm is its ability to generalize from the training dataset to the test data. A classical framework used to study the generalization error in machine learning is PAC learning BID0 BID1. However, the associated bounds using this approach can be conservative. Recently, the notion of uniform stability, introduced in the seminal work of Bousquet and Elisseeff BID2, is leveraged to analyze the generalization error of the stochastic gradient method (SGM) BID3. The in BID3 ) is a substantial step forward, since SGM is widely used in many practical systems. This method is scalable, robust, and widely adopted in a broad range of problems. To accelerate the convergence of SGM, a momentum term is often added in the iterative update of the stochastic gradient BID4. This approach has a long history, with proven benefits in various settings. The heavy-ball momentum method was first introduced by Polyak BID5, where a weighted version of the previous update is added to the current gradient update. Polyak motivated his method by its resemblance to a heavy ball moving in a potential well defined by the objective function. Momentum methods have been used to accelerate the backpropagation algorithm when training neural networks BID6. Intuitively, adding momentum accelerates convergence by circumventing sharp curvatures and long ravines of the sublevel sets of the objective function BID7. For example, Ochs et al. has presented an illustrative example to show that the momentum can potentially avoid local minima BID8. Nesterov has proposed an accelerated gradient method, which converges as O(1/k 2) where k is the number of iterations . However, the Netstrov momentum does not seem to improve the rate of convergence for stochastic gradient (, Section 8.3.3). In this work, we focus on the heavy-ball momentum. Although momentum methods are well known to improve the convergence in SGM, their effect on the generalization error is not well understood. In this work, we first build upon the framework in BID3 to obtain a bound on the generalization error of SGM with momentum (SGMM) for the case of strongly convex loss functions. Our bound is independent of the number of training iterations and decreases inversely with the size of the training set. Secondly, we develop an upper-bound on the optimization error, which quantifies the gap between the empirical risk of SGMM and the global optimum. Our bound can be made arbitrarily small by choosing sufficiently many iterations and a sufficiently small learning rate. Finally, we establish an upper-bound on the expected true risk of SGMM as a function of various problem parameters. We note that the class of strongly convex loss functions appears in several important machine learning problems, including linear and logistic regression with a weight decay regularization term. Other related works: convergence analysis of first order methods with momentum is studied in (; BID11 BID12 BID13 BID14 BID15 BID16 BID17 . Most of these works consider the deterministic setting for gradient update. Only a few works have analyzed the stochastic setting BID15 BID16 BID17 . Our convergence analysis are not directly comparable with these works due to their different assumptions regarding the properties of loss functions. In particular, we analyze the convergence of SGMM for a smooth and strongly convex loss function as in BID3, which is new. First-order methods with noisy gradient are studied in BID18 and references therein. In BID18, the authors show that there exists linear regression problems for which SGM outperforms SGMM in terms of convergence. Our main focus in this work is on the generalization, and hence true risk, of SGMM. We are aware of only one similar work in this regard, which provides stability bounds for quadratic loss functions BID19 . In this paper, we obtain stability bounds for the general case of strongly convex loss functions. In addition, unlike BID19, our show that machine learning models can be trained for multiple epochs of SGMM with bounded generalization errors. We use E[·] to denote the expectation and · to represent the Euclidean norm of a vector. We use lower-case bold font to denote vectors. We use sans-serif font to denote random quantities. Sets and scalars are represented by calligraphic and standard fonts, respectively. We consider a general supervised learning problem, where S = {z 1, · · ·, z n} denotes the set of samples of size n drawn i.i.d. from some space Z with an unknown distribution D. We assume a learning model described by parameter vector w. Let f (w; z) denote the loss of the model described by parameter w on example z ∈ Z. Our ultimate goal is to minimize the true or population risk: DISPLAYFORM0 Since the distribution D is unknown, we replace the objective by the empirical risk, i.e., DISPLAYFORM1 We assume w = A(S) for a potentially randomized algorithm A(·). In order to find an upper-bound on the true risk, we consider the generalization error, which is the expected difference of empirical and true risk: DISPLAYFORM2 Finally, to upper bound g, we consider uniform stability:Definition 1 Let S and S denote two data sets from space Z n such that S and S differ in at most one example. Algorithm A is s -uniformly stable if for all data sets S, S, we have DISPLAYFORM3 It is shown in BID3 ) that uniform stability implies generalization in expectation:Theorem 1 BID3 If A is an s -uniformly stable algorithm, then the generalization error of A is upper-bounded by s.Theorem 1 shows that it is enough to control the uniform stability of an algorithm to upper bound the generalization error. In our analysis, we will assume that the loss function satisfies the following properties. DISPLAYFORM0 We assume that the parameter space Ω is a convex set. Furthermore, for the loss function to be L-Lipschitz and and strongly convex, we further assume that Ω is compact. Since Ω is compact, the SGMM update requires projection. The update rule for projected SGMM is given by: DISPLAYFORM0 where P denotes the Euclidean projection onto Ω, α > 0 is the learning rate 1, µ > 0 is the momentum parameter, i t is a randomly selected index, and f (w t ; z it) is the loss evaluated on sample z it. In SGMM, we run the update iteratively for T steps and let w T denote the final output. Note that there are two typical approaches to select i t. The first approach is to select i t ∈ {1, · · ·, n} uniformly at random at each iteration. The second approach is to permutate {1, · · ·, n} randomly once and then select the examples repeatedly in a cyclic manner. Our are valid for both approaches. The key quantity of interest in this paper is the generalization error for SGMM given by: DISPLAYFORM1 since the randomness in A arises from the choice of i 0, · · ·, i T −1. In the following, we assume that the loss function f (·; z) is β-smooth, L-Lipschitz, and γ-strongly convex for all z. Theorem 2 (Stability bound) Suppose that the SGMM update is executed for T steps with constant learning rate α and momentum µ. Provided that DISPLAYFORM0 The in Theorem 2 implies that the stability bound decreases inversely with the size of the training set. It increases as the momentum parameter µ increases. These properties are also verified in our experimental evaluation. Theorem 3 (Convergence bound) Suppose that the SGMM update is executed for T steps with constant learning rate α and momentum µ. Then we have DISPLAYFORM1 whereŵ T denotes the average of T steps of the algorithm, i.e.,ŵ T = DISPLAYFORM2 Theorem 3 bounds the optimization error, i.e., the expected difference between the empirical risk achieved by SGMM and the global minimum. Upon setting µ = 0 and γ = 0 in FORMULA8, we can recover the classical bound on optimization error for SGM BID20, (, Theorem 5.2). The first two terms in vanish as T increases. The terms with negative sign improve the convergence due to the strongly convexity. The last term depends on the learning rate, α, the momentum parameter µ, and the Lipschitz constant L. This term can be controlled by selecting α sufficiently small. Proposition 1 (Upper-bound on true risk) Suppose that the SGMM update is executed for T steps with constant learning rate α and momentum µ, satisfying the conditions in Theorem 2 and DISPLAYFORM3 T, we have: DISPLAYFORM4 where DISPLAYFORM5 andŵ T as well as the constants W 0, · · ·, W 3 are defined in Theorem 3.Proposition 1 provides a bound on the expected true risk of SGMM in terms of the global minimum of the empirical risk. The bound in FORMULA11 is obtained by combining Theorem 2 and Theorem 3 and minimizing the expression over α. The choice of α simplifies considerably when µ is sufficiently small, as stated in Proposition 1. Due to the page constraint, the proof of this is provided in the supplementary material. Note that the first two terms in vanish as T increases. The last term in vanishes as the number of samples n increases. Following BID3, we track the divergence of two different iterative sequences of update rules with the same starting point. However, our analysis is more involved as the presence of momentum term requires a more careful bound on the iterative expressions. To keep the notation uncluttered, we first consider SGMM without projection and defer the discussion of projection to the end of this proof. Let S = {z 1, · · ·, z n} and S = {z 1, · · ·, z n} be two samples of size n that differ in at most one example. Let w T and w T denote the outputs of SGMM on S and S, respectively. We consider the updates w t+1 = G t (w t) + µ(w t − w t−1) and w t+1 = G t (w t) + µ(w t − w t−1) with G t (w t) = w t − α∇ w f (w t ; z it) and G t (w t) = w t − α∇ w f (w t ; z it), respectively, for t = 1, · · ·, T. We denote δ t ∆ = w t − w t. Suppose w 0 = w 0, i.e., δ 0 = 0. We first establish an upper-bound on E A [δ t+1] in terms of E A [δ t] and E A [δ t−1] in the following lemma, whose proof is provided in the supplementary document. DISPLAYFORM0 Using the of Lemma 1, in the following, we develop an upper bound on E A [δ T]. Let us consider the recursion DISPLAYFORM1 withδ 0 = δ 0 = 0. Upon inspecting it is clear that DISPLAYFORM2 as we simply drop the remainder of positive terms. Substituting into, we have DISPLAYFORM3 where the second inequality holds due to µ ≥ αβγ β+γ − 1 2.Noting that DISPLAYFORM4 where the second expression holds since 0 ≤ µ < αβγ 3(β+γ) is assumed. Applying the L-Lipschitz property on f (·, z), it follows that DISPLAYFORM5 Since this bound holds for all S, S and z, we obtain an upper-bound on the uniform stability and the proof is complete. Our stability bound in Theorem 2 holds for the projected SGMM update because Euclidean projection does not increase the distance between projected points (the argument is essentially analogous to BID3, Lemma 4.6)). In particular, note that Lemma 1 holds for the projected SGMM. Again, we first consider SGMM without projection and discuss the extension to projection at the end of this proof. Our proof is inspired by the convergence analysis in BID15 BID13 for a convex loss function with bounded variance and time-decaying learning rate. Different from these works, we analyze the convergence of SGMM for a smooth and strongly convex loss function with constant learning rate. To facilitate the convergence analysis, we define: DISPLAYFORM0 with p 0 = 0. Substituting into the SGMM update, the parameter recursion is given by DISPLAYFORM1 It follows that DISPLAYFORM2 Substituting p t into, the recursion can be written as DISPLAYFORM3 Upon taking the expectation with respect to i t in FORMULA0 we have DISPLAYFORM4 where we use the fact that ∇ w f (w t ; z it) ≤ L, due to L-Lipschitz, and that E it [∇ w f (w t ; z it)] = ∇ w R S (w t). Furthermore, since R S (·) is a γ-strongly convex function, for all w t and w t−1, we have DISPLAYFORM5 Substituting FORMULA1 in FORMULA0, we have DISPLAYFORM6 Taking expectation over i 0, · · ·, i t for a given S, summing for t = 0, · · ·, T, and rearranging terms, we have DISPLAYFORM7 Since · is a convex function, for all w T and w, we have DISPLAYFORM8 Furthermore, due to convexity of R S (·), we have DISPLAYFORM9 Taking expectation over S, applying inequalities and FORMULA1 into FORMULA1, and substituting w = w * S, we obtain and the proof is complete. Our convergence bound in Theorem 3 can be extended to projected SGMM. Let use denote y t+1 ∆ = w t + µ(w t − w t−1) − α∇ w f (w t ; z it). Then, for any feasible w ∈ Ω, holds for y t+1, i.e., DISPLAYFORM10 Note that the LHS of can be written as DISPLAYFORM11 We note that µw t + (1 − µ)w ∈ Ω for any w ∈ Ω and w t ∈ Ω since Ω is convex. Now in projected SGMM, we have DISPLAYFORM12 since projection a point onto Ω moves it closer to any point in Ω. This shows inequality holds, and the convergence do not change. In this section, we validate the insights obtained in our theoretical in experimental evaluation. Our main goal is to study how adding momentum affects the convergence and generalization of SGM. We study the performance of SGMM when applied to the notMINIST dataset. Please note that similar are provided for the MNIST dataset in the supplementary document. We train a logistic regression model with the weight decay regularization using SGMM for binary classification on the two-class notMNIST dataset that contains the images from letter classes "C" and "J", which leads to a smooth and strongly convex loss function. We set the learning rate α = 0.01. The weight decay coefficient and the minibatch size are set to 0.001 and 10, respectively. We use 100 SGMM realizations to evaluate the average performance. We compare the training and generalization performance of SGM without momentum with that of SGMM under µ = 0.5 and µ = 0.9, which are common momentum values used in practice (, Section 8.3 .2).The generalization error (with respect to cross entropy) and training error versus the number of training samples, n, under SGMM with fixed T = 1000 iterations are shown in FIG0, respectively, for µ = 0, 0.5, 0.9. In FIG1, we plot the generalization error (with respect to classification accuracy) and the training accuracy as a function of the number of training samples for the same dataset. First, we observe that the generalization error (with respect to both cross entropy and classification accuracy) decreases as n increases for all values of µ, which is suggested by our stability upper-bound in Theorem 2. In addition, for sufficiently large n, we observe that the generalization error increases with µ, consistent with Theorem 2. On the other hand, the training error increases as n increases, which is expected. We can observe that adding momentum reduces training error as it improves the convergence rate. The training accuracy also improves by adding momentum as illustrated in FIG1. In order to study the optimization error of SGMM, we show the training error and test error versus the number of epochs, under SGMM trained with n = 500 samples in Figures 3a and 3b, respectively. We plot the classification accuracy for training and test datasets in Figures 4a and 4b, respectively. We observe that the training error decreases as the number of epochs increases for all values of µ, which is consistent with the convergence analysis in Theorem 3. Furthermore, as expected, we see that adding momentum improves the training error and accuracy. However, as the number of epochs increases, we note that the benefit of momentum on the test error and accuracy becomes negligible. This happens because adding momentum also in a higher generalization error thus penalizing the gain in training error. We study the generalization error and convergence of SGMM for the class of strongly convex loss functions, under mild technical conditions. We establish an upper-bound on the generalization error, which decreases with the size of the training set, and increases as the momentum parameter is increased. Secondly, we analyze the convergence of SGMM during training, by establishing an upper-bound on the gap between the empirical risk of SGMM and the global minimum. Our proposed bound reduces to a classical bound on the optimization error of SGM BID20 for convex functions, when the momentum parameter is set to zero. Finally, we establish an upper-bound on the expected difference between the true risk of SGMM and the global minimum of the empirical risk, and illustrate how it scales with the number of training steps and the size of the training set. Although our are established for the case when the learning rate is constant, they can be easily extended to the case when the learning rate decreases with the number of iterations. We also present experimental evaluations on the notMNIST dataset and show that the numerical plots are consistent with our theoretical bounds on the generalization error and the convergence gap.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Stochastic gradient method with momentum generalizes.

An empirical study of variational inference based on chi-square divergence minimization, showing that minimizing the CUBO is trickier than maximizing the ELBO


Question: Represent the scientific article to find a short TLDR

Question: Imitation learning from demonstrations usually relies on learning a policy from trajectories of optimal states and actions. However, in real life expert demonstrations, often the action information is missing and only state trajectories are available. We present a model-based imitation learning method that can learn environment-specific optimal actions only from expert state trajectories. Our proposed method starts with a model-free reinforcement learning algorithm with a heuristic reward signal to sample environment dynamics, which is then used to train the state-transition probability. Subsequently, we learn the optimal actions from expert state trajectories by supervised learning, while back-propagating the error gradients through the modeled environment dynamics. Experimental evaluations show that our proposed method successfully achieves performance similar to (state, action) trajectory-based traditional imitation learning methods even in the absence of action information, with much fewer iterations compared to conventional model-free reinforcement learning methods. We also demonstrate that our method can learn to act from only video demonstrations of expert agent for simple games and can learn to achieve desired performance in less number of iterations. Reinforcement learning(RL) involves training an agent to learn a policy that accomplishes a certain task in an environment. The objective of reinforcement learning is to maximize the expected future reward from a guiding signal. BID11 showed that neural networks can be used to approximate state-action value functions used by an agent to perform discrete control based on a guiding reward. This was demonstrated in Atari games where the score was used as the reward signal. Similarly, continuous control of robotics arm was achieved by BID9 minimizing the distance between end-effector and target location. Following these, other methods such as BID20 were proposed to improve the sample efficiency of modelfree algorithms with theoretical guarantees of policy improvement in each step. These algorithms assume that a guiding reward signal is available for the agent to learn optimal behavior for a certain task. However, in most cases of natural learning, such guiding signal is not present and learning is performed by imitating an expert behavior. Imitation learning involves copying the behavior of an expert agent to accomplish the desired task. In the conventional imitation learning setting, a set of expert trajectories providing states and optimal actions τ = {s 0, a 0, s 1, a 1, ..., s n, a n) performed by an expert agent π E are available but the reward (or cost function), r E (s, a) used to achieve the expert behavior is not available. The goal is to learn a new policy π, which imitates the expert behavior by maximizing the likelihood of given demonstration trajectories. A straightforward way for imitation learning is to direct learn the optimal action to perform given the current state as proposed by; BID2. The policy π can learn to imitate the expert behavior by maximizing likelihood of the condition distribution of action given states p(a|s). This can be achieved by simply training a parameterized function (neural networks for instance) with state and action pairs from the expert trajectories. Since this involves end-to-end supervised learning, training is much more sample-efficient compared to reinforcement learning and overcomes inherent problems in model-free methods such as credit assignment BID22 ). However, since behavior cloning learns optimal action from a single state value only, it is unaware of the future state distribution the current action will produce. Thus, errors are compounded in the future states leading to undesired agent behavior as shown by BID18; BID17. Therefore, numerous training samples are required for behavior cloning to reduce errors in action prediction required for satisfactory imitation learning. The second approach to imitation learning involves setting up exploration in a Markov Decision Process(MDP) setting. The goal then is to recover a reward signal that best explains the expert trajectories. BID12 first introduced Inverse Reinforcement Learning(IRL), where the goal is to find a reward signalr from the trajectories such that the expert is uniquely optimal. After computing this estimated reward signal, usually, a model-free reinforcement learning performed to obtain the desired policy imitating the expert behavior by maximizing the expected discounted reward E π (t γ tr (s t, a t)). While this alleviates the problem of compounding errors as in behavior cloning, BID25 showed that estimating a unique reward function from state and action trajectories is an ill-posed problem. Following the success of Generative Adversarial Networks(GANs) BID3 ) in various fields of machine learning, adversarial learning has also been shown incorporated in the imitation learning framework. The recent work on Generative Adversarial Imitation Leaning or GAIL by BID4 showed that model-free reinforcement learning using the discriminator as a cost function can learn to imitate the expert agent with much less number of demonstrated trajectories compared to behavior cloning. Following the success of GAIL, there have extensions by BID0 to model-based generative imitation learning using a differentiable dynamics model of the environment. Robust imitation policy strategies using a combination of variational autoencoders BID7; BID16 ) and GAIL has also been proposed by BID23.The previous works assume that the expert trajectories consist of both action and state values from the optimal agent. However, optimal actions are usually not available in real-world imitation learning. For example, we often learn tasks like skipping, jump rope, gymnastics, etc. just by watching other expert humans perform the task. In this case, the optimal expert trajectories only consist of visual input, in other words, the consecutive states of the expert human with no action information. We learn to jump rope by trying to reproduce actions that in state trajectories similar to the state trajectories observed from the expert. This requires exploring the environment in a structured fashion to learn the dynamics of the rope (for jump rope) which then enables executing optimal actions to imitate the expert behavior. The recent work of BID10 presents learning from observations only with focus to transferring skills learned from source domain to an unseen target domain, using rewards obtained by feature tracking for model-free reinforcement learning. Inspired by the above method of learning in humans, we present a principled way of learning to imitate an expert from state information only, with no action information available. We first learn a distribution of the next state from the current state trajectory, used to estimate a heuristic reward signal enabling model-free exploration. The state, action and next states information from modelfree exploration is used to learn a dynamics model of the environment. For the case of learning in humans, this is similar to performing actions for replicating the witnessed expert state trajectories, which in turn gives information about the dynamics of the environment. Once this forward model is learned, we try to find the action that maximizes the likelihood of next state. Since the forward model gives a function approximation for the environment dynamics, we can back propagate errors through it to perform model-based policy update by end to end supervised learning. We demonstrate that our proposed network can reach, with fewer iterations, the level close to an expert agent behavior (which is a pre-trained actor network or manually provided by humans), and compare it with reinforcement learning using a hand-crafted reward or a heuristics reward that is based on prediction error of next state learned from the optimal state trajectories of the expert. We summarize the notations used in the paper in this section. Consider a Markov Decision Process (MDP) denoted as (S, A, P, r, ρ 0, γ), where S is the finite set of states, A is the set of possible actions, P: S × A → S is the transition probability distribution and r: S × A → R be the reward signal from state and actions, ρ 0 → R is the initial state distribution and γ ∈ is the discount factor. Let π: S × A → be the policy that gives the conditional distribution of actions given current state, p(a|s) and R(π) = E π [t γ t r t (s t, a t)] is the discounted reward associated with the policy. We consider expert trajectories consisting of only optimal state distribution without any action information, τ E = {s 0, s 1, ..., s n}. The trajectories sampled from model-free exploration is denoted as, τ RL = {s 0, a 0, s 1, a 1, ..., s n, a n}. We use the terms dynamic model, state-transition probability and forward model interchangeably in the paper. f E (s t, a t) denotes the non-differentiable forward model of the environment and f (s t, a t) denotes the differentiable version which is learned during the proposed training procedure. π mf denotes the model-free policy and π mb denotes the model-based policy network. While most previous works use trajectories containing both state and action information to infer a policy that imitates the expert behavior, our problem statement is to imitate an expert from optimal state trajectories only. This setting is common in many natural scenarios where only state information is available. For example, humans learning to swim by seeing videos or observing other experts swimmers, have access to the sensory stream of information containing states only. The optimal actions are obtained by trying to replicate the state trajectories in the real environment via exploration. In this work, we learn a time series predictor of next state given the current state. Subsequently, we estimate a heuristic reward signal at each step, based on the difference of predicted next state by the time series model and the actual next state taken by the policy network to learn a model-free policy with exploration. However, such heuristic methods suffer from the disadvantages of slow modelfree training to reach satisfactory policy and provide no guarantees on convergence. Therefore we resort to model-based policy learning that learns a differentiable model of the environment dynamics used to learn policy by directly supervised learning. The proposed algorithm alternates between dynamic model parameters and policy parameters update using gradient propagation through the differentiable dynamics model. We formalize this setup in the following sections. Following the work in Inverse Reinforcement Learning(IRL), it is possible to frame the imitation learning problem in the MDP framework. The imitation learning problem then reduces to finding a uniquely optimal reward signal that maximizes the likelihood of the observed demonstration trajectories. However, finding a uniquely optimal reward function, even with optimal trajectories containing both state and action information, is an ill-posed problem. The ill-posed nature of optimal reward estimation is further accentuated in the absence of action information in expert trajectories. As such, many solutions for parameterized families of reward estimation models maximizing the likelihood of only expert state trajectories might be sub-optimal to learn the desired task. For model-free policy learning, we use a heuristic reward estimated by the error in next state prediction at each step. We make an assumption that estimating a locally optimal reward maximizing the likelihood of next step predicting, is intuitively globally optimal for learning the desired task. A straight-forward method to obtain such heuristic reward signal from the trajectories, τ s, is to learn a time series predictive model of the next state given the current state, p(s t+1 |s 1:t) from the expert state trajectories using time series modeling. In our case, we use an exponential of the difference between predicted states and actual next state associated with the action predicted by the policy network. DISPLAYFORM0 where k is a constant controlling gain of the reward signal and σ controls the sensitivity of reward to divergence from the predicted state. This reward can be used for guiding any standard model-free reinforcement learning techniques to maximize the expected reward, R(DISPLAYFORM1 We assume on the intuition that locally optimal heuristic reward estimation will be sufficient to ensure global optimality for learning the desired task. Showing the overall architecture of the proposed method. Firstly, model-free policy is updated using reward estimation from next state mismatch which storing the samples in replay buffer. These are used to update the differentiable dynamics model of the environment, which provides the error gradients for end-to-end model-based policy update from state trajectories Consider there are m sets of expert trajectory episode each consisting of T states, given as τ E = {s 0, s 1, ..., s n}, where n = mT. We assume that the trajectories in each episode are independent of each other. For the imitation learning problem, we wish to imitate an expert agent π E from state trajectories τ E. we formulate a maximum likelihood problem for state trajectories given a parameterized model of the agent policy, given as DISPLAYFORM0 where θ represents the parameter of the model. We assume that the random variables state(s t), action(a t) and next state s t+1 form a directed graphical model as shown in figure 2 (b). Following the natural dynamics of environments in reinforcement learning setting, we assume that control action for the agent a t is conditionally independent of other state and actions in the past given the current state. Distribution of the next state is conditionally independent of the other past states given current state and action following the MDP setting. In this framework, we can frame the modelbased policy as an encoder network with action as the latent variable and the dynamics policy as the decoder network predicting the next state. The log-likelihood estimation loss, in this case, can be written as, DISPLAYFORM1 where θ e, theta d are the encoder and decoder parameters respectively. Learning can be performed using (s t, s t+1) pairs from the expert trajectories by minimizing the above loss, sampling action values from the posteriori distribution p(a t |s t) using standard Markov Chain Monte Carlo (MCMC) methods or variational inference methods BID7; BID16. However, the learned action from the encoder in this case will not mimic the actual control commands used to perform the desired task in the control environment. We propose a constrained minimization cost function which enforces the decoder network to mimic the environment dynamics of the agent. The proposed cost function enforces that the decoders model minimizes the loss for dynamics model prediction of next state given the current state and action, whereas the composition of encoder over the decoder minimizes prediction loss of the next state given the current state. This loss function is given as, DISPLAYFORM2 where we θ dyn are the parameters of dynamics model and θ mb are model-based policy network's parameters. Let us denote the first term of the loss term in equation 4 as model-based policy loss, L mb and the second term is referred as dynamics model loss, L dyn. As shown in FIG0, we perform alternate minimizations on the proposed cost function and training on the encoder and decoder are performed on two separate datasets. Firstly, the dynamics model parameters are updated from the experience replay samples gathered during model-free exploration. Subsequently, the updated dynamics model is used as the decoder network in the above formulation with fixed weights while the model-based policy parameters are updated by the gradient, ∇ θ mb L mb. This enforces the encoder network to act as a model-based policy that learns to predict the optimal action given the current state. During implementations, a deterministic encoder and decoder are used and a single action is sampled from the posterior distribution during training. Since our expert trajectories only consist of sensory streams containing state information only, learning from the heuristic reward in a MDP setting can be slow and does not guarantee that the reward is optimal for the desired task. Thus we resort to dynamics model based learning which can propagate error gradients with respect to action information as discussed in the above section. Consider an analogy of a robot learning to navigate through a maze of expert state information only. It must first learn the state transition model p(s t+1 |s t, a t) to navigate through the environment. Once dynamics model is learned, it can obtain the best action that takes the current state in optimal state trajectories to the next state. Solving for the desired action at each state (p(a t |s t)) is a maximum likelihood problem from the expert state trajectories which can be solved by end-to-end supervised learning. Let us assume we have a parameterized model for agent dynamics, given as s t+1 = f (s t, a t ; θ dyn), where θ dyn denotes parameters of the model. During model-free learning, we store the trajectories of (s t, a t, s t+1) that were encountered during exploration by the agent. Let us denotes the trajectories of these triplets as τ RL. For continuous state spaces, the gradient for dynamic model parameters are given as DISPLAYFORM0 which is gradient on the mean squared error loss between model predicted and true next state. For the discrete state space case, we can first maximize the probability of next state given the current state and action using a categorical loss. Any standard stochastic gradient descent algorithms can be used for the above optimizations. However, we use neural networks as function approximators which are shown to have approximation capacity for arbitrary non-linear function, although its nonconvex nature of optimization does always guarantee a solution that is globally optimal. Recent techniques in stochastic gradient descent (; BID6) have alleviated this problem to a large extent. If we assume the dynamics model is ideal in predicting the next state and there exists a unique action to reach the next state from the current state, then the proposed method is identical to behavior cloning, although true action information is not provided by the expert. Therefore, the performance of this method is upper bounded by the performance of behavior cloning model which learns from the true action information between states. In our formulation so far, the entire next state of the agent is predicted by the next state predictor. However, we found that predicting a part of the state which is dependent on action gives better reward structure. This is in line with the work of BID14, where the authors predict φ(s t) as the latent representation of the neural network predicting the action from consecutive states, a t = g(φ(s t), φ(s t+1)). This transformed state value, φ(s t) is also used as the input and output for the dynamics model. This transformation is beneficial for two reasons: (i) It is difficult to learn the dynamics model for high dimensional state information and thus first projecting onto a low dimensional manifold to learn the dynamics model gives a more feasible learning framework.(ii) In case of transferring the learned dynamics model between different tasks that use the same environment, a common state input is required for the dynamics model, which can be achieved such transformation. In case of learning from videos, we use the agent position in the image frames as φ(s t). For the case of linked arm reacher, we will use phi(s t) as the joint angles and joint velocities. We now outline the algorithm based on the above discussed model-based policy learning framework. Sample trajectory from model-free policy τ k ∈ π k mf and add to the replay buffer τ RL Updated the dynamics model parameter using trajectories from the replay buffer DISPLAYFORM0 Update model based parameter from expert trajectories with fixed dynamics model. DISPLAYFORM1 The above algorithm shows an iterative process where in each iteration we first train a model-free algorithm using the heuristic reward function. This step is necessary because we collect a certain amount of system dynamics data, (s t, a t, s t+1), while training the model-free policy. Then, we train a system dynamics model using the above collected data. The action policy is then trained using the system dynamics model in the model-based part, which constitutes one cycle of the training. Subsequently, we repeat this cycle again starting from the model-free part. With each iteration, we collect additional system dynamics data, which in a more precise dynamics model, leading to accurate action policy parameter gradients for updating the model-based policy. The frequency of switching between model-free replay buffer collection and model-based update can be varied depending on the complexity of dynamics model. For state predictions from previous states, we used Long Short-Term Memory Network, proposed by BID5. For the policy and dynamics model, we use neural networks as function approximators. In this work, we assume that the state transformation φ is manually specified in each experiments, although it is possible to learn such representation by learning a common transformation between states that predicts the action, a t = g(φ(s t), φ(s t+1)). We perform experimental evaluations on three kinds of environment, (i) Robotics arm reacher in 2d, (ii) Simple 2d obstacle avoidance based reacher, (iii) Learning to play simple games from raw video demonstrations. In each experiment, we show specific strengths of the proposed algorithm as follows. We use roboschool reacher(; BID1 environment to simulate twolink robotic arm that can move in two-dimensional space. The desired task is to learn reaching a given target from random starting configurations. The arm is controlled using angular torque values of both joints. We use state values consisting of angular position, angular velocity, the end effector location of the robotic link and the position of the target. The robotic arm angles and angular velocities were used as φ(s t), which is the portion of state dependent on action. In this experiment, we assume that the true reward function is known and in addition, we have some state trajectories from the expert policy. The goal is to leverage these existing state trajectories to learn a better policy in less number of steps. Reward signal, consisting of distance potential, electricity cost and the penalty for stuck joint, which is the default reward specified for the environment, was used. Specifically, we show 500 trajectories of optimal states each with 100 steps to learn model-based policy using the proposed method. We used neural networks with hidden layers containing neurons for both model-based and model-free policies. Training was performed over 2000 episodes using Deep Deterministic Policy Gradients(DDPG) proposed by BID9.Figure 3(a) shows the comparison of proposed method against the DDPG algorithm. Our method learns the dynamics model from the model-free exploration, which quickly learns the simple environment dynamics, in this case, thereby learning an optimal policy imitating the state trajectories much faster than model-free training, which is shown in . However, we found this is due to a large number of state trajectories that are shown to the proposed method. Since our performance is upper bounded by behavior cloning , we share the same drawbacks of compounding errors and data-hungry policy learning. In this experiment, we demonstrate that proposed algorithm can be used for direct end-to-end supervised imitation learning on novel tasks without resorting to model-free reinforcement learning. We refer to this setup as one-shot imitation learning, which we demonstrate on a simple toy environment. The environment is shown in FIG1. The environment consists of an agent which can freely move in 2D space. The agent is controlled by continuous position control where the action is the change in (x, y) agent position at each time step. The goal is to reach the target location while avoiding the obstacle. Initially, we train our algorithm on an environment to avoid a single obstacle while reaching the target. We use state information as the absolute position of the agent, target and obstacle, the agent velocity and relative location of obstacle and target with respect to the agent. We use φ(s t) as the agent 2d position in the environment. For expert demonstration, we implemented a manually engineered policy that always avoids the obstacle. We used 1000 number of demonstrations containing only state trajectories to reach a target while avoiding a single obstacle. Out of 1000 demonstrations, 800 are used for training and 200 for validation. We first learn the time series prediction of the next state, used to compute the heuristic reward based using prediction error as discussed in section 2.1. For model-based policy, we use a MLP with hidden units. We use the same policy network for both model-based and modelfree policy. The dynamics model is also modeled as a neural network with a single hidden layer of 8 units for both state and action input. We used a switching frequency of 5 between the model-free and model-based updates. Using these setting for the proposed algorithm, we get a model-based policy and the dynamics model as output. Using the dynamics model obtained from training with demonstrations of single obstacle avoidance, we perform one-shot imitation learning to learn avoidance of two obstacles. The algorithm is presented 500 samples of expert state trajectories for avoiding two obstacles. The model-based policy in the new setting is learned by step 6 of the proposed algorithm 1 using the previously learned dynamics model. Although the state information for the policy networks might change due to an additional obstacle, since φ(s t), which is the agent 2d location, remains same in both cases, we can perform one-shot imitation learning in this case. We compare the with respect to the expert policy and behavior cloning and report the average of test reward on 50 episodes. While the expert policy achieves average test reward of 3036, and behavior cloning achieves 3939 and our imitation learning method gave a reward of 3805. This demonstrates that our proposed algorithm can be used for one-shot imitation in environments with same dynamics and can produce comparable to behavior cloning which was trained from true actions. In this experiment, we learn model-based control policies from raw pixels. We use the python reimplementation of the game Flappy bird BID8 ). In this environment, the agent has to hover through the pipes without collision by flapping its wings. The environment has gravity and with each flap, the agent receives an upward acceleration which sends it to an upward parabolic trajectory. We choose this environment due to its complicated dynamics and show that our proposed method can learn to model this dynamics. We learn action policies from raw videos of just 10 episodes each with 1000 steps. The reward is assumed to be unknown in this case and we estimate the reward by the error in prediction of the next state as mentioned in section 2.1. The control action is a single discrete command of whether to flap the bird's wings or not. We denotes this action space as {+1, −1}. For state information at each step we use 4 consecutive frames resized to (80 × 80 × 4). We also assume that the absolute position of the bird's location is available to us, which we use as φ(s t). This can be also computed by learning a simple object detector from image frames. For the next state predictor, we use an LSTM predictor that outputs the next position of the agent location given the sequence of states witnessed so far. The model-free reward prediction step receives a reward signal based on the difference of the actual next state taken the policy, from the predicted next state by LSTM. This reward is used to train DQN BID11 ) for model-free policy update which collects data to train the dynamics model, which in turn trains the model-based policy network. We also train vanilla DQN using a standard reward, which in this case is 0.1 for each time step and +1 reward if the agent can successfully pass through a pipe. To compare the various methods, we used the reward used by vanilla DQN as a baseline for comparison. For the model-based policy we used a convolutional neural network (CNN) with soft-max output, which is essentially a binary classification task. For the DQN model-free policy, the last layer predicts the q-values and therefore has linearly activation. In this case, we first learn dynamics model, which is approximated as a multi layered perceptron (MLP) with single hidden layer of 16 units for both state and action inputs. It learns to regress the next state from the current state and actions minimizing mean squared error loss. After this, from the expert demonstration of state trajectory(τ E), we find the current optimal action that minimizes the next state a * k = arg min a p(s t+1 − f (s t, a t) | at∈{0,1},{st+1,st}∼τ E. The next step is to find the modelbased policy by behavior cloning on the states (s t ∼ τ E) and current optimal actions, a * k. We found that, the number of +1(flap) actions by the agent are far less frequent compared to the number of -1(no flap) actions which cause an unbalance in distribution. Thus, we used class balancing techniques for learning both model based policy update and direct behavior cloning from true actions (baseline) . Figure 3(b) shows the comparison of proposed model-based method with estimated reward (mb reward pred), behavior cloning (bc), model-free RL using DQN with standard known reward (dqn reward) and with estimated reward (dqn reward pred). It is to be noted that although original reward just provides a constant reward (of 0.1 and +1 bonus) without constant guidance at each step value, estimated reward provides a dense guidance signal which leads to faster convergence as shown in the comparison . We take desired samples from the model-free training, based on prioritized sampling technique with regards to the estimated reward signal, to perform model-based policy update. We found that prioritized sampling was essential to learn a good dynamics model of the environment. The show that the model-based policy learns behavior close to optimal in much fewer steps compared to the model-free counterparts. However, its performance in upper bounded by the behavior cloning method which achieves an average test reward (over 20 iterations) of 49.62. The expert score is 60.9. Since DQN with estimated reward signal learns via exploration in a MDP setting, it can surpass the performance of behavior cloning since the number of expert demonstrations is limited in this case. We presented a model-based imitation learning method that can learn to act from expert state trajectories in the absence of action information. Our method uses trajectories sampled from the modelfree policy exploration to train a dynamics model of the environment. As model-free policy is enriched through time, the forward model can better approximate the actual environment dynamics, which leads to improved gradient flow, leading to better model-based policy update which is trained in a supervised fashion from expert state trajectories. In the ideal case, when dynamics model perfectly approximates the environment, our proposed method is equivalent to behavior cloning, even in the absence of action information. We demonstrate that the proposed method learns the desired policy in less number of iterations compared conventional model-free methods. We also show that once the dynamics model is trained it can be used to transfer learning for other tasks in a similar environment in an end-to-end supervised manner. Future work includes tighter integration of the model-based learning and the model-free learning for higher data efficiency by sharing information between the model-free policy π mf and the model-based policy π mb and between the next state predictor p(s t+1 |s t) and the dynamics model p(s t+1 |s t, a t) and improving the limitations of compounding errors and requirement of large number of demonstration, by adversarial training which can maximize likelihood of future state distributions as well.

Represent the short summary for finding the corresponding scientific article (long)

Learning to imitate an expert in the absence of optimal actions learning a dynamics model while exploring the environment.

Represent the TLDR for finding the corresponding scientific article

Adaptation of an RL agent in a target environment with unknown dynamics is fast and safe when we transfer prior experience in a variety of environments and then select risk-averse actions during adaptation.


Question: Represent the scientific article to find a short TLDR

Question: Recent research has proposed the lottery ticket hypothesis, suggesting that for a deep neural network, there exist trainable sub-networks performing equally or better than the original model with commensurate training steps. While this discovery is insightful, finding proper sub-networks requires iterative training and pruning. The high cost incurred limits the applications of the lottery ticket hypothesis. We show there exists a subset of the aforementioned sub-networks that converge significantly faster during the training process and thus can mitigate the cost issue. We conduct extensive experiments to show such sub-networks consistently exist across various model structures for a restrictive setting of hyperparameters (e.g., carefully selected learning rate, pruning ratio, and model capacity). As a practical application of our findings, we demonstrate that such sub-networks can help in cutting down the total time of adversarial training, a standard approach to improve robustness, by up to 49% on CIFAR-10 to achieve the state-of-the-art robustness. Pruning has served as an important technique for removing redundant structure in neural networks (b; a; ;). Properly pruning can reduce cost in computation and storage without harming performance. However, pruning was until recently only used as a post-processing procedure, while pruning at initialization was believed ineffective (a;). Recently, proposed the lottery ticket hypothesis, showing that for a deep neural network there exist sub-networks, when trained from certain initialization obtained by pruning, performing equally or better than the original model with commensurate convergence rates. Such pairs of sub-networks and initialization are called winning tickets. This phenomenon indicates it is possible to perform pruning at initialization. However, finding winning tickets still requires iterative pruning and excessive training. Its high cost limits the application of winning tickets. Although shows that winning tickets converge faster than the corresponding full models, it is only observed on small networks, such as a convolutional neural network (CNN) with only a few convolution layers. In this paper, we show that for a variety of model architectures, there consistently exist such sub-networks that converge significantly faster when trained from certain initialization after pruning. We call these boosting tickets. We observe the standard technique introduced in for identifying winning tickets does not always find boosting tickets. In fact, the requirements are more restrictive. We extensively investigate underlining factors that affect such boosting effect, considering three stateof-the-art large model architectures: VGG-16 , ResNet-18 , and WideResNet . We conclude that the boosting effect depends principally on three factors: (i) learning rate, (ii) pruning ratio, and (iii) network capacity; we also demonstrate how these factors affect the boosting effect. By controlling these factors, after only one training epoch on CIFAR-10, we are able to obtain 90.88%/90.28% validation/test accuracy (regularly requires >30 training epochs) on WideResNet-34-10 when 80% parameters are pruned. We further show that the boosting tickets have a practical application in accelerating adversarial training, an effective but expensive defensive training method for obtaining robust models against adversarial examples. Adversarial examples are carefully perturbed inputs that are indistinguishable from natural inputs but can easily fool a classifier . We first show our observations on winning and boosting tickets extend to the adversarial training scheme. Furthermore, we observe that the boosting tickets pruned from a weakly robust model can be used to accelerate the adversarial training process for obtaining a strongly robust model. On CIFAR-10 trained with WideResNet-34-10, we manage to save up to 49% of the total training time (including both pruning and training) compared to the regular adversarial training process. Our contributions are summarized as follows: 1. We demonstrate that there exists boosting tickets, a special type of winning tickets that significantly accelerate the training process while still maintaining high accuracy. 2. We conduct extensive experiments to investigate the major factors affecting the performance of boosting tickets. 3. We demonstrate that winning tickets and boosting tickets exist for adversarial training scheme as well. 4. We show that pruning a non-robust model allows us to find winning/boosting tickets for a strongly robust model, which enables accelerated adversarial training process. 2 AND RELATED WORK Network pruning has been extensively studied as a method for compressing neural networks and reducing resource consumption (; ; ; b; ;). However, it was previously believed that pruned networks cannot be trained from the start (b;). Surprisingly, recent research has shown it is possible to prune a neural network at the initialization and still reach similar performance as the full model . Within this category, the lottery ticket hypothesis states a randomly-initialized dense neural network contains a sub-network that is initialized such that, when trained in isolation, learns as fast as the original network and matches its test accuracy. In, an iterative pruning method is proposed to find such sub-networks. Specifically, this approach first randomly initializes the model. The initialization is stored separately and the model is trained in the standard manner until convergence. Then a certain proportion of the weights with the smallest magnitudes are pruned while remaining weights are reset to the previously stored initialization and ready to be trained again. This train-prune-reset procedure is performed several times until the target pruning ratio is reached. Using this pruning method, they show the ing pruned networks can be trained to similar accuracy as the original full networks, which is better than the model with the same pruned structure but randomly initialized. One of the limitations of the lottery ticket hypothesis, as pointed in, is that winning tickets are found by unstructured pruning which does not necessarily yield faster training or executing time. In addition, finding winning tickets requires training the full model beforehand, which is time-consuming as well, especially considering iterative pruning. In this paper, we aim to show that there exists a subset of winning tickets, namely boosting tickets, that not only performs equally well as the original model but also converges much faster. Given a classifier f: X → {1, . . ., k} for an input x ∈ X, an adversarial example x adv is a perturbed version of x such that D(x, x adv) < for some small > 0, yet being mis-classified as f (x) = f (x adv). D(·, ·) is some distance metric which is often an p metric, and in most of the literature ∞ metric is considered, so as in this paper. The procedure of constructing such adversarial examples is often referred to as adversarial attacks. One of the simplest attacks is a single-step method, Fast Gradient Sign Method (FGSM) , manipulating inputs along the direction of the gradient with respect to the outputs:, where Π x+S is the projection operation that ensures adversarial examples stay in the p ball S around x. Although this method is fast, the attack is weak and can be defended easily. On the other hand, its multi-step variant, Projected Gra-dient Descend (PGD), is one of the strongest attacks :, where x is initialized with a random perturbation. Since PGD requires to access the gradients for multiple steps, it will incur high computational cost. On the defense side, currently the most successful defense approach is constructing adversarial examples via PGD during training and add them to the training sets as data augmentation, which is referred to as adversarial training . One caveat of adversarial training is its computational cost due to performing PGD attacks at each training step. Alternatively, using FGSM during training is much faster but the ing model is robust against FGSM attacks but vulnerable against PGD attacks . In this paper, we show it is possible to combine the advantages of both and quickly train a strongly robust model benefited from the boosting tickets. Prior studies have shown success in achieving both compactness and robustness of the trained networks (; ; ; ; ;). However, most of them will either incur much higher training cost or sacrifice robustness from the full model. On the contrary, our approach is able to reduce training time while obtaining similar/higher robust accuracy than the original full network. We first investigate boosting tickets on the standard setting without considering adversarial robustness. In this section, we show that with properly chosen hyperparameters, we are managed to find boosting tickets on VGG-16 and ResNet that can be trained much faster than the original dense network. Detailed model architectures and the setup can be found in Supplementary Section A. To find the boosting tickets, we use a similar algorithm for finding winning tickets, which is briefly described in the previous section and will be detailed here. First, a neural network is randomly initialized and saved in advance. Then the network is trained until convergence, and a given proportion of weights with the smallest magnitudes are pruned, ing in a mask where the pruned weights indicate 0 and remained weights indicate 1. We call this train-and-prune step pruning. This mask is then applied to the saved initialization to obtain a sub-network, which are the boosting tickets. All of the weights that are pruned (where zeros in the mask) will remain to be 0 during the whole training process. Finally, we can retrain the sub-networks. The key differences between our algorithm and the one proposed in to find winning tickets are (i) we use a small learning rate for pruning and retrain the sub-network (tickets) with learning rate warm-up from this small learning rate. In particular, for VGG-16 we choose 0.01 for pruning and warmup from 0.01 to 0.1 for retraining; for ResNet-18 we choose 0.05 for pruning and warmup from 0.05 to 0.1 for retraining; (ii) we find it is sufficient to prune and retrain the model only once instead of iterative pruning for multiple times. In Supplementary Section B, we show the difference of boosting effects brought from the tickets found by iterative pruning and one-shot pruning is negligible. Note warmup is also used in. However, they propose to use warmup from small learning rate to a large one during pruning as well, which hinders the boosting effect as shown in the following experiments. First, we show the existence of boosting tickets for VGG-16 and ResNet-18 on CIFAR-10 in Figure 1 and compare to the winning tickets. In particular, we show the boosting tickets are winning tickets, in the sense that they outperform the randomly initialized models. When compared to the winning tickets, boosting tickets demonstrate equally good performance with a higher convergence rate. Similar on MNIST can be found in Supplementary Section C. To measure the overall convergence rate, early stopping seems to be a good fit in the literature. It is commonly used to prevent overfitting and the final number of steps are used to measure convergence rates. However, early stopping is not compatible with learning rate scheduling we used in our case where the total number of steps is determined before training. This causes two issues in our evaluation in Figure 1: (i) Although the boosting tickets reach a relatively high validation accuracy much earlier than the winning ticket, the training procedure is and d) for winning tickets, boosting tickets, and randomly initialized weights. In both models, the boosting tickets show faster convergence rate and equally good performance as the winning tickets. then hindered by the large learning rate. After the learning rate drops, the performance gap between boosting tickets and winning tickets becomes negligible. As a , the learning rate scheduling obscures the improvement on convergence rates of boosting tickets; (ii) Due to fast convergence, boosting tickets tend to overfit, as observed in ResNet-18 after 50 epochs. To mitigate these two issues without excluding learning rate scheduling, we conduct another experiment where we mimic the early stopping procedure by gradually increasing the total number of epochs from 20 to 100. The learning rate is still dropped at the 50% and 75% stage. In this way, we can better understand the speed of convergence without worrying about overfitting even with learning rate scheduling involved. In figure 2, we compare the boosting tickets and winning tickets in this manner on VGG-16. While the first two plots in Figure 2 show the general trend of convergence, the improvement of convergence rates is much clearer in the last four plots. In particular, the validation accuracy of boosting tickets after 40 epochs is already on pair with the one trained for 100 epochs. Meanwhile, the winning tickets fall much behind the boosting tickets until 100 epochs where two finally match. We further investigate the test accuracy at the end of training for boosting and winning tickets in Table 1. We find the test accuracy of winning tickets gradually increase as we allow for more training steps, while the boosting tickets achieve the highest test accuracy after 60 epochs and start to overfit at 100 epochs. Summarizing the observations above, we confirm the existence of boosting tickets and state the boosting ticket hypothesis: A randomly initialized dense neural network contains a sub-network that is initialized such that, when trained in isolation, converges faster than the original network and other winning tickets while matches their performance. In the following sections, we investigate three major components that affect the boosting effects. As finding boosting tickets requires alternating learning rates, it is natural to assume the performance of boosting tickets relies on the choice of learning rate. Thus, we extensively investigate the influence of various learning rates. We use similar experimental settings in the previous section, where we increase the total number of epochs gradually and use the test accuracy as a measure of convergence rates. We choose four different learning rates 0.005, 0.01, 0.05 and 0.1 for pruning to get the tickets. All of the tickets found by those learning rates obtain the accuracy improvement over randomly reinitialized sub-model and thus satisfy the definition of winning tickets (i.e., they are all winning tickets). As shown in the first two plots of Figure 3, tickets found by smaller learning rates tend to have stronger boosting effects. For both VGG-16 and ResNet-18, the models trained with learning rate 0.1 show the least boosting effects, measured by the test accuracy after 20 epochs of training. On the other hand, training with too small learning rate will compromise the eventual test accuracy at a certain extent. Therefore, we treat the tickets found by learning rate 0.01 as our boosting tickets for VGG-16, and the one found by learning rate 0.05 as for ResNet-18, which converge much faster than all of the rest while achieving the highest final test accuracy. Pruning ratio has been an important component for winning tickets, and thus we investigate its effect on boosting tickets. Since we are only interested in the boosting effect, we use the validation accuracy at early stages as a measure of the strength of boosting to avoid drawing too many lines in the plots. In Figure 4, we show the validation accuracy after the first and fifth epochs of models for different pruning ratios for VGG-16 and ResNet-18. For both VGG-16 and ResNet-18, boosting tickets always reach much higher accuracy than randomly reinitialized sub-models, demonstrating their boosting effects. When the pruning ratio falls into the range from 60% to 90%, boosting tickets can provide the strongest boosting effects which obtain around 80% and 83% validation accuracy after the first and the fifth training epochs for VGG-16 and obtain 76% and 85% validation accuracy for ResNet-18. On the other hand, the increase of validation accuracy between the first training epoch and the fifth training epoch become smaller when boosting effects appear. It indicates their convergence starts to saturate due to the large learning rate at the initial stage and is ready for dropping the learning rate. We finally investigate how model capacity, including the depth and width of models, affects the boosting tickets. We use WideResNet either with its depth or width fixed and vary the other factor. In particular, we keep the depth as 34 and increases the width from 1 to 10, comparing their boosting effect. Then we keep the width as 10 and increase the depth from 10 to 34. The changes of validation accuracy of the models are shown in Figure 5. Overall, Figure 5 shows models with larger capacity have a more significant boosting effect, though the boosting effects keep the same when the depth is larger than 22. Notably, we find the largest model WideResNet-34-10 achieves 90.88% validation accuracy after only one training epoch. Although the lottery ticket hypothesis is extensively studied in and, the same phenomenon in adversarial training setting lacks thorough understanding. In this section, we show two important facts that make boosting tickets suitable for the adversarial scheme: the lottery ticket hypothesis and boosting ticket hypothesis are applicable to the adversarial training scheme; pruning on a weakly robust model allows to find the boosting ticket for a strongly robust model and save training cost. In the following experiment, we use a naturally trained model, that is trained in the standard manner, and two adversarially trained models using FGSM and PGD respectively to obtain the tickets by pruning these models. Then we retrain these pruned models with the same PGD-based adversarial training from the same initialization. In Figure 6, we report the corresponding accuracy on the original validation sets and on the adversarially perturbed validation examples, noted as clean accuracy and robust accuracy. We further train the pruned model from random reinitialization to validate lottery ticket hypothesis. Unless otherwise stated, in all the PGD-based adversarial training, we keep the same setting as. The PGD attacks are performed in 10 steps with step size 2/255 (PGD-10). The PGD attacks are bounded by 8/255 in its ∞ norm. For the FGSM-based adversarial training, the FGSM attacks are bounded by 8/255. Both models trained from the boosting tickets obtained with FGSM-and PGD-based adversarial training demonstrate superior performance and faster convergence than the model trained from random reinitialization. This confirms the lottery ticket hypothesis and boosting ticket hypothesis are applicable to adversarial training scheme on both clean accuracy and robust accuracy. More interestingly, the performance of the models pruned with FGSM-and PGD-based adversarial training are almost the same. This observation suggests it is sufficient to train a weakly robust model with FGSM-based adversarial training for obtaining the boosting tickets and retrain it with stronger attacks such as PGD. This finding is interesting because FGSM-based adversarial trained models will suffer from label leaking problems as learning weak robustness. In fact, the FGSM-based adversarially trained model from which we obtain our boosting tickets has 89% robust accuracy against FGSM but with only 0.4% robust accuracy against PGD performed in 20 steps. However, Figure 6 shows the following PGD-based adversarial retraining on the boosting tickets obtained from that FGSM-based trained model is indeed robust. Further discussions can be found in Section 5. , the authors argued that the lottery ticket hypothesis fails to hold in adversarial training via experiments on MNIST. We show they fail to observe winning tickets because the models they used have limited capacity. In the adversarial setting bounded by L ∞ ≤ 0.3, small models such as a CNN with two convolutional layers used in can not yield even winning tickets when pruning ratio is large. In Figure 7, plot (a) and (b) are the clean and robust accuracy of the pruned models when the pruning ratio is 80%. The pruned model degrades into a trivial classifier where all example are classified into the same class with 11.42%/11.42% valid/test accuracy. However, when we use VGG-16, as shown in plot (c) and (d), the winning tickets are found again. This can be explained as adversarial training requires much larger model capacity than standard training , thus pruning small models could undermine their performance. Since MNIST is a simple dataset, adversarial training converges quickly at the first few epochs for both the tickets and randomly initialized models. Therefore, there is no winning tickets performing obvious boosting effect which we can identify as a boosting ticket on MNIST. We then conduct the same experiment as in Figure 2 but in the adversarial training setting to better show the improved convergence rates. The for validation accuracy and test accuracy are presented in Figure 8 and Table 2 respectively. It suggests it is sufficient to train 60 epochs to achieve similar robust accuracy as the full model trained for 100 epochs. Until now, we have confirmed that boosting tickets exist consistently across different models and training schemes and convey important insights on the behavior of pruned models. However, in the natural training setting, although boosting tickets provide faster convergence, it is not suitable for accelerating the standard training procedure as pruning to find the boosting tickets requires training full models beforehand. On the other hand, the two observations mentioned in Section 4 enable boosting tickets to accelerate adversarial training. In particular, we can find boosting tickets with FGSM-based adversarial training that and they can significantly accelerate the PGD-based adversarial training. Note that the cost of FGSM-based training is only 1/10 times of the standard 10-step PGD-based one and thus is almost negligible compared to the time saved due to the boosting effect. In Table 3, we apply adversarial training to WideResNet-34-10, which has the same structure used in , with the proposed approach for 40, 70 and 100 epochs and report the best accuracy/robust accuracy under various attacks among the whole training process. In particular, we perform 20-step PGD, 100-step PGD as white-box attacks where the attackers have the access to the model parameters. More experimental are included in the Appendix. We report the time consumption for training each model to measure how much time is saved by boosting tickets. We run all the experiments on a workstation with 2 V100 GPUs in parallel. From Table 3 we observe that while our approach requires pruning before training, it is overall faster as it uses FGSM-based adversarial training. In particular, to achieve its best robust accuracy, original Madry et al.'s training method requires 134,764 seconds on WideResNet-34-10. To achieve that, our boosting ticket only requires 69,552 seconds, including 15,462 seconds to find the boosting ticket and 54,090 seconds to retrain the ticket, saving 49% of the total training time. Not knowledge distillation. It may seem that winning tickets and boosting tickets behave like knowledge distillation where the learned knowledge from a large model is transferred to a small model. This conjecture may explain the boosting effects as the pruned model quickly recover the knowledge from the full model. However, the lottery ticket framework seems to be distinctive to knowledge distillation. If boosting tickets simply transfer knowledge from the full model to the pruned model, then an FGSM-based adversarially trained model should not find tickets that improves the robustness of the sub-model against PGD attacks, as the full model itself is vulnerable to PGD attacks. Yet in Section 4.1 we observe an FGSM-based adversarially trained model still leads to boosting tickets that accelerates PGD-based adversarial training. We believe the cause of boosting tickets requires further investigation in the future. Accelerate adversarial training. propose to reduce the training time for PGD-based adversarial training by recycling the gradients computed for parameter updates and constructing adversarial examples. While their approach focuses on reducing the computational time for each epoch, our method focuses more on the convergence rate (i.e., reduce the number of epochs required for convergence). Therefore, our approach is compatible with theirs, making it a promising future direction to combine both to further reduce the training time. In this paper, we investigate boosting tickets, sub-networks coupled with certain initialization that can be trained with significantly faster convergence rate. As a practical application, in the adversarial training scheme, we show pruning a weakly robust model allows to find boosting tickets that can save up to 49% of the total training time to obtain a strongly robust model that matches the state-ofthe-art robustness. Finally, it is an interesting direction to investigate whether there is a way to find boosting tickets without training the full model beforehand, as it is technically not necessary. Setup. We use , weight decay, decreasing learning rate schedules (×0.1 at 50% and 75%), and augmented training data for training models. We try to keep the setting the same as the one used in except we use one-shot pruning instead of iterative pruning. It allows the whole pruning and training process to be more practical in real applications. On CIFAR-10 dataset, we randomly select 5,000 images out of 50,000 training set as validation set and train the models with the rest. The reported test accuracy is measured with the whole testing set. All of our experiments are run on four Tesla V100s, 10 Tesla P100s, and 10 2080 Tis. For all the time-sensitive experiments like adversarial training on WideResNet-34-10 in Section 4.3, we train each model on two Tesla V100s with data parallelism. For the rest ones measuring the final test accuracy, we use one gpu for each model without parallelism. In Table 4, we summarize the number of parameters and parameter sizes of all the model architectures that we evaluate with including VGG-16 , ResNet-18 , and the variance of. Figure 9, we track the training of models obtained from both iterative pruning and one shot pruning. We find the performance of both, in terms of the boosting effects and final accuracy, is indistinguishable. and. We plot the validation accuracy of models from both approaches and the corresponding randomly initialized models. In this section, we report experiment on MNIST for the standard setting, where we use LeNet with two convolutions and two fully connected layers for the classification task. As for MNIST we do not use learning rate scheduling, early stopping is then used to determine the speed of convergence. In 5, we report the epochs when early stopping happens and the test accuracy to illustrate the existence of boosting tickets for MNIST. While winning tickets converge at the 18th epoch, boosting tickets converge at the 11th epoch, indicating faster convergence.. In addition to the experimental reported in the main text, in Table 6 we include for C&W attacks and transfer attacks; where we attack one model with adversarial examples found by 20-step PGD based on other models. We find the adversarial examples generated from one model can transfer to another model with a slight decrease on the robust error. This indicates our models and Madry's model share adversarial examples and further share decision boundaries. Table 6: Best test clean accuracy (the first row), robust accuracy (the second to fourth rows), transfer attack accuracy (the middle four rows), and training time for PGD-based adversarial training (the last four rows) on boosting tickets obtained by FGSM-based adversarial training in various of numbers of epochs on WideResNet-34-10. Overall, our adversarial training strategy based on boosting tickets is able to save up to 49% of the total training time while achieving higher robust accuracy compared to the regular adversarial training on the original full model.

We show the possibility of pruning to find a small sub-network with significantly higher convergence rate than the full model.

Represent the TLDR for finding the corresponding scientific article.

We introduce a type of neural network that is structurally resistant to adversarial attacks, even when trained on unaugmented training sets.  The resistance is due to the stability of network units wrt input perturbations.