Question:
represent the text to find a very short summary (1-2 sentences max)

The provided query could be 'The goal of imitation learning (IL) is to enable a learner to imitate expert behavior given expert demonstrations. Recently, generative adversarial imitation learning (GAIL) has shown significant progress on IL for complex continuous tasks. However, GAIL and its extensions require a large number of environment interactions during training. In real-world environments, the more an IL method requires the learner to interact with the environment for better imitation, the more training time it requires, and the more damage it causes to the environments and the learner itself. We believe that IL algorithms could be more applicable to real-world problems if the number of interactions could be reduced. In this paper, we propose a model-free IL algorithm for continuous control. Our algorithm is made up mainly three changes to the existing adversarial imitation learning (AIL) methods – (a) adopting off-policy actor-critic (Off-PAC) algorithm to optimize the learner policy, (b) estimating the state-action value using off-policy samples without learning reward functions, and (c) representing the stochastic policy function so that its outputs are bounded. Experimental show that our algorithm achieves competitive with GAIL while significantly reducing the environment interactions. Recent advances in reinforcement learning (RL) have achieved super-human performance on several domains BID20 BID21. On most of such domains with the success of RL, the design of reward, that explains what agent's behavior is favorable, is obvious for humans. Conversely, on domains where it is unclear how to design the reward, agents trained by RL algorithms often obtain poor policies and behave worse than what we expect them to do. Imitation learning (IL) comes in such cases. The goal of IL is to enable the learner to imitate expert behavior given the expert demonstrations without the reward signal. We are interested in IL because we desire an algorithm that can be applied to real-world problems for which it is often hard to design the reward. In addition, since it is generally hard to model a variety of real-world environments with an algorithm, and the state-action pairs in a vast majority of realworld applications such as robotics control can be naturally represented in continuous spaces, we focus on model-free IL for continuous control. A wide variety of IL methods have been proposed in the last few decades. The simplest IL method among those is behavioral cloning (BC) BID23 which learns an expert policy in a supervised fashion without environment interactions during training. BC can be the first IL option when enough demonstration is available. However, when only a limited number of demonstrations are available, BC often fails to imitate the expert behavior because of the problem which is referred to compounding error BID25 -inaccuracies compound over time and can lead the learner to encounter unseen states in the expert demonstrations. Since it is often hard to obtain a large number of demonstrations in real-world environments, BC is often not the best choice for real-world IL scenarios. Another widely used approach, which overcomes the compounding error problem, is Inverse Reinforcement Learning (IRL) BID27 BID22 BID0 BID33. Recently, BID15 have proposed generative adversarial imitation learning (GAIL) which is based on prior IRL works. Since GAIL has achieved state-of-the-art performance on a variety of continuous control tasks, the adversarial IL (AIL) framework has become a popular choice for IL BID1 BID11 BID16. It is known that the AIL methods are more sample efficient than BC in terms of the expert demonstration. However, as pointed out by BID15, the existing AIL methods have sample complexity in terms of the environment interaction. That is, even if enough demonstration is given by the expert before training the learner, the AIL methods require a large number of state-action pairs obtained through the interaction between the learner and the environment 1. The sample complexity keeps existing AIL from being employed to real-world applications for two reasons. First, the more an AIL method requires the interactions, the more training time it requires. Second, even if the expert safely demonstrated, the learner may have policies that damage the environments and the learner itself during training. Hence, the more it performs the interactions, the more it raises the possibility of getting damaged. For the real-world applications, we desire algorithms that can reduce the number of interactions while keeping the imitation capability satisfied as well as the existing AIL methods do. The following three properties of the existing AIL methods which may cause the sample complexity in terms of the environment interactions:(a) Adopting on-policy RL methods which fundamentally have sample complexity in terms of the environment interactions.(b) Alternating three optimization processes -learning reward functions, value estimation with learned reward functions, and RL to update the learner policy using the estimated value. In general, as the number of parameterized functions which are related to each other increases, the training progress may be unstable or slower, and thus more interactions may be performed during training.(c) Adopting Gaussian policy as the learner's stochastic policy, which has infinite support on a continuous action space. In common IL settings, we observe action space of the expert policy from the demonstration where the expert action can take on values within a bounded (finite) interval. As BID3 suggests, the policy which can select actions outside the bound may slow down the training progress and make the problem harder to solve, and thus more interactions may be performed during training. In this paper, we propose an IL algorithm for continuous control to improve the sample complexity of the existing AIL methods. Our algorithm is made up mainly three changes to the existing AIL methods as follows:(a) Adopting off-policy actor-critic (Off-PAC) algorithm BID5 to optimize the learner policy instead of on-policy RL algorithms. Off-policy learning is commonly known as the promising approach to improve the complexity.(b) Estimating the state-action value using off-policy samples without learning reward functions instead of using on-policy samples with the learned reward functions. Omitting the reward learning reduces functions to be optimized. It is expected to make training progress stable and faster and thus reduce the number of interactions during training.(c) Representing the stochastic policy function of which outputs are bounded instead of adopting Gaussian policy. Bounding action values may make the problem easier to solve and make the training faster, and thus reduce the number of interactions during training. Experimental show that our algorithm enables the learner to imitate the expert behavior as well as GAIL does while significantly reducing the environment interactions. Ablation experimental show that (a) adopting the off-policy scheme requires about 100 times fewer environment interactions to imitate the expert behavior than the one on-policy IL algorithms require, (b) omitting the reward learning makes the training stable and faster, and (c) bounding action values makes the training faster. We consider a Markov Decision Process (MDP) which is defined as a tuple {S, A, T, R, d 0, γ}, where S is a set of states, A is a set of possible actions agents can take, T: S×A×S → is a transition probability, R: S×A → R is a reward function, d 0: S → is a distribution over initial states, and γ ∈ is a discount factor. The agent's behavior is defined by a stochastic policy π: S×A → and Π denotes a set of the stochastic policies. We denote S E ⊂ S and A E ⊂ A as sets of states and actions observed in the expert demonstration, and S π ⊂ S and A π ⊂ A as sets of those observed in rollouts following a policy π. We will use π E, π θ, β ∈ Π to refer to the expert policy, the learner policy parameterized by θ, and a behavior policy, respectively. Given a policy π, performance measure of π is defined as J (π, R) = E ∞ t=0 γ t R(s t, a t)|d 0, T, π where s t ∈ S is a state that the agent receives at discrete time-step t, and a t ∈ A is an action taken by the agent after receiving s t. The performance measure indicates expectation of the discounted return ∞ t=0 γ t R(s t, a t) when the agent follows the policy π in the MDP. Using discounted state visitation distribution denoted by ρ π (s) = ∞ t=0 γ t P(s t = s|d 0, T, π) where P is a probability that the agent receives the state s at time-step t, the performance measure can be rewritten as J (π, R) = E s∼ρπ,a∼π R(s, a). The state-action value function for the agent following π is defined as DISPLAYFORM0 |T, π, and Q π,ν denotes its approximator parameterized by ν. We briefly describe objectives of RL, IRL, and AIL below. We refer the readers to BID15 for details. The goal of RL is to find an optimal policy that maximizes the performance measure. Given the reward function R, the objective of RL with parameterized stochastic policies π θ: S×A → is defined as follows: DISPLAYFORM0 The goal of IRL is to find a reward function based on an assumption that the discounted returns earned by the expert behavior are greater than or equal to those earned by any non-experts behavior. Technically, the objective of IRL is to find reward functions R ω: S × A → R parameterized by ω that satisfies J (π E, R ω) ≥ J (π, R ω) where π denotes the non-expert policy. The existing AIL methods adopt max-margin IRL BID0 of which objective can be defined as follows: DISPLAYFORM1 2) The objective of AIL can be defined as a composition of the objectives and as follows: DISPLAYFORM2 The objective of Off-PAC to train the learner can be described as follows: DISPLAYFORM0 The learner policy is updated by taking the gradient of the state-action value. BID5 proposed the gradient as follows: provided another formula of the gradient using "re-parameterization trick" in the case that the learner policy selects the action as a = π θ (s, z) with random variables z ∼ P z generated by a distribution P z: DISPLAYFORM1 DISPLAYFORM2 3 ALGORITHMAs mentioned in Section.1, our algorithm (a) adopts Off-PAC algorithms to train the learner policy, (b) estimates state-action value without learning the reward functions, and (c) represents the stochastic policy function so that its outputs are bounded. In this section, we first introduce (b) in 3.1 and describe (c) in 3.2, then present how to incorporate (b) and (c) into (a) in 3.3. In this subsection, we introduce a new IRL objective to learn the reward function in 3.1.1 and a new objective to learn the value function approximator in 3.1.2. Then, we show that combining those objectives derives a novel objective to learn the value function approximator without reward learning in 3.1.3. We define the parameterized reward function as R ω (s, a) = log r ω (s, a), with a function r ω: S×A → parameterized by ω. r ω (s, a) represents a probability that the state-action pairs (s, a) belong to S E × A E. In other words, r ω (s, a) explains how likely the expert executes the action a at the state s. With this reward, we can also define a Bernoulli distribution p ω: Π×S×A → such that p ω (π E |s, a) = r ω (s, a) for the expert policy π E and p ω (π|s, a) = 1 − r ω (s, a) for any other policies π ∈ Π \ {π E} which include π θ and β. A nice property of this definition of the reward is that the discounted return for a trajectory {(s 0, a 0), (s 1, a 1),...} can be written as a log likelihood with p ω (π E |s t, a t): DISPLAYFORM0 Here, we assume Markov property in terms of p ω such that p ω (π E |s t, a t) for t ≥ 1 is independent of p ω (π E |s t−u, a t−u) for u ∈ {1, ..., t}. Under this assumption, the return naturally represents how likely a trajectory is the one the expert demonstrated. The discount factor γ plays a role to make sure the return is finite as in standard RL.The IRL objective can be said to aim at assigning r ω = 1 for state-action pairs (s, a) ∈ S E × A E and r ω = 0 for (s, a) ∈ S π × A π when the same definition of the reward R ω (s, a) = log r ω (s, a) is used. Following this fashion easily leads to a problem where the return earned by the non-expert policy becomes −∞, since log r ω (s, a) = −∞ if r ω (s, a) = 0 and thus log DISPLAYFORM1 The existing AIL methods seem to mitigate this problem by trust region optimization for parameterized value function approximator BID29, and it works somehow. However, we think this problem should be got rid of in a fundamental way. We propose a different approach to evaluate state-action pairs (s, a) ∈ S π × A π. Intuitively, the learner does not know how the expert behaves in the states s ∈ S \ S E -that is, it is uncertain which actions the expert executes in the states the expert has not visited. We thereby define a new IRL objective as follows: DISPLAYFORM2 where H denotes entropy of Bernoulli distribution such that: DISPLAYFORM3 Unlike the existing AIL methods, our IRL objective is to assign p ω (π E |s, a) = p ω (π|s, a) = 0.5 for (s, a) ∈ S π × A π. This uncertainty p ω (π E |s, a) = 0.5 explicitly makes the return earned by the non-expert policy finite. On the other hand, the objective is to assign r ω = 1 for (s, a) ∈ S E × A E as do the existing AIL methods. The optimal solution for the objective satisfies the assumption of IRL: DISPLAYFORM4, even though the objective does not aim at discriminating between (s, a) ∈ S E × A E and (s, a) ∈ S π × A π, As we see in Equation FORMULA7, the discounted return can be represented as a log likelihood. Therefore, a value function approximator Q π θ following the learner policy π θ can be formed as a log probability. We introduce a function q π θ,ν: S×A → parameterized by ν to represent the approximator Q π θ,ν as follows: DISPLAYFORM0 The optimal value function following a policy π satisfies the Bellman equation Q π (s t, a t) = R(s t, a t) + γE st+1∼T,at+1∼π Q π (s t+1, a t+1). Substituting π θ for π, log r ω (s t, a t) for R(s t, a t), and log q π θ,ν (s t, a t) for Q π (s t, a t), the Bellman equation for the learner policy π θ can be written as follows: DISPLAYFORM1 We introduce additional Bernoulli distributions P ν: Π × S × A:→ and P ωνγ: Π × S × A × S × A:→ as follows: DISPLAYFORM2 Using P ν and P ωνγ, the loss to satisfy Equation FORMULA13 can be rewritten as follows: DISPLAYFORM3 We use Jensen's inequality with the concave property of logarithm in Equation FORMULA4. Now we see that the loss L(ω, ν, θ) is bounded by the log likelihood ratio between the two Bernoulli distributions P ν and P ωνγ, and L(ω, ν, θ) = 0 if P ν (π E |s t, a t) = E st+1∼T,at+1∼π θ P ωνγ (π E |s t, a t, s t+1, a t+1).In the end, learning the approximator Q π θ,ν turns out to be matching the two Bernoulli distributions. A natural way to measure the difference between two probability distributions is divergence. We choose Jensen-Shannon (JS) divergence to measure the difference because we empirically found it works better, and thereby the objective to optimize Q π θ,ν can be written as follows: DISPLAYFORM4 where D JS denotes JS divergence between two Bernoulli distributions. Suppose the optimal reward function R ω * (s, a) = log r ω * (s, a) for the objective can be obtained, the Bellman equation FORMULA13 can be rewritten as follows: DISPLAYFORM0 Recall that IRL objective aims at assigning r ω * (s t, a t) = 1 for (s t, a t) ∈ S E × A E and r ω * (s t, a t) = 0.5 for (s t, a t) ∈ S π × A π where π ∈ Π \ {π E}. Therefore, the objective FORMULA9 is rewritten as the following objective using the Bellman equation FORMULA13: DISPLAYFORM1 Thus, r ω * can be obtained by the Bellman equation FORMULA6 as long as the solution for the objective can be obtained. We optimize q π θ,ν (s t, a t) in the same way of objective as follows: DISPLAYFORM2 We use P γ ν instead of P ωνγ in objective unlike the objective. Thus, we omit reward learning that the existing AIL methods require, while learning q π θ,ν (s t, a t) to obtain r ω *. Initialize time-step t = 0 and receive initial state s0 5:while not terminate condition do 6:Execute an action at = π θ (st, z) with z ∼ Pz and observe new state st+1 7:Store a state-action triplet (st, at, st+1) in B β. 8: DISPLAYFORM3 end while 10:for u = 1, t do 11:Sample mini-batches of triplets (s t ′ +1). end for 16: end for Recall that the aim of IL is to imitate the expert behavior. It can be summarized that IL attempts to obtain a generative model the expert has over A conditioned on states in S. We see that the aim itself is equivalent to that of conditional generative adversarial networks (cGANs) BID19. The generator of cGANs can generate stochastic outputs of which range are bounded. As mentioned in Section 1, bounding action values is expected to make the problem easier to solve and make the training faster. In the end, we adopt the form of the conditional generator to represent the stochastic learner policy π θ (s, z). The typical Gaussian policy and the proposed policy representations with neural networks are described in FIG0 Algorithm.1 shows the overview of our off-policy actor-critic imitation learning algorithm. To learn the value function approximator Q π θ,ν, we adopt a behavior policy β as π in the second term in objective We employ a mixture of the past learner policies as β and a replay buffer B β BID20 ) to perform sampling s t ∼ ρ π, a t ∼ π and s t+1 ∼ T. The buffer B β is a finite cache and stores the (s t, a t, s t+1) triplets in a first-in-first-out manner while the learner interacts with the environment. The approximator Q π θ,ν (s t, a t) = log q π θ,ν (s t, a t) takes (−∞, 0]. With the approximator, using the gradient to update the learner policy always punish (or ignore) the learner's actions. Instead, we adopt the gradient which directly uses Jacobian of Q π θ,ν.As do off-policy RL methods such as BID20 and, we use the target value function approximator, of which parameters are updated to track ν, to optimize Q π θ,ν. We update Q π θ,ν and π θ at the end of each episode rather than following each step of interaction. In recent years, the connection between generative adversarial networks (GAN) BID10 and IL has been pointed out BID15 BID6. BID15 show that IRL is a dual problem of RL which can be deemed as a problem to match the learner's occupancy measure BID31 to that of the expert, and that a choice of regularizer for the cost function yields an objective which is analogous to that of GAN. Their algorithm, namely GAIL, has become a popular choice for IL and some extensions of GAIL have been proposed BID1 BID11 BID16. However, those extensions have never addressed reducing the number of interactions during training. There has been a few attempts that try to improve the sample complexity in IL literatures, such as Guided Cost Learning (GCL) BID7. However, those methods have worse imitation capability in comparison with GAIL, as reported by BID8. As detailed in section 5, our algorithm have comparable imitation capability to GAIL while improving the sample complexity. Hester & Osband FORMULA7 proposed an off-policy algorithm using the expert demonstration. They address problems where both demonstration and hand-crafted rewards are given. Whereas, we address problems where only the expert demonstration is given. There is another line of IL works where the learner can ask the expert which actions should be taken during training, such as DAgger BID24, SEARN (Daumé &), SMILe BID25, and AggreVaTe BID26. As opposed to those methods, we do not suppose that the learner can query the expert during training. In our experiments, we aim to answer the following three questions: Q1. Can our algorithm enable the learner to imitate the expert behavior? Q2. Is our algorithm more sample efficient than BC in terms of the expert demonstration? Q3. Is our algorithm more efficient than GAIL in terms of the training time? To answer the questions above, we use five physics-based control tasks that are simulated with MuJoCo physics simulator BID32. See Appendix A for the description of each task. In the experiments, we compare the performance of our algorithm, BC, GAIL, and GAIL initialized by BC 23. The implementation details can be found in Appendix B. We train an agent on each task by TRPO BID28 using the rewards defined in the OpenAI Gym BID2, then we use the ing agent with a stochastic policy as the expert for the IL algorithms. We store (s t, a t, s t+1) triplets during the expert demonstration, then the triplets are used as training samples in the IL algorithms. In order to study the sample efficiency of the IL algorithms, we arrange two setups. The first is sparse sampling setup, where we randomly sample 100 (s t, a t, s t+1) triplets from each trajectory which contains 1000 triplets. Then we perform the IL algorithms using datasets that consist of several 100s triplets. Another setup is dense sampling setup, where we use full (s t, a t, s t+1) triplets in each trajectory, then train the learner using datasets that consist of several trajectories. If an IL algorithm succeeds to imitate the expert behavior in the dense sampling setup whereas it fails in the sparse sampling setup, we evaluate the algorithm as sample inefficient in terms of the expert demonstration. The performance of the experts and the learners are measured by cumulative reward they earned in a trajectory. We run three experiments on each task, and measure the performance during training. Figure 2 shows the experimental in both sparse and dense sampling setup. In comparison with GAIL, our algorithm marks worse performance on Walker2d-v1 and Humanoid-v1 with the datasets of the smallest size in sparse sampling setup, better performance on Ant-v1 in both setups, and competitive performance on the other tasks in both setups. Overall, we conclude that our algorithm is competitive with GAIL with regards to performance. That is, our algorithm enables the learner to imitate the expert behavior as well as GAIL does. BC imitates the expert behavior successfully on all tasks in the dense sampling setup. However, BC often fails to imitate the expert behavior in the sparse sampling setup with smaller datasets. Our algorithm achieves better performance than BC does all over the tasks. It shows that our algorithm is more sample efficient than BC in terms of the expert demonstration. Figure 3 shows the performance plot curves over validation rollouts during training in the sparse sampling setup. The curves on the top row in Figure 3 show that our algorithm denoted by Ours trains the learner more efficiently than GAIL does in terms of training time. In addition, the curves on the bottom row in Figure 3 show that our algorithm trains the learner much more efficiently than GAIL does in terms of the environment interaction. As opposed to BID15 suggestion, GAIL initialized by BC (BC+GAIL) does not improve the sample efficiency, but rather harms the leaner's performance significantly. We conducted additional ablation experiments to demonstrate that our proposed method described in Section.3 improves the sample efficiency. FIG2 shows the ablation experimental on Antv1 task. Ours+OnP, which denotes an on-policy variant of Ours, requires 100 times more interactions than Ours. The with Ours+OnP suggests that adopting off-policy learning scheme instead of on-policy one significantly improves the sample efficiency. Ours+IRL(D) and Ours+IRL(E) are variants of Ours that learn value function approximators using the learned reward function with the objective and FORMULA9, respectively. The with Ours+IRL(D) and Ours+IRL(E) suggests that omitting the reward learning described in 3.1 makes the training stable and faster. The with Ours+GP, which denotes a variant of Ours that adopts the Gaussian policy, suggests that bounding action values described in 3.2 makes the training faster and stable. The with Ours+DP, which denotes a variant of Ours that has a deterministic policy with fixed input noises, fails to imitate the expert behavior. It shows that the input noise variable z in our algorithm plays a role to obtain stochastic policies. In this paper, we proposed a model-free IL algorithm for continuous control. Experimental showed that our algorithm achieves competitive performance with GAIL while significantly reducing the environment interactions. A DETAILED DESCRIPTION OF EXPERIMENT TAB0 summarizes the description of each task, the performance of an agent with random policy, and the performance of the experts. We implement our algorithm using two neural networks with two hidden layers. Each network represents π θ and q ν. For convenience, we call those networks for π θ and q ω as policy network (PN) and Q-network (QN), respectively. PN has 100 hidden units in each hidden layer, and its final output is followed by hyperbolic tangent nonlinearity to bound its action range. QN has 500 hidden units in each hidden layer and a single output is followed by sigmoid nonlinearity to bound the output between. All hidden layers are followed by leaky rectified nonlinearity BID18. The parameters in all layers are initialized by Xavier initialization BID9. The input of PN is given by concatenated vector representations for the state s and noise z. The noise vector, of which dimensionality corresponds to that of the state vector, generated by zero-mean normal distribution so that z ∼ P z = N. The input of QN is given by concatenated vector representations for the state s and action a. We employ RMSProp BID14 for learning parameters with a decay rate 0.995 and epsilon 10 −8. The learning rates are initially set to 10 −3 for QN and 10 −4 for PN, respectively. The target QN with parameters ν ′ are updated so that ν ′ = 10 −3 * ν+(1−10 −3) * ν ′ at each update of ν. We linearly decrease the learning rates as the training proceeds. We set minibatch size of (s t, a t, s t+1) triplets 64, the replay buffer size |B β | = 15000, and the discount factor γ = 0.85. We sample 128 noise vectors for calculating empirical expectation E z∼Pz of the gradient. We use publicly available code (https://github.com/openai/imitation) for the implementation of GAIL and BC. Note that, the number of hidden units in PN is the same as that of networks for GAIL. All experiments are run on a PC with a 3.30 GHz Intel Core i7-5820k Processor, a GeForce GTX Titan GPU, and 32GB of RAM.' and the positive 'In this paper, we proposed a model-free, off-policy IL algorithm for continuous control. Experimental results showed that our algorithm achieves competitive results with GAIL while significantly reducing the environment interactions.' and the negative 'Parameter space noise allows reinforcement learning algorithms to explore by perturbing parameters instead of actions, often leading to significantly improved exploration performance.'

Question: It has long been known that a single-layer fully-connected neural network with an i.i.d. prior over its parameters is equivalent to a Gaussian process (GP), in the limit of infinite network width. This correspondence enables exact Bayesian inference for infinite width neural networks on regression tasks by means of evaluating the corresponding GP. Recently, kernel functions which mimic multi-layer random neural networks have been developed, but only outside of a Bayesian framework. As such, previous work has not identified that these kernels can be used as covariance functions for GPs and allow fully Bayesian prediction with a deep neural network. In this work, we derive the exact equivalence between infinitely wide, deep, networks and GPs with a particular covariance function. We further develop a computationally efficient pipeline to compute this covariance function. We then use the ing GP to perform Bayesian inference for deep neural networks on MNIST and CIFAR-10. We observe that the trained neural network accuracy approaches that of the corresponding GP with increasing layer width, and that the GP uncertainty is strongly correlated with trained network prediction error. We further find that test performance increases as finite-width trained networks are made wider and more similar to a GP, and that the GP-based predictions typically outperform those of finite-width networks. Finally we connect the prior distribution over weights and variances in our GP formulation to the recent development of signal propagation in random neural networks. Deep neural networks have emerged in recent years as flexible parametric models which can fit complex patterns in data. As a contrasting approach, Gaussian processes have long served as a traditional nonparametric tool for modeling. An equivalence between these two approaches was derived in BID17, for the case of one layer networks in the limit of infinite width. Neal (1994a) further suggested that a similar correspondence might hold for deeper networks. Consider a deep fully-connected neural network with i.i.d. random parameters. Each scalar output of the network, an affine transformation of the final hidden layer, will be a sum of i.i.d. terms. As we will discuss in detail below, in the limit of infinite width the Central Limit Theorem 1 implies that the function computed by the neural network (NN) is a function drawn from a Gaussian process (GP). In the case of single hidden-layer networks, the form of the kernel of this GP is well known BID17 BID25 ).This correspondence implies that if we choose the hypothesis space to be the class of infinitely wide neural networks, an i.i.d. prior over weights and biases can be replaced with a corresponding GP prior over functions. As noted by BID25, this substitution enables exact Bayesian inference for regression using neural networks. The computation requires building the necessary covariance matrices over the training and test sets and straightforward linear algebra computations. In light of the resurgence in popularity of neural networks, it is timely to revisit this line of work. We delineate the correspondence between deep and wide neural networks and GPs and utilize it for Bayesian training of neural networks on regression tasks. Our work touches on aspects of GPs, Bayesian learning, and compositional kernels. The correspondence between infinite neural networks and GPs was first noted by BID17 b). BID25 computes analytic GP kernels for single hidden-layer neural networks with error function or Gaussian nonlinearities and noted the use of the GP prior for exact Bayesian inference in regression. BID6 discusses several routes to building deep GPs and observes the degenerate form of kernels that are composed infinitely many times -a point we will return to Section 3.2 -but they do not derive the form of GP kernels as we do. BID10 also discusses constructing kernels equivalent to infinitely wide deep neural networks, but their construction does not go beyond two hidden layers with nonlinearities. Related work has also appeared outside of the GP context but in compositional kernel constructions. BID2 derives compositional kernels for polynomial rectified nonlinearities, which includes the Sign and ReLU nonlinearities, and can be used in GPs; our manner of composing kernels matches theirs, though the context is different. BID4 extends the construction of compositional kernels to neural networks whose underlying directed acyclic graph is of general form. They also prove, utilizing the formalism of dual activations, that compositional kernels originating from fully-connected topologies with the same nonlinearity become degenerate when composed infinitely many times. In a different context than compositional kernels, BID19; BID24 study the same underlying recurrence relation for the specific case of fully-connected networks and bounded nonlinearities. They distinguish regions in hyperparameter space with different fixed points and convergence behavior in the recurrence relations. The focus in these works was to better understand the expressivity and trainability of deep networks. Drawing inspiration from the multi-layer nature of deep neural networks, there is a line of work considering various approaches to stacking GPs, such as deep GPs BID15; BID3; BID11; BID6; BID1 ), which can give rise to a richer class of probabilistic models beyond GPs. This contrasts with our work, where we study GPs that are in direct correspondence with deep, infinitely wide neural networks. BID14 has recently explored the performance of GP models with deep kernels given in BID2, implemented with scalable approximations. However, they do not discuss the equivalence between deep neural networks and GPs with compositional kernels, which constitutes a conceptual contribution of our work. Furthermore, we note that the GP kernels in our work are more general than the compositional kernel construction outlined in BID2 in two respects: (i) we are not limited to rectified polynomials but can deal with general nonlinearities, and (ii) we consider two additional hyperparameters in the kernels, which would correspond to the weight and bias parameter variances in a neural network. Finally, BID8 connects dropout in deep neural networks with approximate Bayesian inference in deep GPs. Another series of recent works BID28 a); BID0 ), termed deep kernel learning, utilize GPs with base kernels which take in features produced by a deep multilayer neural network, and train the ing model end-to-end. Our work differs from these in that our GP corresponds to a multilayer neural network. Additionally, our GP kernels have many fewer parameters, and these parameters correspond to the hyperparameters of the equivalent neural network. We begin by specifying the form of a GP which corresponds to a deep, infinitely wide neural network -hereafter referred to as the Neural Network GP (NNGP) -in terms of a recursive, deterministic computation of the kernel function. The prescription is valid for generic pointwise nonlinearities in fully-connected feedforward networks. We develop a computationally efficient method (Section 2.5) to compute the covariance function corresponding to deep neural networks with fixed hyperparameters. In this work, as a first proof of concept of our NNGP construction, we focus on exact Bayesian inference for regression tasks, treating classification as regression on class labels. While less principled, least-squares classification performs well BID23 and allows us to compare exact inference via a GP to prediction by a trained neural network on well-studied tasks (MNIST and CIFAR-10 classification). Note that it is possible to extend GPs to softmax classification with cross entropy loss BID26; BID21 ), which we aim to investigate in future work. We conduct experiments making Bayesian predictions on MNIST and CIFAR-10 (Section 3) and compare against NNs trained with standard gradient-based approaches. The experiments explore different hyperparameter settings of the Bayesian training including network depth, nonlinearity, training set size (up to and including the full dataset consisting of tens of thousands of images), and weight and bias variance. Our experiments reveal that the best NNGP performance is consistently competitive against that of NNs trained with gradient-based techniques, and the best NNGP setting, chosen across hyperparameters, often surpasses that of conventional training (Section 3, TAB0). We further observe that, with increasing network width, the performance of neural networks with gradient-based training approaches that of the NNGP computation, and that the GP uncertainty is strongly correlated with prediction error. Furthermore, the performance of the NNGP depends on the structure of the kernel, which can be connected to recent work on signal propagation in networks with random parameters BID24. We begin by specifying the correspondence between GPs and deep, infinitely wide neural networks, which hinges crucially on application of the Central Limit Theorem. We review the single-hidden layer case (Section 2.2) before moving to the multi-layer case (Section 2.3). Consider an L-hidden-layer fully-connected neural network with hidden layers of width N l (for layer l) and pointwise nonlinearities φ. Let x ∈ R din denote the input to the network, and let z L ∈ R dout denote its output. The ith component of the activations in the lth layer, post-nonlinearity and postaffine transformation, are denoted x l i and z l i respectively. We will refer to these as the post-and pre-activations. (We let x 0 i ≡ x i for the input, dropping the Arabic numeral superscript, and instead use a Greek superscript x α to denote a particular input α). Weight and bias parameters for the lth layer have components W l ij, b l i, which are independent and randomly drawn, and we take them all to have zero mean and variances σ 2 w /N l and σ 2 b, respectively. GP(µ, K) denotes a Gaussian process with mean and covariance functions µ(·), K(·, ·), respectively. We briefly review the correspondence between single-hidden layer neural networks and GPs BID17 b); BID25 ). The ith component of the network output, z 1 i, is computed as, DISPLAYFORM0 where we have emphasized the dependence on input x. Because the weight and bias parameters are taken to be i.i.d., the post-activations x 1 j, x 1 j are independent for j = j. Moreover, since z 1 i (x) is a sum of i.i.d terms, it follows from the Central Limit Theorem that in the limit of infinite width N 1 → ∞, z 1 i (x) will be Gaussian distributed. Likewise, from the multidimensional Central Limit Theorem, any finite collection of {z DISPLAYFORM1} will have a joint multivariate Gaussian distribution, which is exactly the definition of a Gaussian process. Therefore we conclude that z DISPLAYFORM2, a GP with mean µ 1 and covariance K 1, which are themselves independent of i. Because the parameters have zero mean, we have that µ 1 (x) = E z 1 i (x) = 0 and, DISPLAYFORM3 where we have introduced C(x, x) as in BID17; it is obtained by integrating against the distribution of W 0, b 0. Note that, as any two z 1 i, z 1 j for i = j are joint Gaussian and have zero covariance, they are guaranteed to be independent despite utilizing the same features produced by the hidden layer. The arguments of the previous section can be extended to deeper layers by induction. We proceed by taking the hidden layer widths to be infinite in succession (N 1 → ∞, N 2 → ∞, etc.) as we continue with the induction, to guarantee that the input to the layer under consideration is already governed by a GP. In Appendix C we provide an alternative derivation in terms of Bayesian marginalization over intermediate layers, which does not depend on the order of limits, in the case of a Gaussian prior on the weights. A concurrent work BID5 further derives the convergence rate towards a GP if all layers are taken to infinite width simultaneously, but at different rates. Suppose that z l−1 j is a GP, identical and independent for every j (and hence x l j (x) are independent and identically distributed). After l − 1 steps, the network computes DISPLAYFORM0 As before, z l i (x) is a sum of i.i.d. random terms so that, as N l → ∞, any finite collection {z DISPLAYFORM1 By induction, the expectation in Equation 4 is over the GP governing z DISPLAYFORM2, but this is equivalent to integrating against the joint distribution of only z l−1 i (x) and z l−1 i (x). The latter is described by a zero mean, two-dimensional Gaussian whose covariance matrix has distinct entries DISPLAYFORM3, and K l−1 (x, x). As such, these are the only three quantities that appear in the . We introduce the shorthand DISPLAYFORM4 to emphasize the recursive relationship between K l and K l−1 via a deterministic function F whose form depends only on the nonlinearity φ. This gives an iterative series of computations which can be performed to obtain K L for the GP describing the network's final output. For the base case DISPLAYFORM5; we can utilize the recursion relating K 1 and K 0, where DISPLAYFORM6 In fact, these recurrence relations have appeared in other contexts. They are exactly the relations derived in the mean field theory of signal propagation in fully-connected random neural networks BID19 BID24 ) and also appear in the literature on compositional kernels BID2 BID4 ). For certain activation functions, Equation 5 can be computed analytically BID2 BID4 ). In the case of the ReLU nonlinearity, it yields the well-known arccosine kernel BID2 ) whose form we reproduce in Appendix B. When no analytic form exists, it can instead be efficiently computed numerically, as described in Section 2.5. Here we provide a short review of how a GP prior over functions can be used to do Bayesian inference; see e.g. BID21 for a comprehensive review of GPs. Given a dataset DISPLAYFORM0, we wish to make a Bayesian prediction at test point x * using a distribution over functions z(x). This distribution is constrained to take values z ≡ (z 1, ..., z n) on the training inputs x ≡ (x 1, ..., x n) and, DISPLAYFORM1 where t = (t 1, ..., t n) T are the targets on the training set, and P (t|z) corresponds to observation noise. We will assume a noise model consisting of a Gaussian with variance σ 2 centered at z. If the conditions of Section 2.2 or 2.3 apply, our choice of prior over functions implies that z 1,..., z n, z * are n + 1 draws from a GP and z *, z|x *, x ∼ N (0, K) is a multivariate Gaussian whose covariance matrix has the form DISPLAYFORM2 where the block structure corresponds to the division between the training set and the test point. DISPLAYFORM3 As is standard in GPs, the integral in Equation 7 can be done exactly, ing in z DISPLAYFORM4 where I n is the n × n identity. The predicted distribution for z * |D, x * is hence determined from straightforward matrix computations, yet nonetheless corresponds to fully Bayesian training of the deep neural network. The form of the covariance function used is determined by the choice of GP prior, i.e. the neural network model class, which depends on depth, nonlinearity, and weight and bias variances. We henceforth resume placing a superscript L as in K L to emphasize the choice of depth for the compositional kernel. Given an L-layer deep neural network with fixed hyperparameters, constructing the covariance matrix K L for the equivalent GP involves computing the Gaussian integral in Equation 4 for all pairs of training-training and training-test points, recursively for all layers. For some nonlinearities, such as ReLU, this integration can be done analytically. However, to compute the kernel corresponding to arbitrary nonlinearities, the integral must be performed numerically. The most direct implementation of a numerical algorithm for K L would be to compute integrals independently for each pair of datapoints and each layer. This is prohibitively expensive and costs O n 2 g L(n 2 train + n train n test), where n 2 g is the sampling density for the pair of Gaussian random variables in the 2D integral and n train, n test are the training and test set sizes, respectively. However, by careful pipelining, and by preprocessing all inputs to have identical norm, we can improve this cost to O n 2 g n v n c + L(n 2 train + n train n test), where n v and n c are sampling densities for a variance and correlation grid, as described below. In order to achieve this, we break the process into several steps:1. Generate: pre-activations u = [−u max, · · ·, u max] consisting of n g elements linearly spaced between −u max and u max; variances s = [0, · · ·, s max] with n v linearly spaced elements, where s max < u 2 max; and correlations c = (−1, · · ·, 1) with n c linearly spaced elements. Note that we are using fixed, rather than adaptive, sampling grids to allow operations to be parallelized and reused across datapoints and layers.2. Populate a matrix F containing a lookup table for the function F φ in Equation 5. This involves numerically approximating a Gaussian integral, in terms of the marginal variances s and correlations c. We guarantee that the marginal variance is identical for each datapoint, by preprocessing all datapoints to have identical norm at the input layer, so the number of entries in the lookup table need only be n v n c. These entries are computed as 2: DISPLAYFORM0 3. For every pair of datapoints x and x in layer l, compute K l (x, x) using Equation 5.Approximate the function DISPLAYFORM1 Step 2, where we interpolate into s using the value of K l−1 (x, x), and interpolate into c using DISPLAYFORM2, due to data preprocessing to guarantee constant norm.4. Repeat the previous step recursively for all layers. Bilinear interpolation has constant cost, so this has cost O L(n 2 train + n train n test).This computational recipe allows us to compute the covariance matrix for the NNGP corresponding to any well-behaved nonlinearity φ. All computational steps above can be implemented using accelerated tensor operations, and computation of K L is typically faster than solving the system of linear equations in Equation 8-9. Figure 6 illustrates the close agreement between the kernel function computed numerically (using this approach) and analytically, for the ReLU nonlinearity. It also illustrates the angular dependence of the kernel and its evolution with increasing depth. Finally, note that the full computational pipeline is deterministic and differentiable. The shape and properties of a deep network kernel are purely determined by hyperparameters of the deep neural network. Since GPs give exact marginal likelihood estimates, this kernel construction may allow principled hyperparameter selection, or nonlinearity design, e.g. by gradient ascent on the log likelihood w.r.t. the hyperparameters. Although this is not the focus of current work, we hope to return to this topic in follow-up work. An open source implementation of the algorithm is available at https://github.com/brainresearch/nngp. We compare NNGPs with SGD 3 trained neural networks on the permutation invariant MNIST and CIFAR-10 datasets. The baseline neural network is a fully-connected network with identical width at each hidden layer. Training is on the mean squared error (MSE) loss, chosen so as to allow direct comparison to GP predictions. Formulating classification as regression often leads to good BID22. Future work may involve evaluating the NNGP on a cross entropy loss using the approach in BID26 BID21. Training used the Adam optimizer BID13 ) with learning rate and initial weight/bias variances optimized over validation error using the Google Vizier hyperparameter tuner BID9. Dropout was not used. In future work, it would be interesting to incorporate dropout into the NNGP covariance matrix using an approach like that in BID24. For the study, nonlinearities were chosen to be either rectified linear units (ReLU) or hyperbolic tangent (Tanh). Class labels were encoded as a one-hot, zero-mean, regression target (i.e., entries of -0.1 for the incorrect class and 0.9 for the correct class). We constructed the covariance kernel numerically for ReLU and Tanh nonlinearities following the method described in Section 2.5.Performance: We find that the NNGP often outperforms trained finite width networks. See TAB0 and FIG0. The NNGP often outperforms finite width networks, and neural network performance more closely resembles NNGP performance with increasing width. Test accuracy and mean squared error on MNIST and CIFAR-10 dataset are shown for the best performing NNGP and best performing SGD trained neural networks for given width.' NN-best' denotes the best performing (on the validation set) neural network across all widths and trials. Often this is the neural network with the largest width. We additionally find the performance of the best finite-width NNs, trained with a variant of SGD, approaches that of the NNGP with increasing layer width. This is interesting from at least two, potentially related, standpoints. NNs are commonly believed to be powerful because of their ability to do flexible representation learning, while our NNGP uses fixed basis functions; nonetheless, in our experiments we find no salient performance advantage to the former. It hints at a possible relationship between SGD and Bayesian inference in certain regimes -were the neural networks trained in a fully Bayesian fashion, rather than by SGD, the approach to NNGP in the large width limit would be guaranteed. There is recent work suggesting that SGD can implement approximate Bayesian inference BID16 under certain assumptions. The similarity of the performance of the widest NN in FIG0 with the NNGP suggests that the limit of infinite network width, which is inherent to the GP, is far from being a disadvantage. Indeed, in practice it is found that the best generalizing NNs are in fact the widest. To support this, in FIG1 we show generalization gap from an experiment in which we train 180 fully-connected networks with five hidden layers on CIFAR-10 with a range of layer widths. For this experiment, we trained the networks using a standard cross entropy loss rather than MSE, leading to a slight difference in performance. Uncertainty: One benefit in using a GP is that, due to its Bayesian nature, all predictions have uncertainty estimates (Equation 9). For conventional neural networks, capturing the uncertainty in a model's predictions is challenging BID7. In the NNGP, every test point has an explicit estimate of prediction variance associated with it (Equation 9). In our experiments, we observe that the NNGP uncertainty estimate is highly correlated with prediction error (Figure 3). commonly approach a functionally uninteresting fixed point with depth l → ∞, in that K ∞ (x, x) becomes a constant or piecewise constant map. We now briefly relate our ability to train NNGPs with the convergence of K l (x, x) to the fixed-point kernel. We will be particularly interested in contextualizing our in relation to BID19 For the Tanh nonlinearity, there are two distinct phases respectively called the "ordered" phase and the "chaotic" phase that can be understood as a competition between the weights and the biases of the network. A diagram showing these phases and the boundary between them is shown in Figure 4a. In the ordered phase, the features obtained by propagating an input through the each layer of the recursion become similar for dissimilar inputs. Fundamentally, this occurs because the different inputs share common bias vectors and so all inputs end up just approaching the random bias. In this case the covariance K l (x, x) → q * for every pair of inputs x, x, where q * is a constant that depends only on σ 2 w and σ 2 b. All inputs have unit correlation asymptotically with depth. By contrast in the chaotic phase the weight variance σ 2 w dominates and similar inputs become dissimilar with depth as they are randomly projected by the weight matrices. In this case, the covariance K l (x, x) → q * for x = x but q * c * for x = x. Here c * < 1 is the fixed point correlation. In each of these regimes, there is also a finite depth-scale ξ which describes the characteristic number of layers over which the covariance function decays exponentially towards its fixed point form. Exactly at the boundary between these two regimes is a line in (σ FORMULA0 that this approach to the fixed-point covariance fundamentally bounded whether or not neural networks could successfully be trained. It was shown that initializing networks on this line allowed for significantly deeper neural networks to be trained. For ReLU networks a similar picture emerges, however there are some subtleties due to the unbounded nature of the nonlinearity. In this case for all σ 2 w and σ 2 b, K ∞ (x, x) = q * for all x, x and every point becomes asymptotically correlated. Despite this, there are again two phases: a "bounded" phase in which q * is finite (and nonzero) and an unbounded phase in which q * is either infinite or zero. As in the Tanh case there are depth scales that control the rate of convergence to these fixed points and therefore limit the maximum trainable depth. The phase diagram for the ReLU nonlinearity is also shown in Figure 4b.In a striking analogy with the trainability of neural networks, we observe that the performance of the NNGP appears to closely track the structure from the phase diagram, clearly illustrated in Figure 4. Indeed, we see that as for hyperparameter settings that are far from criticality, the GP is unable to train and we encounter poor test set performance. By contrast, near criticality we observe that our models display high accuracy. Moreover, we find that the accuracy appears to drop more quickly away from the phase boundary with increase in depth L of the GP kernel, K L. To understand this effect we note that information about data will be available to our model only through the difference DISPLAYFORM0 However, as the depth gets larger, this difference becomes increasingly small and at some point can no longer be represented due to numerical precision. At this point our test accuracy begins to quickly degrade to random chance. By harnessing the limit of infinite width, we have specified a correspondence between priors on deep neural networks and Gaussian processes whose kernel function is constructed in a compositional, but fully deterministic and differentiable, manner. Use of a GP prior on functions enables exact Bayesian inference for regression from matrix computations, and hence we are able to obtain predictions and uncertainty estimates from deep neural networks without stochastic gradient-based training. The performance is competitive with the best neural networks (within specified class of fully-connected models) trained on the same regression task under similar hyperparameter settings. While we were able to run experiments for somewhat large datasets (sizes of 50k), we intend to look into scalability for larger learning tasks, possibly harnessing recent progress in scalable GPs BID20; BID12 ). In our experiments, we observed the performance of the optimized neural network appears to approach that of the GP computation with increasing width. Whether gradient-based stochastic optimization implements an approximate Bayesian computation is an interesting question BID16. Further investigation is needed to determine if SGD does approximately implement Bayesian inference under the conditions typically employed in practice. Additionally, the NNGP provides explicit estimates of uncertainty. This may be useful in predicting model failure in critical applications of deep learning, or for active learning tasks where it can be used to identify the best datapoints to hand label. A DRAWS FROM AN NNGP PRIOR FIG5 illustrates the nature of the GP prior for the ReLU nonlinearity by depicting samples of 1D functions z(x) drawn from a ReLU GP, GP(0, K L), with fixed depth L = 10 and (σ Figure 6: The angular structure of the kernel and its evolution with depth. Also illustrated is the good agreement between the kernel computed using the methods of Section 2.5 (blue, starred) and the analytic form of the kernel (red). The depth l in K l runs from l = 0,..., 9 (flattened curves for increasing l), and (σ In the main text, we noted that the recurrence relation Equation 5 can be computed analytically for certain nonlinearities. In particular, this was computed in BID2 for polynomial rectified nonlinearities. For ReLU, the including the weight and bias variance is DISPLAYFORM0 To illustrate the angular form of K l (x, x) and its evolution with l, in Figure 6 we plot K l (θ) for the ReLU nonlinearity, where θ is the angle between x and x with norms such that ||x|| 2 = ||x || 2 = d in. We observe a flattening of the angular structure with increase in depth l, as predicted from the understanding in Section 3.2. Simultaneously, the figure also illustrates the good agreement between the kernel computed using the numerical implementation of Section 2.5 (blue, starred) and the analytic arccosine kernel, Equation 11 (red), for a particular choice of hyperparameters (σ In this section, we present an alternate derivation of the equivalence between infinitely wide deep neural networks and Gaussian process by marginalization over intermediate layers. For this derivation, we take the weight and bias parameters to be drawn from independent Gaussians, with zero mean and appropriately scaled variance. We are interested in finding the distribution p(z L |x) over network outputs z L ∈ R dout×B, conditioned on network inputs x ∈ R din×B, for input dimensionality d in, output dimensionality d out, and dataset size B. Intervening layers will have width N l, z l ∈ R N l+1 ×B for L > l > 0. We define the second moment matrix (here post-nonlinearity) for each layer l to be DISPLAYFORM0 Our approach is to think of intermediate random variables corresponding to these second moments defined above. By definition, K l only depends on z l−1. In turn, the pre-activations z l are described by a Gaussian process conditioned on the second moment matrix K l, DISPLAYFORM1 where DISPLAYFORM2 This correspondence of each layer to a GP, conditioned on the layer's second moment matrix, is exact even for finite width N l because the parameters are drawn from a Gaussian. Altogether, this justifies the graphical model depicted in Figure 7.We will write p(z L |x) as an integral over all the intervening second moment matrices DISPLAYFORM3 This joint distribution can be decomposed as DISPLAYFORM4 The directed decomposition in Equation 16 holds because DISPLAYFORM5 Figure 7: Graphical model for neural network's computation. The sum in Equation 12 for l > 0 is a sum over i.i.d. terms. As N l grows large, the Central Limit Theorem applies, and p K l |K l−1 converges to a Gaussian with variance that shrinks as 1 N l. Further, in the infinite width limit it will go to a delta function, DISPLAYFORM6 with F (·) defined as in FIG5. Similarly, the dependence of K 0 on x can be expressed as a delta function, DISPLAYFORM7 So, in the limit of infinite width, z L |x is described by a Gaussian process with kernel DISPLAYFORM8 We outline details of the experiments for Section 3. For MNIST we use a 50k/10k/10k split of the training/validation/test dataset. For CIFAR-10, we used a 45k/5k/10k split. The validation set was used for choosing the best hyperparameters and evaluation on the test set is reported. For training neural networks hyperparameters were optimized via random search on average 250 trials for each choice of (n train, depth, width, nonlinearity).Random search range: Learning rate was sampled within (10 −4, 0.2) in log-scale, weight decay constant was sampled from (10 −8, 1.0) in log-scale, σ w ∈ [0.01, 2.5], σ b ∈ [0, 1.5] was uniformly sampled and mini-batch size was chosen equally among.For the GP with given depth and nonlinearity, a grid of 30 points evenly spaced from 0.1 to 5.0 (for σ 2 w) and 30 points evenly spaced from 0 to 2.0 (for σ Computation time: We report computation times for NNGP experiments. The grid generation with took 440-460s with 6 CPUs for n g = 501, n v = 501, n c = 500, which was amortized over all the experiments. For full (50k) MNIST, constructing K DD for each layer took 90-140s (depending on CPU generation) running on 64 CPUs. Solving linear equations via Cholesky decomposition took 180-220s for 1000 test points. For all the experiments we used pre-computed lookup tables F with n g = 501, n v = 501, n c = 500, and s max = 100. Default value for the target noise σ 2 was set to 10 −10 and was increased by factor of 10 when Cholesky decomposition failed while solving Equation 8 and 9. We refer to BID21 for standard numerically stable implementation of GP regression. Here we include more from experiments described in Section 3.Uncertainty: Relationship between the target MSE and the GP's uncertainty estimate for smaller training set size is shown in Figure 8. 0.5566 GP-3-3.48-1.52 0.5558

Represent the short summary for finding the corresponding scientific article (long).

We show how to make predictions using deep networks, without training deep networks.

represent the tldr for finding the corresponding scientific article

Fewshot example: "We find movement in function space is not proportional to movement in parameter space during optimization. We propose a new natural-gradient style optimizer to address this." == "To optimize a neural network one often thinks of optimizing its parameters, but it is ultimately a matter of optimizing the function that maps inputs to outputs. Since a change in the parameters might serve as a poor proxy for the change in the function, it is of some concern that primacy is given to parameters but that the correspondence has not been tested. Here, we show that it is simple and computationally feasible to calculate distances between functions in a $L^2$ Hilbert space. We examine how typical networks behave in this space, and compare how parameter $\ell^2$ distances compare to function $L^2$ distances between various points of an optimization trajectory. We find that the two distances are nontrivially related. In particular, the $L^2/\ell^2$ ratio decreases throughout optimization, reaching a steady value around when test error plateaus. We then investigate how the $L^2$ distance could be applied directly to optimization. We first propose that in multitask learning, one can avoid catastrophic forgetting by directly limiting how much the input/output function changes between tasks. Secondly, we propose a new learning rule that constrains the distance a network can travel through $L^2$-space in any one update. This allows new examples to be learned in a way that minimally interferes with what has previously been learned. These applications demonstrate how one can measure and regularize function distances directly, without relying on parameters or local approximations like loss curvature. A neural network's parameters collectively encode a function that maps inputs to outputs. The goal of learning is to converge upon a good input/output function. In analysis, then, a researcher should ideally consider how a network's input/output function changes relative to the space of possible functions. However, since this space is not often considered tractable, most techniques and analyses consider the parameters of neural networks. Most regularization techniques, for example, act directly on the parameters (e.g. weight decay, or the implicit constraints stochastic gradient descent (SGD) places upon movement). These techniques are valuable to the extent that parameter space can be taken as a proxy for function space. Since the two might not always be easily related, and since we ultimately care most about the input/output function, it is important to develop metrics that are directly applicable in function space. In this work we show that it is relatively straightforward to measure the distance between two networks in function space, at least if one chooses the right space. Here we examine L 2 -space, which is a Hilbert space. Distance in L 2 space is simply the expected 2 distance between the outputs of two functions when given the same inputs. This computation relies only on function inference. Using this idea of function space, we first focus on characterizing how networks move in function space during optimization with SGD. Do random initializations track similar trajectories? What happens in the overfitting regime? We are particularly interested in the relationship between trajectories in function space and parameter space. If the two are tightly coupled, then parameter change can be taken as a proxy for function change. This common assumption (e.g. Lipschitz bounds) might not always be the case. Next, we demonstrate two possibilities as to how a function space metric could assist optimization. In the first setting we consider multitask learning, and the phenomenon of catastrophic forgetting that makes it difficult. Many well-known methods prevent forgetting by regularizing how much the parameters are allowed to shift due to retraining (usually scaled by a precision matrix calculated on previous tasks). We show that one can instead directly regularize changes in the input/output function of early tasks. Though this requires a "working memory" of earlier examples, this scheme turns out to be quite data-efficient (and more so than actually retraining on examples from old tasks).In the second setting we propose a learning rule for supervised learning that constrains how much a network's function can change any one update. This rule, which we call Hilbert-constrained gradient descent (HCGD), penalizes each step of SGD to reduce the magnitude of the ing step in L 2 -space. This learning rule thus changes the course of learning to track a shorter path in function space. If SGD generalizes in part because large changes to the function are prohibited, then this rule will have advantages over SGD. Interestingly, HCGD is conceptually related to the natural gradient. As we derive in §3.2.1, the natural gradient can be viewed as ing from constrains changes in a function space measured by the Kullbeck-Leibler divergence. We propose to examine the trajectories of networks in the space of functions defined by the inner product f, g = X f (x)g(x)dµ(x), which yields the following norm: DISPLAYFORM0 Here µ is a measure and corresponds to the probability density of the input distribution X. Note that this norm is over an empirical distribution of data and not over the uniform distribution of all possible inputs. The | · | 2 operator refers to the 2-norm and can apply to vector-valued functions. While we refer to this space as a Hilbert space, we make no use of an inner product and can also speak of this as any normed vector space, e.g. a Banach space. This norm leads to a notion of distance between two functions f and g given by DISPLAYFORM1 Since µ is a density, X dµ = 1, and we can write DISPLAYFORM2 The expectation can be approximated as an empirical expectation over a batch of examples drawn from the input distribution: DISPLAYFORM3 The quality of the empirical distance, of course, will depend on the shape and variance of the distribution of data as well as the form of f and g. In section 2.3, we empirically investigate the quality of this estimator for reasonably sample sizes N. We wish to compare at high level how networks move through parameter and function space. Our first approach is to compare a low-dimensional embedding of the trajectories through these spaces. In FIG2, we take a convolutional neural network and train three random initializations on a 5000-image subset of CIFAR-10. By saving the parameters of the network at each epoch as well as the output on a single large validation batch, we can later compute the 2 parameter distance and the FIG2: Visualization of the trajectories of three random initializations of a network through function space, left, and parameter space, right. The network is a convolutional network trained on a 5,000 image subset of CIFAR-10. At each epoch, we compute the L 2 and 2 distances between all previous epochs, forming two distance matrices, and then recompute the 2D embedding from these matrices using multidimensional scaling. Each point on the plots represents the network at a new epoch of training. The black arrows represent the direction of movement. 2 function distance between the snapshots of network at each epoch. The ing distance matrix is then visualized as a two-dimensional embedding. In parameter space, the networks are initialized at very different points and proceed to diverge yet further from these points. Despite this divergence, each trajectory yields a network that has learned the training data perfectly and generalizes with ∼ 50% accuracy to a test set. This illustrates the wide range of parameter settings that can be used to represent a given neural network function. The behavior of the same initializations in function space is quite different. First, note that all three initializations begin at approximately the same point in function space. This is an intriguing property of random initializations that, rather than encoding entirely random functions, random sets of parameters lead on average to the same function (for related work, see e.g. BID5). The initializations then largely follow an identical path for the initial stage of learning. Different initializations thus learn in similar manners, even if the distance between their parameters diverges. During late-stage optimization, random initializations turn from a shared trajectory and begin to diverge in L 2 space. These differences underlie the general principle that L 2 distances behave differently than 2 distances, and that functional regularization could assist training and reduce overfitting. How well do parameter distances reflect function distances? The answer to this question is relevant for any method that directly considers the behavior of parameters. Certain theoretical analyses, furthermore, desire bounds on function distances but instead find bounds on parameter distances and relate the two with a Lipschitz constant (e.g. BID6). Thus, for theoretical analyses and optimization methods alike, it is important to empirically evaluate how well parameter distances correspond to function distances in typical situations. We can compare these two situations by plotting a change in parameters ∆θ against the corresponding change in the function f θ − f θ+∆θ. In FIG0 we display this relation for several relevant distance during the optimization of a CNN on CIFAR-10. There are three scales: the distance between individual updates, the distance between epochs, and the distance from initialization. What changes strikingly throughout optimization is the relationship between parameter and function space. There is a qualitative difference in the ratio of parameter distances to function distances that visible at all three distance scales. Early epochs generally see larger changes in L 2 space for a given change in parameters. Intriguingly, the ratio of the two distances appears to converge to a single value at late optimization, after test error saturates. This is not because the network ceases to move, as noted above. Rather, the loss landscape shifts such that this ratio become constant. It is also clear from these plots that there is not a consistent positive correlation between the parameter and function distances between any two points on the optimization trajectory. For example, the parameter distance between successive epochs is negatively correlated with the L 2 distance for most of optimization FIG0. The distance from initialization shows a clean and positive relationship, but the relationship changes during optimization. Between successive batches, L 2 distance correlates with parameter distance at late epochs, but less so early in optimization when learning is quickest. Thus, at different stages of optimization, the L 2 / 2 ratio is often quite different. The usage of Batch Normalization (BN) and weight decay in this analysis somewhat affects the trends in the L 2 / 2 ratio. In Appendix A we reproduce these plots for networks trained without BN and without weight decay. The overall message that the L 2 / 2 ratio changes during optimization is unchanged. However, these methods both change the scale of updates, and appear to do so differently throughout optimization, and thus some trends are different. In Appendix B, we also isolate the effect of training data, by reproducing these plots for a CNN trained on MNIST and find similar trends. Overall, the correspondence between parameter and function distances depends strongly on the context. It might be worried that since function space is of infinite dimension, one would require prohibitively many examples to estimate a function distance. However, we find that one can compute a distance between two functions with a relatively small amount of examples. FIG4 shows how the estimated L 2 distance converges with an increasing number examples. In general, we find that only a few hundred examples are necessary to converge to an estimation within a few percent. If, after having been trained on a task, a neural network is retrained on a new task, it often forgets the first task. This phenomenon is termed'catastrophic forgetting'. It is the central difficulty of multitask training as well as applications requiring that learning be done online (especially in non-IID situations). Essentially, new information must be encoded in the network, but the the information pertinent to the previous task must not be overwritten. Most efforts to combat catastrophic forgetting rely on restricting how much parameters can change between tasks. Elastic Weight Consolidation (EWC; BID10), for example, adds a penalty to the loss on a new task B that is the distance from the weights after learning on an earlier FIG4: The variance of the the L 2 estimator is small enough that it can be reasonably estimated from a few hundred examples. In panels A and D, we reproduced L 2 distances seen in the panels of FIG0. As we increase the number of validation examples these distances are computed over, the estimations become more accurate. Panels B and E show the 95% confidence bounds for the estimation; on 95% of batches, the value will lie bewteen these bounds. These bounds can be obtained from the standard deviation of the L 2 distance on single examples. In panel C we show that the standard deviation scales linearly with the L 2 distance when measured between updates, meaning that a fixed batch size will often give similar percentage errors. This is not true for the distance from initialization, in panel F; early optimization has higher variance relative to magnitude, meaning that more examples are needed for the same uncertainty. In the Appendix, we also display the convergence of the L 2 distance estimator between epochs.task A, multiplied by the diagonal of the Fisher information matrix F (calculated on task A): DISPLAYFORM0 This idea is closely related to well-studied approaches to Bayesian online learning, if F is interpreted as a precision matrix BID7, BID15 ). Other similar approaches include that of BID21, who use a more accurate approximation of the Fisher, and Synaptic Intelligence (SI; BID28), which discounts parameter change via a diagonal matrix in which each entry reflects the sum contribution of that parameter to the loss. Each of these method discourages catastrophic forgetting by restricting movement in parameter space between tasks, scaled by a (perhaps diagonal) precision matrix calculated on previous tasks. Using a function space metric, it is not hard to ensure that the network's output function on previous tasks does not change during learning. In this case, the loss for a new task B is modified to be: DISPLAYFORM1 The regularization term is the L 2 distance between the current function f θ B and the function after training on task A, f θ A. Since our function space metric is defined over a domain of examples, we will store a small set of previously seen examples in a working memory, as well as the output on those examples. This memory set will be used to calculate the L 2 distance between the current iteration and the snapshot after training. This is a simple scheme, but novel, and we are not aware of direct precedence in the literature. A working memory approach is employed in related work BID12 BID20 ). Note, however, that storing old examples violates the rules of strict online learning. Nevertheless, for large networks it will be more memory-efficient. EWC, for example, requires storing a snapshot of each parameter at the end of the previous task, as well as a diagonal precision matrix with as many entries as parameters. For the 2 hidden layer network with 400 nodes each that was used in the MNIST task in BID10, this is 1,148,820 new parameters, or as many pixels as 1,465 MNIST images. When each layer has as many as 2,000 nodes, as in FIG4 of BID10, the extra stored parameters are comparable to 15,489 MNIST images. The working memory approach that is required to regularize function change from old tasks is thus comparable or cheaper in memory. We compared the performance of our approach at the benchmark task of permuted MNIST. This task requires a single MLP to learn to classify a sequence of MNIST tasks in which the pixels have been randomly permuted differently on each task. We trained an MLP with 2 hidden layers, 400 nodes each, for 10 epochs on each of 8 such permuted datasets. In FIG5, we display how the test accuracy on the first of 8 tasks degrades with subsequent learning. To build the working memory, we keep 1024 examples from previous tasks, making sure that the number of examples from each task is equal. We also remember the predictions on those examples at the end of training on their originating tasks. To calculate the L 2 distance, we simply re-infer on the examples in working memory, and regularize the distance from the current outputs to the remembered outputs. We chose λ = 1.3 as the regularizing hyperparameter from a logarithmic grid search. In FIG5, we compare this method to four comparison methods. The "ADAM" method is ADAM with a learning rate of 0.001, which nearly forgets the first task completely at the end of the 8 tasks. The "ADAM+retrain" method is augmented with a working memory of 1024 examples that are stored from previous tasks. Every n iterations (we found n = 10 to be best), a step is taken to decrease the loss on the memory cache. This method serves as a control for the working memory concept. We also include EWC and SI as comparisons, using the hyperparameters used in their publications (λ = 500, = c = 0.1). Overall, we found that regularizing the L 2 distance on a working memory cache was more successful than simply retraining on the same cache. It also outperformed EWC, but not SI. Note that these methods store diagonal matrices and the old parameters, and in this circumstance these were larger in memory than the memory cache. Figure 4: Regularizing the L 2 distance from old tasks (calculated over a working memory cache of size 1024) can successfully prevent catastrophic forgetting. Here we display the test performance on the first task as 7 subsequent tasks are learned. Our method outperforms simply retraining on the same cache (ADAM+retrain), which potentially overfits to the cache. Also displayed are ADAM without modifications, EWC, and SI. In this section we propose that the L 2 distance can be used for regularization in a single supervised task. In the space of parameters, SGD is a strongly local update rule and large jumps are generally prohibited. SGD is thus more likely to find solutions that are close to the initialization, and furthermore to trace a path of limited length. This discourages the sampling a large volume of parameter space during optimization. If the mapping between parameter and function space is not already very tight, and locality is important for generalization, then additionally constricting changes in function space should help. On the basis of this logic, we propose a learning rule that directly constrains the path length of the optimization trajectory L 2 space. If a network would have been trained to adjust the parameters θ to minimize some cost C 0, we will instead minimize at each step t a new cost given by: DISPLAYFORM0 Like all regularization terms, this can also be viewed as a Langrangian that satisfies a constraint. Here, this constraint ensures that the change in L 2 -space does not exceed some constant value. To evaluate Equation 1, we can approximate the norm with an empirical expectation over X: DISPLAYFORM1 This cost function imposes a penalty upon the difference between the output of the current network at time t and the proposed network at t + 1. The data x i may derive from some validation batch but must pull from the same distribution X. It would also be possible to use unlabeled data. We can write an update rule to minimize Equation 1 that is a modification of gradient descent. We call the rule Hilbert-constrained gradient descent (HCGD). It minimizes C in Equation 1 via an inner loop of gradient descent. To optimize C via gradient descent, we first replace C 0 with its first order approximation J T ∆θ, where J is the Jacobian. Thus we seek to converge to a ∆θ at each update step, where DISPLAYFORM2 Minimization of the proper ∆θ can be performed in an inner loop by a first order method. We first propose some ∆θ 0 = − J = − ∇ θ C 0 (for learning rate) and then iteratively correct this proposal by gradient descent towards ∆θ. If only one correction is performed, we simply add the derivative of the Hilbert-constraining term after ∆θ 0 has been proposed. We found empirically that a single correction was often sufficient. In Appendix C, we demonstrate that this algorithm does actually decrease the distance traveled in function space, as expected. This algorithm is shown in Algorithm 1. DISPLAYFORM3 Calculate correction 9: DISPLAYFORM4 10: DISPLAYFORM5 return θ t Note that the "proposed update" is presented as an SGD step, but could be a step of another optimizer (e.g. ADAM). In the Appendix, we display an extended version of this algorithm. This version allows for multiple corrective iterations in each step. It also allows for a form of momentum. In standard momentum for SGD, one follows a "velocity" term v which is adjusted at each step with the rule v ← βv + J (e.g. see BID26). For HCGD, we also keep a velocity term but update it with the final Hilbert-constrained update ∆θ rather than J. The velocity is used to propose the initial ∆θ 0 in the next update step. We found that this modification of momentum both quickened optimization and lowered generalization error. The natural gradient turns out to carry a similar interpretation as HCGD, in that the natural gradient also regularizes the change in functions' output distributions. Specifically, the natural gradient can be derived from a penalty upon the change in a network's output distribution as measured by the Kullbeck-Leibler divergence (rather than the L 2 distance).To show this, we start with a similar goal of function regularization and will come upon the natural gradient. Let us seek to regularize the change in a network's output distribution P θ throughout optimization of the parameters θ, choosing the Kullbeck-Leibler (KL) divergence as a measure of similarity between any two distributions. To ensure the output distribution changes little throughout optimization, we define a new cost function DISPLAYFORM0 where C 0 is the original cost function and λ is a hyperparameter that controls the importance of this regularization term. Optimization would be performed with respect to the proposed update θ t+1.Evaluating the KL divergence directly is problematic because it is infeasible to define the output density P θ everywhere. One can obtain a more calculable form by expanding D KL (P θt+1 P θt) around θ t to second order with respect to θ. The Hessian of the KL divergence is the Fisher information metric F. With ∆θ ≡ (θ t+1 − θ t), we can rewrite our regularized cost function as DISPLAYFORM1 To optimize C via gradient descent we first replace C 0 with its first order approximation. DISPLAYFORM2 At each evaluation, J is evaluated before any step is made, and we seek the value of ∆θ that minimizes Equation FORMULA15. By setting the derivative with respect to ∆θ to be zero, we can see that this value is DISPLAYFORM3 When λ = 1 this update is equal to the natural gradient. Thus, the natural gradient emerges as the optimal update when one regularizes the change in the output distribution during learning. In Appendix E, we show how one can approximate the natural gradient with an inner first-order optimization loop, like in HCGD. We note that HCGD is computationally cheaper than the exact natural gradient. It does not require any matrix inversions, nor the calculation of separate per-example gradients. When the validation batch X V is drawn anew for each of n corrective iterations (step 8 in Algorithm 1), HCGD requires an additional two forward passes and one backwards pass for each correction, for a total of 2 + 3n passes each outer step. In addition to being seen as a regularizer of functional change, it in an interesting aside to note that variants of the natural gradient have appeared with many justifications. These include data efficiency, minimizing a regret bound during learning, speeding optimization, and the benefits of whitened gradients. Amari originally developed the natural gradient in the light of information geometry and efficiency BID1; ). If some directions in parameter space are more informative of the network's outputs than others, then updates should be scaled by each dimension's informativeness. Equivalently, if not all examples carry equal information about a distribution, then the update step should be modified to make use of highly informative examples. That is, we wish to find a Fisherefficient algorithm (see BID2). The natural gradient uses the Fisher information matrix to scale the update by parameters' informativeness. There is also a connection between the natural gradient (and thus HCGD) and techniques that normalize and whiten gradients. The term F −1 J, after all, simply ensures that steps are made in a parameter space that is whitened by the covariance of the gradients. Whitening the gradients thus has the effect that SGD becomes more similar to the natural gradient. It appears that many approaches to normalize and whiten activations or gradients have been forwarded in the literature BID19; BID25; BID24; BID3 BID27; BID11; BID23; BID22 ). A similar effect is able to be learned with Batch Normalization, as well BID9 ). By normalizing and whitening the gradients, or by proxy, the activations, these various methods ensure that parameter space is a better proxy for function space. We compared HCGD and SGD on feedforward and recurrent architectures. If it is important that SGD limits changes in function space, and parameter and function space are loosely coupled, then HCGD should improve upon SGD. In all tests, we used a tuned learning rate for SGD, and then used the same learning rate for HCGD. We use values of λ = 0.5 and η = 0.02, generally about 10 times less than the principal learning rate. (For the n = 1 version, λ can be folded into the inner learning rate η. Values were chosen so that λη = 0.01.) We chose the batch size for the "validation" batch to be 256. While the examples in each "validation" batch were different than the training batch, they were also drawn from the train set. All models were implemented in PyTorch BID17 ).We tested HCGD as applied to the CIFAR-10 image classification problem. For reproducibility, we trained a Squeezenet v1.1, a convolutional neural network model with batch normalization optimized for parameter efficiency BID8 ). Overall HCGD does not outperform SGD in the final learning stage when trained with the same learning rate as SGD (initial = 0.1), though it does perform better in the early stage while the learning rate is high (FIG1). When we increase the initial learning rate to = 0.3 (red trace), the training accuracy decreases but the test accuracy is still marginally higher than SGD. Given the difference in relative performance between the high and low learning rate stages, it is possible that HCGD requires a different learning rate schedule to achieve the same level of gradient noise. HCGD thus decreases the test error at a given learning rate, but needs to be trained at a higher learning rate to achieve the same level of gradient noise. We next tested the performance of HCGD on a recurrent task. We trained an LSTM on the sequential MNIST task, in which pixels are input one at a time. The order of the pixels was permuted to further complicate the task. We found that HCGD outperformed SGD (Figure 6 . We used 1 correction step, as before, but found that using more correction steps yielded even better performance. However, HCGD underperformed ADAM. While not the ideal optimizer for this task, the fact that SGD can be improved indicates that SGD does not move as locally in function space as it should. Parameter space thus a poor proxy for function space in recurrent networks. HCGD first proposes an update by SGD, and then corrects it, but the first update step can also be other optimizers. Since Adam worked well for the sequential MNIST task, we tested if Adam could also be improved by taking a step to penalize the change in function space. We found that this is indeed the case, and show the as well in Figure 6 . To differentiate the SGD-and Adam-based methods, we refer to in the figure as SGD+HC and Adam+HC. This combination of Adam and L 2 functional regularization could help to achieve state-of-the-art performance on recurrent tasks. Neural networks encode functions, and it is important that analyses discuss the empirical relationship between function space and the more direct parameter space. Here, we argued that the L 2 Hilbert space defined over an input distribution is a tractable and useful space for analysis. We found that networks traverse this function space qualitatively differently than they do parameter space. Depending on the situation, a distance of parameters cannot be taken to represent a proportional distance between functions. We proposed two possibilities for how the L 2 distance could be used directly in applications. The first addresses multitask learning. By remembering enough examples in a working memory to accurately Figure 6 : Results of a singlelayer LSTM with 128 hidden units trained on the sequential MNIST task with permuted pixels. Shown are the traces for SGD and Adam (both with learning rate 0.01). We then take variants of the HCGD algorithm in which the first proposed step is taken to be an SGD step (SGD+HC) or an Adam step (Adam+HC). For SGD+HC we also show the effect of introducing more iterations n in the SGD+HC step.estimate an L 2 distance, we can ensure that the function (as defined on old tasks) does not change as a new task is learned. This regularization term is agnostic to the architecture or parameterization of the network. We found that this scheme outperforms simply retraining on the same number of stored examples. For large networks with millions of parameters, this approach may be more appealing than comparable methods like EWC and SI, which require storing large diagonal matrices. We also proposed a learning rule that reduces movement in function space during single-task optimization. Hilbert-constrained gradient descent (HCGD) constrains the change in L 2 space between successive updates. This approach limits the movement of the encoded function in a similar way as gradient descent limits movement of the parameters. It also carries a similar intuition as the forgetting application: to learn from current examples only in ways that will not affect what has already been learned from other examples. HCGD can increase test performance at image classification in recurrent situations, indicating both that the locality of function movement is important to SGD and that it can be improved upon. However, HCGD did not always improve , indicating either that SGD is stable in those regimes or that other principles are more important to generalization. This is by no means the only possibility for using an L 2 norm to improve optimization. It may be possible, for example, to use the norm to regularize the confidence of the output function (e.g. BID18). We are particularly interested in exploring if more implicit, architectural methods, like normalization layers, could be designed with the L 2 norm in mind. It interesting to ask if there is support in neuroscience for learning rules that diminish the size of changes when that change would have a large effect on other tasks. One otherwise perplexing finding is that behavioral learning rates in motor tasks are dependent on the direction of an error but independent of the magnitude of that error BID4. This is not expected by most models of gradient descent, but would be expected if the size of the change in the output distribution (i.e. behavior) were regulated to be constant. Regularization upon behavioral change (rather than synaptic change) would predict that neurons central to many actions, like neurons in motor pools of the spinal cord, would learn very slowly after early development, despite the fact that their gradient to the error on any one task (if indeed it is calculated) is likely to be quite large. Given our general resistance to overfitting during learning, and the great variety of roles of neurons, it is likely that some type of regularization of behavioral and perceptual change is at play. Figure A.5: Same as above, but for a network trained without Batch Normalization and also without weight decay. Weight decay has a strong effect. The main effect is that decreases the 2 distance traveled at all three scales (from last update, last epoch, and initialization), especially at late optimization. This explains the left column, and some of the middle and right columns. (It is helpful to look at the "white point" on the color scale, which indicates the point halfway through training. Note that parameter distances continue to change after the white point when WD is not used). An additional and counterintuitive property is that the L 2 distance from the last epoch increases in scale during optimization when WD is not used, but decreases if it is. These comparisons show that WD has a strong effect on the L 2 / 2 ratio, but that this ratio still changes considerable throughout training. This is in line with this paper's motivation to consider L 2 distances directly. Figure B.6: Here we reproduce the of FIG0 and FIG4 for the MNIST task, again using a CNN with batch normalization trained with SGD with momentum. It can be seen first that the majority of function space movement occurs very early in optimization, mostly within the first epoch. The standard deviation of the L 2 estimator, which sets the number of examples needed to accurately estimate a consistent value, is somewhat higher than for CIFAR-10. Finally, at right, it can be seen that the relationship between parameter distance traveled and function distance is similar to that of a CNN on CIFAR-10, include the qualitative change after test error converges (which here is around epoch 1). 2 SPACE Figure C.7: The HCGD algorithm is designed to reduce motion through L2-space. To confirm this, here we plot the cumulative squared distance traveled during optimization for a simple MLP trained on MNIST. This is calculated by the simple cumulative sum of the squared distances between consecutive updates. (The squared distance is nice because Brownian motion will present as a linear increase in its cumulative sum). It can be seen that SGD continues to drift in L2-space during the overfitting regime (around epoch 15, which is when test error saturates), while HCGD plateaus. This indicates that the function has converged to a single location; it ceases to change. With SGD, on the other hand, the network continues to cahnge even long after test error saturates. It is interesting to note that HCGD allows the parameters to continue to drift even though the function has generally converged. This version of the algorithm includes momentum. It also allows for multiple corrections. J ← ∇ θ C 0 (X) Calculate gradients 8: DISPLAYFORM0 ∆θ 0 ← −v Proposed update 10: DISPLAYFORM1 12: DISPLAYFORM2 Update buffer 14:for 1 < j < n do Optional additional corrections 15: ∆θ j ← ∆θ j−1 − η(g L 2) DISPLAYFORM3 18: DISPLAYFORM4 19:θ t ← θ t−1 + ∆θ 20:return θ t In order to better compare the natural gradient to the Hilbert-constrained gradient, we propose a natural gradient algorithm of a similar style. Previous work on the natural gradient has aimed to approximate F −1 as best and as cheaply as possible. This is equivalent to minimizing Equation 2 (i.e. J∆θ + λ 2 ∆θ T F ∆θ) with a single iteration of a second-order optimizer. For very large neural networks, however, it is much cheaper to calculate matrix-vector products than to approximately invert a large matrix. It is possible that the natural gradient may be more accessible via an inner gradient descent, which would be performed during each update step as an inner loop. We describe this idea at high level in Algorithm 2. After an update step is proposed by a standard optimizer, the algorithm iteratively corrects this update step towards the natural gradient. To start with a good initial proposed update, it is better to use a fast diagonal approximation of the natural gradient (such as Adagrad or RMSprop) as the main optimizer. Each additional correction requires just one matrix-vector product after the gradients are calculated. Depending on the quality of the proposed update, the number of iterations required is likely to be small, and even a small number of iterations will improve the update. ∆θ i+1 = ∆θ i − η(J + λF ∆θ i)Step towards DISPLAYFORM0 θ ← θ + ∆θ 8:return θ tSince the Fisher matrix F can be calculated from the covariance of gradients, it never needs to be fully stored. Instead, for an array of gradients G of size (# parameters, # examples), we can write DISPLAYFORM1 The choice of G is an important one. It cannot be a vector of aggregated gradients (i.e. J), as that would destroy covariance structure and would in a rank-1 Fisher matrix. Thus, we must calculate the gradients on a per-example basis. To compute G efficiently it is required that a deep learning framework implement forward-mode differentiation, which is currently not supported in popular frameworks. If we choose G to be the array of per-example gradients on the minibatch, F is known as the'empirical Fisher'. As explained in BID13 and in BID16, the proper method is to calculate G from the predictive (output) distribution of the network, P θ (y|x). This can be done as in BID14 by sampling randomly from the output distribution and re-running backpropagation on these fictitious targets, using (by necessity) the activations from the minibatch. Alternatively, as done in BID16, one may also use unlabeled or validation data to calculate G on each batch."

We mathematically analyze the effect of batch normalization on a simple model and obtain key new insights that applies to general supervised learning.