Question: Represent the scientific article to find a short TLDR

Question: We study the implicit bias of gradient descent methods in solving a binary classification problem over a linearly separable dataset. The classifier is described by a nonlinear ReLU model and the objective function adopts the exponential loss function. We first characterize the landscape of the loss function and show that there can exist spurious asymptotic local minima besides asymptotic global minima. We then show that gradient descent (GD) can converge to either a global or a local max-margin direction, or may diverge from the desired max-margin direction in a general context. For stochastic gradient descent (SGD), we show that it converges in expectation to either the global or the local max-margin direction if SGD converges. We further explore the implicit bias of these algorithms in learning a multi-neuron network under certain stationary conditions, and show that the learned classifier maximizes the margins of each sample pattern partition under the ReLU activation. It has been observed in various machine learning problems recently that the gradient descent (GD) algorithm and the stochastic gradient descent (SGD) algorithm converge to solutions with certain properties even without explicit regularization in the objective function. Correspondingly, theoretical analysis has been developed to explain such implicit regularization property. For example, it has been shown in Gunasekar et al. (2018; 2017) that GD converges to the solution with the minimum norm under certain initialization for regression problems, even without an explicit norm constraint. Another type of implicit regularization, where GD converges to the max-margin classifier, has been recently studied in;; Nacson et al. (2018a); Soudry et al. (2017; 2018) for classification problems as we describe below. Given a set of training samples z i = (x i, y i) for i = 1,..., n, where x i denotes a feature vector and y i ∈ {−1, +1} denotes the corresponding label, the goal is to find a desirable linear model (i.e., a classifier) by solving the following empirical risk minimization problem It has been shown in Nacson et al. (2018a); Soudry et al. (2017; 2018) that if the loss function (·) is monotonically strictly decreasing and satisfies proper tail conditions (e.g., the exponential loss), and the data are linearly separable, then GD converges to the solution w with infinite norm and the maximum margin direction of the data, although there is no explicit regularization towards the maxmargin direction in the objective function. Such a phenomenon is referred to as the implicit bias of GD, and can help to explain some experimental . For example, even when the training error achieves zero (i.e., the ing model enters into the linearly separable region that correctly classifies the data), the testing error continues to decrease, because the direction of the model parameter continues to have an improved margin. Such a study has been further generalized to hold for various other types of gradient-based algorithms. analyzed the convergence of GD with no assumption on the data separability, and characterized the implicit regularization to be in a subspace-based form. The focus of this paper is on the following two fundamental issues, which have not been well addressed by existing studies.• Existing studies so far focused only on the linear classifier model. An important question one naturally asks is what happens for the more general nonlinear leaky ReLU and ReLU models. Will GD still converge, and if so will it converge to the max-margin direction? Our study here provides new insights for the ReLU model that have not been observed for the linear model in the previous studies.• Existing studies mainly analyzed the convergence of GD with the only exceptions; Nacson et al. (2018b) on SGD. did not establish the convergence to the max-margin direction for SGD, and Nacson et al. (2018b) established the convergence to the max-margin solution only epochwisely for cyclic SGD (not iterationwise for SGD under random sampling with replacement). Moreover, both studies considered only the linear model. Here, our interest is to explore the iterationwise convergence of SGD under random sampling with replacement to the max-margin direction, and our can shed insights for online SGD. Furthermore, our study provides new understanding for the nonlinear ReLU and leaky ReLU models. We summarize our main contributions, where our focus is on the exponential loss function under ReLU model. We first characterize the landscape of the empirical risk function under the ReLU model, which is nonconvex and nonsmooth. We show that such a risk function has asymptotic global minima and asymptotic spurious local minima. Such a landscape is in sharp contrast to that under the linear model previously studied in , where there exist only equivalent global minima. Based on the landscape property, we show that the implicit bias property in the course of the convergence of GD can fall into four cases: converges to the asymptotic global minimum along the max-margin direction, converges to an asymptotic local minimum along a local max-margin direction, stops at a finite spurious local minimum, or oscillates between the linearly separable and misclassified regions without convergence. Such a diverse behavior is also in sharp difference from that under the linear model , where GD always converges to the max-margin direction. We then take a further step to study the implicit bias of SGD. We show that the expected averaged weight vector normalized by its expected l 2 norm converges to the global max-margin direction or local max-margin direction, as long as SGD stays either in the linearly separable region or in a region of the local minima defined by a subset of data samples with positive label. The proof here requires considerable new technical developments, which are very different from the traditional analysis of SGD, e.g.,;;;;; BID0 BID1. This is because our focus here is on the exponential loss function without attainable global/local minima, whereas traditional analysis typically assumed that the minimum of the loss function is attainable. Furthermore, our goal is to analyze the implicit bias property of SGD, which is also beyond traditional analysis of SGD.We further extend our analysis to the leaky ReLU model and multi-neuron networks. Implicit bias of gradient descent: studied the implicit bias of GD and SGD for minimizing the squared loss function under bounded global minimum, and showed that some of these algorithms converge to a global minimum that is closest to the initial point. Another collection of papers showed that AdaBoost converges to an approximate max-margin classifier. Soudry et al. (2017; 2018) studied the convergence of GD in logistic regression with linearly separable data and showed that GD converges in direction to the solution of support vector machine at a rate of 1/ ln(t). Nacson et al. (2018a) improved this rate to ln(t)/ √ t under the exponential loss via normalized gradient descent. further showed that steepest descent can lead to margin maximization under generic norms. analyzed the convergence of GD on an arbitrary dataset, and provided the convergence rates along the strongly convex subspace and the separable subspace. Our work studies the convergence of GD and SGD under the nonlinear ReLU model with the exponential loss, as opposed to the linear model studied by all the above previous work on the same type of loss functions. analyzed the average SGD (under random sampling) with fixed learning rate and proved the convergence of the population risk, but did not establish the parameter convergence of SGD in the max-margin direction. Nacson et al. (2018b) established the convergence of cyclic SGD epochwisely in direction to the max-margin classifier at a rate O(1/ ln t). Our work differs from these two studies first in that we study the ReLU model, whereas both of these studies analyzed the linear model. Furthermore, we showed that under SGD with random sampling, the expectation of the averaged weight vector converges in direction to the max-margin classifier at a rate O(1/ √ ln t). There have been extensive studies of the convergence and generalization performance of SGD under various models, of which we cannot provide a comprehensive list due to the space limitations. In general, these type of studies either characterize the convergence rate of SGD or provide the generalization error bounds at the convergence of SGD, e.g.,;; , but did not characterize the implicit regularization property of SGD, such as the convergence to the max-margin direction as provided in our paper. We consider the binary classification problem, in which we are given a set of training samples {z 1, . . ., z n}. Each training sample z i = (x i, y i) contains an input data x i and a corresponding binary label y i ∈ {−1, +1}. We denote I +:= {i : y i = +1} as the set of indices of samples with label +1 and denote I −:= {i : y i = −1} in a similar way. Their cardinalities are denoted as n + and n −, respectively, and are assumed to be non-zero. We consider all datasets that are linearly separable, i.e., there exists a linear classifier w such that y i w x i > 0 for all i = 1,..., n. We are interested in training a ReLU model for the classification task. In specific, for a given input data x, the model outputs σ(w x i), where σ(v) = max{0, v} is the ReLU activation function and w denotes the weight parameters. The predicted label is set to be sgn(w x). Our goal is to learn a classifier by solving the following empirical risk minimization problem, where we adopt the exponential loss. DISPLAYFORM0 The ReLU activation causes the loss function in problem (P) to be nonconvex and nonsmooth. Therefore, it is important to first understand the landscape property of the loss function, which is critical for characterizing the implicit bias property of the GD and SGD algorithms. In order to understand the convergence of GD under the ReLU model, we first study the landscape of the loss function in problem (P), which turns out to be very different from that under the linear activation model. As been shown in; , the loss function in problem (P) under linear activation is convex, and achieves asymptotic global minimum, i.e., ∇L(αw *) α → 0 and L(αw *) α → 0 as the scaling constant α → +∞, only if w * is in the linearly separable region. In contrast, under the ReLU model, the asymptotic critical points can be either global minimum or (spurious) local minimum depending on the training datasets, and hence the convergence property of GD can be very different in nature from that under the linear model. The following theorem characterizes the landscape properties of problem (P). Throughout, we denote the infimum of the objective function in problem (P) as L * = n − n. Furthermore, we call a direction w * asymptotically critical if it satisfies ∇L(αw *)→0 as α → +∞.Theorem 3.1 (Asymptotic landscape property). For problem (P) under the ReLU model, any corresponding asymptotic critical direction w * fall into one of the following cases: DISPLAYFORM0 To further elaborate Theorem 3.1, if w * classifies all data correctly (i.e., item 1), then the objective function possibly achieves global minimum L * along this direction. On the other hand, if w * classifies some data with label +1 as −1 (item 2), then the objective function achieves a sub-optimal value along this direction. In the worst case where all data samples are classified as −1 (item 3), the ReLU unit is never activated and hence the corresponding objective function has constant value 1. We note that the cases in items 2 and 3 may or may not take place depending on specific datasets, but if they do occur, the corresponding w * are spurious (asymptotic) local minima. In summary, the landscape under the ReLU model can be partitioned into different regions, where gradient descent algorithms can have different implicit bias as we show next. In this subsection, we analyze the convergence of GD in learning the ReLU model. At each iteration t, GD performs the update DISPLAYFORM0 where η denotes the stepsize. For the linear model whose loss function has infinitely many asymptotic global minima, it has been shown in that GD always converges to the max-margin direction. Such a phenomenon is regarded as the implicit bias property of GD. Here, for the ReLU model, we are also interested in analyzing whether such an implicit-bias property still holds. Furthermore, since the loss function under the ReLU model possibly contains spurious asymptotic local minima, the convergence of GD under the ReLU model should be very different from that under the linear model. Next, we introduce various notions of margin in order to characterize the implicit bias under the ReLU model. The global max-margin direction of samples in I + is defined as DISPLAYFORM1 Such a notion of max-margin is natural because the ReLU activation function can suppress negative inputs. We note that here w + may not locate in the linearly separable region, and hence it may not be parallel to any (asymptotic) global minimum. As we show next, only when w + is in the linearly separable region, GD may converge in direction to such a max-margin direction under the ReLU model. Furthermore, for each given subset J + ⊆ I +, we define the associated local max-margin direction w DISPLAYFORM2 We further denote the set of asymptotic local minima with respect to J + ⊆ I + (see Theorem 3.1 item 2) as DISPLAYFORM3 Theorem 3.2. Apply GD to solve problem (P) with arbitrary initialization and a small enough constant stepsize. Then, the sequence {w t} t generated by GD falls into one of the following cases. DISPLAYFORM4 DISPLAYFORM5 n, and w t =ŵ + J, where J + = ∅, i.e., GD terminates within finite steps. Theorem 3.2 characterizes various instances of implicit bias of GD in learning the ReLU model, which the nature of the convergence is different from that in learning the linear model. In specific, GD can either converge in direction to the global max-margin direction w + that leads to the global minimum, or converge to the local max-margin direction w + J that leads to a spurious local minimum. Furthermore, it may occur that GD oscillates between the linearly separable region and the misclassified region due to the suppression effect of ReLU function. In this case, GD does not have an implicit bias property and convergence guarantee. We provide two simple examples in the supplementary material to further elaborate these cases. In this subsection, we analyze the convergence property and the implicit bias of SGD for solving problem (P). At each iteration t, SGD samples an index ξ t ∈ {1, . . ., n} uniformly at random with replacement, and performs the update DISPLAYFORM0 Similarly to the convergence of GD characterized in Theorem 3.2, SGD may oscillate between the linearly separable and misclassified regions. Therefore, our major interest here is the implicit bias of SGD when it does converge either to the asymptotic global minimum or local minimum. Thus, without loss of generality, we implicitly assume that w + is in the linearly separable region, and the relevant w + J ∈ W + J. Otherwise, SGD does not even converge. The implicit bias of SGD with replacement sampling has not been studied in the existing literature, and the proof of the convergence and the characterization of the implicit bias requires substantial new technical developments. In particular, traditional analysis of SGD under convex functions requires the assumption that the variance of the gradient is bounded; BID1; BID0. Instead of making such an assumption, we next prove that SGD enjoys a nearlyconstant bound on the variance up to a logarithmic factor of t in learning the ReLU model. Proposition 1 (Variance bound). Apply SGD to solve problem (P) with any initialization. If there exists T such that for all t > T, w t either stays in the linearly separable region, or in W + J, then with stepsize η k = (k + 1) −α where 0.5 < α < 1, the variances of the stochastic gradients sampled by SGD along the iteration path satisfy that for all t, DISPLAYFORM1 Proposition 1 shows that the summation of the norms of the stochastic gradients grows logarithmically fast. This implies that the variance of the stochastic gradients is well-controlled. In particular, if we choose η k = (k+1) −1/2, then the bound in Proposition 1 implies that the term E ∇ (w k, z ξ k) 2 stays at a constant level. Based on the variance bound in Proposition 1, we next establish the convergence rate of SGD for learning the ReLU model. Throughout, we denote w t:= then with the stepsize η k = (k + 1) −α, where 0.5 < α < 1, the averaged iterates generated by SGD satisfies DISPLAYFORM2 If there exist T such that for all t > T, w t stays in W + J, then with the same stepsize DISPLAYFORM3 Theorem 3.3 establishes the convergence rate of the expected risk of the averaged iterates generated by SGD. It can be seen that the convergence of SGD achieves different loss values corresponding to global and local minimum in different regions. The stepsize is set to be diminishing to compensate the variance introduced by SGD. In particular, if α is chosen to be sufficiently close to 0.5, then the convergence rate is nearly of the order O(ln 2 t/ √ t), which matches the standard of SGD in convex optimization up to an logarithmic order. Theorem 3.3 also implies that the convergence of SGD is attained as Ew t → +∞ at a rate of O(ln t). We note that the analysis of Theorem 3.3 is different from that of SGD in traditional convex optimization, which requires the global minimum to be achieved at a bounded point and assumes the variance of the stochastic gradients is bounded by a constant Shalev-Shwartz et al. FORMULA19 −α where 0.5 < α < 1, the sequence of the averaged iterate {w t} t generated by SGD satisfies DISPLAYFORM4 If there exist T such that for all t > T, w t stays in W + J, then with the same stepsize DISPLAYFORM5 Theorem 3.4 shows that the direction of the expected averaged iterate E[w t] generated by SGD converges to the max-margin direction w +, without any explicit regularizer in the objective function. The proof of Theorem 3.4 requires a detailed analysis of the SGD update under the ReLU model and is substantially different from that under the linear model;; Nacson et al. (2018a; b). In particular, we need to handle the variance of the stochastic gradients introduced by SGD and exploit its classification properties under the ReLU model. We next provide an example class of datasets (which has been studied in), for which we show that SGD stays stably in the linearly separable region. Proposition 2. If the linear separable samples {z 1, . . ., z n} satisfy the following conditions given in Combes et al. FORMULA0: DISPLAYFORM6 then there exists at ∈ N such that for all t ≥t the sequence generated by SGD stays in the linearly separable region, as long as SGD is not initialized at the local minima described in item 3 of Theorem 3.1. The leaky ReLU activation takes the form σ(v) = max (αv, v), where the parameter (0 ≤ α ≤ 1). Clearly, leaky ReLU takes the linear and ReLU models as two special cases, respectively corresponding to α = 0 and α = 1. Since the convergence of GD/SGD of the ReLU model is very different from that of the linear model, a natural question to ask is whether leaky ReLU with intermediate parameters 0 < α < 1 takes the same behavior as the linear or ReLU model. It can be shown that the loss function in problem (P) under the leaky ReLU model has only asymptotic global minima achieved by w * in the separable region with infinite norm (there does not exist asymptotic local minima). Hence, the convergence of GD is similar to that under the linear model, where the only difference is that the max-margin classifier needs to be defined based on leaky ReLU as follows. For the given set of linearly separable data samples, we construct a new set of data z * DISPLAYFORM0 +, x * i = αx i, ∀i ∈ I −, and y * i = y i, ∀i ∈ I + ∪ I −. Essentially, the data samples with label −1 are scaled by the parameter α of leaky ReLU. Without loss of generality, we assume that the max-margin classifier for data {x * i} passes through the origin after a proper translation. Then, we define the max-margin direction of data X * as DISPLAYFORM1 Then, following the under the linear model in , it can be shown that GD with arbitrary initialization and small constant stepsize for solving problem (P) under the leaky ReLU model satisfies that L(w) converges to zero, and w converges to the max-margin direction, i.e., lim t→∞ wt wt = w *, with its norm going to infinity. Furthermore, following our of Theorem 3.4, it can be shown that for SGD applied to solve problem (P) with any initialization, if there exists T such that for all t > T w t stays in the linearly separable region, then with the stepsize η k = (k + 1) −α, 0.5 < α < 1, the sequence of the averaged iterate {w t} t generated by SGD satisfies DISPLAYFORM2 Thus, for SGD under the leaky ReLU model, the normalized average of the parameter vector converges in direction to the max-margin classifier. In this subsection, we extend our study of the ReLU model to the problem of training a one-hiddenlayer ReLU neural network with K hidden neurons for binary classification. Here, we do not assume linear separability of the dataset. The output of the network is given by DISPLAYFORM0 where W = [w 1, w 2, · · ·, w K] with each column w k representing the weights of the kth neuron in the hidden layer, DISPLAYFORM1 denotes the weights of the output neuron, and σ(·) represents the entry-wise ReLU activation function. We assume that v is a fixed vector whose entries are nonzero and have both positive and negative values. Such an assumption is natural as it allows the model to have enough capacity to achieve zero loss. The predicted label is set to be the sign of f (x), and the objective function under the exponential loss is given by DISPLAYFORM2 Our goal is to characterize the implicit bias of GD and SGD for learning the weight parameters W of the multi-neuron model. In general, such a problem is challenging, as we have shown that GD may not converge to a desirable classifier even under the single-neuron ReLU model. For this reason, we adopt the same setting as that in (, Corollary 8), which assumes that the activated neurons do not change their activation status and the training error converges to zero after a sufficient number of iterations, but our presented below characterizes the implicit bias of GD and SGD in the original feature space, which is different from that in (, Corollary 8). We define a set of vectors DISPLAYFORM3, where A j i = 1 if the sample x i is activated on the jth neuron, i.e., w j x i > 0, and set A j i = 0 otherwise. Such an A i vector is referred to as the activation pattern of x i. We then partition the set of all training samples into m subsets B 1, B 2, · · ·, B m, so that the samples in the same subset have the same ReLU activation pattern, and the samples in different subsets have different ReLU activation patterns. We call B h, h ∈ [m] as the h-th pattern partition. Let w h = k∈{j:A j h =1} v k w k. Then, for any sample x ∈ B h, the output of the network is given by DISPLAYFORM4 We next present our characterization of the implicit bias property of GD and SGD under the above ReLU network model. We define the corresponding max-margin direction of the samples in B h as DISPLAYFORM5 Then the following theorem characterizes the implicit bias of GD under the multi-neuron network. Theorem 4.1. Suppose that GD optimizes the loss L(W) in eq. to zero and there exists T such that for all t > T, the neurons in the hidden layer do not change their activation status. If A h1 ∧ A h2 = 0 (where "∧" denotes the entry-wise logic operator "AND" between digits zero or one) for any h 1 = h 2, then the samples in the same pattern partition of the ReLU activation have the same label, and DISPLAYFORM6 Differently from (, Corollary 8) which studies the convergence of the vectorized weight matrix so that the implicit bias of GD is with respect to features being lifted to an extended dimensional space, Theorem 4.1 characterizes the convergence of the weight parameters and the implicit bias in the original feature space. In particular, Theorem 4.1 implies that although the ReLU neural network is a nonlinear classifier, f (x) is equivalent to a ReLU classifier for the samples in the same pattern partition (that are from the same class), which converges in direction to the maxmargin classifier w h of those data samples. We next letw k=0 w h (t). Then the following theorem establishes the implicit bias of SGD. Theorem 4.2. Suppose that SGD optimizes the loss L(W) in eq. so that there exists T such that for any t > T, L(W) < 1/n, the neurons in the hidden layer do not change their activation status, and for any h 1 = h 2, A h1 ∧ A h2 = 0. Then, for the stepsize η k = (k + 1) −α, 0.5 < α < 1, the samples in the same pattern partition of the ReLU activation have the same label, and DISPLAYFORM7 Similarly to GD, the averaged SGD in expectation maximizes the margin for every sample partition. At the high level, Theorem 4.1 and Theorem 4.2 imply the following generalization performance of the ReLU network under study. After a sufficiently large number of iterations, the neural network partitions the data samples into different subsets, and for each subset, the distance from the samples to the decision boundary is maximized by GD and SGD. Thus, the learned classifier is robust to small perturbations of the data, ing in good generalization performance. In this paper, we study the problem of learning a ReLU neural network via gradient descent methods, and establish the corresponding risk and parameter convergence under the exponential loss function. In particular, we show that due to the possible existence of spurious asymptotic local minima, GD and SGD can converge either to the global or local max-margin direction, which in the nature of convergence is very different from that under the linear model in the previous studies. We also discuss the extensions of our analysis to the more general leaky ReLU model and multi-neuron networks. In the future, it is worthy to explore the implicit bias of GD and SGD in learning multilayer neural network models and under more general (not necessarily linearly separable) datasets. A PROOF OF THEOREM 3.1The gradient ∇L(w) is given by DISPLAYFORM0 If y i w * x i ≥ 0 for all x i ∈ I + ∪ I −, then as α → +∞, we have,, DISPLAYFORM1 and ∇L(αw DISPLAYFORM2 and ∇L(αw DISPLAYFORM3 The proof is now complete. First consider the case when w + is in linearly separable region and the local minimum does not exist along the updating path. We call the region where all vectors w ∈ R d satisfy w x i < 0 for all i ∈ I − as negative correctly classified region. As shown in , L(w) is non-negative and L-smooth, which implies that DISPLAYFORM0 Based on the above inequality, we have DISPLAYFORM1 which, in conjunction with 0 < η < 2/L, implies that DISPLAYFORM2 Thus, we have ∇L(w k) 2 → 0 as k → +∞. By Theorem 3.1, ∇L(w k) vanishes only when all samples with label −1 are correctly classified, and thus GD enters into the negative correctly classified region eventually and diverges to infinity. Theorem 3 shows that when GD diverges to infinity, it simultaneously converges in the direction of the max-margin classifier of all samples satisfying w t x i > 0. Thus, under our setting, GD either converges in the direction of the global max-margin classifer w +: DISPLAYFORM3 or the local max-margin classifier w DISPLAYFORM4 Next, consider the case when w + is not in linearly separable region, and the local minimum does not exist along the updating path. In such a case, we conclude that GD cannot stay in the linearly separable region. Otherwise, it converges in the direction of w + that is not in linearly separable region, which leads to a contradiction. If the asymptotic local minimum w + J exists, then GD may converge in its direction. If w + J does not exist, GD cannot stay in both the misclassified region and linearly separable region, and thus oscillates between these two regions. In the case when GD reaches a local minimum, by Theorem 3.1, we have ∇L(w *) = 0, and thus GD stops immediately and does not diverges to infinity. Example 1 FIG4. The dataset consists of two samples with label +1 and one sample with label −1. These samples satisfy x 1 x 3 < 0 and x 1 x 2 < 0.For this example, if we initialize GD at the green classifier, then GD converges to the max-margin direction of the sample (x 1, +1). Clearly, such a classifier misclassifies the data sample (x 2, +1). Example 2 FIG4. The dataset consists of one sample with label +1 and one sample with label −1. These two samples satisfy 0 < x 1 x 2 ≤ 0.5 x 2 2.For this example, if we initialize at the green classifier, then GD oscillates around the direction x 2 / x 2 and does not converge. Consider the first iteration. Note that the sample z 3 has label −1, and from the illustration of FIG4 (left) we have w 0 x 3 < 0, w 0 x 2 < 0 and w 0 x 1 > 0. Therefore, only the sample z 1 contributes to the gradient, which is given by DISPLAYFORM0 By the update rule of GD, we obtain that for all t DISPLAYFORM1 By telescoping eq. FORMULA37, it is clear that any w t x 2 < 0 for all t since x 1 x 2 < 0. This implies that the sample z 2 is always misclassified. Since we initialize GD at w 0 such that w 0 x 1 > 0 and w 0 x 2 < 0, the sample z 2 does not contribute to the GD update due to the ReLU activation. Next, we argue that there must exists a t such that w t x 2 > 0. Suppose such t does not exist, we always have w t x 1 = (w 0 + t−1 k=0 exp(−w k x 1)x 1 ) x 1 > 0. Then, the linear classifier w t generated by GD stays between x 1 and x 2, and the corresponding objective function reduces to a linear model that depends on the sample z 1 (Note that z 2 contributes a constant due to ReLU activation). Following from the in; Soudry et al. FORMULA0 for linear model, we conclude that w t converges to the max-margin direction x1 x1 as t → +∞. Since x 1 x 2 > 0, this implies that w t x 2 > 0 as t → +∞, contradicting with the assumption. Next, we consider the t such that w t x 1 > 0 and w t x 2 > 0, the objective function is given by L(w t) = exp(−w t x 1) + exp(w t x 2), and the corresponding gradient is given by DISPLAYFORM0 Next, we consider the case that w t x 1 > 0 for all t. Otherwise, both of x 1 and x 2 are on the negative side of the classifier and GD cannot make any progress as the corresponding gradient is zero. In the case that w t x 1 > 0 for all t, by the update rule of GD, we obtain that DISPLAYFORM1 Clearly, the sequence {w t x 2} t is strictly decreasing with a constant gap, and hence within finite steps we must have w t x 2 ≤ 0. Since SGD stays in the linearly separable region eventually, and hence only the data samples in I + contribute to the gradient update due to the ReLU activation function. For this reason, we reduce the original minimization problem (P) to the following optimization DISPLAYFORM0 which corresponds to a linear model with samples in I +. Similarly, if SGD stays in W + J, only the data samples in J + contribute the the gradient update, the original minimization problem (P) is reduced to DISPLAYFORM1 The proof contains three main steps. Step 1: For any u, bounding the term E w t − u 2: By the update rule of SGD, we have DISPLAYFORM2 where M t = 2η t−1 ∇L(w t−1) − ∇ (w t−1, z ξt), w t−1 − u. By convexity we obtain that ∇L(w t−1), w t−1 − u ≥ L(w t−1) − L(u). Then, eq. further becomes DISPLAYFORM3 Telescoping the above inequality yields that DISPLAYFORM4 Taking expectation on both sides of the above inequality and note that EM t = 0 for all t, we further obtain that DISPLAYFORM5 Note that ≤ 1 whenever the data samples are correctly classified and for all i ∈ I +, x i ≤ B, and without loss of generality, we can assume B < √ 2. Hence, the term E ∇ (w k, z ξ k) 2 can be upper bounded by DISPLAYFORM6 Then, noting that η k ≤ 1, eq. can be upper bounded by DISPLAYFORM7 Next, set u = (ln(t)/γ)ŵ + and note thatŵ + x i ≥ γ for all i ∈ I +, we conclude that DISPLAYFORM8 Substituting this into the above inequality and noting that η k = (k + 1) −α and 0.5 < α < 1, we further obtain that DISPLAYFORM9 Step 2: lower bounding E w t − u 2: Note that only the samples in I + contribute to the update rule. By the update rule of SGD, we obtain that DISPLAYFORM10 which further implies that DISPLAYFORM11 Then, we can lower bound w t − u as DISPLAYFORM12 Taking the expectation of w t − u 2: DISPLAYFORM13 where (i) follows from Jensen's inequality. Step 3: Upper bounding DISPLAYFORM14 Combining the upper bound obtained in step 1 and the lower bound obtained in step 2 yields that DISPLAYFORM15 Solving the above quadratic inequality yields that DISPLAYFORM16 E PROOF OF THEOREM 3.3The proof exploits the iteration properties of SGD and the bound on the variance of SGD established in Proposition 1.We start the proof from eq. FORMULA0, following which we obtain DISPLAYFORM17 Taking the expectation on both sides of the above inequality yields that DISPLAYFORM18 which, after telescoping, further yields that DISPLAYFORM19 DISPLAYFORM20 in conjunction with eq., yields that DISPLAYFORM21 where (i) follows from the fact that DISPLAYFORM22 Thus, we can see that L(w t) decreases to 0 at a rate of O(ln 2 (t)/t 1−α ). If we choose α to be close to 0.5, the best convergence rate that can be achieved is O(ln 2 (t)/ √ t).F PROOF OF THEOREM 3.4 DISPLAYFORM23 We first present four technical lemmas that are useful for the proof of the main theorem. Lemma F.1. Given the stepsize η k+1 = 1/(k + 1) −α and the initialization w 0s, then for t ≥ 1, we have DISPLAYFORM24 Lemma F.2. Let X + represent the data matrix of all samples with the label +1, with each row representing one sample. Then we have: DISPLAYFORM25 ∆ n−1 is the simplex in R n. If the equality holds (i.e., the strong duality holds) at q andŵ +, then they satisfyŵ DISPLAYFORM26 andŵ + is the max-margin classifier of samples with the label +1. DISPLAYFORM27 We next apply the lemmas to prove the main theorem. Taking the expectation of the SGD update rule yields that DISPLAYFORM28 ). Applying the above equation recursively, we further obtain that DISPLAYFORM29 which further leads to DISPLAYFORM30 Next, we prove the convergence of the direction of E[w t] to the max-margin direction as follows. 1 2 DISPLAYFORM31 where (i) follows from Lemma F.4, (ii) follows from Lemma F.3 and (iii) is due to eq.. Since following from Lemma F.1 we have that Ew t = O(ln(t)), the above inequality then implies that DISPLAYFORM32 Proof of Lemma F.1. Since x i ≤ B for all i, we obtain that DISPLAYFORM0 By convexity, we have that EL(w t) ≥ L(Ew t), combining which with the above bounds further yields DISPLAYFORM1 That is, the increasing rate of E w t is at least O(ln(t)).Proof of Lemma F.2. Following from the definition of the max-margin, we have DISPLAYFORM2 where f * (a) = max and g(e) = e, respectively, where a, b, c, d are generic vectors. We also denote ∂f (c) and ∂g(e) the subgradient set of f and g at e and c respectively. By the Fenchel-Rockafellar duality , we obtain that DISPLAYFORM3 In particular, the strong duality holds at q andŵ + if and only if −X + w ∈ ∂f (q) andŵ + ∈ ∂g(X + q). Thus, we conclude thatŵ DISPLAYFORM4 Proof of Lemma F.3. By Taylor's expansion and the update of SGD, we obtain that DISPLAYFORM5 where w = θw k−1 + (1 − θ)w k for certain 0 ≤ θ ≤ 1, and is in the linear separable region. Note that for any v, DISPLAYFORM6 where S is the maximum of L(w) in the linearly separable region. We note that S < +∞ because w → ∞ in the linearly separable region and hence L(w) → 0. Taking the expectation on both sides of eq. and recalling that DISPLAYFORM7 we obtain that DISPLAYFORM8 Denote X ∈ R n×d as the data matrix with each row corresponding to one data sample. The derivative of the empirical risk can be written as ∇L(w) = X T l(w)/n, where l(w) = [(w, z 1), (w, z 2),..., (w, z n)]. Then, we obtain that EL(w p) = 1 n + i∈I + E exp(−w p x i) = 1 n + E(l(w p)) 1 and E∇L(w p) = 1 n + i∈I + E exp(−w p x i)x i = 1 n + X + E(l(w p)).Based on the above relationships and Lemma F.2, we obtain that Proof of Lemma F.4. Define h(y) = ln 1 n + i∈I + exp(y i), and then its dual function h * (q) = ln n + + q i ln(q i) ≤ ln n +. Following from Lemma F.2,ŵ + = 1 γ X +T q. Then, by the FenchelYoung inequality, we obtain that DISPLAYFORM9 (ln E(L(w k)) + ln n + ). Under our ReLU model, in the linearly separable region, the gradient ∇L(w) is given by ∇L(w) = − 1 n n i=1 y i 1 {w xi>0} exp(−y i w x i) x i = − 1 n i∈I + exp(−w x i)x i.Thus, only samples with positive classification output, i.e. σ(w t x ξt) > 0, contribute to the SGD updates. We first prove w t < +∞ when there exist misclassified samples. Suppose, toward contradiction, that w t = +∞ as t → +∞ when misclassified samples exist. Note that DISPLAYFORM0 Since w t is infinite, at least one of the coefficients α i, i = 1, · · ·, n is infinite. No loss of generality, we assume α p = +∞. Then, the inner product DISPLAYFORM1 Based on the data selected in Proposition 2, we obtain for ∀i ∈ I − ∪ I + ∀j ∈ I +, y i x i x j > 0 ∀j ∈ I −, y i x i x j < 0, which, in conjunction with eq., implies that, if there exist j ∈ I +, then the first term in the right side of eq. FORMULA19 is finite, the second term is positive, and the third term is positive and infinite. As a , we conclude that for ∀j ∈ I +, w t x j > 0 as t → +∞. Similarly, we can prove that for ∀j ∈ I −, w t x j ≤ 0 as t → +∞, which contracts that w t x j > 0. Thus, if there exist misclassified samples, then we have w t < +∞.Based on the update rule of SGD, we have, for any j w t+1 x j − w t x j = η exp(−y ξt w t x ξt) y ξt x ξt x j = ξt,j.It can be shown that ∀j ∈ I − ∪ I +, y ξt ξt,j > 0, which, combined with eq., implies that, if one sample is correctly classified at iteration t, it remains to be correctly classified in the following iterations. Next, we prove that when w t < +∞, all samples are correctly classified within finite steps. Define DISPLAYFORM2 Since w t < ∞, there exists a constant C such that w t < C for all t. Let D = max i∈I + x i. Then, we obtain, for any j ∈ I + and ξ t ∈ I +, ξt,j = η exp(−w t x ξt)x ξt x j ≥ η exp(−CD) ++, and for any j ∈ I + and ξ t ∈ I −, ξt,j = −η exp(w t x ξt)x ξt x j ≥ η +−.Combining the above two inequalities ∀j ∈ I + yields ξt,j ≥ η min {exp(−CD) ++, η +− }.

We study the implicit bias of gradient methods in solving a binary classification problem with nonlinear ReLU models.

Represent the short summary for finding the corresponding scientific article (long)

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Model pruning has become a useful technique that improves the computational efficiency of deep learning, making it possible to deploy solutions in resource-limited scenarios. A widely-used practice in relevant work assumes that a smaller-norm parameter or feature plays a less informative role at the inference time. In this paper, we propose a channel pruning technique for accelerating the computations of deep convolutional neural networks (CNNs) that does not critically rely on this assumption. Instead, it focuses on direct simplification of the channel-to-channel computation graph of a CNN without the need of performing a computationally difficult and not-always-useful task of making high-dimensional tensors of CNN structured sparse. Our approach takes two stages: first to adopt an end-to-end stochastic training method that eventually forces the outputs of some channels to be constant, and then to prune those constant channels from the original neural network by adjusting the biases of their impacting layers such that the ing compact model can be quickly fine-tuned. Our approach is mathematically appealing from an optimization perspective and easy to reproduce. We experimented our approach through several image learning benchmarks and demonstrate its interest- ing aspects and competitive performance. Not all computations in a deep neural network are of equal importance. In a typical deep learning pipeline, an expert crafts a neural architecture, which is trained using a prepared dataset. The success of training a deep model often requires trial and error, and such loop usually has little control on prioritizing the computations happening in the neural network. Recently researchers started to develop model-simplification methods for convolutional neural networks (CNNs), bearing in mind that some computations are indeed non-critical or redundant and hence can be safely removed from a trained model without substantially degrading the model's performance. Such methods not only accelerate computational efficiency but also possibly alleviate the model's overfitting effects. Discovering which subsets of the computations of a trained CNN are more reasonable to prune, however, is nontrivial. Existing methods can be categorized from either the learning perspective or from the computational perspective. From the learning perspective, some methods use a dataindependent approach where the training data does not assist in determining which part of a trained CNN should be pruned, e.g. BID7 and, while others use a datadependent approach through typically a joint optimization in generating pruning decisions, e.g., BID4 and BID1. From the computational perspective, while most approaches focus on setting the dense weights of convolutions or linear maps to be structured sparse, we propose here a method adopting a new conception to achieve in effect the same goal. Instead of regarding the computations of a CNN as a collection of separate computations sitting at different layers, we view it as a network flow that delivers information from the input to the output through different channels across different layers. We believe saving computations of a CNN is not only about reducing what are calculated in an individual layer, but perhaps more importantly also about understanding how each channel is contributing to the entire information flow in the underlying passing graph as well as removing channels that are less responsible to such process. Inspired by this new conception, we propose to design a "gate" at each channel of a CNN, controlling whether its received information is actually sent out to other channels after processing. If a channel "gate" closes, its output will always be a constant. In fact, each designed "gate" will have a prior intention to close, unless it has a "strong" duty in sending some of its received information from the input to subsequent layers. We find that implementing this idea in pruning CNNs is unsophisticated, as will be detailed in Sec 4.Our method neither introduces any extra parameters to the existing CNN, nor changes its computation graph. In fact, it only introduces marginal overheads to existing gradient training of CNNs. It also possess an attractive feature that one can successively build multiple compact models with different inference performances in a single round of resource-intensive training (as in our experiments). This eases the process to choose a balanced model to deploy in production. Probably, the only applicability constraint of our method is that all convolutional layers and fully-connected layer (except the last layer) in the CNN should be batch normalized BID9. Given batch normalization has becomes a widely adopted ingredient in designing state-of-the-art deep learning models, and many successful CNN models are using it, we believe our approach has a wide scope of potential impacts. In this paper, we start from rethinking a basic assumption widely explored in existing channel pruning work. We point out several issues and gaps in realizing this assumption successfully. Then, we propose our alternative approach, which works around several numerical difficulties. Finally, we experiment our method across different benchmarks and validate its usefulness and strengths. Reducing the size of neural network for speeding up its computational performance at inference time has been a long-studied topic in the communities of neural network and deep learning. Pioneer works include Optimal Brain Damage BID11 and Optimal Brain Surgeon BID5. More recent developments focused on either reducing the structural complexity of a provided network or training a compact or simplified network from scratch. Our work can be categorized into the former, thus the literature review below revolves around reducing the structural complexity. To reduce the structural complexity of deep learning models, previous work have largely focused on sparsifying the weights of convolutional kernels or the feature maps across multiple layers in a network BID1 BID4. Some recent efforts proposed to impose structured sparsity on those vector components motivated from the implementation perspective on specialized hardware BID18 BID21 BID0 BID10 ). Yet as argued by BID16, regularization-based pruning techniques require per layer sensitivity analysis which adds extra computations. Their method relies on global rescaling of criteria for all layers and does not require sensitivity estimation, a beneficial feature that our approach also has. To our knowledge, it is also unclear how widely useful those works are in deep learning. In Section 3, we discuss in details the potential issues in regularization-based pruning techniques potentially hurting them being widely applicable, especially for those that regularize high-dimensional tensor parameters or use magnitude-based pruning methods. Our approach works around the mentioned issues by constraining the anticipated pruning operations only to batchnormalized convolutional layers. Instead of posing structured sparsity on kernels or feature maps, we enforce sparsity on the scaling parameter γ in batch normalization operator. This blocks the sample-wise information passing through part of the channels in convolution layer, and in effect implies one can safely remove those channels. A recent work by BID8 used a similar technique as ours to remove unimportant residual modules in ResNet by introducing extra scaling factors to the original network. However, some optimization subtleties as to be pointed out in our paper were not well explained. Another recent work called Network-Slimming BID15 ) also aims to sparsify the scaling parameters of batch normalization. But instead of using off-the-shelf gradient learning like theirs, we propose a new algorithmic approach based on ISTA and rescaling trick, improving robustness and speed of the undergoing optimization. In particular, the work of BID15 was able to prune VGG-A model on ImageNet. It is unclear how their work would deal with the γ-W rescaling effect and whether their approach can be adopted to large pre-trained models, such as ResNets and Inceptions. We experimented with the pre-trained ResNet-101 and compared to most recent work that were shown to work well with large CNNs. We also experimented with an image segmentation model which has an inception-like module (pre-trained on ImageNet) to locate foreground objects. In most regularized linear regressions, a large-norm coefficient is often a strong indicator of a highly informative feature. This has been widely perceived in statistics and machine learning communities. Removing features which have a small coefficient does not substantially affect the regression errors. Therefore, it has been an established practice to use tractable norm to regularize the parameters in optimizing a model and pick the important ones by comparing their norms after training. However, this assumption is not unconditional. By using Lasso or ridge regression to select important predictors in linear models, one always has to first normalize each predictor variable. Otherwise, the might not be explanatory. For example, ridge regression penalizes more the predictors which has low variance, and Lasso regression enforces sparsity of coefficients which are already small in OLS. Such normalization condition for the right use of regularization is often unsatisfied for nonconvex learning. For example, one has to carefully consider two issues outlined below. We provides these two cases to exemplify how regularization could fail or be of limited usage. There definitely exist ways to avoid the described failures. Model Reparameterization. In the first case, we show that it is not easy to have fine-grained control of the weights' norms across different layers. One has to either choose a uniform penalty in all layers or struggle with the reparameterization patterns. Consider to find a deep linear (convolutional) network subject to a least square with Lasso: for λ > 0, DISPLAYFORM0 The above formulation is not a well-defined problem because for any parameter set DISPLAYFORM1, one can always find another parameter set {W i} 2n i=1 such that it achieves a smaller total loss while keeping the corresponding l 0 norm unchanged by actually setting DISPLAYFORM2 where α > 1. In another word, for any > 0, one can always find a parameter set DISPLAYFORM3 (which is usually non-sparse) that minimizes the first least square loss while having its second Lasso term less than.We note that gradient-based learning is highly inefficient in exploring such model reparameterization patterns. In fact, there are some recent discussions around this BID3. If one adopts a pre-trained model, and augments its original objective with a new norm-based parameter regularization, the new gradient updates may just increase rapidly or it may take a very long time for the variables traveling along the model's reparameterization trajectory. This highlights a theoretical gap questioning existing sparsity-inducing formulation and actual computational algorithms whether they can achieve widely satisfactory parameter sparsification for deep learning models. Transform Invariance. In the second case, we show that batch normalization is not compatible with weight regularization. The example is penalizing l 1 -or l 2 -norms of filters in convolution layer which is then followed by a batch normalization: at the l-th layer, we let DISPLAYFORM4 where γ and β are vectors whose length is the number of channels. Likewise, one can clearly see that any uniform scaling of W l which changes its l 1 -and l 2 -norms would have no effects on the output x l+1. Alternatively speaking, if one is interested in minimizing the weight norms of multiple layers together, it becomes unclear how to choose proper penalty for each layer. Theoretically, there always exists an optimizer that can change the weight to one with infinitesimal magnitude without hurting any inference performance. As pointed by one of the reviewers, one can tentatively avoid this issue by projecting the weights to the surface of unit ball. Then one has to deal with a non-convex feasible set of parameters, causing extra difficulties in developing optimization for data-dependent pruning methods. It is also worth noting that some existing work used such strategy in a layer-by-layer greedy way BID7.Based on this discussion, many existing works which claim to use Lasso, group Lasso (e.g. BID18 ; BID1), or thresholding (e.g. BID16) to enforce parameter sparsity have some theoretical gaps to bridge. In fact, many heuristic algorithms in neural net pruning actually do not naturally generate a sparse parameterized solution. More often, thresholding is used to directly set certain subset of the parameters in the network to zeros, which can be problematic. The reason is in essence around two questions. First, by setting parameters less than a threshold to zeros, will the functionality of neural net be preserved approximately with certain guarantees? If yes, then under what conditions? Second, how should one set those thresholds for weights across different layers? Not every layer contributes equally in a neural net. It is expected that some layers act critically for the performance but only use a small computation and memory budget, while some other layers help marginally for the performance but consume a lot resources. It is naturally more desirable to prune calculations in the latter kind of layers than the former. In contrast with these existing approaches, we focus on enforcing sparsity of a tiny set of parameters in CNN -scale parameter γs in all batch normalization. Not only placing sparse constraints on γ is simpler and easier to monitor, but more importantly, we have two strong reasons:1. Every γ always multiplies a normalized random variable, thus the channel importance becomes comparable across different layers by measuring the magnitude values of γ; 2. The reparameterization effect across different layers is avoided if its subsequent convolution layer is also batch-normalized. In other words, the impacts from the scale changes of γ parameter are independent across different layers. Nevertheless, our current work still falls short of a strong theoretical guarantee. We believe by working with normalized feature inputs and their regularized coefficients together, one is closer to a more robust and meaningful approach. Sparsity is not the goal, but to find less important channels using sparsity inducing formulation is. We describe the basic principle and algorithm of our channel pruning technique. Pruning constant channels. Consider convolution with batch normalization: DISPLAYFORM0 For the ease of notation, we let γ = γ l. Note that if some element in γ is set to zero, say, γ[k] = 0, its output image x l+1:,:,:,k becomes a constant β k, and a convolution of a constant image channel is almost everywhere constant (except for padding regions, an issue to be discussed later). Therefore, we show those constant image channels can be pruned while the same functionality of network is approximately kept:• If the subsequent convolution layer does not have batch normalization, DISPLAYFORM1 its values (a.k.a. elements in β) is absorbed into the bias term by the following equation DISPLAYFORM2 new, 0, where * γ denotes the convolution operator which is only calculated along channels indexed by non-zeros of γ. Remark that DISPLAYFORM3 • If the subsequent convolution layer has batch normalization, DISPLAYFORM4 instead its moving average is updated as DISPLAYFORM5 Remark that the approximation (≈) is strictly equivalence (=) if no padding is used in the convolution operator *, a feature that the parallel work Liu et al. FORMULA11 does not possess. When the original model uses padding in computing convolution layers, the network function is not strictly preserved after pruning. In our practice, we fine-tune the pruned network to fix such performance degradation at last. In short, we formulate the network pruning problem as simple as to set more elements in γ to zero. It is also much easier to deploy the pruned model, because no extra parameters or layers are introduced into the original model. To better understand how it works in an entire CNN, imagine a channel-to-channel computation graph formed by the connections between layers. In this graph, each channel is a node, their inference dependencies are represented by directed edges. The γ parameter serves as a "dam" at each node, deciding whether let the received information "flood" through to other nodes following the graph. An end-to-end training of channel pruning is essentially like a flood control system. There suppose to be rich information of the input distribution, and in two ways, much of the original input information is lost along the way of CNN inference, and the useful part -that is supposed to be preserved by the network inference -should be label sensitive. Conventional CNN has one way to reduce information: transforming feature maps (non-invertible) via forward propagation. Our approach introduces the other way: block information at each channel by forcing its output being constant using ISTA.ISTA. Despite the gap between Lasso and sparsity in the non-convex settings, we found that ISTA BID2 ) is still a useful sparse promoting method. But we just need to use it more carefully. Specifically, we adopt ISTA in the updates of γs. The basic idea is to project the parameter at every step of gradient descent to a potentially more sparse one subject to a proxy problem: let l denote the training loss of interest, at the (t + 1)-th step, we set DISPLAYFORM6 where ∇ γ l t is the derivative with respect to γ computed at step t, µ t is the learning rate, λ is the penalty. In the stochastic learning, ∇ γ l t is estimated from a mini-batch at each step. Eq. has closed form solution as DISPLAYFORM7 where prox η (x) = max{|x| − η, 0} · sgn(x). The ISTA method essentially serves as a "flood control system" in our end-to-end learning, where the functionality of each γ is like that of a dam. When γ is zero, the information flood is totally blocked, while γ = 0, the same amount of information is passed through in form of geometric quantities whose magnitudes are proportional to γ. Scaling effect. One can also see that if γ is scaled by α meanwhile W l+1 is scaled by 1/α, that is, DISPLAYFORM8 the output x l+2 is unchanged for the same input x l. Despite not changing the output, scaling of γ and W l+1 also scales the gradients ∇ γ l and ∇ W l+1 l by 1/α and α, respectively. As we observed, the parameter dynamics of gradient learning with ISTA depends on the scaling factor α if one decides to choose it other than 1.0. Intuitively, if α is large, the optimization of W l+1 is progressed much slower than that of γ. We describe our algorithm below. The following method applies to both training from scratch or re-training from a pre-trained model. Given a training loss l, a convolutional neural net N, and hyper-parameters ρ, α, µ 0, our method proceeds as follows:1. Computation of sparse penalty for each layer. Compute the memory cost per channel for each layer denoted by λ l and set the ISTA penalty for layer l to ρλ l. Here DISPLAYFORM0 where DISPLAYFORM1 h is the size of input image of the neural network. 3. End-to-End training with ISTA on γ. Train N by the regular SGD, with the exception that γ l s are updated by ISTA, where the initial learning rate is µ 0. Train N until the loss l plateaus, the total sparsity of γ l s converges, and Lasso ρ l λ l γ l 1 converges. 4. Post-process to remove constant channels. Prune channels in layer l whose elements in γ l are zero and output the pruned model N by absorbing all constant channels into subsequent layers (as described in the earlier section.). DISPLAYFORM2 5. γ-W rescaling trick. For γ l s and weights in N which were scaled in Step 2 before training, scale them by 1/α and α respectively (scaling back).6. Fine-tune N using regular stochastic gradient learning. Remark that choosing a proper α as used in Steps 2 and 5 is necessary for using a large µ t · ρ in ISTA, which makes the sparsification progress of γ l s faster. We summarize the sensitivity of hyper-parameters and their impacts for optimization below:• µ (learning rate): larger µ leads to fewer iterations for convergence and faster progress of sparsity. But if if µ too large, the SGD approach wouldn't converge.• ρ (sparse penalty): larger ρ leads to more sparse model at convergence. If trained with a very large ρ, all channels will be eventually pruned.• α (rescaling): we use α other than 1. only for pretrained models, we typically choose α from {0.001, 0.01, 0.1, 1} and smaller α warms up the progress of sparsity. We recommend the following parameter tuning strategy. First, check the cross-entropy loss and the regularization loss, select ρ such that these two quantities are comparable at the beginning. Second, choose a reasonable learning rate. Third, if the model is pretrained, check the average magnitude of γs in the network, choose α such that the magnitude of rescaled γ l is around 100µλ l ρ. We found as long as one choose those parameters in the right range of magnitudes, the optimization progress is enough robust. Again one can monitor the mentioned three quantities during the training and terminate the iterations when all three quantities plateaus. There are several patterns we found during experiments that may suggest the parameter tuning has not been successful. If during the first few epochs the Lasso-based regularization loss keeps decreasing linearly while the sparsity of γs stays near zero, one may decrease α and restart. If during the first few epochs the sparsity of γs quickly raise up to 100%, one may decrease ρ and restart. If during the first few epochs the cross-entropy loss keeps at or increases dramatically to a non-informative level, one may decrease µ or ρ and restart. We experiment with the standard image classification benchmark CIFAR-10 with two different network architectures: ConvNet and ResNet-20. We resize images to 32 × 32 and zero-pad them to 40 × 40. We pre-process the padded images by randomly cropping with size 32 × 32, randomly flipping, randomly adjusting brightness and contrast, and standardizing them such that their pixel values have zero mean and one variance. ConvNet For reducing the channels in ConvNet, we are interested in studying whether one can easily convert a over-parameterized network into a compact one. We start with a standard 4-layer convolutional neural network whose network attributes are specified in Table 1. We use a fixed learning rate µ t = 0.01, scaling parameter α = 1.0, and set batch size to 125.Model A is trained from scratch using the base model with an initial warm-up ρ = 0.0002 for 30k steps, and then is trained by raising up ρ to 0.001. After the termination criterion are met, we prune the channels of the base model to generate a smaller network called model A. We evaluate the classification performance of model A with the running exponential average of its parameters. It is found that the test accuracy of model A is even better than the base model. Next, we start from the pre-trained model A to create model B by raising ρ up to 0.002. We end up with a smaller network called model B, which is about 1% worse than model A, but saves about one third parameters. Likewise, we start from the pre-trained model B to create model C. The detailed statistics and its pruned channel size are reported in Table 1. We also train a reference ConvNet from scratch whose channel sizes are 32-64-64-128 with totally 224,008 parameters and test accuracy being 86.3%. The referenced model is not as good as Model B, which has smaller number of parameters and higher accuracy. We have two major observations from the experiment: When the base network is overparameterized, our approach not only significantly reduces the number of channels of the base model but also improves its generalization performance on the test set. Performance degradation seems unavoidable when the channels in a network are saturated, and our approach gives satisfactory tradeoff between test accuracy and model efficiency. We also want to verify our second observation with the state-of-art models. We choose the popular ResNet-20 as our base model for the CIFAR-10 benchmark, whose test accuracy is 92%. We focus on pruning the channels in the residual modules in ResNet-20, which has 9 convolutions in total. As detailed in Table 2, model A is trained from scratch using ResNet-20's network structure as its base model. We use a warm-up ρ = 0.001 for 30k steps and then train with ρ = 0.005. We are able to remove 37% parameters from ResNet-20 with only about 1 percent accuracy loss. Likewise, Model B is created from model A with a higher penalty ρ = 0.01. Table 2: Comparisons between ResNet-20 and its two pruned versions. The last columns are the number of channels of each residual modules after pruning. We experiment our approach with the pre-trained ResNet-101 on ILSVRC2012 image classification dataset. ResNet-101 is one of the state-of-the-art network architecture in ImageNet Challenge. We follow the standard pipeline to pre-process images to 224×224 for training ResNets. We adopt the pre-trained TensorFlow ResNet-101 model whose single crop error rate is 23.6% with about 4.47 × 10 7 parameters. 2 We set the scaling parameter α = 0.01, the initial learning rate µ t = 0.001, the sparsity penalty ρ = 0.1, and the batch size = 128 (across 4 GPUs). The learning rate is decayed every four epochs with rate 0.86. We create two pruned models from the different iterations of training ResNet-101: one has 2.36 × 10 7 parameters and the other has 1.73 × 10 7 parameters. We then fine-tune these two models using the standard way for training ResNet-101, and report their error rates. The Top-5 error rate increases of both models are less than 0.5%. The Top-1 error rates are summarized in TAB2. To our knowledge, only a few works have reported their performance on this very large-scale benchmark w.r.t. the Top-1 errors. We compare our approach with some recent works in terms of model parameter size, flops, and error rates. As shown in TAB2, our model v2 has achieved a compression ratio more than 2.5 while maintaining more than 1% lower error rates than that of other state-of-the-art models at comparable size of parameters. In the first experiment (CIFAR-10), we train the network from scratch and allocate enough steps for both γ and W adjusting their own scales. Thus, initialization of an improper scale of γ-W is not really an issue given we optimize with enough steps. But for the pre-trained models which were originally optimized without any constraints of γ, the γs scales are often unanticipated. It actually takes as many steps as that of training from scratch for γ to warm up. By adopting the rescaling trick setting α to a smaller value, we are able to skip the warm-up stage and quick start to sparsify γs. For example, it might take more than a hundred epoch to train ResNet-101, but it only takes about 5-10 epochs to complete the pruning and a few more epochs to fine-tune. network param size. flops error (%) ratio resnet-50 pruned BID8 As we have discussed about the two major observations in Section 5.1, a more appealing scenario is to apply our approach in pruning channels of over-parameterized model. It often happens when one adopts a pre-trained network on a large task (such as ImageNet classification) and fine-tunes the model to a different and smaller task BID16. In this case, one might expect that some channels that have been useful in the first pre-training task are not quite contributing to the outputs of the second task. We describe an image segmentation experiment whose neural network model is composed from an inception-like network branch and a densenet network branch. The entire network takes a 224 × 224 image and outputs binary mask at the same size. The inception branch is mainly used for locating the foreground objects while the densenet network branch is used to refine the boundaries around the segmented objects. This model was originally trained on multiple datasets. In our experiment, we attempt to prune channels in both the inception branch and densenet branch. We set α = 0.01, ρ = 0.5, µ t = 2 × 10 −5, and batch size = 24. We train the pre-trained base model until all termination criterion are met, and build the pruned model for fine-tuning. The pruned model saves 86% parameters and 81% flops of the base model. We also compare the fine-tuned pruned model with the pre-trained base model across different test benchmark. Mean IOU is used as the evaluation metric.3 It shows that pruned model actually improves over the base model on four of the five test datasets with about 2% ∼ 5%, while it performs worse than the base model on the most challenged dataset DUT-Omron, whose foregrounds might contain multiple objects. base model pruned model test dataset (#images) mIOU mIOU MSRA10K 83.4% 85.5% DUT-Omron 83.2% 79.1% Adobe Flickr-portrait BID17 FORMULA11 88.6% 93.3% Adobe Flickr-hp BID17 84.5% 89.5% COCO-person BID13 84.1% 87.5% param. size 1.02 × 10 Table 4: mIOU reported on different test datasets for the base model and the pruned model. We proposed a model pruning technique that focuses on simplifying the computation graph of a deep convolutional neural network. Our approach adopts ISTA to update the γ parameter in batch normalization operator embedded in each convolution. To accelerate the progress of model pruning, we use a γ-W rescaling trick before and after stochastic training. Our method cleverly avoids some possible numerical difficulties such as mentioned in other regularization-based related work, hence is easier to apply for practitioners. We empirically validated our method through several benchmarks and showed its usefulness and competitiveness in building compact CNN models. Figure 1: Visualization of the number of pruned channels at each convolution in the inception branch. Colored regions represents the number of channels kept. The height of each bar represents the size of feature map, and the width of each bar represents the size of channels. It is observed that most of channels in the bottom layers are kept while most of channels in the top layers are pruned.

Represent the TLDR for finding the corresponding scientific article

A CNN model pruning method using ISTA and rescaling trick to enforce sparsity of scaling parameters in batch normalization.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the scientific article to find a short TLDR

Question: Stochastic gradient descent (SGD) has been the dominant optimization method for training deep neural networks due to its many desirable properties. One of the more remarkable and least understood quality of SGD is that it generalizes relatively well on unseen data even when the neural network has millions of parameters. We hypothesize that in certain cases it is desirable to relax its intrinsic generalization properties and introduce an extension of SGD called deep gradient boosting (DGB). The key idea of DGB is that back-propagated gradients inferred using the chain rule can be viewed as pseudo-residual targets of a gradient boosting problem. Thus at each layer of a neural network the weight update is calculated by solving the corresponding boosting problem using a linear base learner. The ing weight update formula can also be viewed as a normalization procedure of the data that arrives at each layer during the forward pass. When implemented as a separate input normalization layer (INN) the new architecture shows improved performance on image recognition tasks when compared to the same architecture without normalization layers. As opposed to batch normalization (BN), INN has no learnable parameters however it matches its performance on CIFAR10 and ImageNet classification tasks. Boosting, along side deep learning, has been a very successful machine learning technique that consistently outperforms other methods on numerous data science challenges. In a nutshell, the basic idea of boosting is to sequentially combine many simple predictors in such a way that their combined performance is better than each individual predictor. Frequently, these so called weak learners are implemented as simple decision trees and one of the first successful embodiment of this idea was AdaBoost proposed by. No too long after this, and made the important observation that AdaBoost performs in fact a gradient descent in functional space and re-derived it as such. went on to define a general statistical framework for training boosting-like classifiers and regressors using arbitrary loss functions. Together with they showed that boosting minimizes a loss function by iteratively choosing a weak learner that approximately points in the negative gradient direction of a functional space. Neural networks, in particular deep neural nets with many layers, are also trained using a form of gradient descent. Stochastic gradient descent (SGD) has been the main optimization method for deep neural nets due to its many desirable properties like good generalization error and ability to scale well with large data sets. At a basic level, neural networks are composed of stacked linear layers with differentiable non-linearities in between. The output of the last layer is then compared to a target value using a differentiable loss function. Training such a model using SGD involves updating the network parameters in the direction of the negative gradient of the loss function. The crucial step of this algorithm is calculating the parameter gradients and this is efficiently done by the backpropagation algorithm . Backpropagation has many variations that try to achieve either faster convergence or better generalization through some form of regularization. However, despite superior training outcomes, accelerated optimization methods such as Adam , Adagrad or RMSprop have been found to generalize poorly compared to stochastic gradient descent . Therefore even before using an explicit regularization method, like dropout or batch normalization , SGD shows very good performance on validation data sets when compared to other methods. The prevalent explanation for this empirical observation has been that SGD prefers "flat" over "sharp" minima, which in turn makes these states robust to perturbations. Despite its intuitive appeal, recent work by cast doubt on this explanation. This work introduces a simple extension of SGD by combining backpropagation with gradient boosting. We propose that each iteration of the backpropagation algorithm can be reinterpreted by solving, at each layer, a regularized linear regression problem where the independent variables are the layer inputs and the dependent variables are gradients at the output of each layer, before non-linearity is applied. We call this approach deep gradient boosting (DGB), since it is effectively a layer-wise boosting approach where the typical decision trees are replaced by linear regressors. Under this model, SGD naturally emerges as an extreme case where the network weights are highly regularized, in the L2 norm sense. We hypothesize that for some learning problems the regularization criteria doesn't need to be too strict. These could be cases where the data domain is more restricted or the learning task is posed as a matrix decomposition or another coding problem. Based on this idea we further introduce INN, a novel layer normalization method free of learnable parameters and show that it achieves competitive on benchmark image recognition problems when compared to batch normalization (BN). Consider a learning problem where training examples (x, y) are generated from a probability distribution P (x, y) where x ∈ R N is the space of measurements and y ∈ R is the space of labels. The goal of gradient boosting is to find an approximationF (x) to a function F (x) that minimizes a specified loss function Gradient boosting solves Eq. 1 by seeking functions h t (x), so called weak learners, from a differentiable class H such thatF = t h t (x). Starting from a constant function F 0 and applying steepest descent we can findF by iteratively updating with learning rate γ: The weak learners h t are thus found by fitting the training data to the pseudo-residuals r t = ∇ Ft−1 L(F t−1 (x), y) hence in practice: Next, consider a neural network F (x) with parameters. Similarly, starting from parameter w 0 and following steepest descent the network weights are updated at iteration t: At each iteration, for a fixed layer l the gradient of the parameter w ij can be calculated using the chain rule: where o [l] i is the output of the layer l − 1 at iteration t and position i and the gradients ∂Lt ∂F [l] t are calculated using the backpropagation algorithm. Alternatively, using the gradient boosting formula from Eq. 3 and plugging in t we obtain, after dropping the layer indexes [l] for simplicity: o We choose the weak learner h(o t) in this case to be a linear transformation as well h(o t) = o T t v t and fit it to the pseudo-residuals r t = ∂Lt ∂F [l] t by solving the associated regularized linear regression: The primal form solution of Eq. 7 isv t = (o T t o t + αI) −1 o t r t and plugging it into Eq. 6 yields the following update rule: −1 o t r t where I ∈ R N ×N is the identity matrix. For a given input batch of size B, the update rule becomes: where W t ∈ R n×m is the weight matrix for a given layer, O t ∈ R B×n is the input matrix at the same layer while R t ∈ R B×m is the pseudo-residual matrix ∇ Ft L t. In practice, the inverse of matrix O T t O t can be hard to calculate for neural networks with large number of neurons per layer. In consequence, it is often more convenient to use the dual form formulation of the ridge regression solution of Eq. 7 that involves calculating the inverse of O t O T t instead. This is a well known formulation (e.g. see Chapter 1 of ) and leads to the analogous weight update rule: The weight update formulas from Eq. 8 and 9 have the drawback that they change in magnitude with either the number of neurons in the neural network layer or batch size. This is an undesirable behavior that can lead to differences in scale between weights from layers of different sizes. In consequence, the terms O In the following we will compare the performance of DGB and SGD on a variety of reference data sets and explore the effect of the α parameter using simple dense layer neural networks. A summary of the data sets used is given in The first data set is the MNIST database of handwritten digits comprised of 60,000 28x28 grayscale images for training and 10,000 for testing. In the case of the Higgs data set we used a randomly chosen subset with 1 million samples out of 10 million for training, and 500 thousands samples for testing. This data comes from the field of high energy physics and was generated by Monte Carlo simulations. The third data set (Reuters-21578) is comprised of 11,228 english language news articles from Reuters international news agency, labeled over 46 topics. The forth data set is the Human Resource (HR) data set which was put together by IBM Inc. to predict employee attrition based on 47 features and was randomly split into 1,029 training samples and 441 test samples. The fifth data set is a regression problem for predicting a measure of air quality based on the outputs from an array of chemical sensors together with air temperature and humidity . All experiments were performed until test set convergence using a gradient update momentum of 0.9 and no decay. For simplicity, network architecture was fixed for each data set and all activation functions were rectified linear units (ReLU) . The network weights were initialized using a uniform distribution according to . The loss functions used were cross-entropy loss for classification problems and mean squared loss for regression problems. The performance metrics used were accuracy for classification tasks and root mean squared error (RMSE) for regression tasks. In addition, standard deviations were calculated and reported for all performance measures by repeating each experiment 10 times with different random seeds. For this experiment we found that were relatively invariant to network architecture passed a certain complexity, in consequence we chose a dense neural network with three hidden layers each with 500 ReLU nodes. All models were trained using 100 sample batch size for 25 epochs using a fixed 0.1 learning rate and achieved 100% accuracy on the training set. The two DGB variants were labeled DGB(l) if it used the left pseudo-inverse from Eq. 8 and DGB(r) corresponding to the right pseudo-inverse from Eq. 9. Model α Learning rate Performance 97.99% ± 0.11 DGB(r) 0.01 0.1 96.25% ± 0.29 98.03% ± 0.10 DGB(l) 0.01 0.1 96.86% ± 0.13 Table 2: Performance on the MNIST data set measured as accuracy. Both left and right variants achieve increasingly better test set performance with larger α values culminating with 98.59% average accuracy for the DGB(r) model with α = 1.0. This is marginally better than 98.41% average accuracy for SGD but more importantly this shows how increasing the regularization parameter alpha successfully improves test set performance for an image classification problem prone to over-fitting. Overall DGB(l) and DGB(r) show similar performance with the drawback that for relatively small batch sizes DGB(l) is considerably slower. This is because it needs to calculate the inverse of 500x500 matrix as opposed to calculating the inverse of 100x100 matrix, as is the case for DGB(r). When training on a single V-100 Nvidia GPU one SGD iteration took on average 0.0166 sec, one DGB(r) iteration took 0.0188 sec while in contrast DGB(l) took 0.1390 sec. In consequence, for the rest of experiments we will just use DGB(r) and simply label it DGB. Higgs data set was created using Monte Carlo simulations of particle collisions and has 11 mil samples with 28 features. This is a binary classification problem designed to distinguish between a signal process which produces Higgs bosons and a process which does not. For this exercise we kept only 1 mil samples for training and 500k for testing. All 28 features were normalized to be between 0 and 1. We used an architecture composed of two hidden linear layers with 500 ReLU activations each that was trained using a small batch size of 30 samples over 50 epochs. The learning rate was kept constant at 0.01 for SGD and DGB for high α values and was reduced to 0.001 for α ≤ 0.1. This data set poses a considerable challenge with its relatively few features and large sample size. Performance on the training set is relatively poor and doesn't approach 100% accuracy like in the other experiments. The best test set accuracy is obtained for a smaller α value (Table A1). Reuters is a text categorization data set comprised or articles organized into 46 news topics. It was randomly divided into 8982 training and 2246 test samples that were processed using a binary tokenizer for the top 1,000 most common words. This data was then fed into a dense neural network with two hidden layers each with 500 ReLU nodes and trained for 100 epochs with learning rate fixed at 0.01. Table A2 shows the for experiments run using the SGD optimizer and DGB with α values 1.0, 0.1 and 0.01. This is a relatively small data set with a large number of features that is usually more successfully addressed using more advanced approaches like recurrent neural networks or convolutional neural nets. In this case, the are close for all experiments and only marginally better at 78.05% mean accuracy for DGB with α = 0.1. Similar to the other data sets, min-max normalization was used on all 47 features of the Human resource attrition data (HR). This is a binary classification task designed to predict employee attrition. To this end we employed a dense neural network with one hidden layer composed of 100 ReLU nodes. As before, we used a batch size of 30 samples for speed. All models were trained for 500 epochs with a fixed learning rate of 0.001. With only 1029 training samples and 441 test samples the error bars (i.e. standard deviation) are too big to make a claim that DGB outperforms SGD in this case (see Table A3). The Air quality data set contains 9358 instances of hourly averaged reads from chemical sensors designed to measure air quality. After removing samples with missing values we divided the remaining data equally into 3788 training and 3788 test samples, and all the features were min-max normalized to be between 0 and 1. Unlike the previous experiments this was a regression problem with the goal of predicting concentrations of benzene, a common pollutant associated with road traffic. For this we employed a neural network with two hidden layers with 100 ReLU activations each, trained for 1000 epochs using a fixed learning rate of 0.1 and a batch size of 30 samples. Just as the previous experiments, DGB with larger α values were closer in performance to SGD (Table A4). In this case, relaxing the regularization parameter led to a gradual increase in test set performance with almost half the root mean squared error for DGB with α = 0.001 when compared to SGD. We showed in the previous section that deep gradient boosting outperforms regular stochastic gradient descent in some cases. However, this comes at a cost because DGB gradient updates involve calculating the inverse of a matrix at each step and for each layer. The computational cost for matrix inversion is roughly O(n 3) and this can be significant even for small batch sizes as the ones used in the previous experiments. A straight forward way of speeding matrix inversion is to keep just the diagonal terms. This is equivalent of making the simplified assumption that for a layer input matrix O ∈ R B×n the rows are independent (i.e. k o ik o jk for i = j) in the case of DGB(r) or the columns are independent (i.e. k o ki o kj for i = j) in the case of DGB(l). For a layer with n input dimensions and a sample batch of size B, plugging in the diagonal approximations of matrices O t O T t and O T t O t into Eq. 8 & 9 the gradient updates become: in the case of DGB(r) andô in the case of DGB(l). In the above notationÔ t is the normalized version of layer input matrix O t with elementsô ij. In the following paragraphs, FDGB(r) will refer to fast DGB(r) from Eq. 11 and FDGB(l) will refer to fast DGB(l) from Eq. 12. Convolutional neural networks (CNN) are a class of neural networks particularly suited to analyzing visual imagery. Typically the input at each CNN layer is a tensor of size (B × c × w × h) where B is the batch size, c is the number of feature maps while w and h are the width and height of the map. Instead of using a dense network with different weight connecting each point in the (c × w × h) map, CNNs work by dividing the feature map into usually overlapping sections by scanning the input with a kernel of a size much smaller than (w×h), usually (3×3). In this way CNNs effectively increase the number of input samples and simultaneously decrease the number of weights that need to be learned. This process can be viewed as a form of regularization and it has proven to greatly outperform regular dense networks that are usually prone to over-fitting on image recognition problems. For a given CNN layer, assume we have an input of size (B × c 0 × w 0 × h 0) and an output of size (B × c 1 × w 1 × h 1) that was generated with a kernel of size (p × q). In this case, the equivalent DGB formulation based on Eq. 8 & 9 would have an input matrix O t of size (B × w 1 × h 1, c 0 × p × q). It would be extremely costly to first unfold the input as described and then calculate the matrix inverse of either In consequence, it is advisable to use the fast versions of DGB however, one would still have to unfold the input matrix which is still a costly operation. The key observation for extending FDGB(l) to convolutional neural networks is that for small kernels with a small step size each position in the kernel matrix will "visit" roughly the same pixels in a feature map. Hence, calculating the raw second moment of each feature needs to be done only once per map. After resizing the input tensor O t to (B × w 0 × h 0, c 0) the new normalized inputÔ t becomes:ô It is not unreasonable to assume a small kernel size given that most modern convolutional network architectures use kernels of size for image recognition. It is clear from Eq. 10 and previous paragraphs that DGB's gradient update formulation can also be viewed as input normalization defined as N (X):= X(X T X + αI) −1 or N (X):= (XX T + αI) −1 X followed by a regular SGD update. Hence this process can also be implemented as a separate normalization layer, similar to how other layer normalization work like batch norm (BN) , layer norm (LN) , instance norm (IN) and others. The advantages of these normalization methods are well documented: they accelerate descent by enabling higher learning rates without the risk of divergence which in turn may help avoid local sharp minima , make training more resilient to parameter or input data scale , and they stabilize layer output variance which prevents gradient exposition in very deep networks (a). Interestingly, for formulations that operate across samples like batch norm they also have a regularization effect and virtually eliminate the need for other regularization methods like dropout . This is because training examples are seen in conjunction with other examples in the mini-batch, hence the layer outputs are no longer deterministic. This added noise was found to be beneficial to the generalization of the network. Batch norm, layer norm and instance norm perform a z-score standardization of features/inputs by subtracting the mean and dividing by standard deviation which is typically applied to the output of a layer, before the activation function. This procedure runs the risk of keeping the output of a layer inside the linear regime of activation functions that are symmetric around zero. In order to recover the expressive power of the network, after normalization is performed two new parameters are introduced that scale and shift the output. These parameters are in turn learned during training like the rest of the network weights. Interestingly, the scale and shift parameters help convergence even for neural networks with non-symmetric activation functions (e.g. ReLU). In contrast to these methods, the normalization procedure described here is applied to the layer input, it is not scale invariant and has only one parameter α which is predefined before training, not learned. In addition, "normalization" in our case doesn't refer to the process of making the data look more like a standard normal distribution but instead refers to an algebraic transformation of the data that improves its condition number. In the following we will study the effect of input normalization as defined in the previous paragraph to the simple problem of ordinary least squares solved by gradient descent. Consider a multiple linear regression model arg min w (y − Xw) T (y − Xw) where X ∈ R B×n is an input matrix, y ∈ R B is the output vector and w ∈ R n is the vector of parameters. It is well known that the rate of convergence of gradient descent in this case depends on the condition number of the Hessian H = X T X where high values lead to slow convergence. If X = U ΣV T is the singular value decomposition of input matrix X, where U and V are orthogonal matrices and Σ is a diagonal matrix, then the condition number of where σ max and σ min are the largest and respectively smallest singular values of X. Let N (X):= X(X T X + αI) −1 and arg min w (y − N (X)w) T (y − N (X)w) be the ing linear regression problem after applying input normalization. The new Hessian matrix is After plugging in X = U ΣV T and some linear algebra manipulation the new condition number becomes: This shows that input normalization leads to a smaller condition number of the Hessian matrix in the case of multiple linear regression. The same can be obtained when using the alternative input normalization N (X):= (XX T + αI) −1 X (see Prop. A.1 & A.2 in the appendix). In practice, we will define Input Normalization (INN) using the faster diagonal approximations from Eq. 11 & 12. In addition, similar to other normalization layers we will also mean center the input data. For a single layer network, centering further improves conditioning of the Hessian of the loss , it brings the input data near the critical point of most activation functions thus removing the bias towards certain neurons and, it has a regularization effect by shifting the data in a nondeterministic way during training based on the content of the mini-batch. For an input matrix X ∈ R B,n with elements x ij left input normalization is labeled INN(l) and defined as while right input normalization is labeled INN(r) and defined as In the case of INN(l) formulation the batch statistics ij can cause non-deterministic predictions at test time. Instead, running exponential averages are calculated during training for both m 1 and m 2. These new values are then used at test time. We tested the implicit (i.e. FDGB) and explicit (i.e. INN) input normalization models on the CI-FAR10 data set and compared them to Batch Normalization (BN). We then validated the best performing models on the ImageNet object recognition data set (ILSVRC2015) . In both cases we used versions of the VGG architecture first introduced by and then modified them to accommodate the different models as described in Appendix A. The CIFAR10 data set consists of 60000 images organized into 10 classes. There are 50000 training and 10000 test images. According to Table 3 Table 3: Performance on the CIFAR10 data set measured as accuracy. The ImageNet data set is part of the Large Scale Image Recognition Challenge 2015 and consists of 1,281,167 training and 50,000 validation images organized into 1,000 categories. For this experiment we compared the performance of VGG19 and ResNet101 with batch normalization (VGG19 BN(a) and respectively ResNet101 BN) to the equivalent architecture with input normalization layers (VGG19 INN(l) and respectively ResNet101 INN(l)) (See Appendix A for more details). In addition, for VGG19 BN(b) we disabled the learnable affine parameters which reduces BN to a basic standardization operation. This work introduces Deep Gradient Boosting (DGB), a simple extension of Stochastic Gradient Descent (SGD) that allows for finer control over the intrinsic generalization properties of SGD. We empirically show how DGB can outperform SGD in certain cases among a variety of classification and regression tasks. We then propose a faster approximation of DGB and extend it to convolutional layers (FDGB). Finally, we reinterpret DGB as a layer-wise algebraic manipulation of the input data and implement it as a separate normalization layer (INN). We then test INN on image classification tasks where its performance proves to be on par with batch normalization without the need for additional parameters. A APPENDIX Table A4: Performance on the Air data set measured as root mean squared error. has singular values of the form Let X = U ΣV T be the singular value decomposition of X then: values of the form Let X = U ΣV T be the singular value decomposition of X then: For this experiment we used a version of the VGG11 network introduced by that has 8 convolutional layers followed by a linear layer with 512 ReLU nodes, a dropout layer with probability 0.5 and then a final softmax layer for assigning the classification probabilities. A second version of this architecture (VGG11 BN) has batch normalization applied at the output of each convolutional layer, before the ReLU activation as recommended by We modified this architecture by first removing all the batch normalization and dropout layers. We then either replaced all convolutional and linear layers with ones that implement the fast version of DGB for the FDGB(l) architecture or added INN(l) layers in front of each of the original convolutional and linear layers. Both FDGB(l) and INN(l) models implement input normalization based on the left pseudo-inverse (see Eq. 12 & 16) in order to take advantage of its regularization effect. All weights were initialized according to and were trained using stochastic gradient descent with momentum 0.9 and batch size 128. For the FDGB(l) model the gradients were calculated according to Eq. 12 for linear and 13 for convolutional layers. Training was started with learning rate 0.1 and reduced to 0.01 after 250 epochs and continued for 350 epochs. All experiments were repeated 10 times with different random seeds and performance was reported on the validation set as mean accuracy ± standard deviation. Similar to the CIFAR10 data set we based these experiments on the larger VGG19 architecture of. The VGG19 network has 16 convolutional layers interspersed with max-pool operations, followed by two linear layers with 4096 ReLU nodes and a final softmax layer with 1000 outputs. The original VGG19 model uses dropout regularization after each rectified linear layer. A second version of this architecture (VGG19 BN) has batch normalization applied at the output of each convolutional layer, before the ReLU activation. We tested VGG19 BN with and without affine parameters (VGG19 BN(a) and respectively VGG19 BN(b)). After removing all the dropout and batch normalization layers we created the INN(l) version by adding input normalizations in front of each remaining layer. In order to explore the performance of INN on network architectures with residual connections we employed ResNet101 proposed by He et al. (2016b). The original network uses batch normalization between convolutional layers by default (ResNet101 BN). The INN version of this network (ResNet101 INN(l)) was created by first removing all BN layers, then adding INN operations in front of each convolutional and dense layers, with the exception of down-sample convolutions. Stochastic gradient descent was used to minimize the cross entropy loss of each network over 120 epochs, using an initial learning rate of 0.05, 0.9 momentum, a batch size of 128 images, and 0.0001 weight decay. Every 30 epochs the learning rate was decreased by a factor of 10. All networks were evaluated by computing the top-1 and top-5 validation errors on a held out data set using 10 crops per image.

What can we learn about training neural networks if we treat each layer as a gradient boosting problem?

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.