Question: Represent the article for finding a single-sentence TLDR

Question: In this paper, we study the problem of optimizing a two-layer artificial neural network that best fits a training dataset. We look at this problem in the setting where the number of parameters is greater than the number of sampled points. We show that for a wide class of differentiable activation functions (this class involves most nonlinear functions and excludes piecewise linear functions), we have that arbitrary first-order optimal solutions satisfy global optimality provided the hidden layer is non-singular. We essentially show that these non-singular hidden layer matrix satisfy a ``"good" property for these big class of activation functions. Techniques involved in proving this inspire us to look at a new algorithmic, where in between two gradient step of hidden layer, we add a stochastic gradient descent (SGD) step of the output layer. In this new algorithmic framework, we extend our earlier and show that for all finite iterations the hidden layer satisfies the``good" property mentioned earlier therefore partially explaining success of noisy gradient methods and addressing the issue of data independency of our earlier . Both of these are easily extended to hidden layers given by a flat matrix from that of a square matrix. Results are applicable even if network has more than one hidden layer provided all inner hidden layers are arbitrary, satisfy non-singularity, all activations are from the given class of differentiable functions and optimization is only with respect to the outermost hidden layer. Separately, we also study the smoothness properties of the objective function and show that it is actually Lipschitz smooth, i.e., its gradients do not change sharply. We use smoothness properties to guarantee asymptotic convergence of $O(1/\text{number of iterations})$ to a first-order optimal solution. Neural networks architecture has recently emerged as a powerful tool for a wide variety of applications. In fact, they have led to breakthrough performance in many problems such as visual object classification BID13, natural language processing BID5 and speech recognition BID17. Despite the wide variety of applications using neural networks with empirical success, mathematical understanding behind these methods remains a puzzle. Even though there is good understanding of the representation power of neural networks BID1, training these networks is hard. In fact, training neural networks was shown to be NP-complete for single hidden layer, two node and sgn(·) activation function BID2. The main bottleneck in the optimization problem comes from non-convexity of the problem. Hence it is not clear how to train them to global optimality with provable guarantees. Neural networks have been around for decades now. A sudden resurgence in the use of these methods is because of the following: Despite the worst case by BID2, first-order methods such as gradient descent and stochastic gradient descent have been surprisingly successful in training these networks to global optimality. For example, empirically showed that sufficiently over-parametrized networks can be trained to global optimality with stochastic gradient descent. Neural networks with zero hidden layers are relatively well understood in theory. In fact, several authors have shown that for such neural networks with monotone activations, gradient based methods will converge to the global optimum for different assumptions and settings BID16 BID10 BID11 BID12 ).Despite the hardness of training the single hidden layer (or two-layer) problem, enough literature is available which tries to reduce the hardness by making different assumptions. E.g., BID4 made a few assumptions to show that every local minimum of the simplified objective is close to the global minimum. They also require some independent activations assumption which may not be satisfied in practice. For the same shallow networks with (leaky) ReLU activations, it was shown in that all local minimum are global minimum of the modified loss function, instead of the original objective function. Under the same setting, showed that critical points with large "diversity" are near global optimal. But ensuring such conditions algorithmically is difficult. All the theoretical studies have been largely focussed on ReLU activation but other activations have been mostly ignored. In our understanding, this is the first time a theoretical will be presented which shows that for almost all nonlinear activation functions including softplus, an arbitrary first-order optimal solution is also the global optimal provided certain "simple" properties of hidden layer. Moreover, we show that a stochastic gradient descent type algorithm will give us those required properties for free for all finite number of iterations hence even if the hidden layer variables are data dependent, we still get required properties. Our assumption on data distribution is very general and can be reasonable for practitioners. This comes at two costs: First is that the hidden layer of our network can not be wider than the dimension of the input data, say d. Since we also look at this problem in over-parametrized setting (where there is hope to achieve global optimality), this constraint on width puts a direct upper-bound of d 2 on the number of data points that can be trained. Even though this is a strong upper bound, recent from margin bounds BID19 show that if optimal network is closer to origin then we can get an upper bound on number of samples independent of dimension of the problem which will ensure closeness of population objective and training objective. Second drawback of this general setting is that we can prove good properties of the optimization variables (hidden layer weights) for only finite iterations of the SGD type algorithm. But as it is commonly known, stochastic gradient descent converges to first order point asymptotically so ideally we would like to prove these properties for infinitely many iterations. We compare our to some of the prior work of and. Both of these papers use similar ideas to examine first order conditions but give quite different from ours. They give for ReLU or Leaky ReLU activations. We, on the other hand, give for most other nonlinear activations, which can be more challenging. We discuss this in section 3 in more detail. We also formally show that even though the objective function for training neural networks is nonconvex, it is Lipschitz smooth meaning that gradient of the objective function does not change a lot with small changes in underlying variable. To the best of our knowledge, there is no such formally stated in the literature. discuss similar , but there constant itself depends locally on w max, a hidden layer matrix element, which is variable of the the optimization function. Moreover, there is probabilistic. Our is deterministic, global and computable. This allows us to show convergence for the gradient descent algorithm, enabling us to establish an upper bound on the number of iterations for finding an ε-approximate first-order optimal solution (∇f ≤ ε). Therefore our algorithm will generate an ε-approximate first-order optimal solution which satisfies aforementioned properties of the hidden layer. Note that this does not mean that the algorithm will reach the global optimal point asymptotically. As mentioned before, when number of iterations tend to infinity, we could not establish "good" properties. We discuss technical difficulties to prove such a conjecture in more detail in section 5 which details our convergence . At this point we would also like to point that there is good amount of work happening on shallow neural networks. In this literature, we see variety of modelling assumptions, different objective functions and local convergence . BID15 focuses on a class of neural networks which have special structure called "Identity mapping". They show that if the input follows from Gaussian distribution then SGD will converge to global optimal for population objective of the "identity mapping" network. BID3 show that for isotropic Gaussian inputs, with one hidden layer ReLU network and single non-overlapping convolutional filter, all local minimizers are global hence gradient descent will reach global optimal in polynomial time for the population objective. For the same problem, after relaxing the constraint of isotropic Gaussian inputs, they show that the problem is NP-complete via reduction from a variant of set splitting problem. In both of these studies, the objective function is a population objective which is significantly different from training objective in over parametrized domain. In over-parametrized regime, shows that for the training objective with data coming from isotropic Gaussian distribution, provided that we start close to the true solution and know maximum singular value of optimal hidden layer then corresponding gradient descent will converge to the optimal solution. This is one of its kind of where local convergence properties of the neural network training objective function have studied in great detail. Our differ from available current literature in variety of ways. First of all, we study the training problem in the over-parametrized regime. In that regime, the training objective can be significantly different from population objective. Moreover, we study the optimization problem for many general non-linear activation functions. Our can be extended to deeper networks when considering the optimization problem with respect to outermost hidden layer. We also prove that stochastic noise helps in keeping the aforementioned properties of hidden layer. This , in essence, provides justification for using stochastic gradient descent. Another line of study looks at the effect of over-parametrization in the training of neural networks BID9 ). These are not for the same problem as they require huge amount of over-parametrization. In essence, they require the width of the hidden layer to be greater than number of data points which is unreasonable in many settings. These work for fairly general activations as do our but we require a moderate over-parametrization, width × dimension ≥ number of data population, much more reasonable in practice as pointed before from margin bound . They also work for deeper neural network as do our when optimization is with respect to outermost hidden layer (and aforementioned technical properties are satisfied for all hidden layers). We define set [q]:= {1, . . ., q}. For any matrix A ∈ R a×b, we write vect(A) ∈ R ab×1 as vector DISPLAYFORM0 is the i-th element in vector z. B i (r) represents a l i -ball of radius r, centred at origin. We define component-wise product of two vectors with operator. We say that a collection of vectors, DISPLAYFORM1 Similarly, we say that collection of matrices, DISPLAYFORM2 A fully connected two-layer neural network has three parameters: hidden layer W, output layer θ and activation function h. For a given activation function, h, we define neural network function as DISPLAYFORM3 In the above equation, W ∈ R n×d is hidden layer matrix, θ ∈ R n is the output layer. Finally h: R → R is an activation function. The main problem of interest in this paper is the two-layer neural network problem given by DISPLAYFORM4 In this paper, we assume that DISPLAYFORM5 DISPLAYFORM6 (3.1) can also be written in a matrix vector product form: DISPLAYFORM7 where DISPLAYFORM8 Notice that if matrix D ∈ R nd×N is of full column rank (which implies nd ≥ N, i.e., number of samples is less than number of parameters) then it immediately gives us that s = 0 which means such a stationary point is global optimal. This motivates us to investigate properties of h under which we can provably keep matrix D full column rank and develop algorithmic methods to help maintain such properties of matrix D. Note that similar approach was explored in the works of and. To get the full rank property for matrix D, use leaky ReLu function. Basically this leaky activation function adds noise to entries of matrix D which allows them to show matrix D is full rank and hence all local minimums are global minimums. So this is essentially a change of model. We, on the other hand, do not change model of the problem. Moreover, we look at the algorithmic process of finding W differently. We show that SGD will achieve full rank property of matrix D with probability 1 for all finite iterations. So this is essentially a property of the algorithm and not of the model. Even if that is the case, to show global optimality, we need to prove that matrix D is full column rank in asymptotic sense. That question was partly answered in. They show that matrix D is full column rank by achieving a lower bound on smallest singular value of matrix D. But to get this, they need two facts. First, the activation function has to be ReLu so that they can find the spectrum of corresponding kernel matrix. Second, they require a bound on discrepancy of weights W. These conditions are strong in the sense that they restrict the analysis to a non-differentiable activation function and finding an algorithm satisfying discrepancy constraint on W can be a difficult task. On other hand, our are proved for a simple SGD type algorithm which is easy to implement. But we do not get a lower bound on singular value of D in asymptotic sense. There are obvious pluses and minuses for both types of . For the rest of the discussion, we will assume that n = d (our can be extended to case n ≤ d easily) and hence W is a square matrix. In this setting, we develop the following algorithm whose output is a provable first-order approximate solution. Here we present the algorithm and in next sections we will discuss conditions that are required to satisfy full rank property of matrix D as well as convergence properties of the algorithm. In Algorithm 1, we use techniques inspired from alternating minimization to minimize with respect to θ and W. For minimization with respect to θ, we add gaussian noise to the gradient information. This will be useful to prove convergence of this algorithm. We use randomness in θ to ensure some "nice" properties of W which help us in proving that matrix D generated along the trajectory of Algorithm 1 is full column rank. More details will follow in next section. The algorithm has two loops. An outer loop implements a single gradient step with respect to hidden layer, W. For each outer loop iteration, there is an inner loop which optimizes objective function with respect to θ using a stochastic gradient descent algorithm. In the stochastic gradient descent, we generate a noisy estimated of ∇ θ f (W, θ) as explained below. Let ξ ∈ R d be a vector whose elements are i.i.d. Gaussian random variable with zero mean. Then for a given value of W we define stochastic gradient w.r.t. θ as follows: DISPLAYFORM9 (3.4)Then we know that DISPLAYFORM10. We can choose a constant σ > 0 such that following holds DISPLAYFORM11 Moreover, in the algorithm we consider a case where θ ∈ R. Note that R can be kept equal to R d but that will make parameter selection complicated. In our convergence analysis, we will use DISPLAYFORM12 for some constant R, to make parameter selection simpler. We use prox-mapping P x: R d → R as follows: DISPLAYFORM13 In case R is a ball centred at origin, solution of (3.7) is just projection of x − y on that ball. For case where R = R d then the solution is quantity x − y itself. Initialize N o to predefined iteration count for outer ietaration Initialize N i to predefined iteration count for inner iteration Begin outer iteration: DISPLAYFORM0 Notice that the problem of minimization with respect to θ is a convex minimization problem. So we can implement many procedures developed in the Stochastic optimization literature to get the convergence to optimal value BID18 ). In the analysis, we note that one does not even need to implement complete inner iteration as we can skip the stochastic gradient descent suboptimally given that we improve the objective value with respect to where we started, i.e., DISPLAYFORM1 In essence, if evaluation of f for every iteration is not costly then one might break out of inner iterations before running N i iterations. If it is costly to evaluate function values then we can implement the whole SGD for convex problem with respect to θ as specified in inner iteration of the algorithm above. In each outer iteration, we take one gradient decent step with respect to variable W. We have total of N o outer iterations. So essentially we evaluate DISPLAYFORM2 Overall, this algorithm is new form of alternate minimization, where one iteration can be potentially left suboptimally and other one is only one gradient step. We prove in this section that arbitrary first order optimal points are globally optimal. One does not expect to have an arbitrary first order optimal point because it has to depend on data. We still would like to put our analysis here because that inspires us to consider a new algorithmic framework in Section 3 providing similar for all finite iterations of the algorithm. We say that h: R → R satisfy the condition "C1" if DISPLAYFORM0 One can easily notice that most activation functions used in practice e.g., DISPLAYFORM1 satisfy the condition C1. Note that h (x) also satisfy condition C1 for all of them. In fact, except for very small class of functions (which includes linear functions), none of the continuously differentiable functions satisfy condition C1. We first prove a lemma which establishes that columns of the matrix D (each column is a vector form of d×d matrix itself) are linearly independent when W = I d and h satisfies condition C1. We later generalise it to any full rank W using a simple corollary. The statement of following lemma is intuitive but its proof is technical. DISPLAYFORM2 is full rank with measure 1.This means that if u i in the Problem (2.1) are coming from a Lebesgue measure then by corollary 4.2 DISPLAYFORM3 will be a full rank collection given that we have maintained full rank property of W. Now note that in the first-order condition, given in (3.3), row of matrix D are scaled by constant factors θ[j]'s, j ∈ [d]. Notice that we may assume θ[j] = 0 because otherwise there is no contribution of corresponding j-th row of W to the Problem (2.1) and we might as well drop it entirely from the optimization problem. Hence we can rescale rows of matrix D by factor DISPLAYFORM4 without changing the rank. In essence, corollary 4.2 implies that matrix D is full rank when W is full rank. So by our discussion in earlier section, we show that satisfying first-order optimality is enough to show global optimality under condition C1 for data independent W. Remark 4.4 As a of corollary above one can see that the collection of vectors h(W x i) is full rank under the assumption that W is non-singular, x i ∈ R d are independently sampled from Lebesgue measure and h satisfies condition C1.Remark 4.5 Since collection h(W u i) is also full rank, we can say that z i:= h(W 1 u i) are independent and sampled from a Lebesgue measure for a non-singular matrix W 1. Applying the Lemma to z i, we have collection of matrices g(W 2 z i)z i T are full rank with measure 1 for non-singular W 2 and g satisfying condition C1. So we see that for multiple hidden layers satisfying non-singularity, we can apply full rank property for collection of gradients with respect to outermost hidden layer. has rank min{rank(W)d, N } with measure 1 by removing dependent rows and using remark 4.6. Even though we have proved that collection h(Wis full rank in the previous section, we need such W to be independent of data. In general, any algorithm will use data to find W and it appears that in previous section are not meaningful in practice. However, the analysis of Lemma 4.1 motivates the idea that stochastic noise of θ might help in obtaining the required properties of W and D along the trajectory of Algorithm 1. In this section we first prove that by using random noise in stochastic gradient on θ gives a non-singular W k in every iteration. Then using this fact, we prove that matrix D generated along the algorithm is also full rank. The proof techniques are very similar to proof of Lemma 4.1. Later on, we will also show that overall algorithm will converge to approximate first-order optimal solution to Problem (2.1) by using smoothness properties. It should be noted however that this can not guarantee convergence to a global optimal solution. To prove such a , one needs to analyze the smallest singular value of random matrix D, defined in (3.3). More specifically, we have to show that σ min (D) decreases at the rate slower than the first-order convergence rate of the algorithm so that the overall algorithm converges to the global optimal solution. Even if it is very difficult to prove such a in theory, we think that such an assumption about σ min (D) is reasonable in practice. Now we analyze the algorithm. For the sake of simplicity of notation, let us define 2) where N i is the inner iteration count in Algorithm 1. Essentially ξ [k] contains the record of all random samples used until the k-th outer iteration in Algorithm 1 and ξ j [Ni] contains record of all random samples used in the inner iterations of j-th outer iteration. DISPLAYFORM0 DISPLAYFORM1 where W k are matrices generated by Algorithm 1 and measure Pr{. DISPLAYFORM2 Now that we have proved that W k 's generated by the algorithm are full rank, we show that matrix D generated along the trajectory of the algorithm is full rank for any finite number of iterations. We use techniques inspired from Lemma 4.1 but this time we use Lebesgue measure over Θ rather than data. Over randomness of Θ, we can show that our algorithm will not produce any W such that corresponding matrix D is rank deficient. Since Θ is essentially designed to be independent of data so we will not produce rank deficient D throughout the process of randomized algorithm. DISPLAYFORM3 and v is a random vector with Lebesgue measure in R d . W, Z ∈ R d×d and Z = 0. Let h be a function which follows condition C1. Also assume that W is full rank with measure 1 over randomness of v. Then h(W u i) u Proof. We know that DISPLAYFORM4 Now apply Lemma 5.2 to obtain the required . Note that Lemma 5.3 is very similar to the in section 4. Some remarks are in order. Remark 5.4 As a of Lemma 5.3 above, one can see that collection of vectors h(W k u i) is full rank for all finite iterations of Algorithm 1.Remark 5.5 If we have a neural network with multiple hidden layer, we can assume that inner layers are initialized to arbitrary full rank matrices and we are optimizing w.r.t. outermost hidden layer. Corollary 4.2 and Remark 4.4 give us that input to outermost hidden layer are independent vectors sampled from some lebesgue measure. Then applying Algorithm 1 to optimize w.r.t. outermost hidden layer will give us similar as mentioned in Lemma 5.3.Hence we showed that algorithm will generate full rank matrix D for any finite iteration. Now to prove convergence of the algorithm, we need to analyze the function f (defined in (2.1)) itself. We show that f is a Lipschitz smooth function for any given instance of data {u DISPLAYFORM5 . This will give us a handle to estimate convergence rates for the given algorithm. Lemma 5.6 Assuming that h : R → R is such that its gradients, hessian as well as values are bounded and data DISPLAYFORM6 is given then there exists a constant L such that DISPLAYFORM7 Moreover, a possible upper bound on L can be as follows: DISPLAYFORM8 Remark 5.7 Before staing the proof, we should stress that assumptions on h is satisfied by most activation functions e.g., sigmoid, sigmoid symmetric, gaussian, gaussian symmetric, elliot, elliot symmetric, tanh, Erf. Remark 5.8 Note that one can easily estimate value of L given data and θ. Moreover, if we put constraints on θ 2 then L is constant in every iteration of the algorithm 1. As mentioned in section 3, this will provide an easier way to analyze the algorithm. Lemma 5.9 Assuming that scalar function h is such that |h(·)| ≤ u then there exists L θ s.t. DISPLAYFORM9 Notice that Lemma 5.9 gives us value of L θ irrespective of value of W or data. Also observe that f (W, ·) is convex function since hessian DISPLAYFORM10 which is the sum of positive semidefinite matrices. By Lemma 5.9, we know that f (W, ·) is smooth as well. So we can use following convergence provided by BID14 for stochastic composite optimization. A simplified proof can be found in appendix. Theorem 5.10 Assume that stepsizes β i satisfy 0 < β i ≤ 1/2L θ, ∀ i ≥ 1. Let {θ av i+1} i≥1 be the sequence computed according to Algorithm 1. Then we have, DISPLAYFORM11 5) DISPLAYFORM12 i where θ 1 is the starting point for inner iteration and σ is defined in (3.5). Now we look at a possible strategy of selecting stepsize β i. Suppose we adopt a constant stepsize policy then we have DISPLAYFORM13 we get DISPLAYFORM0 By Lemma 5.6, the objective function for neural networks is Lipschitz-smooth with respect to the hidden layer, i.e., it satisfies eq (5.3). Notice that it is equivalent to saying DISPLAYFORM1 (5.7) Since we have a handle on the smoothness of objective function, we can provide a convergence for the overall algorithm. DISPLAYFORM2 8) where R/2 is the radius of origin centred ball, R in algorithm, defined as R: DISPLAYFORM3 In view of Theorem 5.11, we can derive a possible way of choosing γ k, σ and N i to obtain a convergence . More specifically, if DISPLAYFORM4 L and β k τ is chosen according to (5.6) then we have DISPLAYFORM5 Note that in the algorithm 1, we have proved that having a stochastic noise helps keeping matrix D full rank for all finite iterations. Then in Theorem 5.11, we showed a methodical way of achieving approximate first order optimal. So essentially at the end of the finitiely many steps of algorithm 1, we have a point W which satisfies full rank property of D and is approximately first order optimal. We think this kind of can be extended to variety of different first order methods developed for Lipschitz-smooth non-convex optimization problems. More specifically, accelerated gradient method such as unified accelerated method proposed by BID8 or accelerated gradient method by BID7 can be applied in outer iteration. We can also use stochastic gradient descent method for outer iteration. For this, we need a stochastic algorithm that is designed for non-convex and Lipschitz smooth function optimization. Randomized stochastic gradient method, proposed by BID6, Stochastic variance reduction gradient method (SVRG) by Reddi et al. or Simplified SVRG by Allen-Zhu & Hazan can be employed in outer iteration. Convergence of these new algorithms will follow immediately from the convergence of respective studies. Some work may be needed to prove that they hold matrix D full rank. We leave that for the future work. We also leave the problem of proving a bound on singular value for future. This will close the gap between empirical and theory. Value of Lipschitz constant, L, puts a significant impact on the running time of the algorithm. Notice that if L increases then correspondingly N o and N i increase linearly with L. So we need methods by which we can reduce the value of the estimate of L. One possible idea would be to use l 1 -ball for feasible region of θ. More specifically, if R = B 1 (R/2) then we can possibly enforce sparsity on θ which will allow us to put better bound on L. In this appendix, we provide proofs for auxiliary . The is trivially true for d =1, we will show this using induction on d. DISPLAYFORM0. Note that it suffices to prove independence of vector DISPLAYFORM1 DISPLAYFORM2 Moreover, for any collection satisfying x i ∈ Z i, corresponding collection of vector v i are linearly dependent, i.e., DISPLAYFORM3 Noticing the definition of Z 1, we can choose > 0 s.t. x 1:= x 1 + e 1 ∈ Z 1. Since we ensure that x 1 ∈ Z 1 then by (A.1) we have DISPLAYFORM4 So using (A.1) and (A.2) we get DISPLAYFORM5 2 components of v 1 − v 1 are zero. Let us define: DISPLAYFORM6... DISPLAYFORM7 By definition we have y i ∈ R d−1 are independently sampled from (d − 1)-dimensional Lebesgue measure. So by inductive hypothesis, rank of collection of matrices h(y i)y DISPLAYFORM8 2 then λ 2 = · · · = λ N = 0 with measure 1, then by (A.3) we have w 1 = 0 with measure 1, which is contradiction to the fact that w 1 = 0 with measure 1. This gives us DISPLAYFORM9 Notice that (A.4) in its matrix form can be written as linear system DISPLAYFORM10 By (A.6), we have that vector of λ's lies in the null space of the matrix. Finally by inductive hypothesis and (A.5) we conclude that the dimension of that space is DISPLAYFORM11 2 ∈ R N −1 be the basis of that null space i.e. DISPLAYFORM12 Define t i ∈ R 2d−1 as: DISPLAYFORM13 then we can rewrite (A.3) as DISPLAYFORM14 where DISPLAYFORM15 2 and z 1 part of the equation is already satisfied due to selection of null space. 2 ) are constant. Let us define the set S to be the index set of linearly independent DISPLAYFORM16 DISPLAYFORM17 2 ] and every other row is a linear combination of rows in S. Since (A.8) is consistent so the same combination must be valid for the rows of w 1. Now if N ≤ d 2 − 1 then number of variables in (A.8) is ≤ 2d − 3 but number of equations is 2d − 1, therefore at least two equations are linearly dependent on other equation. This implies last (2d − 2) equations then function must be dependent on each other: DISPLAYFORM18 for some fixed combination α j, β j. If we divide above equation by and take the limit as → 0 then we see that h satisfies following differential equation on interval (a DISPLAYFORM19 which is a contradiction to the condition C1! Clearly this leaves only one case i.e. N = d 2 and (2d − 1) equations must satisfy dependency of the following form for all x1 ∈ (a1, b DISPLAYFORM20 Again by similar arguments, the combination is fixed. Let H(x) = xh(x) then dividing above equation by and taking the limit as → 0, we can see that h satisfies following differential equation: DISPLAYFORM21 DISPLAYFORM22 Here the second statement follows from the fact W is a non-singular matrix. Now by Lemma 4.1 we have that collection h(x i)x i T is linearly independent with measure 1. So DISPLAYFORM23 is linearly independent with measure 1. Since any rotation is U is a full rank matrix so we have the . A.3 PROOF OF LEMMA 5.1 This is true for k = 0 trivially since we are randomly sampling matrix W 0. We now show this by induction on k. Recall that gradient of f (W, θ) with respect to W can be written as DISPLAYFORM24 Notice that in effect, we are multiplying i-th row of the rank one matrix h (W u i)u i T by i-th element of vector θ. So this can be rewritten as a matrix product DISPLAYFORM25 where Θ:= diag{θ[i], i = 1,..., d}. So iterative update of the algorithm can be given as DISPLAYFORM26 Notice that given ξ [k], vector θ k+1 and corresponding diagonal matrix Θ k+1 are found by SGD in the inner loop so θ k+1 is a random vector. More specifically, since {ξ DISPLAYFORM27 induces a Lebesgue measure on random variable DISPLAYFORM28 then W k is deterministic quantity. For the sake of contradiction, take any vector v that is supposed to be in the null space of W k+1 with positive probability. DISPLAYFORM29 Now the last equation is of the form DISPLAYFORM30 Suppose we can find such θ with positive probability. Then we can find hypercuboid Z := {x ∈ R d |a < x < b} such that any θ k+1 in given hypercuboid can solve equation (A.10). By induction we have b = 0. We may assume b = 0. Then to get contradiction on existence of Z, we observe that first equation in (A.10) is: DISPLAYFORM31 can not be 0. Hence we arrive at a contradiction to the assumption that there existed a hypercuboid Z containing solutions of (A.10). Since measure on θ k+1 was induced by {ξ DISPLAYFORM32 A.4 PROOF OF LEMMA 5.2We use induction on d. For d = 1 this is trivially true. Now assume this is true for d − 1. We will show this for d. DISPLAYFORM33 For simplicity of notation define t i := Zu i . Due to simple linear algebraic fact provided by full rank property of W we have rank of collection (h(W u i)u DISPLAYFORM34 For the sake of contradiction, say the collection is rank deficient with positive probability then there exists d-dimensional volume V such that for all v ∈ V, we have h(W u i)u i T is not full rank where DISPLAYFORM35 Without loss of generality, we may assume d-dimensional volume to be a hypercuboid V:= {x ∈ R d |a < x < b} (if not then we can inscribe a hypercuboid in that volume). Let us take v ∈ V and ε small enough such that v:= v + εe 1 ∈ V. Correspondingly we have z i and z i. Note that DISPLAYFORM36 Here DISPLAYFORM37 Similarly we also have v i = c i g i. Now by the act that v, v corresponding to z, z are in V, and our assumption of linear dependence for all v ∈ V we get DISPLAYFORM38. Also by induction on d − 1, we have that DISPLAYFORM39 is an invertible matrix and rewrite one part of equation (A.12) as DISPLAYFORM40. So essentially we have satisfied one part of equations (A.12) and (A.13). Notice that since we are moving only one coordinate of random DISPLAYFORM41 ) (by ε incremental changes) keeping all other elements of v constant so we will have y i as constants which implies g i, G, G are constant. So for the sake of simplicity of notation we define l: DISPLAYFORM42 Now, we look at the remaining part of two equation (A.12),(A.13): DISPLAYFORM43 which can be rewritten as DISPLAYFORM44 After (A.15) − (A.14), we have DISPLAYFORM45 Now note that (A.16), characterizes incremental changes in C, C, µ due to ε. So taking the limit as ε → 0, we have DISPLAYFORM46 Here, last equation is due to product rule in calculus. In (A.17), we see that we have 2d−1 equations and Assume that all the gradients in this proof are w.r.t. W then we know that DISPLAYFORM47 DISPLAYFORM48 where the last inequality follows from Cauchy-Schwarz inequality. DISPLAYFORM49, ∀ i then we are done. Let θ max:= max DISPLAYFORM50 Suppose the Lipschitz constants for the first and second term are L i,L and L i,R respectively. Then DISPLAYFORM51 ) and possible upper bound on value of L would become DISPLAYFORM52 Since the Hessian of scalar function h(·) is bounded so we have h (x) is Lipschitz continuous with constant L h. Let r 1, r 2 be two row vectors then we claim h (r 1 x) − h (r 2 x) 2 ≤ L h x 2. r 1 − r 2 2, ∀ r 1, r 2 because: h (r 1 x) − h (r 2 x) 2 ≤ L h r 1 x − r 2 x ≤ L h x 2 r 1 − r 2 2 From the relation above we have the following: Noting that DISPLAYFORM53 DISPLAYFORM54 we have DISPLAYFORM55 where u 1 and u 2 are upper bounds on scalar functions |h(·)| and |h (·)| respectively and d is rowdimension of W.A.7 PROOF OF THEOREM 5.11We know by Lemma 5.6 that f (·, θ) is a Lipschitz smooth function. So using (5.7) we have DISPLAYFORM56 DISPLAYFORM57 (γ k − L/2γ 2 k). Now taking expectation with respect to ξ [No] (which is defined in (5.1)), we have DISPLAYFORM58 DISPLAYFORM59.

Represent the short summary for finding the corresponding scientific article (long).

This paper talks about theoretical properties of first-order optimal point of two layer neural network in over-parametrized case

Represent the TLDR for finding the corresponding scientific article

We propose a hypothesis for why gradient descent generalizes based on how per-example gradients interact with each other.


Question: Represent the scientific article to find a short TLDR

Question: We introduce the concept of channel aggregation in ConvNet architecture, a novel compact representation of CNN features useful for explicitly modeling the nonlinear channels encoding especially when the new unit is embedded inside of deep architectures for action recognition. The channel aggregation is based on multiple-channels features of ConvNet and aims to be at the spot finding the optical convergence path at fast speed. We name our proposed convolutional architecture “nonlinear channels aggregation networks (NCAN)” and its new layer “nonlinear channels aggregation layer (NCAL)”. We theoretically motivate channels aggregation functions and empirically study their effect on convergence speed and classification accuracy. Another contribution in this work is an efficient and effective implementation of the NCAL, speeding it up orders of magnitude. We evaluate its performance on standard benchmarks UCF101 and HMDB51, and experimental demonstrate that this formulation not only obtains a fast convergence but stronger generalization capability without sacrificing performance. With modern learnable representations such as deep convolutional neural networks (CNNs) matured in many image understanding tasks BID7, human action recognition has received a significant amount of attentions BID11 BID2 BID9 BID3 BID12. Due to the fact that video itself provides an additional temporal clue and that the parameters and the calculations of CNNs grow exponentially, training CNNs with such large-scale parameters in video domain is time-consuming. However, it remains unclear how the effective convergence accelerators could be conducted for the optimal path by formulizing the handcrafted rules. Since videos consist of still images, training tricks and methods, such as Relu, BN, have been shown to transfer to videos directly. Recent theoretical and empirical works have demonstrated the importance of quickly training deep architectures successfully, and the effective convergence accelerators advanced in the 2D image, such as relu BID4 and batch normalization BID5, have been developed for fast convergence. This is in part inspired by observations of the limited GPU memory and computing power, especially when confronting the large-scale video dataset which may introduce a large majority of parameters. Another pipeline of algorithms focuses on the training optimizer of CNNs, for examples, sgd, momentum, nesterov, adagrad and adadelta. However, training CNNs utilizing the large-scale video datasets is still nontrivial in video task, particularly if one seeks a compact but fast long termporal dynamic representation that can be processed efficiently. Our current work reconsiders the means of facilitating convergence of ConvNets to increase the understanding of how to embed some hand-crafted rules inside of CNNs for fast convergence in a more thorough fashion. In addition to the accelerators and effective optimizers, we tend to explore a thorough method causing the value of the loss function to descend rapidly. Intuitively, we argue that CNNs will accelerate training process once the complex relationship across convolutional features channels is modeled, explicitly, by the hand-crafted rules. In the existing units 3D convolution implements a linear partial sum of channels BID6, 3D max-pooling takes the maximum feature by channels and 3D average-pooling make a spatial-channel average of features. Unfortunately, all the 3D units conduct a linear channels aggregation, implicitly and locally. Despite that the implicit linear aggregation has been applied to broad fields, there seems to be less works explicitly taking modeling the complex nonlinear relationship across channels into account. In fact, either one-stream or two-stream algorithms ignore the channel-level encoding. For video recognition task, a very tricky problem is how to train the CNN architectures for the sake of making a lower loss rapidly in the scarcity of videos. We conjecture that there is complex nonlinear relationship among the channels of CNN features. Once this implicit relationship is explicitly modeled, such accomplishment will facilitate converging with faster search to the optimal trajectory. In this paper, we proposed a nonlinear channels aggregation layer (NCAL), which explicitly models the complex nonlinear relationships across channels. Since a standard CNN provides a whole hierarchy of video representations, the first question worthy exploring is where the NACL should take place. For example, we can aggregate the output of the fully-connected layers of CNN architecture pre-trained on videos. A drawback of such implementation is that the convolutional features channels of CNN itself are still implicitly encoded and are unaware of the lower level channels relationship. The alternative is to model the nonlinear channels aggregation of some intermediate network layer. In this case, the lower layers fail to extract the representative features from video sequences, but the upper layers can reason about the overall dynamics in the video. The former is prone to sacrificing the recognition performance while the latter is thus thought of as the appropriate convolutional features for the compact aggregation. Here we build our methods on top of the successful Inception V1 architecture. More specifically, three main contributions are provided in this work. Our first contribution is to introduce the concept of nonlinear channels aggregation for fast convergence. We also show that, in this manner, it is possible to apply the concept of nonlinear channels aggregation to the intermediate layers of a CNN representation by constructing an efficient nonlinear channels aggregation layer (NCAL).Here we build our methods on top of the successful Inception V1 architecture. More specifically, three main contributions are provided in this work. Our first contribution is to introduce the concept of nonlinear channels aggregation for fast convergence. We also show that, in this manner, it is possible to construct an efficient nonlinear channels aggregation by applying the concept of nonlinear channels aggregation to the intermediate layers of the standard CNN. More importantly, it is explicitly and globally that the nonlinear channels relationship is modeled compared to the traditional local and implicit units. Our second contribution is to simplify the process of nonlinear channels aggregation layer (NCAL) and make a fast yet accurate implementation of it. Notably, the proposed NCAL can be embodied inside of any standard CNN architectures, and not break the rest components of structures. More broadly, the proposed NCAL is not limited to action recognition, that is, it can be applied to any task with CNNs. Here we introduce it into action recognition, and leave the explorations of it on the other domains in the future. Our third contribution is to leverage these ideas to construct a novel nonlinear channels aggregation network, perform the training process end-to-end. We show that such nonlinear channels encoding in a fast decline in the value of the loss function of CNNs while obtains efficient and accurate classification of actions in videos. The rest of the paper is organized as follows: Section 2 describes the related works, and section 3 represents the principle of the nonlinear channels aggregation networks (NCAN) and the backward propagation of NCAN. This is followed by the experiments in section 4. Finally, we conclude this paper in Section 6. Owing to the difficulty in training convolutional networks on video dataset, e.g. more parameters and small-scale video sequences, we restrict our works under the premise of action recognition. Previous works related to ours fall into two categories: convolutional networks for action recognition, linear channels aggregation. Convolutional networks for action recognition. Currently one type of the common practices in the mainstream algorithms enables a stack of consecutive video sequences to train the convolutional architectures, which implicitly captures motion characteristics. Another variants built on this is temporal segment networks (TSN), thus ConvNets are trained leveraging the multiple segments BID0 BID1 BID14 BID13 a). The last practice extends the 2-D convolution to capture the motions with extra temporal dimension. In this case so far the extra temporal dimension introduces exponential parameters. Also, we can make finding that most of effective methods are built on all the basic frameworks presented above. Thus, we perform our methods on these fundamental architectures that most of video classification algorithms have in common. Linear channels aggregation. In the traditional ConvNets, the implicit relationship across channels can be captured by the 3D convolution, 3D max pooling, 3D average pooling and 3D weighted-average pooling. Compared with 2-D convolution, max pooling, average pooling and weighted-average pooling, these 3D operators conduct the local linear aggregation on the space and channels level. Nevertheless, the guidelines of these simple linear methods fail to model the explicit, global and nonlinear channels relationships. These local linear encoders, in essence, remain unable to completely represent the complex nonlinear channels functions. The proposed nonlinear channels aggregation network, while also emphasizing this principle, is the first framework for end-to-end fast convergence of CNNs by capturing the global channels dependency. In this section, we give detailed descriptions of performing nonlinear channels aggregation. Specifically, we first introduce the basic linear channels encoding concepts in the framework of video tasks and utilize the available layers to conduct the channel-level linear encoding. Then, we introduce the principle of the nonlinear channels aggregation layer (NCAL) simulating the complex channel-level encodings. Finally we describe the fast optimization and backward propagation of NCAL. We consider how to leverage current units in the standard CNN to comply the channels relationships. Suppose for the moment that different channels in the convolutional network capture various discriminative features, and for the bold hypothesis that there is a nonlinear relationship between them, stacks of the channels in the subsequent layer will skew CNN in favor of the significant correspondence between these appropriate channels. To achieve the goal, current methods, 3D pooling and 3D convolution can be utilized to achieve it. As mentioned in the introduction, pooling would lead to a decline in feature resolution, which would cause the poor performance of recognition. We thus utilize 3D convolution to implement the linear channels relationship. Given video frames V = {V 1, V 2, · · ·, V K} multiple features S = {S 1, S2, · · ·, SK} are denoted as the responses of the immediate layer of CNN. Let h, w and c separately represent the height, width and channels, thus Si ∈ R h * w * c. The response at the spatial position (x, y, z) in the j th feature map of the i th layer, defined as v ij (x, y, z), is given by, DISPLAYFORM0 where I, J and C are the height, width and channels of the kernel, respectively. activate is denoted as the activate functions. Discuss. Since the numbers of channel is arbitrary, 3D operators, such as 3D max, average pooling and 3D convolution, in the subsequent layers can, implicitly, learn the arbitrary partial correspondence between these local channels. Moreover, pooling operators do not define a local correspondence between channels, but filter the significant features in a space-channels cube, finally leading to a decrease in the resolution of convolutional features. Among these local operators, 3D convolution seems to be a relatively better encoding. In fact, we expect that the 3D encoding between channels cannot be imperceptibly influenced by the spatial encoding, then, encoding spatial and channels relationships simultaneously in the 3D cube, locality and linearity, is not the better model constructing the nonlinear functions. DISPLAYFORM1 The nonlinear channels aggregation networks for end-to-end global channels aggregation representations. Frame sequences from a video are taken as inputs and fed forward till the end of the nonlinear channels aggregation layer (NCAL). At the NCAL, the video sequences are encoded by channels aggregation operator to produce the same number of channels with the convolutional features, and then is fed to the next layer in the network. In general, pooling is necessary to make network training computationally feasible and, more fundamentally, to allow information aggregation over large areas of the input features. However, when these implementations are utilized to model the complex channel-level relationships, only linear and local channels representations are modeled in an implicit manner. Nevertheless, these temporal pooling and variants tend to in the reduced resolution, which will achieve coarse classifications. Another downside of the linear channels aggregation is that its locality, inadequate in representing such complex relationship, lacks the global representations across all the channels. To tackle with the problems above, it is desirable to explicitly capture the complex nonlinear relationship encoded in multiple channels. A general design principle of nonlinear channels aggregation layer (NCAL) is that its introduction cannot break the rest of CNN architecture, while keep a unique yet global channel-level encoding for final classification. Denoting F as the function of CNN, then, S i is calculated as follows: DISPLAYFORM0 In particular, let DISPLAYFORM1 ] denote the responses of NCAL, one for each of the K video clips. Then, we utilize the channels aggregation equation to encode the channel-level features to produce a compact feature maps group, DISPLAYFORM2 where G is the linear kernel functions of NCAL, H represents the normalization, m, n represent the spatial location, c denotes the channel location,r DISPLAYFORM3 Instead of many-to-one linear channels aggregation, the NCAL is implement a many-to-many mapping. As shown in Fig.2, each response of NCAL is aggregated by all the features at the same spatial location of S i producing the global yet compact representation, while no extra parameters are involved. Most importantly, in its input and output, the dimensions remain constant, which means NCAL can be positioned anywhere in the feedforward ConvNet architectures consisting of convolutional, fully connected, pooling and nonlinearity layers. Therefore such implementation enables to keep the rest components in the CNN structure constant with the original, and it is critical to utilize the benefit of transfer learning for other tasks. Next, we multiply every representation Ys i with individual convolutional feature S r in the forward propagation pipeline, DISPLAYFORM4 where E r represents the input features of NACL. Eq. keeps the parameters of layers before NCAL module not being greatly offset by the introduced of nonlinear channels aggregation rules. It can be seen that this implementation, avoiding the vanishing gradient problem, enlarges the gradient of loss with respect to the former layers vanishing gradient problem in the following part. Different from linear channels aggregation unit mapping the same spatial location across the multiple channels to a single feature, the NCAL aims to obtain the same number of global outputs as the inputs. Like the temporal linear encoding networks, one can utilize outer product for capturing the interaction of features with each other at all spatial locations, hence leading to a high-dimensional representation. For this reason, the Tensor Sketch algorithm is utilized to projects this high-dimensional space to a lower-dimensional space. We propose nonlinear channels aggregation layer (NCAL) to fit the complex function from all the channels, and to implement it end-to-end in a fast but efficient manner. Consider convolutional features truncated at a convolutional layer for K segments, these feature maps are matrices S = {S 1, S2, · · ·, SK} A channels aggregation function N: S 1,S 2, · ··, S K → Ys 1, Ys 2, · · ·, Ys K, aggregates c channels in each segment to output multiple encoded representations Ys 1, Ys 2, · · ·, Ys K, and this unit can be applied to the output of intermediate convolutional layers. In order to embed NCAL inside of CNNs as an intermediate layer, it is critical to compute the fast forward propagation, the gradients for the back-propagation step and to allow back-prop training through it. First, we implement a skilled and equivalent forward propagation. Based on this, the specific gradient of NCAL is derived for the back-propagation. We divide eq. into two parts, DISPLAYFORM0 DISPLAYFORM1 DISPLAYFORM2 where the normalization process H is removed for simplifying the process. We can find that, as shown in eq. and FORMULA9, such accumulation across all the channels for all the segments suffer enormous computation complexity equivalent to the outer product operators. To tackle the optimization difficulty, we redefine eq. and FORMULA9 to the novel matrix form, DISPLAYFORM3 where W L sr represents the corresponding transformation matrix. More specifically, the corresponding matrix form can be formulized as, DISPLAYFORM4 All the steps we have done so far, whether Y After this, the back-propagation step can be conducted by the gradient, DISPLAYFORM5 Discuss. Next, we discuss the difference of nonlinear channels aggregation layer compared with the traditional outer product. The benefit of the original outer product is that the ing feature captures multiplicative interactions at corresponding spatial locations. The main drawback of this feature is its high dimensionality. Once the ing features are followed by a fully-connected layer, many more parameters will be involved in the CNN, prone to network over-fitting. So far we can find that the NCAL, as well as the outer product, captures the global information representing the whole channels range and the NCAL is optimized by a fast yet compact implementation of matrix. Not only that, but each value in the weighting kernel in the NCAL is a single mapping for channels while the corresponding value in the outer product is many-to-one mapping, that is, outer product between the whole channels range views each channel equally important. In our proposed methods, all the responses of NCAL are obtained, only once, by the multiplication of transformation matrix with the convolutional feature maps of an intermediate layer which the outer-product implementation that every response of NACL is aggregated by every channel of one intermediate layer is transformed to matrix multiplication. We first give a detailed description for the experimental setup used in the paper. Then we evaluate and compare the performances of NCAN architectures in the following sections. Finally the change curse of loss with iteration is depicted for further analyzing its effect. UCF101. UCF101 is an action recognition dataset, collected from YouTube, with 101 video categories BID10. The videos in 101 action categories are grouped into 25 units, each unit containing 4-7 videos of an action. With large variations in camera motion, object appearance and pose, viewpoint, clutter , etc, it is a challenging dataset. HMDB51. HMDB51 is a large human motion dataset, containing 6849 clips divided into 51 action categories BID8. Due to that HMDB 51 is extracted from commercial movies as well as YouTube, it can represent a sufficient reflect of light conditions, situations and surroundings, which are close to the realistic video. Experimental setup. In view of the efficiency and accuracy, our architectures are built on the Inception V1, and pre-trained on ImageNet dataset. With the pre-trained model initializing our architectures all the layers are fine-tuned with the learning rate set to 10 −2, decrease it to 10 −3 after 12 epochs and stop the training process after 20 epochs. We position the NCAL behind the different convolutional layer, thus, various nonlinear aggregation networks with NCAL in different location are constructed to explore where the NCAL should be placed. In this section, we sample a single image from a video for training and evaluate the performances of CNN with NACL in various convolutional layers. Table I reports the experimental of CNNs with NCAL in various location on the split 1 of UCF101. We can see that the NCAL following the last convolutional layer outperforms the reproduction of the original architecture, and the NCAL performs well when placed behind the latter convolutional layer. As shown in Table I, the more distinctive the convolutional features are, the higher, on accuracy of recognition, is the effect of CNN with the NCAL. Based on this, we stack multiple frames to train CNNs with NCAL, embedded inside of the final convolutional layer, and report the performances on the three splits of UCF101 and HMDB51. Among the NCANs the CNN with multiple NCALs turn out to be most competitive. In practice, on accuracy of recognition, our NCANs make marginal improvements of 0.3%-0.5% (single frame in table I) and 0.3%-0.7% (stacked frames in table II). This can be ascribed to that the channel-level aggregation maps the convolutional maps to another distinctive feature space, in which each feature map is encoded by the hand-crafted rules of feature enhancement. Another intuitive conjecture is that the NCAL captures the global nonlinear property across the channels, ing in the improvement of accuracy. However, the purpose of this paper is not to boost the performance of action recognition but to facilitate convergence without scarifying classification accuracy. As another interesting noting, no extra parameters are updated in the NACL, in comparison to CNN which purely consisting of convolutional layer, pooling layer and fully-connected layers and so on. Thus, our models are constructed simple, and computationally efficient. Moreover, there is no need rebuilding the CNN models when introducing the NCAL. In the case seen in the training process of CNN, the network will almost lead to a multi-fold increase in training time once trained by stacking the still images. Therefore, it is critical to arrive at the same loss within fewer iterations. we evaluate in FIG2 Among the 6 splits of UCF101 and HMDB51, the proposed NACNs achieve superior performance of convergence than the standard CNN which is shown in FIG3. We conclude that nonlinear channels aggregation is a high-quality encoding of channels relationship, while to be a global fitting to complex channels functions as it aggregates all the channels features to a new dimension space. From a modeling perspective, we are interested in answering the following questions: what hand-crafted rules are best at taking advantage of global channels relationships in standard CNN architecture? How do such hand-crafted rules influence the convergence of a CNN? We limit the hand-crafted rule under the constraint of channels enhancement, and examine these questions empirically by taking a nonlinear channels aggregation to model the complex and nonlinear relationships across the channels. From a practical standpoint, currently there are few methods that conduct the global yet explicit channels dependency because exploring global channels relationship are significantly more difficult to ascertain and model. That is, most of channels relationships are local, vague and implicit. For the explicit and global channels aggregation, we propose nonlinear channels aggregation operations in the section 3.1. To keep the rest parts of the convolutional structure constant with the original CNNs, both linear and nonlinear channels aggregations use the convolutional features as input and the same number of channels as its output. Such setting is significant to extend the channels aggregation to other domains in the future.4.4 Why NCAL can facilitate network convergence. In addition to the convergence comparison of NCAN with the standard CNN, a theoretical analysis why NCAL can facilitate training network should be considered. We therefore perform a qualitative analysis in the forward and backward propagations. As explained in the previous section, the NCAL can be conducted as a nonlinear combination of data points along the channels and coefficients can be computed by using the eq.. In our work, if the normalization H is ignored, these parameters can also be defined as a one-to-one mapping in Equ..Each item in eq. FORMULA0 is greater than 1 when θ ∈ [0, In practice, owing to the periodicity of sinusoidal and cosine functions, the mapping process may in the mistake that the feature in different channels is mapped to the same value. This mapping mistake will make CNN confused and lose the ability of recognizing the features with characteristics with periodic relations, which reduces the accuracy of recognition. To tackle this problem, we propose the kernel function of NCAL G rescales θ to a single-valued interval θ ∈ ( 0, π 2), such that each feature in the convolutional is enhanced and aggregated by distinctive transformation items. For the back-propagation, the gradients have been derived in eq. FORMULA0, and fortunately is a symmetric positive definite matrix. It is interesting to note that the proposed nonlinear channels aggregation layer not only captures the global channels relationship but enlarges the gradient in the back-propagation, reducing the risk of vanishing gradients. We, furthermore, perform an analysis of nonlinear channels aggregation layer (NCAL). In general, representations on later convolutional layers tend to be somewhat local, where channels correspond to specific, natural parts instead of being dimensions in a completely distributed coding. That said, not all channels correspond to natural parts, ing in a possible and different decomposition of discriminative features than humans might expect. We can even assume that partial channels are useless or interfering, inevitably, this will become a key point that leads to retarded network convergence. In particular, NCANs perform well on CNNs with methods facilitating convergence, for instance, BN, dropout and optimization strategies, which indicate that the NCAL is complementary to the existing training tricks. We present nonlinear channels aggregation, a powerful and new, yet simple concept in the context of deep learning that captures the global channels relationship. We introduce a novel nonlinear channels aggregation layer (NCAL) and make a fast yet accurate implementation of NCAL, which allows us to embed the principle of complex channels encoding to the mainstream CNN architectures and back-propagate the gradients through NCALs. Experiments on video sequences demonstrate the effective power of nonlinear channels aggregation on facilitating training CNNs. In this paper we fit the complex channels relationships by capturing the global channels aggregation. Still, there seems to be some possible research directions that can be further expanded, modeling the nonlinear functions across channels. In the future it is beneficial to explore multiple-scale channel-levels by pyramid coding across channels. In sublimation, we can embed any hand-crafted rules, channels aggregation in the mainstream architectures, to making CNN working as we expect.

An architecture enables CNN trained on the video sequences converging rapidly

Represent the short summary for finding the corresponding scientific article (long)

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent text

Question: We present a new method for black-box adversarial attack. Unlike previous methods that combined transfer-based and scored-based methods by using the gradient or initialization of a surrogate white-box model, this new method tries to learn a low-dimensional embedding using a pretrained model, and then performs efficient search within the embedding space to attack an unknown target network. The method produces adversarial perturbations with high level semantic patterns that are easily transferable. We show that this approach can greatly improve the query efficiency of black-box adversarial attack across different target network architectures. We evaluate our approach on MNIST, ImageNet and Google Cloud Vision API, ing in a significant reduction on the number of queries. We also attack adversarially defended networks on CIFAR10 and ImageNet, where our method not only reduces the number of queries, but also improves the attack success rate. The wide adoption of neural network models in modern applications has caused major security concerns, as such models are known to be vulnerable to adversarial examples that can fool neural networks to make wrong predictions . Methods to attack neural networks can be divided into two categories based on whether the parameters of the neural network are assumed to be known to the attacker: white-box attack and black-box attack. There are several approaches to find adversarial examples for black-box neural networks. The transfer-based attack methods first pretrain a source model and then generate adversarial examples using a standard white-box attack method on the source model to attack an unknown target network (; ; ; a). The score-based attack requires a loss-oracle, which enables the attacker to query the target network at multiple points to approximate its gradient. The attacker can then apply the white-box attack techniques with the approximated gradient (; a;). A major problem of the transfer-based attack is that it can not achieve very high success rate. And transfer-based attack is weak in targeted attack. On the contrary, the success rate of score-based attack has only small gap to the white-box attack but it requires many queries. Thus, it is natural to combine the two black-box attack approaches, so that we can take advantage of a pretrained white-box source neural network to perform more efficient search to attack an unknown target black-box model. In fact, in the recent NeurIPS 2018 Adversarial Vision Challenge , many teams transferred adversarial examples from a source network as the starting point to carry out black-box boundary attack . N Attack also used a regression network as initialization in the score-based attack (a). The transferred adversarial example could be a good starting point that lies close to the decision boundary for the target network and accelerate further optimization. P-RGF used the gradient information from the source model to accelerate searching process. However, gradient information is localized and sometimes it is misleading. In this paper, we push the idea of using a pretrained white-box source network to guide black-box attack significantly further, by proposing a method called TRansferable EMbedding based Black-box Attack (TREMBA). TREMBA contains two stages: train an encoder-decoder that can effectively generate adversarial perturbations for the source network with a low-dimensional embedding space; apply NES (Natural Evolution Strategy) of to the low-dimensional embedding space of the pretrained generator to search adversarial examples for the target network. TREMBA uses global information of the source model, capturing high level semantic adversarial features that are insensitive to different models. Unlike noise-like perturbations, such perturbations would have much higher transferablity across different models. Therefore we could gain query efficiency by performing queries in the embedding space. We note that there have been a number of earlier works on using generators to produce adversarial perturbations in the white-box setting (; ;). While black-box attacks were also considered there, they focused on training generators with dynamic distillation. These early approaches required many queries to fine-tune the classifier for different target networks, which may not be practical for real applications. While our approach also relies on a generator, we train it as an encoder-decoder that produces a low-dimensional embedding space. By applying a standard black-box attack method such as NES on the embedding space, adversarial perturbations can be found efficiently for a target model. It is worth noting that the embedding approach has also been used in AutoZOOM . However, it only trained the autoencoder to reconstruct the input, and it did not take advantage of the information of a pretrained network. Although it also produces structural perturbations, these perturbations are usually not suitable for attacking regular networks and sometimes its performance is even worse than directly applying NES to the images . TREMBA, on the other hand, tries to learn an embedding space that can efficiently generate adversarial perturbations for a pretrained source network. Compared to AutoZOOM, our new method produces adversarial perturbation with high level semantic features that could hugely affect arbitrary target networks, ing in significantly lower number of queries. We summarize our contributions as follows: 1. We propose TREMBA, an attack method that explores a novel way to utilize the information of a pretrained source network to improve the query efficiency of black-box attack on a target network. 2. We show that TREMBA can produce adversarial perturbations with high level semantic patterns, which are effective across different networks, ing in much lower queries on MNIST and ImageNet especially for the targeted attack that has low transferablity. 3. We demonstrate that TREMBA can be applied to SOTA defended models . Compared with other black-box attacks, TREMBA increases success rate by approximately 10% while reduces the number of queries by more than 50%. There have been a vast literature on adversarial examples. We will cover the most relevant topics including white-box attack, black-box attack and defense methods. White-Box Attack White-box attack requires the full knowledge of the target model. It was first discovered by that adversarial examples could be found by solving an optimization problem with L-BFGS . Later on, other methods were proposed to find adversarial examples with improved success rate and efficiency (; ; b;). More recently, it was shown that generators can also construct adversarial noises with high success rate . Black-Box Attack Black-box attack can be divided into three categories: transfer-based, score-based and decision-based. It is well known that adversaries have high transferablity across different networks (a). Transfer-based methods generate adversarial noises on a source model and then transfer it to an unknown target network. It is known that targeted attack is harder than untargeted attack for transfer-based methods, and using an ensemble of source models can improve the success rate . Score-based attack assumes that the attacker can query the output scores of the target network. The attacker usually uses sampling methods to approximate the true gradient (; a; a; . AutoZOOM tried to improve the query efficiency by reducing the sampling space with a bilinear transformation or an autoencoder . (b) incorporated data and time prior to accelerate attacking. In contrast to the gradient based method, used combinatorial optimization to achieve good efficiency. In decision-based attack, the attacker only knows the output label of the classifier. Boundary attack and its variants are very powerful in this setting (; . In NeutIPS 2018 Adversarial Vision Challenge , some teams combined transfer-based attack and decision-based attack in their attacking methods . And in a similar spirit, N Attack also used a regression network as initialization in score-based attack (a). Gradient information from the surrogate model could also be used to accelerate the scored-based attack . Defense Methods Several methods have been proposed to overcome the vulnerability of neural networks. Gradient masking based methods add non-differential operations in the model, interrupting the backward pass of gradients. However, they are vulnerable to adversarial attacks with the approximated gradient (; a). Adversarial training is the SOTA method that can be used to improve the robustness of neural networks. Adversarial training is a minimax game. The outside minimizer performs regular training of the neural network, and the inner maximizer finds a perturbation of the input to attack the network. The inner maximization process can be approximated with FGSM , PGD , adversarial generator etc. Moreover, feature denoising can improve the robustness of neural networks on ImageNet . be an input, and let F (x) be the output vector obtained before the softmax layer. We denote F (x) i as the i-th component for the output vector and y as the label for the input. For un-targeted attack, our goal is to find a small perturbation δ such that the classifier predicts the wrong label, i.e. arg max F (x + δ) = y. And for targeted attack, we want the classifier to predicts the target label t, i.e. arg max F (x + δ) = t. The perturbation δ is usually bounded by p norm: δ p ≤ ε, with a small ε > 0. Adversarial perturbations often have high transferablity across different DNNs. Given a white-box source DNN F s with known architecture and parameters, we can transfer its white-box adversarial perturbation δ s to a black-box target DNN F t with reasonably good success rate. It is known that even if x + δ s fails to be an adversarial example, δ s can still act as a good starting point for searching adversarial examples using a score-based attack method. This paper shows that the information of F s can be further utilized to train a generator, and performing search on its embedding space leads to more efficient black-box attacks of an unknown target network F t. Adversarial perturbations can be generated by a generator network G. We explicitly divide the generator into two parts: an encoder E and a decoder D. The encoder takes the origin input x and output a latent vector z = E(x), where dim(z) dim(x). The decoder takes z as the input and outputs an adversarial perturbation δ = ε tanh(D(z)) with dim(δ) = dim(x). In our new method, we will train the generator G so that δ = ε tanh(G(x)) can fool the source network F s. Suppose we have a training set {(x 1, y 1),..., (x n, y n)}, where x i denotes the input and y i denotes its label. For un-targeted attack, we train the desired generator by minimizing the hinge loss used in the C&W attack : And for targeted, we use where t denotes the targeted class and κ is the margin parameter that can be used to adjust transferability of the generator. A higher value of κ leads to higher transferability to other models . We focus on ∞ norm in this work. By adding point-wise tanh function to an unnormalized output D(z), and scaling it with ε, δ = ε tanh(D(z)) is already bounded as δ ∞ < ε. Therefore we employ this transformation, so that we do not need to impose the infinity norm constraint explicitly. While hinge loss is employed in this paper, we believe other loss functions such the cross entropy loss will also work. Given a new black-box DNN classifier F t (x), for which we can only query its output at any given point x. As in (a;), we can employ NES to approximate the gradient of a properly defined surrogate loss in order to find an adversarial example. Denote the surrogate loss by L, rather than calculating ∇ δ L(x+δ, y) directly, NES update δ by using The expectation can be approximated by taking finite samples. And we could use the following equation to iteratively update δ: where η is the learning rate, b is the minibatch sample size, ω k is the sample from the gaussian distribution and [−ε,ε] represents a clipping operation, which projects δ onto the ∞ ball. The sign function provides an approximation of the gradient, which has been widely used in adversarial attack (a;). However, it is observed that more effective attacks can be obtained by removing the sign function (b). Therefore in this work, we remove the sign function from Eqn and directly use the estimated gradient. Instead of performing search on the input space, TREMBA performs search on the embedding space z. The generator G explores the weakness of the source DNN F s so that D produces perturbations that can effective attack F s. For a different unknown target network F t, we show that our method can still generate perturbations leading to more effective attack of F t. Given an input x and its label y, we choose a starting point z 0 = E(x). The gradient of z t given by NES can be estimated as: where ν k is the sample from the gaussian distribution N (z t, σ 2). Moreover, z t is updated with stochastic gradient descent. The detailed procedure is presented in Algorithm 1. We do not need to do projection explicitly since δ already satisfies δ ∞ < ε. Next we shall briefly explain why applying NES on the embedding space z can accelerate the search process. Adversarial examples can be viewed as a distribution lying around a given input. Usually this distribution is concentrated on some small regions, making the search process relatively slow. After training on the source network, the adversarial perturbations of TREMBA would have high level semantic patterns that are likely to be adversarial patterns of the target network. Therefore searching over z is like searching adversarial examples in a lower dimensional space containing likely adversarial patterns. The distribution of adversarial perturbations in this space is much less concentrated. It is thus much easier to find effective adversarial patterns in the embedding space. We evaluated the number of queries versus success rate of TREMBA on undefended network in two datasets: MNIST and ImageNet . Moreover, we evaluated the efficiency of our method on adversarially defended networks in CIFAR10 and ImageNet. We also attacked Google Cloud Vision API to show TREMBA can generalize to truly black-box model. 1 We used the hinge loss from Eqn 1 and 2 as the surrogate loss for un-targeted and targeted attack respectively. We compared TREMBA to four methods: NES: Method introduced by (a), but without the sign function for reasons explained earlier. Trans-NES: Take an adversarial Algorithm 1 Black-Box adversarial attack on the embedding space Input: Target Network F t; Input x and its label y or the target class t; Encoder E; Decoder D; Standard deviation σ; Learning rate η; Sample size b; Iterations T; Bound for adversarial perturbation ε Output: Adversarial perturbation δ 1: z 0 = E(x) 2: for t = 1 to T do 3: perturbation generated by PGD or FGSM on the source model to initialize NES. AutoZOOM: Attack target network with an unsupervised autoencoder described in . For fair comparisons with other methods, the strategy of choosing sample size was removed. P-RGF: Prior-guided random gradient-free method proposed in . The P-RGF D (λ *) version was compared. We also combined P-RGF with initialization from Trans-NES PGD to form a more efficient method for comparison, denoted by Trans-P-RGF. Since different methods achieve different success rates, we need to compare their efficiency at different levels of success rate. For method i with success rate s i, the average number of queries is q i for all success examples. Let q * denote the upper limit of queries, we modified the average number of queries to be q * which unified the level of success rate and treated queries of failure examples as the upper limit on the number of queries. Average queries sometimes could be misleading due to the the heavy tail distribution of queries. Therefore we plot the curve of success rate at different query levels to show the detailed behavior of different attacks. The upper limit on the number of queries was set to 50000 for all datasets, which already gave very high success rate for nearly all the methods. Only correctly classified images were counted towards success rate and average queries. And to fairly compare these methods, we chose the sample size to be the same for all methods. We also added momentum and learning decay for optimization. And we counted the queries as one if its starting point successfully attacks the target classifier. The learning rate was fine-tuned for all algorithms. We listed the hyperparameters and architectures of generators and classifiers in Appendix B and C. We trained four neural networks on MNIST, denoted by ConvNet1, ConvNet1*, ConvNet2 and FCNet. ConvNet1* and ConvNet1 have the same architecture but different parameters. All the network achieved about 99% accuracy. The generator G was trained on ConvNet1* using all images from the training set. Each attack was tested on images from the MNIST test set. The limit of ∞ was ε = 0.2. We performed un-targeted attack on MNIST. Table 1 lists the success rate and the average queries. Although the success rate of TREMBA is slightly lower than Trans-NES in ConvNet1 and FCNet, their success rate are already close to 100% and TREMBA achieves about 50% reduction of queries compared with other attacks. In contrast to efficient attack on ImageNet, P-RGF and Trans-P-RGF behaves very bad on MNIST. Figure 4.1 shows that TREMBA consistently achieves higher success rate at nearly all query levels. We randomly divided the ImageNet validation set into two parts, containing 49000 and 1000 images respectively. The first part was used as the training data for the generator G, and the second part was used for evaluating the attacks. We evaluated the efficiency of all adversarial attacks on VGG19 , Resnet34 , DenseNet121 and MobilenetV2 . All networks were downloaded using torchvision package. We set ε = 0.03125. ) as the source model to improve transferablity for both targeted and un-targeted attack. TREMBA, Trans-NES, P-RGF and Trans-P-RGF all used the same source model for fair comparison. We chose several target class. Here, we show the of attacking class 0 (tench) in Table 2 and Figure 2. And we leave the of attacking other classes in Appendix A.1. The average queries for TREMBA is about 1000 while nearly all the average queries for other methods are more than 6000. TREMBA also achieves much lower queries for un-targeted attack on ImageNet. The is shown in Appendix A.2 due to space limitation. And we also compared TREMBA with CombOpt in the Appendix A.9. Figure 3 shows the adversarial perturbations of different methods. Unlike adversarial perturbations produced by PGD, the perturbations of TREMBA reflect some high level semantic patterns of the targeted class such as the fish scale. As neural networks usually capture such patterns for classification, the adversarial perturbation of TREMBA would be more easy to transfer than the noise-like perturbation produced by PGD. Therefore TREMBA can search very effectively for the target network. More examples of perturbations of TREMBA are shown in Appendix A.3. We performed attack on different ensembles of source model, which is shown in Appendix A.4. TREMBA outperforms the other methods in different ensemble model. And more source networks lead to better transferability for TREMBA, Trans-NES and Trans-P-RGF. Varying ε: We also changed ε and performed attack on ε = 0.02 and ε = 0.04. As shown in Appendix A.5, TREMBA still outperforms the other methods despite using the G trained on ε = 0.03125. We also show the of TREMBA for commonly used ε = 0.05. Sample size and dimension the embedding space: To justify the choice of sample size, we performed a hyperparameter sweep over b and the is shown in Appendix A.6. And we also changed the dimension of the embedding space for AutoZOOM and Trans-P-RGF. As shown in Appendix A.7, the performance gain of TREMBA does not purely come from the diminishing of dimension of the embedding space. This section presents the for attacking defended networks. We performed un-targeted attack on two SOTA defense methods on CIFAR10 and ImageNet. MNIST is not studied since it is already robust against very strong white-box attacks. For CIFAR10, the defense model was going through PGD minimax training . We directly used their model as the source network 2, denoted by WResnet. To test whether these methods can transfer to a defended network with a different architecture, we trained a defended ResNeXt using the same method. For ImageNet, we used the SOTA model 3 from . We used "ResNet152 Denoise" as the source model and transfered adversarial perturbations to the most robust "ResNeXt101 DenoiseAll". Following the previous settings, we set ε = 0.03125 for both CIFAR10 and ImageNet. As shown in Table 3, TREMBA achieves higher success rates with lower number of queries. TREMBA achieves about 10% improvement of success rate while the average queries are reduced by more than 50% on ImageNet and by 80% on CIFAR10. The curves in Figure 4 (a) and 4(b) show detailed behaviors. The performance of AutoZOOM surpasses Trans-NES on defended models. We suspect that low-frequency adversarial perturbations produced by AutoZOOM will be more suitable to fool the defended models than the regular networks. However, the patterns learned by AutoZOOM are still worse than adversarial patterns learned by TREMBA from the source network. An optimized starting point for TREMBA: z 0 = E(x) is already a good starting point for attacking undefended networks. However, the capability of generator is limited for defended networks . Therefore, z 0 may not be the best starting point we can get from the defended source network. To enhance the usefulness of the starting point, we optimized z on the source network by gradient descent and found The method is denoted by TREMBA OSP (TREMBA with optimized starting point). Figure 4 shows TREMBA OSP has higher success rate at small query levels, which means its starting point is better than TREMBA. We also attacked the Google Cloud Vision API, which was much harder to attack than the single neural network. Therefore we set ε = 0.05 and perform un-targeted attack on the API, changing the top1 label to whatever is not on top1 before. We chose 10 images for the ImageNet dataset and set query limit to be 500 due to high cost to use the API. As shown Table 4, TREMBA achieves much higher accuracy success rate and lower number of queries. We show the example of successfully attacked image in Appendix A.8. We propose a novel method, TREMBA, to generate likely adversarial patterns for an unknown network. The method contains two stages: training an encoder-decoder to generate adversarial perturbations for the source network; search adversarial perturbations on the low-dimensional embedding space of the generator for any unknown target network. Compared with SOTA methods, TREMBA learns an embedding space that is more transferable across different network architectures. It achieves two to six times improvements in black-box adversarial attacks on MNIST and ImageNet and it is especially efficient in performing targeted attack. Furthermore, TREMBA demonstrates great capability in attacking defended networks, ing in a nearly 10% improvement on the attack success rate, with two to six times of reductions in the number of queries. TREMBA opens up new ways to combine transfer-based and score-based attack methods to achieve higher efficiency in searching adversarial examples. For targeted attack, TREMBA requires different generators to attack different classes. We believe methods from conditional image generation may be combined with TREMBA to form a single generator that could attack multiple targeted classes. We leave it as a future work. A EXPERIMENT A.1 TARGETED ATTACK ON IMAGENET Figure 9 shows of the targeted attack on dipper, American chameleon, night snake, ruffed grouse and black swan. TREMBA achieves much higher success rate than other methods at almost all queries level. We used the same source model from targeted attack as the source model for un-targeted attack. We report our evaluation in Table 5 and Figure 5. Compared with Trans-P-RGF, TREMBA reduces the number of queries by more than a half in ResNet34, DenseNet121 and MobilenetV2. Searching in the embedding space of generator remains very effective even when the target network architecture differs significantly from the networks in the source model. Figure 5: The success rate of un-targeted black-box adversarial attack at different query levels for undefended ImageNet models. Figure 10 shows some examples of adversarial perturbations produced by TREMBA. The first column is one image of the target class and other columns are examples of perturbations (amplified by 10 times). It is easy to discover some features of the target class in the adversarial perturbation such as the feather for birds and the body for snakes. We chose two more source ensemble models for evaluation. The first ensemble contains VGG16 and Squeezenet. And the second ensemble is consist of VGG16, Squeezenet and Googlenet. Figure 6 shows our for targeted attack for ImageNet. We only compared Trans-NES PGD and Trans-P-RGF since they are the best variants from Trans-NES and P-RGF. Figure 6: We show the success rate at different query levels for targeted attack for different ensemble source networks. V represents VGG16; S represents Squeezenet; G represents Googlenet; R represents Resnet18 A.5 VARYING ε We chose ε = 0.02 and ε = 0.04 and performed targeted attack on ImageNet. Although TREMBA used the same model that is trained on ε = 0.03125, it still outperformed other methods, which shows that TREMBA can also generalize to different strength of adversarial attack with different ε. For the commonly used ε = 0.05, TREMBA also performs well. The are shown in Table 6, Table 7, and Figure 8. We performed a hyperparameter sweep over b on Densenet121 on un-targeted attack on ImageNet. b = 20 may not be the best choice Trans-NES, but it is not the best for TREMBA, either. Generally, the performance is not very sensitive to b, and TREMBA will also outperform other methods even if we fine-tune the sample size for all the methods. We slightly changed the architecture of the autoencoder by adding max pooling layers and changing the number of filters and perform un-targeted attack on ImageNet. More specifically, we added additional max pooling layers after the first and the fourth convolution layers and changed the number of filters of the last layer in the encoder to be 8. Thus, the dimension of the embedding space would be 8 × 8 × 8. And we also changed the factor of bilinear sampling in the decoder. The remaining settings are the same in Appendix A.2. As shown in Table 9, this autoencoder is even worse than the original autoencoder despite small dimension of the embedding space. In addition, we also changed to dimension of the data-dependent prior of Trans-P-RGF to match the dimension of TREMBA, whose performance is also not better than before. They show that simply diminishing the size of the embedding space may not lead to better performance. The performance gain of TREMBA comes beyond the effect of diminishing the dimension of the embedding space. A.8 EXAMPLES OF ATTACKING GOOGLE CLOUD VISION API Figure 11 shows one example of attacking Google Cloud Vision API. TREMBA successfully make the shark to be classified as green. CombOpt is one of the SOTA score-based black-box attack. We compared our method with it on the targeted and un-targeted attack on Imagenet. The targeted attack is 0 and ε = 0.03125. As shown in Table 10 and Table 11, TREMBA requires much lower queries than CombOpt. It demonstrates the great improvement by combining the transfer-based and score-based attack. B ARCHITECTURE OF CLASSIFIERS AND GENERATORS B.1 CLASSIFIER Table 12 lists the architectures of ConvNet1, ConvNet2 and FCNet. The architecture of ResNeXt used in CIFAR10 is from https://github.com/prlz77/ResNeXt.pytorch. We set the depth to be 20, the cardinality to be 8 and the widen factor to be 4. Other architectures of classifiers are specified in the corresponding paper. B.2 GENERATOR Table 13 lists the architectures of generator for three datasets. For AutoZOOM, we find our architectures are not suitable and use the same generators in the corresponding paper. We trained the generators with learning rate starting at 0.01 and decaying half every 50 epochs. The whole training process was 500 epochs. The batch size was determined by the memory of GPU. Specifically, we set batch size to be 256 for MNIST and CIFAR10 defense model, 64 for ImageNet model. All large κ will work well for our method and we chose κ = 200.0. All the experiments were performed using pytorch on NVIDIA RTX 2080Ti. Table 14 to 19 list the hyperparameters for all the algorithms. The learning rate was fine-tuned for all the algorithms. We set sample size b = 20 for all the algorithms for fair comparisons. Table 18: Hyperparameters for TREMBA. Un-targeted Targeted

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We present a new method that combines transfer-based and scored black-box adversarial attack, improving the success rate and query efficiency of black-box adversarial attack across different network architectures.

Represent the TLDR for finding the corresponding scientific article.

This paper identifies classes of problems for which adversarial examples are inescapable, and derives fundamental bounds on the susceptibility of any classifier to adversarial examples.