Question: Represent the article for finding a single-sentence TLDR

Question: We study model recovery for data classification, where the training labels are generated from a one-hidden-layer fully -connected neural network with sigmoid activations, and the goal is to recover the weight vectors of the neural network. We prove that under Gaussian inputs, the empirical risk function using cross entropy exhibits strong convexity and smoothness uniformly in a local neighborhood of the ground truth, as soon as the sample complexity is sufficiently large. This implies that if initialized in this neighborhood, which can be achieved via the tensor method, gradient descent converges linearly to a critical point that is provably close to the ground truth without requiring a fresh set of samples at each iteration. To the best of our knowledge, this is the first global convergence guarantee established for the empirical risk minimization using cross entropy via gradient descent for learning one-hidden-layer neural networks, at the near-optimal sample and computational complexity with respect to the network input dimension. Neural networks have attracted a significant amount of research interest in recent years due to the success of deep neural networks BID18 in practical domains such as computer vision and artificial intelligence BID24 BID15 BID27. However, the theoretical underpinnings behind such success remains mysterious to a large extent. Efforts have been taken to understand which classes of functions can be represented by deep neural networks BID7 BID16 BID0 ), when (stochastic) gradient descent is effective for optimizing a non-convex loss function BID8, and why these networks generalize well BID1.One important line of research that has attracted extensive attention is a model-recovery setup, i.e., given that the training samples (x i, y i) ∼ (x, y) are generated i.i.d. from a distribution D based on a neural network model with the ground truth parameter W, the goal is to recover the underlying model parameter W, which is important for the network to generalize well BID22. Previous studies along this topic can be mainly divided into two types of data generations. First, a regression problem, for example, assumes that each sample y is generated as y = 1 K K k=1 φ(w k x), where w k ∈ R d is the weight vector of the kth neuron, 1 ≤ k ≤ K, and the input x ∈ R d is Gaussian. This type of regression problem has been studied in various settings. In particular, BID28 studied the single-neuron model under ReLU activation, BID38 ) studied the onehidden-layer multi-neuron network model, and BID19 ) studied a two-layer feedforward networks with ReLU activations and identity mapping. Second, for a classification problem, suppose each label y ∈ {0, 1} is drawn under the conditional distribution P(y = 1|x) = 1 K K k=1 φ(w k x), where w k ∈ R d is the weight vector of the kth neuron, 1 ≤ k ≤ K, and the input x ∈ R d is Gaussian. Such a problem has been studied in BID21 in the case with a single neuron. For both the regression and the classification settings, in order to recover the neural network parameters, all previous studies considered (stochastic) gradient descent over the squared loss, i.e., qu (W ; x, y) = DISPLAYFORM0 Furthermore, previous studies provided two types of statistical guarantees for such model recovery problems using the squared loss. More specifically, BID38 showed that in the local neighborhood of the ground truth, the Hessian of the empirical loss function is positive definite for each given point under independent high probability event. Hence, their guarantee for gradient descent to converge to the ground truth requires a fresh set of samples at every iteration, thus the total sample complexity will depend on the number of iterations. On the other hand, studies such as BID21 BID28 establish certain types of uniform geometry such as strong convexity so that resampling per iteration is not needed for gradient descent to have guaranteed linear convergence as long as it enters such a local neighborhood. However, such a stronger statistical guarantee without per-iteration resampling have only been shown for the squared loss function. In this paper, we aim at developing such a strong statistical guarantee for the loss function in eq., which is much more challenging but more practical than the squared loss for the classification problem. This study provides the first performance guarantee for the recovery of one-hidden-layer neural networks using the cross entropy loss function, to the best of our knowledge. More specifically, our contributions are summarized as follows.• For multi-neuron classification problem with sigmoid activations, we show that, if the input is Gaussian, the empirical risk function f n (W) = 1 n n i=1 (W ; x i) based on the cross entropy loss in eq. is uniformly strongly convex in a local neighborhood of the ground truth W of size O(1/K 3/2) as soon as the sample size is O(dK 5 log 2 d), where d is the input dimension and K is the number of neurons.• We further show that, if initialized in this neighborhood, gradient descent converges linearly to a critical point W n (which we show to exist), with a sample complexity of O(dK 5 log 2 d), which is near-optimal up to a polynomial factor in K and log d. Due to the nature of quantized labels here, the recover of W is only up to certain statistical accuracy, and W n converges to W at a rate of O(dK 9/2 log n/n) in the Frobenius norm. Furthermore, such a convergence guarantee does not require a fresh set of samples at each iteration due to the uniform strong convexity in the local neighborhood. To obtain -accuracy, it requires a computational complexity of O(ndK 2 log(1/)).• We adopt the tensor method proposed in BID38, and show it provably provides an initialization in the neighborhood of the ground truth. In particular, our proof replaces the homogeneous assumption on activation functions in BID38 ) by a mild condition on the curvature of activation functions around W, which holds for a larger class of activation functions including sigmoid and tanh. In order to analyze the challenging cross-entropy loss function, our proof develops various new machineries in order to exploit the statistical information of the geometric curvatures, including the gradient and Hessian of the empirical risk, and to develop covering arguments to guarantee uniform concentrations. Our technique also yields similar performance guarantees for the classification problem using the squared loss in eq., which we omit due to space limitations, as it is easier to analyze than cross entropy. Due to page limitations we focus on the most relevant literature on theoretical and algorithmic aspects of learning shallow neural networks via nonconvex optimization. The parameter recovery viewpoint is relevant to the success of non-convex learning in signal processing problems such as matrix completion, phase retrieval, blind deconvolution, dictionary learning and tensor decomposition BID31 BID6 BID13 BID30 BID2, to name a few. The statistical model for data generation effectively removes worst-case instances and allows us to focus on average-case performance, which often possess much benign geometric properties that enable global convergence of simple local search algorithms. The studies of one-hidden-layer network model can be further categorized into two classes, landscape analysis and model recovery. In the landscape analysis, it is known that if the network size is large enough compared to the data input, then there are no spurious local minima in the optimization landscape, and all local minima are global BID3 BID25 BID23. For the case with multiple neurons (2 ≤ K ≤ d) in the under-parameterized setting, the work of Tian BID33 studied the landscape of the population squared loss surface with ReLU activations. In particular, there exist spurious bad local minima in the optimization landscape BID26 even at the population level. Zhong et. al. BID38 provided several important characterizations for the local Hessian for the regression problem for a variety of activation functions for the squared loss. In the model recovery problem, the number of neurons is smaller than the dimension of inputs. In the case with a single neuron (K = 1), under Gaussian input, BID28 showed that gradient descent converges linearly when the activation function is ReLU, i.e. φ(z) = max{z, 0}, with a zero initialization, as long as the sample complexity is O(d) for the regression problem. On the other end, BID21 showed that when φ(·) has bounded first, second and third derivatives, there is no other critical points than the unique global minimum (within a constrained region of interest), and (projected) gradient descent converges linearly with an arbitrary initialization, as long as the sample complexity is O(d log 2 d) with sub-Gaussian inputs for the classification problem using the squared loss. Moreover, BID38 shows that the ground truth From a technical perspective, our study differs from all the aforementioned work in that the cross entropy loss function we analyze has a very different form. Furthermore, we study the model recovery classification problem under the multi-neuron case, which has not been studied before. Finally, we note that several papers study one-hidden-layer or two-layer neural networks with different structures under Gaussian input. For example, BID9 b; BID37 ) studied the non-overlapping convolutional neural network, BID19 ) studied a two-layer feedforward networks with ReLU activations and identity mapping, and BID11 introduced the Porcupine Neural Network. These are not directly comparable to ours since both the networks and the loss functions are different. The rest of the paper is organized as follows. Section 2 describes the problem formulation. Section 3 presents the main on local geometry and local linear convergence of gradient descent. Section 4 discusses the initialization method. Numerical examples are demonstrated in Section 5, and finally, are drawn in Section 6.Throughout this paper, we use boldface letters to denote vectors and matrices, e.g. w and W. The transpose of W is denoted by W, and W, W F denote the spectral norm and the Frobenius norm. For a positive semidefinite (PSD) matrix A, we write A 0. The identity matrix is denoted by I. The gradient and the Hessian of a function f (W) is denoted by ∇f (W) and ∇ 2 f (W), respectively. Let σ i (W) denote the i-th singular value of W. Denote · ψ1 as the sub-exponential norm of a random variable. We use c, C, C 1,... to denote constants whose values may vary from line to line. For nonnegative functions f (x) and g(x), f (x) = O (g(x)) means there exist positive constants c and a such that f (x) ≤ cg(x) for all x ≥ a; f (x) = Ω (g(x)) means there exist positive constants c and a such that f (x) ≥ cg(x) for all x ≥ a. We first describe the generative model for training data, and then describe the gradient descent algorithm for learning the network weights. Suppose we are given n training samples {(x i, y i)} n i=1 ∼ (x, y) that are drawn i.i.d., where x ∼ N (0, I). Assume the activation function is sigmoid, i.e. φ (z) = 1/(1 + e −z) for all z. Conditioned on x ∈ R d, we consider the classification setting, where y is mapped to a discrete label using the one-hidden layer neural network model as follows: DISPLAYFORM0 and P(y = 0|x) = 1 − P(y = 1|x), where K is the number of neurons. Our goal is to estimate W = [w 1, · · ·, w K], via minimizing the following empirical risk function: DISPLAYFORM1 where (W ; x):= (W ; x, y) is the cross entropy loss, i.e., the negative log-likelihood function, i.e., DISPLAYFORM2 With slight abuse of notation, we denote the gradient and Hessian of (W ; x) with respect to the vector w. To estimate W, since is a highly nonconvex function, vanilla gradient descent with an arbitrary initialization may get stuck at local minima. Therefore, we implement the gradient descent algorithm with a well-designed initialization scheme that is described in detail in Section 4. The update rule is given as DISPLAYFORM0, where η is the step size. The algorithm is summarized in Algorithm 1. DISPLAYFORM1 We note that throughout the execution of the algorithm, the same set of training samples is used which is the standard implementation of gradient descent. This is in sharp contrast to existing work such as BID38 that employs the impractical scheme of resampling, where a fresh set of training samples is used at every iteration of gradient descent. Before stating our main , we first introduce an important quantity regarding φ(z) that captures the geometric properties of the loss function, distilled in BID38. Figure 1: ρ (σ) for sigmoid activation. DISPLAYFORM0 Note that the definition here is different from that in (b, Property 3 .2) but consistent with (b, Lemma D.4) which removes the third term in (b, Property 3.2). For the activation function considered in this paper, the first two terms suffice. We depict ρ(σ) as a function of σ in a certain range for the sigmoid activation in Fig. 1. It is easy to observe that ρ(σ) > 0 for all σ > 0. We first characterize the local strong convexity of f n (W) in a neighborhood of the ground truth W. Let B (W, r) denote a Euclidean ball centered at W ∈ R d×K with a radius r, i.e. DISPLAYFORM0 Let σ i:= σ i (W) denote the i-th singular value of W. Let the condition number be κ = σ 1 /σ K, and λ = DISPLAYFORM1 The following theorem guarantees the Hessian of the empirical risk function f n (W) in the local neighborhood of W is positive definite with high probability. Theorem 1. For the classification model with sigmoid activation function, assume W F ≤ 1, then there exists some constant C, such that if DISPLAYFORM2 then with probability at least 1 − d −10, for all W ∈ B(W, r), DISPLAYFORM3 hold, where r:= min DISPLAYFORM4 We note that all column permutations of W are equivalent global minima of the loss function, and Theorem 1 applies to all such permutation matrices of W. The proof of Theorem 1 is outlined in Appendix A. Theorem 1 guarantees that the Hessian of the empirical cross-entropy loss function f n (W) is positive definite (PD) in a neighborhood of the ground truth W, as long as ρ(σ K) > 0 (i.e. W is full-column rank), when the sample size n is sufficiently large for the sigmoid activation. The bounds in Theorem 1 depend on the dimension parameters of the network (n and K), as well as the activation function and the ground truth (ρ(σ K), λ). As a special case, suppose W is composed of orthonormal columns with ρ(σ K) = O, κ = 1, λ = 1. Then, Theorem 1 guarantees DISPLAYFORM5, as soon as the sample complexity n = Ω(dK 5 log 2 d). The sample complexity is order-wise near-optimal in d up to polynomial factors of K and log d, since the number of unknown parameters is dK. For the classification problem, due to the nature of quantized labels, W is no longer a critical point of f n (W). By the strong convexity of the empirical risk function f n (W) in the local neighborhood of W, there can exist at most one critical point in B(W, r), which is the unique local minimizer in B (W, r) if it exists. The following theorem shows that there indeed exists such a critical point W n, which is provably close to the ground truth W, and gradient descent converges linearly to W n. Theorem 2. For the classification model with sigmoid activation function, and assume W F ≤ 1, there exist some constants C, C 1 > 0 such that if the sample size n ≥ C · dK DISPLAYFORM0 then with probability at least 1 − d −10, there exists a unique critical point W n in B(W, r) with DISPLAYFORM1 Moreover, if the initial point W 0 ∈ B (W, r), then gradient descent converges linearly to W n, i.e. DISPLAYFORM2 where DISPLAYFORM3, as long as the step size DISPLAYFORM4 Similarly to Theorem 1, Theorem 2 also holds for all column permutations of W. The proof can be found in Appendix B. Theorem 2 guarantees that there exists a critical point W n in B(W, r) which converges to W at the rate of O(K 9/4 d log n/n), and therefore W can be recovered consistently as n goes to infinity. Moreover, gradient descent converges linearly to W n at a linear rate, as long as it is initialized in the basin of attraction. To achieve -accuracy, i.e. DISPLAYFORM5 requires a computational complexity of O ndK 2 log (1/), which is linear in n, d and log(1/). Our initialization adopts the tensor method proposed in BID38. In this section, we first briefly describe this method, and then present the performance guarantee of the initialization with remarks on the differences from that in BID38. This subsection briefly introduces the tensor method proposed in BID38, to which a reader can refer for more details. We first define a product ⊗ as follows. If v ∈ R d is a vector and I is the identity matrix, then DISPLAYFORM0 Definition 3. Let α ∈ R d denote a randomly picked vector. We define P 2 and P 3 as follows: DISPLAYFORM1 1 where j 2 = min{j ≥ 2|M j = 0}, and P 3 = M j3 (I, I, I, α, · · ·, α), where j 3 = min{j ≥ 3|M j = 0}.We further denote w = w/ w. The initialization algorithm based on the tensor method is summarized in Algorithm 2, which includes two major steps. Step 1 first estimates the direction of each column of W by decomposing P 2 to approximate the subspace spanned by {w 1, w 2, · · ·, w K} (denoted by V), then reduces the third-order tensor P 3 to a lower-dimension tensor R 3 = P 3 (V, V, V) ∈ R K×K×K, and applys non-orthogonal tensor decomposition on R 3 to output the estimate s i V w i, where s i ∈ {1, −1} is a random sign. Step 2 approximates the magnitude of w i and the sign s i by solving a linear system of equations. For the classification problem, we make the following technical assumptions, similarly in (b, Assumption 5. 3) for the regression problem. Assumption 1. The activation function φ(z) satisfies the following conditions: DISPLAYFORM0 2. At least one of M 3 and M 4 is non-zero. Furthermore, we do not require the homogeneous assumption ((i.e., φ(az) = a p z for an integer p)) required in BID38, which can be restrictive. Instead, we assume the following condition on the curvature of the activation function around the ground truth, which holds for a larger class of activation functions such as sigmoid and tanh. Assumption 2. Let l 1 be the index of the first nonzero M i where i = 1,..., 4. For the activation function φ (·), there exists a positive constant δ such that m l1,i (·) is strictly monotone over the interval (w i − δ, w i + δ), and the derivative of m l1,i (·) is lower bounded by some constant for all i. We next present the performance guarantee for the initialization algorithm in the following theorem. Theorem 3. For the classification model, under Assumptions 1 and 2, if the sample size n ≥ dpoly (K, κ, t, log d, 1/), then the output W 0 ∈ R d×K of Algorithm 2 satisfies DISPLAYFORM1 with probability at least 1 − d−Ω(t).The proof of Theorem 3 consists of (a) showing the estimation of the direction of W is sufficiently accurate and (b) showing the approximation of the norm of W is accurate enough. Our proof of part (a) is the same as that in BID38 ), but our argument in part (b) is different, where we relax the homogeneous assumption on activation functions. More details can be found in the supplementary materials in Appendix C. In this section, we first implement gradient descent to verify that the empirical risk function is strongly convex in the local region around W. If we initialize multiple times in such a local region, it is expected that gradient descent converges to the same critical point W n, with the same set of training samples. Given a set of training samples, we randomly initialize multiple times, and then calculate the variance of the output of gradient descent. Denote the output of the th run as w DISPLAYFORM0, where L = 20 is the total number of random initializations. Adopted in BID21, it quantifies the standard deviation of the estimator W n under different initializations with the same set of training samples. We say an experiment is successful, if SD n ≤ 10 −2.Figure 2 (a) shows the successful rate of gradient descent by averaging over 50 sets of training samples for each pair of n and d, where K = 3 and d = 15, 20, 25 respectively. The maximum iterations for gradient descent is set as iter max = 3500. It can be seen that as long as the sample complexity is large enough, gradient descent converges to the same local minima with high probability. We next show that the statistical accuracy of the local minimizer for gradient descent if it is initialized close enough to the ground truth. Suppose we initialize around the ground truth such that W 0 − W F ≤ 0.1 · W F. We calculate the average estimation error as DISPLAYFORM0 Monte Carlo simulations with random initializations. FIG1 shows the average estimation error with respect to the sample complexity when K = 3 and d = 20, 35, 50 respectively. It can be seen that the estimation error decreases gracefully as we increase the sample size and matches with the theoretical prediction of error rates reasonably well. We further compare the performance of gradient descent algorithm applied to both the cross entropy loss and the squared loss, respectively. As shown in FIG1, when K = 3, d = 20, cross entropy loss with gradient descent achieves a much lower error than the squared loss. Clearly, the cross entropy loss is favored in the classification problem over the squared loss. In this paper, we have studied the model recovery of a one-hidden-layer neural network using the cross entropy loss in a multi-neuron classification problem. In particular, we have characterized the sample complexity to guarantee local strong convexity in a neighborhood (whose size we have characterized as well) of the ground truth when the training data are generated from a classification model. This guarantees that with high probability, gradient descent converges linearly to the ground truth if initialized properly. In the future, it will be interesting to extend the analysis in this paper to more general class of activation functions, particularly ReLU-like activations; and more general network structures, such as convolutional neural networks BID10 BID37. To begin, denote the population loss function as DISPLAYFORM0 where the expectation is taken with respect to the distribution of the training sample (x; y).The proof of Theorem 1 follows the following steps:1. We first show that the Hessian ∇ 2 f (W) of the population loss function is smooth with respect to ∇ 2 f (W) (Lemma 1);2. We then show that ∇ 2 f (W) satisfies local strong convexity and smoothness in a neighborhood of W, B(W, r) with appropriately chosen radius by leveraging similar properties of ∇ 2 f (W) (Lemma 2); 3. Next, we show that the Hessian of the empirical loss function ∇ 2 f n (W) is close to its popular counterpart ∇ 2 f (W) uniformly in B(W, r) with high probability (Lemma 3).4. Finally, putting all the arguments together, we establish ∇ 2 f n (W) satisfies local strong convexity and smoothness in B(W, r).We will first show that the Hessian of the population risk is smooth enough around W in the following lemma. Lemma 1. For sigmoid activations, assume W F ≤ 1, we have DISPLAYFORM1 holds for some large enough constant C, when W − W F ≤ 0.7.The proof is given in Appendix D.2. Lemma 1 together with the fact that ∇ 2 f (W) be lower and upper bounded, will allow us to bound ∇ 2 f (W) in a neighborhood around ground truth, given below. Lemma 2 (Local Strong Convexity and Smoothness of Population Loss). For sigmoid activations, there exists some constant C, such that DISPLAYFORM2 holds for all W ∈ B(W, r) with r:= min DISPLAYFORM3 The proof is given in Appendix D.3. The next step is to show the Hessian of the empirical loss function is close to the Hessian of the population loss function in a uniform sense, which can be summarized as following. Lemma 3. For sigmoid activations, there exists constant C such that as long as n ≥ C · dK log dK, with probability at least 1 − d −10, the following holds DISPLAYFORM4 where r:= min DISPLAYFORM5 The proof can be found in Appendix D.4.The final step is to combine Lemma 3 and Lemma 1 to obtain Theorem 1 as follows,Proof of Theorem 1. By Lemma 3 and Lemma 2, we have with probability at least DISPLAYFORM6 As long as the sample size n is set such that DISPLAYFORM7 holds for all W ∈ B (W, r). Similarly, we have DISPLAYFORM8 holds for all W ∈ B (W, r). We have established that f n (W) is strongly convex in B(W, r) in Theorem 1, thus there exists at most one critical point in B(W, r). The proof of Theorem 2 follows the steps below:1. We first show that the gradient ∇f n (W) concentrates around ∇f (W) in B(W, r) (Lemma 4), and then invoke BID21, Theorem 2) to guarantee there indeed exists a critical point W n in B(W, r);2. We next show W n is close to W and gradient descent converges linearly to W n with a properly chosen step size. The following lemma establishes that ∇f n (W) uniformly concentrates around ∇f (W). Lemma 4. For sigmoid activation function, assume W F ≤ 1, there exists constant C such that as long as n ≥ CdK log(dK), with probability at least 1 − d −10, the following holds DISPLAYFORM0 where r:= min DISPLAYFORM1 Notice that for the population risk function, f (W), W is the unique critical point in B(W, r) due to local strong convexity. With Lemma 3 and Lemma 4, we can invoke (, Theorem 2), which guarantees the following. Corollary 1. There exists one and only one critical point W n ∈ B (W *, r) that satisfies DISPLAYFORM2 We first show that W n is close to W. By the intermediate value theorem, ∃W ∈ B (W, r) such that DISPLAYFORM3 where the last inequality follows from the optimality of W n. By Theorem 1, we have DISPLAYFORM4 On the other hand, by the Cauchy-Schwarz inequality, we have DISPLAYFORM5 where the last line follows from Lemma 4. Plugging FORMULA0 and FORMULA0 into FORMULA0, we have DISPLAYFORM6 Now we have established there indeed exists a critical point in B(W, r). We can establish local linear convergence of gradient descent as below. Let W t be the estimate at the t-th iteration. According to the update rule, we have DISPLAYFORM7 Moreover, by the fundamental theorem of calculus BID17, ∇f n (W t) can be written as DISPLAYFORM8. By Theorem 1, we have DISPLAYFORM9 where DISPLAYFORM10 and H max = C. Therefore, we have DISPLAYFORM11 Hence, DISPLAYFORM12 as long as we set η < DISPLAYFORM13. In summary, gradient descent converges linearly to the local minimizer W n. The proof contains two parts. Part (a) proves that the estimation of the direction of W is sufficiently accurate, which follows the arguments similar to those in BID38 and is only briefly summarized below. Part (b) is different, where we do not require the homogeneous condition for the activation function, and instead, our proof is based on a mild condition in Assumption 2. We detail our proof in part (b).We first define a tensor operation as follows. For a tensor T ∈ R n1×n2×n3 and three matrices A ∈ R n1×d1, B ∈ R n2×d2, C ∈ R n3×d3, the (i, j, k)-th entry of the tensor T (A, B, C) is given by DISPLAYFORM0 (a) In order to estimate the direction of each w i for i = 1,..., K, BID38 shows that for the regression problem, if the sample size n ≥ dpoly (K, κ, t, log d), then DISPLAYFORM1 holds with high probability. Such a also holds for the classification problem with only slight difference in the proof as we describe as follows. The main idea of the proof is to bound the estimation error of P 2 and R 3 via Bernstein inequality. For the regression problem, Bernstein inequality was applied to terms associated with each neuron individually, and the bounds were then put together via triangle inequality in BID38, whereas for the classification problem here, we apply Bernstein inequality to terms associated with all neurons all together. Another difference is that the label y i of the classification model is bounded by nature, whereas the output y i in the regression model needs to be upper bounded via homogeneously bounded conditions of the activation function. A reader can refer to BID38 for the details of the proof for this part.(b) In order to estimate w i for i = 1,..., K, we provide a different proof from BID38, which does not require the homogeneous condition on the activation function, but assumes a more relaxed condition in Assumption 2.We define a quantity Q 1 as follows: DISPLAYFORM2 where l 1 is the first non-zero index such that M l1 = 0. For example, if l 1 = 3, then Q 1 takes the following form DISPLAYFORM3 where w = w/ w and by definition m 3,i (DISPLAYFORM4 Clearly, Q 1 has information of w i, which can be estimated by solving the following optimization problem: DISPLAYFORM5 where each entry of the solution takes the form DISPLAYFORM6 In the initialization, we substitute Q 1 (estimated from training data) for Q 1, V u i (estimated in part (a)) for s i w i into FORMULA55, and obtain an estimate β of β. We then substitute β for β and V u i for s i w i into to obtain an estimate a i of w i via the following equation DISPLAYFORM7 Furthermore, since m l1,i (x) has fixed sign for x > 0 and for l 1 ≥ 1, s i can be estimated correctly from the sign of β i for i = 1,..., K.For notational simplicity, let β 1,i:= DISPLAYFORM8 2, and then FORMULA0 and FORMULA57 DISPLAYFORM9 By Assumption 2 and, there exists a constant δ > 0 such that the inverse function g(·) of m 3,1 (·) has upper-bounded derivative in the interval (β 1,i − δ, β 1,i + δ), i.e., |g (x)| < Γ for a constant Γ. By employing the in BID38, if the sample size n ≥ dpoly (K, κ, t, log d), then Q 1 and Q 1, V u i and s i w i can be arbitrarily close so that |β DISPLAYFORM10 Thus, by and mean value theorem, we obtain DISPLAYFORM11 where ξ is between β 1,i and β 1,i, and hence |g (ξ)| < Γ. Therefore, DISPLAYFORM12, which is the desired . We introduce some useful definitions and that will be used in the proofs. The first one is the definition of norms of random variable, i.e. Definition 4 (Sub-gaussian and Sub-exponential norm). The sub-gaussian norm of a random variable X, denotes as X ψ2, is defined as DISPLAYFORM0 and the sub-exponential norm of X, denoted as X ψ1, is defined as DISPLAYFORM1 The definition is summarized from (, Def 5.7,Def 5.13), and if X ψ2 is upper bounded, then X is a sub-gaussian random variable and it satisfies DISPLAYFORM2 Next we provide the calculations of the gradient and Hessian of E [(W ; DISPLAYFORM3 where if j = l, DISPLAYFORM4 and if j = l, DISPLAYFORM5 Next we will evaluate ∆ j,l . From FORMULA28 we can write the hessian block more concisely as DISPLAYFORM6 where g j,l (W) = ξ j,l (W) (p(W)(1−p(W))) 2 ∈ R, and then by the mean value theorem, we can write g j,l (W) as DISPLAYFORM7 where W = η · W + (1 − η) W for some η ∈. Thus we can calculate ∆ j,l as DISPLAYFORM8 and plug it back to we can obtain DISPLAYFORM9 for the third equality we have used the fact that DISPLAYFORM10 since the variable of g j,l W is in the form of w i x. and for the last two inequalities, we have used Cauchy-Schwarz inequality. Our next goal is to upper bound E T 2 j,l,k. Further since DISPLAYFORM11 which aligns with x and the scalar coefficient is upper bounded by DISPLAYFORM12 are all upper bounded, thus we leave only the denominator. And then DISPLAYFORM13 holds for some constant C, where the second inequality follows from Lemma 5.Lemma 5. Let x ∼ N (0, I), t = max {w 1 2, · · · w K 2} and z ∈ Z such that z ≥ 1, for the sigmoid activation function φ (x) = 1 1+e −x, the following DISPLAYFORM14 holds for a large enough constant C which depends on the constant z. Plugging FORMULA1 into FORMULA1, we can obtain DISPLAYFORM15 Further since e DISPLAYFORM16, where we have used the assumption that W F ≤ 1 thus we can conclude that if DISPLAYFORM17 holds for some constant C. Proof. We will first present upper and lower bounds of the Hessian of the population risk at ground truth, i.e. ∇ 2 f (W), and then apply Lemma 1 to obtain a uniform bound in the neighborhood of W. As a reminder, DISPLAYFORM0 and let a = [a 1, · · ·, a K] ∈ R dK, we can write DISPLAYFORM1 the second inequality holds due to the fact that DISPLAYFORM2, and the last inequality follows from (b, Lemmas D.4 and D.6).Further more, we can uppder bound ∇ 2 f (W) as DISPLAYFORM3 where for the third and fourth inequality we have used the fact that φ w i x 1 − φ w i x ≤ 1 4and DISPLAYFORM4 Thus together with the lower bound we can conclude that DISPLAYFORM5 From Lemma 1, we have DISPLAYFORM6 therefore, when W − W F ≤ 0.7 and DISPLAYFORM7 i.e., when W − W F ≤ min DISPLAYFORM8 κ 2 λ, 0.7 for some constant C, we have DISPLAYFORM9 Moreover, within the same neighborhood, by the triangle inequality we have DISPLAYFORM10 D.4 PROOF OF LEMMA 3Proof. We adapt the analysis in BID21 to our setting. Let N be the -covering number of the Euclidean ball B (W, r). It is known that log N ≤ dK log (3r/) BID35. Let W = {W 1, · · ·, W N} be the -cover set with N elements. For any W ∈ B (W, r), let j (W) = argmin j∈[N] W − W j(W) F ≤ for all W ∈ B (W, r).For any W ∈ B (W, r), we have DISPLAYFORM11 Hence, we have DISPLAYFORM12 where the events A t, B t and C t are defined as DISPLAYFORM13 DISPLAYFORM14 DISPLAYFORM15 In the sequel, we will bound the terms P (A t), P (B t), and P (C t), separately.1. Upper bound P (B t). Before continuing, let us state a simple technical lemma that is useful for our proof, whose proof can be found in BID21. Let G i = v, ∇ 2 (W ; x i) − E ∇ 2 (W ; x) v where E[G i] = 0. Let a = a 1, · · ·, a K ∈ R dK. Then we can show that G i ψ1 is upper bounded, which we summariz as follows. Lemma 7. There exists some constant C such that DISPLAYFORM16

Represent the short summary for finding the corresponding scientific article (long)

We provide the first theoretical analysis of guaranteed recovery of one-hidden-layer neural networks under cross entropy loss for classification problems.

Represent the TLDR for finding the corresponding scientific article

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question:
represent the scientific article to find a short tldr!
------
E.g. Humans have the remarkable ability to correctly classify images despite possible degradation. Many studies have suggested that this hallmark of human vision from the interaction between feedforward signals from bottom-up pathways of the visual cortex and feedback signals provided by top-down pathways. Motivated by such interaction, we propose a new neuro-inspired model, namely Convolutional Neural Networks with Feedback (CNN-F). CNN-F extends CNN with a feedback generative network, combining bottom-up and top-down inference to perform approximate loopy belief propagation. We show that CNN-F's iterative inference allows for disentanglement of latent variables across layers. We validate the advantages of CNN-F over the baseline CNN. Our experimental suggest that the CNN-F is more robust to image degradation such as pixel noise, occlusion, and blur. Furthermore, we show that the CNN-F is capable of restoring original images from the degraded ones with high reconstruction accuracy while introducing negligible artifacts. Convolutional neural networks (CNNs) have been widely adopted for image classification and achieved impressive prediction accuracy. While state-of-the-art CNNs can achieve near-or super-human classification performance, these networks are susceptible to accuracy drops in the presence of image degradation such as blur and noise, or adversarial attacks, to which human vision is much more robust. This weakness suggests that CNNs are not able to fully capture the complexity of human vision. Unlike the CNN, the human's visual cortex contains not only feedforward but also feedback connections which propagate the information from higher to lower order visual cortical areas as suggested by the predictive coding model. Additionally, recent studies suggest that recurrent circuits are crucial for core object recognition. A recently proposed model extends CNN with a feedback generative network, moving a step forward towards more brain-like CNNs. The inference of the model is carried out by the feedforward only CNN. We term convolutional neural networks with feedback whose inference uses no iterations as CNN-F (0 iterations). The generative feedback models the joint distribution of the data and latent variables. This methodology is similar to how human brain works: building an internal model of the world. Despite the success of CNN-F (0 iterations) in semi-supervised learning and out-of-distribution detection, the feedforward only CNN can be a noisy inference in practice and the power of the rendering top-down path is not fully utilized. A neuro-inspired model that carries out more accurate inference is therefore desired for robust vision. Our work is motivated by the interaction of feedforward and feedback signals in the brain, and our contributions are: We propose the Convolutional Neural Network with Feedback (CNN-F) with more accurate inference. We perform approximated loopy belief propagation to infer latent variables. We introduce recurrent structure into our network by feeding the generated image from the feedback process back into the feedforward process. We term the model with k-iteration inference as CNN-F (k iterations). In the context without confusion, we will use the name CNN-F for short in the rest of the paper. We demonstrate that the CNN-F is more robust to image degradation including noise, blur, and occlusion than the CNN. In particular, our experiments show that CNN-F experiences smaller accuracy drop compared to the corresponding CNN on degraded images. We verify that CNN-F is capable of restoring degraded images. When trained on clean data, the CNN-F can recover the original image from the degraded images at test time with high reconstruction accuracy. Convolutional Neural Network with Feedback (CNN-F) is a generative model that generates images by coarse-to-fine rendering using the features computed by the corresponding CNN. Latent variables in CNN-F account for the uncertainty of the rendering process. The prior distribution of those latent variables is designed to capture the dependencies between them across layers. Inference for the optimal latent variables given image x and label y matches a feedforward CNN in CNN-F (0 iterations) (see Fig. 1). We provide mathematical description of CNN-F (0 iterations) below. Let h be the generated image, y ∈ {1, ..., K} be object category. z = {t, s}, = 1,..., L are the latent variables at layer, where t defines translation of rendering templates based on the position of local maximum from Maxpool, and s decides whether to render a pixel or not based on whether it is activated (ReLU) in the feed-forward CNN. T (t) denotes the translation matrix corresponding to the translation latent variable t. W are rendering templates, where W is the weight matrix at layer in the corresponding CNN. h is the intermediate rendered image at layer. The generation process in CNN-F (0 iterations) is given by: The dependencies among latent variables {z} 1:L across different layers are captured by the structured prior π z|y Softmax η exp(η), and b corresponds the bias after convolutions in CNN. Under the assumption that the intermediate rendered images {h} 1:L are nonnegative, the joint maximum a posteriori (JMAP) inference of latent variable z in CNN-F (0 iterations) is a CNN. Convolutional Neural Networks with Feedback using k-iteration inference [CNN-F (k iterations)] performs approximated loopy belief propagation on CNN-F for k times (see Fig. 1). Inference of latent variables is performed by propagating along both directions of the model. In the following of this session, we will use CNN-F to denote CNN-F (k iterations) for short. Inheriting the notation for the formulation in the CNN-F (0 iterations), we formulate CNN-F as follows. The generation process of the top-down pathway in CNN-F is the same as in the CNN-F (0 iterations), i.e. h(− 1) = T (t)W (s h). Different from the CNN-F (0 iterations), the generated image h in the CNN-F is fed back to the bottomup pathway for approximated loopy belief propagation. In other words, the CNN-F performs bottom-up followed by top-down inference such that the information at later layers in the CNNs can be used to update the noisy estimations at the early layers in the same network. Specifically, the feedforward process in the CNN-F is g = W AdaPool{AdaRelu(g( − 1))} + b, where g denotes the network activations at layer. The top-down messages correct for the noisy bottom-up inference by the adaptive operators Input: Input image x. and object class y *. where W is the rendering template at layer, and b is the parameters of the structured prior π z|y at layer. T (t) is the translation matrix corresponding to the translation latent variable t. Repeat step 2 -3 until convergence or early stopping. (see Algorithm 1): We study the robustness and image restoration performance of CNN-F (10 iterations). Additionally, we observe the disentanglement of information stored in latent variables. In this section, we will refer to CNN-F (10 iterations) as CNN-F. We train a 4 layer CNN and CNN-F (10 iterations) of corresponding architecture on the clean MNIST train set. For the architecture, we use 3 convolutional layers followed by 1 fully connected layer. We use 5x5 convolutional kernel for each convolutional layer with 8 channels in the first layer followed by 16 channels in the second layer followed by 8 channels in the third layer. We use instance norm between layers to normalize the input. We test the models on degraded test set images. The CNN trained has test accuracy 99.1% while CNN-F has test accuracy 95.26%. Our experimental study shows that the iterative inference in CNN-F promotes disentanglement of latent factors across layers. In particular, we observe that the latent variables at each layer in CNN-F captures different essences of the reconstructed image. For example, in the case of MNIST digits, those essences are different strokes that form the digits. Those strokes differ from each other in their location, styles, or angles. In our experiment, we trained a CNN-F with 3 convolutional layers on MNIST. Then, we sent an MNIST image of digit 0 and an MNIST image of digit 1 into the trained networks and collected their corresponding sets of latent variables. We denote z k to be the estimated latent variables from the image of digit 1 at layer k = 1, 2, 3 in CNN-F. Figure 2 illustrates that each set of latent variables z k captures strokes at a particular location in digit 1. In the first column of Figure 2, we use latent variables z 3 at the top layer in CNN-F to reconstruct the image. Similarly, in the second column of Figure 2, in addition to z 3, we add z 2 into the reconstruction. We observe that the latent variables z 3 capture the center of the digit 1 while the latent variables z 2 try to extend the digits to both ends. Finally, we include z 1 into the reconstruction and observe that it completes the digit by filling in the two ends. This observation suggests that CNN-F and its iterative inference algorithm lead to effective disentanglement of latent factors across the layers. Robustness Table 1 shows the accuracy and percent accuracy drop on noisy, blurry and occluded input. The accuracy of CNN-F drops less compared to CNN of same architecture, indicating that CNN-F is more robust. Image Restoration Table 2 shows CNN-F's reconstruction of images with added gaussian noise, blur, and occlusion. CNN-F is able to denoise, deblur, and do some degree of inpainting in on the degraded images. We note that with more iterations of feedback, the reconstructed image becomes more clean. The ability of CNN-F to restore images is consistent with studies in neuroscience which suggest that feedback signals contribute to automatic sharpening of images. For example, Abdelhack and Kamitani showed that the neural representation of blurry images is more similar to the latent representation of the clean version from a deep neural network than the latent representation of the blurry image. CNN-F is able to sharpen blurry images, which is consistent with this study. to understand better the role of feedback in robust vision. To compare CNN-F with neuronal/psychological data, we will scale up the training to ImageNet. A more challenging scenario for robust vision is adversarial attack. We will study the robustness of the proposed CNN-F under various types of adversarial attacks. We also plan to measure the similarity between the latent representations of the CNN-F with neural activity recorded from the brain in order to access whether CNN-F is a good model for human vision. We propose the Convolutional Neural Networks with Feedback (CNN-F) which consists of both a classification pathway and a generation pathway similar to the feedforward and feedback connections in human vision. Our model uses approximate loopy belief propagation for inferring latent variables, allowing for messages to be propagated along both directions of the model. We also introduce recurrency by passing the reconstructed image and predicted label back into the network. We show that CNN-F is more robust than CNN on corrupted images such as noisy, blurry, and occluded images and is able to restore degraded images when trained only on clean images. == CNN-F extends CNN with a feedback generative network for robust vision. != We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.

Question: With the deployment of neural networks on mobile devices and the necessity of transmitting neural networks over limited or expensive channels, the file size of trained model was identified as bottleneck. We propose a codec for the compression of neural networks which is based on transform coding for convolutional and dense layers and on clustering for biases and normalizations. With this codec, we achieve average compression factors between 7.9–9.3 while the accuracy of the compressed networks for image classification decreases only by 1%–2%, respectively. Deep neural networks spread to many scientific and industrial applications (1; 2; 3; 4). Often, the necessity of large amounts of training data, long training duration and the computational complexity of the inference operation are noted as bottlenecks in deep learning pipelines. More recently, the memory footprint of saved neural networks was recognized as challenge for implementations in which neural networks are not executed on servers or in the cloud but on mobile devices or on embedded devices. In these use cases, the storage capacities are limited and/or the neural networks need to be transmitted to the devices over limited transmission channels (e.g. app updates). Therefore, an efficient compression of neural networks is desirable. General purpose compressors like Deflate (combination of Lempel-Ziv-Storer-Szymanski with Huffman coding) perform only poorly on neural networks as the networks consist of many slightly different floating-point weights. In this paper, we propose a complete codec pipeline for the compression of neural networks which relies on a transform coding method for the weights of convolutional and dense layers and a clusteringbased compression method for biases and normalizations. Our codec provides high coding efficiency, negligible impact on the desired output of the neural network (e.g. accuracy), reasonable complexity and is applicable to existing neural network models, i.e. no (iterative) retraining is required. Several related works were proposed in the literature. These works mainly rely on techniques like quantization and pruning. The tensorflow framework provides a quantization method to convert the trained floating-point weights to 8 bit fixed-point weights. We will demonstrate that considerable coding gains on top of those due to quantization can be achieved by our proposed methods. Han et al. proposed the Deep Compression framework for the efficient compression of neural networks BID4. In addition to quantization, their method is based on an iterative pruning and retraining phase. In contrast to Deep Compression, we aim at transparent compression of existing network models without the necessity of retraining and without modifying the network architecture. It is known from other domains like video coding that transparent coding and coding modified content are different problems (6; 7). Iandola et al. propose a novel network architecture called SqueezeNet which particularly aims at having as few weights in the network as possible BID7. We will demonstrate that our method can still reduce the size of this already optimized SqueezeNet network by a factor of up to 7.4. It is observable that the filters in neural networks contain structural information not completely different from blocks in natural pictures. Reasoned by this observation, the encoder base for convolutional filters consists of a two-dimensional discrete cosine transform (2D DCT) followed by a quantization step. This combination is often referred to as transform coding. For the DCT, the transformation block size is set accordingly to the size of the filter (e.g. a 7 × 7 DCT for a 7 × 7 filter). Subsequent to the transformation, the coefficients are quantized. The bit depth of the quantizer can be tuned according to the needs of the specific application. Typical values are 5-6 bit/coefficient with only a small accuracy impact. The weights of dense layers (also referred to as fully-connected layers) and of 1 × 1 convolutions (no spatial filtering but filtering over the depth of the previous layer, typically used in networks for depth reduction) are arranged block-wise prior to transform coding. K-means clustering is used for the coding of the biases and normalizations. The number of clusters is set analogously to the quantizer bit depth according to the quality settings. Code books are generated for biases and normalizations. Thereby, the usage of the clustering algorithm is beneficial if less bits are needed for coding the quantizer indices and the code book itself than for coding the values directly. The clustering approach has the advantage that the distortion is smaller than for uniform quantization. In consequence, the accuracy of the network is measured to be higher for a given number of quantizer steps. However, the occurrence of code book indices is also more uniformly distributed. Due to the higher entropy of this distribution, the compression factor is considerably smaller (see Sec. 3). In particular the Burrow-Wheeler transform and the move-to-front transform which are both invoked for entropy coding are put at a disadvantage by the uniform distribution. We chose to use use the same number of quantizer steps for all parameters. For this reason the clustering was chosen for those network parameters which are too sensible to the higher distortion caused by uniform quantization. The processed data from the transform coding and from the clustering are entropy coded layer-wise using BZip2, serialized and written to the output file. In addition, meta data is stored. It includes the architecture of the layers in the network, shapes and dimensions of the filters, details on the block arrangements, scaling factors from the pre-scaling, scaling factors and offsets from the quantizer, and the code books for the clustering. We evaluate our method using four image classification networks: ResNet50, GoogLeNet, AlexNet BID8 and SqueezeNet BID7. Rate-distortion analysis is a typical procedure for the evaluation of compression algorithms BID9. The performance of neural networks for image classifi- cation is usually measured using the Top-5 accuracy. Therefore, we measure the distortion as decrease of the accuracy after compressing the networks. We use the compression factor (uncompressed file size /compressed file size) to assess the compression efficiency. The networks are encoded, decoded and then used for the image classification downstream pipeline. As data, we use the ILSVRC-2012 validation set (50,000 images in 1,000 classes). To study which algorithms from our overall pipeline contribute how much the the final , we evaluate three subsets of our technology: In the first subset, only quantization is applied to the network weights. In the second subset, we apply the clustering algorithm to all parameters of all layers. In the third set, we use our complete pipeline with transform coding and clustering. The ing RD curves are visualized in FIG1. The findings for the two state-of-the-art networks (FIG1 and 2(b)) are quite clear: The for the complete pipeline are superior to those of the subsets. This indicates that all methods in the pipeline have a reason for existence and their coding gains are to some extend additive. Compression factors of ten or higher are observed without a considerable decrease in accuracy. AlexNet has the special property that it contains an extraordinary high number of weights and that more than 90% of the weights are located in the first dense layer. As suggested by Han et al. BID4, this disadvantage in the design of the network can only be fixed by pruning and retraining. Hence, we observe in FIG1 (c) that transform coding and clustering do not bring any gain on-top of quantization. Interestingly, our methods enables compression factors of more then five without much decrease in accuracy even for SqueezeNet in FIG1.This is an indication that our framework is also beneficial for networks with memory-optimized architecture. From the underlying data of FIG1, we calculate numerical values for the compression factor. In average, our codec achieves compression factors of 7.9 and 9.3 for accuracy decreases of 1% and 2%, respectively. We analyze the computational complexity of our algorithms by measuring the encoder and decoder run times using our unoptimized Python code. For the final codec (transform coding for convolutional and dense weights, clustering for biases and normalizations), we measure 29s for the encoder and 7.6s for the decoder. In this paper, we proposed a codec for the compression of neural networks which is based on transform coding and clustering. The codec enables a low-complexity and high efficient transparent compression of neural networks. The impact on the neural network performance is negligible.

Our neural network codec (which is based on transform coding and clustering) enables a low complexity and high efficient transparent compression of neural networks.

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.