Question:
represent the scientific article to find a short tldr:
------
E.g.
"We propose a robust Bayesian deep learning algorithm to infer complex posteriors with latent variables. Inspired by dropout, a popular tool for regularization and model ensemble, we assign sparse priors to the weights in deep neural networks (DNN) in order to achieve automatic “dropout” and avoid over-fitting. By alternatively sampling from posterior distribution through stochastic gradient Markov Chain Monte Carlo (SG-MCMC) and optimizing latent variables via stochastic approximation (SA), the trajectory of the target weights is proved to converge to the true posterior distribution conditioned on optimal latent variables. This ensures a stronger regularization on the over-fitted parameter space and more accurate uncertainty quantification on the decisive variables. Simulations from large-p-small-n regressions showcase the robustness of this method when applied to models with latent variables. Additionally, its application on the convolutional neural networks (CNN) leads to state-of-the-art performance on MNIST and Fashion MNIST datasets and improved resistance to adversarial attacks. Bayesian deep learning, which evolved from Bayesian neural networks (; BID4, provides an alternative to point estimation due to its close connection to both Bayesian probability theory and cutting-edge deep learning models. It has been shown of the merit to quantify uncertainty BID6, which not only increases the predictive power of DNN, but also further provides a more robust estimation to enhance AI safety. Particularly, BID5 BID3 described dropout as a variational Bayesian approximation. Through enabling dropout in the testing period, the randomly dropped neurons generate some amount of uncertainty with almost no added cost. However, the dropout Bayesian approximation is variational inference (VI) based thus it is vulnerable to underestimating uncertainty. MCMC, known for its asymptotically accurate posterior inference, has not been fully investigated in DNN due to its unscalability in dealing with big data and large models. Stochastic gradient Langevin dynamics (SGLD) , the first SG-MCMC algorithm, tackled this issue by adding noise to a standard stochastic gradient optimization, smoothing the transition between optimization and sampling. Considering the pathological curvature that causes the SGLD methods inefficient in DNN models, BID15 proposed combining adaptive preconditioners with SGLD (pSGLD) to adapt to the local geometry and obtained state-of-the-art performance on MNIST dataset. To avoid SGLD's random-walk behavior, BID3 proposed using stochastic gradient Hamiltonian Monte Carlo (SGHMC), a second-order Langevin dynamics with a large friction term, which was shown to have lower autocorrelation time and faster convergence BID2. used SGHMC with GANs BID8 ) to achieve a fully probabilistic inference and showed the Bayesian GAN model with only 100 labeled images was able to achieve 99.3% testing accuracy in MNIST dataset.;; provided theoretical interpretations of SGLD from the perspective of non-convex optimization, echoing the empirical fact that SGLD works well in practice. When the number of predictors exceeds the number of observations, applying the spike-and-slab priors is particularly powerful and efficient to avoid over-fitting by assigning less probability mass on We denote the decaying learning rate at time k by (k), the entire data by DISPLAYFORM0, where d i = (x i, y i), the log of posterior by L(β), ∇ as the gradient of any function in terms of β. The minibatch of data B is of size n with indices S = {s 1, s 2, ..., s n}, where s i ∈ {1, 2, ..., N}. Stochastic gradient ∇L(β) from a mini-batch of data B randomly sampled from D is used to approximate the true gradient ∇L(β):∇L(β) = ∇ log P(β) + N n i∈S ∇ log P(d i |β).The stochastic gradient Langevin dynamics (no momentum) is formed as follows: DISPLAYFORM1 where τ > 0 denotes the temperature, G is a positive definite matrix to precondition the dynamics. It has been shown that SGLD asymptotically converges to a stationary distribution π(β|D) ∝ e τ L(β) . As τ increases, the algorithm tends towards optimization with underestimated uncertainty. Another variant of SG-MCMC, SGHMC BID3 ), proposes to use second-order Langevin dynamics to generate samples: DISPLAYFORM2 where r is the momentum item, M is the mass,B is an estimate of the error variance from the stochastic gradient, C is a user-specified friction term to counteracts the noisy gradient. Dropout has been proven successful, as it alleviates over-fitting and provides an efficient way of making bagging practical for ensembles of countless sub-networks. Dropout can be interpreted as assigning the Gaussian mixture priors on the neurons BID7. To mimic Dropout in our Bayesian CNN models, we assign the spike-and-slab priors on the most fat-tailed weights in FC1 FIG0. From the Bayesian perspective, the proposed robust algorithm distinguishes itself from the dropout approach in treating the priors: our algorithm keeps updating the priors during posterior inference, rather than fix it. The inclusion of scaled mixture priors in deep learning models were also studied in BID1; BID15 with encouraging . However, to the best of our knowledge, none of the existing SG-MCMC methods could deal with complex posterior with latent variables. Intuitively, the Bayesian formulation with model averaging and the spike-and-slab priors is expected to obtain better predictive performance through averages from a "selective" group of "good" submodels, rather than averaging exponentially many posterior probabilities roughly. For the weight priors of the rest layers (dimension u), we just assume they follow the standard Gaussian distribution, while the biases follow improper uniform priors. Similarly to the hierarchical prior in the EM approach to variable selection (EMVS) (Rořková and), we assume the weights β ∈ R p in FC1 follow the spike-and-slab mixture prior DISPLAYFORM0 where DISPLAYFORM1 for each j and 0 < v 0 < v 1. By introducing the latent variable γ j = 0 or 1, the mixture prior is represented as DISPLAYFORM2 The interpretation is: if γ j = 0, then β j is close to 0; if γ j = 1, the effect of β j on the model is intuitively large. The likelihood of this model given a mini-batch of data {( DISPLAYFORM3 where ψ(x i ; β) can be a mapping for logistic regression or linear regression, or a mapping based on a series of nonlinearities and affine transformations in the deep neural network. In the classification formulation, y i ∈ {1, . . ., K} is the response value of the i-th example. In addition, the variance σ 2 follows an inverse gamma prior DISPLAYFORM4 The i.i.d. Bernoulli prior is used since there is no structural information in the same layer. DISPLAYFORM5 Finally, our posterior density follows DISPLAYFORM6 The EMVS approach is efficient in identifying potential sparse high posterior probability submodels on high-dimensional regression (Rořková and) and classification problem . These characteristics are helpful for large neural network computation, thus we refer the stochastic version of the EMVS algorithm as Expectation Stochastic-Maximization (ESM). Due to the existence of latent variables, optimizing π(β, σ 2, δ|B) directly is difficult. We instead iteratively optimize the "complete-data" posterior log π(β, σ 2, δ, γ|B), where the latent indicator γ is treated as "missing data".More precisely, the ESM algorithm is implemented by iteratively increasing the objective function DISPLAYFORM0 where E γ|· denotes the conditional expectation DISPLAYFORM1 ) at the k-th iteration, we first compute the expectation of Q, then alter (β, σ, δ) to optimize it. For the conjugate spike-slab hierarchical prior formulation, the objective function Q is of the form DISPLAYFORM2 where DISPLAYFORM3 spike-and-slab priors in the "sparse" layer, e.g. FC1 DISPLAYFORM4 Gaussian priors in other layers DISPLAYFORM5 and DISPLAYFORM6 The physical meaning of E γ|· γ j in Q 2 is the probability ρ j, where ρ ∈ R p, of β j having a large effect on the model. Formally, we have DISPLAYFORM0 where DISPLAYFORM1 Bernoulli prior enables us to use P(γ j = 1|δ DISPLAYFORM2 The other conditional expectation comes from a weighted average κ j, where κ ∈ R p . DISPLAYFORM3 Since there is no closed-form optimal solution for β here, to optimize Q 1 with respect to β, we use Adam , a popular algorithm with adaptive learning rates, to train the model. In order to optimize Q 1 with respect to σ, by denoting DISPLAYFORM0 as V, following the formulation in and Rořková and we have: DISPLAYFORM1 To optimize Q 2, a closed-form solution can be derived from Eq. and Eq.. DISPLAYFORM2 Algorithm 1 SGLD-SA with spike and slab priors DISPLAYFORM3 Initialize: DISPLAYFORM4 end for The EMVS algorithm is designed for linear regression models, although the idea can be extended to nonlinear models. However, when extending to nonlinear models, such as DNNs, the M-step will not have a closed-form update anymore. A trivial implementation of the M-step will likely cause a local-trap problem. To tackle this issue, we replace the E-step and the M-step by SG-MCMC with the prior hyperparameters tuned via stochastic approximation BID0: DISPLAYFORM0 where DISPLAYFORM1 is the gradient-like function in stochastic approximation (see details in Appendix A.2), g θ (·) is the mapping detailed in Eq., Eq., Eq. and Eq. to derive the optimal θ based on the current β, the step size ω (k) can be set as A(k + Z) −α with α ∈ (0.5, 1]. The interpretation of this algorithm is that we sample β (k+1) fromL(β (k), θ (k) ) and adaptively optimize θ (k+1) from the mapping g θ (k). We expect to obtain an augmented sequence as follows: DISPLAYFORM2 We show the (local) L 2 convergence rate of SGLD-SA below and present the details in Appendix B.Theorem 1 (L 2 convergence rate). For any α ∈ and any compact subset Ω ∈ R 2p+2, under assumptions in Appendix B.1, the algorithm satisfies: there exists a constant λ such that DISPLAYFORM3 where t(Ω) = inf{k : DISPLAYFORM4 Corollary 1. For any α ∈ and any compact subset Ω ∈ R 2p+2, under assumptions in Appendix B.2, the distribution of β (k) in FORMULA1 converges weakly to the invariant distribution e L(β,θ *) as → 0.The key to guaranteeing the consistency of the latent variable estimators is from stochastic approximation and the fact of DISPLAYFORM5 The non-convex optimization of SGLD , in particular the ability to escape shallow local optima , ensures the robust optimization of the latent variables. Furthermore, the mappings of Eq., Eq., Eq. and Eq. all satisfy the assumptions on g in a compact subset Ω (Appendix B), which enable us to apply theorem 1 to SGLD-SA. Because SGHMC proposed by BID3 is essentially a second-order Langevin dynamics and yields a stationary distribution given an accurate estimation of the error variance from the stochastic gradient, the property of SGLD-SA also applies to SGHMC-SA. Corollary 2. For any α ∈ and any compact subset Ω ∈ R 2p+2, under assumptions in Appendix B.2, the distribution of β (k) from SG-MCMC-SA converges weakly to the invariant distribution π(β, ρ *, κ *, σ *, δ * |D) as → 0. The posterior average given decreasing learning rates can be approximated through the weighted DISPLAYFORM0 to avoid over-emphasizing the tail end of the sequence and reduce the variance of the estimator.; BID2 showed a theoretical optimal learning rate (k) ∝ k −1/3 for SGLD and k −1/5 for SGHMC to achieve faster convergence for posterior average, which are used in Sec. 4.1 and Sec. 4.2 respectively. 4.1 SIMULATION OF LARGE-P-SMALL-N REGRESSION SGLD-SA can be applied to the (logistic) linear regression cases, as long as u = 0 in Eq.. We conduct the linear regression experiments with a dataset containing n = 100 observations and p = 1000 predictors. N p (0, Σ) is chosen to simulate the predictor values X (training set) where DISPLAYFORM0 Response values y are generated from Xβ + η, where β = (β 1, β 2, β 3, 0, 0, ..., 0) and η ∼ N n (0, 3I n). To make the simulation in Rořková and more challenging, we assume DISPLAYFORM1 We introduce some hyperparameters, but most of them are uninformative, e.g. ν ∈ {10 −3, 1, 10 3} makes little difference in the test set performance. Sensitivity analysis shows that three hyperparameters are important: v 0, a and σ, which are used to identify and regularize the over-fitted space. We fix τ = 1, λ = 1, ν = 1, v 1 = 100, δ = 0.5, b = p and set a = 1. The learning rates for SGLD-SA and SGLD are set to (k) = 0.001 × k To implement SGLD-SA, we perform stochastic gradient optimization by randomly selecting 50 observations and calculating the corresponding gradient in each iteration. We simulate 500, 000 samples from the posterior distribution and at the same time keep optimizing the latent variables. EMVS is implemented with β directly optimized each time. We also simulate a group of the test set with 1000 observations (display 50 in FIG5 (e)) in the same way as generating the training set to evaluate the generalizability of our algorithm. Tab.1 shows that EMVS frequently fails given bad initializations, while SGLD-SA is fairly robust to the hyperparameters. In addition, from, we can see SGLD-SA is the only algorithm among the three that quantifies the uncertainties of β 1, β 2 and β 3 reasonably and always gives more accurate posterior average (FIG5); by contrast, the estimated response values y from SGLD is close to the true values in the training set (FIG5), but are far away from them in the testing set (2(e)), indicating the over-fitting problem of SGLD without proper regularization. For the simulation of SGLD-SA in logistic regression to demonstrate the advantage of SGLD-SA over SGLD and ESM, we leave the in Appendix C. We implement all the algorithms in Pytorch and run the experiments on GeForce GTX 1080 GPUs. The first DNN we use is a standard 2-Conv-2-FC CNN model (FIG0) of 670K parameters (see details in Appendix D.1). The first set of experiments is to compare methods on the same model without using other complication, such as data augmentation (DA) or batch normalization (BN) BID12. We refer to the general CNN without dropout as Vanilla, with 50% dropout rate applied to the green neurons (FIG0 as Dropout. Vanilla and Dropout models are trained q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q with Adam with Pytorch default parameters (with learning rate 0.001). We use SGHMC as a benchmark method as it is also sampling-based and has a close relationship with the popular momentum based optimization approaches in DNN. SGHMC-SA differs from SGHMC in that SGHMC-SA applies the spike-and-slab priors to the FC1 layer while SGHMC just uses the standard normal priors. The hyperparameters v 1 = 1, v 0 = 1 × 10 −3 and σ = 0.1 in SGHMC-SA are used to regularize the over-fitted space, and a, b are set to p to obtain a moderate "sparsity" to resemble dropout, the step size is ω (k) = 0.1 × (k + 1000) DISPLAYFORM0. We use training batch size 1000 and a thinning factor 500 to avoid a cumbersome system, and the posterior average is applied to each Bayesian model. Temperatures are tuned to achieve better (Appendix D.2).The four CNN models are tested on the classical MNIST and the newly introduced Fashion MNIST (FMNIST) dataset. Performance of these models is shown in Tab.2. Compared with SGHMC, our SGHMC-SA outperforms SGHMC on both datasets. We notice the posterior averages from SGHMC-SA and SGHMC obtain much better performance than Vanilla and Dropout. Without using either DA or BN, SGHMC-SA achieves 99.60% which even outperforms some state-of-the-art models, such as stochastic pooling (99.53%) , Maxout Network (99.55%) BID9 and pSGLD (99.55%) BID15. In F-MNIST, SGHMC-SA obtains 93.01% accuracy, outperforming all other competing models. To further test the maximum performance of SGHMC-SA, we apply DA and BN to the following experiments (see details in Appendix D.3) and refer the datasets with DA as aMNIST and aFMNIST. All the experiments are conducted using a 2-Conv-BN-3-FC CNN of 490K parameters. Using this model, we obtain 99.75% on aMNIST (300 epochs) and 94.38% on aFMNIST (1000 epochs). The are noticeable because posterior model averaging is essentially conducted on a single Bayesian neural network. We also conduct the experiments based on the ensemble of five networks and refer them as aMNIST-5 and aFMNIST-5 in Tab. 2. We achieve 99.79% on aMNIST-5 using 5 small Bayesian neural networks each with 2 thinning samples (4 thinning samples in aFMNIST-5), which is comparable with the state-of-the-art performance . Continuing with the setup in Sec. 4.2, the third set of experiments focus on evaluating model robustness. We expect less robust models perform considerably well on a certain dataset due to over-tuning; however, as the degree of adversarial attacks increases, the performance decreases sharply. In contrast, more robust models should be less affected by these adversarial attacks. We apply the Fast Gradient Sign method BID10 to generate the adversarial examples with one single gradient step as in's study: DISPLAYFORM0 where ζ ranges from 0.1, 0.2,..., 0.5 to control the different levels of adversarial attacks. Similar to the setup in the adversarial experiments by BID16, we normalize the adversarial images by clipping to the range. As shown in FIG6 and FIG6, there is no significant difference among all the four models in the early phase. As the degree of adversarial attacks arises, the images become vaguer as shown in FIG6 and FIG6. In this scenario the performance of Vanilla decreases rapidly, reflecting its poor defense against adversarial attacks, while Dropout performs better than Vanilla. But Dropout is still significantly worse than the sampling based methods SGHMC-SA and SGHMC. The advantage of SGHMC-SA over SGHMC becomes more significant when ζ > 0.25.In the case of ζ = 0.5 in MNIST where the images are hardly recognizable, both Vanilla and Dropout models fail to identify the right images and their predictions are as worse as random guesses. However, SGHMC-SA model achieves roughly 11% higher than these two models and 1% higher than SGHMC, which demonstrates the strong robustness of our proposed SGHMC-SA. Overall, SGHMC-SA always yields the most robust performance. We propose a mixed sampling-optimization method called SG-MCMC-SA to efficiently sample from complex DNN posteriors with latent variables and prove its convergence. By adaptively searching and penalizing the over-fitted parameter space, the proposed method improves the generalizability of deep neural networks. This method is less affected by the hyperparameters, achieves higher prediction accuracy over the traditional SG-MCMC methods in both simulated examples and real applications and shows more robustness towards adversarial attacks. Interesting future directions include applying SG-MCMC-SA towards popular large deep learning models such as the residual network BID11 on CIFAR-10 and CIFAR-100, combining active learning and uncertainty quantification to learn from datasets of smaller size and proving posterior consistency and the consistency of variable selection under various shrinkage priors concretely. The Fokker-Planck equation (FPE) can be formulated from the time evolution of the conditional distribution for a stationary random process. Denoting the probability density function of the random process at time t by q(t, β), where β is the parameter, the stochastic dynamics is given by DISPLAYFORM0 Let q(β) = lim t→∞ q(t, β). If lim t→∞ ∂ t q(t, β) = 0, then DISPLAYFORM1. In other words, lim t→∞ q(t, β) ∝ e L(β), i.e. q(t, β) gradually converges to the Bayesian posterior e L(β). Robbins-Monro algorithm is the first stochastic approximation algorithm to deal with the root finding problem. Given the random output of H(θ, β) with respect to β, our goal is to find θ * such that DISPLAYFORM0 where E θ * denotes the expectation with respect to the distribution of β given parameter θ *. To implement the Robbins-Monro Algorithm, we can generate iterates of the form as follows: DISPLAYFORM1 Note that in this algorithm, H(θ, β) is the unbiased estimator of h(θ), that is DISPLAYFORM2 If there exists an antiderivative Q(θ, β) that satisfies H(θ, β) = and E θ [Q(θ, β)] is concave, it is equivalent to solving the stochastic optimization problem max θ∈Θ E θ [Q(θ, β)]. In contrast to Eq. and Eq., the general stochastic approximation algorithm is intent on solving the root of the integration equation DISPLAYFORM0 where θ ∈ Θ, β ∈ B, for a subset B ∈ B, the transition kernel Π θ (β (k), B), which converges to the invariant distribution f θ (β), satisfies that DISPLAYFORM1. The stochastic approximation algorithm is an iterative recursive algorithm consisting of two steps: DISPLAYFORM2 is not equal to 0 but decays to 0 as k → ∞.To summarise, H(θ, β) is a biased estimator of h(θ) in finite steps, but as k → ∞, the bias decreases to 0. In the SG-MCMC-SA algorithm FORMULA1, DISPLAYFORM3 B CONVERGENCE ANALYSIS The stochastic gradient Langevin Dynamics with a stochastic approximation adaptation (SGLD-SA) is a mixed half-optimization-half-sampling algorithm to handle complex Bayesian posterior with latent variables, e.g. the conjugate spike-slab hierarchical prior formulation. Each iteration of the algorithm consists of the following steps: Sample β (k+1) using SGLD based on the current θ (k), i.e. DISPLAYFORM0 Optimize θ (k+1) from the following recursion DISPLAYFORM1 where g θ (k) (·) is some mapping to derive the optimal θ based on the current β. DISPLAYFORM2. In this formulation, our target is to find θ * such that h(θ DISPLAYFORM3 as k → ∞, this algorithm falls to the category of the general stochastic approximation. To provide the local L 2 upper bound for SGLD-SA, we first lay out the following assumptions:Assumption 1 (Step size and Convexity). {ω (k) } k∈N is a positive decreasing sequence of real numbers such that DISPLAYFORM0 There exist constant δ > 0 and θ DISPLAYFORM1 with additionally DISPLAYFORM2 Then for any α ∈ and suitable A and B, a practical ω (k) can be set as DISPLAYFORM3 Assumption 2 (Existence of Markov transition kernel). For any θ ∈ Θ, in the mini-batch sampling, there exists a family of noisy Kolmogorov operators {Π θ} approximating the operator (infinitesimal) of the Ito diffusion, such that every Kolmogorov operator Π θ corresponds to a single stationary distribution f θ, and for any Borel subset A of β, we have DISPLAYFORM4 Assumption 3 (Compactness). For any compact subset Ω of Θ, we only consider θ ∈ Ω such that DISPLAYFORM5 Note that the compactness assumption of the latent variable θ is not essential, the assumption that the variable is in the compact domain is not only reasonable, but also simplifies our proof. In addition, there exists constants C 1 (Ω) and C 2 (Ω) so that DISPLAYFORM6 Assumption 4 (Solution of Poisson equation). For all θ ∈ Θ, there exists a function µ θ on β that solves the Poisson equation DISPLAYFORM7 For any compact subset Ω of Θ, there exist constants C 3 (β, Ω) and C 4 (β, Ω) such that for all θ, θ ∈ Ω, DISPLAYFORM8 Remark: For notation simplicity, we write C 1 (Ω) as C 1, C 2 (Ω) as C 2,... in the following context. Lemma 1 is a restatement of Lemma 25 (page 447) from BID0. Lemma 1. Suppose k 0 is an integer which satisfies with DISPLAYFORM9 Then for any k > k 0, the sequence {Λ K k} k=k0,...,K defined below is increasing and bounded by 2ω DISPLAYFORM10 Lemma 2 is an extension of Lemma 23 (page 245) from BID0 Lemma 2. There exist λ 0 and k 0 such that for all λ ≥ λ 0 and k ≥ k 0, the sequence u DISPLAYFORM11 From FORMULA2, we can denote a positive constant ∆ + as lim k→∞ inf 2δω DISPLAYFORM12. Then can be simplified as DISPLAYFORM13 There exist λ 0 and k 0 such that for all λ > λ 0 and k > k 0, holds. Note that in practical case when C 1 is small, finding a suitable λ 0 will not be a problem. Theorem 1 (L 2 convergence rate). Suppose that Assumptions 1-4 hold, for any compact subset Ω ∈ Θ, the algorithm satisfies: there exists a constant λ such that DISPLAYFORM14 where t(Ω) = inf{k : DISPLAYFORM15 Proof. Denote T (k) = θ (k) − θ *, with the help of FORMULA2 and Poisson equation FORMULA2, we deduce that DISPLAYFORM16 First of all, according to FORMULA1 and FORMULA2, we have DISPLAYFORM17 DISPLAYFORM18 Conduct the decomposition of D3 similar to Theorem 24 (p.g. 246) from BID0 and Lemma A.5 . DISPLAYFORM19 (ii) From FORMULA3 and FORMULA3 respectively, we deduce that DISPLAYFORM20 Thus there exists C 4 = 2C 4 C 0 such that DISPLAYFORM21 (iii) D3-3 can be further decomposed to D3-3a and D3-3b DISPLAYFORM22 ) with a constant C 3 = 4C 0 C 3 which satisfies that DISPLAYFORM23 Finally, add all the items D1, D2 and D3 together, for some C 1 = C 3 + C 4, we have DISPLAYFORM24 Moreover, from FORMULA3 and FORMULA3, there exists a constant C 5 such that DISPLAYFORM25 Lemma 3 is an extension of Lemma 26 (page 248) from BID0.Lemma 3. Let {u (k) } k≥k0 as a sequence of real numbers such that for all k ≥ k 0, some suitable constants C 1 and C 2 DISPLAYFORM26 and assume there exists such k 0 that DISPLAYFORM27 Then for all k > k 0, we have DISPLAYFORM28 Proof of Theorem 1 (Continued). From Lemma 2, we can choose λ 0 and k 0 which satisfy the conditions and DISPLAYFORM29 From Lemma 3, it follows that for all k > k 0 DISPLAYFORM30 From and the increasing property of Λ k j in Lemma 1, we have DISPLAYFORM31 Therefore, given the sequence u (k) = λ 0 ω (k) that satisfies conditions FORMULA3, FORMULA5 and Lemma 3, for any k > k 0, from FORMULA1 and FORMULA2, we have DISPLAYFORM32 where λ = λ 0 + 6C 5. In addition to the previous assumptions, we make one more assumption on the stochastic gradients to guarantee that the samples converge to the posterior conditioned on the optimal latent variables: Assumption 5 (Gradient Unbiasedness and Smoothness). For all β ∈ B and θ ∈ Θ, the mini-batch of data B, the stochastic noise ξ, which comes from∇L(β, θ) − ∇L(β, θ), is a white noise and independent with each other. In addition, there exists a constant l ≥ 2 such that the following conditions hold: DISPLAYFORM0 For all θ, θ ∈ Θ, there exists a constant M > 0 such that the gradient is M-smooth: DISPLAYFORM1 Corollary 1. For all α ∈, under assumptions 1-5, the distribution of β (k) converges weakly to the invariant distribution e L(β,θ *) as → 0.Proof. The proof framework follows from section 4 of. In the context of stochastic noise ξ (k), we ignore the subscript of and only consider the case of τ = 1. Since θ DISPLAYFORM2 converges to θ * in SGLD-SA and the gradient is M-smooth FORMULA5, we transform the stochastic gradient from ∇L(DISPLAYFORM3, therefore Eq. FORMULA2 can be written as DISPLAYFORM4 Using Eq. FORMULA3, the characteristic function of DISPLAYFORM5 Then the characteristic function of DISPLAYFORM6 Rewrite β (k+1) as β (k+), the characteristic function of β (t+) is the characteristic function of DISPLAYFORM7 With the fact exp(x) = 1 + x + O(x 2), we can get DISPLAYFORM8 Therefore, For any integrable function f, set F as the Fourier transform defined by DISPLAYFORM9 DISPLAYFORM10 The inverse Fourier transform of F[f (x)] and the l-th order derivatives of f (l) (x) is DISPLAYFORM11 Combine Eq. FORMULA5, Eq. FORMULA6 and Eq. FORMULA1, we arrive at the following simplified equation: DISPLAYFORM12 Since DISPLAYFORM13 Finally, we have proved that the distribution of β (k) converges weakly to the invariant distribution e L(β,θ *) as → 0. Now we conduct the experiments on binary logistic regression. The setup is similar as before, except n is set to 500, Σ i,j = 0.3 |i−j| and η ∼ N (0, I/2). We set the learning rates in SGLD-SA and SGLD to 0.01 × k FIG11, FIG11 and FIG11 demonstrate the posterior distribution of SGLD-SA is significantly better than that of SGLD. As shown in FIG11, SGLD-SA is the best method to regulate the over-fitting space and provides the most reasonable posterior mean. Table. 3 illustrates the predictive power of SGLD-SA is overall better than the other methods and robust to different initializations. FIG11 and FIG11 show that the over-fitting problem of SGLD when p > n in logistic regression and the algorithm fails to regulate the over-fitting space; We observe SGLD-SA is able to resist over-fitting and always yields reproducible . q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q The first DNN we use is a standard 2-Conv-2-FC CNN: it has two convolutional layers with a 2 × 2 max pooling after each layer and two fully-connected layers. The filter size in the convolutional layers is 5 × 5 and the feature maps are set to be 32 and 64, respectively BID13 ). The fully-connected layers (FC) have 200 hidden nodes and 10 outputs. We use the rectified linear unit (ReLU) as activation function between layers and employ a cross-entropy loss. The second DNN is a 2-Conv-BN-3-FC CNN: it has two convolutional layers with a 2 × 2 max pooling after each layer and three fully-connected layers with batch normalization applied to the first FC layer. The filter size in the convolutional layers is 4 × 4 and the feature maps are both set to 64. We use 256 × 64 × 10 fully-connected layers. In practice, we observe a suitable temperature setting is helpful to improve the classification accuracy. For example, by setting τ = 100 in the second DNN (see Appendix D.1) we obtain 99.70% on aMNIST. To account for the scale difference of weights in different layers, we apply different temperatures to different layers based on different standard deviations of the gradients in each layer and obtain the in Tab. 2. The MNIST dataset is augmented by randomCrop: randomly crop each image with size 28 and padding 4, random rotation: randomly rotate each image by a degree in [−15•, +15•], normalization: normalize each image with empirical mean 0.1307 and standard deviation 0.3081.The FMNIST dataset is augmented by randomCrop: same as MNIST, randomHorizontalFlip: randomly flip each image horizontally, normalization: same as MNIST." == "a robust Bayesian deep learning algorithm to infer complex posteriors with latent variables" != "This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets."

Question: We propose a method, called Label Embedding Network, which can learn label representation (label embedding) during the training process of deep networks. With the proposed method, the label embedding is adaptively and automatically learned through back propagation. The original one-hot represented loss function is converted into a new loss function with soft distributions, such that the originally unrelated labels have continuous interactions with each other during the training process. As a , the trained model can achieve substantially higher accuracy and with faster convergence speed. Experimental based on competitive tasks demonstrate the effectiveness of the proposed method, and the learned label embedding is reasonable and interpretable. The proposed method achieves comparable or even better than the state-of-the-art systems. Most of the existing methods of neural networks use one-hot vector representations for labels. The one-hot vector has two main restrictions. The first restriction is the "discrete distribution", where each label is distributed at a completely different dimension from the others. The second restriction is the "extreme value" based representation, where the value at each dimension is either 1 or 0, and there is no "soft value" allowed. Those deficiencies may cause the following two potential problems. First, it is not easy to measure the correlation among the labels due to the "discrete distribution". Not being able to measure the label correlation is potentially harmful to the learned models, e.g., causing the data sparseness problem. Given an image recognition task, the image of the shark is often similar to the image of the dolphin. Naturally, we expect the two labels to be "similar". Suppose that we have a lot of training examples for shark, and very few training examples for dolphin. If the label shark and the label dolphin have similar representations, the prediction for the label dolphin will suffer less from the data sparsity problem. Second, the 0/1 value encoding is easy to cause the overfitting problem. Suppose A and B are labels of two similar types of fishes. One-hot label representation prefers the ultimate separation of those two labels. For example, if currently the system output probability for A is 0.8 and the probability for B is 0.2, it is good enough to make a correct prediction of A. However, with the one-hot label representation, it suggests that further modification to the parameters is still required, until the probability of A becomes 1 and the probability of B becomes 0. Because the fish A and the fish B are very similar in appearance, it is probably more reasonable to have the probability 0.8 for A and 0.2 for B, rather than completely 1 for A and 0 for B, which could lead to the overfitting problem. We aim to address those problems. We propose a method that can automatically learn label representation for deep neural networks. As the training proceeds, the label embedding is iteratively learned and optimized based on the proposed label embedding network through back propagation. The original one-hot represented loss function is softly converted to a new loss function with soft distributions, such that those originally unrelated labels have continuous interactions with each other during the training process. As a , the trained model can achieve substantially higher accuracy, faster convergence speed, and more stable performance. The related prior studies include the traditional label representation methods BID7 BID10 BID1, the "soft label" methods BID22, and the model distillation methods BID9 ).Our method is substantially different from those existing work, and the detailed comparisons are summarized in Appendix E. The contributions of this work are as follows:• Learning label embedding and compressed embedding: We propose the Label Embedding Network that can learn label representation for soft training of deep networks. Furthermore, some large-scale tasks have a massive number of labels, and a naive version of label embedding network will suffer from intractable memory cost problem. We propose a solution to automatically learn compressed label embedding, such that the memory cost is substantially reduced.• Interpretable and reusable: The learned label embeddings are reasonable and interpretable, such that we can find meaningful similarities among the labels. The proposed method can learn interpretable label embeddings on both image processing tasks and natural language processing tasks. In addition, the learned label embeddings can be directly adapted for training a new model with improved accuracy and convergence speed.• General-purpose solution and competitive : The proposed method can be widely applied to various models, including CNN, ResNet, and Seq-to-Seq models. We conducted experiments on computer vision tasks including CIFAR-100, CIFAR-10, and MNIST, and on natural language processing tasks including LCSTS text summarization task and IWSLT2015 machine translation task. Results suggest that the proposed method achieves significantly better accuracy than the existing methods (CNN, ResNet, and Seq-to-Seq). We achieve comparable or even better than the state-of-the-art systems on those tasks. A neural network typically consists of several hidden layers and an output layer. The hidden layers map the input to the hidden representations. Let's denote the part of the neural network that produces the last hidden representation as h = f (x) where x is the input of the neural network, h is the hidden representation, and f defines the mapping from the input to the hidden representation, including but not limited to CNN, ResNet, Seq-to-Seq, and so on. The output layer maps the hidden representation to the output, from which the predicted category is directly given by an argmax operation. The output layer typically consists of a linear transformation that maps the hidden representation h to the output z: DISPLAYFORM0 where o represents the linear transformation. It is followed by a softmax operation that normalizes the output as z, so that the sum of the elements in z is 1, which is then interpreted as a probability distribution of the labels: z = softmax(z) The neural network is typically trained by minimizing the cross entropy loss between the true label sdistribution y and the output distribution as the following: DISPLAYFORM1 where m is the number of the labels. In the following, we will use y to denote the true label category, y to denote the one-hot distribution of y, x to denote softmax(x), and H(p, q) to denote the cross entropy between p and q, where p is the distribution that the model needs to approximate, e.g., y in, and q is the distribution generated by the model, e.g., z in. The label embedding is supposed to represent the semantics, i.e. similarity between labels, which makes the length of each label embedding to be the number of the labels m. The embedding is denoted by E ∈ R m×m where m is the number of the labels. Each element in a label embedding vector represents the similarity between two labels. For example, in label y's embedding vector e = E y, the i-th value represents the similarity of label y to label i. To learn the embeddings, a reasonable approach would be to make the label embedding e = E y close to the output z in of the neural network, whose predicted label is y, as the output distribution of the model contains generalization information learned by the neural network. In turn, the label embedding can be used as a more refined supervisory signal for the learning of the model. However, the aforementioned approach affects the learning of the model, in terms of the discriminative power. In essence, the model is supposed to distinguish the inputs, while the label embedding is supposed to capture the commonness of the labels based on the inputs, and the two goals are in conflict. To avoid the conflict, we propose to separate the output representation. One output layer, denoted by o 1, is used to differentiate the hidden representation as normal, which is used for predicting, and the other output layer, denoted by o 2, focuses more on learning the similarity of the hidden representation, from which the label embedding is learned: DISPLAYFORM0 The two output layers share the same hidden representation, but have independent parameters. They both learn from the one-hot distribution of the true label: DISPLAYFORM1 DISPLAYFORM2 In back propagation, the gradient from z 2 is kept from propagating to h, so the learning of the o 2 does not affect the hidden representation. By doing this, the discriminative power of o 1 is maintained and even enhanced by the using of label embedding. In the meanwhile, the label embedding obtains a more stable learning target. The label embedding is then learned by minimizing the cross entropy loss between the normalized embedding e = softmax(e) and the normalized output z 2 = softmax(z 2): DISPLAYFORM3 However, the above approach does not scale properly during the training, as the output z 2 becomes too close to the one-hot distribution y, and the label embedding fails to capture the similarity between labels. To solve this, we apply the softmax with temperature τ to soften the distribution of the normalized z 2, which is computed by DISPLAYFORM4 and the loss becomes DISPLAYFORM5 In the following we will use z 2 to denote the softmax with temperature. By applying a higher temperature, the label embedding gains more details of the output distribution, and the elements in an embedding vector other than the label-based one, i.e. the elements off the diagonal, are better learned. However, the annealed distribution also makes the difference between the incorrect labels closer. To solve the problem, we further propose to regularize the normalized output, so that the highest value of the distribution does not get too high, and the difference between labels is kept: DISPLAYFORM6 If p equals to 1 or 2, the loss is a hinge L1 or L2 regularization. The learned embedding is in turn used in the training of the network by making the output close to the learned embedding. This is done by minimizing the cross entropy loss between the normalized output and the normalized label embedding: DISPLAYFORM7 As a fine-grained distribution of the true label is learned by the model, a faster convergence is achieved, and risk of overfitting is also reduced. In summary, the final objective of the proposed method is as follows: FIG1 shows the overall architecture of the proposed method. Various kinds of neural networks are compatible to generate the hidden representation. In our experiments, we used CNN, ResNet, and Seq-to-Seq. However, the choice may not be limited to those architectures. Moreover, although the output architecture is significantly re-designed, the computational cost does not increase much, as the added operations are relatively cheap in computation. DISPLAYFORM8 When there is a massive number of labels (e.g., over 20,000 labels for neural machine translation), the embedding E takes too much memory. Suppose we have a neural machine translation task with 50,000 labels, then the label embedding is a 50,000 × 50,000 matrix. The embedding matrix alone will take up approximately 9,536MB, which is not suitable for GPU. To alleviate this issue, we propose to re-parameterize the embedding matrix to a product of two smaller matrices, A and B: DISPLAYFORM0 where m is the number of the labels, and h is the size of the "compressed" label embedding. The label embedding for label y is computed as the following: DISPLAYFORM1 where A y means taking out the y-th row from the matrix A. The ing vector e is an mdimensional vector, and can be used as label embedding to substitute the corresponding part in the final loss of a normal label embedding network. The matrix A can be seen as the "compressed" label embeddings, where each row represents a compressed label embedding, and the matrix B can be seen as the projection that reconstructs the label embeddings from the "compressed" forms. This technique can reduce the space needed to store the label embeddings by a factor of m 2h. Considering the previous example, if h = 100, the space needed is reduced by 250x, from 9,536MB to about 38.15MB. We conduct experiments using different models (CNN, ResNet, and Seq-to-Seq) on diverse tasks (computer vision tasks and natural language processing tasks) to show that the proposed method is general-purpose and works for different types of deep learning models. The CIFAR-100 BID13 ) dataset consists of 60,000 32×32 color images in 100 classes containing 600 images each. The dataset is split into 50,000 training images and 10,000 test images. Each image comes with a "fine" label (the class to which it belongs) and a "coarse" label (the superclass to which it belongs). The CIFAR-10 dataset BID13 ) has the same data size as CIFAR-100, that is, 60,000 32×32 color images, split into 50,000 training images and 10,000 test images, except that it has 10 classes with 6,000 images per class. The MNIST handwritten digit dataset BID14 consists of 60,000 28×28 pixel gray-scale training images and additional 10,000 test examples. Each image contains a single numerical digit. We select the first 5,000 images of the training images as the development set and the rest as the training set. consists of more than 2,400,000 social media text-summary pairs. It is split into 2,400,591 pairs for training, 10,666 pairs for development data, and 1,106 pairs for testing. Following, the evaluation metric is ROUGE-1, ROUGE-2 and ROUGE-L BID16. The dataset is from the International Workshop on Spoken Language Translation 2015. The dataset consists of about 136,000 English-Vietnam parallel sentences, constructed from the TED captions. It is split into training set, development set and test set, with 133,317, 1,268 and 1,553 sentence pairs respectively. The evaluation metric is BLEU score BID23. For CIFAR-100 and CIFAR-10, we test our method based on ResNet with 18 layers and 8 layers, respectively, following the settings in BID8. For MNIST, the CNN model consists of two convolutional layers, one fully-connected layer, and another fully-connected layer as the output layer. The filter size is 5 × 5 in the convolutional layers. The first convolutional layer contains 32 filters, and the second contains 64 filters. Each convolutional layer is followed by a max-pooling layer. Following common practice, we use ReLU BID6 as the activation function of the hidden layers. For LCSTS and IWSLT2015, we test our approach based on the sequence-to-sequence model. Both the encoder and decoder are based on the LSTM unit, with one layer for LCSTS and two layer for IWSLT2015. Each character or word is represented by a random initialized embedding. For LCSTS, the embedding size is 400, and the hidden state size of the LSTM unit is 500. For IWSLT2015, the embedding size is 512, and the hidden state size of the LSTM unit is 512. We use beam search for IWSLT2015, and the beam size is 10. Due to the very large label sets, we use the compressed label embedding network (see Section 2.2) for both tasks. Although there are several hyper-parameters introduced in the proposed method, we use a very simple setting for all tasks, because the proposed method is robust in our experiments, and simply works well without fine-tuning. We use temperature τ = 2 for all the tasks. For simplicity, we use the L1 form of the hinge loss of o 2, and α is set to 0.9 for all the tasks. We use the Adam optimizer for all the tasks, using the default hyper-parameters. For CIFAR-100, we divide the learning rate by 5 at epoch 40 and epoch 80. As shown in the previous work BID8, dividing the learning rate at certain iterations proves to be beneficial for SGD. We find that the technique also applies to Adam. We do not apply this technique for CIFAR-10 and MNIST, because the are similar with or without the technique. The experiments are conducted using INTEL Xeon 3.0GHz CPU and NVIDIA GTX 1080 GPU. We run each configuration 20 times with different random seeds for the CV tasks. For the tasks without development sets, we report the at the final epoch. For the ones with development sets, we report the test at the epoch that achieves the best score on development set. First, we show on CIFAR-100 and CIFAR-10, which are summarized in TAB1. As we can see, the proposed method achieves much better . On CIFAR-100, the proposed method achieves 12.4% error reduction ratio from the baseline (ResNet-18). On CIFAR-10, the proposed method achieves 19.5% error reduction ratio from the baseline (ResNet-8). The training time per epoch is similar to the baselines. The of MNIST are summarized in TAB1. As we can see, the proposed method achieves the error rate reduction of over 32%.The detailed error rate curves are shown in FIG2. The 20 repeated runs are shown in lighter color, and the averaged values are shown in deeper color. As we can see from FIG2, the proposed method achieves better convergence speed than the ResNet and CNN baselines. This is because the label embedding achieves soft training of the model, where the conflict of the features of similar labels are alleviated by the learned label embeddings. The learned label embeddings enables the model to share common features when classifying the similar labels, because the supervisory signal contains the information about similarity, thus making the learning easier. Besides, the model is not required to distinguish the labels completely, which avoids unnecessary subtle update of the parameters. In addition, we can see that by using label embedding the proposed method has much more stable training curves than the baselines. The fluctuation of the proposed method is much smaller than the baselines. As the one-hot distribution forces the label to be completely different from others, the original objective seeks unique indicators for the labels, which are hard to find and prone to 17.7 8.5 15.8 Seq2seq (C) 21.5 8.9 18.6 Seq2seq-Attention (W) 26.8 16.1 24.1 Seq2seq-Attention (C) 29.9 17.4 27.2 Seq2seq-Attention (C) (our implementation) 30.1 17.9 27.2 Seq2seq-Attention-LabelEmb (C) (our proposal) 31.7 (+1.6) 19.1 (+1.2) 29.1 (+1.9) overfitting, thus often leading the training astray. The proposed method avoids that by softening the target distribution, so that the features used are not required to be unique, and more common but essential features can be selected, which stabilizes the learning compared to the original objective. DISPLAYFORM0 The proposed method achieves comparable or even better than the state-of-the-art systems. More detailed comparisons to the high performance systems are in Appendix D. It would be interesting to check the learned label embeddings from those datasets. FIG3 shows the learned label embeddings from the CIFAR-100, CIFAR-10, and MNIST tasks, respectively. For the CIFAR-100 task, as we can see, the learned label embeddings are very interesting. Since we don't have enough space to show the heatmap of all of the 100 labels, we randomly selected three groups of labels, with 15 labels in total. For example, the most similar label for the label "bottle" is "can". For the label "bowl", the two most similar labels are "cup" and "plate". For the label "man", the most similar label is "woman", and the second most similar one is "boy".For the CIFAR-10 task, as we can see, the learned label embeddings are also meaningful. For example, the most similar label for the label "automobile" is "truck". For the label "cat", the most similar label is "dog". For the label "deer", the most similar label is "horse". For the MINST task, there are also interesting patterns on the learned label embeddings. Those heatmaps of the learned labels demonstrate that our label embedding learning is reasonable and can indeed reveal rational similarities among diversified labels. The learned embedding can also be used to directly trained a new model on the same task, with improved accuracy and faster convergence, which we will show in Appendix C. First, we show experimental on the LCSTS text summarization task. The are summarized in TAB3. The performance is measured by ROUGE-1, ROUGE-2, and ROUGE-L. As we can see, the proposed method performs much better compared to the baselines, with ROUGE-1 score of 31.7, ROUGE-2 score of 19.1, and ROUGE-L score of 29.1, improving by 1.6, 1.2, and 1.9, respectively. In addition, the of the baseline implemented by ourselves are competitive BLEU Stanford NMT BID18 23.3 NMT (greedy) BID19 25.5 NMT (beam=10) BID19 26.1 Seq2seq-Attention FORMULA6 25.7 Seq2seq-Attention-LabelEmb (beam=10) 26.8 (+1.1) with previous work. In fact, in terms of all of the three metrics, our implementation consistently beats the previous work, and the proposed method could further improve the . Then, we show experimental on the IWSLT2015 machine translation task. The are summarized in TAB4. We measure the quality of the translation by BLEU, following common practice. The proposed method achieves better BLEU score than the baseline, with an improvement of 1.1 points. To our knowledge, 26.8 is the highest BLEU achieved on the task, surpassing the previous best 26.1 BID19. From the experimental , it is clear that the compressed label embedding can improve the of the Seq-to-Seq model as well, and works for the tasks, where there is a massive number of labels. The label embedding learned in compressed fashion also carries semantic similarities. We report the sampled similarities in TAB5. As shown in TAB5, the learned label embeddings capture the semantics of the label reasonably well. For example, the word "đỏ" (red) is most similar to the colors, i.e. "màu" (color), "red" (red), "xanh" (blue), "đen" (black), and "vàng" (yellow). The word "mưa (rain)" is most similar to "bão" (storm), "trời" (sky), "gió" (wind), "cơn" (storm), "nước" (water), which are all semantically related to the natural phenomenon "rain". The of the label embeddings learned in a compressed fashion demonstrate that the re-parameterization technique is effective in saving the space without degrading the quality of the learned label embeddings. They also prove that the proposed label embedding also works for NLP tasks. We propose a method that can learn label representation during the training process of deep neural networks. Furthermore, we propose a solution to automatically learn compressed label embedding, such that the memory cost is substantially reduced. The proposed method can be widely applied to different models. We conducted experiments on CV tasks including CIFAR-100, CIFAR-10, and MNIST, and also on natural language processing tasks including LCSTS and IWSLT2015. Results suggest that the proposed method achieves significant better accuracies than the existing methods (CNN, ResNet, and Seq-to-Seq). Moreover, the learned label embeddings are reasonable and interpretable, which provides meaningful semantics of the labels. We achieve comparable or even better with the state-of-the-art systems on those tasks. To achieve good performance, there are some additional considerations for the proposed method. First, when learning the label embedding, if the current output from the model is wrong, which often happens when the training just begins, the true label's embedding should not learn from the output from the model. This is because the information is incorrect to the learning of the label embedding, and should be neglected. This consideration can be particularly useful to improve the performance under the circumstances where the model's prediction is often wrong during the start of the training, e.g. the CIFAR-100 task and the neural machine translation task. Second, we suggest using the diagonal matrix as the initialization of the label embedding matrix. By using the diagonal matrix, we provide a prior to the label embedding that one label's embedding should be the most similar to the label itself, which could be useful at the start of the training and beneficial for the learning. We also conducted experiments on MNIST, using the MLP model. The MLP model consists of two 500-dimensional hidden layers and one output layer. The other settings are the same as the CNN model. The experimental are summarized in TAB6. As we can see, the proposed label embedding method achieves better performance than the baseline, with an error rate reduction over 24%. All the are the averaged error rates over 20 repeated experiments, and the standard deviation are also shown. FIG4 shows the detailed error rate curve of the MLP model. The 20 repeated runs are shown in light color, and the averaged values are shown in deeper color. As shown, the proposed method also works for MLP, and the are consistently better than the baselines. As the same with the CNN model, the proposed method converges faster than the baseline. In the following section, we will show that the learned label embedding is not only reasonable, but also useful for applications. For example, the learned label embedding can be directly used as finegrained true label distribution to train a new model on the same dataset. For this purpose, the new model's objective function contains two parts, i.e., the original one-hot label based cross entropy objective, together with a label embedding based cross entropy objective. We call this model PreTrained Label Embedding Network. The Label Embedding Network means that the network uses label embedding to improve the training of the network, and the difference of the pre-trained label embedding network from the one presented in Section 2.1 is that in the pre-trained label embedding network, the label embedding is pre-trained and fixed, thus eliminating the need for learning the embedding, while in the label embedding network, the label embedding is learned during the training. In implementation, there are two main differences. First, the label embedding E is fixed and requires no learning. Second, the sub-network o 2, which learns the label embedding, is removed -because there is no need to learn the label embedding again. Thus, the pre-trained label embedding network has the loss function as follows:Loss(x, y; θ) = H(y, z 1) + H(e, z 1)The pre-trained label embedding network is illustrated in Figure 5. Figure 6 shows the of the pre-trained label embedding network, whose label embedding is learned by a normal label embedding network. As we can see, pre-trained label embedding network can achieve much better than the baseline, with faster convergence. It shows that the learned label embedding is effective in improving the performance of the same model and the label embedding indeed contains generalization information, which provides a more refined supervised signal to the model. In this way, the learned label embeddings can be saved and be reused to improve the training of different models on the task, and there is no need to learn the label embedding again. For the CIFAR-100 task, the error rate is typically from 38% to 25% BID4 BID17 BID26 BID27 BID8 BID24 BID3 BID5 BID21. BID4 For CIFAR-10 task, the error rate is typically from 15% to 7% BID28 BID17 BID26 BID24 BID15 BID27 BID8 BID3 BID4. Further improvement can be achieved by finetuning the model and the optimization method BID30 For MNIST task, plain convolutional networks typically achieve error rates ranging widely from more than 1.1% to around 0.4% BID4 BID25 BID27. Data augmentation and other more complicated models can further improve the performance of the models BID28 BID2 BID5, which we believe also work for our method. Srivastava et al. FORMULA0 achieves 0.57% error rate by using Highway Network. BID21 achieves 0.48% error rate by using LSUV initialization for FitNets. Our CNN model achieves the averaged error rate of 0.55%. If considering a good run, our model achieves 0.42%. The prior studies on label representation in deep learning are limited. Existing label representation methods are mostly on traditional methods out of deep learning frameworks. Those label representation methods also adopt the name of label embedding. However, the meaning is different from that in the sense of deep learning. Those label representation methods intend to obtain a representation function for labels. The label representation vector can be data independent or learned from existing information, including training data BID29, auxiliary annotations BID0, class hierarchies BID1, or textual descriptions BID20. For example, in BID10, the label embedding is fixed and is set independently from the data by random projections, and several regressors are used to learn to predict each of the elements of the true label's embedding, which is then reconstructed to the regular one-hot label representation for classification. Another example is the Canonical Correlation Analysis (CCA), which seeks vector a and vector b for random variables X and Y, such that the correlation of the variables a X and b Y is maximized, and then b Y can be regarded as label embeddings BID7 ).There are several major differences between those methods and our proposed method. First, most of those methods are not easy to adapt to deep learning architectures. As previously introduced, those methods come with a totally different architecture and their own learning methods, which are not easy to extend to general-purpose models like neural networks. Instead, in the proposed method, label embedding is automatically learned from the data by back propagation. Second, the label representation in those methods is not adapting during the training. In BID10, the label embedding is fixed and randomly initialized, thus revealing none of the semantics between the labels. The CCA method is also not adaptively learned from the training data. In all, their learned label representation lacks interaction with other model parameters, while label embeddings obtained from our proposed method both reveal the semantics of the labels and interact actively with the other parts of the model by back propagation. There have also been prior studies on so-called "soft labels". The soft label methods are typically for binary classification BID22, where the human annotators not only assign a label for an example, but also give information on how confident they are regarding the annotation. The side information can be used in the learning procedure to alleviate the noise from the data and produce better . The main difference from our method is that the soft label methods require additional annotation information (e.g., the confidence information of the annotated labels) of the training data, while our method does not need additional annotation information, and the "soft" probability is learned during the training in a simple but effective manner. Moreover, the proposed method is not restricted to binary classification. There have also been prior studies on model distillation in deep learning that uses label representation to better compress a big model into a smaller one. In deep learning, it's common sense that due to the non-convex property of the neural network functions, different initialization, different data order, and different optimization methods would cause varied of the same model. Model distillation BID9 ) is a novel method to combine the different instances of the same model into a single one. In the training of the single model, its target distribution is a combination of the output distributions of the previously trained models. Our method is substantially different compared with the model distillation method. The motivations and designed architectures are both very different. The model distillation method adopts a pipeline system, which needs to first train a large model or many different instances of models, and then use the label representation of the baseline models to provide better supervisory signals to re-train a smaller model. This pipeline setting is very different from our single-pass process setting. Our method also enables the ability to learn compressed label embedding for an extremely large number of labels. Moreover, for a given label, the label representation in their method is different from one example to another. That is, they do not provide a universal label representation for a label, which is very different compared with our setting.

Represent the short summary for finding the corresponding scientific article (long)

Learning Label Representation for Deep Networks

Represent the TLDR for finding the corresponding scientific article

To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.