Question: Represent the article for finding a single-sentence TLDR

Question: We analyze the joint probability distribution on the lengths of the vectors of hidden variables in different layers of a fully connected deep network, when the weights and biases are chosen randomly according to Gaussian distributions, and the input is binary-valued. We show that, if the activation function satisfies a minimal set of assumptions, satisfied by all activation functions that we know that are used in practice, then, as the width of the network gets large, the ``length process'' converges in probability to a length map that is determined as a simple function of the variances of the random weights and biases, and the activation function. We also show that this convergence may fail for activation functions that violate our assumptions. second layer may not be independent, even for some permissible φ like the ReLU. The of this section contradict claims made in BID13 BID9. Section 5 describes some simulation experiments verifying some of the findings of the paper, and illustrating the dependence among the values of the hidden nodes. Our analysis of the convergence of the length map borrows ideas from Daniely, et al. BID1, who studied the properties of the mapping from inputs to hidden representations ing from random Gaussian initialization. Their theory applies in the case of activation functions with certain smoothness properties, and to a wide variety of architectures. Our analysis treats a wider variety of values of σ w and σ b, and uses weaker assumptions on φ. For n ∈ N, we use [n] to denote the set {1, 2, . . ., n}. If T is a n × m × p tensor, then, for i ∈ [n], let T i,:,: = T i,j,k jk, and define T i,j,:, etc., analogously. Consider a deep fully connected width-N network with D layers. Let W ∈ R D×N ×N. An activation function φ maps R to R; we will also use φ to denote the function from R N to R N obtained by applying φ componentwise. Computation of the neural activity vectors x 0,:,..., x D,: ∈ R N and preactivations h 1,:,..., h D,: ∈ R N proceeds in the standard way as follows:h,: = W,:,: x −1,: + b,: x,: = φ(h,:), for = 1,..., D.We will study the process arising from fixing an arbitrary input x 0,: ∈ {−1, 1} N and choosing the parameters independently at random: the entries of W are sampled from Gauss 0, DISPLAYFORM0 Note that for all ≥ 1, all the components of h,: and x,: are identically distributed. For the purpose of defining a limit, assume that, for a fixed, arbitrary function χ: N → {−1, 1}, for finite N, we have x 0,: = (χ,..., χ(N)). For > 0, if the limit exists (in the sense of "convergence in distribution"), let x be a random variable whose distribution is the limit of the distribution of x,1 as N goes to infinity. Define h and q similarly. If P and Q are probability distributions, then d T V (P, Q) = sup E P (E) − Q(E), and if p and q are their densities, DISPLAYFORM0 In this section we characterize the length map of the hidden nodes of a deep network, for all activation functions satisfying the following assumptions. Definition 1 An activation function φ is permissible if, (a) the restriction of φ to any finite interval is bounded; (b) |φ(x)| = exp(o(x 2)) as |x| gets large. 1; and (c) φ is measurable. Conditions (b) and (c) ensure that a key integral can be computed. DISPLAYFORM0 If φ is permissible, then, since φ(cz) 2 exp(−z 2 /2) is integrable for all c, we have that q 0,...,q D,r 0,...,r D are well-defined finite real numbers. The following theorem shows that the length map q 0,..., q D converges in probability toq 0,...,q D.Theorem 2 For any permissible φ, σ w, σ b ≥ 0, any depth D, and any, δ > 0, there is an N 0 such that, for all N ≥ N 0, with probability 1 − δ, for all ∈ {0, ..., D}, we have |q −q | ≤.The rest of this section is devoted to proving Theorem 2. Our proof will use the weak law of large numbers. For any random variable X with a finite expectation, and any, δ > 0, there is an N 0 such that, for all N ≥ N 0, if X 1,..., X N are i.i.d. with the same distribution as X, then DISPLAYFORM0 In order to divide our analysis into cases, we need the following lemma, whose proof is in Appendix B.Lemma 4 If φ is permissible and not zero a.e., for all σ w > 0, for all ∈ {0, ..., D},q > 0 and r > 0.We will also need a lemma that shows that small changes in σ lead to small changes in Gauss(0, σ 2).Lemma 5 (see BID7) There is an absolute constant C such that, for all σ 1, σ 2 > 0, DISPLAYFORM1 The following technical lemma is proved in Appendix C. If φ is permissible, for all 0 < r ≤ s, for all β > 0, there is an a ≥ 0 such that, for all q ∈ [r, s], DISPLAYFORM0 Armed with these lemmas, we are ready to prove Theorem 2.First, if φ is zero a.e., or if σ w = 0, Theorem 2 follows directly from Lemma 3, together with a union bound over the layers. Assume for the rest of the proof that φ(x) is not zero a.e., and that σ w > 0, so thatq > 0 andr > 0 for all. DISPLAYFORM1,i. Our proof of Theorem 2 is by induction. The inductive hypothesis is that, for any, δ > 0 there is an N 0 such that, if N ≥ N 0, then, with probability 1 − δ, for all ≤, |q −q | ≤ and |r −r | ≤.The base case holds because q 0 =q 0 = r 0 =r 0 = 1, no matter what the value of N is. Now for the induction step; choose > 0, 0 < < min{q /4,r} and 0 < δ ≤ 1/2. (Note that these choices are without loss of generality.) Let ∈ (0,) take a value that will be described later, using quantities from the analysis. By the inductive hypothesis, whatever the value of, there is an N 0 such that, if N ≥ N 0, then, with probability 1 − δ/2, for all ≤ − 1, we have |q −q | ≤ and |r −r | ≤. Thus, to establish the inductive step, it suffices to show that, after conditioning on the random choices before the th layer, if |q −1 −q −1 | ≤, and |r −1 −r −1 | ≤, there is an N such that, if N ≥ N, then with probability at least 1 − δ/2 with respect only to the random choices of W,:,: and b,:, that |q −q | ≤ and |r −r | ≤. Given such an N, the inductive step can be satisfied by letting N 0 be the maximum of N 0 and N.Let us do that. For the rest of the proof of the inductive step, let us condition on outcomes of the layers before layer, and reason about the randomness only in the th layer. Let us further assume that |q −1 −q −1 | ≤ and |r −1 −r −1 | ≤. DISPLAYFORM2 Since we have conditioned on the values of h −1,1,..., h −1,N, each component of h,i is obtained by taking the dot-product of x −1,: = φ(h −1,:) with W,i,: and adding an independent b,i. Thus, conditioned on h −1,1,..., h −1,N, we have that h,1,..., h,N are independent. Also, since x −1,: is fixed by conditioning, each h,i has an identical Gaussian distribution. Since each component of W and b has zero mean, each h,i has zero mean. Choose an arbitrary i ∈ [N]. Since x −1,: is fixed by conditioning and W,i,1,..., W,i,N and b,i are independent, DISPLAYFORM3 We wish to emphasize the q is determined as a function of random outcomes before the th layer, and thus a fixed, nonrandom quantity, regarding the randomization of the th layer. By the inductive hypothesis, we have DISPLAYFORM4 The key consequence of this might be paraphrased by saying that, to establish the portion of the inductive step regarding q, it suffices for q to be close to its mean. Now, we want to prove something similar for r. We have DISPLAYFORM5 which gives DISPLAYFORM6 Since |q −q | ≤ σ 2 w and we may choose to ensure ≤q 2σ 2 w, we haveq /2 ≤ q ≤ 2q.For β > 0 and κ ∈ (0, 1/2) to be named later, by Lemma 6, we can choose a such that, for all q ∈ [q /2, 2q], DISPLAYFORM7 We claim that DISPLAYFORM8 So now we are trying to bound DISPLAYFORM9 Using changes of variables, we have DISPLAYFORM10 But since, for κ < 1/2, conditioning on an event of probability at least 1 − κ only changes a distribution by total variation distance at most 2κ, and therefore, applying Lemma 5 along with the fact that |q −q | ≤ σ 2 w, for the constant C from Lemma 5, we get DISPLAYFORM11 Tracing back, we have DISPLAYFORM12 If κ = min{24M, Recall that q is an average of N identically distributed random variables with a mean between 0 and 2q (which is therefore finite) and r is an average of N identically distributed random variables, each with mean between 0 andr + /2 ≤ 2r. Applying the weak law of large numbers (Lemma 3), there is an N such that, if N ≥ N, with probability at least 1 − δ/2, both |q − E[q]| ≤ /2 and |r − E[r]| ≤ /2 hold, which in turn implies |q −q | ≤ and |r −r | ≤, completing the proof of the inductive step, and therefore the proof of Theorem 2. In this section, we show that, for some activation functions, the probability distribution of hidden nodes can have some surprising properties. In this subsection, we will show that the hidden variables are sometimes not Gaussian. Our proof will refer to the Cauchy distribution. Definition 2 A distribution over the reals that, for x 0 ∈ R and γ > 0, has a density f given by FIG1 DISPLAYFORM0 DISPLAYFORM1 So, for all N, h 2,1 is Cauchy(0, √ N). Suppose that h 2,1 converged in distribution to some distribution P. Since the cdf of P can have at most countably many discontinuities, we can cover the real line by a countable set of finite-length FIG1,... whose endpoints are points of continuity for P. Since Cauchy(0, DISPLAYFORM2 Thus, the probability assigned by P to the entire real line is 0, a contradiction. The following contradicts a claim made on line 8 of Section A.1 of BID13 .Theorem 10 If φ is either the ReLU or the Heaviside function, then, for every σ w > 0, σ b ≥ 0, and N ≥ 2, FIG1 are not independent. Proof: We will show that E[h DISPLAYFORM0, which will imply that h 2,1 and h 2,2 are not independent. As mentioned earlier, because each component of h 1,: is the dot product of x 0,: with an independent row of W 1,:,: plus an independent component of b 1,:, the components of h 1,: are independent, and since x 1,: = φ(h 1,:), this implies that the components of x 1,: are independent. Since each row of W 1,:,: and each component of the bias vector has the same distribution, x 1,: is i.i.d. We have DISPLAYFORM1 The components of W 2,:,: and x 1,:, along with b 2,1, are mutually independent, so terms in the double sum with i = j have zero expectation, and E[h DISPLAYFORM2 . For a random variable x with the same distribution as the components of x 1,:, this implies DISPLAYFORM3 Similarly, DISPLAYFORM4 Putting this together with, we have ) and Gauss(0, σ 2) for σ estimated from the data (shown in red). Now, we calculate the difference using for the Heaviside and ReLU functions. DISPLAYFORM5 Heaviside. Suppose φ is Heaviside function, i.e. φ(z) is the indicator function for z > 0. In this case, since the components of h 1,: are symmetric about 0, the distribution of x 1,: is uniform over DISPLAYFORM6 DISPLAYFORM7 dz. By symmetry this is DISPLAYFORM8 Similarly, E[DISPLAYFORM9 2 . Plugging these into we get that, in the case the φ is the ReLU, that DISPLAYFORM10 completing the proof. Here, we show, informally, that for φ at the boundary of the second condition in the definition of permissibility, the recursive formula defining the length mapq breaks down. Roughly, this condition cannot be relaxed. For any α > 0, if φ is defined by φ(x) = exp(αx 2), there exists a σ w, σ b s.t.q,r is undefined for all ≥ 2. For each N ∈ {10, 100, 1000}, we (a) initialized the weights 100 times, (b) plotted the histograms of all of the values of h [2, :], along with the Cauchy(0, √ N) distribution from the proof of Proposition 9, and Gauss(0, σ 2) for σ estimated from the data. Consistent with the theory, the Cauchy(0, √ N) distribution fits the data well. To illustrate the fact that the values in the second hidden layer are not independent, for N = 1000 and the parameters otherwise as in the other experiment, we plotted histograms of the values seen in the second layer for nine random initializations of the weights in FIG5. When some of the values in the first hidden layer have unusually small magnitude, then the values in the second hidden layer coordinately tend to be large. This is in contrast with the claim made at the end of Section 2.2 of BID9. Note that this is consistent with Theorem 2 establishing convergence in probability for permissible φ, since the φ used in this experiment is not permissible.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We prove that, for activation functions satisfying some conditions, as a deep network gets wide, the lengths of the vectors of hidden variables converge to a length map.

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: Represent the scientific article to find a short TLDR

Question: Data augmentation is one of the most effective approaches for improving the accuracy of modern machine learning models, and it is also indispensable to train a deep model for meta-learning. However, most current data augmentation implementations applied in meta-learning are the same as those used in the conventional image classification. In this paper, we introduce a new data augmentation method for meta-learning, which is named as ``Task Level Data Augmentation'' (referred to Task Aug). The basic idea of Task Aug is to increase the number of image classes rather than the number of images in each class. In contrast, with a larger amount of classes, we can sample more diverse task instances during training. This allows us to train a deep network by meta-learning methods with little over-fitting. Experimental show that our approach achieves state-of-the-art performance on miniImageNet, CIFAR-FS, and FC100 few-shot learning benchmarks. Once paper is accepted, we will provide the link to code. Although the machine learning systems have achieved a human-level ability in many fields with a large amount of data, learning from a few examples is still a challenge for modern machine learning techniques. Recently, the machine learning community has paid significant attention to this problem, where few-shot learning is the common task for meta-learning (e.g., ; ; ;). The purpose of few-shot learning is to learn to maximize generalization accuracy across different tasks with few training examples. In a classification application of the few-shot learning, tasks are generated by sampling from a conventional classification dataset; then, training samples are randomly selected from several classes in the classification dataset. In addition, a part of the examples is used as training examples and testing examples. Thus, a tiny learning task is formed by these examples. The meta-learning methods are applied to control the learning process of a base learner, so as to correctly classify on testing examples. Data augmentation is widely used to improve the training of deep learning models. Usually, the data augmentation is regarded as an explicit form of regularization;;. Thus, the data augmentation aims at artificially generating the training data by using various translations on existing data, such as: adding noises, cropping, flipping, rotation, translation, etc. The general idea of data augmentations is increasing the number of data by change data slightly to be different from original data, but the data still can be recognized by human. The new data involved in the classes are identical to the original data. However, the minimum units of meta-learning are the tasks rather than data. Increasing the data of original class cannot increase the types of task instances. Therefore, "Task Aug" increases the data that can be clearly recognized as the different classes as the original data. With novel classes, the more diverse task instances can be generated. This is important for the meta-learning, since metalearning models must predict unseen classes during the testing phase. Therefore, a larger number of classes is helpful for models to generate task instances with different classes. In this work, the natural images are augmented by being rotated 90, 180, 270 degrees (we show examples in Figure 1). We compare two cases, 1) the new images are converted to the classes of original images and 2) the new images are separated to the new classes. The proposed method is evaluated by experiments with the state of art meta-learning . The experimental analysis shows that Task Aug can reduce over-fitting and improve the performance, while the conventional data augmentation (referred to Data Aug) of rotation, which converts the novel data into the classes of original data, does not improve the performance and even causes the worse . In the comparative experiments, Task Aug achieves the best accuracy of the meta-learning methods applied. Besides, the best of our experiments exceed the current state-of-art over a large margin. Meta-learning involves two hierarchies learning processes: low-level and high-level. The low-level learning process learns to deal with general tasks, often termed as the "inner loop"; and the highlevel learning process learns to improve the performance of a low-level task, often termed as the "outer loop". Since models are required to handle sensory data like images, deep learning methods are often applied for the "outer loop". However, the machine learning methods applied for the "inner loop" are very diverse. Based on different methods in the "inner loop", meta-learning can be applied in image recognition;;;; , image generation;; , reinforce learning; , and etc. This work focuses on few-shot learning image recognition based on meta-learning. Therefore, in the experiment, the methods applied in the "inner loop" are able to classify data, and they are K-nearest neighbor (KNN), Support Vector Machine (SVM) and ridge regression, respectively;;. Previous studies have introduced many popular regularization techniques to few-shot learning from deep learning, such as weight decay, dropout, label smooth, and data augmentation. Common data augmentation techniques for image recognition are usually designed manually and the best augmentation strategies depend on dataset. However, in natural color image datasets, random cropping and random horizontal flipping are the most common. Since the few-shot learning tasks consist of natural color images, the random horizontal flipping and random cropping are applied in few-shot learning. In addition, color (brightness, contrast, and saturation) jitter is often applied in the works of few-shot learning;. Other data augmentation technologies related to few-shot learning include generating samples by few-shot learning and generating samples for few-shot learning. The former tried to synthesize additional examples via transferring, extracting, and encoding to create the data of the new class, that are intra-class relationships between pairs of reference classes' data instances;. The later tried to apply meta-learning in a few-shot generation to generate samples from other models. In addition to these two types of studies, the data augmentation technology most closed to the new proposed approach is applied to Omniglot dataset, which consists of handwritten words. They created the novel classes by rotating the original images 90, 180 and 270 degrees. However, this approach cannot be applied for the natural color image directly, and we will explain the reasons and the solutions in Section 3. We adopt the formulation purposed by to describe the N -way K-shot task. A few-shot task contains many task instances (denoted by T i), each instance is a classification problem consisting of the data sampled from N classes. The classes are randomly selected from a classes set. The classes set are split into M tr, M val and M test for a training class set C tr, a validation classes set C val, and a test classes set C test. In particular, each class cannot overlap others (i.e., the classes used during testing are unseen classes during training). Data is randomly sampled from C tr, C val and C test, so as to create task instances for training meta-set S tr, validation meta-set S val, and test meta-set S test, respectively. The validation and testing meta-sets are used for model selection and final evaluation, respectively. The data in each task instance, T i, are divided into training examples D tr and validation examples D val. Both of them only contains the data from N classes which sampled from the appropriate classes set randomly (for a task instance applied during training, the classes form a subset of the training classes set C tr). In most settings, the training set.. K} consists of K data instances from each class, this processing usually called as a "shot". The validation set, D val, consists of several other data instances from the same classes, this processing is usually called as a "query". An evaluation is provided for generalization performance on the N classification task instance D tr. Note that: the validation set of a task instance D val (for optimizing model during "outer loop") is different from the held-out validation classes set C val and meta-set S val (for model selection). This work is to increase the size of the training classes set, M tr, by rotating all images within the training classes set with 90, 180, 270 degrees. The size, M tr, is increased for three times. In the Omniglot dataset consisting of handwritten words , this approach works well, since it can rotate a handwritten word multiple of 90 degrees and treat the new one as another word; in addition, it is really possible that the novel word is similar to some words, which are not included in the training classes but existed. However, for natural images, it is not the same cause. For examples, the images in the third line of Figure 1 are difficult to identify which images are rotated. Moreover, the images are rarely rotated in the photos taken by humans. Despite the two problems, the fundamental features of the novel images can provide useful information. We assign novel classes that contain smaller weights than the original classes, so as to make models prioritize learning the features of the original classes, and make the features of the novel classes as a supplement to prevent the augmented data from taking up large capacity in the model. The smaller weights are implemented in two ways, 1) lower probability and 2) delay selecting the novel classes. For a class in a task instance, the probability of the class coming from the novel classes is p, and the probability coming from the original classes is 1 − p. Besides, The initial p is set to 0, then linearly rises from 0 to p max after generating T task instances. The max probability p max is set lower than the proportion of the novel classes in all classes to make each novel class have a lower probability than each original class. The whole process of Task Aug on a classes set is summarized in Algorithm 1 and Figure 2. In this work, we also compare the methods with the training protocol with ensemble method in addition to the standard training protocol, which choosing a model by the validation set. The training protocol with an ensemble method use the models with different training epoch to Algorithm 1 Task Level Data Augmentation. Require: Classes set C = {c 1, c 2, . . ., c M}; Max possibility for Task Aug p max; The delay to Task Aug T; The current count t; The number of ways, shots and queries N, K, H 1: Rotate all x ∈ {x|(x, y) ∈ D} 90r degrees 17: an ensemble model, in order to better use the models obtained in a single training process, and this approach has been proved to be valid for meta-learning by experiments. We adopt this ensemble method. However, unlike and that we did not use cyclic annealing for learning rate and any methods to select models. We directly took the average of the prediction of all models, which are saved according to an interval of 1 epoch. In Section 4, the methods with this ensemble approach are marked by "+ens". We evaluate the proposed method on few-shot learning tasks. In order to ensure fair, both the of baseline and Task Aug were run in our own environment. The comparative experiment is designed to answer the following questions: Is Task Aug able to improve the performance of meta-learning? How much should the probably for the novel classes be set? Will converting the novel data into the classes of the original data cause worse , which are generated by being rotated 90, 180, 270 degrees?, we used ResNet-12 network in our experiments. The ResNet-12 network had four residual blocks which contains three 3 × 3 convolution, batch normalization and Leaky ReLU with 0.1 negative slope. One 2 × 2 max-pooling layer is used for reducing the size of the feature map. The numbers of the network channels were 64, 160, 320 and 640, respectively. DropBlock regularization is used in the last two residual blocks, the conventional dropout is used in the first two residual blocks. The block sizes of DropBlock were set to 2 and 5 for CIFAR derivatives and ImageNet derivatives, respectively. In all experiments, the dropout possibility was set to 0.1. The global average pooling was not used for the final output of the last residual block. For ProtoNets, we did not use a higher way for training than testing like. Instead, the equal number of shot and way were used in both training and evaluation, and its output multiplied by a learnable scale before the softmax following;, For M-SVM, we set training shot to 5 for CIFAR-FS; 15 for FC100; and 15 for miniImageNet; regularization parameter of SVM was set to 0.1; and a learnable scale was used following. We did not use label smoothing like, because we did not find that label smoothing can improve the performance in our environment. This was also affirmed from the author's message on GitHub, that Program language packages and environment might affect of the meta-learning method. For R2-D2, we set the same training shot as for M-SVM, and used a learnable scale and bias following. It was different from we used a fixed regularization parameter of ridge regression which was set to 50 because has confirmed that making it learnable might not be helpful. Last, for all methods, each class in a task instance contained 6 test (query) examples during training and 15 test (query) examples during testing. Stochastic gradient descent (SGD) was used. , we set weight decay and Nesterov momentum to 0.0005 and 0.9, respectively. Each mini-batch contained 8 task instances. The meta-learning model was trained for 60 epochs, and 1000 mini-batchs for each epoch. We set the initial learning rate to 0.1, then multiplied it by 0.06, 0.012, and 0.0024 at epochs 20, 40 and 50, respectively, as in. The , which are marked by "ens" were used the 60 models saved after each epoch to become an ensemble model. For the final epoch, the training classes set was augmented by the validation classes set. We chose the model at the epoch where we got the best model during training on the training classes set only. The of the final run are marked by "+val" in this subsection. For data augmentation, we adopted random random crop, horizontal flip, and color (brightness, saturation, and contrast) jitter data augmentation following the work of;. We set p max to 0.5 for CIFAR-FS and FC100; 0.25 for miniImageNet; and T was set to 80000 for all experiments. The FC100 are also derived from CIFAR-100 , and the 100 classes are grouped into 20 superclasses. The training, validation, and testing classes contain 60 classes from 12 superclasses, 20 classes from 4 superclasses, and 20 classes from 4 superclasses, respectively. The target is to minimize the information overlap between classes to make it more challenging than current few-shot classification tasks. Same as CIFAR-FS, there are 600 nature color images of size 32 × 32 in each class. Results. In Table 1, we compare our with the previous studies, and the table shows that the highest accuracies of our experiments exceeded the current state-of-art accuracies from 3% to 5%. Besides, Table 2 and Table 3 summarize the on the CIFAR-FS and FC100 5-way tasks, and in most cases our method rises accuracy by 0.5%-3%. Figure 3: The accuracies (%) on meta-test sets with varying probability p max for the novel classes. The 95% confidence interval is denoted by the shaded region. In general, the performance of Task Aug on most of the regimes is better than Data Aug and baseline. To identify whether the rotation multi 90 degrees for Task Aug is better than that for Data Aug, we analyzed the experiment on CIFAR-FS and miniImageNet. The linear rising of p was also used for Data Aug, and T = 80000 for both Task Aug and Data Aug. In the analysis, the training classes set was not augmented by the validation classes set. As shown in Figure 3, we observed that: with p max, the accuracy rises at first, reaches the peaks between 0.25 and 0.5, then declines and reaches baseline when p max = 0.75 at the end, which is the proportion of the novel classes in all classes. On the other hand, the rotation multi 90 degrees for Data Aug can not improve or even cause worse performance.

Represent the short summary for finding the corresponding scientific article (long)

We propose a data augmentation approach for meta-learning and prove that it is valid.

Represent the TLDR for finding the corresponding scientific article!

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.


Question: Represent the scientific article to find a short TLDR

Question: In this paper, we present a general framework for distilling expectations with respect to the Bayesian posterior distribution of a deep neural network, significantly extending prior work on a method known as ``Bayesian Dark Knowledge. " Our generalized framework applies to the case of classification models and takes as input the architecture of a ``teacher" network, a general posterior expectation of interest, and the architecture of a ``student" network. The distillation method performs an online compression of the selected posterior expectation using iteratively generated Monte Carlo samples from the parameter posterior of the teacher model. We further consider the problem of optimizing the student model architecture with respect to an accuracy-speed-storage trade-off. We present experimental investigating multiple data sets, distillation targets, teacher model architectures, and approaches to searching for student model architectures. We establish the key that distilling into a student model with an architecture that matches the teacher, as is done in Bayesian Dark Knowledge, can lead to sub-optimal performance. Lastly, we show that student architecture search methods can identify student models with significantly improved performance. Deep learning models have shown promising in the areas including computer vision, natural language processing, speech recognition, and more (; a; b; ;). However, existing point estimation-based training methods for these models may in predictive uncertainties that are not well calibrated, including the occurrence of confident errors. It is well-known that Bayesian inference can often provide more robust posterior predictive distributions in the classification setting compared to the use of point estimation-based training. However, the integrals required to perform Bayesian inference in neural network models are also well-known to be intractable. Monte Carlo methods provide one solution to representing neural network parameter posteriors as ensembles of networks, but this can require large amounts of both storage and compute time . To help overcome these problems, introduced an interesting model training method referred to as Bayesian Dark Knowledge. In the classification setting, Bayesian Dark Knowledge attempts to compress the Bayesian posterior predictive distribution induced by the full parameter posterior of a "teacher" network into a "student" network. The parameter posterior of the teacher network is represented through a Monte Carlo ensemble of specific instances of the teacher network (the teacher ensemble), and the analytically intractable posterior predictive distributions are approximated as Monte Carlo averages over the output of the networks in the teacher ensemble. The major advantage of this approach is that the computational complexity of prediction at test time is drastically reduced compared to computing Monte Carlo averages over a large ensemble of networks. As a , methods of this type have the potential to be much better suited to learning models for deployment in resource constrained settings. In this paper, we present a Bayesian posterior distillation framework that generalizes the Bayesian Dark Knowledge approach in several significant directions. The primary modeling and algorithmic contributions of this work are: we generalize the target of distillation in the classification case from the posterior predictive distribution to general posterior expectations; we generalize the student architecture from being restricted to match the teacher architecture to being a free choice in the distillation procedure. The primary empirical contributions of this work are evaluating the distillation of both the posterior predictive distribution and expected posterior entropy across a range of models and data sets including manipulations of data sets that increase posterior uncertainty; and evaluating the impact of the student model architecture on distillation performance including the investigation of sparsity-inducing regularization and pruning for student model architecture optimization. The key empirical findings are that distilling into a student model that matches the architecture of the teacher, as in , can be sub-optimal; and student architecture optimization methods can identify significantly improved student models. We note that the significance of generalizing distillation to arbitrary posterior expectations is that it allows us to capture a wider range of useful statistics of the posterior that are of interest from an uncertainty quantification perspective. As noted above, we focus on the case of distilling the expected posterior entropy in addition to the posterior predictive distribution itself. When combined with the entropy of the posterior predictive distribution, the expected posterior entropy enables disentangling model uncertainty (epistemic uncertainty) from fundamental uncertainty due to class overlap (aleatoric uncertainty). This distinction is extremely important in determining why predictions are uncertain for a given data case. Indeed, the difference between these two terms is the basis for the Bayesian active learning by disagreement (BALD) score used in active learning, which samples instances with the goal of minimizing model uncertainty . The remainder of this paper is organized as follows. In the next section, we begin by presenting material and related work in Section 2. In Section 3, we present the proposed framework and associated Generalized Posterior Expectation Distillation (GPED) algorithm. In Section 4, we present experiments and . Additional details regarding data sets and experiments can be found in Appendix A, with supplemental included in Appendix B. In this section we present material on Bayesian inference for neural networks, and related work on approximate inference, and model compression and pruning. Let p(y|x, θ) represent the probability distribution induced by a deep neural network classifier over classes y ∈ Y = {1, .., C} given feature vectors x ∈ R D. The most common way to fit a model of this type given a data set D = {(x i, y i)|1 ≤ i ≤ N } is to use maximum conditional likelihood estimation, or equivalently, cross entropy loss minimization (or their penalized or regularized variants). However, when the volume of labeled data is low, there can be multiple advantages to considering a full Bayesian treatment of the model. Instead of attempting to find the single (locally) optimal parameter set θ * according to a given criterion, Bayesian inference uses Bayes rule to define the posterior distribution p(θ|D, θ 0) over the unknown parameters θ given a prior distribution P (θ|θ 0) with prior parameters θ 0 as seen in Equation 1. For prediction problems in machine learning, the quantity of interest is typically not the parameter posterior itself, but the posterior predictive distribution p(y|x, D, θ 0) obtained from it as seen in Equation 2. The primary problem with applying Bayesian inference to neural network models is that the distributions p(θ|D, θ 0) and p(y|x, D, θ 0) are not available in closed form, so approximations are required, which we discuss in the next section. Most Bayesian inference approximations studied in the machine learning literature are based on variational inference (VI) or Markov Chain Monte Carlo (MCMC) methods . In VI, an auxiliary distribution q φ (θ) is defined to approximate the true parameter posterior p(θ|D, θ 0). The variational parameters φ are selected to minimize the Kullback-Leibler (KL) divergence between q φ (θ) and p(θ|D, θ 0). first studied applying VI to neural networks. later presented a method based on stochastic VI with improved scalability. In the closely related family of expectation propagation (EP) methods , present an online EP algorithm for neural networks with the flexibility of representing both continuous and discrete weights. Hernández- present the probabilistic backpropagation (PBP) algorithm for approximate Bayesian learning of neural network models, which is an example of an assumed density filtering (ADF) algorithm that, like VI and EP, generally relies on simplified posterior densities. The main drawback of VB, EP, and ADF is that they all typically in biased posterior estimates for complex posterior distributions. MCMC methods provide an alternative family of sampling-based posterior approximations that are unbiased, but are often computationally more expensive to use at training time. MCMC methods allow for drawing a correlated sequence of samples θ t ∼ p(θ|D, θ 0) from the parameter posterior. These samples can then be used to approximate the posterior predictive distribution as a Monte Carlo average as shown in Equation 3. Neal addressed the problem of Bayesian inference in neural networks using Hamiltonian Monte Carlo (HMC) to provide a set of posterior samples. A bottleneck with this method is that it uses the full dataset when computing the gradient needed by HMC, which is problematic for larger data sets. While this scalability problem has largely been solved by more recent methods such as stochastic gradient Langevin dynamics (SGLD) , the problem of needing to compute over a large set of samples when making predictions at test or deployment time remains. Bayesian Dark Knowledge is precisely aimed at reducing the test-time computational complexity of Monte Carlo-based approximations for neural networks. In particular, the method uses SGLD to approximate the posterior distribution using a set of posterior parameter samples. These samples can be thought of as an ensemble of neural network models with identical architectures, but different parameter values. This posterior ensemble is used as the "teacher" in a distillation process that trains a single "student" model to match the teacher ensemble's posterior predictive distribution . The major advantage of this approach is that it can drastically reduce the test time computational complexity of posterior predictive inference relative to using a Monte Carlo average computed using many samples. Finally, we note that with the advent of Generative Adversarial Networks , there has also been work on generative models for approximating posterior sampling. and both propose methods for learning to generate samples that mimic those produced by SGLD. However, while these approaches may provide a speed-up relative to running SGLD itself, the ing samples must still be used in a Monte Carlo average to compute a posterior predictive distribution in the case of Bayesian neural networks. This is again a potentially costly operation and is exactly the computation that Bayesian Dark Knowledge addresses. As noted above, the problem that Bayesian Dark Knowledge attempts to solve is reducing the test-time computational complexity of using a Monte-Carlo posterior to make predictions. In this work, we are particularly concerned with the issue of enabling test-time speed-storage-accuracy trade-offs. The relevant material includes methods for network compression and pruning. Previous work has shown that overparameterised deep learning models tend to show much better learnability. Further, it has also been shown that such overparameterised models rarely use their full capacity and can often be pruned back substatially without significant loss of generality (; ; ; ; ; ;).; are some examples that use Group LASSO regularization at their core. use hierarchical priors to prune neurons instead of weights. An advantage of these methods over ones which induce connection-based sparsity is that these methods directly produce smaller networks after pruning (e.g., fewer units or channels) as opposed to networks with sparse weight matrices. This makes it easier to realize the ing computational savings, even on platforms that do not directly support sparse matrix operations. In this section, we describe our proposed framework for distilling general Bayesian posterior expectations for neural network classification models and discuss methods for enabling test-time speed-storage-accuracy trade-offs for flexible deployment of the ing models. There are many possible inferences of interest given a Bayesian parameter posterior P (θ|D, θ 0). We consider the general case of inferences that take the form of posterior expectations as shown in Equation 4 where g(y, x, θ) is an arbitrary function of y, x and θ. Important examples of functions g(y, x, θ) include g(y, x, θ) = p(y|x, θ), which in a posterior expectation yielding the posterior predictive distribution p(y|x, D, θ 0), as used in Bayesian Dark Knowledge; g(y, x, θ) = C y =1 p(y |x, θ) log p(y |x, θ), which yields the posterior predictive entropy H(y|x, D, θ 0) 1; and g(y, x, θ) = p(y|x, θ)(1 − p(y|x, θ)), which in the posterior marginal variance σ 2 (y|x, D, θ 0). While the posterior predictive distribution p(y|x, D, θ 0) is certainly the most important posterior inference from a predictive standpoint, the entropy and variance are also important from the perspective of uncertainty quantification. Our goal is to learn to approximate posterior expectations E p(θ|D,θ 0) [g(y, x, θ)] under a given teacher model architecture using a given student model architecture. The method that we propose takes as input the teacher model p(y|x, θ), the prior p(θ|θ 0), a labeled data set D, an unlabeled data set D, the function g(y, x, θ), a student model f (y, x|φ), an online expectation estimator, and a loss function (·, ·) that measures the error of the approximation given by the student model f (y, x|φ). Similar to , we propose an online distillation method based on the use of the SGLD sampler. We describe all of the components of the framework in the sections below, and provide a complete description of the ing method in Algorithm 1. We define the prior distribution over the parameters p(θ|θ 0) to be a spherical Gaussian distribution centered at µ = 0 with precision τ (we thus have θ 0 = [µ, τ]). We define S to be a minibatch of size M drawn from D. θ t denotes the parameter set sampled for the teacher model at sampling iteration t, while η t denotes the step size for the teacher model at iteration t. The Langevin noise is denoted by z t ∼ N (0, η t I). The sampling update for SGLD is given by: 1 Note that the posterior predictive entropy represents the average entropy integrated over the parameter posterior. It is not equal to the entropy of the posterior predictive distribution p(y|x, D, θ 0) in general. Distillation Procedure: For the distillation learning procedure, we make use of a secondary unlabeled data set D = {x i |1 ≤ i ≤ N}. This data set could use feature vectors from the primary data set D, or a larger data set. We note that due to autocorrelation in the sampled teacher model parameters θ t, we may not want to run a distillation update for every Monte Carlo sample drawn. We thus use two different iteration indices: t for SGLD iterations and s for distillation iterations. On every distillation step s, we sample a minibatch S from D of size M. For every data case i in S, we update an estimateĝ yis of the posterior expectation using the most recent parameter sample θ t, obtaining an updated estimateĝ yis+1 ≈ E p(θ|D,θ 0) [g(y, x, θ)] (we discuss update schemes in the next section). Next, we use the minibatch of examples S to update the student model. To do so, we take a step in the gradient direction of the regularized empirical risk of the student model as shown below where α s is the student model learning rate, R(φ) is the regularizer, and λ is the regularization hyper-parameter. We next discuss the estimation of the expectation targetsĝ yis. Expectation Estimation: Given an explicit collection of posterior samples θ 1,..., θ s, the standard Monte Carlo estimate of However, this estimator requires retaining the sequence of samples θ 1,..., θ s, which may not be feasible in terms of storage cost. Instead, we consider the application of an online update function. We define m is to be the count of the number of times data case i has been sampled up to and including distillation iteration s. An online update function U (ĝ yis, θ t, m is) takes as input the current estimate of the expectation, the current sample of the model parameters, and the number of times data case i has been sampled, and produces an updated estimate of the expectationĝ yis+1. Below, we define two different versions of the function. U s (ĝ yis, θ t, m is), updatesĝ yis using the current sample only, while U o (ĝ yis, θ t, m is) performs an online update equivalent to a full Monte Carlo average. We note that both update functions provide unbiased estimates of The online update U o will generally in much lower variance in the estimated values ofĝ yis, but it comes at the cost of needing to explicitly maintain the expectation estimatesĝ yis across learning iterations, increasing the storage cost of the algorithm. It is worthwhile noting that the extra storage and computation cost required by U o grows linearly in the size of the training set for the student. By contrast, the fully stochastic update is memoryless in terms of past expectation estimates, so the estimated expectationsĝ yis do not need to be retained across iterations ing in a space savings. We show a complete description of the proposed method in Algorithm 1. The algorithm takes as input the teacher model p(y|x, θ), the parameters of the prior P (θ|θ 0), a labeled data set D, an unlabeled data set D, the function g(y, x, θ), the student model f (y, x|φ), an online expectation estimator U (ĝ yis, θ t, m is), a loss function (·, ·) that measures the error of the approximation given by f (y, x|φ), a regularization function R and regularization hyper-parameter λ, minibatch sizes M and M, the thinning interval parameter H, the SGLD burn-in time parameter B and step size schedules for the step sizes η t and α s. We note that the original Bayesian Dark Knowledge method is recoverable as a special case of this framework via the the choices g(y, x, θ) = p(y|x, θ), (p, q) = −p log(q), U = U s and p(y|x, θ) = f (y, x, φ) (e.g., the architecture of the student is selected to match that of the teacher). The original approach also uses a distillation data set D obtained from D by adding randomly Initialize s = 0, φ 0, θ 0,ĝ yi0 = 0, m i0 = 0,η 0 3: Sample S from D with |S| = M 5: if mod (t, H) = 0 and t > B then Sample S from D with |S | = M 8: end for 12: 13: end if end for 16: end procedure generated noise to instances from D on each distillation iteration, taking advantage of the fact that the choice U = U s means that no aspect of the algorithm scales with |D |. Our general framework allows for other trade-offs, including reducing the variance in the estimates ofĝ yis at the cost of additional storage in proportion to |D |. We also note that note that the loss function (p, q) = −p log(q) and the choice g(y, x, θ) = p(y|x, θ) are somewhat of a special case when used together as even when the full stochastic expectation update U s is used, the ing distillation parameter gradient is unbiased. To distill posterior entropy, we set g(y, x, θ) = y∈Y p(y|x, θ) log p(y|x, θ), U = U o and (h, h) = |h − h |. One of the primary motivations for the original Bayesian Dark Knowledge approach is that it provides an approximate inference framework that in significant computational and storage savings at test time. However, a drawback of the original approach is that the architecture of the student is chosen to match that of the teacher. As we will show in Section 4, this will sometimes in a student network that has too little capacity to represent a particular posterior expectation accurately. On the other hand, if we plan to deploy the student model in a low resource compute environment, the teacher architecture may not meet the specified computational constraints. In either case, we need a general approach for selecting an architecture for the student model. To begin to explore this problem, we consider to basic approaches to choosing student model architectures that enable trading off test time inference speed and storage for accuracy. A helpful aspect of the distillation process relative to a de novo architecture search problem is that the architecture of the teacher model is available as a starting point. As a first approach, we consider wrapping the proposed GPED algorithm with an explicit search over a set of student models that are "close" to the teacher. Specifically, we consider a search space obtained by starting from the teacher model and applying a width multiplier to the width of every fully connected layer and a kernel multiplier to the number of kernels in every convolutional layer. While this search requires exponential time in the number of layers, it provides a baseline for evaluating other methods. As an alternative approach with better computational complexity, we leverage the regularization function R(φ) included in the GPED framework to prune a large initial network using group 1 / 2 regularization . To apply this approach, we first must partition the parameters in the parameter vector φ across K groups G k. The form of the regularizer is As is well-established in the literature, this regularizer causes all parameters in a group to go to zero simultaneously when they are not needed in a model. To use it for model pruning for a unit in a fully connected layer, we collect all of that unit's inputs into a group. Similarly, we collect all of the incoming weights for a particular channel in a convolution layer together into a group. If all incoming weights associated with a unit or a channel have magnitude below a small threshold, we can explicitly remove them from the model, obtaining a more compact architecture. We also fine-tune our models after pruning. Finally, we note that any number of weight compressing, pruning, and architecture search methods could be combined with the GPED framework. Our goal is not to exhaustively compare such methods, but rather to demonstrate that GPED is sensitive to the choice of student model to highlight the need for additional research on the problem of selecting student model architectures. In this section, we present experiments and evaluating the proposed approach using multiple data sets, posterior expectations, teacher model architectures, student model architectures and basic architecture search methods. We begin by providing an overview of the experimental protocols used. Data Sets: We use the MNIST and CIFAR10 data sets as base data sets in our experiments. In the case of MNIST, posterior predictive uncertainty is very low, so we introduce two different modifications to explore the impact of uncertainty on distillation performance. The first modification is simply to subsample the data. The second modification is to introduce occlusions into the data set using randomly positioned square masks of different sizes, ing in masking rates from 0% to 86.2%. For CIFAR10, we only use sub-sampling. Full details for both data sets and the manipulations applied can be found in Appendix A.1. We evaluate a total of three teacher models in this work: a three-layer fully connected network (FCNN) for MNIST matching the architecture used by , a four-layer convolutional network for MNIST, and a five-layer convolutional network for CIFAR10. Full details of the teacher model architectures are given in Appendix A.2. For exhaustive search for student model architectures, we use the teacher model architectures as base models and search over a space of layer width multipliers K 1 and K 2 that can be used to expand sets of layers in the teacher models. A full description of the search space of student models can be found in Appendix A.2. Distillation Procedures: We consider distilling both the posterior predictive distribution and the posterior entropy, as described in the previous section. For the posterior predictive distribution, we use the stochastic expectation estimator U s while for entropy we used the full online update U o. We allow B = 1000 burn-in iterations and total of T = 10 6 training iterations. The prior hyper-parameters, learning rate schedules and other parameters vary by data set or distillation target and are fully described in Appendix A.2. For this experiment, we use the MNIST and CIFAR10 datasets without any subsampling or masking. For each dataset and model, we consider separately distilling the posterior predictive distribution and the posterior entropy. We fix the architecture of the student to match that of the teacher. To evaluate the performance while distilling the posterior predictive distribution, we use the negative log-likelihood (NLL) of the model on the test set. For evaluating the performance of distilling posterior entropy, we use the mean absolute difference between the teacher ensemble's entropy estimate and the student model output on the test set. The are given in Table 1. First, we note that the FCNN NLL on MNIST closely replicate the in , as expected. We also note that the error in the entropy is low for both the FCNN and CNN architectures on MNIST. However, the student model fails to match the NLL of the teacher on CIFAR10 and the entropy MAE is also relatively high. In Experiment 2, we will investigate the effect of increasing uncertainty on models applied to both data sets, while in Experiment 3 we will search for student model architectures that improve performance. No. of training samples Difference between teacher and student posterior entropy estimates on test data set. In the plots above, S denotes the student and T denotes the teacher. This experiment builds on Experiment 1 by exploring methods for increasing posterior uncertainty on MNIST (sub-sampling and masking) and CIFAR10 (sub-sampling). We consider the cross product of four sub-sampling rates and six masking rates for MNIST and three sub-sampling rates for CIFAR10. We consider the posterior predictive distribution and posterior entropy distillation targets. For the posterior predictive distribution we report the negative log likelihood (NLL) of the teacher, and the NLL gap between the teacher and student. For entropy, we report the mean absolute error between the teacher ensemble and the student. All metrics are evaluated on held-out test data. We also restrict the experiment to the case where the student architecture matches the teacher architecture, mirroring the Bayesian Dark Knowledge approach. In Figure 1, we show the for the convolutional models on MNIST and CIFAR10 respectively. The FCNN are similar to the CNN on MNIST and are shown in Figure 4 in Appendix B. In Appendix B, we also provide a performance comparison between the U o and U s estimators while distilling posterior expectations. As expected, the the NLL of the teacher decreases as the data set size decreases. We observe that changing the number of training samples has a similar effect on NLL gap for both CIFAR10 and MNIST. More specifically, for any fixed masking rate of MNIST (and zero masking rate for CIFAR10), we can see that the NLL difference between the student and teacher decreases with increasing training data. However, for MNIST we can see that the teacher NLL increases much more rapidly as a function of the masking rate. Moreover, the gap between the teacher and student peaks for moderate values of the masking rate. This fact is explained through the observation that when the masking rate is low, posterior uncertainty is low, and distillation is relatively easy. On the other hand, when the masking rate is high, the teacher essentially outputs the uniform distribution for every example, which is very easy for the student to represent. As a , the moderate values of the masking rate in the hardest distillation problem and thus the largest performance gap. For varying masking rates, we see exactly the same trend for the gap in posterior entropy predictions on MNIST. However, the gap for entropy prediction increases as a function of data set size for CIFAR10. Finally, as we would expect, the performance of distillation using the U o estimator is almost always better than that of the U s estimator (refer Appendix B). The key finding of this experiment is simply that the quality of the approximations provided by the student model varies as a function of properties of the underlying data set. Indeed, restricting the student architecture to match the teacher can sometimes in significant performance gaps. In the next experiment, we address the problem of searching for improved student model architectures. In this experiment, we compare the exhaustive search to the group 1 / 2 (group lasso) regularizer combined with pruning. For the pruning approach, we start with the largest student model considered under exhaustive search, and prune back from there using different regularization parameters λ, leading to different student model architectures. We present in terms of performance versus computation time (estimated in FLOPS), as well as performance vs storage cost (estimated in number of parameters). As performance measures for the posterior predictive distribution, we consider accuracy and negative log likelihood. For entropy, we use mean absolute error. In all cases are reported on test data. We consider both fully connected and convolutional models. Figure 2 shows for negative the log likelihood (NLL) of the convolutional model on MNIST with masking rate 29% and 60,000 training samples. We select this setting as illustrative of a difficult case for posterior predictive distribution distillation. We plot NLL vs FLOPS and NLL vs storage for all points encountered in each search. The solid blue line indicates the Pareto frontier. First, we note that the baseline student model (with architecture matching the teacher) from Experiment 2 on MNIST achieves an NLL of 0.469 at approximately 0.48 × 10 6 FLOPs and 0.03 × 10 6 parameters on this configuration of the data set. We can see that both methods for selecting student architectures provide a highly significant improvement over the baseline student architectures. On MNIST, the NLL is reduced to 0.30. Further, we can also see that the group 1 / 2 approach is able to obtain much better NLL at the same computation and storage cost relative to the exhaustive search method. Lastly, the group 1 / 2 method is able to obtain models on MNIST at less than 50% the computational cost needed by the baseline model with only a small loss in performance. Results for other models and distillation targets show similar trends and are presented in Appendix B. Additional experimental details are given in Appendix A.2. In summary, the key finding of this experiment is that the capacity of the student model has a significant impact on the performance of the distillation procedure, and methods for optimizing the student architecture are needed to achieve a desired speed-storage-accuracy trade-off. We have presented a framework for distilling expectations with respect to the Bayesian posterior distribution of a deep neural network that generalizes the Bayesian Dark Knowledge approach in several significant directions. Our show that the performance of posterior distillation can be highly sensitive to the architecture of the student model, but that basic architecture search methods can help to identify student model architectures with improved speed-storage-accuracy trade-offs. There are many directions for future work including considering the distillation of a broader class of posterior statistics including percentiles, assessing and developing more advanced student model architecture search methods, and applying the framework to larger state-of-the-art models. A DATASETS AND MODEL DETAILS As noted earlier in the paper, the original empirical investigation of Bayesian Dark Knowledge for classification focused on the MNIST data set . However, the models fit to the MNIST data set have very low posterior uncertainty and we argue that it is thus a poor benchmark for assessing the performance of posterior distillation methods. In this section, we investigate two orthogonal modifications of the standard MNIST data set to increase uncertainty: reducing the training set size and masking regions of the input images. Our goal is to produce a range of benchmark problems with varying posterior predictive uncertainty. We also use the CIFAR10 data set in our experiments and employ the same subsampling technique. The full MNIST dataset consists of 60,000 training images and 10,000 test images, each of size 28 × 28, distributed among 10 classes. As a first manipulation, we consider sub-sampling the labeled training data to include 10,000, 20,000, 30,000 or all 60,000 data cases in the primary data set D when performing posterior sampling for the teacher model. Importantly, we use all 60,000 unlabeled training cases in the distillation data set D. This allows us de-couple the impact of reduced labeled training data on posterior predictive distributions from the effect of the amount of unlabeled data available for distillation. As a second manipulation, we generate images with occlusions by randomly masking out parts of each available training and test image. For generating such images, we randomly choose a square m × m region (mask) and set the value for pixels in that region to 0. Thus, the masking rate for a 28 × 28 MNIST image corresponding to the mask of size m × m is given by r = m×m 28×28. We illustrate original and masked data in Figure 3. We consider a range of square masks ing in masking rates between 0% and 86.2%. The full CIFAR10 dataset consists of 50,000 training images and 10,000 test images, each of size 32 × 32 pixels. We sub-sample the data into a primary training sets D containing 10,000, 20,000, and 50,000 images. As with MNIST, the sub-sampling is limited to training the teacher model only and we utilize all the 50,000 unlabeled training images in the distillation data set D. To demonstrate the generalizability of our methods to a range of model architectures, we run our experiments with both fully-connected, and convolutional neural networks. We note that our goal in this work is not to evaluate the GPED framework on state-of-the-art architectures, but rather to provide illustrative and establish methodology for assessing the impact of several factors including the level of uncertainty and the architecture of the student model. Teacher Models: We begin by defining the architectures used for the teacher model as follows: We use a 3-layer fully connected neural network. The architecture used is: Input For a CNN, we use two consecutive sets of 2D convolution and maxpooling layers, followed by two fully-connected layers. The architecture used is: Input(1,)-Conv(num kernels=10, kernel size=4, stride=1) -MaxPool(kernel size=2) -Conv(num kernels=20, kernel size=4, stride=1) -MaxPool(kernel size=2) -FC -FC (output). Similar to the CNN architecture used for MNIST, we use two consecutive sets of 2D convolution and max-pooling layers followed by fully-connected layers. In the architectures mentioned above, the "output" size will change depending on the expectation that we're distilling. For classification, the output size will be 10 for both datasets, while for the case of entropy, it will be 1. We use ReLU non-linearities everywhere between the hidden layers. For the final output layer, softmax is used for classification. In the case of entropy, we use an exponential activiation to ensure positivity. The student models used in our experiments use the above mentioned architectures as the base architecture. For explicitly searching the space of the student models, we use a set of width multipliers starting from the teacher architecture. The space of student architectures corresponding to each teacher model defined earlier is given below. The width multiplier values of K 1 and K 2 are determined differently for each of the experiments, and thus will be mentioned in later sections. Model and Distillation Hyper-Parameters: We run the distillation procedure using the following hyperparameters: fixed teacher learning rate η t = 4 × 10 −6 for models on MNIST and η t = 2 × 10 −6 for models on CIFAR10, teacher prior precision τ = 10, initial student learning rate α s = 10 −3, student dropout rate p = 0.5 for fully-connected models on MNIST (and zero otherwise), burn-in iterations B = 1000, thinning interval H = 100 for distilling predictive means and H = 10 for distilling entropy values, and total training iterations T = 10 6. For training the student model, we use the Adam algorithm (instead of plain steepest descent as indicated in Algorithm 1) and set a learning schedule for the student such that it halves its learning rate every 200 epochs for models on MNIST, and every 400 epochs for models on CIFAR10. Also, note that we only apply the regularization function R(φ s) while doing Group 1 / 2 pruning. Otherwise, we use dropout as indicated before. Hyper-parameters for Group 1 / 2 pruning experiments: For experiments involving group 1 / 2 regularizer, the regularization strength values λ are chosen from a log-scale ranging from 10 −8 to10−3. When using Group 1 / 2 regularizer, we do not use dropout for the student model. The number of fine-tuning epochs for models on MNIST and CIFAR100 are 600 and 800 respectively. At the start of fine-tuning, we also reinitialize the student learning rate α t = 10 −4 for fully-connected models and α t = 10 −3 for convolutional models. The magnitude threshold for pruning is = 10 −3. Supplemental Results for Experiment 2: Robustness to Uncertainty In Figure 4, we demonstrate the of Experiment 2 (Section 4.3), on fully-connected networks for MNIST. Additionally, in Tables [2- Figure 11 : Accuracy-Storage-Computation tradeoff while using CNNs on CIFAR10 with subsampling training data to 20,000 samples. (a) Test accuracy using posterior predictive distribution vs FLOPS found using exhaustive search. (b) Test accuracy using posterior predictive distribution vs FLOPS found using group 1 / 2 with pruning. (c) Test accuracy using posterior predictive distribution vs storage found using exhaustive search. (d) Test accuracy using posterior predictive distribution vs storage found using group 1 / 2 with pruning. The optimal student model for this configuration is obtained with group 1 / 2 pruning. It has approximately 5.4× the number of parameters and 5.6× the FLOPS of the base student model.

A general framework for distilling Bayesian posterior expectations for deep neural networks.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: represent the article for finding a single-sentence tldr

Question: Variational Autoencoders (VAEs) have proven to be powerful latent variable models. How- ever, the form of the approximate posterior can limit the expressiveness of the model. Categorical distributions are flexible and useful building blocks for example in neural memory layers. We introduce the Hierarchical Discrete Variational Autoencoder (HD-VAE): a hi- erarchy of variational memory layers. The Concrete/Gumbel-Softmax relaxation allows maximizing a surrogate of the Evidence Lower Bound by stochastic gradient ascent. We show that, when using a limited number of latent variables, HD-VAE outperforms the Gaussian baseline on modelling multiple binary image datasets. Training very deep HD-VAE remains a challenge due to the relaxation bias that is induced by the use of a surrogate objective. We introduce a formal definition and conduct a preliminary theoretical and empirical study of the bias. Unsupervised learning has proven powerful at leveraging vast amounts of raw unstructured data (; ; ;). Through unsupervised learning, latent variable models learn the explicit likelihood over an unlabeled dataset with an aim to discover hidden factors of variation as well as a generative process. An example hereof, is the Variational Autoencoder (VAE) that exploits neural networks to perform amortized approximate inference over the latent variables. This approximation comes with limitations, both in terms of the latent prior and the amortized inference network . It has been proposed to go beyond Gaussian priors and approximate posterior using, for instance, autoregressive flows , a hierarchy of latent variables (Sønderby et al., 2016; Maaløe et al., 2016 Maaløe et al.,, 2019, a mixture of priors or discrete distributions (van den ; ; ; b,a;). Current state-of-the-art deep learning models are trained on web-scaled datasets and increasing the number of parameters has proven to be a way to yield remarkable . Nonetheless, time complexity and GPU memory are scarce resources, and the need for both resources increases linearly with the depth of neural network. and showed that large memory layers are an effective way to increase the capacity of a model while reducing the computation time. showed that discrete variational distributions are analogous to neural memory , which can be used to improve generative models . Also, memory values are yet another way to embed data, allowing for applications such as one-shot transfer learning and semi-supervised learning that scales . Depth promises to bring VAEs to the next frontier (Maaløe et al., 2019). However, the available computing resources may shorten that course. Motivated by the versatility and the scalability of discrete distributions, we introduce the Hierarchical Discrete Variational Autoencoder. HD-VAE is a VAE with a hierarchy of factorized categorical latent variables. In contrast to the existing discrete latent variable methods, our model (a) is hierarchical, (b) trained using Concrete/Gumbel-Softmax, (c) relies on a conditional prior that is learned end-to-end and (d) uses a variational distribution that is parameterized as a large stochastic memory layer. Despite being optimized for a biased surrogate objective we show that a shallow HD-VAE outperforms the baseline Gaussian-based models on multiple binary images datasets in terms of test log-likelihood. This motivates us to introduce a definition of the relaxation bias and to measure how it is affected by the configuration of latent variables. Hierarchical VAE Hierarchical VAEs define a model p θ (x, z) = p θ (x|z)p θ (z) where x is an observed variable and z = {z 1, . . ., z L} is a hierarchy of latent variables so that p θ (z) is factorized into L layers. The inference model q φ (z|x) usually exploits the inverse dependency structure. A vanilla hierarchical VAE in the following model: The choice of the VAE architecture is independent of the choice of the variational family and deeper models can easily be defined (see appendix F). Variational Neural Memory Each stochastic layer consists of N categorical random variables with K class probabilities π = {π 1, . . ., π K} and can be parametrized as a memory layer. recently proposed a scalable approach to attention-based memory layers that can be directly translated to the stochastic setting: Each categorical distribution is parametrized by factored keys {k 1, . . ., k K}, k i ∈ R d 1 and a parametric query model Q(h). If {v 1, ..., v K}, v i ∈ R d 2 are the memory values, for c ∈ R and i = 1,..., K, then the output of the memory layer for one variable is Optimization We wish to maximize the Evidence Lower Bound (ELBO): The subscript of L denotes the number of importance weighted samples. Guided by the analysis of Sønderby et al., we chose to use the Concrete/GumbelSoftmax relaxation for differentiable, approximate sampling of categorical variables. A relaxed categorical sample can be obtained as where {g i} are i.i.d. samples drawn from Gumbel, and τ ∈ R * + is a temperature parameter. As in the categorical case, the output of the memory layer is a convex combination of the memory values weighted by the entries ofz: The relaxed samplesz follow a Concrete/Gumbel-Softmax distribution q τ φ which depends on τ and converges to the categorical distribution q τ =0 φ = q φ as τ → 0 which is equivalent to applying the Gumbel-Max trick to soft samples, meaning z = H(z), H = one hot • arg max. When we extend the definition of f θ,φ to the domain of the relaxed samples, as in appendix D, the surrogate objective that is maximized becomes which is not guaranteed to be a lower bound of log p θ (x). Hence, we are interested in the relaxation bias that we define as: where is the original ELBO. If f θ,φ is a κ-Lipschitz for z, we can derive an upper bound for the relaxation bias as well as a new log-likelihood bound (relaxed ELBO) by adding a corrective term to the surrogate objective (derivation in appendix C). For a one layer Ladder Variational Autoencoder (LVAE), it in the following bounds: This new bound shows that, if the model is unconstrained, the relaxation bias is free to grow and that it grows with the number of discrete variables. In section 4.2, we provide empirical supporting the monotonically increasing property of the relaxation bias with regards to the number of stochastic units. Table 1: Sample estimates of the KL(q φ,θ (z|x)||p θ (z)) and the ELBO for 1000 importance weighted hard samples (τ = 0) using the same LVAE architecture and hyperparameters across all datasets. discrete normal discrete normal discrete normal discrete normal discrete normal discrete normal discrete normal discrete normal discrete normal discrete normal discrete normal discrete normal. To the best of our knowledge, HD-VAE is the only work that attempts to transform memory layers into a general purpose variational distribution. We trained HD-VAE for different number of layers of latent variables using the surrogate objective defined in the equation 5. In this experiment, we observe that HD-VAE consistently outperforms the baseline Gaussian model for multiple datasets and different number of latent layers (table 1). This shows that using variational memory layers yields a more flexible model than for the VAE with a Gaussian prior and the same number of latent variables. Furthermore, optimizing latent variable models is challenging (Sønderby et al., 2016;). In this experiment, the measured KL is higher for the discrete model, suggesting a well-tempered optimization behavior. Finally, we observe that increasing the depth of HD-VAE consistently improves on the log-likelihood, with a limit of three layers latent layers. The relaxation bias (section 2) may increase with the number of discrete latent variables. We trained HD-VAE for different numbers of stochastic units and different depths on Binarized MNIST using the surrogate objective defined in the equation 5. We measured the relaxation bias δ τ =0.1 on the test set (figure 1, table 4). The relaxation bias monotonically increases with the total number of discrete latent variables for different numbers of latent variables. This may explain why we found that HD-VAE with a large number of latent variables is not yet competitive with the Gaussian counterparts. In this preliminary research, we have introduced a design for variational memory layers and shown that it can be exploited to build hierarchical discrete VAEs, that outperform Gaussian prior VAEs. However, without explicitly constraining the model, the relaxation bias grows with the number of latent layers, which prevents us from building deep hierarchical models that are competitive with state-of-the-art methods. In future work we will attempt to harness the relaxed-ELBO to improve the performance of the HD-VAE further. Optimization During training, we mitigate the posterior collapse using the freebits strategy with λ = 2 for each stochastic layer. A dropout of 0.5 is used to avoid overfitting. We linearly decrease the temperature τ from 0.8 to 0.3 during the first 2 · 10 5 steps and from 0.3 to 0.1 during the next 2 · 10 5 steps. We use the Adamax optimizer with initial learning rate of 2 · 10 −3 for all parameters except for the memory values that are trained using a learning rate of 2 · 10 −2 to compensate for sparsity. We use a batch size of 128. All models are trained until they overfit and we evaluate the log-likelihood using 1000 importance weighted samples . Despite its large number of parameters, HD-VAE seems to be more robust to overfitting, which may be explained by the sparse update of the memory values. Runtime Sparse CUDA operations are currently not used, which means there is room to make HD-VAE more memory efficient. Even during training, one may truncate the relaxed samples to benefit from the sparse optimizations. The table 3 shows the average elapsed time training iteration as well as the memory usage for a 6 layers LVAE with 6 × 16 stochastic units and K = 16 2 and batch size of 128. Table 4: Measured one-importance-weighted ELBO on binarized MNIST for a LVAE model with different number of layers and different numbers of stochastic units using relaxed (τ = 0.1) and hard samples (τ = 0). We report N = L l=1 n l, where n l relates to the number of latent variables at the layer l and we set K = 256 for all the variables. Let x be an observed variable, and consider a VAE model with one layer of N categorical latent variables z = {z 1, . . ., z N} each with K classes. The generative model is p θ (x, z) and the inference model is q φ (z|x). For a temperature parameter τ > 0, the equivalent relaxed concrete variables are denotedẑ = {ẑ 1, . . .,ẑ N},ẑ i ∈ K. We define H = one hot • arg max and , using the Gumbel-Max trick, one can notice that We now assume that f θ,φ,x is κ-Lipschitz for L 2. Then, by definition, The relaxation bias can therefore be bounded as follows: Furthermore, we can define the adjusted Evidence Lower Bound for relaxed categorical variables (relaxed-ELBO): As shown by the experiment presented in the section 4.2, the quantity L τ >0 1 (θ, φ) appears to be a positive quantity. Furthermore, as the model attempts to exploit the relaxation of z to maximize the surrogate objective, one may consider that is a tight bound of δ τ (θ, φ), meaning that the relaxed-ELBO is a tight lower bound of the ELBO. The relaxed-ELBO is differentiable and may enable automatic control of the temperature as left and right terms of the relaxed-ELBO seek respectively seek for high and low temperature. κ-Lipschitz neural networks can be designed using Weight Normalization or Spectral Normalization . Nevertheless handling residual connections and multiple layers of latent variables is not trivial. We note however that in the case of a one layer VAE, one only needs to constrain the VAE decoder to be κ-Lispchitz as the surrogate objective is computed as In the appendix E, we show how the relaxed-ELBO can be extended to multiple layers of latent variables in the LVAE setting. Appendix D. Defining f θ,φ on the domain of the relaxed Categorical Variablesz f θ,φ is only defined for categorical samples. For relaxed samplesz, we define f θ,φ as:. The introduction of the function H is necessary as the terms (b) and (c) are only defined for categorical samples. This expression remains valid for hard samplesz. During training, relaxing the expressions (b) and (c) can potentially yield gradients of lower variance. In the case of a single categorical variable z described by the set of K class probabilities π = {π 1, ...π K}. One can define: Alternatively, asides from being a relaxed Categorical distribution, the Concrete/GumbelSoftmax also defines a proper continuous distribution. When treated as such, this in a proper probabilistic model with continuous latent variables, and the objective is unbiased. In that case, the density is given by We consider now an LVAE model: In the following, we will leave the conditioning on x implicit for convenience. The ELBO estimated with relaxed samples (relaxed-ELBO) is: The correct ELBO can be rewritten as follows: In this section we define the VAE (; ;), the LVAE (Sønderby et al., 2016) and BIVA (Maaløe et al., 2019). All models are characterized by a generative model p θ (x, z) = p θ (x|z)p θ (z) and can be coupled with any variational distribution. Variational Autoencoder (VAE) Variational Autoencoder with Skip-Connections (Skip-VAE) Ladder Variational Autoencoder (LVAE) Ladder Variational Autoencoder with Skip-Connections (Skip-LVAE) Bidirectional Variational Autoencoder (BIVA)

represent the tldr for finding the corresponding scientific article!

In this paper, we introduce a discrete hierarchy of categorical latent variables that we train using the Concrete/Gumbel-Softmax relaxation and we derive an upper bound for the absolute difference between the unbiased and the biased objective.

Represent the short summary for finding the corresponding scientific article (long)

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.