Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We design simple and quantifiable testing of global translation-invariance in deep learning models trained on the MNIST dataset. Experiments on convolutional and capsules neural networks show that both models have poor performance in dealing with global translation-invariance; however, the performance improved by using data augmentation. Although the capsule network is better on the MNIST testing dataset, the convolutional neural network generally has better performance on the translation-invariance. Convolutional neural networks (CNN) have achieved state-of-the-art performance than the human being on many computer vision tasks BID6; BID2. The deep learning community trend to believe that the success of CNN mainly due to two key features in CNN, reduced computation cost with weight sharing in convolutional layers and generalization with local invariance in subsampling layers BID7; BID8. Due to convolutional layers are'place-coded' equivariant and max-pooling layers are local invariant BID1, CNN has to learn different models for different viewpoints which need big data and expensive cost. More Generalization model should be able to train on a limited range of viewpoints and getting good performance on a much more wider range. Capsule network is robust in dealing with different viewpoints BID3 BID9; BID4. Capsules are a group of neurons which includes the pose, colour, lighting and deformation of the visual entity. Capsule network aims for'rate-coded' equivariance because it's the weights that code viewpoint-invariant knowledge, not the neural activities. Viewpoint changes in capsule network are linear effects on the pose matrices of the parts and the whole between different capsules layers. However, it still unclear whether capsule networks be able to generalize for global translation invariance. Visualize and Quantify the translation-invariance in deep learning model are essential for understanding the architectural choices and helpful for developing Generalization model that is invariant to viewpoint changes. An analysis using translation-sensitivity map for MNIST digit dataset has been used to investigate translation invariance in CNN BID5. In this paper, we introduce a simple method to test the performance of global translation-invariance in convolutional and capsule neural network models trained on the MNIST dataset. Global translational invariance (GTI) of a deep learning model trained on the MINST data which can test by using a simple testing dataset. All images in the GTI testing dataset generated by shifting the centre of mass of a Helvetica font digit from top left to bottom right one pixel each time as shown in FIG0. The TI testing images size is 28 × 28 which the same size as MNIST images. Locations of the centre of mass for each digit has 18 columns and 14 rows. In total there are 18 × 14 × 10 = 2520 testing images, which cover all the possible cases of translational translations. We train all deep learning models using MNIST training dataset which includes 60000 samples, and testing on both MNIST testing dataset with 10000 samples and GTI testing dataset with 2520 samples. Nearly all the images in the MNIST dataset are located at the centre of canvas while the GTI dataset distributes uniformly on the canvas. Compare to translation-sensitivity maps method based on average on numerous images of each class in MNIST testing dataset and shift it to all possible translations , our way is robust to overcome random noise due to miss labelling in the MNIST testing dataset and our GTI training dataset is much smaller. Since the GTI training dataset is identical for testing all different models, it can capture the tiny difference in those models. Another advantage of the GTI dataset is it natural to quantify the global invariance because the accuracy of model predictions on the GTI testing dataset reflects the ability of the model to dealing with global translational invariance. The CNN model has nine layers as shown in FIG1. Layer 1, 2, 4, 5 are convolutional layers with 32, 32, 64, and 64 channels respectively. Each layer has 3 × 3 filters and a stride of 1. Layer 3 and 6 are max-pooling layers, and layer 7, 8, 9 are fully connected layers with size 256, 128 and 10 respectively. Dropout with the drop rate of 0.5 applies to the first max-pooling layer and all the fully connected layers. The total number of parameters is 361578 which is about 23 times smaller than the Capsule networks. Except the last layer's activation function is softmax, all the other layers use ReLU. The optimizer is Adam with default parameters in Keras, and the objective function is cross entropy loss. The of CNN shown in FIG2 and TAB0. The CNN models trained on MNIST data, thus it achieves very high accuracy on the MNIST testing set. However, the model trained on the MNIST training dataset without any data augmentation has only 42.16% accuracy on GTI testing dataset, which implies CNN's performance on dealing with global translational invariance is abysmal. As we can see in the left picture of FIG2, images with the digit's centre of mass around the centre of the canvas predicted correctly, and images with the number at the corner assigned to an incorrect class. Those images around the centre are classified correctly due to the max-pooling layers preserve the local invariance in feature maps. Since only MNIST images used to train the model, and the model could not accurately predict those images shift toward the corner in GTI dataset, which strongly suggests that CNN is'place-code' equivariant. 5 5 5 8 0 5 0 0 0 6 6 6 6 6 6 1 7 7 5 5 5 2 2 2 2 2 6 6 6 6 6 6 6 6 7 7 5 5 5 2 2 2 2 2 2 6 6 6 6 6 1 7 7 7Without Augmentation Shift 20% in training data To improve the performance of CNN on the GTI dataset, we train the MNIST with data augmentation by shifting the image from the centre in x and y-direction. The accuracy on GTI testing dataset increase to 98.05% by randomly moving the centre of an MNIST training image in x and y-direction up to 30% of width or hight. Data Augmentation of training dataset to improve the performance imply place-code' equivariance in CNN, because those neurons at the corner of feature maps are activated when the model start to see training sample with objects at the edge. We test GTI dataset on CapsNet with the same architecture in BID9 bashed on the. The CapsNet has 8.2M parameters that are about 23 times larger than the CNN. We trained the model with Adam optimizer and using exponential decay of learning rate with decay parameter 0.9. We use margin lass with the same parameters in BID9. We also add reconstruction loss but scale down it by 0.0005.Capsule network is robust in viewpoints invariance of object recognition; however, our experiment implies capsule network's performance on global invariance has yet to be improved. As we can see in the left of FIG3, the model trained on the MNIST without data augmentation fails to predict the class correctly and also generated incorrect images if the digit close to the edge. Data augmentation in MNIST training dataset helps the CapsNet achieve better accuracy on the GTI dataset. The right of FIG3 is an example trained on MNIST with 20% shifting. Nearly all images being predicted correctly except those close to the edge. It's interesting that the generated images look like handwriting even the input images are Helvetica font. CNN's performance on GTI dataset generally better than the CapsNet as shown in Figure 5. The accuracy on CNN is always better than the CapsNet with different amount of shifting, even the convolutional layers in CapsNet using wider receptive fields. Since we remove the max-pooling layers in the CapsNet and the convolutional layers in the CapsNet is'place-coded' equivalent, which could be the reason that the CapsNet has lower performance on the GTI dataset. We believe there is much room to improve for the CapsNet to handling translational invariance. We introduce a simple GTI testing dataset for deep learning models trained on MNIST dataset. The goal is to get a better understanding of the ability of CNN and CapsNet to dealing with global translational invariance. Although the current version of CapsNet could not handle global translational invariance without data augmentation, we still believe CapsNet architecture potentially better than CNN on dealing with global translational invariance because capsules could train to learn all viewpoint no matter it receives the information for the centre or the edge. Our testing method is sample Figure 5: GTI dataset accuracy of models trained on CNN and CapsNet with different amount of random shifting in MNIST training dataset.and quantifiable, and it easy to implement for other datasets of computer vision tasks by taking a clear and correct labelled image from each class and apply the translational shifting to cover all possible cases.

Testing of global translational invariance in Convolutional and Capsule Networks

Represent the short summary for finding the corresponding scientific article (long)

A method to binarize both weights and activations of a deep neural network that is efficient in computation and memory usage and performs better than the state-of-the-art.


Question: represent the scientific article to find a short tldr

Question: Gaussian processes are ubiquitous in nature and engineering. A case in point is a class of neural networks in the infinite-width limit, whose priors correspond to Gaussian processes. Here we perturbatively extend this correspondence to finite-width neural networks, yielding non-Gaussian processes as priors. The methodology developed herein allows us to track the flow of preactivation distributions by progressively integrating out random variables from lower to higher layers, reminiscent of renormalization-group flow. We further develop a perturbative prescription to perform Bayesian inference with weakly non-Gaussian priors. Gaussian processes model many phenomena in the physical world. A prime example is Brownian motion (Brown, 1828), modeled as the integral of Gaussian-distributed bumps exerted on a pointlike solute . The theory of elementary particles also becomes a Gaussian process in the free limit where interactions between particles are turned off, and manybody systems as complex as glasses come to be Gaussian in the infinite-dimensional, mean-field, limit . In the context of machine learning, pointed out that a class of neural networks give rise to Gaussian processes in the infinite-width limit, which can perform exact Bayesian inference from training to test data . They occupy a corner of theoretical playground wherein the karakuri of neural networks is scrutinized (; ; ; ;). In reality, Gaussian processes are but mere idealizations. Brownian particles have finite-size structure, elementary particles interact, and many-body systems respond nonlinearly. In order to understand rich phenomena exhibited by these real systems, Gaussian processes rather serve as starting points to be perturbed around. Indeed many edifices in theoretical physics are build upon the successful treatment of non-Gaussianity, with a notable example being renormalization-group flow (; ; ;). In the quest to elucidate behaviors of real neural networks away from the infinite-width limit, it is thus natural to wonder if the similar treatment of non-Gaussianity yields equally elegant and powerful machinery. Here we set out on this program, perturbatively treating finite-width corrections to neural networks. Prior distributions of outputs are obtained through progressively integrating out preactivation of neurons layer by layer, yielding non-Gaussian priors. Intriguingly, intermediate recursion relations and their derivation resemble renormalization-group flow . Such a recursive approach further enables us to treat finite-width corrections on Bayesian inference and their regularization effects, with arbitrary activation functions. The rest of the paper is structured as follows. In Section 2 we review and set up basic concepts. Our master recursive formulae (R1,R2,R3) are derived in Section 3, which control the flow of preactivation distributions from lower to higher layers. After an interlude with concrete examples in Section 4, we extend the Gaussian-process Bayesian inference method to non-Gaussian priors in Section 5 and use the ing scheme to study inference of neural networks at finite widths. We conclude in Section 6 with dreams. In this paper we study real finite-width neural networks in the regime where the number of neurons in hidden layers is asymptotically large whereas input and output dimensions are kept constant. Let us focus on a class of neural networks termed multilayer perceptrons, with model parameters, θ = b i, W i,j, and an activation function, σ. For each input, x ∈ R n0, a neural network outputs a vector, z(x; θ) = z (L) ∈ R n L, recursively defined as sequences of preactivations through i,j x j for i = 1,..., n 1, , we assume priors for biases and weights given by independent and identically distributed Gaussian distributions with zero means, E b i,j = 0, and variances Higher moments are then obtained by Wick's contractions . For instance, For those unfamiliar with Wick's contractions and connected correlation functions (a.k.a. cumulants), a pedagogical review is provided in Appendix A as our formalism heavily relies on them. In the infinite-width limit where n 1, n 2,..., n L−1 → ∞ (but finite n 0 and n L), it has been argued -with varying degrees of rigor (; ;) -that the prior distribution of outputs is governed by the Gaussian process with a kernel and all the higher moments given by Wick's contractions. In particular, there exists a recursive formula that lets us evaluate this kernel for any pair of inputs [c.f. Equation (R1)]. Importantly, once the values of the kernel are evaluated for all the pairs of..,ND, consisting of N R training inputs with target outputs and N E test inputs with unknown targets, we can perform exact Bayesian inference to yield mean outputs as predictions for N E test data [c.f. Equation (GPM)]. This should be contrasted with stochastic gradient descent (SGD) optimization , through which typically a single estimate for the optimal model parameters of the posterior, θ, is obtained and used to predict outputs for test inputs; Bayesian inference instead marginalizes over all model parameters, performing an ensemble average over the posterior distribution . We shall now study real finite-width neural networks in the regime n 1,..., n L−1 ∼ n 1. 1 At finite widths, there are corrections to Gaussian-process priors. In other words, a whole tower of 1 Note that input and output dimensions, n0 and nL, are arbitrary. To be precise, defining n1,..., nL−1 ≡ µ1n,..., µL−1n, we send n 1 while keeping C , µ1,..., µL−1, n0, and nL constants, and compute the leading 1/n corrections. In particular it is crucial to keep the number of outputs nL constant in order to consistently perform Bayesian inference within our approach. nontrivial preactivation correlation functions beyond the kernel, collectively dictate the distribution of preactivations. Our aim is to trace the flow of these distributions progressively and cumulatively all the way up to the last layer whereat Bayesian inference is executed. More specifically, we shall inductively and self-consistently show that two-point preactivation correlation functions take the form and connected four-point preactivation correlation functions is symmetric under α 1 ↔ α 2, α 3 ↔ α 4, and (α 1, α 2) ↔ (α 3, α 4). At the first layer the preactivation distribution is exactly Gaussian for any finite widths and hence Equations (KS) and (V) are trivially satisfied, with = 0, and Obtained in Section 3 are the recursive formulae that link these core kernel, self-energy, and fourpoint vertex at the -th layer to those at the (+ 1)-th layer while in Section 5 these tensors at the last layer = L are used to yield the leading 1/n correction for Bayesian inference at finite widths. Our Schwinger operator approach is orthogonal to the replica approach by and, unlike the planar diagrammatic approach by , applies to general activation functions, made possible by accumulating corrections layer by layer rather than dealing with them all at once. More substantially, in contrast to these previous approaches, we here study finite-width effects on Bayesian inference and find that the renormalization-group picture naturally emerges. As auxiliary objects in recursive steps, let us introduce activation correlation functions Our basic strategy is to establish relations zigzagging between sets of preactivation correlation functions and sets of activation correlation functions, keeping track of leading finite-width corrections. Below, relations G → H are obtained by integrating out preactivations while relations H → G (+1) are obtained by integrating out biases and weights. At first glance the algebra in this paper may look horrifying but repeated applications of Wick's contractions are all there is to it. The are summarized in Section 3.2. 2 In the main text we place tildes on objects that depend only on sample indices α's in order to distinguish them from those that depend both on sample indices α's and neuron indices i's. 3 Given that the means of biases and weights are zero, G The remaining task is to relate preactivation correlations G to activation correlations H within the same layer, which will complete the zigzag relation (ZIGZAG) for these correlation functions. 4 Once the sorcery of Wick's contractions and connected correlation functions is mastered, it is simple to derive the following combinatorial hack (Appendix A.4): viewing prior preactivations as a random (n N D)-dimensional vector and defining the Gaussian integral with the kernel z, the prior average capture 1/n corrections due to selfenergy and four-point vertex, respectively, and are defined as where the sample indices are lowered by using the inverse core kernel as a metric, meaning Using the above hack, we can evaluate the activation correlations by straightforward algebra with -you guessed it -Wick's contractions. As the Gaussian integral is diagonal in the neuron index i, we just need to disentangle cases with repeated and unrepeated neuron indices. The solution for this exercise is in Appendix B: this is the most cumbersome algebra in the paper and the ability to perform it certifies the graduation from the magical school of Wick's crafts and wizardly cumulants. 4 The nontrivial parts of the inductive proof for Equations (KS) and (V) are to show (i) that the right-hand side of Equation is finite as n → ∞, (ii) that the leading contribution of Equation is the Gaussianprocess kernel, and (iii) that higher-point connected preactivation correlation functions are all suppressed by O 1 n 2, all of which are verified in obtaining the recursive equations. See Appendix B for a full proof. Denoting the Gaussian integral with the core kernel z..,ND, and plugging in of Appendix B into Equations and, we arrive at our master recursion relations, and For = 1, a special note about the ratio n n −1 is in order: even though n 0 stays constant while n 1 1, the terms proportional to that ratio are identically zero due to the complete Gaussianity (R0). The preactivation distribution in the first layer (R0) sets the initial condition for the flow from lower to higher layers dictated by these recursive equations. Once recursed up to the last layer = L, the ing distribution of outputs z = z (L) can be succinctly encoded by the probability distribution with the potential, and (D1) By now, the reader should be deriving this through Wick's contractions without solicitation. It is important to note that n L is constant and thus H 1 [z] can consistently be treated perturbatively. 5 If nL were of order n 1, the potential H would become a large-n vector model, for which we would have to sum the infinite series of bubble diagrams . The recursive relations obtained above can be evaluated numerically [or sometimes analytically for ReLU ], which is a perfectly adequate approach: at the leading order it involves four-dimensional Gaussian integrals at most. Here, continuing the theme of wearing out Wick's contractions, we develop an alternative analytic method that works for any polynomial activations , providing another perfectly cromulent approach. For a general polynomial activation of degree p, σ(z) = p k=0 a k z k, the nontrivial term in Equation (R1) can be expanded as Each term can then be evaluated by Wick's contractions and the same goes for all the terms in Equations (R2) and (R3). Below and in Appendix C, we illustrate this procedure with simple examples. When the activation function is linear, σ(z) = z, multilayer perceptrons go under the aweinspiring name of deep linear networks . Setting C = 0 and C W = 1 for simplicity, our recursion relations reduce to K α1,α2, and S α1,α2 Solving them yields the layer-independent core kernel and zero self-energy and the linearly layer-dependent four-point vertex. It succinctly reproduces the that can be obtained through planar diagrams in this special setup . Quadratic activation is worked out in Appendix C.1. The recursion relations simplify drastically for the case of a single input, N D = 1, as worked out in detail in Appendix C.2. For instance, for rectified linear unit (ReLU) activation with C b = 0 and C W = 2, we obtain the layer-independent core kernel, zero self-energy, and the four-point vertex 2. Interestingly, as for deep linear networks, the factor (1/n) appears again. This factor has also been found by , which provides guidance for network architectural design through its minimization. We generalize this factor for monomial activations in Appendix C.2.1 Here we put our theory to the test. For concreteness, take a single black-white image of handwritten digits with 28-by-28 pixels (i.e. n 0 = 784) from the MNIST dataset without preprocessing, set depth L = 3, bias variance C b = 0, weight variance C W = C W, and widths (n 0, n 1, n 2, n 3) = (784, n, 2n, 1), and use activations σ(z) = z (linear) with C W = 1 and max(0, z) (ReLU) with C W = 2. In Figure 1, for each width-parameter n of the hidden layers we record the prior distribution of outputs over 10 6 instances of Gaussian weights and compare it with the theoretical prediction -obtained by cranking the knob from the initial condition (R0) through the recursion relations (R1-R3) to the distribution (D0-D2). The prior distribution becomes increasingly non-Gaussian as networks narrow and the deviation from the Gaussian-process prior is correctly captured by our theory. Higher-order perturbative calculations are expected to systematically improve the quality -and extend the range -of the agreement. Additional experiments are performed in Appendix C.3, which further corroborates our theory. Figure 1: Comparison between theory and experiments for prior distributions of outputs for a single input. The agreement between our theoretical predictions (smooth thick lines) and experimental data (rugged thin lines) is superb, correctly capturing the initial deviations from Gaussian processes at n = ∞ (black), all the way down to n ∼ 10 for linear activation and to n ∼ 30 for ReLU activation. Let us take off from the terminal point of Section 3: we have obtained the recursive equations (R0-R3) for the Gaussian-process kernel and the leading finite-width corrections and codified them in the weakly non-Gaussian prior distributions with ≡..,n L. We shall develop a formalism to infer outputs for test inputs a lá Bayes, perturbatively extending the textbook by. For field theorists, our calculation is just a tree-level field calculation in disguise. Taking the liberty of notations, we let the number of input-data arguments dictate the summation over sample indices α inside the potential H, and denote the joint probabilities Given the training targets y R, the posterior distribution of test outputs are given by Bayes' rule: The leading Gaussian-process contributions can be segregated out through the textbook manipulation [c.f. Appendix D.1]: denoting the full Gaussian-process kernel in the last layer as and the Gaussian-process posterior mean prediction as and defining a fluctuation For any function F, its expectation over the Bayesian posterior (Bayes) then turns into where the deviation kernel (δz E) and the normalization factor In particular the mean posterior output is given by ], recalling Equation (D2) for H 1, and using Wick's contractions for one last time, the mean prediction becomes With additional manipulations in Appendix D, this expression is simplified into the actionable form that is amenable to use in practice. For illustration, there, we also show a simple preliminary experiment, which indicates the 1/n regularization effect for sufficiently large width n and small amount of training data N R. This is in line with expectations that finite widths ameliorate overfitting and that non-Gaussian priors increase the expressivity of neural functions, but additional large-scale extensive experiments would be desirable in the future. In this paper, we have developed the perturbative formalism that captures the flow of preactivation distributions from lower to higher layers. The spiritual resemblance between our recursive equations and renormalization-group flow equations in high-energy and statistical physics is highly appealing. It would be exciting to investigate the structure of fixed points away from the Gaussian asymptopia and fully realize the dream articulated by beyond their limited example of a mapping between two antiquated techniques -the audacious hypothesis that neural networks wash away microscopic irrelevancies and extract relevant features. In addition we have developed the perturbative Bayesian inference scheme universally applicable whenever prior distributions are weakly non-Gaussian, and have applied it to the specific cases of neural networks at finite widths. In light of finite-width regularization effects, it would be prudent to revisit the empirical comparison between SGD optimization and Bayesian inference at finite widths , especially for convolutional neural networks. Finally, given surging interests in SGD dynamics within the large-width regime (; ; ;), it would be natural to adapt our formalism for investigating corrections to neural tangent kernels, and even nonperturbatively aspire to capture a phase transition out of lazy-learning into feature-learning regimes. Welcome to the magical school of Wick's crafts and wizardly cumulants. Here is all you need to know in order to follow the calculations in the paper. In the main text, Wick's contractions are used trivially for integrating out biases and weights as straightforward applications of Appendix A.1 while they are used more nontrivially for integrating out preactivations, with concepts of cumulants reviewed in Appendix A.2 and A.3, culminating in a hack derived in Appendix A.4. For most parts, we shall forget about the neuron index i for pedagogy and put them back in at the very end. For Gaussian-distributed variables z = {z α} α=1,...,N with a kernel K α,α, moments (S1) For any odd m such moments identically vanish. For even m, Isserlis-Wick's theorem states that where the sum is over all the possible pairings of m variables, For a proof, see for example. In order to understand and use the theorem, it is instructive to look at a few examples: and Given general (not necessarily Gaussian) random variables, connected correlation functions are defined inductively through where the sum is over all the possible subdivisions of m variables into s > 1 clusters of sizes (ν 1, . . ., ν s) as (k νs). In order to understand the definition, it is again instructive to look at a few examples. Assuming that all the odd moments vanish, Rearranging them in particular yields If these examples do not suffice, here is yet another example to chew on: and hence We emphasize that these are just renderings of the definition (S6). The power of this definition will be illustrated in the next two subsections. We often encounter situations with the hierarchy where 1 is a small perturbative parameter and here again odd moments are assumed to vanish. Often comes with the hierarchical structure is the asymptotic limit → 0 where with the Gaussian kernel K α1,α2 at zero and the leading self-energy correction S α1,α2. Let us also denote the leading four-point vertex For instance this hierarchy holds for weakly-coupled field theories -from which we are importing names such as self-energy, vertex, and metric -and, in this paper, such hierarchical structure is inductively shown to hold for prior preactivations z with = 1 n −1 in the regime n 1,..., n L−1 ∼ n 1. Note that, by definition, K α1,α2 and S α1,α2 are symmetric under α 1 ↔ α 2 and V α1,α2,α3,α4 is symmetric under permutations of (α 1, α 2, α 3, α 4). With the review of connected correlation functions passed us, we can now readily see that where in the last equality Wick's theorem was used backward. So far we have reviewed the standard technology of Wick's contractions, connected correlation functions, and all that. Here is one sorcery, which lets us magically evaluate E [z α1 · · · z αm] by mindless repetitions of Wick's contractions. Throughout, we shall assume the hierarchical structure (S12), 7 which is the inductive assumption in the main text, and start from its consequence (CLUSTER). Below, let us use the inverse kernel K −1 α1,α2 as a metric to lower indices: First note that 7 More precisely, we shall only use the weaker assumption that for m ≥ 6 along with Equations (S13) and (S14). where the symmetry α 1 ↔ α 2 of S α1,α2 was used. Hence, defining we obtain, for one term in Equation (CLUSTER), The similar algebraic exercise renders the other term in Equation (CLUSTER) to be In summary, for any function In order to get the expressions used in the main text at the -th layer, we need only to put back neuron indices i by replacing α → (α, i), identify = 1 n −1, and use the inductive assumptions (KS) The operators in Equations (OS') and (OV') then become i.e., the operators in Equations (OS) and (OV) in the main text. In this Appendix, we provide a full inductive proof for one of the main claims in the paper, streamlined in the main text. Namely, we assume at the -th layer that Equations (KS) and (V) hold and that all the higher-point connected preactivation correlation functions are of order O 1 n 2 -which are trivially true at = 1 -and prove the same for the (+ 1)-th layer. We assume the full mastery of Appendix A or, conversely, this section can be used to test the mastery of wicked tricks. First, trivial Wick's contractions yield Studiously disentangling cases with different numbers of repetitions in neuron indices (j 1, . . ., j k), we notice that at order O 1 n, terms without repetition or with only one repetition contribute, finding where we used the inductive hierarchical assumption at the -th layer, i.e., its consequence (HACK) and denoted a single-neuron random vectorz = {z α} α=1,...,ND and the Gaussian integral with the core kernel z As special cases, we obtain expressions advertised in the main text to be contained in this Appendix: Assembling everything, In particular, completing our inductive proof. Note that B (α1,α2),(α3,α4) = O 1 n. Nowhere in our derivation had we assumed anything about the form of activation functions. The only potential exceptions to our formalism are exponentially growing activation functions -which we have never seen in practice -that would make the Gaussian integrals unintegrable. Let us take multilayer perceptrons with quadratic activation, σ(z) = z 2, and study the distributions of preactivations in the second layer as another illustration of our technology. From the master recursion relations (R1-R3) with the initial condition (R0), Wickology yields, and =0. where. These expressions are used in Appendix D.2 for the experimental study of finite-width corrections on Bayesian inference. The recursive relations simplify drastically for the case of a single input, N D = 1. Setting C b = 0 for simplicity and dropping α index, our recursive equations reduce to, and (S34) Under review as a conference paper at ICLR 2020 For monomial activations, σ(z) = z p, such as in deep linear networks and quadratic activations ,, and (S37) In particular the four-point vertex solution is given by The factor 1 n p 2 generalizes the factor 1 n for linear and ReLU activations. , this factor guides us to narrow hidden layers as we pass through nonlinear activations. ReLU activation, σ(z) = max(0, z), can also be worked out for a single input through Wick's contractions, noting that the Gaussian integral is halved, yielding, and (S41) Setting C W = 2 for simplicity, these equations can be solved, leading to, and (S44) Here is an extended version of experiments in Section 4.3. As in the main text, take a single blackwhite image of hand-written digits from the MNIST dataset as an n 0 = 784-dimensional input, without preprocessing. Set bias variance C and max(0, z) (ReLU) with C W = 2. For all three cases, we consider both depth L = 2 with widths (n 0, n 1, n 2) = (784, n, 1) and depth L = 3 with widths (n 0, n 1, n 2, n 3) = (784, n, 2n, 1). As in Figure 1, in Figure S1, for each width-parameter n of the hidden layers we record the prior distribution of outputs over 10 6 instances of Gaussian weights and compare it with the theoretical prediction. Results again corroborate our theory. Figure S1: Comparison between theory and experiments for prior distributions of outputs for a single input. Our theoretical predictions (smooth thick lines) and experimental data (rugged thin lines) agree, correctly capturing the initial deviations from the Gaussian processes (black, n = ∞), at least down to n = n with n ∼ 10 for linear cases, n ∼ 30 for ReLU cases and depth L = 2 quadratic case, and n ∼ 100 for depth L = 3 quadratic case. This also illustrates that nonlinear activations quickly amplify non-Gaussianity. This expression simplifies drastically through the identity which can be checked explicitly, recalling RR K RE. Incidentally, this identity can also be used to prove Equation (GP∆). Now equipped with this identity, recalling φ Finally, denoting the matrix inside the parenthesis to be are actionable, i.e., easy to program. It turns out that for deep linear networks the leading finite-width correction given above vanishes, and the first correction is likely to show up at higher order in 1/n asymptotic expansion, which is not carried out in this paper. Here we instead use the L = 2 multilayer perceptron with the quadratic activation for illustration, plugging Equations (S30,S31,S32) into Equations (NGPM') and (NGPM"). Set C Figure S2 indicate the regularization effects of finite widths, at least when the number of training samples, N R, is small, ing in peak performance at finite widths. Figure S2: Test accuracy for N E = 10000 MNIST test data as a function of the inverse width = 1/n L−1 of the hidden layer with quadratic activation. For each number N R of subsampled training data, the is averaged over 10 distinct choices of such subsamplings. For small numbers of training data, finite widths in regularization effects, improving the test accuracy.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

We develop an analytical method to study Bayesian inference of finite-width neural networks and find that the renormalization-group flow picture naturally emerges.

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Distillation is a method to transfer knowledge from one model to another and often achieves higher accuracy with the same capacity. In this paper, we aim to provide a theoretical understanding on what mainly helps with the distillation. Our answer is "early stopping". Assuming that the teacher network is overparameterized, we argue that the teacher network is essentially harvesting dark knowledge from the data via early stopping. This can be justified by a new concept, Anisotropic In- formation Retrieval (AIR), which means that the neural network tends to fit the informative information first and the non-informative information (including noise) later. Motivated by the recent development on theoretically analyzing overparame- terized neural networks, we can characterize AIR by the eigenspace of the Neural Tangent Kernel(NTK). AIR facilities a new understanding of distillation. With that, we further utilize distillation to refine noisy labels. We propose a self-distillation al- gorithm to sequentially distill knowledge from the network in the previous training epoch to avoid memorizing the wrong labels. We also demonstrate, both theoret- ically and empirically, that self-distillation can benefit from more than just early stopping. Theoretically, we prove convergence of the proposed algorithm to the ground truth labels for randomly initialized overparameterized neural networks in terms of l2 distance, while the previous was on convergence in 0-1 loss. The theoretical ensures the learned neural network enjoy a margin on the training data which leads to better generalization. Empirically, we achieve better testing accuracy and entirely avoid early stopping which makes the algorithm more user-friendly. Deep learning achieves state-of-the-art in many tasks in computer vision and natural language processing. Among these tasks, image classification is considered as one of the fundamental tasks since classification networks are commonly used as base networks for other problems. In order to achieve higher accuracy using a network with similar complexity as the base network, distillation has been proposed, which aims to utilize the prediction of one (teacher) network to guide the training of another (student) network. , the authors suggested to generate a soft target by a heavy-duty teacher network to guide the training of a light-weighted student network. More interestingly,; proposed to train a student network parameterized identically as the teacher network. Surprisingly, the student network significantly outperforms the teacher network. Later, it was suggested by Zagoruyko & Komodakis (2016a);; to transfer knowledge of representations, such as attention maps and gradients of the classifier, to help with the training of the student network. In this work, we focus on the distillation utilizing the network outputs;; Yang et al. (2018a);; Yang et al. (2018b). To explain the effectiveness of distillation, suggested that instead of the hard labels (i.e one-hot vectors), the soft labels generated by the pre-trained teacher network provide extra information, which is called the "Dark Knowledge". The "Dark knowledge" is the knowledge encoded by the relative probabilities of the incorrect outputs.;; Yang et al. (2018a), the authors pointed out that secondary information, i.e the semantic similarity between different classes, is part of the "Dark Knowledge", and observed that the "Dark Knowledge" can help to refine noisy labels. In this paper, we would like to answer the following question: can we theoretically explain how neural networks learn the Dark Knowledge? Answering this question will help us to understand the regularization effect of distillation. In this work, we assume that the teacher network is overparameterized, which means that it can memorize all the labels via gradient descent training Du et al. (2018b; a);;. In this case, if we train the overparameterized teacher network until convergence, the network's output coincides exactly with the ground truth hard labels. This is because the logits corresponding to the incorrect classes are all zero, and hence no "Dark knowledge" can be extracted. Thus, we claim that the core factor that enables an overparameterized network to learn "Dark knowledge" is early stopping. What's more,;; observed that "Dark knowledge" represents the discrepancy of convergence speed of different types of information during the training of the neural network. Neural network tends to fit informative information, such as simple pattern, faster than non-informative and unwanted information such as noise. Similar phenomenon was observed in the inverse scale space theory for image restoration;;;. In our paper, we call this effect Anisotropic Information Retrieval (AIR). With the aforementioned interpretation of distillation, We further utilize AIR to refine noisy labels by introducing a new self-distillation algorithm. To extract anisotropic information, we sequentially extract knowledge from the output of the network in the previous epoch to supervise the training in the next epoch. By dynamically adjusting the strength of the supervision, we can theoretically prove that the proposed self-distillation algorithm can recover the correct labels, and empirically the algorithm achieves the state-of-the-art on Fashion MNIST and CIFAR10. The benefit brought by our theoretical study is twofold. Firstly, the existing approach using large networks; often requires a validation set to early terminate the network training. However, our analysis shows that our algorithm can sustain long training without overfitting the noise which makes the proposed algorithm more user-friendly. Secondly, our analysis is based on an 2 -loss of the clean labels which enables the algorithm to generate a trained network with a bigger margin and hence generalize better. We summarize our contributions as follows • This paper aims to understand distillation theoretically (i.e. understand the regularization effect of distillation). Distillation works due to the soft targets generated by the teacher network. Based on the observation that the overparameterized network can exactly fit the one-hot labels which contain no dark knowledge, we theoretically justify that early stopping is essential for an overparameterized teacher network to extract dark knowledge from the hard labels. This provides a new understanding of the regularization effect of distillation. • This is the first attempt to theoretically understand the role of distillation in noisy label refinery using overparameterized neural networks. Inspired by, we utilize distillation to propose a self-distillation algorithm to train a neural network under label corruption. The algorithm is theoretically guaranteed to recover the unknown correct labels in terms of the 2 -loss rather than the previous 0-1-loss. This enables the algorithm to generate a trained network whose output has a bigger margin and hence generalizes better. Furthermore, our algorithm does not need a validation set to early stop during training, which makes it more hyperparameter friendly. The theoretical understanding of the overparameterized networks encourages us to use large models which empirically produce better . 2.1 NO EARLY STOPPING, NO DARK KNOWLEDGE As mentioned in the introduction, an overparameterized teacher network is able to extract dark knowledge from the on-hot hard labels because of early stopping. In this section we present an experiment to verify this effect of early stopping, where we use a big model as a teacher to teach a smaller model as the student. In this experiment, we train a WRN-28 Zagoruyko & Komodakis (2016b) Figure 1: A good teacher may not be able to produce a good student. To maximize the effectiveness of distillation, you should early stop your epoch at a proper time. As we can see, the teacher model does not suffer from overfitting during the training, while it does not always educate a good student model either. As suggested by Yang et al. (2018a), a more tolerant teacher educates better students. The teacher model trained with 80 epochs produces the best which indicates that early stopping of the bigger model can extract more informative information for distillation. In this paper, we introduce a new concept called Anisotropic Information Retrieval (AIR), which means to exploit the discrepancy of the convergence speed of different types of information during the training of an overparameterized neural network. An important observation of AIR is that informative information tends to converge faster than non-informative and unwanted information such as noise. Selective bias of an iterative algorithm to approximate a function has long been discovered in different areas. For example, observed that iterative linear equation solver fits the low frequency component first, and they proposed a multigrid algorithm to exploit this property. In image processing,;;; proposed inverse scale space methods that recover image features earlier in the iteration and noise comes back later. In kernel learning, early stopping of gradient descent is equivalent to the ridge regression;; , which means that bias from the eigenspace corresponding to the larger eigenvalues is reduced quicker by the gradient descent. Under the neural network setting,; observed that neural networks find low frequency patterns more easily. discovered that noisy labels can slow down the training.; studied the memorization effects of the deep networks, and revealed that, during training, neural networks first memorize the data with clean labels and later with the wrong labels. In the next subsection, we will characterize AIR of overparametrized neural networks using the. , the authors introduced the Neural Tangent Kernel to characterize the trajectory of gradient descent algorithm learning infinitely wide neural networks. Denote f (θ, x) ∈ R as the output of neural network with θ ∈ R n being the trainable parameter and x ∈ R p the input. Consider training the neural network using the 2 -loss on the dataset {( We use the gradient descent θ t+1 = θ t − η∇l(θ t) to train the neural network. Let u t = (f (θ t, x i)) i∈[n] ∈ R n be the network outputs on all the data {x i} at iteration t and y = (y i) i∈ [n]. It was shown by Du et al. (2018b);; that the evolution of the error u t − y can be formulated in a quasi-linear form, It is known thatĤ t − H t = o with respect to d, where. Note that H * is a symmetric positive semi-definite matrix. Assume that λ 1 > · · · > λ n ≥ 0 are its n eigenvalues and e 1, e 2, · · ·, e n are the corresponding eigenvectors. The eigenvectors are orthogonal < e i, e j >= 0 and. Consider the evolution of the projection of the loss function in different eigenspaces We can see that the component lies in the eigenspace with a larger eigenvalue converges faster. Therefore, AIR describes the phenomenon that the gradient descent algorithm searches for information components corresponding to different eigenspaces at different rates.;;; has shown that that one possible reason of neural network's good generalization property is that neural network fits useful information faster. Thus in our paper, we regard informative information as the eigenspaces associated with the largest few eigenvalues of NTK. Figure 2: Components of label noise in the largest five eigensapces of NTK are decreasing. We denote the projection of supervision signal to the eigenspace with a larger eigenvalue as useful information. In Figure 2, We calculate the ratio of the172norm of the label vector provided by the self-distillation algorithm lies in the top-5 eigenspace as a representative of informative information. We can see that the informative information decreases when the noise level increases. This motivates us to further explore how "Dark Knowledge" helps with label refinery. Supervised learning requires high quality labels. However, due to noisy crowd-sourcing platforms and data augmentation pipeline , it is hard to acquire entirely clean labels for training. On the other hand, successive deep models often have huge capacities with millions or even billions of parameters. Such huge capacity enables the network to memorize all the labels, right or wrong , which makes learning deep neural networks with noisy labels a challenging task.;; pointed out that neural networks often fit the clean labels before the noisy ones during training. theoretically showed that early stopping can clean up label noise with overparametrized neural networks. This, together with our understanding of distillation with AIR, inspired us to use distillation for noisy label refinery. We shall introduce a new self-distillation algorithm with theoretically guaranteed recovery of the clean labels under suitable assumptions. Training deep models on datasets with label corruption is an important and challenging problem that has attracted much attention lately. , the authors proposed to regularize the Local Intrinsic Dimensionality of deep representations to detect label noise. proposed a joint optimization framework to simultaneously optimize the network parameters and output labels. In the literature of distillation, first utilized distillation to refine noisy labels of ImageNet, while Yang et al. (2018b) introduced a distillation method to complete teacherstudent training in one generation. The latter is most related to our proposed self-distillation algorithm. However, the difference is that their model aims to ensemble diverse models in one training trajectory but ours aims to utilize AIR to refine noisy labels during the training. It is known that the label noise lies in the eigenspaces associated to small eigenvalues;. used early stopping to remove label noise. However, early stopping is hard to tune and sometimes leads to unsatisfactory . In this section, we proposed a self-distillation algorithm with an excellent empirical performance and a theoretical guarantee to recover the correct labels under certain conditions but without the requirement of early stopping. From the perspective of AIR, we observe that the knowledge learned in early epochs is informative information (i.e. the eigenspaces associated with the largest few eigenvalues of NTK) and can be used to refine the training for later epochs. In other words, the algorithm distills knowledge sequentially to guide the training of the model in later epochs by the knowledge distilled by the model from earlier epochs. The informative information learned during early epochs is, in some sense, "low frequency information", which is the core factor to enable the model to generalize well. A nice property of the self-distillation algorithm is that it generates the final model in one generation (i.e. single-round training), which has almost no additional computational cost compared to normal training. The proposed self-distillation algorithm is given by Algorithm 1. Here, the function h(·) in the algorithm is the label function. It can either be a hard label function such as hardmax, or a soft label function such as softmax with a certain temperature. The choice of h(·) and interpolation coefficient α t depends on the usage of the self-distillation algorithm. If Randomly initialize the network. we want to clean up label noise, we normally choose h(·) to be hardmax or softmax with a low temperature. The weight α t is chosen to be adaptively decreasing corresponding to the increase of our confidence on the learned model at current epoch. The introduction of h(·) helps to boost AIR and the information gained from the previous epoch. In this section, we provide a theoretical justification of the performance of the self-distillation algorithm with overparameterized neural networks. Here, we only consider binary classification task with label ∈ {−1, +1}. • We consider a dataset with n data:. (x i, y i) are the input data and its associated label seen by the model whileỹ i is the unobserved ground truth label. The pair (x i, y i) with y i =ỹ i is called a clean data, otherwise it is called a corrupted data. • We assume that {x i} i∈ [n] contains points with unit Euclidean norm and has K clusters. The data is randomly i.i.d sampled from a distribution P. Let n l be the number of points in the lth cluster. In our analysis, we ssume that number of data in each cluster is balanced in the sense that n l ≥ c low n K for constant c low > 0 and all of the data is bounded, i.e. • For each of the K clusters, we assume that all the input data lie within the Euclidean ball B(c l,), where c l is the center with unit Euclidean norm and > 0 is the radius. • Assume that the data in the same cluster has the same ground truth labelỹ. For the lth cluster, we denote ρ l the proportion of the data with wrong labels. Let ρ = max{ρ i : i ∈ [K]} and assume that ρ < 1 2. • A dataset satisfying the above assumptions is called an (, ρ) dataset. The above definition of dataset follows that of the previous work;. It is reasonable to assume that ρ < 1 2 in order to ensure the correct labels dominate each cluster. In this work, we consider two-layers neural networks. For input data x ∈ R d, the output of the neural network f is: where W ∈ R k×d is the weight matrix and φ is the activation function applied to W x entry-wise. We suppose k is even and fix the output layer by assigning half of the entries T, we simply denote the output vector on the data matrix as Following the previous work on training overparameterized neural network Du et al. (2018b), we consider the MSE loss Definition 2. For a data matrix D ∈ m×d, we denote λ(D) the small eigenvalue of the neural network covariance matrix The above definition reveals the matching score of the model and data. We denote C = [c 1, c 2, . . ., c K] T the matrix composed by the center of the cluster. We denote Λ = min(λ(C), λ(X)) for simplification. We are now ready to present our main theorem that establishes the convergence of the proposed self-distillation algorithm to the ground truth labels under certain conditions. Theorem 1. Assume that |φ|, |φ (·)| and |φ (·)| are bounded with upper bound Γ ≥ 1. We fix a learning rate η = 1 2Γ 2 n for the gradient descent. Assume that the sequence α t monotonically decreases to 0. Furthermore, we have two slow-decay conditions on α t where ) and T 2 = inf{t : For the self-distillation algorithm, if the following two conditions for the radius and the width k are satisfied then for random initialization W 0 ∼ N k×d, with probability 1 − δ, we have: where W t is the parameter generated by the full-batch self-distillation algorithm at iteration t. Theorem 2. Combining Theorem 1 and , with failure probability δ ∈, using the self-distillation algorithm and neural network described in Theorem 1 and under the same condition, we have Compared to previous on noisy label , our made the following improvements. Firstly, our algorithm fits the ground truth labels without the help of early stopping as long as a mild condition on α t is satisfied. Secondly, while previous work only ensures the algorithm to yield correct class labels on training set, our state the 2 convergence of the outputs to the ground truth labels. As a , the solution that our algorithm finds tends to have larger margin which leads to better generalization. This is shown in Theorem 2. and will be supported by our empirical studies in the next subsection. 3.4 NOISY LABEL REFINERY Figure 4: Training on CIFAR10 with 40% noise injection. The normal training suffers from over-fitting, while selfdistillation does not. (Note that we are conducting cosine learning rate scheduling. The learning rate is extremely small at the end of learning.) In this section, we conduct experiments on the self-distillation algorithm. In the experiments, we applied our algorithm on corrupted Fashion MNIST and CI-FAR10. At noise level p, every data in the original training dataset is chosen and assigned a symmetric noisy label with probability p. We test our algorithm for p = 0.2, 0.4, 0.6 and 0.8. The test accuracy is calculated with respect to the ground truth labels. We adopted the shake-shake network of with 32 channels and cross entropy loss. We trained the network by momentum stochastic gradient descent (SGD) with batch size 128, momentum 0.9 and a weight decay of 1e-4. We schedule the learning rate following cosine learning rate with a maximum learning rate 0.2 and minimum learning rate 0. In order to ensure convergence, we trained the models for 600 epochs. , mean subtraction, horizontal random flip, 32 × 32 random crops after padding with 4 pixels on each side from the padded image is performed as the data augmentation process. For testing, We only do evaluation on the original 32 × 32 image. In self-distillation, we adaptively adjust α t by setting 1 − α t = λ * accuracy, where accuracy is the accuracy calculated on the current batch. The direct ratio λ is the only tuning parameter. For CIFAR10, We simply set λ as 1 when the noisy level is low, such as p = 0, 0.2 and 0.4. When the noisy level p = 0.6 and 0.8, we take λ = 1.5. For Fashion MNIST, We simply set λ as 0.6, 1, 1, 1.4, 1.6 respectively when p = 0, 0.2, 0.4, 0.6, 0.8. We report the average final test accuracy of 3 runs for Fashion MNIST and CIFAR10 in Figure 5: Self-distillation always gains information. Distillation can benefit from more than just early stopping. Distillation has the ability to enhance the AIR and thus can extract more dark knowledge from the data via early stopping. For example, enhanced AIR by adjusting the temperature in the softmax layer. In self-distillation, the label function h(·) is proposed to amplify the information gained in the earlier epochs so that the knowledge gained in the earlier epochs is preserved. Thus, self-distillation does not require early stopping which makes the algorithm more user-friendly. On the other hand, the selfdistillation algorithm dynamically enhances AIR which enables it to achieve 2 convergence and thus better generalization. Again, we use the ratio of the norm of the label vector which lies in the top-5 eigenspace as a representative of informative information. We call the subtraction of informative information corresponding to the label vector provided by self-distillation algorithm and original label vector as information gain. From 5 we can see that the information gain is mostly larger than zero during the training of self-distillation algorithm. This phenomenon indicates that the supervision signal of self-distillation algorithm gains more information than directly using the noisy label. To sum up, a well-designed distillation algorithm can enjoy a regularization effect beyond early stopping and is able to gain more knowledge from the data. This paper provided an understanding of distillation using overparameterized neural networks. We observed that such neural networks posses the property of Anisotropic Information Retrieval (AIR), which means the neural network tends to fit the infomrative information (i.e. the eigenspaces associated with the largest few eigenvalues of NTK) first and the non-informative information later. Through AIR, we further observed that distillation of the Dark Knowledge is mainly due to early stopping. Based on this new understanding, we proposed a new self-distillation algorithm for noisy label refinery. Both theoretical and empirical justifications of the performance of the new algorithm were provided. Our analysis is based on the assumption that the teacher neural network is overparameterized. When the teacher network is not overparameterized, the network will be biased towards the label even without early stopping. It is still an interesting and unclear problem that whether the bias can provide us with more information. For label refinery, our analysis is mostly based on the symmetric noise setting. We are interested in extending our analysis to the asymmetric setting. A PROOF DETAILS A.1 NEURAL NETWORK PROPERTIES As preliminaries, we first discuss some properties of the neural network. We begin with the jacobian of the one layer neural network x → v φ(W x), the Jacobian matrix with respect to W takes the form First we borrow Lemma 6.6, 6.7, 6.8 from and Theorem 6.7, 6.8 from. T be a data matrix made up of data with unit Euclidean norm. Assuming that λ(X) > 0, the following properties hold., at random Gaussian initialization W 0 ∼ N k×d, with probability at least 1 − δ, we have T in whichx i corresponds to the center of cluster including x i. What's more, we define the matrix of cluster center C = [c 1, c 2, . . ., c K] T. Assuming that λ(C) > 0, the following properties hold. •, at random Gaussian initialization W 0 ∼ N k×d, with probability at least 1 − δ, we have • range(J(W,X)) ⊂ S + for any parameter matrix W. Then, we gives out the perturbation analysis of the Jacobian matrix. Lemma 3. Let X be a -clusterable data matrix with its center matrixX. For parameter matrices W,W, we have Proof. We bound J(W, X) − J(W,X) by The first term is bounded by Lemma 1. As to the second term, we bound it by Combining the inequality above, we get Lemma 4. Let X be a -clusterable data matrix with its center matrixX. We assume W 1, W 2 have a upper bound c √ k. Then for parameter matrices W 1, W 2,W 1,W 2, we have Proof. By the definition of average Jacobian, we have A.2 PROVE OF THE THEOREM First, we introduce the proof idea of our theorem. Our proof of the theorem divides the learning process into two stages. During the first stage, we aim to prove that the neural network will give out the right classification, i.e. the 0-1-loss converges to 0. The proof in this part is modified from. Furthermore, we proved that training 0-1-loss will keep 0 until the second stage starts and the margin at the first stage will larger than 1−2ρ 2. During the second stage, we prove that the neural networks start to further enlarge the margin and finally the 2 loss starts to converge to zero. has shown that this dynamic can be illustrated by the average Jacobian. Definition 3. We define the average Jacobian for two parameters W 1 and W 2 and data matrix X as The residualr = f (θ) − y, r = f (θ) − y obey the following equation r = (I − ηC(θ))r In our proof, we project the residual to the following subspace Definition 4. Let {x i} n i=1 be a -clusterable dataset and {x i} n i=1 be the associated cluster centers, that is,x i = c l iff x i is from lth cluster. We define the support subspace S + as a subspace of dimension K, dictated by the cluster membership as follows. Let Λ l ⊂ {1, 2, · · ·, n} be the set of coordinates i such that = c l. Then S + is characterized by Definition 5. We define the minimum eigenvalue of a matrix B on a subspace S σ min (B, S) = min where P S is the projection to the space S. Recall the generation process of the dataset Definition 6. (Clusterable Dataset Descriptions) • We assume that {x i} i∈[n] contains points with unit Euclidean norm and has K clusters. Let n l be the number of points in the lth cluster. Assume that number of data in each cluster is balanced in the sense that n l ≥ c low n K for constant c low > 0. • For each of the K clusters, we assume that all the input data lie within the Euclidean ball B(c l,), where c l is the center with unit Euclidean norm and > 0 is the radius. • A dataset satisfying the above assumptions is called an -clusterable dataset. First, we reduct the dataset to its cluster center, i.e. = 0 for -clusterable dataset. Lemma 6. We fix the label function be a -clusterable dataset and {x i} n i=1 be the associated cluster centers, that is,x i = c l iff x i is from lth cluster. We denote the data matrix X andX. We denote α =. We set the learning rate η = min(where Θ is maximum of the residual norm during the optimization. We suppose along the optimization path we have α ≤ J(W,X)v ≤ β for all v ∈ S +. We set T 1 = log 1− ηα 2 4 1−2ρ 8 r0 2, wherer 0 = P S+ (f (W 0,X) − y 0 ) is the projected residual on the space S +. Then ∀t ≥ T 1, we have • The neural network can learn the true label sgn (f) (W t,X) =ỹ. • The weight vector will be close to its initialization for all iterations Proof. We denote C 1 max t≥0 2 √ n(α t − α t+1). We denote J t = J(W t,X). Follow the step of gradient descent, we have We denote G t = J(W t+1, W t,X)J(W t,X) T. Then the dynamic of gradient descent can be written by We consider the dynamic of the residual r t r t+1 = (I − ηG t)r t + y t − y t+1. We project the residual on S +r t+1 = (I − ηG t)r t +ȳ t −ȳ t+1. Thus, the norm of the residual can be bounded by Utilizing the following lemma Lemma 7. (Claim2.) Let P S+ be the projection matrix to S +, then the following inequality holds Therefore, as long as η ≤ α Lβ rt 2 holds, we have When By simple calculation, we have After T 1 = log 1− As a , all the data have been classified correctly at iteration T 1 and the following inequality holds For the next step, we use induction to prove that For t = T 1, it has already been proved. We assume the claim is correct for arbitrary t, we establish the induction for t + 1. By applying projection to S + on equation 7, we have By applying lemma 7, we have Given that h(f t) =ỹ, we haveȳ Thus, for f t+1 we have By the definition of h(·), we deduce that h(f t+1) =ỹ. Back to equation 8, we have Finally, we estimate the total variation of the parameter. Combining equation 11 and 12, we can conclude that inequality holds for all t ≥ 0. After taking sum on both sides for t = 0, 1, 2,..., we have By simple calculation, we get Combining equation 6, we have Then we further to adapt the above theorem to the -clusterable dataset by a pertubation analysis. Since we have a simple inequality α a ≥ α b, in the following discussion, we simply replace the α a in the previous by α b, the also holds. be a -clusterable dataset and {x i} n i=1 be the associated cluster centers, that is,x i = c l iff x i is from lth cluster. We denote the data matrix X andX. For the same initialization W 0 =W 0, we run the self-distillation algorithm on X and tX respectively. We denote the parameter matrix W t andW t for t ≥ 0. We denote α = c low nΛ. We set the learning rate η = min(where Θ is maximum of the residual norm during the optimization. We denote c √ k the upper bound of the Frobenius norm of the parameter matrxi and we set M = c + 1. We set T 2 = inf{t : α t < 1 24 √ n}. Then if the following conditions hold Proof. We introduce the following notations. We can conclude the following inequalities from lemma 3 and lemma 4 Thus the parameters are updated by gradient descent, we have Also, we have To sum up, if we can guarantee that p t ≤ 1−2ρ 4 holds for t ≤ T 2, we have For r t 2, we have r t 2 ≤ Tr t 2 + r t − Tr t 2 Thus we have the following inequality for p t We claim that if the following conditions for and k hold It is obviously when t = 0. We further suppose the inequalities hold for an arbitrary t satisfying t < T 2, we have because of the condition on k ensures that LΘd t ≤ M ΓΘ √ n. When it comes to p t+1, we have because of the conditions on and k ensure that Ld T2 ≤ M Γ √ n and p t ≤ Θ. Now we are ready to finalize the proof of our main theorem. Proof. We denote C 2 max s≥T2 2 √ n(α s − α s+1). We take Θ = (C 3 Γ log 8 δ + 1) √ n. C 3 is the constant in Hoeffding's inequality. By lemma 6, we have Θ ≥ max t≥0 T r t 2 with probability 1 − δ 4. Combining lemma 6 and 8, we have f (W t, X) =ỹ for T 1 ≤ t ≤ T 2 and Similar to the proof in 6, we consider the gradient descent on original dataset X after T 2. We proof the following claim by induction For s = T 2, it has already been proved. We assume the claim is correct for arbitrary s, we establish the induction for s + 1. By equation 7, we have By applying lemma 7, we have f (W s+1,X) −ȳ s 2 ≤ r s 2 ≤ 5 8 (1 − 2ρ) + 1 24. Given that h(f (W s, X)) =ỹ, we have y s (i)ỹ(i) ≥ 1 − 2α T2, i = 1, 2,..., n. Thus, for f (W s+1, x i) we have As a , we have f (W s+1, X) =ỹ. Furthermore, we can bound r s+1 2 by r s+1 2 ≤ (I − ηG s)r s 2 + (1 − α s) h(f (W s, X)) − h(f (W s+1, X)) 2 + (α s − α s+1) ȳ − h(f (W s+1, X)) 2 ≤(1 − ηα To sum up, we have the following inequality holds for all s ≥ T 2 r s+1 2 ≤ (1 − ηα 2 2) r s 2 + 2 √ n(α s − α s+1). After taking sum on both sides for s ≥ T 2, we have By lemma 1 and lemma 2, as long as k ≥ To ensure α lower bounding the eigenvalue of the gram matrix, we need to verify that That is to say Another condition related to R is the condition on M. We require By Bernstein's inequality, we have with probability 1 − δ/4. On the condition that k ≥ R 2 (acutually one can show that we can choose M = d + C 4 log 8 δ + 2. We take these constants to the lemma 8. Firstly, for we have ---512 bn fc. ---10 dropout Table 2 : Architecture of the student network. After each convolution layer, there is a Rectified Linear Unit(ReLU) layer. Experiment In Section2.3 and Section3.5 For these two experiments, we modify CIFAR10 to a binary classification task. We choose class 2, 7 to be the positive class and the others to be negative. We train resnet56 with MSE loss. We set batch size 128, momentum 0.9, weight decay 5e − 4 and learning rate 0.1. In the experiment in section2.3, we fetch a batch of data (batch size=128) from the testset and randomly corrupted the label by noise level 0, 0.1, 0.2, 0.3, 0.4, 0.5. We plot the ratio of the norm of the label vector which lies in the subspaces corresponding to top-5 eigenvalues of NTK. In the experiment in section2.3, we calculate the ratio of the norm of the label vector which lies in the subspaces corresponding to top-5 eigenvalues of NTK firstly. We calculate the ratio of the norm of the label vector provided by the self-distillation algorithm lies in the top-5 eigenspace. We calculate the difference of the latter ratio and the former ratio and called it information gain. We plot the information gain of the first 1500 iterations.

Represent the TLDR for finding the corresponding scientific article

theoretically understand the regularization effect of distillation. We show that early stopping is essential in this process. From this perspective, we developed a distillation method for learning with corrupted Label with theoretical guarantees.

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Lifelong learning poses considerable challenges in terms of effectiveness (minimizing prediction errors for all tasks) and overall computational tractability for real-time performance. This paper addresses continuous lifelong multitask learning by jointly re-estimating the inter-task relations (\textit{output} kernel) and the per-task model parameters at each round, assuming data arrives in a streaming fashion. We propose a novel algorithm called \textit{Online Output Kernel Learning Algorithm} (OOKLA) for lifelong learning setting. To avoid the memory explosion, we propose a robust budget-limited versions of the proposed algorithm that efficiently utilize the relationship between the tasks to bound the total number of representative examples in the support set. In addition, we propose a two-stage budgeted scheme for efficiently tackling the task-specific budget constraints in lifelong learning. Our empirical over three datasets indicate superior AUC performance for OOKLA and its budget-limited cousins over strong baselines. Instead of learning individual models, learning from multiple tasks leverages the relationships among tasks to jointly build better models for each task and thereby improve the transfer of relevant knowledge between the tasks, especially from information-rich tasks to information-poor ones. Unlike traditional multitask learning, where the tasks are presented simultaneously and an entire training set is available to the learner , in lifelong learning the tasks arrives sequentially BID27 ). This paper considers a continuous lifelong learning setting in which both the tasks and the examples of the tasks arrive in an online fashion, without any predetermined order. Following the online setting, particularly from BID24 BID7, at each round t, the learner receives an example from a task, along with the task identifier and predicts the output label for the example. Subsequently, the learner receives the true label and updates the model(s) as necessary. This process is repeated as we receive additional data from the same or different tasks. Our approach follows an error-driven update rule in which the model for a given task is updated only when the prediction for that task is in error. Lifelong learning poses considerable challenges in terms of effectiveness (minimizing prediction errors for all tasks) and overall computational tractability for real-time performance. A lifelong learning agent must provide an efficient way to learn new tasks faster by utilizing the knowledge learned from the previous tasks and also not forgetting or significantly degrading performance on the old tasks. The goal of a lifelong learner is to minimize errors as compared to the full ideal hindsight learner, which has access to all the training data and no bounds on memory or computation. This paper addresses lifelong multitask learning by jointly re-estimating the inter-task relations from the data and the per-task model parameters at each round, assuming data arrives in a streaming fashion. We define the task relationship matrix as output kernels in Reproducing Kernel Hilbert Space (RKHS) on multitask examples. We propose a novel algorithm called Online Output Kernel Learning Algorithm (OOKLA) for lifelong learning setting. For a successful lifelong learning with kernels, we need to address two key challenges: learn the relationships between the tasks (output kernel) efficiently from the data stream and bound the size of the knowledge to avoid memory explosion. The key challenge in learning with a large number of tasks is to adaptively learn the model parameters and the task relationships, which potentially change over time. Without manageability-efficient updates at each round, learning the task relationship matrix automatically may impose a severe computational burden. In other words, we need to make predictions and update the models in an efficient real time manner. We propose simple and quite intuitive update rules for learning the task relationship matrix. When we receive a new example, the algorithm updates the output kernel when the learner made a mistake by computing the similarity between the new example and the set of representative examples (stored in the memory) that belongs to a specific task. If the two examples have similar (different) labels and high similarity, then the relationship between the tasks is increased (decreased) to reflect the positive (negative) correlation and vice versa. To avoid the memory explosion associated with the lifelong learning setting, we propose a robust budget-limited version of the proposed algorithm that efficiently utilizes the relationship between the tasks to bound the total number of representative examples in the support set. In addition, we propose a two-stage budgeted scheme for efficiently tackling the task-specific budget constraints in lifelong learning. It is worth noting that the problem of lifelong multitask learning is closely related to online multitask learning. Although the objectives of both online multitask learning and lifelong learning are similar, one key difference is that the online multitask learning, unlike in the lifelong learning, may require that the number of tasks be specified beforehand. In recent years, online multitask learning has attracted extensive research attention BID0; BID10; BID16 BID7; BID24 BID17. We evaluate our proposed methods with several state-of-the-art online learning algorithms for multiple tasks. Throughout this paper, we refer to our proposed method as online multitask learning or lifelong learning. There are many useful application areas for lifelong learning, including optimizing financial trading as market conditions evolve, email prioritization with new tasks or preferences emerging, personalized news, and spam filtering, with evolving nature of spam. Consider the latter, where some spam is universal to all users (e.g. financial scams), some messages might be useful to certain affinity groups, but spam to most others (e.g. announcements of meditation classes or other special interest activities), and some may depend on evolving user interests. In spam filtering each user is a "task," and shared interests and dis-interests formulate the inter-task relationship matrix. If we can learn the matrix as well as improving models from specific spam/not-spam decisions, we can perform mass customization of spam filtering, borrowing from spam/not-spam feedback from users with similar preferences. The primary contribution of this paper is precisely the joint learning of inter-task relationships and its use in estimating per-task model parameters in a lifelong learning setting. Most existing work in online learning of multiple task focuses on how to take advantage of task relationships. To achieve this, BID16 imposed a hard constraint on the K simultaneous actions taken by the learner in the expert setting, BID1 used matrix regularization, and BID10 proposed a global loss function, as an absolute norm, to tie together the loss values of the individual tasks. Different from existing online multitask learning models, our paper proposes an intuitive and efficient way to learn the task relationship matrix automatically from the data, and to explicitly take into account the learned relationships during model updates. BID7 assumes that task relationships are available a priori. However often such taskrelation prior knowledge is either unavailable or infeasible to obtain for many applications especially when the number of tasks K is large BID28 ) and/or when the manual annotation of task relationships is expensive BID15 ). BID24 formulated the learning of task relationship matrix as a Bregman-divergence minimization problem w.r.t. positive definite matrices. The model suffers from high computational complexity as semi-definite programming is required when updating the task relationship matrix at each online round. We show that with a different formulation, we can obtain a similar but much cheaper updating rule for learning the inter-task weights. BID17 proposed an efficient method for learning the task relationship matrix using the cross-task performance measure, but their approach learns only the positive correlation between the tasks. Our proposed approach learns positive and negative correlations between the tasks for robust transfer of knowledge from the previously learned tasks. Recent work in output kernel learning estimate the task covariance matrix in RKHS space, inferred it directly from the data BID12 BID25 BID14 ). The task covariance matrix is called the output kernel defined on the tasks, similar to the scalar kernel on the inputs. Most recently, BID14 showed that for a class of regularization functions, we can efficiently learn this output kernel. Unfortunately most of the proposed methods for learning output kernels require access to the entire data for the learning algorithm, a luxury unavailable in online learning and especially in the lifelong learning setting. Unlike in online multitask learning, most lifelong learning approaches use a single model for all the tasks or reuse the models from the previous tasks to build a model for the new task BID2 FORMULA0 ). These approaches either increase the computation time on iterations where we encounter a novel task or reduce the prediction power of the model learned from the previous tasks due to catastrophic forgetting. To the best of our knowledge, relationships among the tasks has not been successfully exploited in the lifelong learning setting due to the difficulty in learning a positive semi-definite task relationship matrix in large-scale applications. This paper provides an efficient way to learn the task relationship matrix in the lifelong learning setting. Let ((x t, i t), y t ) be the example received by the learner from the task i t (at the time step t) where we assume that the x t ∈ X and y t is its corresponding true label. The task i t can be a new task or one seen by the learner in the previous iterations. We denote by [N] the consecutive integers ranging from 1 to N. In this paper, we do not assume that the number of tasks is known to the learner ahead of time, an important constraint in lifelong learning problems. Let K be the number of tasks seen so far until the current iteration t. For brevity, we consider a binary classification problem for each task y t ∈ {−1, +1}, but the methods generalize to multi-class cases and are also applicable to regression tasks. We assume that the learner made a mistake if y t =ŷ t whereŷ t is the predicted label. Our approach follows a mistake-driven update rule in which the model for a given task is updated only on rounds where the learner predictions differ from the true label. Let K: X × X → R (kernel on input space) and Ω: N × N → R (output kernel) be symmetric, positive semi-definite (p.s.d) multitask kernel functions and denote H as their corresponding RKHS of functions with the norm · H on multitask examples BID26; BID12 BID14 ). Using the above notation, we can define a kernel representation of an example based on a set of representative examples collected on the previous iterations (prototypes). Formally, given an example x ∈ X, its kernel representation can be written using this set:x −→ {K(x, x s): s ∈ S} S is the set of stored examples for which the learner made a mistake in the past. The set S is called the support set. The online classification function is then defined as the weighted sum of the kernel combination of the examples in the support set. To account for the examples from the different tasks, we consider both the kernel on the input space K and the output kernel Ω in our classification function. DISPLAYFORM0 We set α s = y s. The predicted label for a new example is computed from the linear combination of the labels of the examples from the support set S weighted by their input similarity K and the task similarity Ω to the new example. Using the kernel trick, one can write: DISPLAYFORM1 Note that, in the above representation, we need to learn both the support set S and the output kernel Ω from the data. As explained in the previous section, for a successful lifelong learning with kernels, we need to address two key challenges: learn the relationships between the tasks (output kernel) efficiently from the data arriving in an online fashion and bound the size of the support set S to avoid memory explosion. We address these two challenges in the following sections. DISPLAYFORM2 (or) DISPLAYFORM3 end end Our objective function for the lifelong learning problem is given as follows: DISPLAYFORM0 where (·) is some loss function such as hinge loss or logistic loss, R(·) is the regularization on the task relationship matrix Ω and λ is the regularization parameter. Note that f in the above equation depends on Ω. In order to reduce the time taken for each time-step, we require an efficient update to the task relationship matrix Ω. Following the work of BID14 in the batch setting, we consider a subset of regularization functions R for which we can efficiently learn the task covariance matrix. Consider the dual function of the above equation, at time-step t (see BID4 ; BID14): DISPLAYFORM1 When we consider the entry-wise l p norm between Ω and Ω (t−1) from the previous iteration as our regularization i.e., R(Ω, DISPLAYFORM2 we get the update function in Equation 2. Similarly, if we consider the generalized KL-divergence between Ω and Ω (t−1) i.e., R(Ω, FIG1) itk, we get the update function in Equation 3. Unlike in the previous work, we update only the row (and the corresponding column) of the task relationship matrix Ω specific to the task i t, which significantly reduces the time taken per example. DISPLAYFORM3 We can see that the update equations are simple and quite intuitive. For a given new example (x t, i t) at round t, the algorithm updates Ω itk (for some k ∈ [K]) by computing the similarity between the new example and the examples in the support set S that belongs to the task k. If the two examples have similar (different) labels and high similarity K(x t, x s), then the Ω itk is increased to reflect the positive (negative) correlation and vice versa. A value close to 0 implies no significant relationship between the tasks. The update to the Ω itk is normalized by the regularization parameter λ for scaling. It is worth noting that our update equations do not violate the p.s.d constraints on Ω in Equation 5.If Ω from the previous iteration is a p.s.d matrix and the update is a p.s.d matrix (as it is computed using the Gram matrix of the example from the previous iteration), the sum and Hadamard product of two p.s.d matrices satisfy the p.s.d constraint (using the Schur Product Theorem).Algorithm 1 outlines the key steps in our proposed method. We write f ((x t, i t)) as f it (x t) for notational convenience. At each time-step t, the learner receives an example x t ∈ X and predicts the output label y t usingŷ t = sign(f it (x t)). We update both the support set S and the output kernel Ω it· when the learner makes a mistake. DISPLAYFORM4 Find an example to remove arg max DISPLAYFORM5 as in Algorithm 1. end end Algorithm 3: Two-Stage Budgeted Learning Initialize: DISPLAYFORM6 In Algorithm 1, we can see that both the classification function f and the update equations for Ω use the support set S. When the target function changes over time, the support set S grows unboundedly. This leads to serious computational and runtime issues especially in the lifelong learning setting. The most common solution to this problem is to impose a bound on the number of examples in the support set S. There are several budget maintenance strategies proposed recently BID6; BID11; BID18 ). Unfortunately these schemes cannot be directly used in our setting due to the output kernels in our learning formulation. BID5 proposed multitask variants of these schemes but they are impractical for the lifelong learning setting. We follow a simple support set removal schemes based on BID8. In single-task setting, when the number of examples in the support set S exceeds the limit (say B), a simple removal scheme chooses an example x r with the highest confidence from S. The confidence of an example x r is measured using y r f(x r) after removing x r from the support set S. DISPLAYFORM0 We extend the above approach to the multitask and lifelong learning settings. Since the support set S is shared by all the tasks, we choose an example x r with high confidence to remove from each task function f k, weighted by the relationship among the tasks. The objective function to choose the example is shown in Equation 6. We show in the experiment section that this simple approach is efficient and performs significantly better than the state-of-the-art budget maintenance strategies. Algorithm 2 shows pseudocode of the proposed budgeted learning algorithm. In lifelong learning setting, the number of tasks is typically large. The support set S may have hundreds or thousands of examples from all the tasks. Each task does not use all the examples from the support set S. For example, in movie recommendations task, recommendation for each user (task) can be characterized by just a few movies (subset of examples) in the support set S. Motivated by this observation, we propose a two-stage budgeted learning algorithm for the lifelong learning setting. Algorithm 3 shows pseudocode of the proposed two-stage budgeted learning algorithm. In addition to the support set S, we maintain task-specific support set T k. We choose the budget for each task (say L) where L <<< B. Similar to the removal strategies for S, we remove an example from T k when |T k | > L and replace with an example from the set S − T k. The proposed two-stage approach provides better runtime complexity compared to the budgeted algorithm in Algorithm 2. Since only a subset of tasks may hold an example from S, the removal step in Equation 6 requires only a subset of tasks for choosing an example. This improves the runtime per iteration significantly when the number of tasks is large. One may consider a different budget size for each task L k based on the complexity of the task. In addition, the proposed two-stage budgeted learning algorithm provides an alternative approach to using state-of-the-art budget maintenance strategies. For example, it is easier to use the Projectron algorithm BID18 ) on T k, rather than on S. We will further explore this line of research in our future work. In this section, we evaluate the performance of our algorithms. All reported are averaged over 10 random runs on permutations of the training data. Unless otherwise specified, all model parameters are chosen via 5-fold cross validation. We use three benchmark datasets, commonly used for evaluating online multitask learning. Details are given below:Newsgroups Dataset 2 consists of 20 tasks generated from two subject groups: comp and talk.politics. We paired two newsgroups, one from each subject (e.g.,comp.graphics vs talk.politics.guns), for each task. In order to account for positive/negative correlation between the tasks, we randomly choose one of the newsgroups as positive (+) or negative (−) class. Each post in a newsgroup is represented by a vocabulary of approximately 60K unique features. 3 We use the dataset obtained from ECML PAKDD 2006 Discovery challenge for the spam detection task. We used the task B challenge dataset, which consists of labeled training data from the inboxes of 15 users. We consider each user as a single task and the goal is to build a personalized spam filter for each user. Each task is a binary classification problem: spam (+) or non-spam (−) and each example consists of approximately 150K features representing term frequency of the word occurrences. Some spam is universal to all users (e.g. financial scams), but some messages might be useful to certain affinity groups and spam to most others. Such adaptive behavior of each user's interests and dis-interests can be modeled efficiently by utilizing the data from other users to learn per-user model parameters. 4 We also evaluated our algorithm on product reviews from amazon. The dataset contains product reviews from 25 domains. We consider each domain as a binary classification task. Reviews with rating > 3 were labeled positive (+), those with rating < 3 were labeled negative (−), reviews with rating = 3 are discarded as the sentiments were ambiguous and hard to predict. Similar to the previous datasets, each example consists of approximately 350K features representing term frequency of the word occurrences. We choose 2000 examples (100 posts per task) for 20 Newsgroups, 1500 emails for spam (100 emails per user inbox) and 2500 reviews for sentiment (100 reviews per domain) as training set for our experiments. Note that we intentionally kept the size of the training data small to simulate the lifelong learning setting and drive the need for learning from previous tasks, which diminishes as the training sets per task become large. Since these datasets have a class-imbalance issue (with few (+) examples as compared to (−) examples), we use average Area Under the ROC Curve (AU C) as the performance measure on the test set. To evaluate the performance of our proposed algorithm (OOKLA), we use the three datasets (Newsgroups, Spam and Sentiment) for evaluation and compare our proposed methods to 5 baselines. We implemented Perceptron and Passive-Aggressive algorithm (PA) BID9 for online multitask learning. Both Perceptron and PA learn independent model for each task. These two baselines do not exploit the task-relationship or the data from other tasks during model update. Next, we implemented two online multitask learning related to our approach: FOML -initializes Ω with fixed weights BID7, Online Multitask Relationship Learning (OMTRL) BID24 -learns a task covariance matrix along with task parameters. Since OMTRL requires expensive calls to SVD routines, we update the task-relationship matrix every 10 iterations. In addition, we compare our proposed methods against the performance of Online Smooth Multitask Learning (OSMTL) which learns a probabilistic distribution over all tasks, and adaptively refines the distribution over time BID17. We implement two versions of our proposed algorithm with different update rules for the task-relationship matrix: OOKLA-sum (Equation 2 OOKLA with sum update) OOKLA-exp (Equation 3 OOKLA with exponential update) as shown in Algorithm 1. TAB0 summarizes the performance of all the above algorithms on the three datasets. In addition to the AU C scores, we report the average total number of support vectors (nSV) and the CPU time taken for learning from one instance (Time).From the table, it is evident that both OOKLA-sum and OOKLA-exp outperform all the baselines in terms of both AU C and nSV. This is expected for the two default baselines (Perceptron and PA). The update rule for FOML is similar to ours but using fixed weights. The justify our claim that learning the task-relationship matrix adaptively leads to improved performance. As expected, both OOKLA and OSMTL consume less or comparable CPU time than the subset of baselines which take into account learning inter-task relationships. Unlike in the OMTRL algorithm that recomputes the task covariance matrix every iteration using expensive SVD routines, the task-relationship matrix in our proposed methods (and OSMTL) are updated independently for each task. We implement the OSMTL with exponential update for our experiments as it has shown to perform better than the other baselines. One of the major drawbacks of OSMTL is that it learn only the positive correlations between the tasks. The performance of OSMTL worsens when the tasks are negatively correlated. As we can see from the table, our proposed methods outperform OSMTL significantly in the Newsgroup dataset. TAB1 compares the proposed methods with different budget schemes and budget sizes in terms of test set AU C scores and the runtime. We use OOKLA-sum for this experiment. We set the value of B to {50, 100, 150} for all the datasets. We compare our proposed budgeted learning algorithm (Algorithm 2) with the following state-of-the-art algorithms for online budgeted learning: TAB1 shows both the test set AUC scores (first line) and time taken for learning from one instance (including the removal step). It is evident from the table, our proposed budgeted learning algorithm for online multitask learning significantly outperforms the other state-of-the-art budget schemes on most settings. Our proposed algorithm uses the relationship between the tasks efficiently to choose the next example for removal. Finally, we evaluate the performance of the proposed two-stage budgeted scheme compared to the Algorithm 2. To study the effect of different budget sizes L, we compute the cumulative mistake rate which uses all the examples from the support set S. We observe similar trend in the test set AUC scores. On average, we achieved over 16% improvement in running time compared to the budget maintenance scheme in Algorithm 2. We believe that the time consumption and the performance improvement will be even better for applications with larger numbers of tasks. We proposed a novel lifelong learning algorithm using output kernels. The proposed method efficiently learns both the model and the inter-task relationships at each iteration. Our update rules for learning the task relationship matrix, at each iteration, were motivated by the recent work in output kernel learning. In order to handle the memory explosion from an unbounded support set in the lifelong learning setting, we proposed a new budget maintenance scheme that utilizes the task relationship matrix to remove the least-useful (high confidence) example from the support set. In addition, we proposed a two-stage budget learning scheme based on the intuition that each task only requires a subset of the representative examples in the support set for efficient learning. It provides a competitive and efficient approach to handle large number of tasks in many real-life applications. The effectiveness of our algorithm is empirically verified over several benchmark datasets, outperforming several competitive baselines both in the unconstrained case and the budget-limited case, where selective forgetting was required.

a novel approach for online lifelong learning using output kernels.

Represent the TLDR for finding the corresponding scientific article

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.