Question: Represent the following document

Question: Approaches to continual learning aim to successfully learn a set of related tasks that arrive in an online manner. Recently, several frameworks have been developed which enable deep learning to be deployed in this learning scenario. A key modelling decision is to what extent the architecture should be shared across tasks. On the one hand, separately modelling each task avoids catastrophic forgetting but it does not support transfer learning and leads to large models. On the other hand, rigidly specifying a shared component and a task-specific part enables task transfer and limits the model size, but it is vulnerable to catastrophic forgetting and restricts the form of task-transfer that can occur. Ideally, the network should adaptively identify which parts of the network to share in a data driven way. Here we introduce such an approach called Continual Learning with Adaptive Weights (CLAW), which is based on probabilistic modelling and variational inference. Experiments show that CLAW achieves state-of-the-art performance on six benchmarks in terms of overall continual learning performance, as measured by classification accuracy, and in terms of addressing catastrophic forgetting. Continual learning (CL), sometimes called lifelong or incremental learning, refers to an online framework where the knowledge acquired from learning tasks in the past is kept and accumulated so that it can be reused in the present and future. Data belonging to different tasks could potentially be non i.i.d. (; ; ; ; ;). A continual learner must be able to learn a new task, crucially, without forgetting previous tasks (; ; ;). In addition, CL frameworks should continually adapt to any domain shift occurring across tasks. The learning updates must be incremental -i.e, the model is updated at each task only using the new data and the old model, without access to all previous data (from earlier tasks) -due to speed, security and privacy constraints. A compromise must be found between adapting to new tasks and enforcing stability to preserve knowledge from previous tasks. Excessive adaptation could lead to inadvertent forgetting of how to perform earlier tasks. Indeed, catastrophic forgetting is one of the main pathologies in continual learning (; ; ; ; ; a; ; ; ; ; ; ;). Many approaches to continual learning employ an architecture which is divided a priori into (i) a slowly evolving, global part; and (ii) a quickly evolving, task-specific, local part. This is one way to enable multi-task transfer whilst mitigating catastrophic forgetting, which has proven to be effective (b; ;), albeit with limitations. Specifying a priori the shared global, and task-specific local parts in the architecture restricts flexibility. As more complex and heterogeneous tasks are considered, one would like a more flexible, data-driven approach to determine the appropriate amount of sharing across tasks. Here, we aim at automating the architecture adaptation process so that each neuron of the network can either be kept intact, i.e. acting as global, or adapted to the new task locally. Our proposed variational inference framework is flexible enough to learn the range within which the adaptation parameters can vary. We introduce for each neuron one binary parameter controlling whether or not to adapt, and two parameters to control the magnitude of adaptation. All parameters are learnt via variational inference. We introduce our framework as an expansion of the variational continual learning algorithm , whose variational and sequential Bayesian nature makes it convenient for our modelling and architecture adaptation procedure. Our modelling ideas can also be applied to other continual learning frameworks, see the Appendix for a brief discussion. We highlight the following contributions: A modelling framework which flexibly automates the adaptation of local and global parts of the (multi-task) continual architecture. This optimizes the tradeoff between mitigating catastrophic forgetting and improving task transfer. A probabilistic variational inference algorithm which supports incremental updates with adaptively learned parameters. The ability to combine our modelling and inference approaches without any significant augmentation of the architecture (no new neurons are needed). State-of-the-art in six experiments on five datasets, which demonstrate the effectiveness of our framework in terms of overall accuracy and reducing catastrophic forgetting. We briefly discuss three related approaches to continual learning: (a) regularisation-based, (b) architecture-based and (c) memory-based. We provide more details of related work in Section A in the Appendix. (a) A complementary approach to CLAW is the regularisation-based approach to balance adaptability with catastrophic forgetting: a level of stability is kept via protecting parameters that greatly influence the prediction against radical changes, while allowing the rest of the parameters to change without restriction (; ; ; ; ; ; c). The elastic weight consolidation (EWC) algorithm by is a seminal example, where a quadratic penalty is imposed on the difference between parameter values of the old and new tasks. One limitation is the high level of hand tuning required. (b) The architecture-based approach aims to deal with stability and adaptation issues by a fixed division of the architecture into global and local parts (b; ; ; ; ; ; b). (c) The memory-based approach relies on episodic memory to store data (or pseudo-data) from previous tasks (; ; ; ; ; ; ; ; ; ; van de ; ;). Limitations include overheads for tasks such as data storage, replay, and optimisation to select (or generate) the points. CLAW can as well be seen as a combination of a regularisation-based approach (the variational inference mechanism) and a modelling approach which automates the architecture building process in a data-driven manner, avoiding the overhead ing from either storing or generating data points from previous tasks. CLAW is also orthogonal to (and simple to combine with, if needed) memory-based methods. In this paper, we use Variational Continual Learning as the underlying continual learning framework. However, our methods apply to other frameworks, see Appendix (Section A.1). VCL is a variational Bayesian framework where the posterior of the model parameters θ is learnt and updated continually from a sequence of T datasets, {x, where t = 1, 2, . . ., T and N t is the size of the dataset associated with the t-th task. More specifically, denote by p(y|θ, x) the probability distribution returned by a discriminative classifier with input x, output y and parameters θ., we approximate the intractable posterior p(θ|D 1:t) after observing the first t datasets via a tractable variational distribution q t as: where q 0 is the prior p, p(t), and Z t is the normalizing constant which does not depend on θ but only on the data D. This framework allows the approximate posterior q t (θ) to be updated incrementally from the previous approximate posterior q t−1 (θ) in an online fashion. In VCL, the approximation in is performed by minimizing the following KL-divergence over a family Q of tractable distributions: This framework can be enhanced to further mitigate catastrophic forgetting by using a coreset , i.e. a representative set of data from previously observed tasks that can serve as memory and can be revisited before making a decision. As discussed in the Related Work, this leads to overhead costs of memory and optimisation (selecting most representative data points). Previous work on VCL considered simple models without automatic architecture building or adaptation. In earlier CL approaches, the parts of the network architecture that are shared among the learnt tasks are designated a priori. To alleviate this rigidity and to effectively balance adaptation and stability, we propose a multi-task, continual model in which the adaptation of the architecture is data-driven by learning which neurons need to be adapted as well as the maximum adaptation capacity for each. All the model parameters (including those used for adaptation) are estimated via an efficient variational inference algorithm which incrementally learns from data of the successive tasks, without a need to store (nor generate) data from previous tasks and with no expansion in the network size. With model parameters θ, the overall variational objective we aim at maximising at task with index t is equivalent to the following online marginal likelihood: We propose a framework where the architecture, whose parameters are θ, is flexibly adapted based on the available tasks, via a learning procedure that will be described below. With each task, we automate the adaptation of the neuron contributions. Both the adaptation decisions (i.e. whether or not to adapt) and the maximum allowed degree of adaptation for every neuron are learnt. We refer to the binary adaptation variable as α. There is another variable s that is learnt in a multi-task fashion to control the maximum degree of adaptation, such that the expression b = s 1+e −a − 1 limits how far the task-specific weights can differ from the global weights, in case the respective neuron is to be adapted. The parameter a depicts unconstrained adaptation, as described later. 2 We illustrate the proposed model to perform this adaptation by learning the probabilistic contributions of the different neurons within the network architecture on a task-by-task basis. We follow this with the inference details. Steps of the proposed modeling are listed as follows: • For a task T, the classifier that we are modeling outputs: • The task-specific weights w T can be expressed in terms of their global counterparts as follows: The symbol • denotes an element-wise (Hadamard) multiplication. • For each task T and each neuron j at layer i, α T i,j is a binary variable which indicates whether the corresponding weight is adapted (α T i,j = 1) or unadapted (α T i,j = 0). Initially assume that the adaptation probability α T i,j follows a Bernoulli distribution with probability p i,j 3, α T i,j ∼ Bernoulli(p i,j). Since this Bernoulli is not straightforward to optimise, and to adopt a scalable inference procedure based on continuous latent variables, we replace this Bernoulli with a Gaussian that has an equivalent mean and variance from which we draw α T i,j. For the sake of attaining higher fidelity than what is granted by a standard Gaussian, we base our inference on a variational Gaussian estimation. Though in a context different from continual learning and with different estimators, the idea of replacing Bernoulli with an equivalent Gaussian has proven to be effective with dropout (; . The approximation of the Bernoulli distribution by the corresponding Gaussian distribution is achieved by matching the mean and variance. The mean and variance of the Bernoulli distribution are p i,j, p i,j (1 − p i,j), respectively. A Gaussian distribution with the same mean and variance is used to fit α • The variable b T controls the strength of the adaptation and it limits the range of adaptation via: So that the maximum adaptation is s. The variable a T is an unconstrained adaptation value, similar to that in . The addition of 1 is to facilitate the usage of a probability distribution while still keeping an adaptation range allowing for the attenuation or amplification of each neuron's contribution. • Before facing the first dataset and learning task t = 1, the prior on the weights q 0 (w) = p(w) is chosen to be a log-scale prior, which can be expressed as: p(log |w|) ∝ c, where c is a constant. The log-scale prior can alternatively be described as: At a high level, adapting neuron contributions can be seen as a generalisation of attention mechanisms in the context of continual learning. Applying this adaptation procedure to the input leads to an attention mechanism. However, our approach is more general since we do not apply it only to the very bottom (i.e. input) layer, but throughout the whole network. We next show how our variational inference mechanism enables us to learn the adaptation parameters. We describe the details related to the proposed variational inference mechanism. The adaptation parameters are included within the variational parameters. The (unadapted version of the) model parameters θ consist of the weight vectors w. To automate adaptation, we perform inference on p i,j, which would have otherwise been a hyperparameter of the prior (; ;). Multiplying w by (1 + bα) where α is distributed according to, then from with random noise variable ∼ N: From and, the corresponding KL-divergence between the variational posterior of w, q(w|γ) and the prior p(w) is as follows. The subscripts are removed when q in turn is used as a subscript for improved readability. The variational parameters are γ i,j and p i,j. where the switch from to is due to the entropy computation of the Gaussian q(w i,j |γ i,j) defined in. The switch from to is due to using a log-scale prior, similar to Appendix C in and to Section 4.2 in . E q(w|γ) log | | is computed via an accurate approximation similar to equation in , with slightly different values of k 1, k 2 and k 3. This is a very close approximation via numerically pre-computing E q(w|γ) log | | using a third degree polynomial ). This is the form of the KL-divergence between the approximate posterior after the first task and the prior. Afterwards, it is straightforward to see how this KL-divergence applies for the subsequent tasks in a manner similar to, but while taking into account the new posterior form and original prior. The KL-divergence expression derived in is to be minimised. By minimising with respect to p i,j and then using samples from the respective distributions to assign values to α i,j, adapted contributions of each neuron j at each layer i of the network are learnt per task. Values of p i,j are constrained between 0 and 1 during training via projected gradient descent. Using to express the value of b i,j, and neglecting the constant term therein since it does not affect the optimisation, the KL-divergence in is equivalent to: Values of a i,j are straightforwardly learnt by minimising with respect to a i,j. This subsection explains how to learn the maximum adaptation variable s i,j. Values of the maximum s i,j of the logistic function defined in are learnt from multiple tasks. For each neuron j at layer i, there is a general value s i,j and another value that is specific for each task t, referred to as s i,j,t. This is similar to the meta-learning procedure proposed in . The following procedure to learn s is performed for each task t such that: (i) the optimisation performed to learn a task-specific value s i,j,t benefits from the warm initialisation with the general value s i,j rather than a random initial condition; and then (ii) the new information obtained from the current task t is ultimately reflected back to update the general value s i,j. • First divide the sample N t into two halves. For the first half, depart from the general value of s i,j as an initial condition, and use the assigned data examples from task t to learn the taskspecific values s i,j,t for the current task t. For neuron j at layer i, refer to the second term in, The set of parameters θ contains s as well as other parameters, but we focus here on s in the f notation since the following procedure is developed to optimise s. Also, refer to the loss of the (classification) function f as Err(f) = CE(f (x, θ) y), where CE stands for the cross-entropy: • Now use the second half of the data from task t to update the general learnt value s i,j: Where ω 1 and ω 2 are step-size parameters. When testing on samples from task t after having faced future tasks t + 1, t + 2,..., the value of s i,j used is the learnt s i,j,t. There is only one value per neuron, so the overhead ing from storing such values is negligible. The key steps of the algorithm are listed in Algorithm 1. Input: A sequence of T datasets, {x, where t = 1, 2, . . ., T and N t is the size of the dataset associated with the t-th task. Output: q t (θ), where θ are the model parameters. Initialise all p(|w i,j |) with a log-scale prior, as in. for the current task t. for i = 1... # layers do for j = 1... # neurons at layer i do Compute p i,j using stochastic gradient descent on. Compute s i,j,t using. Update the corresponding general value s i,j using. end for end for end for At task t, the algorithmic complexity of a single joint update of the parameters θ based on the additive terms in 2 ), where L is the number of layers in the network, D is the (largest) number of neurons within a single layer, E is the number of samples taken from the random noise variable, and M is the minibatch size. Each α is obtained by taking one sample from the corresponding p, so that does not in an overhead in terms of the complexity. Our experiments mainly aim at evaluating the following: (i) the overall performance of the introduced CLAW, depicted by the average classification accuracy over all the tasks; (ii) the extent to which catastrophic forgetting can be mitigated when deploying CLAW; and (iii) the achieved degree of positive forward transfer. The experiments demonstrate the effectiveness of CLAW in achieving state-of-the-art continual learning measured by classification accuracy and by the achieved reduction in catastrophic forgetting. We also perform ablations in Section D in the Appendix which exhibit the relevance of each of the proposed adaptation parameters. We perform six experiments on five datasets. The datasets in use are: MNIST , notMNIST , Fashion-MNIST , Omniglot and CIFAR-100 . We compare the obtained by CLAW to six different state-of-the-art continual learning algorithms: the VCL algorithm (original form and one with a coreset), the elastic weight consolidation (EWC) algorithm , the progress and compress (P&C) algorithm, the reinforced continual learning (RCL) algorithm , the one referred to as functional regularisation for continual learning (FRCL) using Gaussian processes and the learn-to-grow (LTG) algorithm (b). Our main metric is the all-important classification accuracy. We consider six continual learning experiments, based on the MNIST, notMNIST, Fashion-MNIST, Omniglot and CIFAR-100 datasets. The introduced CLAW is compared to two VCL versions: VCL with no coreset and VCL with a 200-point coreset assembled by the K-center method , EWC, P&C, RCL, FRCL (its TR version) and LTG 4. All the reported classification accuracy values reflect the average classification accuracy over all tasks the learner has trained on so far. More specifically, assume that the continual learner has just finished training on a task t, then the reported classification accuracy at time t is the average accuracy value obtained from testing on equally sized sets each belonging to one of the tasks 1, 2,..., t. For all the classification experiments, statistics reported are averages of ten repetitions. Statistical significance and standard error of the average classification accuracy obtained after completing the last two tasks of each experiment are displayed in Section E in the Appendix. As can be seen in Figure 1, CLAW achieves state-of-the-art classification accuracy in all the six experiments. The minibatch size is 128 for Split MNIST and 256 for all the other experiments. More detailed descriptions of the of every experiment are given next: Permuted MNIST Using MNIST, Permuted MNIST is a standard continual learning benchmark (a; ;). For each task t, the corresponding dataset is formed by performing a fixed random permutation process on labeled MNIST images. This random permutation is unique per task, i.e. it differs for each task. For the hyperparameter λ of EWC, which controls the overall contribution from previous data, we experimented with two values, λ = 1 and λ = 100. We report the latter since it has always outperformed EWC with λ = 1 in this experiment. EWC with λ = 100 has also previously produced the best EWC classification . In this experiment, fully connected single-head networks with two hidden layers are used. There are 100 hidden units in each layer, with ReLU activations. Adam is the optimiser used in the 6 experiments with η = 0.001, β 1 = 0.9 and β 2 = 0.999. Further experimental details are given in Section C in the Appendix. Results of the accumulated classification accuracy, averaged over tasks, on a test set are displayed in Figure 1a. After 10 tasks, CLAW achieves significantly (check the Appendix) higher classification than all the competitors. Split MNIST In this MNIST based experiment, five binary classification tasks are processed in the following sequence: 0/1, 2/3, 4/5, 6/7, and 8/9 . The architecture used consists of fully connected multi-head networks with two hidden layers, each consisting of 256 hidden units with ReLU activations. As can be seen in Figure 1b, CLAW achieves the highest classification accuracy. Split Fashion-MNIST Fashion-MNIST is a dataset whose size is the same as MNIST but it is based on different (and more challenging) 10 classes. The five binary classification tasks here are: T-shirt/Trouser, Pullover/Dress, Coat/Sandals, Shirt/Sneaker, and Bag/Ankle boots. The architecture used is the same as in Split notMNIST. In most of the continual learning tasks (including the more significant, later ones) CLAW achieves a clear classification improvement (Figure 1d). Omniglot This is a sequential learning task of handwritten characters of 50 alphabets (a total of over 1,600 characters with 20 examples each) belonging to the Omniglot dataset . We follow the same way via which this task has been used in continual learning before ); handwritten characters from each alphabet constitute a separate task. We thus have 50 tasks, which also allows to evaluate the scalability of the frameworks in comparison. The model used is a CNN. To deal with the convolutions in CLAW, we used the idea proposed and referred to as the local reparameterisation trick by Kingma et al. (2014;, where a single global parameter is employed per neuron activation in the variational distribution, rather than employing parameters for every constituent weight element 5 . Further details about the CNN used are given in Section C. The automatically adaptable CLAW achieves better classification accuracy (Figure 1e). This dataset consists of 60,000 colour images of size 32 × 32. It contains 100 classes, with 600 images per class. We use a split version CIFAR-100. Similar to , we perform a 20-task experiment with a disjoint subset of five classes per task. CLAW achieves significantly higher classification accuracy (Figure 1f) -also higher than the previous state of the art on CIFAR-100 by. Details of the used CNN are in Section C. A that can be taken from Figure 1 (a-f) is that CLAW consistently achieves state-of-the-art (in all the 6 experiments). It can also be seen that CLAW scales well. For instance, the difference between CLAW and the best competitor is more significant with Split notMNIST than it is with the first two experiments, which are based on the smaller and less challenging MNIST. Also, CLAW achieves good with Omniglot and CIFAR-100. To assess catastrophic forgetting, we show how the accuracy on the initial task varies over the course of the training procedure on the remaining tasks. Since Omniglot (and CIFAR-100) contain a larger number of tasks: 50 tasks, i.e. 49 remaining tasks after the initial task, this setting is more relevant for Omniglot and CIFAR-100. We nonetheless display the for Split MNIST, Split notMNIST, Split Fashion-MNIST, Omniglot and CIFAR-100. As can be seen in Figure 2, CLAW (at times jointly) achieves state-of-the-art performance retention degrees. Among the competitors, P&C and LTG also achieve high performance retention degrees. An empirical that can be made out of this and the previous experiment, is that CLAW achieves better overall continual learning , partially thanks to the way it addresses catastrophic forgetting. The idea of adapting the architecture by adapting the contributions of neurons of each layer also seems to be working well with datasets like Omniglot and CIFAR-100, giving directions for imminent future work where CLAW can be extended for other application areas based on CNNs. The purpose of this experiment is to assess the impact of learning previous tasks on the current task. In other words, we want to evaluate whether an algorithm avoids negative transfer, by evaluating the relative performance achieved on a unique task after learning a varying number of previous tasks. From Figure 3, we can see that CLAW achieves state-of-the-art in 4 out of the 5 experiments (at par in the fifth) in terms of avoiding negative transfer. We introduced a continual learning framework which learns how to adapt its architecture from the tasks and data at hand, based on variational inference. Rather than rigidly dividing the architecture into shared and task-specific parts, our approach adapts the contributions of each neuron. We achieve The impact of learning previous tasks on a specific task (the last task) is inspected and used as a proxy for evaluating forward transfer. This is performed by evaluating the relative performance achieved on a unique task after learning a varying number of previous tasks. This means that the value at x-axis = 1 refers to the learning accuracy of the last task after having learnt solely one task (only itself), the value at 2 refers to the learning accuracy of the last task after having learnt two tasks (an additional previous task), etc. Overall, CLAW achieves state-of-the-art in 4 out of the 5 experiments (at par in the fifth) in terms of avoiding negative transfer. Best viewed in colour. that without having to expand the architecture with new layers or new neurons. Results of six different experiments on five datasets demonstrate the strong empirical performance of the introduced framework, in terms of the average overall continual learning accuracy and forward transfer, and also in terms of effectively alleviating catastrophic forgetting. We begin by briefly summarising the contents of the Appendix below: • Related works are described in Section A, followed by a brief discussion on the potential applicability of CLAW to another continual learning (CL) framework in Section A.1. • In Section E, we provide the statistical significance and standard error of the average classification accuracy obtained after completing the last two tasks from each experiment. • Further experimental details are given in Section C. • In Section D and Figures 4-8, we display the of performed ablations which manifest the relevance of each adaptation parameter. A complementary approach to CLAW, which could be combined with it, is the regularisation-based approach to balance adaptability with catastrophic forgetting: a level of stability is kept via protecting parameters that greatly influence the prediction against radical changes, while allowing the rest of the parameters to change without restriction . In , the regulariser is based on synapses where an importance measure is locally computed at each synapse during training, based on their respective contributions to the change in the global loss. During a task change, the less important synapses are given the freedom to change whereas catastrophic forgetting is avoided by preventing the important synapses from changing . The elastic weight consolidation (EWC) algorithm, introduced by , is a seminal example of this approach where a quadratic penalty is imposed on the difference between parameter values of the old and new tasks. One limitation of EWC, which is rather alleviated by using minibatch or stochastic estimates, appears when the output space is not low-dimensional, since the diagonal of the Fisher information matrix over parameters of the old task must be computed, which requires a summation over all possible output labels (; ; . In addition, the regularisation term involves a sum over all previous tasks with a term from each and a hand-tuned hyperparameter that alters the weight given to it. The accumulation of this leads to a lot of hand-tuning. The work in is based on penalising confident fitting to the uncertain knowledge by a maximum entropy regulariser. Another seminal algorithm based on regularisation, which can be applied to any model, is variational continual learning (VCL) which formulates CL as a sequential approximate (variational) inference problem. However, VCL has only been applied to simple architectures, not involving any automatic model building or adaptation. The framework in incrementally matches the moments of the posterior of a Bayesian neural network that has been trained on the first and then the second task, and so on. Other algorithms pursue regularisation approaches based on sparsity (; . For example, the work in (c) encourages sparsity on the neuron activations to alleviate catastrophic forgetting. The l 2 distance between the top hidden activations of the old and new tasks is used for regularisation in . This approach has achieved good , but is computationally expensive due to the necessity of computing at least a forward pass for every new data point through the network representing the old task . Other regularisation-based continual learning algorithms include . Another approach is the architecture-based one where the principal aim is to administer both the stability and adaptation issues via dividing the architecture into reusable parts that are less prone to changes, and other parts especially devoted to individual tasks (b; ; ; ; ; a;). To learn a new task in the work by Rusu et al. (2016a), the whole network from the previous task is first copied then augmented with a new part of the architecture. Although this is effective in eradicating catastrophic forgetting, there is a clear scalability issue since the architecture growth can be prohibitively high, especially with an increasing number of tasks. The work introduced in (b) bases its continual learning on neural architecture search, whereas the representation in is optimised such that online updates minimize the error on all samples while limiting forgetting. The framework proposed by interestingly aims at solving this neural architecture structure learning problem, while balancing the tradeoff between adaptation and stability, via designed reinforcement learning (RL) strategies. When facing a new task, the optimal number of neurons and filters to add to each layer is cast as a combinatorial optimisation problem solved by an RL strategy whose reward signal is a function of validation accuracy and network complexity. Another RL based framework is the one presented by where catastrophic forgetting is mitigated at multiple time scales via RL agents with a synaptic model inspired by neuroscience. Bottom layers (those near the input) are generally shared among the different tasks, while layers near the output are task-specific. Since the model structure is usually divided a priori and no automatic architecture learning nor adaptation takes place, alteration on the shared layers can still cause performance loss on earlier tasks due to forgetting . A clipped version of maxout networks is developed in where parameters are partially shared among examples. The method in is based a dynamic network expansion accomplished by a generative adversarial network. The memory-based approach, which is the third influential approach to address the adaptationcatastrophic forgetting tradeoff, relies on episodic memory to store data (or pseudodata) from previous tasks (; ; ; ;). A major limitation of the memory-based approach is that data from previous tasks may not be available in all real-world problems (; . Another limitation is the overhead ing from the memory requirements, e.g. storage, replay, etc. In addition, the optimisation required to select the best observation to replay for future tasks is a source of further overhead . In addition to the explicit replay form, some works have been based on generative replay (; ; ; ; ; ; van de ;). train a deep generative model based on generative adversarial networks (b;) to mimic past data. This mitigates the aforementioned problem, albeit at the added cost of the training of the generative model and sharing its parameters. Alleviating catastrophic forgetting via replay mechanisms has also been adopted in reinforcement learning, e.g. . A similar approach was introduced by where gradients of the previous task (rather than data examples) are stored so that a trust region consisting of gradients of all previous tasks can be formed to reduce forgetting. Other algorithms based on replay mechanisms include (a; . Equivalent tradeoffs to the one between adaptation and stability can be found in the literature since the work in , in which a balance was needed to resolve the stabilityplasticity dilemma, where the latter refers to the ability to rapidly adapt to new tasks. The works introduced in (; shed light on the tradeoff between adaptation and stability, where they explore measures of intransigence and forgetting. The former refers to the inability to adapt to new tasks and data, whereas an increase in the latter clearly signifies an instability problem. Other recent works tackling the same tradeoff include where the transfer-interference (interference is catastrophic forgetting) tradeoff is optimised for the sake of maximising transfer and minimising interference by an algorithm based on experience replay and meta-learning. Other recent algorithms include the ORACLE algorithm by , which addresses the sensitivity of a continual learner to the order of tasks it encounters by establishing an order robust learner that represents the parameters of each task as a sum of task-shared and task-specific parameters. The algorithm in achieves functional regularisation by performing approximate inference over the function (instead of parameter) space. They use a Gaussian process obtained by assuming the weights of the last neural network layer to be Gaussian distributed. Our model is also related to the multi-task learning approach (; ; ;). As mentioned in the main document, ideas of the proposed CLAW can be applied to continual learning frameworks other than VCL. The latter is more relevant for the inference part of CLAW since both are based on variational inference. As per the modeling ideas, e.g. the binary adaptation parameter depicting whether or not to adapt, and the maximum allowed adaptation, these can be integrated within other continual learning frameworks. For example, the algorithm in utilises reinforcement learning to adaptively expand the network. The optimal number of nodes and filters to be added is cast as a combinatorial optimisation problem. In CLAW, we do not expand the network. As such, an extension of the work in can be inspired by CLAW where not only the number of nodes and filters to be added is decided for each task, but also a soft and more general version where an adaptation based on the same network size is performed such that the network expansion needed in can be further moderated. In this section, we provide information about the statistical significance and standard error of CLAW and the competing continual learning frameworks. In Table 1, we list the average accuracy values (Figure 1 in the main document) obtained after completing the last two tasks from each of the six experiments. A bold entry in Table 1 denotes that the classification accuracy of an algorithm is significantly higher than its competitors. Significance are identified using a paired t-test with p = 0.05. Each average accuracy value is followed by the corresponding standard error. Average classification accuracy ing from CLAW is significantly higher than its competitors on the 6 experiments. 99.2 ± 0.2 % 93. 98.7 ± 0.3 % 95.8 ± 0.4 % 96.9 ± 0.5 % 92.9 ± 0.4 % 97.8 ± 0.4 % 97.7 ± 0.2 % 96.1 ± 0.6 % 97.8 ± 0.3 % Split notMNIST (task 5) 98.4 ± 0.2 % 92.1 ± 0.3 % 96.0 ± 0.3 % 92.3 ± 0.4 % 96.9 ± 0.5 % 97.3 ± 0.5 % 95.2 ± 0.7 % 97.4 ± 0.3 % Split Fashion-MNIST (task 4) 93.2 ± 0.2 % 90.0 ± 0.3 % 90.7 ± 0.2 % 89.4 ± 0.4 % 91.4 ± 0.3 % 91.1 ± 0.3 % 90.4 ± 0.2 % 92.5 ± 0.4 % Split Fashion-MNIST (task 5) 92.5 ± 0.2 % 88.0 ± 0.2 % 88.5 ± 0.4 % 87.6 ± 0.3 % 90.8 ± 0.2 % 89.7 ± 0.4 % 87.7 ± 0.4 % 91.1 ± 0.3 % Omniglot (task 49) 84.5 ± 0.2 % 81.1 ± 0.3 % 81.8 ± 0.3 % 78.2 ± 0.3 % 82.8 ± 0.2 % 80.1 ± 0.4 % 79.9 ± 0.3 % 83.6 ± 0.3 % Omniglot (task 50) 84.6 ± 0.3 % 80.7 ± 0.3 % 81.1 ± 0.4 % 77.3 ± 0.3 % 82.7 ± 0.3 % 80.2 ± 0.4 % 79.8 ± 0.5 % 83.5 ± 0.3 % CIFAR-100 (task 19) 95.6 ± 0.3 % 78.7 ± 0.4 % 80.8 ± 0.3 % 63.1 ± 0.5 % 68.3 ± 0.6 % 63.7 ± 0.6 % 77.4 ± 0.7 % 86.2 ± 0.4 % CIFAR-100 (task 20) 95.6 ± 0.3 % 77.2 ± 0.4 % 79.9 ± 0.4 % 62.4 ± 0.4 % 65.5 ± 0.6 % 60.4 ± 0.6 % 76.8 ± 0.6 % 85.6 ± 0.5 % Here are some additional details about the datasets in use: The MNIST dataset is used in both the Permuted MNIST and Split MNIST experiments. The MNIST (Mixed National Institute of Standards and Technology) dataset is a handwritten digit dataset. Each MNIST image consists of 28 × 28 pixels, which is also the pixel size of the notMNIST and Fashion-MNIST datasets. The MNIST dataset contains a training set of 60,000 instances and a test set of 10,000 instances. As mentioned in the main document, each experiment is repeated ten times. Data is randomly split into three partitions, training, validation and test. A portion of 60% of the data is reserved for training, 20% for validation and 20% for testing. Statistics reported are the averages of these ten repetitions. Number of epochs required per task to reach a saturation level for CLAW (and the bulk of the methods in comparison) was 10 epochs for all experiments except for Omniglot and CIFAR-100 (15 epochs). Used values of ω 1 and ω 2 are 0.05 and 0.02, respectively. For Omniglot, we used a network similar to the one used in, which consists of 4 blocks of 3 × 3 convolutions with 64 filters, followed by a ReLU and a 2 × 2 max-pooling. The same CNN is used for CIFAR-100. CLAW achieves clearly higher classification accuracy on both Omniglot and CIFAR-100 (Figures 1e and 1f). The plots displayed in this section empirically demonstrate how important the main adaptation parameters are in achieving the classification performance levels reached by CLAW. In each of the Figures 4-9, the classification performance of CLAW is compared to the following three cases: 1) when the parameter controlling the maximum degree of adaptation is not learnt in a multi-task fashion, i.e. when the respective general value s i,j is used instead of s i,j,t. 2) when adaptation always happens, i.e. the binary variable denoting the adaptation decision is always activated. 3) when adaptation never takes place. The differences in classification accuracy between CLAW and each of the other three plots in Figures 4-9 empirically demonstrate the relevance of each adaptation parameter.

Represent the short summary for finding the corresponding scientific article (long).

A continual learning framework which learns to automatically adapt its architecture based on a proposed variational inference algorithm.

represent the short summary for finding the corresponding scientific article (long)

To give you a sense - "We present novel distillation techniques that enable training student models with different vocabularies and compress BERT by 60x with minor performance drop." should be close to "Pre-trained deep neural network language models such as ELMo, GPT, BERT and XLNet have recently achieved state-of-the-art performance on a variety of language understanding tasks. However, their size makes them impractical for a number of scenarios, especially on mobile and edge devices. In particular, the input word embedding matrix accounts for a significant proportion of the model's memory footprint, due to the large input vocabulary and embedding dimensions. Knowledge distillation techniques have had success at compressing large neural network models, but they are ineffective at yielding student models with vocabularies different from the original teacher models. We introduce a novel knowledge distillation technique for training a student model with a significantly smaller vocabulary as well as lower embedding and hidden state dimensions. Specifically, we employ a dual-training mechanism that trains the teacher and student models simultaneously to obtain optimal word embeddings for the student vocabulary. We combine this approach with learning shared projection matrices that transfer layer-wise knowledge from the teacher model to the student model. Our method is able to compress the BERT-BASE model by more than 60x, with only a minor drop in downstream task metrics, ing in a language model with a footprint of under 7MB. Experimental also demonstrate higher compression efficiency and accuracy when compared with other state-of-the-art compression techniques. Recently, contextual-aware language models such as ELMo , GPT , BERT and XLNet have shown to greatly outperform traditional word embedding models including Word2Vec and GloVe in a variety of NLP tasks. These pre-trained language models, when finetuned on downstream language understanding tasks such as sentiment classification , natural language inference and reading comprehension , have achieved state-of-the-art performance. However, the large number of parameters in these models, often above hundreds of millions, makes it impossible to host them on resource-constrained tasks such as doing real-time inference on mobile and edge devices. Besides utilizing model quantization techniques which aim to reduce the floating-point accuracy of the parameters, significant recent research has focused on knowledge distillation techniques. Here, the goal is to train a small-footprint student model by borrowing knowledge, such as through a soft predicted label distribution, from a larger pre-trained teacher model. However, a significant bottleneck that has been overlooked by previous efforts is the input vocabulary size and its corresponding word embedding matrix, often accounting for a significant proportion of all model parameters. For instance, the embedding table of the BERT BASE model, comprising over 30K WordPiece tokens (b), accounts for over 21% of the model size. While there has been existing work on reducing NLP model vocabulary sizes , distillation techniques cannot utilize these, since they require the student and teacher models to share the same vocabulary and output space. This profoundly limits their potential to further reduce model sizes. We present two novel ideas to improve the effectiveness of knowledge distillation, in particular for BERT, with the focus on bringing down model sizes to as much as a few mega-bytes. Our model is among the first to propose to use a significantly smaller vocabulary for the student model learned during distillation. In addition, instead of distilling solely on the teacher model's final-layer outputs, our model leverages layer-wise teacher model parameters to directly optimize the parameters of the corresponding layers in the student model. Specifically, our contributions are: • Dual Training: Our teacher and student models have different vocabularies and incompatible tokenizations for the same sequence. To address this during distillation, we feed the teacher model a mix of teacher vocabulary-tokenized and student vocabulary-tokenized words within a single sequence. Coupled with the masked language modeling task, this encourages an implicit alignment of the teacher and student WordPiece embeddings, since the student vocabulary embedding may be used as context to predict a word tokenized by the teacher vocabulary and vice versa. • Shared Variable Projections: To minimize the loss of information from reducing the hidden state dimension, we introduce a separate loss to align the teacher and student models' trainable variables. This allows for more direct layer-wise transfer of knowledge to the student model. Using the combination of dual training and shared variable projections, we train a 12-layer highlycompressed student BERT model, achieving a maximum compression ratio of ∼61.94x (with 48 dimension size) compared to the teacher BERT BASE model. We conduct experiments for measuring both generalized language modeling perspective and for downstream tasks, demonstrating competitive performance with high compression ratios for both families of tasks. Research in neural network model compression has been concomitant with the rise in popularity of neural networks themselves, since these models have often been memory-intensive for the hardware of their time. Work in model compression for NLP applications falls broadly into four categories: matrix approximation, parameter pruning/sharing, weight quantization and knowledge distillation. A family of approaches seeks to compress the matrix parameters of the models by low-rank approximation i.e. the full-rank matrix parameter is approximated using multiple low-rank matrices, thereby reducing the effective number of model parameters. Another line of work explores parameter pruning and sharing-based methods (; ; ;), which explore the redundancy in model parameters and try to remove redundant weights as well as neurons, for a variety of neural network architectures. Model weight quantization techniques (; a;) focus on mapping model weights to lower-precision integers and floating-point numbers. These can be especially effective with hardware supporting efficient low-precision calculations. More recently, apply quantization to BERT-based transformer models. Knowledge distillation differs from the other discussed approaches: the smaller student model may be parametrized differently from the bigger teacher model, affording more modeling freedom. Teaching a student model to match the soft output label distributions from a larger model alongside the hard ground-truth distribution works well for many tasks, such as machine translation and language modeling . Not limited to the teacher model outputs, some approaches perform knowledge distillation via attention transfer , or via feature maps or intermediate model outputs (; ;). More relevant to current work, and employ variants of these techniques to BERT model compression by reducing the number of transformer layers. However, as explained before, these approaches are not immediately applicable to our setting due to incompatible teacher and student model vocabularies, and do not focus sufficiently on the embedding matrix size . Our knowledge distillation approach is centered around reducing the number of WordPiece tokens in the model vocabulary. In this section, we first discuss the rationale behind this reduction and the challenges it introduces, followed by our techniques, namely dual training and shared projection. (Right) A student BERT model trained from scratch with smaller vocab (5K) and hidden state dimension (e.g., 48). During distillation, the teacher model randomly selects a vocabulary to segment each input word. The red and green square nexts to the transformer layers indicate trainable parameters for both the student and teacher models -note that our student models have smaller model dimensions. The projection matrices U and V, shown as having representative shapes, are shared across all layers for model parameters that have the same dimensions. We follow the general knowledge distillation paradigm of training a small student model from a large teacher model. Our teacher model is a 12-layer uncased BERT BASE, trained with 30522 WordPiece tokens and 768-dimensional embeddings and hidden states. We denote the teacher model parameters by θ t. Our student model consists of an equal number of transformer layers with parameters denoted by θ s, but with a smaller vocabulary as well as embedding/hidden dimensions, illustrated in Figure 1. Using the same WordPiece algorithm and training corpus as BERT, we obtain a vocabulary of 4928 WordPieces, which we use for the student model. WordPiece tokens (b) are sub-word units obtained by applying a greedy segmentation algorithm to the training corpus: a desired number (say, D) of WordPieces are chosen such that the segmented corpus is minimal in the number of WordPieces used. A cursory look at both vocabularies reveals that 93.9% of the WordPieces in the student vocabulary also exist in the teacher vocabulary, suggesting room for a reduction in the WordPiece vocabulary size from 30K tokens. Since we seek to train a general-purpose student language model, we elect to reuse the teacher model's original training objective to optimize the student model, i.e., masked language modeling and next sentence prediction, before any fine-tuning. In the former task, words in context are randomly masked, and the language model needs to predict those words given the masked context. In the latter task, given a pair of sentences, the language model predicts whether the pair is consistent. However, since the student vocabulary is not a complete subset of the teacher vocabulary, the two vocabularies may tokenize the same words differently. As a , the outputs of the teacher and student model for the masked language modeling task may not align. Even with the high overlap between the two vocabularies, the need to train the student embedding from scratch, and the change in embedding dimension precludes existing knowledge distillation techniques, which rely on the alignment of both models' output spaces. As a , we explore two alternative approaches that enable implicit transfer of knowledge to the student model, which we describe below. During distillation, for a given training sequence input to the teacher model, we propose to mix the teacher and student vocabularies by randomly selecting (with a probability p DT, a hyperparameter) tokens from the sequence to segment using the student vocabulary, with the other tokens segmented using the teacher vocabulary. As illustrated in Figure 1, given the input context ['I', 'like', 'machine', 'learning'], the words'I' and'machine' are segmented using the teacher vocabulary (in green), while'like' and'learning' are segmented using the student vocabulary (in blue). Similar to cross-lingual training in , this encourages alignment of the representations for the same word as per the teacher and student vocabularies. This is effected through the masked language modeling task: the model now needs to learn to predict words from the student vocabulary using context words segmented using the teacher vocabulary, and vice versa. The expectation is that the student embeddings can be learned effectively this way from the teacher embeddings as well as model parameters θ t. Note that we perform dual training only for the teacher model inputs: the student model receives words segmented exclusively using the student vocabulary. Also, during masked language modeling, the model uses different softmax layers for the teacher and the student vocabularies depending on which one was used to segment the word in question. Relying solely on teacher model outputs to train the student model may not generalize well . Therefore, some approaches utilize and try to align the student model's intermediate predictions to those of the teacher . In our setting, however, since the student and teacher model output spaces are not identical, intermediate model outputs may prove hard to align. Instead, we seek to directly minimize the loss of information from the teacher model parameters θ t to the student parameters θ s with smaller dimensions. We achieve this by projecting the model parameters into the same space, to encourage alignment. More specifically, as in Figure 1, we project each trainable variable in θ t to the same shape as the corresponding variable in θ s. For example, for all the trainable variables θ t with shape 768×768, we learn two projection matrices U ∈ R d×768 and V ∈ R 768×d to project them into the corresponding space of the student model variable θ t, where d is the student model's hidden dimension. U and V are common to all BERT model parameters of that dimensionality; in addition, U and V are not needed for fine-tuning or inference after distillation. In order to align the student variable and the teacher variable's projection, we introduce a separate mean square error loss defined in Equation 1, where ↓ stands for down projection (since the projection is to a lower dimension). The above loss function aligns the trainable variables in the student space. Alternatively, we can project trainable variables in θ s to the same shape as in θ t. This way, the loss function in Equation 2, (↑ denotes up projection) can compare the trainable variables in the teacher space. 3.4 OPTIMIZATION OBJECTIVE Our final loss function includes, in addition to an optional projection loss, masked language modeling cross-entropy losses for the student as well as the teacher models, since the teacher model is trained with dual-vocabulary inputs and is not static. P (y i = c|θ s) and P (y i = c|θ t) denote the stu-dent and teacher model prediction probabilities for class c respectively, and 1 denotes an indicator function. Equations 3 and 4 below define the final loss L f inal, where is a hyperparameter. To evaluate our knowledge distillation approach, we design two classes of experiments. First, we evaluate the distilled student language models using the masked word prediction task on an unseen evaluation corpus, for an explicit evaluation of the language model. Second, we fine-tune the language model by adding a task-specific affine layer on top of the student language model outputs, on a suite of downstream sentence and sentence pair classification tasks. This is meant to be an implicit evaluation of the quality of the representations learned by the student language model. We describe these experiments, along with details on training, implementation and our baselines below. During the distillation of the teacher BERT model to train the student BERT language model, we utilize the same corpus as was used to train the teacher i.e. BooksCorpus and English Wikipedia, with whitespaces used tokenize the text into words. We only use the masked language modeling task to calculate the overall distillation loss from Section 3.3, since the next sentence prediction loss hurt performance slightly. Dual training is enabled for teacher model inputs, with p DT, the probability of segmenting a teacher model input word using the student vocabulary, set to 0.5. For experiments including shared projection, the projection matrices U and V utilized Xavier initialization . The loss weight coefficient is set to 1 after tuning. It is worth noting that in contrast to a number of existing approaches, we directly distill the teacher BERT language model, not yet fine-tuned on a downstream task, to obtain a student language model that is task-agnostic. For downstream tasks, we fine-tune this distilled student language model. Distillation is carried out on Cloud TPUs in a 4x4 pod configuration 1 (32 TPU cores overall). We optimized the loss using LAMB for 250K steps, with a learning rate of 0.00125 and batch size of 4096. Depending on the student model dimension, training took between 2-4 days. We evaluate three variants of our distilled student models: with only dual training of the teacher and student vocabularies (DualTrain) and with dual training along with down-projection (DualTrain + SharedProjDown) or up-projection (DualTrain + SharedProjUp) of the teacher model parameters. For each of these configurations, we train student models with embedding and hidden dimensions 48, 96 and 192, for 9 total variants, each using a compact 5K-WordPiece vocabulary. Table 1 presents some statistics on these models' sizes: our smallest model contains two orders of magnitude fewer parameters, and requires only 1% floating-point operations when compared to the BERT BASE model. For the language modeling evaluation, we also evaluate a baseline without knowledge distillation (termed NoKD), with a model parameterized identically to the distilled student models but trained directly on the teacher model objective from scratch. For downstream tasks, we compare with NoKD as well as Patient Knowledge Distillation (PKD) from , who distill the 12-layer BERT BASE model into 3 and 6-layer BERT models by using the teacher model's hidden states. Table 1: A summary of our student models' sizes compared to BERT BASE. #Params indicates the number of parameters in the student model, model size is measured in megabytes, and FLOPS ratio measures the relative ratio of floating point operations required for inference on the model. For explicit evaluation of the generalized language perspective of the distilled student language models, we use the Reddit dataset 2 to measure word mask prediction accuracy of the student models, since the language used on Reddit is different from that in the training corpora. The dataset is preprocessed similarly to the training corpora, except we do not need to tokenize it using the teacher vocabulary, since we only run and evaluate the student models. For implicit evaluation on downstream language understanding tasks, we fine-tune and evaluate the distilled student models on three tasks from the GLUE benchmark : • Stanford Sentiment Treebank (SST-2) , a two-way sentence sentiment classification task with 67K training instances, • Microsoft Research Paraphrase Corpus (MRPC) , a two-way sentence pair classification task to identify paraphrases, with 3.7K training instances, and • Multi-Genre Natural Language Inference (MNLI) , a three-way sentence pair classification task with 393K training instances, to identify premise-hypothesis relations. There are separate development and test sets for genre-matched and genre-mismatched premisehypothesis pairs; we tune our models solely on the genre-matched development set. For all downstream task evaluations, we fine-tune for 10 epochs using LAMB with a learning rate of 0.0002 and batch size of 32. Since our language models are trained with a maximum sequence length of 128 tokens, we do not evaluate on reading comprehension datasets such as SQuAD or RACE , which require models supporting longer sequences. Table 2: Masked language modeling task accuracy for the distilled student models and a fine-tunefrom-scratch baseline. We observe consistently better performance for our proposed approaches. Table 2 contains masked word prediction accuracy figures for the different models and the NoKD baseline. We observe that dual training significantly improves over the baseline for all model dimensions, and that both shared projection losses added to dual training further improve the word prediction accuracy. It is interesting to note that for all model dimensions, SharedProjUp projecting into the teacher space outperforms SharedProjDown, significantly so for dimension 48. Expectedly, there is a noticeable performance drop going from 192 to 96 to 48-dimensional hidden state models. Table 3: Results of the distilled models, the teacher model and baselines on the downstream language understanding task test sets, obtained from the GLUE server, along with the size parameters and compression ratios of the respective models compared to the teacher BERT BASE. MNLI-m and MNLI-mm refer to the genre-matched and genre-mismatched test sets for MNLI. Note that because of the differing teacher and student model vocabularies, masked word prediction accuracy for the teacher BERT BASE model is not directly comparable with the student models. Table 3 shows on the downstream language understanding tasks, as well as model sizes, for our approaches, the BERT BASE teacher model, and the PKD and NoKD baselines. We note that models trained with our proposed approaches perform strongly and consistently improve upon the identically parametrized NoKD baselines, indicating that the dual training and shared projection techniques are effective, without incurring significant losses against the BERT BASE teacher model. Comparing with the PKD baseline, our 192-dimensional models, achieving a higher compression rate than either of the PKD models, perform better than the 3-layer PKD baseline and are competitive with the larger 6-layer baseline on task accuracy while being nearly 5 times as small. Another observation we make is that the performance drop from 192-dimensional to 96-dimensional models is minimal (less than 2% for most tasks). For the MRPC task, in fact, the 96-dimensional model trained with dual training achieves an accuracy of 80.5%, which is higher than even the PKD 6-layer baseline with nearly 12 times as many parameters. Finally, our highly-compressed 48-dimensional models also perform respectably: the best 48-dimensional models are in a similar performance bracket as the 3-layer PKD model, a model 25 times larger by memory footprint. Shared projections and model performance: We see that for downstream task performance, dual training still consistently improves upon the direct fine-tuning approach for virtually all experiments. The effect of shared variable projection, however, is less pronounced, with consistent improvements visible only for MRPC and for the 48-dimensional models i.e. the smallest dataset and models respectively in our experiments. This aligns with our intuition for variable projection as a more direct way to provide a training signal from the teacher model internals, which can assume more importance for a low-data or small-model scenario. However, for larger models and more data, the linear projection of parameters may be reducing the degrees of freedom available to the model, since linear projection is a fairly simple function to align the teacher and student parameter spaces. A related comparison of interest is between up-projection and down-projection of the model variables: we note up-projection does visibly better on the language modeling task and slightly better on the downstream tasks. The parameters of a well-trained teacher model represent a high-quality local minimum in the teacher space, which may be easier to search for during up-projection. Vocabulary size tradeoffs: Issues with input vocabulary size are peculiar to problems in natural language processing: they do not always apply to other areas such as computer vision, where a small fixed number of symbols can encode most inputs. There has been some work on reducing input vocabulary sizes for NLP, but typically not targeting model compression. One concern with reducing the vocabularies of NLP models is it pushes the average tokenized sequence lengths up, making model training harder. In this work, however, we consider classification tasks on shorter texts, which are not as affected by input sequence lengths as, say, tasks such as machine translation are. Furthermore, many real-world applications revolve around short text inputs, which is why a better trade-off between vocabulary size and sequence lengths may be worthwhile for such applications. Order of distillation and fine-tuning: Most of the existing work on distilling language models such as BERT and reporting on downstream tasks, including some of the baselines in this work, first fine-tune a teacher model on the downstream tasks, and then distill this model. Our goal in this work, however, is to explore the limits to which BERT's language modeling capacity itself, and how much of it is driven by its large WordPiece vocabulary. We leave experiments on distilling fine-tuned teacher models, potentially yielding better on downstream tasks, to future work. We proposed two novel ideas to improve the effectiveness of knowledge distillation for BERT, focusing on using a significantly smaller vocabulary, as well as smaller embedding and hidden dimensions for the student BERT language models. Our dual training mechanism encourages implicit alignment of the teacher and student WordPiece embeddings, and shared variable projection allows for the faster and direct layer-wise transfer of knowledge to the student BERT model. Combining the two techniques, we trained a series of highly-compressed 12-layer student BERT models. Experiments on these models, to evaluate both generalized language perspective and four standardized downstream tasks, demonstrate the effectiveness of our proposed methods on both model accuracy and compression efficiency. One future direction of interest is to combine our approach with existing work to reduce the number of layers in the student models and explore other approaches such as low-rank matrix factorization to transfer model parameters from the teacher space to the student space. In addition, taking into account the frequency distribution of the WordPiece tokens while training embeddings may help optimize the model size further."

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.


Question: Represent the article for finding a single-sentence TLDR

Question: High-dimensional data often lie in or close to low-dimensional subspaces. Sparse subspace clustering methods with sparsity induced by L0-norm, such as L0-Sparse Subspace Clustering (L0-SSC), are demonstrated to be more effective than its L1 counterpart such as Sparse Subspace Clustering (SSC). However, these L0-norm based subspace clustering methods are restricted to clean data that lie exactly in subspaces. Real data often suffer from noise and they may lie close to subspaces. We propose noisy L0-SSC to handle noisy data so as to improve the robustness. We show that the optimal solution to the optimization problem of noisy L0-SSC achieves subspace detection property (SDP), a key element with which data from different subspaces are separated, under deterministic and randomized models. Our provide theoretical guarantee on the correctness of noisy L0-SSC in terms of SDP on noisy data. We further propose Noisy-DR-L0-SSC which provably recovers the subspaces on dimensionality reduced data. Noisy-DR-L0-SSC first projects the data onto a lower dimensional space by linear transformation, then performs noisy L0-SSC on the dimensionality reduced data so as to improve the efficiency. The experimental demonstrate the effectiveness of noisy L0-SSC and Noisy-DR-L0-SSC. High-dimensional data often lie in or close to low-dimensional subspaces. Sparse subspace clustering methods with sparsity induced by 0 -norm, such as 0 -Sparse Subspace Clustering (0 -SSC) , are demonstrated to be more effective than its 1 counterpart such as Sparse Subspace Clustering (SSC). However, these 0 -norm based subspace clustering methods are restricted to clean data that lie exactly in subspaces. Real data often suffer from noise and they may lie close to subspaces. We propose noisy 0 -SSC to handle noisy data so as to improve the robustness. We show that the optimal solution to the optimization problem of noisy 0 -SSC achieves subspace detection property (SDP), a key element with which data from different subspaces are separated, under deterministic and randomized models. Our provide theoretical guarantee on the correctness of noisy 0 -SSC in terms of SDP on noisy data. We further propose Noisy-DR-0 -SSC which provably recovers the subspaces on dimensionality reduced data. Noisy-DR-0 -SSC first projects the data onto a lower dimensional space by linear transformation, then performs noisy 0 -SSC on the dimensionality reduced data so as to improve the efficiency. The experimental demonstrate the effectiveness of noisy 0 -SSC and Noisy-DR-0 -SSC. Clustering is an important unsupervised learning procedure for analyzing a broad class of scientific data in biology, medicine, psychology and chemistry. On the other hand, high-dimensional data, such as facial images and gene expression data, often lie in low-dimensional subspaces in many cases, and clustering in accordance to the underlying subspace structure is particularly important. For example, the well-known Principal Component Analysis (PCA) works perfectly if the data are distributed around a single subspace. The subspace learning literature develops more general methods that recover multiple subspaces in the original data, and subspace clustering algorithms aim to partition the data such that data belonging to the same subspace are identified as one cluster. Among various subspace clustering algorithms, the ones that employ sparsity prior, such as Sparse Subspace Clustering (SSC) and 0 -Sparse Subspace Clustering (0 -SSC) , have been proven to be effective in separating the data in accordance with the subspaces that the data lie in under certain assumptions. Sparse subspace clustering methods construct the sparse similarity matrix by sparse representation of the data. Subspace detection property (SDP) defined in Section 4.1 ensures that the similarity between data from different subspaces vanishes in the sparse similarity matrix, and applying spectral clustering on such sparse similarity matrix leads to compelling clustering performance. prove that when the subspaces are independent or disjoint, SDP can be satisfied by solving the canonical sparse linear representation problem using data as the dictionary, under certain conditions on the rank, or singular value of the data matrix and the principle angle between the subspaces. SSC has been successfully applied to a novel deep neural network architecture, leading to the first deep sparse subspace clustering method. Under the independence assumption on the subspaces, low rank representation Liu et al. (2010; is also proposed to recover the subspace structures. Relaxing the assumptions on the subspaces to allowing overlapping subspaces, the and the LowRank Sparse Subspace Clustering achieve subspace detection property with high probability. The geometric analysis in shows the theoretical on subspace recovery by SSC. In the following text, we use the term SSC or 1 -SSC exchangeably to indicate the Sparse Subspace Clustering method in . Real data often suffer from noise. Noisy SSC proposed in handles noisy data that lie close to disjoint or overlapping subspaces. While 0 - has guaranteed clustering correctness via subspace detection property under much milder assumptions than previous subspace clustering methods including SSC, it assumes that the observed data lie in exactly in the subspaces and does not handle noisy data. In this paper, we present noisy 0 -SSC, which enhances 0 -SSC by theoretical guarantee on the correctness of clustering on noisy data. It should be emphasized that while 0 -SSC on clean data empirically adopts a form of optimization problem robust to noise, it lacks theoretical analysis on the correctness of 0 -SSC on noisy data. In this paper, the correctness of noisy 0 -SSC on noisy data in terms of the subspace detection property is established. Our analysis is under both deterministic model and randomized models, which is also the model employed in the geometric analysis of . Our randomized analysis demonstrates potential advantage of noisy 0 -SSC over its 1 counterpart as more general assumption on data distribution can be adopted. Moreover, we present Noisy Dimensionality Reduced 0 -Sparse Subspace Clustering (Noisy-DR-0 -SSC), an efficient version of noisy 0 -SSC which also enjoys robustness to noise. Noisy-DR-0 -SSC first projects the data onto a lower dimensional space by random projection, then performs noisy 0 -SSC on the dimensionality reduced data. Noisy-DR-0 -SSC provably recovers the underlying subspace structure in the original data from the dimensionality reduced data under deterministic model. Experimental demonstrate the effectiveness of both noisy 0 -SSC and Noisy-DR-0 -SSC. We use bold letters for matrices and vectors, and regular lower letter for scalars throughout this paper. The bold letter with superscript indicates the corresponding column of a matrix, e.g. A i is the i-th column of matrix A, and the bold letter with subscript indicates the corresponding element of a matrix or vector. · F and · p denote the Frobenius norm and the vector p -norm or the matrix p-norm, and diag(·) indicates the diagonal elements of a matrix. H T ⊆ R d indicates the subspace spanned by the columns of T, and A I denotes a submatrix of A whose columns correspond to the nonzero elements of I (or with indices in I without confusion). σ t (·) denotes the t-th largest singular value of a matrix, and σ min (·) indicates the smallest singular value of a matrix. supp(·) is the support of a vector, P S is an operator indicating projection onto the subspace S. We hereby introduce the notations for subspace clustering on noisy data considered in this paper. The uncorrupted data matrix is denoted by Y = [y 1, . . ., y n] ∈ R d×n, where d is the dimensionality and n is the size of the data. The uncorrupted data Y lie in a union of is the additive noise. x i = y i + n i is the noisy data point that is corrupted by the noise n i. n k = n, and denote the corresponding columns in X by X (k). The data X are normalized such that each column has unit 2 -norm in our deterministic analysis. We consider deterministic noise model where the noise Z is fixed and max n i ≤ δ. Note that our analysis can be extended to a random noise model which is common and also considered by noisy , and the random noise model assumes that columns of Z are sampled i.i.d. and max n i ≤ δ with high probability. Note that such random noise model does not require spherical symmetric noise as that in. 0 - proposes to solve the following 0 sparse representation problem and it proves that the subspace detection property defined in Definition 1 is satisfied with the globally optimal solution to. We resort to solve the 0 regularized sparse approximation problem below to handle noisy data for 0 -SSC, which is the optimization problem of noisy 0 -SSC: The definition of subspace detection property for noisy 0 -SSC and noiseless 0 -SSC, i.e. 0 -SSC on noiseless data, is defined in Definition 1 below. Definition 1. (Subspace detection property for noisy and noiseless 0 -SSC) Let Z * be the optimal solution to. The subspaces {S k} K k=1 and the data X satisfy subspace detection property for noisy 0 -SSC if Z i is a nonzero vector, and nonzero elements of Z i correspond to the columns of X from the same subspace as y i for all 1 ≤ i ≤ n. Similarly, in the noiseless setting where X = Y, let Z * be the optimal solution to. The subspaces {S k} K k=1 and the data X satisfy the subspace detection property for noiseless 0 -SSC if Z i is a nonzero vector, and nonzero elements of Z i correspond to the columns of X that from the same subspace as y i for all 1 ≤ i ≤ n. We say that subspace detection property holds for x i if nonzero elements of Z * i correspond to the data that lie in the same subspace as y i, for either noisy 0 -SSC or noiseless 0 -SSC. Similar to , we introduce the deterministic, semi-random and fullyrandom models for the analysis of noisy 0 -SSC. • Deterministic Model: the subspaces and the data in each subspace are fixed. • Semi-Random Model: the subspaces are fixed but the data are independent and identically distributed in each of the subspaces. • Fully-Random Model: both the subspaces and the data of each subspace are independent and identically distributed. The data in the above definitions refer to clean data without noise. We refer to semi-random model and fully-random model as randomized models in this paper. The theoretical on the subspace detection property for noisy 0 -SSC are presented in this section under deterministic model and randomized models. All the data Y are normalized to have unit norm for illustration purpose, so they lie on the surface of the sphere. S 1 and S 2 are two subspaces in the three-dimensional ambient space. The subspace spanned by y i ∈ S 1 and y j ∈ S 2 is an external subspace, and the intersection of this external subspace and S 1 is a dashed line y i OA. We introduce the definition of general position and external subspace before our analysis on noisy 0 -SSC. The assumption of general condition is rather mild. In fact, if the data points in X (k) are independently distributed according to any continuous distribution, then they almost surely in general position. Let the distance between a point x ∈ R d and a subspace S ⊆ R d be defined as d(x, S) = inf y∈S x− y 2, the definition of external subspaces is presented as follows. Figure 1 illustrates an example of external subspace. spanned by a set of linear independent points {y ij} L j=1 ⊆ Y is defined to be an external subspace of y if. The point y is said to be away from its external subspaces if are the set of all external subspaces of y of dimension no greater than d for y, i.e. }. All the data points in Y (k) are said to be away from the external subspaces if each of them is away from the its associated external spaces. Remark 1. (Subspace detection property holds for noiseless 0 -SSC under the deterministic model) It can be verified that the following statement is true. Under the deterministic model, suppose data is noiseless, If all the data points in Y (k) are away from the external subspaces for any 1 ≤ k ≤ K, then the subspace detection property for 0 -SSC holds with the optimal solution Z * to. To present our theoretical of the correctness of noisy 0 -SSC, we also need the definitions of the minimum restricted eigenvalue and the subspace separation margin, which are defined as follows. In the following analysis, we employ β to denote the sparse code of datum x i so that a simpler notation other than Z i is dedicated to our analysis. Definition 4. The minimum restricted eigenvalue of the uncorrupted data is defined as for r ≥ 1. In addition, the normalized minimum restricted eigenvalue of the uncorrupted data is defined byσ We have the following perturbation bound for the distance between a data point and the subspaces spanned by noisy and noiseless data, which is useful to establish the conditions when the subspace detection property holds for noisy 0 -SSC. Lemma 1. Let β ∈ R n and Y β has full column rank. Suppose δ <σ Y,r where r = β 0, then X β is a full column rank matrix, and The optimization problem of noisy 0 -SSC is separable. For each 1 ≤ i ≤ n, the optimization problem with respect to the sparse code of i-th data point is Lemma 2 shows that the optimal solution to the noisy 0 -SSC problem is also that to a 0 -minimization problem with tolerance to noise. Lemma 2. Let nonzero vector β * be the optimal solution to the noisy 0 -SSC problem for point then β * is the optimal solution to the following sparse approximation problem with the uncorrupted data as the dictionary: where c * Define B(x i, c 0) = {x : x − x i ≤ c 0} be the ball centered at x i with radius c 0. If B(x i, c 0) is away from the corresponding confusion area, i.e. all the external subspaces in H yi,d k, then subspace detection property holds with the solution to a proper sparse approximation problem where x i is approximated by the uncorrupted data, as shown in the following Lemma. Lemma 3. Suppose Y is in general position and Then the subspace detection property holds for x i with the optimal solution to the following sparse approximation problem, denoted by β *, i.e. nonzero elements of β * correspond to the columns of X from the same subspace as y i. Now we use the above to present the main on the correctness of noisy 0 -SSC. Theorem 1. (Subspace detection property holds for noisy 0 -SSC) Let nonzero vector β * be the optimal solution to the noisy 0 -SSC problem for point x i with β * 0 = r * > 1, and c * Then the subspace detection property holds for x i with β *. Here τ 0, τ 1,σ * Y and σ * X are defined in Lemma 2. Remark 2. When δ = 0 and there is no noise in the data X, the conditions for the correctness of noisy 0 -SSC in Theorem 1 almost reduce to that for noiseless 0 -SSC. To see this, the conditions are reduced to B(y i, c *) ∩ H = ∅, which are exactly the conditions required by noiseless 0 -SSC, namely data are away from the external subspaces by choosing λ → 0 and it follows that c * = 0. While Theorem 1 establishes geometric conditions under which the subspace detection property holds for noisy 0 -SSC, it can be seen that these conditions are often coupled with the optimal solution β * to the noisy 0 -SSC problem. In the following theorem, the correctness of noisy 0 -SSC is guaranteed in terms of λ, the weight for the 0 regularization term in, and the geometric conditions independent of the optimal solution to. Let M i > 0 be the minimum distance between y i ∈ S k and its external subspaces when y i is away from its external subspaces, i.e. The following two quantities related to the spectrum of clean and noisy data, µ r and σ X,r, are defined as follows with r > 1 for the analysis in Theorem 2. Theorem 2. (Subspace detection property holds for noisy 0 -SSC under deterministic model, with conditions in terms of λ) Let nonzero vector β * be the optimal solution to the noisy 0 -SSC problem for point x i with β * 0 = r *, n k ≥ d k + 1 for every 1 ≤ k ≤ K, and there exists 1 < r 0 ≤ d such that 1 < r * ≤ r 0. Suppose Y is in general position, y i ∈ S k for some 1 ≤ k ≤ K, δ < min 1≤r<r0σY,r, and and Then if where λ 0 max{λ 1, λ 2} and λ1 inf{0 < λ < 1 : the subspace detection property holds for x i with β * . Here M i, µ r0 and σ X,r0 are defined in, and respectively. Remark 3. The two conditions and and hold, λ 1 and λ 2 can always be chosen in accordance with and. Remark 4. It can be observed from condition that noisy 0 -SSC encourages sparse solution by a relatively large λ so as to guarantee the subspace detection property. This theoretical finding is consistent with the empirical study shown in the experimental . In this subsection, the correctness of noisy 0 -SSC is analyzed when the clean data in each subspace are distributed at random. We assume that the data in subspace S (k) are i.i.d. isotropic samples on sphere of radius √ d k centered at the origin according to some continuous distribution, for In addition, for each 1 ≤ k ≤ K, we assume that the following condition holds: (a) There exists a constant M ≥ 1 such that for any t > 0, any y ∈ Y (k), and any vector v with unit 2 -norm, Intuitively, condition (a) requires that the projection of any data point onto arbitrary unit vector is bounded from both sides with relatively large probability. This condition is also required in to derive lower bound for the least singular value of a random matrix with independent isotropic columns. In order to meet the conditions in Theorem 2 so as to guarantee the subspace detection property under randomized models, the following lemma is presented and it provides the geometric concentration inequality for the distance between a point y ∈ Y (k) and any of its external subspaces. It renders a lower bound for M i, namely the minimum distance between y i ∈ S k and its external subspaces. Lemma 4. Under randomized models, given 1 ≤ k ≤ K and y ∈ Y (k), suppose H ∈ H yi,d k is any external subspace of y. Then for any t > 0, We then have the following regarding to the subspace detection property of noisy 0 -SSC under randomized models. 0 -SSC under randomized models, with conditions in terms of λ) Under randomized models, let nonzero vector β * be the optimal solution to the noisy 0 -SSC problem for point x i with β * 0 = r *, n k ≥ d k + 1 for every 1 ≤ k ≤ K, and there exists 1 < r 0 ≤ d such that 1 < r * ≤ r 0. Suppose the data in each subspace are i.i.d. isotropic samples according to some continuous distribution that satisfies condition (a).. For t > 0 such that 1 − 2t and where λ 0 max{λ 1, λ 2} and λ 1 inf{0 < λ < 1 : Then with probability at least 2), the subspace detection property holds for x i with β *. Remark 5. Note that there is no assumption on the distribution of subspaces in Theorem 3, so it is not required that the subspaces should have uniform distribution, an is required in the geometric analysis of 1 - and its noisy version. In addition, while; 4 NOISY 0 -SSC ON DIMENSIONALITY REDUCED DATA: NOISY-DR-0 -SSC Albeit the theoretical guarantee and compelling empirical performance of noisy 0 -SSC to be shown in the experimental , the computational cost of noisy 0 -SSC is high with the high dimensionality of the data. In this section, we propose Noisy Dimensionality Reduced 0 -SSC (Noisy-DR-0 -SSC) which performs noisy 0 -SSC on dimensionality reduced data. The theoretical guarantee on the correctness of Noisy-DR-0 -SSC under deterministic model as well as its empirical performance are presented. Noisy-DR-0 -SSC performs subspace clustering by the following two steps: 1) obtain the dimension reduced dataX = PX with a linear transformation P ∈ R p×d (p < d). 2) perform noisy 0 -SSC on the compressed dataX: If p < d, Noisy-DR-0 -SSC operates on the compressed dataX rather than on the original data, so that the efficiency is improved. High-dimensional data often exhibits low-dimensional structures, which often leads to low-rankness of the data matrix. Intuitively, if the data is low rank, then it could be safe to perform noisy 0 -SSC on its dimensionality reduced version by the linear projection P, and it is expected that P can preserve the information of the subspaces contained in the original data as much as possible, while effectively removing uninformative dimensions. To this end, we propose to choose P as a random projection induced by randomized low-rank approximation of the data. The key idea is to obtain an approximate low-rank decomposition of the data. Using the random projection induced by such low-rank approximation as the linear transformation P, the clustering correctness hold for Noisy-DR-0 -SSC with a high probability. Randomized algorithms are efficient and they have been extensively studied in the computer science and numerical linear algebra literature. They have been employed to accelerate various numerical matrix computation and matrix optimization problems, including random projection for matrix decomposition Formally, a random matrix T ∈ R n×p is generated such that each element T ij is sampled independently according to the Gaussian distribution N. QR decomposition is then performed on XT to obtain the basis of its column space, namely XT = QR where Q ∈ R d×p is an orthogonal matrix of rank p and R ∈ R p×p is an upper triangle matrix. The columns of Q form the orthogonal basis for the sample matrix XT. An approximation of X is then obtained by projecting X onto the column space of XT: QQ X = QW =X where W = Q X ∈ R p×n. In this manner, a randomized low-rank decomposition of X is achieved as follows: We present probabilistic on the correctness of Noisy-DR-0 -SSC using the random projection induced by randomized low-rank decomposition of the data X, namely P = Q, in Theorem 4. In the sequel,x = Px for any x ∈ R n. To guarantee the subspace detection property on the dimensionality-reduced dataX, it is crucial to make sure that the conditions, such as and in Theorem 2, still hold after the linear transformation. We denote byβ * the optimal solution to. We also define the following quantities in the analysis of the subspace detection property, which correspond to M i,σ Y,r, σ X,r and µ r used in the analysis on the original data: where Hỹ is all the external subspaces ofỹ i with dimension no greater thand k in the transformed space by P.σỸ Theorem 4. (Subspace detection property holds for Noisy-DR-0 -SSC under deterministic model) Let nonzero vector β * be the optimal solution to the noisy 0 -SSC problem for point x i with β * 0 = r *, n k ≥ d k + 1 for every 1 ≤ k ≤ K, and there exists 1 < r 0 ≤ d such that 1 < r * ≤ r 0. Suppose Y is in general position, δ < min 1≤r<r0σY,r, andM i,δ M i − δ. Suppose the following conditions hold: for all y i ∈ S k and 1 ≤ k ≤ K. then with probability at least 1 − 6e −p, the subspace detection property holds forx i withβ *. Herẽ M i,μ r andσX,r0 are defined in, and respectively. We employ Proximal Gradient Descent (PGD) to optimize the objective function of noisy 0 -SSC and Noisy-DR-0 -SSC. For example, in the k-th iteration of PGD for problem, the variable β is updated according to where g(β) x i − Xβ 2 2, T θ is an element-wise hard thresholding operator: It is proved in that the sequence {β (k) } generated by PGD converges to a critical point of, denoted byβ. Let β * be the optimal solution to. Theorem 5 in to problem shows that the β * −β 2 is bounded. Theorem 5 establishes the conditions under whichβ is also the optimal solution to. The following theorem demonstrates thatβ = β * if λ is two-side bounded andβ min = min t:βt =0 |β t | is sufficiently large. Theorem 5. (Conditions that the sub-optimal solution by PGD is also globally optimal) If and µ thenβ = β *. We demonstrate the performance of noisy 0 -SSC and Noisy-DR-0 -SSC, with comparison to other competing clustering methods including K-means (KM), Spectral Clustering (SC), noisy SSC, Sparse Manifold Clustering and Embedding (SMCE) and. With the coefficient matrix Z obtained by the optimization of noisy 0 -SSC or Noisy-DR-0 -SSC, a sparse similarity matrix is built by W =, and spectral clustering is performed on W to obtain the clustering . Two measures are used to evaluate the performance of different clustering methods, i.e. the Accuracy (AC) and the Normalized Mutual Information (NMI). We use randomized rank-p decomposition of the data matrix in Noisy-DR-0 -SSC with p = min{d,n} 10. It can be observed that noisy 0 -SSC and Noisy-DR-0 -SSC always achieve better performance than other methods in Table 1, including the noisy SSC on dimensionality reduced data (Noisy DR-SSC). Throughout all the experiments we find that the best clustering accuracy is achieved whenever λ is chosen by 0.5 < λ < 0.95, justifying our theoretical finding claimed in Remark 4 and in Theorem 5. More experimental on the CMU Multi-PIE data are shown in Table 2. For all the methods that involve random projection, we conduct the experiments for 30 times and report the average performance. Note that the cluster accuracy of SSC-OMP on the extended Yale-B data set is reported according to. The time complexity of running PGD for noisy 0 -SSC and Noisy-DR-0 -SSC are O(T nd) and O(T pd) respectively, where T is the maximum iteration number. The actual running time of both algorithms confirms such time complexity, and we observe that Noisy-DR-0 -SSC is always more than 8.7 times faster than noisy 0 -SSC with the same number of iterations. We present provable noisy 0 -SSC that recovers subspaces from noisy data through 0 -induced sparsity in a robust manner, with the theoretical guarantee on its correctness in terms of subspace detection property under both deterministic and randomized models. Experimental shows the superior performance of noisy 0 -SSC. We also propose Noisy-DR-0 -SSC which performs noisy 0 -SSC on dimensionality reduced data and still provably recovers the subspaces in the original data. Experiment demonstrate the effectiveness of both noisy 0 -SSC and Noisy-DR-0 -SSC. β = 0. Perform the above analysis for all 1 ≤ i ≤ n, we can prove that the subspace detection property holds for all 1 ≤ i ≤ n. The following proposition is used for proving Lemma 1. Lemma B. (Perturbation of distance to subspaces) Let A, B ∈ R m×n are two matrices and rank(A) = r, rank(B) = s. Also, E = A − B and E 2 ≤ C, where · 2 indicates the spectral norm. Then for any point x ∈ R m, the difference of the distance of x to the column space of A and B, i.e. |d(x, Proof. Note that the projection of x onto the subspace H A is AA + x where A + is the Moore-Penrose pseudo-inverse of the matrix A, so d(x, H A) equals to the distance between x and its projection, namely It follows that According to the perturbation bound on the orthogonal projection in; , Since EA, combining and So that is proved. Proof of Lemma 1. We have yi = xi − ni, and σmin(It follows that σmin(X β) ≥ σ Y,r − √ rδ > 0 and X β has full column rank. Also, X β − Y β 2 ≤ X β − Y β F ≤ √ rδ. According to Lemma B, A.3 PROOF OF LEMMA 2 Proof. xi − Xβ * 2 2 + λ β * 0 ≤ xi − X0 2 2 + λ 0 0 = 1 ⇒ c * = xi − Xβ * 2 < 1. We first prove that β * is the optimal solution to the sparse approximation problem min β β 0 s.t. xi − Xβ 2 ≤ c *, βi = 0. To see this, suppose there is a vector β such that xi − Xβ 2 ≤ c * and β 0 < β that y − xi 2 ≤ c0 since c0 ≥ d(xi, S k). Also, d k points in Y (k) can linearly represent y since Y (k) is in general position, and it follows that β and V A have orthonormal columns with U A U A = V A V A = I. Then QA = U QA ΣV QA is the singular value decomposition of QA with U QA = QU A and V QA = V A. This is because the columns of U QA are orthonormal since the columns Q are orthonormal: U QA U QA = U A Q QU A = I, and Σ is a diagonal matrix with nonnegative diagonal elements. It follows that σmin(QA) = σmin(A) for any A ∈ R p×q. For a point xi = yi + ni, after projection via P, we have the projected noiseñi = Pni. Because the magnitude of the noise in the projected data is also bounded by δ. Also, Let β ∈ R n,Ỹ β = PY β with β 0 = r. Then σmin(QỸ β) = σmin(Ỹ β)). Since Therefore, it follows from that if thenỸ is also in general position. In addition, since λ ≥ 1 r 0, we have λ β * 0 ≤ L ≤ 1, and it follows that β * 0 ≤ 1 λ ≤ r0. Based on we have |σỸ,r −σ Y,r | ≤ Cp,p 0 + 2δ it follows that δ < min 1≤r<r 0σỸ,r because δ < min 1≤r<r 0σ Y,r − Cp,p 0 − 2δ √ r0. Again, for β ∈ R n with β 0 = r ≤ r0, we have |σmin(X β) − σmin(X β)| = |σmin(QX β) − σmin(X β)| It can be verified that |σX,r − σ X,r | ≤ Cp,p 0 Combining and Lemma D, noting that σ X,r 0 − Cp,p 0, since we haveM i,δ where yi ∈ S k. Based on and, we haveμ

Represent the TLDR for finding the corresponding scientific article

We propose Noisy-DR-L0-SSC (Noisy Dimension Reduction L0-Sparse Subspace Clustering) to efficiently partition noisy data in accordance to their underlying subspace structure.

Represent the short summary for finding the corresponding scientific article (long).

To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.